Lewati ke isi

SeaweedFS private object-storage runbook

Status: operational baseline for CBT dev and staging, 2026-09-13. This is not production certification, a DR certification, or authorization to admit real participants.

Scope and security boundary

Camera evidence is held in a private S3-compatible SeaweedFS deployment. The database holds the object key, checksum, metadata and lifecycle state; it must not hold evidence binary content. No SeaweedFS endpoint is published through Nginx.

Each environment has its own single-node master, volume, filer and S3 gateway, all bound to 127.0.0.1. The CBT API reaches its matching S3 endpoint over loopback only. The application identity is restricted to explicit S3 actions; anonymous S3 access is not configured. The S3 gateway permits CORS only from that environment's CBT origin.

Environment API Seaweed S3 Master Volume Filer Metrics
staging 8010 9001 9333 9080 8888 9324
dev 8011 9003 9334 9081 8889 9325

The systemd units are cbt-<environment>-seaweedfs.service. API units depend on their matching SeaweedFS unit, not on the retired S3Proxy unit.

Version and installation provenance

The installed binary is SeaweedFS 4.46, downloaded from the official SeaweedFS GitHub release as linux_amd64.tar.gz. Its release asset SHA-256 is f4c654a72353c36bd8ae03c8879b81588dc29ffee0fa87594d8716a3d9347ad5.

Before an upgrade, use the official release page to record the selected version, asset checksum, release notes and compatibility assessment. Verify the downloaded checksum before replacing /usr/local/bin/weed; preserve the previous binary and do not combine an untested binary upgrade with an event. Run the verification sequence below first in staging, then create a separate immutable CBT certification candidate before any production authorization.

Configuration and secret handling

The following files are host-local operational configuration, not repository source and not evidence attachments:

Path Required protection Purpose
/home/scola/cbt-<environment>/config/secrets.env 0600 S3 credential and WEED_* secret values
/home/scola/cbt-<environment>/config/security.toml 0600 non-secret Seaweed security policy
/home/scola/cbt-<environment>/config/seaweed-s3.json 0600 S3 identity/action policy
/home/scola/cbt-<environment>/seaweed-data 0700 volume data
/home/scola/cbt-<environment>/seaweed-meta 0700 master/filer metadata

Use the environment-variable names emitted by the official weed scaffold -config=security command: WEED_JWT_SIGNING_KEY, WEED_JWT_SIGNING_READ_KEY, WEED_JWT_FILER_SIGNING_KEY, WEED_JWT_FILER_SIGNING_READ_KEY, WEED_S3_SSE_KEK, and WEED_S3_SSE_KEK_PASSPHRASE. The KEK is a unique, randomly generated 256-bit value per environment. The passphrase protects the stored KEK. Never place either value in source control, logs, a shell history, manifests or a test artifact.

The KEK and passphrase are recovery-critical: loss or replacement can make previous SSE-S3 evidence unrecoverable. Restore them only from the approved secret backup together with the corresponding data and metadata backup.

The CBT application has S3_SERVER_SIDE_ENCRYPTION=AES256. This is fail-closed in production configuration; it may not be cleared to work around an S3 compatibility issue.

Bootstrap and service operations

Do not expose the S3, filer, master, volume or metrics ports publicly. Do not enable a public UI. The service must use these invariants:

  • -ip=127.0.0.1 and -ip.bind=127.0.0.1;
  • -s3.autoCreateBucket=false and -s3.allowDeleteBucketNotEmpty=false;
  • environment-specific S3 CORS origin only;
  • master telemetry disabled;
  • filer directory metadata and volume UI disabled;
  • a dedicated data and metadata directory per environment.

After a controlled configuration change:

sudo systemctl daemon-reload
sudo systemctl restart cbt-staging-seaweedfs.service
sudo systemctl restart cbt-staging-api.service cbt-staging-celery.service
systemctl is-active cbt-staging-seaweedfs.service cbt-staging-api.service cbt-staging-celery.service

Use the equivalent cbt-dev-* names only for dev. Do not restart the public production CBT service under this procedure.

Required verification

Run every check on staging after an install, upgrade, restore, credential rotation or configuration change, then repeat in dev when applicable.

  1. Confirm the Seaweed S3 listener is only 127.0.0.1:<environment-port> and Nginx has no route to it.
  2. With the application S3 credential, explicitly create the approved bucket if absent; perform a unique put_object with ServerSideEncryption='AES256'; head_object must report AES256; read back the exact bytes and delete the probe.
  3. Verify an unsigned S3 request receives HTTP 401 or 403.
  4. Run the application's put_evidence and delete_evidence round trip, not merely a generic S3 probe.
  5. Restart the API and worker after the storage unit is ready; check https://cbt-<environment>.gcgscola.id/health and inspect the new service journal for traceback, critical, error or exception events.

The 2026-09-13 baseline passed all five checks in dev and staging with SeaweedFS 4.46 and the candidate 4be119a68e81c4903180b9c526d80cfe81bb26c2. This evidence proves the listed smoke checks only; it does not replace R6 load, failure, DR, security-review or UAT evidence.

Rollback and recovery constraints

The former S3Proxy configuration and object roots were archived before this migration under each environment's backups/ directory. They are rollback material only for the empty/simulated pre-migration state; do not use them to overwrite SeaweedFS evidence written after cutover.

For a configuration or binary failure before real evidence is admitted, stop the affected CBT API and worker, preserve a copy of seaweed-data, seaweed-meta and the secret record, restore the previous known-good service definition, and verify the five checks above. A source/service rollback does not decrypt, migrate or restore data.

This single-node baseline is not a disaster-recovery design. R6 still requires an approved off-host encrypted backup target, tested restoration of data, metadata and secrets, evidence tombstone/hold restoration, measured RPO/RTO, and a failure/load campaign. Do not claim PITR, high availability, or hundreds of-student readiness from this installation alone.

References