SeaweedFS private object-storage runbook
Status: operational baseline for CBT dev and staging, 2026-09-13. This is not production certification, a DR certification, or authorization to admit real participants.
Scope and security boundary
Camera evidence is held in a private S3-compatible SeaweedFS deployment. The database holds the object key, checksum, metadata and lifecycle state; it must not hold evidence binary content. No SeaweedFS endpoint is published through Nginx.
Each environment has its own single-node master, volume, filer and S3 gateway,
all bound to 127.0.0.1. The CBT API reaches its matching S3 endpoint over
loopback only. The application identity is restricted to explicit S3 actions;
anonymous S3 access is not configured. The S3 gateway permits CORS only from
that environment's CBT origin.
| Environment | API | Seaweed S3 | Master | Volume | Filer | Metrics |
|---|---|---|---|---|---|---|
| staging | 8010 | 9001 | 9333 | 9080 | 8888 | 9324 |
| dev | 8011 | 9003 | 9334 | 9081 | 8889 | 9325 |
The systemd units are cbt-<environment>-seaweedfs.service. API units depend
on their matching SeaweedFS unit, not on the retired S3Proxy unit.
Version and installation provenance
The installed binary is SeaweedFS 4.46, downloaded from the official
SeaweedFS GitHub release as linux_amd64.tar.gz. Its release asset SHA-256 is
f4c654a72353c36bd8ae03c8879b81588dc29ffee0fa87594d8716a3d9347ad5.
Before an upgrade, use the official release page to record the selected
version, asset checksum, release notes and compatibility assessment. Verify
the downloaded checksum before replacing /usr/local/bin/weed; preserve the
previous binary and do not combine an untested binary upgrade with an event.
Run the verification sequence below first in staging, then create a separate
immutable CBT certification candidate before any production authorization.
Configuration and secret handling
The following files are host-local operational configuration, not repository source and not evidence attachments:
| Path | Required protection | Purpose |
|---|---|---|
/home/scola/cbt-<environment>/config/secrets.env |
0600 |
S3 credential and WEED_* secret values |
/home/scola/cbt-<environment>/config/security.toml |
0600 |
non-secret Seaweed security policy |
/home/scola/cbt-<environment>/config/seaweed-s3.json |
0600 |
S3 identity/action policy |
/home/scola/cbt-<environment>/seaweed-data |
0700 |
volume data |
/home/scola/cbt-<environment>/seaweed-meta |
0700 |
master/filer metadata |
Use the environment-variable names emitted by the official weed scaffold
-config=security command: WEED_JWT_SIGNING_KEY,
WEED_JWT_SIGNING_READ_KEY, WEED_JWT_FILER_SIGNING_KEY,
WEED_JWT_FILER_SIGNING_READ_KEY, WEED_S3_SSE_KEK, and
WEED_S3_SSE_KEK_PASSPHRASE. The KEK is a unique, randomly generated
256-bit value per environment. The passphrase protects the stored KEK. Never
place either value in source control, logs, a shell history, manifests or a
test artifact.
The KEK and passphrase are recovery-critical: loss or replacement can make previous SSE-S3 evidence unrecoverable. Restore them only from the approved secret backup together with the corresponding data and metadata backup.
The CBT application has S3_SERVER_SIDE_ENCRYPTION=AES256. This is
fail-closed in production configuration; it may not be cleared to work around
an S3 compatibility issue.
Bootstrap and service operations
Do not expose the S3, filer, master, volume or metrics ports publicly. Do not enable a public UI. The service must use these invariants:
-ip=127.0.0.1and-ip.bind=127.0.0.1;-s3.autoCreateBucket=falseand-s3.allowDeleteBucketNotEmpty=false;- environment-specific S3 CORS origin only;
- master telemetry disabled;
- filer directory metadata and volume UI disabled;
- a dedicated data and metadata directory per environment.
After a controlled configuration change:
sudo systemctl daemon-reload
sudo systemctl restart cbt-staging-seaweedfs.service
sudo systemctl restart cbt-staging-api.service cbt-staging-celery.service
systemctl is-active cbt-staging-seaweedfs.service cbt-staging-api.service cbt-staging-celery.service
Use the equivalent cbt-dev-* names only for dev. Do not restart the public
production CBT service under this procedure.
Required verification
Run every check on staging after an install, upgrade, restore, credential rotation or configuration change, then repeat in dev when applicable.
- Confirm the Seaweed S3 listener is only
127.0.0.1:<environment-port>and Nginx has no route to it. - With the application S3 credential, explicitly create the approved bucket
if absent; perform a unique
put_objectwithServerSideEncryption='AES256';head_objectmust reportAES256; read back the exact bytes and delete the probe. - Verify an unsigned S3 request receives HTTP 401 or 403.
- Run the application's
put_evidenceanddelete_evidenceround trip, not merely a generic S3 probe. - Restart the API and worker after the storage unit is ready; check
https://cbt-<environment>.gcgscola.id/healthand inspect the new service journal for traceback, critical, error or exception events.
The 2026-09-13 baseline passed all five checks in dev and staging with
SeaweedFS 4.46 and the candidate 4be119a68e81c4903180b9c526d80cfe81bb26c2.
This evidence proves the listed smoke checks only; it does not replace R6 load,
failure, DR, security-review or UAT evidence.
Rollback and recovery constraints
The former S3Proxy configuration and object roots were archived before this
migration under each environment's backups/ directory. They are rollback
material only for the empty/simulated pre-migration state; do not use them to
overwrite SeaweedFS evidence written after cutover.
For a configuration or binary failure before real evidence is admitted, stop
the affected CBT API and worker, preserve a copy of seaweed-data,
seaweed-meta and the secret record, restore the previous known-good service
definition, and verify the five checks above. A source/service rollback does
not decrypt, migrate or restore data.
This single-node baseline is not a disaster-recovery design. R6 still requires an approved off-host encrypted backup target, tested restoration of data, metadata and secrets, evidence tombstone/hold restoration, measured RPO/RTO, and a failure/load campaign. Do not claim PITR, high availability, or hundreds of-student readiness from this installation alone.
References
- SeaweedFS official installation and S3 documentation: https://github.com/seaweedfs/seaweedfs
- Official release
4.46: https://github.com/seaweedfs/seaweedfs/releases/tag/4.46 - CBT disaster-recovery runbook