Observability dan Alerting — OS-v1.0
Last verified: 2026-09-12. Target specification; alerts/probes belum disertifikasi. Owner: platform/operations, OS-011/072. Baseline.
1. Probe dan identity
Existing /api/v1/health dan /api/v1/health/metrics perlu diverifikasi terhadap implementasi. Pisahkan liveness (process), readiness (required dependencies) dan operational metrics. Health publik minimal; metrics/topology/identity sensitif memerlukan operator authorization.
JSON health/metrics bukan otomatis Prometheus exposition. OS-011 memilih exporter/format atau collector yang tepat dan contract tests; jangan menyetel scraper Prometheus ke JSON endpoint tanpa adapter. Tidak ada Helm/Kubernetes requirement v1.
Release manifest/probe mengikat source tree/FE build/BE runtime/migration/launcher/SEB config/policy. Jangan mengungkap runtime path, secrets atau full topology ke peserta.
2. Metrics dan alert ownership
| Area | Metrik minimum | Severity/tindakan |
|---|---|---|
| Answers | p95/p99 latency, accepted/conflict/rejected/timeout, retry age, ACK reconciliation | SLO breach → technical lead; ACK loss → SEV-1 |
| Attempts | active by lifecycle, start/submit bursts, overdue active, duplicate receipt/revision violations | Integrity anomaly → SEV-1; do not auto-adjust deadline |
| Identity/secure | auth/MFA/revocation failures, direct access/replay/version failures | Security triage; rate limits tidak membatalkan valid ACK |
| Presence | heartbeat age, offline/reconnect, room fan-out lag/rebuild | Proctor operational attention; bukan cheating verdict |
| Camera/evidence | readiness fail reason, capture/upload accepted/expired/failed, bytes/queue age, access denials | Storage failure alert; bounded retries; separate answer path |
| Audit/outbox | commit failure, oldest undelivered, retry/dead-letter, dedupe | Critical audit failure fail transaction; delivery retry alert |
| Database/storage | pool/locks/slow query/CPU/IO/disk, Redis, object-store failures | Investigate under approved budget, jangan menaikkan pool otomatis |
| Backups/retention | base/WAL age, recoverable point, failed expiry/deletion, holds, restore/tombstone validation | RPO risk dan privacy control failure → assigned owner |
| Scoring/results | pending grade/review, revisions, snapshot/publish failures | Grading lead; no publish bypass |
Warning thresholds/durations, monitoring freshness, retry budgets dan escalation contact di OP-01/DEC-11. Autosave SLO target di capacity document; alert calibration harus diuji agar request volume rendah tidak menghasilkan angka rate menyesatkan. Jangan mengubah policy atau resource otomatis tanpa approved procedure.
3. Logs dan dashboards
Structured correlation across HTTP/job/audit. Redact token/password/answer text/image bytes/participant PII/Zoom passcodes; jangan gunakan attempt IDs sebagai label metrics cardinality tak terbatas. Safe identifiers dapat berada pada restricted diagnostic log sesuai policy.
Dashboards role-scoped; grader tidak bisa membuka identity/camera melalui observability. Operator runbooks diuji lewat alert injection; setiap alert punya owner/severity/action dan timestamped evidence. Availability telemetry sendiri tidak membuktikan integrity/capacity.
4. Verification
OS-072 menunjukkan simulated alert → routing owner → safe action → recovery, termasuk storage outage dan stale deadline sweep. I7 mengikat hasil observability ke exact deployed candidate; tidak melaporkan “monitoring aktif” hanya dari keberadaan file konfigurasi.