Lewati ke isi

Observability dan Alerting — OS-v1.0

Last verified: 2026-09-12. Target specification; alerts/probes belum disertifikasi. Owner: platform/operations, OS-011/072. Baseline.

1. Probe dan identity

Existing /api/v1/health dan /api/v1/health/metrics perlu diverifikasi terhadap implementasi. Pisahkan liveness (process), readiness (required dependencies) dan operational metrics. Health publik minimal; metrics/topology/identity sensitif memerlukan operator authorization.

JSON health/metrics bukan otomatis Prometheus exposition. OS-011 memilih exporter/format atau collector yang tepat dan contract tests; jangan menyetel scraper Prometheus ke JSON endpoint tanpa adapter. Tidak ada Helm/Kubernetes requirement v1.

Release manifest/probe mengikat source tree/FE build/BE runtime/migration/launcher/SEB config/policy. Jangan mengungkap runtime path, secrets atau full topology ke peserta.

2. Metrics dan alert ownership

Area Metrik minimum Severity/tindakan
Answers p95/p99 latency, accepted/conflict/rejected/timeout, retry age, ACK reconciliation SLO breach → technical lead; ACK loss → SEV-1
Attempts active by lifecycle, start/submit bursts, overdue active, duplicate receipt/revision violations Integrity anomaly → SEV-1; do not auto-adjust deadline
Identity/secure auth/MFA/revocation failures, direct access/replay/version failures Security triage; rate limits tidak membatalkan valid ACK
Presence heartbeat age, offline/reconnect, room fan-out lag/rebuild Proctor operational attention; bukan cheating verdict
Camera/evidence readiness fail reason, capture/upload accepted/expired/failed, bytes/queue age, access denials Storage failure alert; bounded retries; separate answer path
Audit/outbox commit failure, oldest undelivered, retry/dead-letter, dedupe Critical audit failure fail transaction; delivery retry alert
Database/storage pool/locks/slow query/CPU/IO/disk, Redis, object-store failures Investigate under approved budget, jangan menaikkan pool otomatis
Backups/retention base/WAL age, recoverable point, failed expiry/deletion, holds, restore/tombstone validation RPO risk dan privacy control failure → assigned owner
Scoring/results pending grade/review, revisions, snapshot/publish failures Grading lead; no publish bypass

Warning thresholds/durations, monitoring freshness, retry budgets dan escalation contact di OP-01/DEC-11. Autosave SLO target di capacity document; alert calibration harus diuji agar request volume rendah tidak menghasilkan angka rate menyesatkan. Jangan mengubah policy atau resource otomatis tanpa approved procedure.

3. Logs dan dashboards

Structured correlation across HTTP/job/audit. Redact token/password/answer text/image bytes/participant PII/Zoom passcodes; jangan gunakan attempt IDs sebagai label metrics cardinality tak terbatas. Safe identifiers dapat berada pada restricted diagnostic log sesuai policy.

Dashboards role-scoped; grader tidak bisa membuka identity/camera melalui observability. Operator runbooks diuji lewat alert injection; setiap alert punya owner/severity/action dan timestamped evidence. Availability telemetry sendiri tidak membuktikan integrity/capacity.

4. Verification

OS-072 menunjukkan simulated alert → routing owner → safe action → recovery, termasuk storage outage dan stale deadline sweep. I7 mengikat hasil observability ke exact deployed candidate; tidak melaporkan “monitoring aktif” hanya dari keberadaan file konfigurasi.