SOP-080 — Incident response & SLO/SLA monitoring

SOP · owner: Operations (Site Reliability Engineer); SOP-system compliance reviewed by the Process Engineer. Layer: SOP/Runbook (docs/canon.md). Two procedures, one Interface Contract: (A) declare and command a P1/P2 incident, and (B) review SLO/SLA compliance on a cadence. Stub-first until platform go-live — the intranet has no runtime to defend; the live learning platform is what this protects (data layer deferred, ADR-0006).

Interface Contract

• Inputs: manifest verification result ← from Archivist · trigger: manifest verified
• Outputs: integrity/verification result logged → to self · trigger: verification recorded
• Outputs: approved incident SOP / runbook / post-mortem → to Archivist · trigger: artifact approved
• Handoff trigger (in): the Archivist reports a manifest verification result (the live integrity seam today)
• Handoff trigger (out): the integrity result is recorded against the reliability log
• Definition of Done before handoff:
  • [ ] verification result (0 drift / drift) recorded with timestamp
  • [ ] for an incident: record opened, owners notified, timeline owned to resolution
  • [ ] post-mortem blameless (no individual named), ≤48h, action items have named owners
  • [ ] Process-Engineer-reviewed for SOP-system compliance; archived (0 drift)
  • [ ] learnings captured to docs/knowledge/ if the incident produced one

Pending / Wave-4 seams (correct, not defects). Two real seams have no registry trigger yet and are proposed to Operations (see the SRE Phase 0 report), not invented inline: (1) incident-declaration → Customer Success Director (learner comms) — CS is Wave 4, so the edge reconciles pending (reconciler R3) until then; (2) SLO-monitoring-requirement → Engineering Director (telemetry implementation).

Purpose

Run a clean, decisive incident command so nothing falls through the seams, and review platform availability against defined SLO/SLA targets so breaches surface immediately, not at the next report cycle.

When to use

(A) A platform incident is detected or reported. (B) On a recurring cadence (monthly, or after any P1/P2) to review SLO/SLA compliance.

Flow — (A) incident command

flowchart LR
  D["Detect / report"] --> C{"Classify"}
  C -->|P3| E["Engineering task<br/>(no command)"]
  C -->|P1/P2| O["Open record<br/>+ notify owners"]
  O --> Cmd["Command<br/>own timeline"]
  Cmd --> R["Resolve<br/>(fix live + verified)"]
  R --> PM["Blameless post-mortem<br/>≤48h"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  classDef d fill:#0f766e,color:#fff,stroke:#134e4a
  class D,E,O,Cmd,R,PM s
  class C d

Text fallback: detect/report → classify (P3 = standard engineering task, no command; P1/P2 = command) → open the incident record and notify owners (Customer Success owns learner comms, Engineering the fix, Security any security thread) → hold command and own the timeline → resolve once the fix is live and verified → author a blameless post-mortem within 48 hours.

Steps — (A) incident command

1. Classify severity (on-call runbook criteria): P1 = full outage / data-affecting; P2 = significant degradation; P3 = minor / cosmetic (no command — engineering task).
2. Open the record immediately: reports/incident-<YYYY-MM-DD>-<slug>.md — declaration time, severity, known scope, first-responder, comms log.
3. Notify owners per the on-call runbook: Customer Success Director (learner comms + status cadence), Engineering Director (diagnosis/fix), Security Director (if any security dimension — hand off the investigation thread).
4. Hold command. Maintain the timeline; drive to resolution without crossing into fix implementation or security investigation. Escalate to Process Engineer (and Principal) if the incident cannot resolve within its severity-class SLA.
5. Declare resolution once Engineering confirms the fix is live and verified; notify Customer Success to close learner comms; record resolution time and final scope.
6. Hand the record to the Archivist within 24h; author a blameless post-mortem (reports/postmortem-<YYYY-MM-DD>-<slug>.md) within 48h with named action-item owners.

Steps — (B) SLO/SLA monitoring

1. Define targets (availability %, MTTR by severity, MTBF) in docs/sop/platform-sla.md (stub-first until go-live); specify monitoring requirements to the Engineering Director (implementation is theirs).
2. On cadence, pull availability/MTTR data; compare against targets.
3. Produce reports/sla-<YYYY-MM>.md (actual vs target, breach count, trend). Surface any breach immediately to the Process Engineer and Principal — not at the next cycle.
4. Hand the report to the Archivist.

Roles / approvers

Guardrails

• Command, don’t fix. Code fixes are Engineering’s; security investigation is Security’s; learner messages are Customer Success’s. SRE coordinates and owns the timeline.
• Blameless is non-negotiable. Post-mortems name contributing factors and process gaps, never individuals.
• A runbook never walked through is not a runbook — rehearse against dry-runs pre-traffic.
• No personal information (PA-only). No fabricated readiness — keep platform runbooks stub-first until go-live; do not over-build pre-traffic (GR-4).

See also: .claude/agents/site-reliability-engineer.md (JD) · docs/units/operations/manuals/site-reliability-engineer.md · docs/units/operations/job-aids/JA-080-incident-runbook-quickref.md · interface-contract-spec.md §4 (the manifest verified seam) · docs/operations.md.