<!-- last-reviewed: 2026-06-01 · owner: Security & Privacy (Security Director) -->

SOP-083 — Security incident runbook (first response)

SOP / runbook · owner: Security & Privacy (Security Director) (content + policy); Operations (Process Engineer) (operational wiring). Layer: SOP/Runbook (docs/canon.md). The org’s first-response procedure for a suspected or confirmed security/privacy incident. Authored to close the named-overdue docs/sop/security-incident.md deliverable (Phase 0 D1). Policy frame: hr/policies/security.md, SOP-004 (docs/sop/secrets-management.md).

Interface Contract

• Inputs: incident signal (leaked secret / breach / external-compliance failure) ← from <any agent / Security Analyst / monitoring> · trigger: review requested
• Outputs: remediation requirement (the fix bar) → to Engineering Director · trigger: requirements issued
• Outputs: incident report + risk record → to Archivist · trigger: artifact approved
• Outputs: incident posture / runbook change → to Documentarian · trigger: document changed
• Outputs: escalation (active incident / unaccepted-risk decision) → to Principal · trigger: (exempt — Principal edge, interface-contract-spec.md §3.1 R2)
• Handoff trigger (in): a suspected or confirmed security/privacy incident is reported or detected
• Handoff trigger (out): the incident is contained, root-caused, remediation requirements are issued, and the report + risk record are archived
• Definition of Done before handoff:
  • [ ] Severity triaged (SEV-1 … SEV-3 below) and the Principal notified immediately for SEV-1/SEV-2
  • [ ] Containment done first (revoke/rotate/disable) — before root-cause analysis
  • [ ] Remediation requirement issued to the implementing owner (requirements issued)
  • [ ] Dated incident report (timeline · scope · root cause · remediation · residual risk) handed to the Archivist (artifact approved)
  • [ ] Learnings captured to docs/knowledge/security/

Triggers are drawn verbatim from docs/handoff-triggers.md (review requested, requirements issued, artifact approved, document changed). The Principal escalation edge is exempt (Principal) (§3.1 R2). A dedicated incident-detection trigger (incident detected) is proposed to Operations (see below) — until registered, the in-edge reuses review requested.

Purpose

Give the org a single, boring, well-rehearsed first response so that when a secret leaks, a breach is suspected, or an external-compliance control fails, the people involved contain first, analyze second, and document always — rather than improvising under pressure.

Severity triage (set the response speed)

Flow

flowchart LR
  A["Signal:<br/>incident reported/detected"] --> B["Triage SEV<br/>notify Principal (1/2)"]
  B --> C["CONTAIN first<br/>revoke / rotate / disable"]
  C --> D["Root cause<br/>+ scope the blast radius"]
  D --> E["requirements issued<br/>→ Engineering (the fix bar)"]
  E --> F["Incident report + risk record<br/>→ Archivist"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  class A,B,C,D,E,F s

Text fallback: a signal arrives → triage severity and notify the Principal for SEV-1/2 → contain first (revoke the credential at the provider, rotate, or disable the failing path — never just delete the commit) → root-cause and scope the blast radius → issue the remediation requirement to Engineering (requirements issued, the bar, not the code) → file the dated incident report + risk record to the Archivist (artifact approved).

Steps

1. Detect & triage. Classify SEV-1/2/3. For SEV-1/2, notify the Principal immediately (escalation is not held for the next touchpoint — JD reporting rule).
2. Contain first, analyze second.
   • Leaked secret: revoke at the provider, then replace in the managed store (SOP-004 §Guardrails — revoked first, then replaced; do not just delete the commit). Update the secret register date.
   • Failing external-compliance path: disable it (e.g. keep the Slack deletion cron disabled — H2) until the control is fixed.
   • Suspected agent compromise / tool-grant abuse: flag the grant to the Principal (SOP-082); do not let the surface stay trusted.
3. Root-cause + scope. What was exposed, for how long, to whom, via which boundary? Map it against the relevant threat model (SOP-081).
4. Issue remediation requirement. Hand the fix bar (acceptance criterion) to Engineering (requirements issued) — we define what “fixed” means; they write the code. For pipeline/wiring, hand to the Process Engineer.
5. Document + archive. Dated incident report: timeline, scope, root cause, remediation status, residual risk (accepted by whom). Hand to the Archivist (artifact approved); capture the learning to docs/knowledge/security/.
6. Verify closure. The Security Analyst reviews the fix (SOP-084); the control is not closed until the acceptance criterion is met.

Standing live incident on the books — H2 (ADR-0017 Slack ToS)

SEV-2, ACCEPTED (reports/phase0-reviews/coo-cross-cutting-log.md H2). The app_uninstalled handler swallows DB-delete failures and returns 200 (intranet/src/pages/api/slack/events.ts:301-308); ToS-required deletion can silently fail. Containment in force: the deletion cron stays disabled until the channel_category/archived_at migrations + fallback are complete; the CRON_SECRET-gated cleanup-check cron is the partial fallback. Ownership split: the code fix is Engineering’s (the acceptance bar: the deletion failure no longer returns a silent 200 — issued via requirements issued); the policy/risk record and this incident posture are ours; the Security Analyst is the PR-review eyes (SOP-084). This SOP does not close H2 — it holds its containment and acceptance record until remediated.

Roles / approvers

Run by the Security Director (incident commander). The Principal is notified immediately for SEV-1/2 and accepts residual risk on the record (GR-3). Engineering implements the fix; the Process Engineer wires any pipeline/monitoring change; the Security Analyst verifies closure; the Archivist preserves the report.

Proposed triggers (to Operations, not invented inline)

• incident detected (any agent / monitoring → Security) — a security/privacy incident signal opening this runbook. Until registered, reuse review requested; the in-edge reads pending if a distinct phrase is needed. Request via interface-contract-spec.md §1.2.

Guardrails

• Contain before you analyze. A revoked credential and a disabled failing path buy the time to root-cause safely.
• Never just delete the commit for a leaked secret — revoke + rotate (SOP-004).
• Requirements, not code — issue the fix bar; do not write product code.
• No silent risk acceptance — residual risk is named and accepted by the Principal on the record (GR-3).
• Personal information is PA-only — an incident touching personal records routes through the PA; we do not inspect the data.

Job aid: (△ — the SEV-triage table + contain-first sequence above is the runbook checklist; a separate card may be seeded later, GR-4.) See also: docs/units/security-privacy/manuals/security-director.md · SOP-004 · SOP-081 · SOP-084 · hr/policies/security.md · reports/phase0-reviews/coo-cross-cutting-log.md (H2) · reports/phase0-reviews/security-director-codebase-review.md (D1, D7).