SOP-087 — Four-test stratified-sample procedure (anti-theater gate)

SOP · owner: Quality & Risk. Layer: SOP/Runbook (docs/canon.md). The independent anti-theater gate for the documentation program (master plan §1.3, R1/R11). Fixes the sample frame · size · stratification rule so the sample is a repeatable procedure, not a recollection — this closes prior Q&R finding G-2 (“‘stratified’ was asserted but the frame/size/stratification were undefined”, 2026-06-01-wave1-qr-sample.md §1).

Interface Contract

• Inputs: a completed wave / artifact corpus for assurance ← from <authoring agents/Leads of the wave> · trigger: review requested
• Outputs: four-test sample verdict + findings → to <wave owners / COO> · trigger: review passed
• Outputs: approved sample report → to Archivist · trigger: artifact approved
• Handoff trigger (in): a wave (or named artifact set) is declared complete and handed for the independent four-test sample
• Handoff trigger (out): the verdict (Pass / Pass with conditions / Does not pass) + executability rate + falsifiability defects + N/A-sample result are issued to the wave owners
• Definition of Done before handoff:
  • [ ] Sample frame enumerated (the wave’s full deliverable set)
  • [ ] Size computed (≥20% of the named deliverables, min the cell-coverage floor below)
  • [ ] Stratification hit on all three dimensions (every cell sampled); members oversampled
  • [ ] Each sampled artifact assessed on all four tests (executable · non-redundant · falsifiable · interlocking) with citations
  • [ ] A sample of N/A justifications assessed (R11) — not only job-aid executability
  • [ ] Both scripted gates confirmed (executed, or logic-traced with the limit disclosed)
  • [ ] Executability pass rate reported against the ≥90% target (master plan §10)
  • [ ] One verdict; findings routed to owners; report handed to Archivist (artifact approved)

Purpose

Provide the program’s only independent proof besides the scripted reconciliation gate that a wave is real work, not theater (R1) — applied on a stratified sample, never by self-attestation (R11). Director approval is recorded but is first-line reviewing first-line; this sample is the independent check.

When to use

Run once per wave checkpoint (the wave is declared complete) and on any artifact set explicitly handed for a four-test review. The cadence is in reports/audit-schedule.md.

The sample-design rule (frame · size · stratification — the G-2 fix)

Frame (the population)

Enumerate the wave’s full named deliverable set before sampling: every unit charter / operating-model, every role manual, every SOP (each carrying an ## Interface Contract), every job aid, every playbook seeded this wave, the Phase-0 reviews, the wave’s knowledge entries, and the mechanism artifacts touched this wave (docs/handoff-triggers.md block, docs/sop/registry.md, docs/handoff-register.md, scripts/handoff-reconcile.mjs, scripts/km-index-check.mjs). The frame is written into the report so the sample is reproducible.

Size

≥ 20% of the named deliverable set, and never below the cell-coverage floor below (whichever is larger). A stratum where a defect appears is oversampled (sampling may expand a stratum on a hit — Wave-1 precedent). Record the count and the % in the report.

Stratification (three dimensions — every cell sampled)

Chosen so a thin-member or a single-unit failure cannot hide:

Cell-coverage floor (the minimum, regardless of the 20%):

• every unit’s charter / operating-model in the wave,
• ≥1 Lead SOP and ≥1 member SOP,
• ≥1 job aid per altitude that has one,
• every wave knowledge entry,
• the two gate scripts read line-by-line,
• ≥1 sampled N/A justification per unit that declares one (R11).

The four tests (applied to each sampled artifact)

1. Executable — a competent stranger / fresh instance could do the job from the job aid without the incumbent. Report the executability pass rate (target ≥90%, master plan §10).
2. Non-redundant — links to canon rather than restating it. Copy-pasted policy/rule text is a defect, not coverage.
3. Falsifiable — named files, exact triggers, exact commands; specific enough to be wrong. (The Wave-1 F-1 stale-number defect is the archetype this test catches.)
4. Interlocking — Interface-Contract inputs/outputs name real counterparties who agree (verbatim trigger match, mirrored direction). Trace the wave’s high-risk seams by hand — do not defer to the GREEN reconciliation, because the gate has a fan-in blind spot (risk R-003): a deleted fan-in producer reads GREEN.

Flow

flowchart LR
  A["1. Enumerate the frame"] --> B["2. Compute size (≥20% + floor)"]
  B --> C["3. Allocate to every cell<br/>(oversample members)"]
  C --> D["4. Apply the 4 tests<br/>+ cite"]
  D --> E["5. Sample N/A justifications (R11)"]
  E --> F["6. Confirm both gates<br/>(execute or trace + disclose)"]
  F --> G["7. Verdict + route + Archivist"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  class A,B,C,D,E,F,G s

Text fallback: enumerate the frame → compute size (≥20% and the cell floor) → allocate the sample so every stratification cell is hit, oversampling members → apply the four tests to each sampled artifact with citations → sample the N/A justifications (R11) → confirm both scripted gates (execute them, or logic-trace with the limit disclosed and ask Operations/PMO to machine-attest) → issue one verdict, route findings to owners, hand the report to the Archivist.

Steps

1. Enumerate the frame in the report (the wave’s full deliverable set).
2. Compute the size — max(20% of the frame, the cell-coverage floor); record count + %.
3. Allocate the sample to hit every cell of all three dimensions; oversample member-authored artifacts; list each sampled artifact with its (altitude, author line, unit) cell.
4. Apply the four tests to each sampled artifact, citing file:line for each verdict. Compute the executability pass rate.
5. Sample N/A justifications (R11). Verify each holds against the authoritative artifact-matrix.md — a disguised empty/un-done slot is a defect.
6. Confirm both scripted gates. Run node scripts/handoff-reconcile.mjs and node scripts/km-index-check.mjs if a shell is available; otherwise logic-trace them against source + spec, label the result a trace not an execution (risk R-006), and request Operations/PMO paste literal stdout (incl. a RED break-test) into the checkpoint record. Include the falsifiable break-test class (a trigger mutation turns the gate RED; note the fan-in producer-deletion that does not, R-003).
7. Issue one verdict — Pass / Pass with conditions / Does not pass — with the executability rate and the defect list.
8. Route + archive. Findings to owners (proposals, not directives); open gaps to the control-gap registry (SOP-086); emit review passed; hand the report to the Archivist (artifact approved).

Roles / approvers

• Q&R Director designs and runs the sample and issues the verdict (sole, independent reviewer).
• Wave owners / Leads act on findings.
• Operations / PMO machine-attest the gate stdout when Q&R cannot execute it.
• COO receives the wave verdict; Principal only on an unaddressed Critical (JD).

Guardrails

• The sample is independent — Q&R edits nothing it reviews. Findings are proposals routed to owners.
• No self-attestation closes a phase (R11). This sample + the scripted gate are the only two proofs; Director approval is recorded but not independent.
• Do not defer to GREEN. The reconciliation gate proves declared edges interlock, not that every expected producer is present (R-003). Trace named seams by hand.
• Disclose what you could not execute (R-006). A trace is labelled a trace; the green run is machine-attested by Operations/PMO.
• Oversample members, never under-sample a unit — the whole point is that a thin member or a single unit cannot hide.
• Links to canon, never restates it (CONTRIBUTING.md).

See also: 2026-06-01-wave1-qr-sample.md / 2026-06-01-wave2-qr-sample.md (worked applications) · docs/units/quality-risk/charter.md · SOP-085 / SOP-086 · master plan §1.3 / §9 / §10 / §11 (R1/R7/R11) · interface-contract-spec.md §3.1.1 · reports/risk-register.md (R-003, R-006).