Response to the External Review — Agent-Canon Program (2026-06-08)
Management response to
reviews/2026-06-08-external-review-agent-canon-program.md(External Review Partners, verdict B+). Every finding F1–F10 and recommendation P0–P5 is addressed below with the action taken and where it landed. Directed by the Principal: “act on all findings.”
Findings → actions
| Ref | Sev | Finding | Action taken | Status |
|---|---|---|---|---|
| F1 | Med | Structural “present” = file-existence, not content | auditor.py: content-aware scoring — a dimension earns credit only with ≥ MIN_SUBSTANTIVE_CHARS real content (aggregated across the dimension’s files); hollow stubs score 0 and are flagged. Eval c34 + hollow-agent fixture prove it. |
Fixed |
| F2 | Med | “10/10 (structural)” conflated with “1/10 (content)” under one label | Fleet output relabelled: score_kind: structural, a Certified (content-judged) column, and a banner that a structural verdict is not a certification. Emitted the content-judged Mode-A re-audit for engineering-director (…2026-06-08-postupskill-engineering-director-audit.json) — the symmetric counterpart to the 1/10. |
Fixed |
| F3 | Med | Held-out eval tests tooling, not verdict-vs-ground-truth accuracy | Added an accuracy case (c35): classifier verdicts vs. labelled fixtures = 100% on a 4-agent set. Charter’s ≥90% agent-judgment target marked “not yet measured” (needs model-in-loop). | Partly fixed (structural accuracy measured; agent-judgment accuracy still pending) |
| F4 | Med | Behavioural 6/6 self-administered, no preserved transcript | Transcripts + judge rationales preserved: engineering-director/eval/behavioural-runs/2026-06-08.md. Independence caveat recorded; independent re-grade still open. |
Partly fixed (reproducible now; independent re-grade pending) |
| F5 | High | Whole loop in-house, no independent break | (a) The external review itself is the first independent break, now on file. (b) Certification rule: “conformant” requires a content-judged audit on file, spot-checkable by the retainer (RUNBOOK §Independence). © Burn-down plan assigns the Auditor as verifier distinct from execution. The structural gap (org grades itself) is acknowledged, not eliminated — see open items. | Mitigated |
| F6 | Low | GR-9 generalized from n=1 | Burn-down plan §2: the first T1 upskill after engineering-director is the explicit GR-9 generalization test before the rule is treated as settled. | Scheduled |
| F7 | High | 1/42 conformant, no costed delivery plan | reports/2026-06-08-fleet-canon-burndown-plan.md — tiered (T0/T1/T2), sequenced, costed (~19 sessions), with a per-cycle certified-conformant burn-down metric. |
Fixed (plan exists; execution is the program) |
| F8 | Med | Per-agent upskill doesn’t scale; no tiering | Tier policy defined (burn-down §1). Scaffolding generator scoped as P3/TP-4 (next lever). | Policy fixed; scaffolding scheduled |
| F9 | Med | Structural proxy gameable (5 stub files → 10/10) | Closed by F1’s content-aware check + the certification rule: a flip needs real content and a content audit. hollow-agent fixture asserts a 5-stub agent scores non-conformant. |
Fixed |
| F10 | Low | SPOFs: bespoke YAML fallback parser could diverge | Added a smoke test (c36) comparing the fallback parser to PyYAML on the live rubric — which caught a real bug (every non-final dimension dropped its last clause; charter 5 vs 6). Bug fixed in _parse_rubric_minimal. |
Fixed (+ latent bug found & fixed) |
Recommendations → disposition
- P0 (separate the two scores) — done (F1/F2): structural sweep relabelled; content-judged re-audit on file; certification rule.
- P1 (ground-truth accuracy eval) — partly (F3): structural-classifier accuracy measured (c35); agent-judgment accuracy harness remains open.
- P2 (costed tiered burn-down plan) — done (F7/F8).
- P3 (scaffolding generator + content-aware check) — content-aware check done (F1/F9); scaffolding generator scheduled (next lever).
- P4 (reproducible behavioural score) — transcripts preserved (F4); independent re-grade open.
- P5 (re-validate GR-9 on agent #2; harden SPOFs) — GR-9 #2 scheduled (F6); YAML SPOF done (F10).
Open items carried forward (honest residue)
- Agent-judgment accuracy eval (F3/P1) — measuring the auditor agent’s Mode-A verdicts against Principal-graded labels needs a model-in-the-loop harness. Not yet built.
- Independent re-grade (F4/P4) — have the retainer (or a fresh judge) re-grade a behavioural sample.
- Structural independence (F5) — the org still builds and runs its own auditor; the durable fix is periodic external spot-checks (this retainer) of certified claims, not a one-time review.
- Scaffolding generator (P3/TP-4) and GR-9 #2 validation (F6) — both tied to the burn-down program.
Verification of this response: auditor eval 36/36 (21 held-out), archive clean.