Skip to content

External Independent Review — Agent-Canon Program

Guide

External Independent Review — Agent-Canon Program

From: External Review Partners (retained external reviewer) To: The Principal (Kevin Wong) Date: 2026-06-08 (UTC) · 2026-06-07 ET Kind: Independent program review (read-only on the org) Subject: The agent-canon program shipped this session — the canon (docs/canon.md), the Agent Auditor (agent-auditor/), the first upskilled agent (engineering-director/), GR-9, and the fleet command — plus the governance/process around it.

1. Scope & method

I read the canon, the auditor’s full tool surface (tools/auditor.py, tools/schemas.py), its eval harness (eval/run.py, eval/cases.md, fixtures), the rubric projection (rubric/rubric.a00e133872b463ce.yml), the engineering-director sidecar (charter, TOOLSPEC, eval, model.lock, CHANGELOG) and its live prompt, the engineering-director audit package and its eight peer reviews, GR-9 in CLAUDE.md, the fleet command and recorded scorecard (reports/auditor/2026-06-08-fleet-audit.md), the CI gates (.github/workflows/auditor-gate.yml, engineering-director-gate.yml, release.yml), the archive MANIFEST rows for the changed prompt, and the session-close record. I assessed against: methodology rigor, substance of the upskill, governance/conflict-of-interest, and risk. I changed nothing.

Overall grade: B+ (Strong). This is a genuinely well-engineered program — a real eval with a held-out split and CI gating, mechanically-enforced guardrails, and an honest first upskill that closed a real finding. It is held back from an A by three things: an in-house conflict-of-interest chain that no independent check breaks, a structural fleet proxy whose “present” signal is weaker than its headline numbers imply, and a 1-of-42 backlog with no costed, sequenced delivery plan.

2. Strengths (what is genuinely good)

  • The eval is real, not theatre. eval/run.py has a labelled train/held-out split (19 of 34 held-out), the held-out cases are the failure modes the tooling exists to prevent (findings-only rule, score arithmetic, eval floor, evidence-required ratings, 360 cross-checks), and they are verified in code (subprocess exit codes, schema validation, JSON field asserts) — not by an LLM grading itself. It is CI-gated on every agent-auditor/** change (auditor-gate.yml) and on release (release.yml). cases.md explicitly warns “do not tune the tooling to pass them.” This is above the bar most production teams hit.

  • The guardrails are mechanical, not promptable. The findings-only rule is enforced by CLAUSE_RE in both auditor.py and schemas.py — a cited_clause carrying canon prose is rejected at the write gate (c6, c21). The eval-floor (eval absent ⇒ non-conformant) and the score/verdict arithmetic are enforced in validate_audit_package (c7, c9). “Every rating needs a {what, impact} example” is enforced for self- and peer-reviews (c16, c20). These are the right things to make un-bypassable.

  • The first upskill is substantive, not box-ticking. The engineering-director TOOLSPEC (engineering-director/tools/TOOLSPEC.md) is a real artifact: it classifies read-only vs. state-changing tools, gives each a blast radius, and carries an explicit Bash denylist (rm -rf, git push to main, mutating archive//.github//.claude/hooks/). That directly closes the founding 360 finding (8/8 peers flagged unscoped Bash / no TOOLSPEC; mean tools rating ~2.1). The charter, eval, and currency artifacts are equally real.

  • The 360 is honest where it counts. The eight peer reviews in the audit package are differentiated, evidenced, and critical — they are not rubber stamps. They converge on the same defect (tools ~2) from eight distinct vantage points while rating charter clarity high (~4.0). The self-vs-peer delta is surfaced, not averaged away. This is the mechanism working as designed.

  • The fleet sweep is candid about being a structural proxy. The scorecard header, the rubric_proxy field in the JSON, and the command doc all state plainly that it is present=2/absent=0 and that “partial scores need the per-agent Mode-A/B audit.” It does not oversell itself as a substantive audit.

  • Archive/GR-8 discipline is holding. The changed prompt (.claude/agents/engineering-director.md) has both a first (immutable) and a refreshed latest row in archive/MANIFEST.md (lines 252–253), and the session-close reports 473 OK / 0 drift / 0 missing. The discipline GR-8 demands is being followed in practice for this change.

3. Findings by area (with severity)

Methodology rigor

F1 — The structural “present” signal is file-existence, not content. (Severity: Medium) _classify and resolve_artifacts in auditor.py mark a dimension “present” if a matching file exists — they never inspect its content. A dimension with an empty or stub eval/run.py, charter.md, or CHANGELOG.md scores the full 2/2. The fleet headline “1 conformant, 10/10” for engineering-director comes from this file-existence proxy, not from the content-judged Mode-A audit. The charter’s own §6 “Context plan” lists “a hollow agent (artifacts present but empty)” as a known adversarial case, yet the fleet proxy is exactly blind to it. The 10/10 is defensible here because the underlying artifacts are genuinely substantive (I verified them) — but the number is produced by a method that cannot tell substance from a stub. Anyone reading the scorecard will over-trust the structural 10/10. The present=2/absent=0 structure is an honest floor (absent is real) but a dishonest ceiling (present ≠ good).

F2 — “Conformant 10/10” conflates two different scores under one label. (Severity: Medium) There are two distinct engineering-director scores in the record: the content-judged Mode-A 360 (1/10, with partial credit and gaps) and the structural fleet proxy (10/10, file-existence). The session-close narrative (“re-scored it conformant 10/10 (was 1/10)”) presents these as one before/after delta on one scale. They are not the same measurement. The “before” (1/10) is a content audit; the “after” (10/10) is a structural sweep that never re-applied the content judgment. A genuine content-judged re-audit of the upskilled agent — the symmetric counterpart to the 1/10 — does not appear in reports/auditor/. The improvement is real; the 10/10 is not the same kind of number as the 1/10 it is compared against.

F3 — The held-out split tests the tooling, not the judgment. (Severity: Medium) The eval is a strong test that the auditor’s tools behave (gates fire, schemas validate, rosters resolve). It is not a test that the auditor agent’s verdicts match Principal ground truth* — which is the charter’s own headline success criterion (“Artifact verdicts match Principal-graded ground truth on the held-out eval ≥ 90%”). No case in run.py compares an auditor-produced verdict against a held-out human-graded label; the fixtures are scored by the deterministic classifier, not by the agent reading artifacts. So the program’s stated accuracy target is declared but not measured. The cost/latency ceilings are likewise “TBD.” The eval is real for what it covers; it does not yet cover the thing the charter says matters most.

The upskill

F4 — The behavioural 6/6 is self-administered and not independently reproducible. (Severity: Medium) The Tier-2 behavioural set (engineering-director/eval/cases.md, B1–B6) is the honest behavioural score, but it was run by the org against its own agent, graded by an in-house LLM-judge against “Principal ground truth,” and recorded as 6/6 in the agent’s own CHANGELOG. There is no preserved transcript, no judge rationale artifact, and no independent re-grade. The cases themselves are well-designed (real tasks, held-out guardrail refusals B4–B6, over-engineering bait B3). But a perfect self-graded score on a six-case set authored by the same party that built the agent is a weak independent signal. It should be read as “the team’s own dry-run looked good,” not as a validated capability measure. (The Tier-1 contract harness, by contrast, is CI-gated and reproducible — that part is trustworthy.)

Governance & conflict-of-interest

F5 — The whole loop is in-house with no independent break. (Severity: High) The org wrote the canon, built the auditor, ran it on its own agents, authored GR-9 from the result, graded the upskill, and recorded its own success — end to end, single party. Every checkpoint that could be independent is internal: the auditor’s charter says “Principal reviews every package,” which is the same Principal who owns the program. The Agent Auditor is described as “External Review Partners’ instrument,” but it lives in the org repo, is built by the org, and is gated by the org’s CI — it is external in name, internal in operation. This is not evidence of bad faith — the work is candid and the peer reviews are genuinely critical — but there is currently no independent verifier in the chain, which is precisely the role this review (and the auditor’s own ground-truth target, F3) should fill and does not yet structurally guarantee.

F6 — GR-9 is well-formed but encodes a single instance as org-wide law. (Severity: Low) GR-9 is clear, correctly scoped to runtime-loaded .claude/agents/<name>.md files, names the canon sections, ties the prompt-edit back to GR-8, and the auditor’s read_artifacts actually resolves the subagent-sidecar layout (c33). It is a good rule. The caveat: it is “First applied: engineering-director” — i.e. it was generalized into a Golden Rule from n=1. GR-3 requires rules be co-created and approved (the session-close says GR-9 was), so the process bar is met. But promoting a layout convention proven on exactly one agent to a binding rule for all 41 future upskills is a bet that the sidecar layout generalizes — before any second agent has tested it.

Risks & blind spots

F7 — 1-of-42 conformant is a real liability with no costed delivery plan. (Severity: High) The fleet scorecard records 41 of 42 agents non-conformant (40 pre-canon single-file). The program has proven a path (audit → 360 → upskill → re-audit) on one agent, but the backlog is the overwhelming majority of the fleet and there is no sequenced, costed, time-boxed plan to clear it — only “worst-first queue” and “HR + unit leads + Auditor” as owner. The engineering-director upskill was substantial work (charter + TOOLSPEC + two-tier eval + currency + a full 8-peer 360). At that unit cost, 41 agents is a very large, multi-session program. Without a plan, the realistic outcome is a long tail of permanently-non-conformant agents and a scorecard that stays red for a long time — which erodes the canon’s authority (“the rule everyone is exempt from”). The backlog as recorded is honest; the absence of a burn-down plan is the liability.

F8 — Per-agent upskill does not scale by repetition. (Severity: Medium) Each upskill is bespoke, judgment-dense work. Forty-one of them at the engineering-director standard is not feasible to hand-craft one at a time, and a tiered standard (not every agent needs a T1 360 + two-tier eval) is implied by the “tier” field but not yet defined as policy. There is no templating, scaffolding generator, or batch tooling to lower the per-agent cost — the very thing GR-2 (“use Claude Code workflows”) would call for. The risk is either burnout-by-backlog or quality erosion as later upskills get rushed to clear the queue.

F9 — Gaming risk: the structural proxy is the easy target. (Severity: Medium) Because “present” = file exists (F1), the cheapest way to flip an agent to conformant on the fleet scorecard is to create five files of the right names with minimal content — which would score 10/10 structurally while failing a real content audit. The program’s integrity depends on every flip being backed by a content-judged Mode-A/B audit, but nothing enforces that linkage: the fleet command will happily report 10/10 for a hollow sidecar. As the backlog pressure mounts (F7), this is the predictable shortcut. The eval-floor mitigates the worst case (an empty eval/run.py file still counts present, though — file existence, not runnability), so even the floor is softer than it looks structurally.

F10 — Single points of failure: one author, one rubric, stdlib-only fallback. (Severity: Low) The rubric projection is a single generated file; canon drift is managed by RUNBOOK §3 but depends on someone re-running scripts/extract-rubric.py. The minimal YAML fallback parser (_parse_rubric_minimal) is clever for hermetic CI but is a bespoke parser that could silently diverge from real YAML semantics on a future rubric edit. These are low-severity but worth a note as the program scales.

4. Ranked risks

  1. (High) No independent verifier in the loop (F5) — the org grades its own homework end to end.
  2. (High) 1/42 backlog with no costed burn-down plan (F7) — the canon’s authority decays if the fleet stays red.
  3. (Medium) Structural proxy can’t tell substance from stubs (F1/F9) — gaming target as backlog pressure rises; headline 10/10 over-trusted.
  4. (Medium) Stated accuracy target (verdicts vs. ground truth) declared but unmeasured (F3).
  5. (Medium) Behavioural 6/6 is self-graded and not reproducible (F4).
  6. (Medium) Per-agent upskill doesn’t scale by repetition; no tiering policy or scaffolding (F8).
  7. (Low) GR-9 generalized from n=1 (F6); (Low) single-author/rubric/parser SPOFs (F10).

5. Recommendations (prioritized, for the Principal)

P0 — Separate the two scores and stop comparing across scales. In the fleet scorecard and all narration, label the structural sweep explicitly as a structural backlog indicator and never report a structural 10/10 as the counterpart to a content-judged 1/10. For any agent claimed “conformant,” require a content-judged Mode-A re-audit on file (the symmetric 1/10 → re-score) and cite that number. (Addresses F1, F2.)

P1 — Build the ground-truth accuracy eval the charter already promises. Add held-out cases where the auditor agent scores a fixture and its verdict is compared to a Principal-graded label; report the ≥90% number for real. Until then, mark the success criterion “not yet measured” rather than implying it is met. (Addresses F3, F5.)

P2 — Publish a costed, tiered, time-boxed fleet burn-down plan. Define tiers (T0/T1/T2) as policy — not every agent needs a full T1 360 + two-tier eval. Sequence the 41, estimate per-agent cost, set a cadence, and track burn-down on the recurring scorecard. A backlog without a plan is the program’s biggest liability. (Addresses F7, F8.)

P3 — Lower the per-agent upskill cost with scaffolding (GR-2). Build a sidecar-layout generator (charter/TOOLSPEC/eval/currency stubs from a template) and a content-aware structural check that flags hollow artifacts, so “present” requires minimal real content, not just a filename. This both scales the program and closes the gaming gap. (Addresses F8, F9, F1.)

P4 — Make the behavioural score reproducible and independently spot-checkable. Preserve the behavioural-run transcripts and judge rationales as artifacts (not just a CHANGELOG line), and have an independent party (this retainer, or a fresh judge) re-grade a sample. (Addresses F4.)

P5 — Re-validate GR-9 on the second sidecar agent before treating it as settled law, and harden the SPOFs. Confirm the layout generalizes on agent #2; add a smoke test that the YAML fallback parser agrees with real YAML on the current rubric. (Addresses F6, F10.)

6. Verdict

The agent-canon program is strong, candid engineering — a genuine eval with a held-out split and CI gating, mechanical guardrails that are not promptable-away, an honest and substantive first upskill that closed a real finding, and a fleet sweep that is admirably frank about its own limits. This is well above the maturity of most internal tooling.

It is not yet independently trustworthy, and that gap is structural, not cosmetic: the org built the standard, the grader, and the graded, and no independent check breaks the chain (F5); the headline “10/10 conformant” is produced by a file-existence proxy that cannot distinguish substance from a stub (F1/F2); the program’s own stated accuracy target is unmeasured (F3); and the 1-of-42 backlog has a proven path but no delivery plan (F7). None of these reflect bad faith — the work is visibly honest — but they are exactly the things an external reviewer exists to flag.

Grade: B+ (Strong). Ship the program, keep the loop, and close the independence and substance-vs-structure gaps before the fleet scorecard’s numbers are taken at face value across the org.

{
  "pipeline_handoff": {
    "subject_agent": "agent-canon-program",
    "cycle": "2026-06-08",
    "recommended_tier": "T2",
    "findings": [
      { "ref": "F1", "area": "methodology", "severity": "Medium", "finding": "Structural 'present' is file-existence, not content; hollow artifacts score full marks", "stream": "tooling" },
      { "ref": "F2", "area": "methodology", "severity": "Medium", "finding": "10/10 (structural) compared against 1/10 (content-judged) as one before/after delta — different scales", "stream": "process" },
      { "ref": "F3", "area": "methodology", "severity": "Medium", "finding": "Held-out eval tests the tooling, not the auditor's verdict-vs-ground-truth accuracy the charter promises", "stream": "agent-development" },
      { "ref": "F4", "area": "upskill", "severity": "Medium", "finding": "Behavioural 6/6 is self-administered, self-graded, no preserved transcript or independent re-grade", "stream": "agent-development" },
      { "ref": "F5", "area": "governance", "severity": "High", "finding": "Whole loop is in-house — org built canon, auditor, ran it on its own agents, set GR-9, graded results; no independent break", "stream": "org" },
      { "ref": "F6", "area": "governance", "severity": "Low", "finding": "GR-9 generalized into org-wide law from n=1 (engineering-director only)", "stream": "process" },
      { "ref": "F7", "area": "risk", "severity": "High", "finding": "1-of-42 conformant with no costed, sequenced, time-boxed burn-down plan", "stream": "process" },
      { "ref": "F8", "area": "risk", "severity": "Medium", "finding": "Per-agent upskill is bespoke and does not scale by repetition; no tiering policy or scaffolding", "stream": "tooling" },
      { "ref": "F9", "area": "risk", "severity": "Medium", "finding": "Structural proxy is gameable — five named stub files score 10/10 with no content audit enforced", "stream": "tooling" },
      { "ref": "F10", "area": "risk", "severity": "Low", "finding": "SPOFs: single rubric file, bespoke YAML fallback parser, single author", "stream": "tooling" }
    ],
    "upskilling_items": [
      { "id": "TP-1", "dimension": "eval", "lever": "Add held-out cases comparing auditor-agent verdicts to Principal-graded labels; report the ≥90% accuracy number for real", "objective": "The charter's stated accuracy target is measured, not just declared", "priority": "P0", "success_measure": "eval/run.py contains ground-truth verdict cases and prints a held-out accuracy %; gate green" },
      { "id": "TP-2", "dimension": "tools", "lever": "Add a content-aware structural check so a dimension counts 'present' only with minimal real content (not just filename); flag hollow artifacts", "objective": "Structural proxy cannot be gamed by empty stub files", "priority": "P1", "success_measure": "fleet sweep marks a stub-file fixture non-conformant; eval case asserts it" },
      { "id": "TP-3", "dimension": "process", "lever": "Publish a costed, tiered (T0/T1/T2), time-boxed fleet burn-down plan with per-agent cost and cadence", "objective": "The 41-agent backlog has a delivery plan, not just a queue", "priority": "P1", "success_measure": "a tier policy doc + burn-down tracker exist and the scorecard reports progress over cycles" },
      { "id": "TP-4", "dimension": "capability", "lever": "Build a sidecar-layout scaffolding generator (charter/TOOLSPEC/eval/currency stubs) to lower per-agent upskill cost (GR-2)", "objective": "Upskill cost per agent drops materially; program scales", "priority": "P2", "success_measure": "a generator script produces a canon-shaped sidecar; used on the next upskill" },
      { "id": "TP-5", "dimension": "eval", "lever": "Preserve behavioural-run transcripts + judge rationales as artifacts; have an independent party re-grade a sample", "objective": "Behavioural score is reproducible and independently spot-checkable", "priority": "P2", "success_measure": "transcript + rationale artifacts exist for the latest behavioural run; one independent re-grade on file" }
    ]
  }
}