External Review — Historian Audit Assessment

Commissioned by: The Principal (Kevin Wong) Prepared by: External Review Partners Date: 2026-06-04 Subject: reports/auditor/2026-06-04-historian-audit.json Cross-reference: .claude/agents/historian.md

Executive Summary

The Auditor’s first real audit — the historian — is mechanically correct. The verdict (Non-conformant, 1/10, pre_canon=true) is defensible and consistent with the Auditor’s own spec. The score arithmetic is valid. The cited clauses are real clause IDs from the rubric projection. The pre_canon flag is correctly applied. There are no false-Present verdicts, no canon prose leaks, and the package validates against the output specification in prompt.v1.md.

That said, this review identifies three genuine understatements, two methodology concerns relevant to Phase-2 planning, and a recommendation on upskilling that the Principal should establish as policy before running more audits.

Overall assessment of the audit: sound but blunt. Fit for purpose as a Phase-1 Mode-A run. Not yet rich enough to drive prioritization decisions.

Section 1: Audit Quality

Arithmetic and scoring — correct

Total score: 0 (charter) + 1 (prompt) + 0 (tools) + 0 (eval) + 0 (currency) = 1. Eval is Absent, so the eval floor triggers. Non-conformant verdict is mandatory regardless of total. Score of 1 also independently meets the Non-conformant threshold (< 6). Both grounds apply; floor_triggered: true is accurate.

Cited clauses — accurate and appropriately chosen

Each cited clause is a valid clause ID from the rubric projection at rubric.a00e133872b463ce.yml. For Absent dimensions, the Auditor cites the existence clause of the dimension (e.g., 04:eval-suite-exists for eval, 01:one-sentence-job for charter). This is a reasonable convention — cite the first failure rather than attempting to enumerate all downstream failures for something that doesn’t exist. Acceptable.

The Partial verdict on prompt, citing 02:versioned-header, is accurate. The historian’s single-file prompt has domain content (altitude levels, guardrails, sources, differentiation from Archivist) but lacks the versioned header, role section, instructions section, output_specification, and constraints sections. 02:versioned-header is the first missing element in the canonical order.

Pre-canon flag — correctly applied

The historian is a single-file .claude/agents/historian.md agent with no charter, toolspec, eval, or currency directory. The pre_canon flag is warranted. The gap descriptions correctly note “predates the canon” without characterizing the agent as defying it.

False-Present errors — none detected

No dimension is scored Present or Partial when it should be Absent. This is the most expensive error class per the Auditor’s own charter. Clean.

Section 2: What the Auditor Missed or Understated

2.1 Bash is an unjustified tool for this role

The historian’s frontmatter lists six tools: Read, Write, Edit, Bash, Grep, Glob.

The rubric penalizes this under 01:minimal-tools and 03:distinct-purpose, but because charter and toolspec are Absent, these checks collapse into dimension-level absent verdicts. The substantive concern — that Bash is unjustified for a narrative-curation role — is invisible in the audit output.

The historian’s own prompt says “Stay in your lane. You write history/ documents.” Bash gives it the ability to execute arbitrary shell commands, which contradicts that guardrail and overlaps with Write and Edit for any plausible historian task. This is a real risk that should appear in the gap description for the tools dimension, even though the dimension itself is Absent. The gap text reads: “No TOOLSPEC.md exists… Tools are listed in frontmatter.” It should also read: “Bash tool present in frontmatter is unjustified for a narrative-curation role; it overlaps with Write/Edit and expands the blast radius beyond what the role requires.”

The Auditor did not flag this. It is an understatement.

2.2 Partial=1 is too blunt to be useful for pre-canon agents

The historian’s prompt dimension received Partial=1. Under the scoring rubric, Partial=1 is the same score a two-line stub would receive if it technically contained domain content. The historian’s prompt is substantively well-written: it defines an altitude model (logline through full chronicle), differentiates the historian from the Archivist, names its sources, and provides a five-step process with guardrails.

A partial score of 1 does not communicate to the Principal whether the underlying prompt is “good work that needs structural wrapping” or “a weak placeholder.” For pre-canon agents in particular — where the entire diagnosis collapses into a row of 0s and one 1 — this distinction is load-bearing for upskilling decisions.

The Auditor cannot fix this within the current rubric (Partial is a rubric-level construct, not an Auditor-level one), but it should be flagged as a gap in the scoring model rather than silently absorbed by the blunt instrument.

2.3 pre_canon is binary; the signal it carries is imprecise

pre_canon=true tells the Principal that the agent predates the canon. It does not tell the Principal whether the agent is a well-built legacy agent that needs structural scaffolding or a weak agent that needs substantive rebuilding. The historian falls into the first category: coherent job definition, clear ownership, well-structured process, genuine guardrails. Many pre-canon agents will fall into the second.

Without a qualitative signal — even a single Auditor’s-note field in the package — the Principal cannot prioritize upskilling from audit packages alone. This is a Phase-1 design gap, not a Phase-1 execution error by the Auditor.

Section 3: Process Observations for Phase-2 Planning

3.1 n=1 does not validate methodology

This is the first real audit. The eval fixture conformant audit (2026-06-03) exercised the tooling against a synthetic known-answer case. The historian audit is the first real subject. One real audit is insufficient to validate that the rubric discriminates meaningfully between agents or that the Auditor’s verdicts will correlate with the Principal’s ground-truth assessments. Before Phase-2 expands scope, run three to five more real audits and compare verdicts to Principal-assigned ratings.

3.2 Mode-A is structural; the Principal must not read “Non-conformant” as “bad agent”

The audit correctly scores whether the canonical artifact structure is in place. It does not assess whether the historian is good at its job. Non-conformant (pre_canon) is the expected verdict for every pre-canon single-file agent in the KLMS org — the charter explicitly names this as informative, not a failure. The Principal should treat the 1/10 score as a baseline datum, not a quality judgment. Nothing in the audit package communicates this. A covering note or a standard field (e.g., interpretation_note: "pre-canon agents are expected non-conformant; this verdict reflects structural absence, not role quality") would reduce misread risk.

3.3 The cited_clause-for-Absent-dimensions convention is emergent, not specified

The Auditor’s pattern of citing the existence clause as the cited_clause for Absent dimensions (e.g., 01:one-sentence-job when charter is entirely absent) is reasonable but is not written down anywhere. The prompt’s example for the archivist audit establishes this pattern by illustration. If future auditors or a Phase-2 automated tool deviates from it, comparability across audit packages breaks. The convention should be stated explicitly in the prompt’s <constraints> section.

3.4 No narrative channel for substantive observations

The findings-only rule is correct as a canon-leakage guardrail. But it eliminates any channel for the Auditor to surface substantive observations that the rubric does not capture — such as the Bash tool concern in 2.1 above. Phase-2 should consider a separate auditor_notes field (non-scored, non-rubric-governed) for substantive observations that are outside the scoring model. This field would be clearly distinguished from scored findings and would not carry rubric language.

Section 4: Recommendation — Upskilling the Historian

Direct answer: do not run full canon upskilling on the historian first.

Full canon upskilling requires: charter.md with sign-off gate, versioned prompt with XML section structure, TOOLSPEC.md with ACI checklist, eval/run.py with held-out split and CI gate, model.lock, CHANGELOG.md, RUNBOOK.md. This is the right standard for agents with high-stakes tool access, broad scope, or org-wide impact. The historian writes narrative history files to history/. The risk profile does not justify eval harness overhead.

The more urgent action is for the Principal to establish a tiering policy before running more audits. Without it, every pre-canon agent produces the same 1/10 verdict with no signal on what to do. A minimal working tiers model:

• T1 (full canon): agents with state-changing tools, broad scope, or security relevance. Full charter + versioned prompt + toolspec + eval + currency infrastructure.
• T2 (lightweight canon-aligned): agents with narrow scope, low risk, no eval harness warranted. Requires: one-sentence job statement, versioned prompt header, minimal stated tools with justifications, named owner.
• T0 (acknowledged legacy): single-file pre-canon agents that will not be upskilled. Noted in a registry; audited for flag/risk but not scored against T1 criteria.

For the historian specifically: the right intervention is small — remove Bash from the tools list, add a versioned header to the prompt file, add a one-sentence-job statement to the frontmatter description. That brings it to T2-compliant with one hour of work. Building an eval harness for narrative-curation output would be gold-plating.

The Auditor’s 1/10 verdict accurately reflects canon non-conformance. That verdict should not determine where upskilling effort goes. Tiering policy should.

Findings Summary

Verdict on the Auditor

The Auditor ran its first real audit correctly. The mechanics are sound. The pre-canon path was handled as specified. No false-Presents, no canon prose leaks.

The gaps above are not execution errors — they are the natural ceiling of a Phase-1 Mode-A scope. The Auditor was built for structural artifact scoring, and that is what it delivered. The understatements (Bash tool concern, blunt Partial score) are limitations of the scoring model, not of the Auditor’s execution.

The Principal should act on F4 (tiering policy) before commissioning further audits. Without it, the audit program produces correct but undifferentiated verdicts that cannot drive prioritization.

External Review Partners — permanent retainer Delivered to the Principal — 2026-06-04 For Archivist preservation upon Principal direction

Pipeline Handoff

{
  "pipeline_handoff": {
    "subject_agent": "historian",
    "cycle": "2026-06-04",
    "recommended_tier": "T2",
    "findings": [
      { "ref": "F1", "area": "Tools", "severity": "Medium", "finding": "Bash tool unjustified for narrative-curation role; overlaps Write/Edit, expands blast radius", "stream": "tooling" }
    ],
    "upskilling_items": [
      { "id": "TP-1", "dimension": "tools", "lever": "Remove Bash from tools list", "objective": "Eliminate unjustified shell access from narrative-curation agent", "priority": "P1", "success_measure": "Bash absent from frontmatter tools list; reviewer confirms" },
      { "id": "TP-2", "dimension": "prompt", "lever": "Add versioned header to prompt file", "objective": "T2 requirement: vN.N — model — owner — tier visible in the file", "priority": "P2", "success_measure": "Versioned header comment present in historian.md" },
      { "id": "TP-3", "dimension": "charter", "lever": "Sharpen one-sentence job statement in frontmatter description", "objective": "T2 requirement: description opens with a single clear sentence ending with a period", "priority": "P2", "success_measure": "Description first sentence is a complete, unambiguous job statement" }
    ]
  }
}