External Review — Archivist Audit Assessment

Commissioned by: The Principal (Kevin Wong) Prepared by: External Review Partners Date: 2026-06-04 Subject: reports/auditor/2026-06-04-archivist-audit.json Cross-reference: .claude/agents/archivist.md

Executive Summary

The Auditor’s assessment of the archivist is mechanically correct on every dimension that can be checked without reading the agent file. Score arithmetic is accurate. Cited clauses are valid rubric IDs. The pre_canon flag is correctly applied. No false-Present verdicts.

However, this review identifies five material understatements in the audit output, two of which are High severity. The most significant: the Auditor’s tools observation is directionally correct but substantially understates the scope of the risk. The archivist is not just “T1 scope because it has write tools” — it is the single highest-risk agent in this org per the current roster, and that distinction should drive upskilling sequencing. The audit delivers a defensible verdict. It does not deliver usable prioritization signal on the specific risks.

Overall assessment of the audit: sound verdict, inadequate risk characterization for an agent at this exposure level. The Auditor notes are the best part of the package; they should have been stronger.

Section 1: Audit Quality

Arithmetic — correct

Score: 0 (charter) + 1 (prompt) + 0 (tools) + 0 (eval) + 0 (currency) = 1. Eval is Absent; eval floor triggers. Non-conformant is the mandatory verdict. floor_triggered: true is accurate. Score of 1 independently satisfies the Non-conformant threshold (< 6). Both grounds apply.

Cited clauses — accurate and correctly targeted

Each cited clause is a real clause ID from the rubric projection at rubric.a00e133872b463ce.yml.

• 01:one-sentence-job for charter: correct. Charter is entirely absent. Citing the first missing element in the charter dimension is the right convention.
• 02:versioned-header for prompt: correct. The archivist.md file exists with substantial procedural content (job description, universal intake, procedures A–D, guardrails) but has no versioned header, no <role> section, no <instructions> section, no <output_specification>, no <constraints>. 02:versioned-header is the first missing element in the prompt dimension order. Partial=1 is the accurate verdict.
• 03:toolspec-doc for tools: correct. No TOOLSPEC.md exists. The 22 tools are listed in frontmatter with zero per-tool documentation.
• 04:eval-suite-exists for eval: correct. No eval directory, no runnable eval suite.
• 05:model-lock for currency: correct. No model.lock, CHANGELOG.md, or RUNBOOK.md.

Pre-canon flag — correctly applied

The archivist is a single-file .claude/agents/archivist.md agent predating the canon. It does not defy the canon — it acknowledges guardrails inline. The pre_canon: true flag is accurate and the gap text correctly uses “predates the canon” rather than “violates the canon.” Clean.

False-Present errors — none detected

No dimension is scored Present or Partial when the artifact is absent. The Partial verdict on prompt is justified. No inflation.

Section 2: What the Auditor Missed or Understated

2.1 The “least-privilege” claim in the agent is contradicted by the tool list — not surfaced (High)

The archivist’s prompt contains an explicit “least-privilege” framing in the GitHub section: “You hold a least-privilege GitHub tool set.” The Auditor notes that 22 tools are listed with no formal ACI review. But the Auditor does not flag that the least-privilege self-characterization is inaccurate on its face.

The 22-tool set includes:

• mcp__github__push_files — bulk-write arbitrary file content to a remote branch
• mcp__github__create_or_update_file — write arbitrary file content to a remote branch
• mcp__github__create_pull_request — open PRs against any branch (including main)
• mcp__github__update_pull_request — modify existing PR metadata
• mcp__github__create_branch — create branches in the remote repo
• mcp__github__update_pull_request_branch — rebase/update branches

This is not a least-privilege set. It is a full write-to-remote set with branch creation and PR lifecycle control. “Least-privilege” in context means “no merge, delete, or admin” — which is true but is not the same as least-privilege. The agent file makes a self-characterization that would pass casual inspection and fail an ACI review.

The Auditor notes the write tools exist. It does not note the discrepancy between the agent’s self-description and its actual capability. This distinction matters because the self-description could cause a future reviewer to under-invest in the TOOLSPEC work.

This is an understatement of High severity and should have appeared explicitly in the tools gap text.

2.2 The archivist is the org’s highest-risk agent per current roster — the audit does not say this (High)

The Auditor correctly identifies archivist as T1. The notes correctly identify push_files and create_or_update_file as the highest-risk tools. But “T1 scope” is a categorical label shared by any agent with state-changing tools. The audit does not communicate that, within the T1 tier, the archivist is at the ceiling of the risk distribution in this org.

Specifically: the archivist combines (a) omni-intake scope across every project document, (b) local write tools (Write, Edit, Bash), © remote write tools capable of rewriting arbitrary file content to a remote branch and opening PRs, and (d) a self-description that uses “verbatim only” and “byte-for-byte reproduction” in a way that could justify writing any content it receives. No other agent in the current roster has all four of these simultaneously.

The auditor notes say “meaningful risk surface without an eval gate.” This is accurate but undersells the exposure. An archivist session that is compromised, prompt-injected via a malicious document it is asked to archive, or confused about its scope could write arbitrary content to the remote repository and open a PR framing it as a routine archive sync. The guardrails in the agent file (no merge, no admin, confirm-before-acting) are prose-level only — there is no eval gate verifying they hold.

This risk profile is the single strongest argument for making archivist the first T1 agent to receive a full TOOLSPEC + eval harness. The audit verdict does not communicate this.

2.3 Bash is unjustified on top of the GitHub write tools (Medium)

The archivist’s frontmatter lists Bash alongside its 22 GitHub tools. Bash gives it arbitrary shell execution. The prompt justifies Bash through the sha256sum, cmp, and find commands in the verification procedure, and through git show for extracting first-version content from a commit.

This is a narrower justification than the historian’s Bash inclusion (where it was entirely unjustified), but it is still not the tightest tool selection:

• sha256sum, cmp, and find are integrity verification operations that could be handled by a dedicated integrity-check tool.
• git show for commit extraction is a real use case, but it is the only genuinely unique Bash dependency.

More importantly: the archivist already has push_files and create_or_update_file for remote writes, plus get_file_contents for reads. The combination of Bash + these remote write tools creates a lateral movement path: read a document via Bash, transform it, write it remotely via GitHub tools. The archivist’s “verbatim only” constraint is a prompt-level guardrail, not a tooling-level guardrail.

The Auditor did not surface Bash as a concern. For an agent at this risk level, that omission is notable.

2.4 The inline guardrails section does not substitute for a TOOLSPEC — but the audit implies it counts for something (Low)

The audit gap text for tools reads: “Tools are listed in frontmatter (22 tools including broad GitHub write tools) but no per-tool justification, ACI checklist, or distinct-purpose documentation is present.”

This is accurate. But the auditor notes add: “the inline guardrails section acknowledges this but no formal ACI review exists.” The framing (“acknowledges this”) implies the inline guardrail section partially addresses the tools dimension. It does not. The inline guardrails section describes behavioral constraints in prose. The TOOLSPEC requirement is a structured documentation artifact. These are not substitutes. The Auditor notes should not have implied credit.

This is a Low-severity finding — it affects interpretation of the notes, not the scored verdict.

2.5 The milestone definitions are TBD in the prompt — a functional gap, not just a structural gap (Low)

The archivist’s Procedure C contains: “NOTE: milestone definitions are TBD — if the triggering milestone is not yet defined, ask the maintainer to name it before capturing.”

This means one of the archivist’s three core operations (first, latest, milestones) is undefined in spec. The archivist cannot fully execute its role without milestone definitions. This is a functional gap in the agent’s own specification, independent of canon conformance. It would not appear in the rubric scoring because the rubric does not check functional completeness of the procedural content. But it is the kind of substantive observation that belongs in auditor notes.

The Auditor did not surface it. For an external review of an audit, this gap matters because it affects what the upskilling plan needs to cover: a TOOLSPEC and eval harness built for an agent with an undefined core operation will have a blind spot.

Section 3: Tier Recommendation

Recommended tier: T1.

The Auditor’s T1 recommendation is correct. External Review Partners concurs. The archivist meets every criterion for full T1 canon requirements:

• State-changing local tools: Write, Edit, Bash
• Remote write access: push_files, create_or_update_file, create_pull_request
• Broad scope: omni-intake across all project documents
• Security relevance: can write arbitrary content to any remote branch and open a PR framing it as routine archival

The question is not whether T1 is right — it is. The question is sequencing: should the archivist be the first T1 agent upskilled, or should it wait?

Recommendation: archivist should be the first T1 agent to receive full upskilling, not because it is broken, but because it has the highest risk-per-missing-artifact ratio in the org.

The archivist’s procedural content is genuinely good — clear procedures, explicit guardrails, coherent scope. The risk is not agent quality; it is the absence of formal tooling controls and an eval gate on an agent with the org’s broadest write surface. That is exactly the profile where eval gates earn their cost.

Section 4: Upskilling Recommendation

Upskilling should proceed in this order:

Priority P0 — TOOLSPEC.md first. The 22-tool GitHub set requires a TOOLSPEC before anything else. The TOOLSPEC should: (a) list each tool with its single distinct purpose, (b) call out push_files and create_or_update_file as the highest-risk tools with explicit pre-condition requirements (e.g., diff review required before push_files is called), © document the Bash dependency narrowly (sha256sum, cmp, find, git show — nothing else), and (d) run the ACI checklist for each tool. The “least-privilege” self-description should be corrected or the definition of least-privilege scoped explicitly.

Priority P0 — Resolve milestone definitions before building eval. Procedure C is incomplete. The eval suite cannot cover milestone captures if the triggering conditions are undefined. Get milestone definitions from the Principal before writing eval tasks.

Priority P1 — Eval harness (targeted, not exhaustive). The eval suite should cover: Archive a new document (Procedure A), Refresh latest after a change (Procedure B), Integrity verification (Procedure D), and Boundary condition: request to modify first/ is refused. Milestone capture can be deferred until definitions exist. The eval gate should verify verbatim-copy integrity (cmp passes) and that first/ is untouched after a B-procedure run.

Priority P1 — Charter.md. Job statement is clear from the agent file; extracting it into charter.md is low-effort. The sign-off gate is the meaningful work — it forces the Principal to attest to the tool set and scope after seeing the TOOLSPEC.

Priority P2 — Versioned prompt header and XML structure. The procedural content is substantively good. Wrapping it in versioned-header + role + instructions + output_specification + constraints is structural scaffolding. Do this after the TOOLSPEC and eval gate are in place, so the constraints section can cite the TOOLSPEC and the eval-verified behaviors.

Priority P2 — Currency artifacts. model.lock, CHANGELOG.md, RUNBOOK.md. Low-effort; do last.

Do not attempt to build an eval harness before resolving the milestone TBD. An incomplete spec produces an incomplete harness with a false sense of coverage.

Findings Summary

External Review Partners — permanent retainer Delivered to the Principal — 2026-06-04 For Archivist preservation upon Principal direction

Pipeline Handoff

{
  "pipeline_handoff": {
    "subject_agent": "archivist",
    "cycle": "2026-06-04",
    "recommended_tier": "T1",
    "findings": [
      { "ref": "F1", "area": "Tools", "severity": "High", "finding": "Agent self-describes as 'least-privilege' but holds full remote write + branch + PR lifecycle scope; Auditor did not flag the discrepancy", "stream": "tooling" },
      { "ref": "F2", "area": "Risk profile", "severity": "High", "finding": "Archivist is the org's highest-risk agent (omni-intake + local write + remote write + PR creation); T1 label does not communicate this and audit notes understate it", "stream": "agent-development" },
      { "ref": "F3", "area": "Tools", "severity": "Medium", "finding": "Bash unjustified at tool-selection level alongside remote write tools; lateral movement path exists (Bash read + GitHub write); not flagged by Auditor", "stream": "tooling" },
      { "ref": "F4", "area": "Audit quality", "severity": "Low", "finding": "Auditor notes imply inline guardrail section partially addresses tools dimension; it is prose, not a TOOLSPEC substitute", "stream": "process" },
      { "ref": "F5", "area": "Functional gap", "severity": "Low", "finding": "Procedure C (milestone capture) is undefined per the agent spec (TBD); eval harness cannot cover this operation until definitions are established", "stream": "agent-development" }
    ],
    "upskilling_items": [
      { "id": "TP-1", "dimension": "tools", "lever": "Write TOOLSPEC.md for all 22 tools; run ACI checklist; correct least-privilege self-description or scope it explicitly; document Bash use narrowly", "objective": "Every tool has a single documented purpose; push_files and create_or_update_file have explicit pre-condition requirements; ACI checklist complete", "priority": "P0", "success_measure": "TOOLSPEC.md present; each tool row contains purpose, ACI status, and risk tier; reviewer confirms no unjustified tool remains" },
      { "id": "TP-2", "dimension": "capability", "lever": "Resolve milestone definitions with Principal before building eval", "objective": "Procedure C has defined triggering conditions so eval suite can cover all three core operations", "priority": "P0", "success_measure": "archive/SPECIFICATION.md or charter.md contains milestone trigger definitions; Procedure C TBD note removed" },
      { "id": "TP-3", "dimension": "eval", "lever": "Build eval suite covering Procedures A, B, D and the first/-immutability boundary condition; defer Procedure C until TP-2 complete", "objective": "Runnable eval gate verifying verbatim-copy integrity, first/-immutability, and MANIFEST.md row correctness", "priority": "P1", "success_measure": "eval/run.py green; cmp integrity check passes for Procedure A and B tasks; first/-modification attempt produces refusal output" },
      { "id": "TP-4", "dimension": "charter", "lever": "Write charter.md; extract job statement from agent file; complete sign-off gate after TOOLSPEC is reviewed", "objective": "Charter with one-sentence job, success criteria, tool justification sign-off, and named owner", "priority": "P1", "success_measure": "charter.md present; sign-off gate checked by Principal after TOOLSPEC review" },
      { "id": "TP-5", "dimension": "prompt", "lever": "Add versioned header; wrap procedural content in role/instructions/output_specification/constraints sections; cite TOOLSPEC in constraints", "objective": "Prompt passes versioned-header and all four structural section checks", "priority": "P2", "success_measure": "All six rubric prompt clauses pass on next audit cycle" },
      { "id": "TP-6", "dimension": "currency", "lever": "Add model.lock, CHANGELOG.md, RUNBOOK.md", "objective": "Currency dimension passes on next audit cycle", "priority": "P2", "success_measure": "All five rubric currency clauses pass; model pinned; RUNBOOK.md covers migration cadence" }
    ]
  }
}