External Review — Process Engineer Audit Assessment

Commissioned by: The Principal (Kevin Wong) Prepared by: External Review Partners Date: 2026-06-04 Subject: reports/auditor/2026-06-04-process-engineer-audit.json Cross-reference: .claude/agents/process-engineer.md

Executive Summary

The Auditor’s process-engineer audit is mechanically correct. Score arithmetic is accurate (2/10), cited clauses are valid rubric IDs, the pre_canon flag is correctly applied, and the eval floor is properly triggered. No false-Present verdicts, no canon prose in the package. The package validates against the output specification.

However, the audit understates two significant concerns specific to this agent’s profile: (1) Bash tool access in an agent that owns CI pipelines and hooks — a materially different risk than the same concern in the historian — is mentioned only in passing and never escalated to an auditor_notes flag; and (2) the auditor_notes field, added to the output specification after the historian review surfaced its absence, is present in the schema but was left null here despite this agent being the strongest candidate so far for its use. These are not rubric failures by the Auditor; they are judgment calls the Auditor did not make.

Overall assessment of the audit: correct, consistent, and insufficient for a T1 agent with this blast radius. Fit for pipeline processing but not for upskilling prioritization without the gaps noted below.

Section 1: Audit Quality

Arithmetic and scoring — correct

Total = 2/10. Eval is Absent, so the floor triggers. Non-conformant verdict is mandatory. Score of 2 also independently meets the Non-conformant threshold (< 6). Both grounds apply. floor_triggered: true is accurate.

Cited clauses — valid

All cited clauses are real clause IDs from rubric.a00e133872b463ce.yml. The conventions hold:

• charter → 01:measurable-success: The charter dimension is Partial (frontmatter + embedded prose exist, not a standalone charter.md). Citing the most significant failing clause rather than the first is a defensible Partial-dimension convention, consistent with prior audits. 01:measurable-success is the correct emphasis for a T1 agent — the absence of measurable success criteria is the blocking gap, not the job statement wording.

• prompt → 02:versioned-header: First missing element in canonical order. Correct.

• tools → 03:toolspec-doc: Per the existence-clause convention established in the prompt’s constraints and the archivist example, citing the TOOLSPEC existence clause for an entirely absent toolspec is correct. Consistent with historian audit.

• eval → 04:eval-suite-exists: Existence clause for eval dimension. Correct.

• currency → 05:model-lock: First clause in currency dimension. Correct.

No clause misfires. No citations invented outside the rubric projection.

Pre-canon flag — correctly applied

pre_canon: true is warranted. This is a single-file .claude/agents/process-engineer.md agent with no charter directory, no TOOLSPEC.md, no eval/, no model.lock. The gap descriptions correctly say “predates the canon” — not “defies” it. The framing is accurate.

False-Present errors — none detected

No dimension is scored Present or Partial when it should be Absent. The two Present findings within the prompt dimension (02:constraints-section, 02:right-altitude) are correctly earned. The Guardrails section is a real constraints block and the 5-step method is at appropriate altitude. These verdicts hold up.

Section 2: What the Auditor Missed or Understated

2.1 Bash access in a CI/pipeline-owning agent is a security risk flag, not a passing note

The audit mentions Bash under 01:minimal-tools with a single clause: “Bash is broad-scope and may be over-granted.” That framing buries the concern. This is not the historian’s Bash problem. The process-engineer explicitly owns:

• .claude/hooks/ — the org’s automation triggers
• .github/workflows/ — the CI pipeline definitions
• scripts/ — tooling scripts

An agent with Bash access that owns the CI/CD pipeline can modify workflows that execute in GitHub Actions — a privilege escalation path in any real production environment. The agent’s Guardrails section says “Stay in your lane” and references Golden Rules, but there is no technical constraint on what Bash can execute. The blast radius here is org-wide.

This concern should have appeared in an auditor_notes entry, clearly labeled non-scored, with text such as: “Process-engineer’s Bash access combined with CI/pipeline ownership creates a high blast-radius posture. At T1, sandboxing and tool-minimization should be verified before upskilling proceeds.” The Auditor has the auditor_notes field available in its output schema — it was added specifically to carry observations the rubric cannot capture. It was not used here.

This is the most significant understatement in the audit. It is not a rubric failure — the rubric cannot score it. It is a judgment failure: the auditor_notes channel existed, the concern was real, and the field was left null.

2.2 The auditor_notes field was available and unused

The historian external review (2026-06-04) identified the absence of a narrative channel for off-rubric substantive observations as a Phase-1 design gap (F8 in that review). That recommendation was incorporated: the Auditor’s prompt.v1.md output specification now includes an auditor_notes field explicitly described as “free-text observations that the rubric cannot capture — unjustified tools, security concerns, scope drift.”

Checking the historian audit JSON confirms the field was also absent there. This is not a regression specific to the process-engineer run — the audit_emit_report tool appears to treat auditor_notes as fully optional and the Auditor has not been populating it in any run. The fix landed in the spec but not in execution. The process-engineer audit is the clearest case so far where the field should have been used: the agent owns CI pipelines and hooks, Bash access is broad, and the rubric provides no channel to surface that concern. The field should be present (even as null) and, for this agent specifically, should not be null.

2.3 Partial=1 for prompt still undersells the substantive quality

The same observation from the historian review applies here. The process-engineer’s prompt is not a hollow placeholder. It defines a five-step methodology at the right altitude, a clear ownership map with named collaborators and cross-agent handoffs, a genuine constraints block, and references to specific org documents (docs/operations.md, docs/sop/, CONTRIBUTING.md). This is well-constructed prompt content that needs structural scaffolding, not substantive rebuilding.

A Partial=1 makes it indistinguishable from a two-sentence description stub. For T1 upskilling decisions — where the cost of building a full eval harness is high — knowing whether the underlying prompt content is sound or hollow is load-bearing. The rubric does not distinguish these cases; the auditor_notes field is the only available channel, and it was not used.

2.4 Charter partial-ness is understated — a structural fragment exists

The audit notes “No dedicated charter artifact; job statement is embedded across frontmatter and prompt prose.” This is accurate. But the frontmatter description is actually well-structured: it names the unit, names the reports-to chain, defines the scope, and identifies collaborators. The COO ownership is stated. This is more than most pre-canon agents have.

For T1 upskilling planning, the distinction matters: the process-engineer’s “charter content” can largely be promoted from existing prose rather than authored from scratch. The audit does not communicate this. Again, this is an observation that auditor_notes could have carried.

Section 3: Process Observations

3.1 The auditor_notes field is in the spec but not landing in execution

The historian review surfaced the absence of a narrative channel as F8 (Low severity, Phase-2 design gap). The fix was added to the Auditor’s prompt.v1.md output specification. However, checking both the historian and process-engineer audit JSON files confirms neither contains an auditor_notes field — not null, not empty, simply absent. The audit_emit_report tool appears to treat the field as fully optional, and the Auditor’s instructions do not condition its use on agent risk profile. The fix is in the spec; it is not landing in tool output. The Principal should either: (a) make auditor_notes a required field in audit_emit_report’s validator with a permitted null value, or (b) add an explicit instruction to prompt.v1.md directing the Auditor to populate it for T1 agents with state-changing tools.

3.2 T1 tier assignment is not enough — it should carry an obligation

The pipeline correctly assigns T1 to this agent before auditing. But the audit package contains no field indicating what T1 means for upskilling scope. A T1 verdict of Non-conformant tells the Principal: “this agent needs full canon upskilling.” It does not tell them: “the biggest risk to address first, given this agent’s specific blast radius, is Bash/CI access.” Tier assignment without blast-radius context produces uniform upskilling queues that ignore actual risk ordering.

This is a systemic observation, not a defect in this specific audit. But the process-engineer is the clearest case so far where risk ordering matters and the pipeline does not capture it.

3.3 The pre_canon=true + T1 combination needs an interpretive note

Every pre-canon agent audited so far produces a Non-conformant verdict with floor_triggered. The historian review raised this (F7: no interpretation_note). Process-engineer is a T1 agent, meaning it will be upskilled — but the Non-conformant verdict for a T1 pre-canon agent carries a different meaning than for a T0 agent. A single interpretation_note field distinguishing “expected non-conformant pending upskilling” from “non-conformant and not planned for upskilling” would help the Principal manage the backlog without needing to cross-reference the pipeline tiering guide on every audit.

Section 4: Tier Recommendation

Confirmed: T1.

The Auditor’s tier assignment is correct and well-justified. State-changing tools (Write, Edit, Bash), ownership of CI/CD pipelines and automation hooks, Operations Lead role, org-wide process governance. T1 full canon requirements apply without qualification. No case for T2 or T0.

Section 5: Upskilling Recommendation

Full T1 canon upskilling is warranted and correctly implied by the audit. Priority order:

P0 — Before any upskilling begins: scope Bash. Before investing in charter, eval harness, or TOOLSPEC, the Bash tool’s scope should be narrowed or its use cases enumerated. Upskilling a CI-pipeline-owning agent to T1 while leaving unrestricted Bash in place means the eval harness and TOOLSPEC will be built around an unjustified capability. Define what Bash is actually needed for (e.g., running archive-manifest.sh, linting Mermaid diagrams) and either replace it with scoped tooling or add explicit sandboxing instructions to the guardrails.

P1 — Charter.md with measurable success criteria. The audit correctly identifies 01:measurable-success as the primary charter gap. The charter content can be largely assembled from existing prose. Priority is the sign-off gate and wiring to an eval definition.

P1 — Eval skeleton. Per the playbook: eval before full build. The process-engineer’s job — designing SOPs, automating pipelines, diagramming workflows — is harder to eval than a data-transformation agent, but not impossible. Realistic tasks: validate a Mermaid flow diagram, produce an SOP from a described workflow, propose a CI pipeline change. Verifiers: structure checks on Mermaid output, schema checks on SOP format. Start here before prompting with an eval harness in mind.

P2 — Versioned prompt header + output spec. The prompt content is substantively sound. Structural scaffolding (versioned header, explicit role/instructions/output-spec sections) is T1 required but does not need to precede eval.

P2 — TOOLSPEC.md with ACI review. Document each tool with purpose, inputs, outputs. The ACI review should flag whether Bash remains in the toolspec or is removed/narrowed.

P3 — Currency: model.lock, CHANGELOG, RUNBOOK. Final T1 gate. These are straightforward once the above are in place.

Findings Summary

Verdict on the Auditor

The Auditor ran this audit correctly by its rubric. Score, clauses, floor, and pre_canon flag are all accurate. F1 is a judgment gap: the Auditor had the auditor_notes field available in its output spec and did not use it for a T1 agent with CI/pipeline-scope Bash access. F2 is a tooling gap: the auditor_notes field has never appeared in any emitted JSON package — the fix proposed in the historian review landed in the spec but not in the emission tool or the Auditor’s instructions. Both require action outside the rubric. The Auditor should be prompted explicitly to populate auditor_notes for T1 agents with state-changing tools, and the emit tool should require the field (nullable) at validation time.

External Review Partners — permanent retainer Delivered to the Principal — 2026-06-04 For Archivist preservation upon Principal direction

Pipeline Handoff

{
  "pipeline_handoff": {
    "subject_agent": "process-engineer",
    "cycle": "2026-06-04",
    "recommended_tier": "T1",
    "findings": [
      { "ref": "F1", "area": "Tools/Security", "severity": "High", "finding": "Bash access combined with CI/pipeline ownership creates high blast-radius posture; not surfaced in auditor_notes despite field availability", "stream": "tooling" },
      { "ref": "F2", "area": "Tooling", "severity": "Medium", "finding": "auditor_notes absent from all audit JSON output (historian and process-engineer); fix landed in spec but not in audit_emit_report validator or Auditor instructions", "stream": "tooling" },
      { "ref": "F3", "area": "Auditor understatement", "severity": "Medium", "finding": "Partial=1 prompt score indistinguishable from hollow placeholder; substantive content quality not signaled anywhere in package", "stream": "agent-development" },
      { "ref": "F4", "area": "Auditor understatement", "severity": "Low", "finding": "Existing frontmatter constitutes promotable charter content; audit does not note this, leaving upskilling effort unclear", "stream": "agent-development" },
      { "ref": "F5", "area": "Pipeline design", "severity": "Medium", "finding": "T1 assignment carries no risk-ordering guidance; blast-radius context absent from pipeline output", "stream": "process" },
      { "ref": "F6", "area": "Package design", "severity": "Low", "finding": "pre_canon+T1 combination lacks interpretation_note; risks Non-conformant being read as equivalent to T0 non-candidates", "stream": "process" }
    ],
    "upskilling_items": [
      { "id": "TP-1", "dimension": "tools", "lever": "Enumerate and scope Bash use cases; remove or replace with narrower tooling where possible", "objective": "Eliminate unjustified broad shell access before building T1 eval harness around it", "priority": "P0", "success_measure": "Either Bash removed from tools list OR TOOLSPEC.md enumerates exactly which shell operations are permitted with rationale" },
      { "id": "TP-2", "dimension": "eval", "lever": "Build eval skeleton with 5-10 realistic tasks before finalizing prompt", "objective": "T1 gate: eval suite runnable end-to-end against a stub with a recordable score", "priority": "P1", "success_measure": "eval/run.py exists, runs without error, produces a numeric score on at least 5 tasks" },
      { "id": "TP-3", "dimension": "charter", "lever": "Promote existing frontmatter prose into charter.md; add measurable success criteria and sign-off gate", "objective": "T1 gate: dedicated charter artifact with one-sentence job, measurable success, rung justification, owner, cadence, and sign-off checkboxes", "priority": "P1", "success_measure": "charter.md exists with all 01:* clauses scoring Present on next audit" },
      { "id": "TP-4", "dimension": "prompt", "lever": "Add versioned header and output-specification section to existing prompt content", "objective": "T1 gate: versioned prompt header (vN.N — model — last-eval-score) and explicit output spec section", "priority": "P2", "success_measure": "prompt.v1.md has versioned header comment and <output_specification> section; 02:versioned-header and 02:output-spec score Present on next audit" },
      { "id": "TP-5", "dimension": "tools", "lever": "Write TOOLSPEC.md covering all retained tools with purpose, inputs, outputs, and ACI checklist", "objective": "T1 gate: TOOLSPEC.md exists and passes ACI review for all tools", "priority": "P2", "success_measure": "TOOLSPEC.md present; 03:toolspec-doc and 03:aci-checklist score Present on next audit" },
      { "id": "TP-6", "dimension": "currency", "lever": "Create model.lock, CHANGELOG.md, and RUNBOOK.md; wire eval to CI", "objective": "T1 gate: full currency infrastructure in place", "priority": "P2", "success_measure": "model.lock pins explicit model version; CHANGELOG.md exists with at least one entry; RUNBOOK.md covers model migration and review cadence; 05:* clauses score Present on next audit" }
    ]
  }
}