Skip to content

SOP-067 — AI-feature build + eval-harness execution

Guide

<!-- last-reviewed: 2026-06-01 · owner: AI Systems (AI Engineer) -->

SOP-067 — AI-feature build + eval-harness execution

SOP · owner: AI Systems (AI Engineer). Layer: SOP/Runbook (docs/canon.md). The Director owns what to build and how to evaluate (SOP-065/066); this is how the implementation gets done and the harness gets run. Implementation backlog: docs/knowledge/ai-systems-retrieval-review-2026-05-31.md — linked, not restated.

Interface Contract

  • Inputs: architecture spec / eval harness ← from <AI Systems Director> · trigger: review requested
  • Outputs: per-call AI-usage / token event (cost instrumentation) → to Financial Analyst · trigger: usage data ready
  • Outputs: eval results report (to interpret) → to AI Systems Director · trigger: review requested
  • Outputs: AI-output contract (retrieval / generation) → to Platform/Staff Engineer · trigger: review requested
  • Outputs: approved implementation note / eval report → to Archivist · trigger: artifact approved
  • Handoff trigger (in): the AI Systems Director hands a spec to build or a harness to run
  • Handoff trigger (out): the increment is built with cost instrumentation wired, or the harness is run and a results report delivered to the Director to interpret
  • Definition of Done before handoff:
    • [ ] Spec read in full; ambiguities/blockers surfaced to the Director before coding
    • [ ] Implemented in the smallest correct increment; matches surrounding conventions (TS strict, 2-space, ESM)
    • [ ] Per-call cost instrumentation wired from the start — token in/out, cache-hit, model, usd_equiv source (the producer of usage data ready)
    • [ ] Verified against the spec — happy path + ≥2 failure modes (rate limit, empty result); real output reported, no paraphrasing
    • [ ] For an eval run: all cases executed, scores collected, written results report produced — harness logic not modified (surface problems instead)
    • [ ] AI-output contract agreed with the Platform Engineer (when staffed) before changing it
    • [ ] Code committed; produced docs handed to the Archivist (0 drift); learnings captured to docs/knowledge/ if any

H4 producer note (COO log H4). The usage data ready output is the per-call instrumentation that feeds Finance’s AI-usage ledger (docs/sop/SOP-028-ledger-upkeep.md consumes it). The interpretation/model-mix half is the Director’s (SOP-065); this is the raw numbers producer. It is blocked until platform/ exists — the contract is declared now, the code lands with the platform build. Trigger phrases drawn from docs/handoff-triggers.md.

Purpose

Turn the AI Systems Director’s architecture into working, production-quality code with full cost visibility, and run eval harnesses faithfully — so the AI layer is real, not a whiteboard, and its quality is measured before it ships.

When to use

When the Director hands a spec to build (embedding generation, vector index, retrieval pipeline, LLM-call orchestration) or a harness to run. If the work is designing the harness or choosing a model, that is the Director’s (SOP-066 / SOP-065) — surface the need, do not redesign.

Flow

flowchart LR
  R["review requested<br/>(spec / harness)"] --> RD["Read spec;<br/>surface blockers"]
  RD --> B{"Build or<br/>run?"}
  B -->|build| IMPL["Smallest correct increment<br/>+ cost instrumentation"]
  IMPL --> VER["Verify: happy +<br/>≥2 failure modes"]
  VER --> FEED["usage data ready<br/>→ Financial Analyst"]
  B -->|run| RUN["Run all cases;<br/>collect scores"]
  RUN --> REP["review requested<br/>→ Director (interpret)"]
  VER --> ARCH["artifact approved<br/>→ Archivist"]
  REP --> ARCH
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  classDef d fill:#0f766e,color:#fff,stroke:#134e4a
  class R,RD,IMPL,VER,FEED,RUN,REP,ARCH s
  class B d

Text fallback: a review requested (spec or harness) arrives → read it fully and surface blockers before coding → build: implement the smallest correct increment with per-call cost instrumentation, verify the happy path plus ≥2 failure modes, emit the usage data ready instrumentation event to the Financial Analyst → run: execute all cases, collect scores, hand a written results report back to the Director to interpret (review requested) → archive produced docs on approval (artifact approved).

Steps

  1. Read the spec before a line of code. Identify ambiguities, missing constraints, implementation blockers; ask the Director before coding.
  2. Audit existing code (Grep/Glob/Read) — never introduce a new pattern where one fits; never duplicate Platform-Engineer-owned infra.
  3. Implement the smallest correct increment — match conventions; no speculative features.
  4. Wire cost instrumentation from the start — token/cost on every LLM and embedding call; this is the producer of usage data ready (not a follow-up task).
  5. Verify against the spec — happy path + ≥2 failure modes; report real output.
  6. For an eval run — stand up the runner, execute all cases, collect scores, produce a written results report. Do not modify harness logic — surface execution problems to the Director (SOP-066).
  7. Hand off — commit code; agree the AI-output contract with the Platform Engineer (when staffed); results report to the Director (review requested); produced docs to the Archivist (artifact approved); capture learnings.

Roles / approvers

Run by the AI Engineer under the AI Systems Director’s architecture. Architectural decisions arising from implementation are flagged to the Director for ADR authorship (the Director owns ADRs, SOP-065). The Director approves the AI Engineer’s eval report / implementation note for preservation (recorded, not an independence proof — R11).

Guardrails

  • Do not own prompt architecture, model selection, or eval design — the Director does (flag concerns before implementing, do not silently change the spec).
  • Do not modify harness logic — surface an unrunnable harness, do not redesign it.
  • Do not set scoring weights / ranking rubrics — Data & Analytics owns business scoring; implement mechanics only.
  • Do not write general application code — deliver a clean AI-output contract at the boundary; the Platform Engineer owns the application layer.
  • Synthetic / anonymized fixtures — never personal data (PA-only, ORG.md).
  • Never write into archive/; hand off to the Archivist.
  • Draw triggers from the registry; never invent inline.

Job aid: docs/units/ai-systems/job-aids/JA-067-run-eval-harness.md. See also: SOP-065 (model selection) · SOP-066 (eval design) · docs/knowledge/ai-systems-retrieval-review-2026-05-31.md (implementation backlog) · docs/sop/SOP-028-ledger-upkeep.md (consumes usage data ready) · docs/handoff-triggers.md · interface-contract-spec.md §3.1.