Skip to content

SOP-066 — Eval design

Guide

<!-- last-reviewed: 2026-06-01 · owner: AI Systems (AI Systems Director) -->

SOP-066 — Eval design

SOP · owner: AI Systems (AI Systems Director). Layer: SOP/Runbook (docs/canon.md). The Director designs the harness; the AI Engineer executes it (SOP-067). Telemetry is Data & Analytics’ territory; we own the eval logic on top of their signals — not restated here.

Interface Contract

  • Inputs: prompt change / new AI feature to evaluate ← from self (or SOP-064 / SOP-065) · trigger: review requested
  • Outputs: eval harness (methodology + cases) → to AI Engineer · trigger: review requested
  • Outputs: eval harness for CI integration → to Process Engineer · trigger: review requested
  • Outputs: approved eval report → to Archivist · trigger: artifact approved
  • Handoff trigger (in): a prompt change is about to ship, or a new AI feature needs a correctness harness
  • Handoff trigger (out): the harness (success criteria + method + cases) is authored and handed to the AI Engineer to run and the Process Engineer to wire into CI
  • Definition of Done before handoff:
    • [ ] Success criteria defined — specific enough that a non-author can score
    • [ ] Eval method chosen (LLM-as-judge / human review / automated assertions / hybrid) and matched to stakes
    • [ ] ≥5 cases per significant behavior — positive, negative, edge, adversarial
    • [ ] Runner contract specified (input shape, scoring output, pass/fail threshold) so the AI Engineer can execute it (SOP-067)
    • [ ] Baseline established — before/after comparison defined; no prompt change ships without it
    • [ ] Harness handed to the AI Engineer + Process Engineer (review requested); report archived on completion (artifact approved)
    • [ ] Learnings captured to docs/knowledge/ if any

Trigger phrases drawn from docs/handoff-triggers.md: review requested (handing the harness to the AI Engineer / Process Engineer for execution and CI wiring) and artifact approved (the eval report to the Archivist fan-in sink).

Purpose

Build the harness that scores prompt quality, agent behavior, and AI-feature correctness, so no prompt change ships without a before/after comparison and no AI feature serves output a harness cannot score.

When to use

Before any prompt change ships (the before/after gate), or when a new AI feature (generation, classification, retrieval) needs a correctness harness. If the harness already exists and just needs running, that is SOP-067 (AI Engineer).

Flow

flowchart LR
  R["review requested<br/>(prompt change / feature)"] --> SC["Define success<br/>criteria"]
  SC --> ME["Choose eval<br/>method"]
  ME --> CA["Write ≥5 cases<br/>(pos/neg/edge/adversarial)"]
  CA --> RC["Specify runner<br/>contract + baseline"]
  RC --> HO["review requested →<br/>AI Engineer (run) + Process Engineer (CI)"]
  HO --> ARCH["artifact approved<br/>→ Archivist (report)"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  class R,SC,ME,CA,RC,HO,ARCH s

Text fallback: a review requested (prompt change or new feature) arrives → define success criteria a non-author can score → choose the eval method matched to stakes → write at least five cases per significant behavior (positive, negative, edge, adversarial) → specify the runner contract and the before/after baseline → hand the harness to the AI Engineer to run (SOP-067) and the Process Engineer to wire into CI (review requested) → archive the report on completion (artifact approved).

Steps

  1. Define success criteria — what a correct, high-quality output looks like and the failure modes, specific enough that a non-author can score it.
  2. Choose the eval method — LLM-as-judge, human review, automated assertions, or hybrid; match rigor to stakes (a customer-facing generation prompt warrants more than an internal classifier).
  3. Write the cases — minimum five per significant behavior: positive examples, negative examples, edge cases, adversarial inputs.
  4. Specify the runner contract — input shape, scoring output, pass/fail threshold — so the AI Engineer can stand it up without redesigning it (SOP-067).
  5. Establish the baseline — define the before/after comparison; a prompt change never ships without one (charter rule 2).
  6. Hand off — the harness to the AI Engineer (run) and the Process Engineer (CI) via review requested; the eval report to the Archivist on approval (artifact approved); capture learnings.

Roles / approvers

Designed by the AI Systems Director; executed by the AI Engineer (SOP-067); CI-wired by the Process Engineer. The Director interprets results; the AI Engineer does not. The Director approves its own report for preservation (recorded, not an independence proof — R11).

Guardrails

  • No ship without an eval (charter rule 2).
  • Director designs; AI Engineer runs — keep the design/execution split clean (the AI Engineer surfaces an unrunnable harness, does not redesign it).
  • Eval logic only, not telemetry — Data & Analytics owns the pipelines; we own the scoring logic on top.
  • Synthetic / anonymized fixtures — never personal data (PA-only, ORG.md).
  • Draw triggers from the registry; never invent inline.

Job aid: the execution card is the AI Engineer’s (docs/units/ai-systems/job-aids/JA-067-run-eval-harness.md); design is judgement-led (N/A as a standalone Director job aid). See also: SOP-067 (execution) · docs/units/ai-systems/charter.md · docs/generation-policy.md §3 (the scoring path) · docs/handoff-triggers.md · interface-contract-spec.md.