Skip to content

SOP-065 — Model selection + matrix upkeep

Guide

<!-- last-reviewed: 2026-06-01 · owner: AI Systems (AI Systems Director) -->

SOP-065 — Model selection + matrix upkeep

SOP · owner: AI Systems (AI Systems Director). Layer: SOP/Runbook (docs/canon.md). The decision matrix lives in docs/ai/model-selection.md (authored under this SOP) — not restated here. GR-1 (“best model always”, CLAUDE.md) is the standing prior.

Interface Contract

  • Inputs: AI-usage / token feed ← from <Data & Analytics usage producer> · trigger: usage data ready
  • Inputs: model-mix / cost data ← from Financial Analyst · trigger: finance summary ready
  • Outputs: interpreted model-mix AI-usage event → to Financial Analyst · trigger: usage data ready
  • Outputs: model-selection matrix / ADR added or moved → to Documentarian · trigger: document changed
  • Outputs: approved AI doc (matrix / ADR) → to Archivist · trigger: artifact approved
  • Handoff trigger (in): a new AI workload or model question arises, or Data & Analytics delivers an AI-usage/token feed, or the Financial Analyst delivers model-mix cost data
  • Handoff trigger (out): the tier is chosen with its rationale recorded in docs/ai/model-selection.md (and an ADR for a standing choice), and/or the interpreted model-mix usage event is delivered to the Financial Analyst for the ledger
  • Definition of Done before handoff:
    • [ ] Workload characterized (latency, quality ceiling, context length, call frequency, extended-thinking value)
    • [ ] Tier chosen against the matrix; GR-1 the prior — any deviation from the best available model names the tradeoff in writing
    • [ ] Standing choice recorded as an ADR (next free docs/adr/ number — never a stale-assumed one)
    • [ ] For the H4 edge: model-mix interpreted from the D&A feed; interpreted AI-usage event delivered to the Financial Analyst (usage data ready)
    • [ ] Matrix / ADR handed to the Archivist (0 drift); Documentarian notified if a doc was added/moved (document changed)
    • [ ] Learnings captured to docs/knowledge/ if any

H4 cross-unit wiring (COO log H4). This SOP carries the AI Systems half of the AI-usage-ledger seam. usage data ready governs Data & Analytics / requesting unit → Financial Analyst (docs/handoff-triggers.md); we consume D&A’s raw feed (input), interpret the model-mix, and emit an interpreted AI-usage event to the Financial Analyst (output) — SOP-028’s declared input (docs/sop/SOP-028-ledger-upkeep.md). We also consume finance summary ready (model-mix / cost data), matching SOP-029’s existing output to the AI Systems Director (docs/sop/SOP-029-utilization-report.md). No new trigger needed — both phrases are Wave 1 registry additions. The seam reconciles GREEN once D&A authors its producer side this wave; producer code lands when platform/ is built (AI Engineer instrumentation, SOP-067).

Purpose

Map every AI workload to the right model tier with a recorded rationale, so GR-1 is verifiable (not asserted) and any deviation from the best available model carries its tradeoff in writing — and interpret the org’s model-mix usage so Finance can record AI cost in the ledger.

When to use

When a new AI workload or “which model for X?” question arises; when a standing model choice needs an ADR; or when an AI-usage/token feed (usage data ready) or model-mix cost data (finance summary ready) arrives for interpretation.

Flow

flowchart LR
  Q["model question /<br/>usage data ready /<br/>finance summary ready"] --> C["Characterize<br/>workload"]
  C --> M["Apply matrix<br/>(GR-1 prior)"]
  M --> T["State tradeoff<br/>(if deviating)"]
  T --> REC["Record in matrix<br/>+ ADR if standing"]
  Q --> MIX{"H4 feed?"}
  MIX -->|yes| INT["Interpret model-mix"]
  INT --> FA["usage data ready<br/>→ Financial Analyst"]
  REC --> ARCH["artifact approved → Archivist<br/>+ document changed → Documentarian"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  classDef d fill:#0f766e,color:#fff,stroke:#134e4a
  class Q,C,M,T,REC,INT,FA,ARCH s
  class MIX d

Text fallback: a model question or an H4 feed arrives → characterize the workload → apply the matrix with GR-1 as the prior → state the tradeoff if deviating → record in docs/ai/model-selection.md and, for a standing choice, an ADR. If the input is an H4 feed, interpret the model-mix and emit an interpreted usage data ready event to the Financial Analyst. On approval, hand the matrix/ADR to the Archivist (artifact approved) and notify the Documentarian (document changed).

Steps

  1. Characterize the workload — latency sensitivity, quality ceiling, context length, call frequency, whether extended thinking adds value.
  2. Apply the matrix (docs/ai/model-selection.md) — map workload → tier. If the matrix lacks the row, add it under this SOP.
  3. State the tradeoff — what quality/capability a cheaper tier gives up, what cost a larger tier pays. GR-1 is the prior; justify any deviation in writing.
  4. Record the decision — update the matrix; for a standing architectural choice, draft an ADR at the next free docs/adr/ number (check the live ledger — the retrieval review’s “ADR-0014/0015” are now taken; the collision demon, playbook).
  5. For an H4 feed: consume the D&A usage data ready token feed (and/or the Financial Analyst’s finance summary ready cost data), interpret the model-mix, and emit an interpreted usage data ready AI-usage event to the Financial Analyst for the ledger. Interpretation only — do not build the telemetry pipeline (D&A) or approve spend (Finance).
  6. Hand off — matrix/ADR to the Archivist on approval (artifact approved); notify the Documentarian if a doc was added/moved (document changed); capture learnings.

Roles / approvers

Authored and run by the AI Systems Director. ADRs are approved by the Principal (docs/canon.md RACI). Spend implications go to Finance (we recommend; Finance approves — charter rule 8). The Director approves its own matrix update for preservation (recorded, not an independence proof — R11).

Guardrails

  • GR-1 is the prior — best model always; deviation names the tradeoff (charter rule 1).
  • Number ADRs against the live ledger — never a stale assumption (charter rule 7).
  • Interpret, do not build telemetry — D&A owns the pipeline; we consume its output.
  • Recommend cost, do not approve spend — Finance decides (charter rule 8).
  • Draw triggers from the registry; never invent inline.

Job aid: N/A — judgement-led; the matrix is the durable reference, not a cheat-sheet. See also: docs/ai/model-selection.md · CLAUDE.md (GR-1) · docs/units/ai-systems/charter.md · docs/sop/SOP-028-ledger-upkeep.md · docs/sop/SOP-029-utilization-report.md · docs/units/finance/charter.md · docs/handoff-triggers.md · interface-contract-spec.md §3.1.