Skip to content

SOP-062 — Pipeline upkeep (incl. the AI-usage / token feed)

Guide

<!-- last-reviewed: 2026-06-01 · owner: Data & Analytics (Analytics Engineer) -->

SOP-062 — Pipeline upkeep (incl. the AI-usage / token feed)

SOP · owner: Data & Analytics (Analytics Engineer). Layer: SOP/Runbook (docs/canon.md). The AI-usage feed’s field meanings and frozen column order live in docs/financial-system.md — not restated. Plan-phase / phase-gated note: no product data layer or PostHog exists yet (ADR-0006); the Analytics Engineer is hired when real learner data flows. Until then the DA Director runs this directly, stub-first against the planned platform.

Interface Contract

  • Inputs: pipeline spec / measurement plan ← from Data & Analytics Director · trigger: briefing posted
  • Outputs: AI-usage / token-count event feed → to Financial Analyst · trigger: usage data ready
  • Outputs: pipeline / data-quality artifact → to Archivist · trigger: artifact approved
  • Handoff trigger (in): a pipeline spec is approved, or a monitoring cycle is due
  • Handoff trigger (out): the AI-usage feed (12-column AI-ledger shape) is ready for the Financial Analyst to record; the pipeline artifact is ready for preservation
  • Definition of Done before handoff:
    • [ ] Pipeline implemented/verified against the DA Director’s spec — names, keys, types exact (no drift)
    • [ ] AI-usage feed serialized to the frozen 12-column shape (id, ts, ledger, unit_or_agent, venture, phase, type, qty, unit, usd_equiv, source, note)
    • [ ] AI rows carry ledger=ai, qty = token/request count, unit=tokens (or requests), model in note; source and ts present (a row missing either is a reliability defect, financial-system.md:38-39)
    • [ ] usd_equiv left empty / unset — DA emits counts; Finance computes the dollar (qty × rate). A dollar in the feed is a boundary defect
    • [ ] Data-quality check run (volume / latency / required-field completeness); anomalies escalated to the DA Director immediately
    • [ ] Learnings captured to docs/knowledge/ if any
    • [ ] Approved against the frozen exemplar; handed to the Archivist (0 drift)

Trigger usage data ready is registered (docs/handoff-triggers.md, Wave-1 block — governs Data & Analytics / requesting unit → Financial Analyst). This output is the raw-counts producer half of the H4 seam (token counts; no usd_equiv). It is absorbed by the Financial Analyst’s declared fan-in sink in SOP-028 (money transaction / AI-usage event ← from <usage-data producer> · usage data ready), which also absorbs the AI Systems interpreted half (SOP-065) and the AI Engineer per-call instrumentation (SOP-067) — so all three reconcile as matched (fan-in), not a false OVERLAP (§3.1 R1). No new trigger is proposed.

Purpose

Keep the telemetry pipeline running to spec — ingestion, dashboards, funnels, cohorts — monitor data quality, and emit the AI-usage / token-count feed to the Financial Analyst in the shape Finance already froze, so the AI-Usage ledger has a real source and the H4 seam closes on the producer side.

When to use

A pipeline spec is approved and needs implementing/maintaining; a data-quality monitoring cycle is due; or AI-usage events need to be emitted for the ledger.

Flow

flowchart LR
  A["Spec approved /<br/>monitoring due"] --> B["Implement / verify<br/>pipeline to spec"]
  B --> C["Collect AI-usage<br/>events (token counts)"]
  C --> D["Serialize to frozen<br/>12-column AI-ledger shape"]
  D --> E{"usd_equiv empty?<br/>source &amp; ts present?"}
  E -->|no| X["Fix — counts only,<br/>required fields present"]
  E -->|yes| F["Emit feed →<br/>Financial Analyst<br/>(usage data ready)"]
  B --> G["Run data-quality<br/>checks"]
  G --> H{"Anomaly?"}
  H -->|yes| I["Escalate to<br/>DA Director now"]
  F --> J["Hand artifact<br/>to Archivist"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  classDef d fill:#0f766e,color:#fff,stroke:#134e4a
  class A,B,C,D,X,F,G,I,J s
  class E,H d

Text fallback: spec approved / monitoring due → implement or verify the pipeline to spec → collect AI-usage events (token counts) → serialize to the frozen 12-column AI-ledger shape → check: is usd_equiv left empty and are source and ts present? (no → fix: counts only, required fields present; yes → emit the feed to the Financial Analyst as usage data ready). In parallel, run data-quality checks → on any anomaly, escalate to the DA Director immediately → hand the pipeline artifact to the Archivist.

Steps

  1. Read the spec in full (Read/Grep). Identify every ambiguity; surface it to the DA Director before building — never implement an ambiguous requirement.
  2. Audit what exists — current pipeline/config and related files.
  3. Implement / verify the pipeline faithfully: event names, property keys, types, cardinality, destination routing, privacy annotations — exactly as specified.
  4. Build the AI-usage feed. For each AI-usage event, write a row in the frozen 12-column shape: ledger=ai, qty=count, unit=tokens/requests, model in note, unit_or_agent, venture, phase, ts, source. Leave usd_equiv empty — Finance computes it. Field meanings: docs/financial-system.md (linked, not restated).
  5. Emit the feed to the Financial Analyst with trigger usage data ready.
  6. Run data-quality checks — volume range, latency, required-field completeness. Flag any metric outside threshold (volume drop >20%, required-field missing >5%, any event silent).
  7. Escalate anomalies immediately to the DA Director — do not hold for a scheduled report.
  8. Hand off the pipeline/data-quality artifact to the Archivist; capture any learning.

Roles / approvers

Run by the Analytics Engineer (or the DA Director while the role is unhired — phase-gated). The Engineer implements and verifies; the DA Director owns the taxonomy, spec, and any schema decision. Application capture code is the Platform Engineer’s — the Engineer writes the brief (SOP-063 coordination) and verifies.

Guardrails

  • Schema fidelity is non-negotiable — exact names/keys/types; a one-character drift is a data split.
  • Counts only, never dollars — usd_equiv is Finance’s; emitting it is a defect.
  • Required fields present — a row missing source, ts, or usd_equiv’s placeholder is a reliability defect per financial-system.md; for DA feed rows, usd_equiv is intentionally empty and Finance fills it.
  • No schema improvisation — a decision not in the taxonomy is a stop-and-escalate.
  • No application code — that is the Platform Engineer.
  • No Principal personal data — PA-only; follow the DA Director’s privacy annotations.
  • Do not restate docs/financial-system.md — link to it.

Job aid: JA-062-pipeline-quality-checklist.md. See also: SOP-060 (event schema) · SOP-063 (instrumentation verification) · SOP-028 (Finance ledger upkeep — the consuming side) · docs/financial-system.md.