Skip to content

SOP-061 — Scoring & ranking definition

Guide

<!-- last-reviewed: 2026-06-01 · owner: Data & Analytics (Data & Analytics Director) -->

SOP-061 — Scoring & ranking definition

SOP · owner: Data & Analytics (Data & Analytics Director). Layer: SOP/Runbook (docs/canon.md). The retrieval/ranking flywheel lives in docs/concept.md (lines 94–114) — not restated. Plan-phase note: no outcome data flows yet (ADR-0006); this authors the rubric stub-first against the planned platform.

Interface Contract

  • Inputs: ranking / scoring need ← from self (DA Director) or Product / Innovation · trigger: briefing posted
  • Inputs: chunk-tagged manifest (to score / rank) ← from <Growth (Video Producer)> · trigger: chunk manifest ready
  • Outputs: scoring rubric + ranking spec → to Archivist · trigger: artifact approved
  • Outputs: rubric / definition change notice → to Documentarian · trigger: document changed
  • Handoff trigger (in): content/exercises/learner-progress need an evaluation or ranking rule
  • Handoff trigger (out): the rubric, weights, threshold, and edge cases are approved and ready for preservation / implementation
  • Definition of Done before handoff:
    • [ ] Inputs enumerated (completion, quiz, rating, recall, time-on-task, freshness, fit)
    • [ ] The formula or rubric is explicit, with weights stated and justified
    • [ ] The top-K quality / freshness / fit threshold is specified (concept.md:108)
    • [ ] Edge cases defined: cold-start, sparse data, ties, stale content, sub-threshold candidates
    • [ ] Acceptance criterion for a passing implementation is stated (falsifiable)
    • [ ] The feedback path is named (which signals flow back to re-rank — concept.md:104)
    • [ ] No model/prompt tuning (that is AI Systems); no dollars
    • [ ] Learnings captured to docs/knowledge/ if any
    • [ ] Approved against the frozen exemplar; handed to the Archivist (0 drift)

Purpose

Define trustworthy scoring and ranking logic — the rubric, weights, threshold, and edge cases — that drive content ranking and adaptive sequencing, so the retrieval flywheel ranks candidates by outcome evidence rather than guesswork, and the feedback loop has a defined re-ranking path.

When to use

Any time content, exercises, or learner progress need a defined evaluation or ranking rule — including the retrieval step (“score + rank candidates”, concept.md:101) and the top-K quality gate (“below threshold → fallback”, concept.md:108).

Flow

flowchart LR
  A["Name the ranking<br/>question"] --> B["Enumerate inputs<br/>(outcome signals)"]
  B --> C["Specify rubric /<br/>formula + weights"]
  C --> D["Set top-K<br/>threshold"]
  D --> E["Define edge cases<br/>(cold-start, sparse, ties)"]
  E --> F["State acceptance<br/>criterion"]
  F --> G["Name the<br/>feedback path"]
  G --> H["Hand to Archivist"]
  classDef s fill:#1d4ed8,color:#fff,stroke:#1e3a8a
  class A,B,C,D,E,F,G,H s

Text fallback: name the ranking question → enumerate the outcome-signal inputs → specify the rubric/formula and its weights → set the top-K quality threshold → define edge cases (cold-start, sparse data, ties, stale, sub-threshold) → state the acceptance criterion for a passing implementation → name the feedback path (which signals flow back to re-rank) → hand to the Archivist.

Steps

  1. Name the ranking question. What is being ranked, for whom, to what end?
  2. Enumerate inputs. The outcome signals: completion, quiz score, user rating, recall rate, time-on-task, freshness, fit. Tie each to an event in the SOP-060 taxonomy (no input without a measurable source).
  3. Specify the rubric/formula. State the weights and justify them; a weight with no rationale is a vanity weight.
  4. Set the threshold. The top-K quality / freshness / fit cutoff below which the fallback path triggers (concept.md:108).
  5. Define edge cases. Cold-start (no outcome data), sparse data, ties, stale content, all-candidates-sub-threshold. An undefined edge case is a future incident.
  6. State the acceptance criterion. The falsifiable condition a passing implementation must meet (e.g. “given fixture set F, top-K matches expected rank within tolerance T”).
  7. Name the feedback path. Which signals flow back to re-rank, and how often (concept.md:104,113-114).
  8. Hand off. Rubric + spec to the Archivist; definition-change notice to the Documentarian; capture any learning.

Roles / approvers

Run by the Data & Analytics Director. Implementation is Engineering’s (DA does not write product code); measurement-strategy disputes escalate to the Principal. Scoring that depends on an AI-produced quality signal measures the model output — it does not tune the model (AI Systems’ boundary).

Guardrails

  • No vanity weight — every weight carries a rationale tied to an outcome.
  • No model/prompt tuning — DA measures AI output quality; AI Systems owns the model.
  • No dollars — scoring is about learning outcomes, not money (Finance’s domain).
  • Every edge case defined — cold-start and sparse-data are the default early state.
  • No Principal personal data — PA-only.
  • Do not restate docs/concept.md — link to it.

Job aid: JA-061-scoring-rubric-checklist.md. See also: SOP-060 (event schema — the input signals) · docs/concept.md · docs/units/data-analytics/charter.md.