<!– chunk-id: C-004 title: How I turned an unimplementable “retrieve chunks tagged to X” into a queryable skill ontology tags: [retrieval, ontology, taxonomy, tagging-schema, system-design, query-resolution] format-version: 1.0 –>

C-004 — How I turned an unimplementable “retrieve chunks tagged to X” into a queryable skill ontology

HOOK

My product concept said the platform would “retrieve chunks tagged to X” when a learner asks to learn X. An operational review caught the problem in one sentence: there is no field called X to query. The retrieval system — the core of the entire product — was specified as a feature that could not be built, because “X” was a free-text string the user types, not anything stored on a chunk.

MAP

• Inputs: A concept document whose retrieval loop reads “user asks to learn X → retrieve chunks tagged to X → score → assemble lesson,” plus three existing chunks (C-001, C-002, C-003) that had no subject tags at all — only positional script tags like [HOOK] and [RADIO] that describe where content sits in an episode, not what it teaches.
• Lap: Ran a thoroughness review of the concept against what it would take to build. Found that “tagged to X” had no schema behind it: no list of valid topics, no levels, no prerequisite structure, nothing queryable.
• Radio: “Retrieve chunks tagged to X” is not implementable as written — the positional tags that exist ([HOOK], [RADIO]) are not retrievable subject fields. There is no field named X on any chunk.
• Setup Change: Defined a hierarchical skill ontology — 25 nodes across 5 domains at depth-2 — gave every node a stable dotted ID, and added a skill_ids[] field to every chunk pointing at those IDs. Query resolution became: free-text → resolved skill_id → set-intersection match against chunk skill_ids[].
• Delta: Queryable skill nodes went from 0 to 25; chunks carrying a retrievable subject tag went from 0 to 3 (all three existing chunks retagged).

RADIO

The concept read like it already had a retrieval system. It named a “library,” it named “rubrics/test sets by ontology,” it tagged episode scripts with [HOOK] [MAP] [SCOREBOARD] [RADIO] [DELTA] [PIT] [CTA]. All of that created the impression that “retrieve chunks tagged to X” was a small implementation detail.

It was not. Read literally, the system had no subject tags of any kind. The bracket tags describe an episode’s structure — where the hook is, where the radio call is — not its topic. “X” in “tagged to X” referred to a skill the user types as a string. Nothing on a chunk could be matched against that string, because no chunk carried a subject field. The plain-language version: the spec described querying a column that did not exist in the table.

Red: The first implicit design — inherited straight from the concept — was to treat X as a free-text topic the user types and match it directly. That fails immediately: two chunks about the same skill drift to different strings (“prompt grounding” vs. “constraint instructions”), and a query for one silently misses the other. A free-text tag space with no controlled vocabulary is not a retrieval index; it is a place retrieval goes to return nothing.

SETUP CHANGE

Single change: replace the free-text “X” with a controlled skill ontology — a fixed set of nodes, each with a stable ID — and tag every chunk with the IDs it teaches (skill_ids[]).

Concretely, the ontology is:

• 5 top-level domains: prompt-eng, eval-testing, iteration, sys-design, production.
• Depth-2: each domain is a root node plus 3–4 subskill nodes — 25 nodes total (5 roots + 20 subskills).
• Dotted IDs: every node has a stable, machine-queryable ID, e.g. sys-design.least-privilege.
• skill_ids[] on every chunk: retrieval matches the session’s resolved skill IDs against each chunk’s skill_ids[] by set intersection.

Why this specifically, and why depth-2 and not depth-1: depth-1 (five domain roots only) cannot tell “I want to learn about evaluation” apart from “I want to learn how to design a test set” — both collapse to eval-testing, and retrieval at that resolution is no better than a single coarse topic tag. Depth-2 lets the second query resolve to eval-testing.test-set-design and pull the right chunk. Going deeper than depth-2 was deferred: a depth-3 node is only created once two or more chunks share a depth-2 node but teach meaningfully different subskills inside it. You add the node when an episode demands it — the content produces the node, not the reverse.

DELTA

Before: Retrieval spec was “retrieve chunks tagged to X.” Queryable subject fields on a chunk: 0. Valid skill nodes defined: 0. X was a free-text string with no controlled vocabulary. Existing chunks (C-001, C-002, C-003) carried positional tags only — nothing a topic query could match.

After: A 25-node depth-2 ontology (5 domain roots + 20 subskill nodes). Every chunk carries skill_ids[] drawn from those 25 IDs; all three existing chunks were retagged (C-001 → 3 production nodes; C-002 → 4 sys-design/production nodes; C-003 → 2 iteration nodes). Free-text queries resolve to a node via a two-stage resolver: deterministic keyword/alias match first, LLM classification only as a fallback, and the classifier is constrained to return a valid ID from the list or the literal "no-match" — never an invented one.

Observable: The ontology is docs/skill-ontology.md (25 nodes, with a node-ID quick-reference appendix). The retagged metadata is visible in each chunk’s sidecar JSON skill_ids field and in chunks/index.json. A query for a skill now has a concrete field to match against; before, it had none.

PIT WALL NOTES

• “Tagged to X” is not a spec until X is a column. A retrieval feature is only real when you can name the field the query matches against. If the noun in “retrieve by X” does not correspond to a stored, controlled field on the record, you have described a wish, not a query. Find the missing column before you build the search.
• A controlled vocabulary beats free text the moment you have two authors or two sessions. Free-text tags drift — the same idea gets tagged three ways and retrieval fragments. A fixed node list with stable IDs is the cheapest insurance against silent retrieval misses. Pay the taxonomy cost up front; it is smaller than the cost of queries that quietly return nothing.
• Constrain the LLM resolver to a closed set, or it will invent IDs. When you fall back to an LLM to map a fuzzy query to a node, the prompt must force the output to be a valid ID from the provided list or an explicit "no-match". An unconstrained classifier will hallucinate a plausible-looking ID that does not exist, and a downstream lookup on a fabricated ID fails in a way that is hard to trace.

CTA

Use this when a spec says “retrieve / route / classify by X” and you cannot yet name the field X lives in.

The artifact is (1) a SkillNode schema you can paste and fill, and (2) the two-stage query resolver — deterministic-first, LLM-fallback — with the exact closed-set classification prompt.

Part 1 — Node schema (paste and define your nodes)

interface SkillNode {
  id:             string;            // stable dotted ID: "domain.parent.child"
  label:          string;            // human-readable, sentence case
  domain:         string;            // one of your top-level domain IDs
  prerequisites:  SkillNode["id"][]; // IDs that must be understood first
  subskills:      SkillNode["id"][]; // child IDs (populated at depth-2+)
  misconceptions: string[];          // named wrong beliefs a lesson must address
  aliases:        string[];          // common phrasings for keyword resolution
  level:          "L1" | "L2" | "L3";
}

Rules that kept the tree honest:

• Define 3–5 domains, expand to depth-2 before launch (root + ≥2 subskills each).
• A node’s id never changes once content references it.
• Add a depth-3 node only when ≥2 chunks share a depth-2 node but teach different subskills.

Part 2 — Two-stage query resolver

resolve(query):
  q = lowercase(query)

  # Stage 1 — deterministic. Match q against every node's label + aliases.
  hits = [node for node in NODES if q matches node.label or any(q matches a for a in node.aliases)]
  if len(hits) == 1:
      return hits[0].id

  # Stage 2 — LLM fallback ONLY when stage 1 is 0 or >1 matches.
  return classify_with_llm(query, NODES)   # closed-set prompt below

Part 3 — Closed-set classification prompt (the anti-hallucination clause)

You map a learner's free-text query to exactly one skill node ID.

Query: "{{QUERY}}"

Valid node IDs (choose exactly one, or "no-match"):
{{for each node}} - {{node.id}} — {{node.label}} — addresses: {{node.misconceptions}}

Return ONLY one of:
  - a single ID copied verbatim from the list above, or
  - the literal string "no-match"

Do not invent an ID. Do not return an ID that is not in the list. If no node
fits, return "no-match" so the system can escalate instead of guessing.

A "no-match" result should escalate (ask a human / flag the gap), never auto-create a node.

SELF-SCORE

Rubric

Total: 94/100

Passing threshold: >= 70/100

Verdict: PASS

Disqualifier check: No dimension scores zero. Node count (25), domain count (5), depth-2, and the per-chunk retag counts are all traceable to docs/skill-ontology.md and the chunk sidecar JSONs. No invented tools or APIs. Auto-fail not triggered.

Honest weakness: The DELTA reports structural facts (0 → 25 nodes, 0 → 3 retagged chunks), not a retrieval-accuracy measurement. I did not run a before/after retrieval test on a query set, because no live retrieval system exists yet — so the claim is “the field now exists to query,” not “retrieval precision improved by N%.” That stronger claim is unearned until there is a running index to measure.