AI Systems Retrieval Review — IDEA-0001 Platform Infrastructure

Prepared by: AI Systems Director

Date: 2026-05-31

Audience: Engineering Director (first-route work), CPTO

Status: Recommendation — ready for Engineering implementation

Context

ADR-0013 established platform/ as a Next.js 15 App Router surface for the IDEA-0001 “Show the Work” learning platform. The ADR explicitly deferred two infrastructure decisions: chunk storage and learner state persistence. This review examines the retrieval infrastructure implications of the full spec set — ADR-0013, docs/instrumentation.md, docs/skill-ontology.md, docs/generation-policy.md, and specs/IDEA-0001-show-the-work.json — and delivers concrete recommendations for each deferred question.

1. CHUNK STORE

Current state

chunks/index.json is a flat JSON index containing 3 chunk records. Each record carries: chunk_id, skill_ids[], level, rubric_score, last_validated, source, deprecated, and a file path to the .md content. The retrieval query pattern specified in docs/instrumentation.md §3 is: match on skill_ids set intersection + level equality, then rank by rubric_score desc, with freshness_days as a freshness gate. K=5 candidates retrieved per session.

Recommendation: JSON index is sufficient through ~200 chunks

A vector store is not needed for v1 and should not be introduced preemptively. Here is why:

The query is structured, not semantic. Every retrieval operation starts with a resolved skill_id — a validated dot-notation identifier from a 25-node ontology. The resolution from free-text query to skill_id already happens upstream via the two-stage keyword + LLM classification defined in docs/skill-ontology.md §5 Q3. By the time the chunk store is queried, the input is a structured key (skill_id + level), not a natural-language string. Semantic similarity search adds no value to a lookup that is already operating on canonical identifiers.

The ranking function is deterministic. rubric_score is a float in [0, 100]. freshness_days is an integer derived from last_validated. Both are fields in the existing index record. Ranking K=5 candidates on these two signals is a sort operation — O(n) scan of the index, not a nearest-neighbor search.

The library scale is tiny and grows slowly. The v1 ontology has 25 nodes. Episodes are solo-produced; the spec’s success criteria require 10 passing chunks within 60 days of first publish. A realistic 12-month library ceiling for a solo creator is 50–100 chunks. Even at 200 chunks, a linear scan of a JSON index (each record is ~300 bytes) reads ~60 KB — well within a single Node.js import or a server-side fs.readFile call.

Threshold where a vector store becomes necessary:

A vector store becomes justified when any of the following conditions is met:

1. Chunk count exceeds 200 — at this scale, a linear scan of the JSON index on every session start introduces measurable latency (>50ms for the index read alone on cold starts), and the ontology likely has enough depth-3 nodes that a “nearby skill” fallback (see below) becomes valuable.

2. The skill ontology grows beyond ~60 nodes — if depth-3 nodes are added and learners begin querying at a granularity the two-stage classifier cannot resolve to a single node reliably, vector similarity over chunk skill_ids embeddings would provide a graceful fallback instead of the current hard no-match escalation.

3. Cross-skill queries become a product requirement — e.g. “show me content that bridges eval-testing and iteration” is a natural-language query that cannot be resolved to a single skill_id. If the platform supports compound queries, semantic retrieval over chunk content becomes necessary.

4. The generation policy produces a backlog of generated chunks that diverge from ontology nodes — generated chunks may address subtleties not captured by the 25-node tree, making exact-match retrieval imprecise.

None of these thresholds apply today or within the v1 horizon. Keep chunks/index.json as the canonical chunk store. Extend it with the misconceptions_addressed[] field (already specified in docs/instrumentation.md §2 but not yet present in the index records) so that the misconception-boost signal is available at retrieval time without reading individual chunk .json files.

One extension worth making now: add a chunk_type field to the index records to distinguish HOOK-lead chunks from other types. The spec calls out “HOOK chunk + RADIO chunk from different source episodes” as a valid multi-chunk assembly. If chunk_type is not indexed, the assembly logic must open each .md file to determine type, which defeats the purpose of the index.

2. DATABASE SPLIT

Recommendation: Two ADRs, not one

ADR-0013 deferred “database: chunk storage and learner progress persistence” as a single decision. It should be split into two separate ADRs. The reason is that the two stores have fundamentally different access patterns, retention requirements, and scaling profiles — forcing them into one decision produces an over-constrained choice.

Store A — Chunk metadata (the index)

• Access pattern: read-heavy, write-rare. Chunks are written when a new episode lands (weekly at most) or when a generated chunk is accepted. Reads happen on every session start (K=5 candidates retrieved).
• Retention: indefinite. Deprecated chunks are soft-deleted, not purged. The full chunk history is a content asset.
• Consistency requirement: eventual is acceptable. A newly accepted chunk not appearing in retrieval for one session is a minor quality issue, not a correctness failure.
• Scale: 200 chunks is the 12-month ceiling. This is a file-system or lightweight key-value problem, not a database problem. A flat JSON index or a small Postgres table (same Neon instance as Store B) are both appropriate. No vector extensions needed for v1 (see §1).
• Recommendation: Continue with chunks/index.json for v1. When the platform requires a real database (auth, per-user state), migrate the chunk index into a chunks table in the same Neon Postgres instance — no separate store required. Capture this as ADR-0014: Chunk metadata store.

Store B — Learner state (LessonSession)

• Access pattern: write-on-session-open + write-on-session-close, read on per-user history queries. Append-only by spec (docs/instrumentation.md §1).
• Retention: 90-day hard delete on raw session rows; aggregated chunk weights kept indefinitely (docs/instrumentation.md §5 Q4). This retention differential means two tables with different lifecycle management, or a single table with a TTL policy.
• Consistency requirement: strong. user_id is a real authenticated identity; per-user learning history (which chunks a user passed) is a correctness requirement for adaptive routing, not a nice-to-have.
• Scale: user_id is required (authenticated-only, §5 Q1), so LessonSession rows are tied to real accounts. At early scale this is negligible; the schema is designed for Postgres row storage.
• Recommendation: A single Neon Postgres instance with two tables: lesson_sessions (90-day TTL on raw rows) and chunk_weights (indefinite). Auth integration (deferred per ADR-0013) gates this store — do not build LessonSession persistence until auth is in place. Capture this as ADR-0015: Learner state persistence.

Why separate ADRs matter: the chunk store decision (ADR-0014) can be made and implemented now — it unblocks the first retrieval route Engineering is building. The learner state decision (ADR-0015) is blocked on the auth ADR. Conflating them into one ADR would either hold up the chunk store decision waiting for auth, or force a premature auth choice.

3. EMBEDDING STRATEGY

Recommendation: No embeddings in v1. Pre-compute node embeddings when threshold is crossed.

Given the recommendation in §1 (JSON index sufficient through 200 chunks), no embedding infrastructure is needed for v1.

When the chunk count or ontology size crosses the thresholds in §1, here is the embedding strategy that fits the system’s structure:

What to embed: Embed the chunk’s skill_ids[] + misconceptions_addressed[] concatenated with the chunk’s HOOK section (first 200 tokens of content). Do not embed the full chunk body. The reason: the retrieval problem is not “find the most semantically similar chunk to this raw query.” The query is already resolved to a skill_id before hitting the chunk store. The embedding is needed only for the fallback case — when a query resolves to no-match or to an ambiguous multi-node result, and the system needs to find the closest available chunk across node boundaries.

Embedding model: Use Anthropic’s embedding API if available at v2 time, or OpenAI text-embedding-3-small. The choice is not load-bearing for v1 since no embeddings are generated yet. What matters is that embeddings are pre-computed at chunk ingestion time (not at query time), stored as a vector column in Postgres (pgvector extension on Neon), and updated only when a chunk’s skill_ids or misconceptions_addressed change.

Why skill_ids are already structured metadata: the 25-node ontology with dot-notation IDs is a strong prior that reduces the embedding problem significantly. A vector store would not be doing semantic retrieval over raw text; it would be doing nearest-neighbor lookup over a space where node IDs are already the primary discriminant. This means the embedding space will be well-clustered by domain (all production.* nodes near each other) and the effective dimensionality is low. A small embedding model at low dimensions is appropriate — no need for a high-capacity embedding model.

Avoid hybrid search in v1. Hybrid search (BM25 + vector) adds operational complexity with no retrieval quality payoff at this scale. The structured metadata retrieval path is exact and sufficient.

4. RANKING STRATEGY

Recommendation: rubric_score + freshness_days is sufficient for v1. Do NOT introduce a learned ranking signal on day one.

The ranking function specified in docs/instrumentation.md §3 is:

1. Filter by skill_ids set intersection + level equality + rubric_score >= 70 + deprecated = false
2. Sort by rubric_score desc
3. Apply freshness gate: if top candidate has freshness_days > 180, fire generation trigger

This is a deterministic, interpretable ranking function. It is the right choice for v1 for three reasons:

The library has 3 chunks. There is no ranking problem to solve at this scale. Every skill node that has coverage at all has at most one or two chunks. The K=5 candidate retrieval will return fewer than K for most queries. A learned ranking model trained on 3 chunks would be overfit noise.

The signal integrity rules require minimum 5 accepted sessions per chunk before weight updates. The chunk_weight_update guard in docs/instrumentation.md §4 Rule 3 means the ranking feedback loop cannot fire until each chunk has accumulated 5 qualifying sessions. The platform will be in production for weeks before any chunk crosses this threshold. Building a learned ranker before the minimum signal threshold is met is premature by definition.

Rubric score is a proxy for learner value that is already calibrated. docs/generation-policy.md §5 Q2 specifies a 10-sample calibration pass before auto-serve is enabled. The rubric score is not an arbitrary quality proxy — it is a human-calibrated measure of clarity, correctness, usefulness, ethos, and completeness. Using it as the primary ranking signal is the correct choice until learner outcome signals are statistically meaningful.

When to introduce a learned signal: After a chunk accumulates 5+ accepted sessions with outcome_pass data, the aggregated chunk_weights store (defined in docs/instrumentation.md §5 Q4) contains three signals: mean_outcome_quiz, mean_outcome_rating, and outcome_pass_rate. At that point, a simple weighted blend is appropriate:

ranking_score = (0.6 × rubric_score)
              + (0.2 × outcome_pass_rate × 100)
              + (0.1 × mean_outcome_quiz)
              + (0.1 × mean_outcome_rating × 20)
              - freshness_penalty

where freshness_penalty = max(0, (freshness_days - 90) × 0.1) to begin penalizing staleness before the 180-day hard gate triggers.

Do not introduce this blend until at least 3 chunks have crossed the 5-session minimum. Before that point, the blend weight would be dominated by rubric_score anyway, and the added complexity introduces bugs for no quality gain.

One addition worth making now: the misconception-boost signal specified in docs/instrumentation.md §2 (a chunk addressing at least one misconception from the matched SkillNode scores higher) is cheap to implement as a sort tie-breaker within the rubric_score-sorted results. This is pure metadata and does not require learner outcome data. Add it to the v1 ranking as a tiebreaker: ORDER BY rubric_score DESC, misconception_match_count DESC.

5. CLAUDE API INTEGRATION POINT

Recommendation: Server Action for generation; Route Handler for auto-score

The generation policy (docs/generation-policy.md) contains two Claude API call types:

A. Chunk generation (Section 2 generation prompt)

Use a Server Action.

Rationale: chunk generation is triggered within a learner session flow — the user submits an intent query, the retrieval step fires, the generation trigger evaluates, and if triggered, generation runs. This is a user-initiated, session-scoped operation. Server Actions are the idiomatic Next.js 15 App Router pattern for mutations triggered by user interaction. The generation prompt is a 2000-token max call (per spec) on the most capable Claude model available (GR-1). The response is not streamed to the learner — the learner receives a “best available candidate” while generation runs asynchronously and the result is queued for author review (generation-policy §5 Q3). This means the Server Action can fire the Claude API call, write the result to the review queue, and return the best-available candidate to the client — all without holding a streaming response open.

Do not use an edge function for generation. Edge functions have a 25-50 second execution timeout depending on the Vercel plan tier, and a 2000-token generation call at standard Claude API latency (typically 8-15 seconds on complex prompts) is at risk of hitting that ceiling under load. Server Actions on Vercel run as Node.js lambdas with a longer timeout (configurable to 300 seconds on Pro).

B. Auto-score / ethos alignment gate (Section 3 automated scoring)

Use a Route Handler (/api/chunks/score).

Rationale: the ethos alignment gate (Step 3 of the gate procedure) applies the rubric’s automated proxies to a generated chunk. This is an internal pipeline step, not a user-facing interaction. A Route Handler is the correct pattern for internal API calls between platform services — the generation pipeline (Server Action) calls the scoring endpoint, the scoring endpoint calls Claude for the LLM-as-judge dimension checks (Dimension 2 correctness flagging), and returns a structured score object. Route Handlers are easy to call from both Server Actions and from the author’s review queue UI.

The auto-score endpoint also serves the Human Review Queue: when an author uses the Revise flow (generation-policy §4), the revised chunk is re-scored by calling the same /api/chunks/score Route Handler. A Route Handler is the right abstraction for a reusable scoring service.

C. Skill classification fallback (Stage 2 LLM classification, skill-ontology §5 Q3)

Use a Server Action.

Rationale: the two-stage skill resolution (keyword match → LLM fallback) is triggered at session start, inline with the user’s query submission. It is a user-facing latency-sensitive call. The LLM classification prompt is small (the query + 25 node labels + misconceptions — roughly 1500 tokens) and the expected response is a single skill_id string. A Server Action keeps this in the same request lifecycle as the session creation, avoids a round-trip through a separate endpoint, and benefits from Next.js’s built-in request deduplication for identical queries in the same render pass.

Summary table:

Prompt caching note: the generation system prompt in docs/generation-policy.md Section 2 is 1,400+ tokens and is identical across all generation calls (only the {{DOUBLE_BRACES}} variables change). Mark the static system prompt as a cache-breakpoint prefix. At the current generation rate (max 3 auto-generated chunks per SkillNode per 7-day window × 25 nodes = at most 75 generation calls per week at full quota), prompt caching on the generation system prompt will reduce input token costs by approximately 70-80% on cache hits. This is a day-one implementation, not a future optimization — the caching breakpoint should be placed at the end of the static system prompt content, before the === WHAT YOU ARE GENERATING === section where the variables begin.

Implementation Order for Engineering

1. Extend chunks/index.json — add misconceptions_addressed[] and chunk_type fields to existing records and to the index schema. This makes the ranking and assembly logic complete without a database. (No ADR required — this is an index schema update, not an architectural decision.)

2. Build the retrieval route — a Server Component in platform/ that reads chunks/index.json, filters by skill_ids + level, sorts by rubric_score desc with misconception tiebreaker, applies the freshness gate, and returns K=5 candidates. This is pure TypeScript — no database, no vector store, no Claude API call. The generation trigger check is a condition on the filtered result.

3. Wire the skill classification Server Action — two-stage: keyword match against node label fields (loaded from docs/skill-ontology.md at build time or as a static import), then LLM fallback via Claude API. Add prompt cache breakpoint on the node list prefix.

4. Wire the generation Server Action — calls Claude API with the generation prompt from docs/generation-policy.md Section 2, runs the structural gate checks (Steps 1-2), calls the auto-score Route Handler (Step 3), and on gate pass writes the pending chunk to the review queue.

5. Build the auto-score Route Handler — /api/chunks/score. Accepts a chunk body, applies the 5-dimension automated proxies, returns a structured score object. Add prompt cache breakpoint on the rubric dimension instructions.

6. File ADR-0014 (chunk metadata store) and ADR-0015 (learner state persistence) as separate decisions, sequenced so ADR-0014 does not block on the auth decision that gates ADR-0015.

Open Questions This Review Does Not Resolve

• Auth ADR — LessonSession requires user_id, which requires an auth decision. The auth ADR (Clerk vs. NextAuth, per ADR-0013 consequences) should be filed before ADR-0015 (learner state) is finalized.
• docs/rubric.md — referenced extensively in docs/generation-policy.md but not read for this review (not in scope). Engineering should confirm the automated scoring proxies in generation-policy §3 Step 3 match the rubric dimension definitions before implementing the auto-score Route Handler.
• chunks/ serving — chunk .md files are currently in the repo. At v1 scale (< 200 chunks), serving them from the repo via Next.js static file serving or fs.readFile in a Server Component is appropriate. If chunks grow beyond ~500KB total, evaluate Vercel Blob storage.