Skip to content

ADR-0016: Data-layer policy — when to use .md files vs. Neon

Guide

ADR-0016: Data-layer policy — when to use .md files vs. Neon

Status: Accepted
Date: 2026-05-31
Approved by: Kevin Wong (Principal)
Related: ADR-0014 (Neon + Drizzle data layer), ADR-0015 (Slack live chatter feed)

Context

KLMS uses two storage systems:

  1. Markdown files (the repo) — single source of truth for all human-authored knowledge, documents, and policies. Rendered at build time by intranet/scripts/sync-content.mjs into static JSON artifacts.

  2. Neon (Postgres + Drizzle) — introduced in ADR-0014. Currently holds four tables: chunks, skill_nodes, chatter_channels, chatter_messages.

As the org grows, new data categories will be created (finance ledgers, risk register, lesson sessions, etc.). Without an explicit policy, each decision will be made ad-hoc and may be inconsistent. This ADR establishes the principle and a per-category inventory so future contributors (human or AI) have a clear default.

Decision

The governing principle: prose stays in .md; structured, queryable, or write-heavy data goes to Neon.

A data category should move to (or stay in) Neon when at least two of the following signals apply:

Signal Description
Write-heavy Data changes frequently, from multiple sources, not just deliberate git commits
Relational / queryable You need filtering, joins, or aggregations — not just “list all”
Real-time / freshness matters Stale data (from the last deploy) causes a user-facing problem
Not human-authored The canonical author is a process or system, not a person writing prose
Rigid schema The data has a fixed shape that markdown frontmatter approximates poorly

If only one signal applies, the default is .md. Convenience (e.g., “SQL feels cleaner”) is not a sufficient reason to migrate — git-versioned .md files have their own advantages (human-readable, diffable, audit-trailed by git history, zero infra cost).

Per-category inventory

Already in Neon — correct placement

Table Signals that justify Neon
chunks Rigid schema (10 fields); machine-scored; will be queried by platform retrieval system
skill_nodes Relational edges (prerequisites[]); reference data for skill resolution queries
chatter_channels / chatter_messages Write-heavy (every Slack message); real-time freshness required; system-authored

Currently in .md — keep as-is

Category Paths Why .md is correct
Foundational docs docs/, hr/, ADRs, SOPs Human prose; changes in days/weeks; git history is the audit trail; 144 docs sync in < 1 s
Reports & briefings reports/YYYY-MM-DD-*.md 1–2 writes/day by the PA; build-time dashboard read is sufficient today
Org roster / agent definitions ORG.md, .claude/agents/*.md Claude Code requires .md agent files; infrequent writes; Notion is the secondary view
Chunks prose chunks/*.md Human-readable companion to the Neon metadata; prose does not belong in a DB column
Archive / MANIFEST archive/MANIFEST.md Consumed by Bash scripts for integrity checks; Neon adds no value
Specs & reviews specs/, reviews/ Low-cardinality, one-off artifacts; no query need beyond list

Migration candidates — Neon when trigger condition is met

Category Current location Trigger condition Signals that will apply
Finance ledgers finance/ledger-money.csv, finance/ledger-ai.csv When Finance unit runs queries across ledgers (cross-rate reporting, per-venture P&L, budget vs. actuals) Relational/queryable; rigid schema; append-only
Risk register reports/risk-register.md When entry count exceeds ~30 rows or Quality & Risk unit needs runtime filtering Rigid schema; relational (owner, severity, status); queryable

When a trigger condition is reached, the migration owner opens a new ADR that specifies the table schema and migration plan, then moves the data.

Consequences

Positive:

  • Contributors have a clear default — no ad-hoc debates on each new data category
  • Neon stays lean; schema complexity is introduced only when genuinely justified
  • Repo stays the human-readable source of truth for prose and policy

Negative / trade-offs:

  • Two migration candidates (finance, risk) will eventually require schema design and a data migration — this cost is deferred, not eliminated
  • The “two-signal” threshold is a heuristic; edge cases will require judgment

Not decided here:

  • The specific schema for finance_transactions or risk_entries — deferred to their respective migration ADRs
  • Real-time dashboard briefing freshness — if the PA publishes 3+ briefings/day and staleness becomes a complaint, a /api/briefing endpoint (similar to /api/chatter) is the natural next step