<!– chunk-id: C-006 title: How I stopped click-through gaming from inflating a learning platform’s ranking signal tags: [eval, ranking, signal-integrity, anti-gaming, outcome-metrics, feedback-loop] format-version: 1.0 –>

C-006 — How I stopped click-through gaming from inflating a learning platform’s ranking signal

HOOK

My platform ranks lessons by how learners do on them — a high rating or quiz score pushes a chunk up. The first version had a hole you could drive through: open a lesson, immediately close it, leave a 5-star rating, and the ranking treats that as a strong positive signal. A chunk nobody actually read could climb to the top because the rating event fired without any engagement behind it.

MAP

• Inputs: A feedback loop where each closed lesson session logs three outcome signals — outcome_completion (fraction of the lesson viewed, 0.0–1.0), outcome_quiz (0–100), and outcome_rating (1–5) — and those signals update a chunk’s ranking weight.
• Lap: Walked the path from “session closes” to “chunk weight changes” looking for signals that could move ranking without representing real learning.
• Radio: A rating or quiz event can fire with outcome_completion near 0 — a learner who opened and closed a lesson without reading it still produces a ranking-moving signal.
• Setup Change: Added a completion gate — a session’s outcome signals are ignored for ranking unless outcome_completion >= 0.7. Completion is the only input to the gate; a rating or quiz alone does not satisfy it.
• Delta: Sessions that can move a chunk’s ranking without 70%+ engagement went from “any session with a rating or quiz” to none.

RADIO

The ranking loop trusted three signals equally: completion, quiz, rating. The intent was that a learner who rates a lesson highly, or scores well on its quiz, is telling you the lesson worked. For an engaged learner, that holds.

The hole is the learner who is not engaged. Red: in the first design, a session that logged outcome_rating = 5 updated the chunk’s ranking weight regardless of outcome_completion. So the sequence open-lesson → close-immediately → tap 5 stars produced outcome_completion ≈ 0.0, outcome_rating = 5 — and the loop counted it as a strong endorsement. Worse, it is the cheapest possible action: no reading required. A ranking signal that rewards the lowest-effort behavior will be dominated by low-effort behavior, whether the gaming is deliberate or just idle tapping.

A second, quieter version of the same failure: an expert. Someone who already knows the skill resolves to an advanced (L3) level for that topic, glances at a beginner (L1) chunk, and rates it 5 — “yep, that’s correct.” That rating is honest but it inflates the chunk’s standing for beginners, who are the people the L1 chunk is actually ranked for. The signal is real; it is just attributed to the wrong audience.

SETUP CHANGE

Single change: a session’s outcome signals are excluded from ranking unless outcome_completion >= 0.7. Completion is measured as a scroll/view progress fraction and is the only input to this gate — quiz score and star rating do not satisfy it on their own. A session below 0.7 completion is still logged (it is a valid record), but its chunk_weight_update flag is set to false, so it never reaches the ranking model.

Why this specific gate, and why completion as the sole input: completion is the one signal that is expensive to fake. A rating is one tap; a quiz can be clicked through; but viewing 70% of a lesson takes the time it takes. Gating the trusted signals (rating, quiz) behind the costly signal (completion) means a ranking-moving event now requires the behavior you actually care about — engagement — before it counts. I gated on completion rather than tightening the rating scale or de-weighting low ratings because those still let a zero-engagement session through; the completion gate removes the entire class of click-through events at the door.

(The expert-inflation case is handled by a separate level-stratification rule — only signals from a session at the chunk’s own level update that chunk’s weight — but that is a second change and belongs in its own chunk. This chunk is the completion gate.)

DELTA

Before: chunk_weight_update fired on any closed session that carried a quiz score or a rating, with no completion floor. A session with outcome_completion = 0.0 and outcome_rating = 5 updated the chunk’s ranking weight. Click-through endorsements counted. Sessions able to move ranking without real engagement: all of them with a rating or quiz.

After: chunk_weight_update = true requires outcome_completion >= 0.7 as a necessary condition (alongside the other integrity rules). The same 0.0-completion, 5-star session is now logged with chunk_weight_update = false and contributes nothing to ranking. Sessions able to move ranking without 70%+ engagement: zero.

Observable: The rule is docs/instrumentation.md §4, Rule 1 (“Completion gate (anti-gaming)”), and it appears in the §4 summary table: outcome_completion < 0.7 → chunk_weight_update = false. The derived outcome_pass field encodes the same floor: it is true only if outcome_completion >= 0.7 AND (outcome_quiz >= 80 OR outcome_rating >= 4).

PIT WALL NOTES

• A feedback signal that rewards the cheapest action will be dominated by the cheapest action. Before you wire any outcome into a ranking or reward loop, ask: what is the lowest-effort way to produce a strong positive signal here? If the answer doesn’t involve the behavior you actually want, you have a gaming hole — close it before launch, not after the rankings are already skewed.
• Gate your easy-to-fake signals behind a hard-to-fake one. Ratings and quizzes are cheap to emit; sustained engagement is not. Requiring the costly signal (completion) as a precondition for counting the cheap ones (rating, quiz) raises the price of a fake endorsement to the price of actually doing the thing. Pick the signal that is expensive to fake and make it the gate.
• Log the rejected signal; don’t delete it. A gamed or low-engagement session is still data. Keep the record, set the “counts toward ranking” flag to false, and you stay able to audit, recalibrate the threshold, or study the behavior later — without letting it touch the live ranking. Separate “is this a valid log” from “does this update the model.”

CTA

Use this when an outcome signal (rating, vote, score, thumbs-up) feeds a ranking, reward, or recommendation loop and the signal can be emitted without the engagement it is supposed to represent.

The artifact is a copyable signal-integrity ruleset and the single boolean that decides whether a session is allowed to move your ranking.

Part 1 — The eligibility boolean (gate every outcome signal through this)

# A session may update ranking weights only if ALL hold:
chunk_weight_update =
        (outcome_completion >= 0.7)                       # Rule 1: completion gate (anti-gaming)
    AND (session.level == chunk.level)                    # Rule 2: level stratification (anti-inflation)
    AND (chunk.accepted_session_count >= 5)               # Rule 3: minimum signal before any weight change
    AND (outcome_quiz is not null OR outcome_rating is not null)  # Rule 4: there is an actual signal
    AND not (chunk.source == "generated" and not chunk.author_validated)  # Rule 5: probation for machine-made content

# If any clause is false: log the session, set chunk_weight_update = false, do NOT touch ranking.

Part 2 — Pre-launch gaming-hole checklist (run before any feedback loop goes live)

[ ] What is the LOWEST-effort action that produces a strong positive signal?
       → If it isn't the behavior you want, add a gate.
[ ] Is every cheap signal (rating, vote, click) gated behind a costly one (completion, dwell, verified action)?
[ ] Can a wrong-audience user (expert rating beginner content) inflate the wrong ranking?
       → Stratify: only same-segment signals update that segment's weight.
[ ] Does a single session move ranking a lot? → Require a minimum accepted-session count first.
[ ] Are rejected signals LOGGED (not dropped) so thresholds can be recalibrated later?
[ ] For machine-generated items: are their signals on probation until a human validates the item?

SELF-SCORE

Rubric

Total: 94/100

Passing threshold: >= 70/100

Verdict: PASS

Disqualifier check: No dimension scores zero. The 0.7 completion floor, Rule 1, the summary-table row, and the outcome_pass formula are all traceable to docs/instrumentation.md §4. No invented tools or APIs. Auto-fail not triggered.

Honest weakness: The DELTA is a logic before/after (“a 0.0-completion 5-star session counted → now it doesn’t”), not a measured drop in ranking distortion, because no learner sessions have run yet — the platform is pre-launch. So this documents a gaming hole closed by specification before any real session could exploit it. The honest claim is “the class of click-through signals can no longer reach ranking,” not “gaming dropped by N%.”