Modern AI Pro · Level 2 · Track B · Project B1

Version-aware RAG: outdated documents never reach the answer

Old and current versions of an operations runbook read almost identically, so similarity search keeps serving outdated thresholds, ports and on-call teams. One metadata filter, where={"superseded": False}, removes them before ranking.

48.8% → 0%superseded chunks retrieved
0/9 → 9/9version-conflict cases passed
7/7 → 7/7exact-value cases (unchanged)
+0.9 mslatency cost of the filter

Try it: filter OFF vs ON

Your question is embedded in your browser with the same model as the Python system (all-MiniLM-L6-v2). The page then searches the 17 stored chunks twice: once by similarity alone, once with the version filter. The first search downloads the model (~90 MB, cached afterwards).

Model not loaded yet; it loads on the first search.

Filter OFF: similarity only

—

Filter ON: superseded = False

—

The 21 frozen test cases

Written and committed before the fix. These are the measured results from the Python system, retrieval only, top 4. Click a question to run it live above.

IDTypeQuestionOFFONSuperseded retrieved OFF → ON

How it works

Every stage existed in the class lab except the version filter. The metadata comes from a separate manifest.json, so the embedded text is exactly what a reader sees.

16 runbooks→load→+ version metadata→chunk→embed (MiniLM)→Chroma
question→embed→VERSION FILTER→top-4 by similarity→LLM answer (Python version only)

On this page, the stored chunk vectors come from the Python pipeline, and questions are embedded in your browser with transformers.js (fp32). Checked before publishing: for all 21 test questions, filter OFF and ON, the browser returns the same top 4 in the same order as the Python system (42/42).

The corpus: 8 operations runbooks, two versions each

Example runbooks for a fictional company's messaging platform. Each pair shares wording and differs in its facts. Old versions carry no "superseded" banner; only the metadata knows.

RunbookCurrent fileOld fileExample of what changed (old → current)