Build a Small RAG Pipeline and Measure What It Misses

A grounded answer has two jobs behind it: find useful evidence, then use that evidence correctly. If you only inspect the final sentence, it is hard to tell which job failed.

This lesson builds a small retrieval-and-answer pipeline you can inspect end to end. Six synthetic documents feed a keyword retriever, a generation-input builder and an extractive answer function. The baseline deliberately misses a synonym query. You will see that miss in the retrieval score instead of hiding it behind a plausible answer.

Download the corpus, questions and runnable pipeline. It uses Python's standard library and makes no model or embedding request. The answer function copies a retrieved sentence, providing a deterministic substitute at the generation boundary. These results do not measure an LLM.

1. Run the baseline

Use Python 3.10 or newer. Extract the ZIP, keep its files together, and run:

python3 rag_lesson.py

Use python if that is your Python 3 command. Open the output alongside corpus.json and questions.json. The recorded September 17, 2026 result is:

The first two numbers are intentionally imperfect. The code's assertions preserve that observed baseline. A passing program does not mean every question was answered correctly.

2. Start with a corpus you can audit

Each document has an ID, version, tenant, active flag and text. The corpus contains:

These documents are already short passages. There is no ingestion or chunking service hidden in the download. For a larger corpus, preserve source IDs, access metadata and versions when creating chunks. Otherwise you lose the information needed to filter and cite them.

The historical 30-day rule is useful as a failure fixture. A query containing “legacy refund policy” matches it well, but the current application should not use an inactive rule simply because it has more overlapping words.

3. Filter before ranking

The retrieve function first selects active documents belonging to the requesting tenant or the public collection. Only then does it score overlap:

eligible = [
    doc for doc in corpus
    if doc["active"] and doc["tenant"] in {tenant, "public"}
]
query = terms(question)
ranked = [
    {**doc, "score": len(query & terms(doc["text"]))}
    for doc in eligible
]

terms lowercases text and extracts alphanumeric tokens. Results with zero overlap are discarded. Remaining passages are sorted by descending overlap, then ID, and limited to two.

This is a keyword baseline, not BM25, semantic search or a recommendation to use overlap counts for every application. Its advantage here is visibility: you can reproduce a score by looking at two short strings.

The tenant comes from the application-side fixture, not from an answer returned by the generator. A real service must derive it from trusted identity and enforce access again wherever evidence is cached or stored. Matching a tenant string in this program demonstrates filtering, not authentication.

4. Inspect what reaches generation

generation_input creates a compact payload containing an instruction, the question and the retrieved evidence. Each evidence item includes only its ID, version and text. The result file records this exact payload for every question.

For “refund requests,” the selected evidence includes:

{
  "id": "refund-v2",
  "version": 2,
  "text": "Refund requests must be made within 14 days."
}

The deterministic extractive_generator copies the first result's text into the answer and its citation. With no results, it returns answer: null and citation: null.

That is the point where an LLM adapter could later receive the question and passages. Anthropic's contextual-retrieval article explains retrieval as supplying relevant knowledge to a generation prompt, and evaluates retrieval separately. This example keeps that separation while using a simple local answer function.

5. Check the answer against supplied evidence

The validator requires exactly answer and citation. A citation must identify a document and version actually present in that request's retrieved results, and its quote must occur in that document. For this extractive task, the answer must equal the quoted text.

The worked response is:

{
  "answer": "Refund requests must be made within 14 days.",
  "citation": {
    "id": "refund-v2",
    "version": 2,
    "quote": "Refund requests must be made within 14 days."
  }
}

Three injected failures exercise different checks: a citation to the other tenant's rule, a nonexistent “30 days” quote attributed to the current rule, and a 90-day answer attached to a valid 14-day quote. All are rejected.

An extractive answer can still be irrelevant to the question. Citation validity establishes where text came from. It does not establish that the passage answers what the user asked. The question fixtures therefore have separate reference answers.

6. Explain the failed question

The seven baseline questions produce these outcomes:

The synonym miss accounts for both imperfect metrics. Four of five answerable questions retrieve their reference document. Six of seven total questions match the expected answer or abstention. Questions without an accessible answer stay out of retrieval recall's denominator but remain in the answer/abstention measurement.

With one reference document per question, this recall calculation is simple. A task requiring several passages needs a relevance set and a scoring rule that accounts for all of them.

7. Improve one case without declaring victory

Keep the original files. As an exercise, normalize the exact phrase money back to refund before tokenizing. The missed query should now retrieve refund-v2. After inspecting the output, update the script's expected metric tuple from (5, 4, 6, 3) to (5, 5, 7, 3) and preserve both result files. The tuple records answerable questions, reference retrieval hits, answer matches and rejected adversarial outputs, in that order.

That is a targeted vocabulary fix, not proof of a good general retriever. Add new questions such as “return my payment” before deciding whether the change generalizes. Evaluate semantic or hybrid retrieval on the same access-filtered corpus and reference questions if the keyword approach keeps missing how users ask.

Next, temporarily remove the active-document filter and run “legacy refund policy.” The old rule becomes a stronger lexical match. Restore the filter afterward. This failure comes from source lifecycle policy, so changing models would not fix it.

Your deliverable is the corpus, the seven-question result table, one improved query and one deliberately reproduced lifecycle failure. Continue to persistent memory to store a user's preferences across runs without confusing them with source evidence.

Related Guides

• Three Prompt Contracts You Can Check with Evaluation Fixtures
• Enterprise AI Knowledge Management: A Practical Selection and Pilot Guide
• Enterprise AI Agent Platforms: How an FDE Should Evaluate the Shortlist