Build a Small RAG Pipeline and Measure What It Misses

Say a chatbot answers a customer's refund question with the wrong deadline. Two different things could have gone wrong. It may have pulled the wrong document, or it may have pulled the right one and used it badly. If you only look at the final answer, you can't tell which.

This lesson builds a small RAG pipeline you can inspect from start to finish. RAG, short for retrieval-augmented generation, means finding relevant documents first and then answering from them.

Six made-up documents feed three parts: a keyword search, a step that packages what the answer step gets to see, and an answer function that copies a sentence from the evidence. The baseline misses one question on purpose, where the user words it with a synonym. You'll see that miss in the retrieval score instead of having it hidden behind an answer that sounds fine.

Download the corpus, questions and runnable pipeline. It uses only Python's standard library and never calls a model or an embedding service. The answer function copies a retrieved sentence, standing in for the model with something that gives the same result every time. So these results don't measure an LLM.

1. Run the baseline

Use Python 3.10 or newer. Extract the ZIP, keep its files together, and run:

python3 rag_lesson.py

Use python if that's your Python 3 command. Open the output next to corpus.json and questions.json. The run recorded on September 17, 2026 gave:

The first two numbers are imperfect on purpose, and the code's assertions lock in that baseline. So a program that passes doesn't mean every question was answered right.

2. Start with a corpus you can audit

Each document has an ID, a version, a tenant (the customer it belongs to), an active flag and its text. The corpus has:

These documents are already short passages. Nothing in the download quietly splits long files into chunks. If you do split a larger set of documents, keep each chunk's source ID, access details and version. Without them you can't filter or cite the chunks later.

The old 30-day rule is there to cause a failure. A question containing "legacy refund policy" matches it well. But the app shouldn't use an inactive rule just because it shares more words with the question.

3. Filter before ranking

The retrieve function first keeps only active documents that belong to the asking tenant or to the public collection. Only then does it score how many words each one shares with the question:

eligible = [
    doc for doc in corpus
    if doc["active"] and doc["tenant"] in {tenant, "public"}
]
query = terms(question)
ranked = [
    {**doc, "score": len(query & terms(doc["text"]))}
    for doc in eligible
]

terms lowercases the text and pulls out the words and numbers. Documents that share no words are dropped. The rest are sorted by how many words they share, then by ID, and only the top two are kept.

This is plain keyword matching. It isn't BM25 or semantic search, and it isn't a suggestion to count shared words in every app. Its advantage here is that nothing is hidden. You can check any score by looking at two short strings.

The tenant comes from the app's own test setup, not from anything the answer step returns. A real service has to get it from a trusted login, and check access again anywhere evidence is cached or stored. Matching a tenant string in this program shows filtering. It doesn't prove who the user is.

4. Inspect what reaches generation

generation_input builds a small package with an instruction, the question and the retrieved evidence. Each evidence item carries only its ID, version and text. The result file records this exact package for every question.

For "refund requests," the evidence includes:

{
  "id": "refund-v2",
  "version": 2,
  "text": "Refund requests must be made within 14 days."
}

The extractive_generator copies the top result's text into both the answer and the citation. If nothing was retrieved, it returns answer: null and citation: null.

This is the spot where a real LLM could later receive the question and passages. Anthropic's contextual-retrieval article describes retrieval as handing relevant knowledge to the prompt that writes the answer, and it scores retrieval on its own. This example keeps those two parts separate too, with a simple local answer function where the model would go.

5. Check the answer against supplied evidence

The validator requires exactly two fields, answer and citation. A citation has to name a document and version that were actually retrieved for this request, and its quote has to appear in that document. For this copy-a-sentence task, the answer must equal the quote.

The worked response is:

{
  "answer": "Refund requests must be made within 14 days.",
  "citation": {
    "id": "refund-v2",
    "version": 2,
    "quote": "Refund requests must be made within 14 days."
  }
}

Three planted failures test different checks. One cites the other tenant's rule. One quotes "30 days" from the current rule, which doesn't say that. One gives a 90-day answer next to a valid 14-day quote. All three are rejected.

A copied answer can still miss the question. A valid citation shows where the text came from. It doesn't show that the text answers what the user asked. That's why each test question also has its own reference answer.

6. Explain the failed question

The seven baseline questions turn out like this:

The synonym miss explains both imperfect numbers. Four of five answerable questions find their reference document. Six of seven questions match the expected answer or correctly abstain, which means the system says it has no answer.

Questions with no accessible answer are left out of the retrieval score, called recall, because there's nothing to find. They still count in the answer measurement.

With one reference document per question, recall is simple to work out. A task that needs several passages needs a list of every relevant one and a scoring rule that counts them all.

7. Improve one case without declaring victory

Keep the original files. As an exercise, replace the exact phrase money back with refund before splitting the question into words. The missed question should now find refund-v2.

Check the output, then update the script's expected numbers from (5, 4, 6, 3) to (5, 5, 7, 3) and keep both result files. In order, those four numbers are answerable questions, reference documents found, answers matched, and planted bad outputs rejected.

That fixes one phrase. It doesn't prove the search is good in general. Add new questions such as "return my payment" before deciding whether the fix holds up. If keyword matching keeps missing how users phrase things, try semantic or hybrid search on the same filtered documents and the same reference questions.

Next, remove the active-document filter for a moment and run "legacy refund policy." The old rule now wins, because it shares more words with the question. Put the filter back afterward. This failure comes from the rule about which documents are current, so switching models wouldn't fix it.

You should end up with the corpus, the seven-question result table, one improved question and one failure you reproduced on purpose. Next, persistent memory stores a user's preferences across runs without mixing them up with source documents.

Related Guides

• Three Prompt Contracts You Can Check with Evaluation Fixtures
• Enterprise AI Knowledge Management: Choose and Pilot
• Enterprise AI Agent Platforms: An FDE's Shortlist Test