RAG Evaluation

RAG evaluation tells you whether a change (new chunk size, embedding model, reranker or prompt) actually made your system better. Without it you are guessing. Evaluate retrieval and generation separately so you know which part to fix.

Build a Small Test Set

Write 20 to 100 realistic questions, each with the ID of the chunk (or page) that contains the answer and a reference answer. Real user questions from logs are the best source.

test_set = [
    {"q": "How many annual leave days?", "expected": "handbook.pdf-4", "answer": "20 days"},
    {"q": "Can I carry over leave?",     "expected": "handbook.pdf-5", "answer": "Up to 5 days"},
]

(no output: this is the evaluation data)

Retrieval Metrics: Hit Rate and MRR

def evaluate_retrieval(test_set, search, k=4):
    hits, rr = 0, 0.0
    for case in test_set:
        ids = search(case["q"], k)                     # returns a list of chunk ids
        if case["expected"] in ids:
            hits += 1
            rr += 1 / (ids.index(case["expected"]) + 1)
    n = len(test_set)
    return {"hit_rate": hits / n, "mrr": rr / n}

fake_search = lambda q, k: ["handbook.pdf-4", "handbook.pdf-9"] if "annual" in q else ["x", "handbook.pdf-5"]
print(evaluate_retrieval(test_set, fake_search))

{'hit_rate': 1.0, 'mrr': 0.75}

Hit rate is how often the right chunk appears in the top k. MRR (mean reciprocal rank) also rewards putting it near the top: rank 1 scores 1, rank 2 scores 0.5. The fake search here stands in for your real retriever.

Generation Metrics with an LLM Judge

• Faithfulness (groundedness): is every claim supported by the retrieved context?
• Answer relevance: does the answer address the question?
• Correctness: does it match the reference answer?
• Context precision/recall: were the retrieved chunks relevant and complete?

A Simple Faithfulness Judge

import ollama

def judge_faithful(context, answer):
    prompt = (f"Context:\n{context}\n\nAnswer:\n{answer}\n\n"
              "Is every claim in the answer supported by the context? Reply YES or NO.")
    r = ollama.chat(model="qwen3", options={"temperature": 0},
                    messages=[{"role": "user", "content": prompt}])
    return "YES" in r["message"]["content"].upper()

print(judge_faithful("Annual leave is 20 days.", "You get 25 days of leave."))

Sample output:
False

Evaluation Tools

Open-source libraries such as Ragas and DeepEval implement these metrics and can use local models as judges. LangSmith and Langfuse store datasets and track scores across experiments. Whatever tool you use, keep the test set fixed and change one thing at a time.

Frequently asked questions

How do you evaluate a RAG system?

Measure retrieval (hit rate, MRR, recall@k) against labelled questions, and measure answers for faithfulness, relevance and correctness, often with an LLM judge.

What is faithfulness in RAG?

Whether the answer's claims are supported by the retrieved context. Low faithfulness means the model is hallucinating or using outside knowledge.

How many test questions do I need?

Even 20 to 50 good questions reveal big problems. Grow the set over time from real user queries.