How to tell whether your RAG assistant is any good

A RAG assistant that answers fluently is not necessarily a RAG assistant that answers correctly. Fluency is the model's strength; correctness depends on the whole pipeline. Without measurement you will ship changes that feel better and quietly make things worse. Here is a practical way to measure, using nothing more complicated than a spreadsheet to start.

Split the system, then measure each half

A RAG system has two jobs, and they fail in different ways:

1. Retrieval: did we fetch the right passages?
2. Generation: given those passages, did the answer stay faithful to them and address the question?

If you only grade final answers, you cannot tell which half to fix. Grade both.

Step 1: build an evaluation set

Collect 30 to 100 real questions, ideally from actual users or the people who will use the system. For each, record:

• The question, in the user's own words.
• The document and passage that contain the answer.
• A short reference answer, or key facts the answer must include.
• Include some questions that should not be answerable from your documents.

This set is worth more than any tool. Keep it in version control and grow it every time you find a failure.

Step 2: measure retrieval

For each question, run only the retrieval step and check whether the known correct passage appears in the top results.

• Hit rate at k: the share of questions where the right passage is in the top k results.
• Mean reciprocal rank: rewards putting the right passage near the top.

def hit_rate_at_k(eval_set, retrieve, k=5):
    hits = 0
    for item in eval_set:
        results = retrieve(item["question"], top_k=k)
        if any(r["source_id"] == item["source_id"] for r in results):
            hits += 1
    return hits / len(eval_set)

If retrieval is weak, no prompt improvement will save you. Fix chunking, embeddings, search settings or metadata first.

Step 3: measure the answers

For each question, review the generated answer on three separate questions:

• Correct: does it match the reference facts?
• Grounded (faithful): is every claim supported by the retrieved passages, or did the model add something from its own memory?
• Complete and relevant: does it answer what was asked, without padding?

Score each on a simple scale (for example pass, partial, fail) and keep the notes. Human review of a modest sample beats an elaborate automatic score you have not verified.

Step 4: test the "I don't know" behaviour

Ask the questions that cannot be answered from your documents. A good assistant declines or says the information is not available; a poor one invents something plausible. Track the share of unanswerable questions it handles correctly. This is a safety property, not a nicety.

Using a model as a judge, carefully

You can ask another model to grade answers against the reference. It scales well, and it has pitfalls: it tends to prefer longer, confident answers, and it can share the blind spots of the system it grades. If you use it, calibrate: compare its scores with human scores on a sample, and re-check whenever you change the judge prompt.

Track it over time

• Re-run the evaluation after every change to chunking, embeddings, prompts, models or documents.
• Record the scores in a table with the date and what changed.
• Watch for regressions in questions that used to pass, not only the average.

Common mistakes

1. Testing with three questions the developer wrote and calling it evaluated.
2. Only checking answers that look good.
3. Changing several things at once, so you cannot tell what helped.
4. Never testing unanswerable questions.
5. Trusting an automatic judge without comparing it to human review.

A minimal starter table

Try it yourself

Write 20 questions about a document you know well, five of which the document cannot answer. Run your assistant, fill in the table, and pick the single most common failure. Fix only that, re-run the same 20, and see whether the table improved. That loop is the whole discipline.