How to Evaluate a RAG System: Retrieval First, Answer Second
By Numan ·
Separate retrieval quality from answer quality when testing a RAG application.
How to Evaluate a RAG System: Retrieval First, Answer Second
A RAG answer can fail for two different reasons: the search did not return the right evidence, or the model used good evidence badly. Microsoft's RAG evaluator guide (https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators) separates retrieval and groundedness measures. The OpenAI accuracy guide (https://developers.openai.com/api/docs/guides/optimizing-llm-accuracy) uses the same diagnostic split.
Start with a question set
Collect questions that reflect actual users: a simple lookup, a multi-document comparison, an outdated fact, an ambiguous request, and a question whose answer is absent. For each, record the source document or expected “not found” outcome. Do not evaluate only examples the system was built to pass.
Inspect retrieval before generation
For each question, save the top retrieved passages and their source identifiers. Ask whether the needed evidence was returned, whether irrelevant passages dominate, and whether access control filtered private records correctly. A fluent answer cannot repair missing evidence.
Then inspect the answer
Compare the answer with the retrieved passages. Mark unsupported claims, missing caveats, stale facts, and incorrect citations. Include a case where the correct behavior is to state uncertainty. If the answer is wrong despite good context, adjust the prompt or model behavior; if context is wrong, work on indexing, filters, or ranking.
Make changes one at a time
Keep the question set fixed while testing a change, and log the previous and new result. A threshold that removes noisy passages may also remove the only relevant passage. The objective is better answers on the tasks your users bring, not a higher score on a single easy example.
Primary sources: Microsoft RAG evaluators (https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators) · OpenAI accuracy guide…