RAG evaluation has two independent axes: retrieval quality (did we get the right chunks?) and generation quality (given the chunks, did we produce the right answer?). Confusing them is the most common evaluation mistake in the field.
- 01Distinguish retrieval and generation metrics.
- 02Name three retrieval metrics.
- 03Name three generation metrics.
Retrieval metrics
Recall@k: fraction of queries where the correct chunk is in the top k. Mean Reciprocal Rank: 1/rank of the first correct chunk. Precision@k: fraction of returned chunks that are relevant.
Generation metrics
Faithfulness: does the answer cite the retrieved chunks correctly? Answer correctness: does the answer match ground truth? Answer relevance: does the answer address the question? Frameworks: RAGAS, TruLens, DeepEval.
Why the separation matters
A wrong answer with correct retrieval is a generation bug. A wrong answer with wrong retrieval is a retrieval bug. Aggregate 'RAG accuracy' hides both. Report them independently or you will fix the wrong system.
- Retrieval and generation are two systems; measure them separately.
- RAGAS, TruLens, DeepEval are the standard toolkits.