RAG Evaluation: Metrics That Measure Answer Quality

A RAG answer can fail in two completely different places. The retrieval stage can miss the relevant passage, in which case no amount of prompting will fix the answer. Or retrieval can succeed and the generation stage can still distort, ignore, or embellish what it was given.

If you measure only end-to-end answer quality, you can watch scores drop without knowing whether to fix the index, the chunker, the reranker, or the prompt. Splitting measurement into retrieval metrics and generation metrics turns 'this system got worse' into a target.

Retrieval evaluation asks one question: of the passages that should have been found, how many made it into the candidate list? A handful of metrics describe different aspects of that answer.

The right metric depends on the downstream design. If a reranker gets a final look at the candidates, recall at k is the priority; if the top result is used directly, hit rate and MRR carry more weight.

Once good passages reach the model, the answer must be faithful — every claim traceable to the retrieved context — and relevant, meaning it genuinely addresses the question rather than retrieved adjacent trivia.

Faithfulness is the critical one for enterprise use because it connects directly to trust and compliance. A fluent answer that quietly contradicts its own sources is worse than a short one that stays accurate, so most production evaluations weight it heavily.

A useful evaluation set reflects the real distribution of questions: the topics, phrasings, and difficulty users actually throw at the system. Collecting real queries from logs is the strongest source; synthesized questions are a useful supplement but should be reviewed by people who know the domain.

Each item needs a question, one or more passages judged relevant, and a reference for what a good answer contains. The labeling effort feels like overhead until a regression run shows you exactly which corpus change broke retrieval — then it pays for itself.

Using a strong model to score faithfulness and relevance is now common, and it scales far better than manual review. Judges can compare candidate answers to reference context and flag unsupported statements quickly, and they are consistent enough to track trends over time.

But an LLM judge shares blind spots with the system being judged. It can reward fluent but hollow answers, tolerate style over substance, and drift when the judge model is upgraded. The prudent pattern is a strong judge for screening, with a human review layer on a stratified sample to catch what the judge misses.

Evaluation only pays off if it runs regularly. Wire a regression suite into the pipeline so every change to chunking, retrieval, or prompts is measured before it ships, and keep the dataset versioned alongside the code.

In production, track the same metrics on real traffic plus user feedback such as thumbs-down rates and 'this did not help' responses. Quality signals from live users often surface problems a test set never imagined.