RAG Evaluation

Roadmaps › AI Engineering

Scenario

Your team swaps the embedding model from text-embedding-ada-002 to text-embedding-3-small. The developer says 'it feels better'. A week later, user complaints spike. Before you read on, predict: what information would tell you definitively whether the change helped or hurt?

Think about the two phases of a RAG system — then read on to learn how to measure each one separately.

Mental model

RAG evaluation is grading an open-book exam in two parts: did the student find the right pages (retrieval), and did they copy the answer correctly without making things up (generation)?

Because RAG has two distinct phases, it fails in two distinct ways. If retrieval fails, the LLM is starved of facts. If generation fails, the LLM ignores the facts and hallucinates. You cannot evaluate RAG with a single 'is the answer good?' number. The RAG Triad uses three separate metrics — Context Relevance, Faithfulness, and Answer Relevance — each grading a different part of the pipeline. You use an LLM-as-a-judge to score these at scale.

Deep dive

Traditional software is deterministic — you write unit tests with exact expected values. LLMs are probabilistic — you write evaluations ('evals'). A RAG pipeline is complex: a single 'thumbs down' from a user tells you nothing about whether the vector search failed or the LLM hallucinated. You must evaluate the retrieval step and the generation step separately.

Evaluate retrieval and generation as separate steps — a single end-to-end score hides the root cause.

**The RAG triad** — The industry standard approach to evaluating RAG pipelines isolates failures into three distinct metrics: Context Relevance, Faithfulness, and Answer Relevance. Popular open-source evaluation frameworks like **RAGAS** and **TruLens** are built entirely around this triad, providing ready-to-use LLM-as-a-judge functions that calculate these scores automatically.

Frameworks like RAGAS and TruLens automate the RAG triad to isolate pipeline failures.

**Metric 1: Context Relevance** — grades the retriever. Given the user's question, did the retrieved chunks actually contain relevant information? If the user asks about 'billing' and you retrieve paragraphs about 'password resets', this score is low. A low Context Relevance score means you need to fix chunking, embeddings, or reranking — not the model.

Context Relevance: did the database return documents that relate to the question?

**Metric 2: Faithfulness** — grades the LLM. Given the retrieved chunks, did the model use only those facts, or did it add information it invented? If the chunks say 'SLA is 99%' and the model says 'SLA is 99.9%', faithfulness is low. A low faithfulness score means you need to tighten the system prompt, lower temperature, or use a stronger model.

Faithfulness: did the LLM stick strictly to the retrieved facts without hallucinating?

**Metric 3: Answer Relevance** — grades the final output. Did the answer actually address the user's question? A model can be perfectly faithful to context (no hallucinations) but still reply 'I don't know' because the retrieved context was irrelevant. That is a faithful but unhelpful answer. Answer Relevance catches this case.

Answer Relevance: did the final output actually satisfy what the user asked?

How do you score hundreds of test queries? Human grading is too slow. You use **LLM-as-a-judge**: write a strict grading prompt asking a capable model (like GPT-4o) to score the triad on a 0–1 or 1–5 scale, given the question, context, and answer. This is the industry-standard approach for automated, scalable evaluation.

Use a strong LLM to automatically score the RAG Triad across hundreds of test cases.

Code examples

LLM-as-a-judge: evaluating faithfulness

from openai import OpenAI
client = OpenAI()

def evaluate_faithfulness(question: str, context: str, generated_answer: str) -> int:
    judge_prompt = f"""
You are an impartial judge. Determine if the GENERATED_ANSWER
is entirely faithful to the CONTEXT — no facts outside the context.

CONTEXT: {context}
GENERATED_ANSWER: {generated_answer}

Reply with exactly '1' if faithful, or '0' if it contains hallucinations.
"""
    response = client.chat.completions.create(
        model='gpt-4o',        # always use a capable model for grading
        messages=[{'role': 'user', 'content': judge_prompt}],
        temperature=0.0        # deterministic grading
    )
    return int(response.choices[0].message.content.strip())

The judge evaluates the relationship between context and answer — not the quality of the question. Running this over 500 test questions gives you a numeric faithfulness score. If you change the system prompt and faithfulness drops from 0.95 to 0.78, you know not to ship that change.

Key points

Common mistakes

Glossary

RAG Triad
The three metrics used to grade a RAG system: did it find useful documents, did it stick to those documents, and did it actually answer the question?
context relevance
A metric that checks whether the documents pulled from your database actually contain information relevant to the user's question.
faithfulness
A metric that checks whether the model's answer was derived strictly from the retrieved context — or whether the model added facts it invented.

Recall questions

Questions & answers

Your team changed chunk overlap from 50 tokens to 200 tokens. User satisfaction dropped. How would you use the RAG Triad to diagnose what went wrong?

Run the golden dataset through automated evals and compare the three triad scores before and after the change. Chunk overlap primarily affects retrieval. If Context Relevance dropped, larger overlapping chunks added noise and confused the vector search. If Context Relevance stayed the same but Faithfulness dropped, the retrieved chunks are now so large the LLM's attention is diluted across too much text, causing hallucinations.

Approach: Tie the specific code change (chunking) to the specific metric it impacts (Context Relevance). Show how triad metrics isolate which layer of the pipeline degraded.

You are building an automated eval suite using LLM-as-a-judge. What are two critical requirements for the judge model to ensure trustworthy evaluations?

First, the judge model must be more capable than the model being evaluated — use a frontier model like GPT-4o so it can reliably detect subtle hallucinations. Second, the judge prompt must be highly specific with a binary or discrete scale and explicit criteria. An open-ended 'is this good?' prompt causes the judge itself to produce inconsistent scores. Set temperature=0 for deterministic grading.

Approach: Highlight the need for a stronger judge and a constrained grading rubric. A judge that produces variable scores is useless for regression detection.

Continue learning

Previous: Context Injection & Prompt Assembly

Next: Evaluation & Test Sets

Return to AI Engineering Roadmap