Evaluation & Test Sets

Roadmaps › AI Engineering

Scenario

You tweak your system prompt to make the bot more concise. It works on your three test queries. You deploy it. The bot now refuses all complex questions, replying 'I must be concise.' Before you read on, predict: how many test queries would you need to catch this?

Think about coverage — then read on to understand how golden datasets and automated evals prevent silent regressions.

Mental model

Evaluation is unit testing for vibes. It turns 'this looks good to me' into a repeatable number you can track across every prompt change.

Standard code either works or crashes. LLM outputs exist on a spectrum of quality. Evaluation is the process of building a golden dataset of (input, expected behavior) pairs, running every pipeline change against it, and scoring the results. Since you cannot do exact string matching on natural language, you use a strong LLM as a judge to grade whether the output contains the expected facts or behavior.

Deep dive

When you build an AI app, your first instinct is to 'vibe check' — run a few queries manually and eyeball the outputs. This does not scale. A change that fixes one edge case almost always breaks another. You need automated, repeatable measurement. Every prompt or model change must be validated against the same standard test suite before it ships.

Manual vibe checks miss silent regressions. Automated evals catch them before deployment.

The foundation of evaluation is the **golden dataset** — a static list of 50–200 diverse inputs and the expected behaviors or key facts you want to see in the output. It must cover easy questions, hard multi-step reasoning, adversarial inputs ('ignore all instructions'), and edge cases. Without this dataset, you have nothing to measure against and no way to prove improvement.

A golden dataset of inputs and expected facts is your evaluation ground truth.

You cannot use exact string matching to grade LLM outputs. If the expected answer is 'Paris' and the model outputs 'The capital is Paris, France', the string match fails even though the answer is correct. Instead, use **LLM-as-a-judge**: a meta-prompt asks a stronger model to determine whether the output contains the expected fact or satisfies the grading criterion.

LLM-as-a-judge grades semantic meaning, not exact characters — essential for natural language output.

An eval run loops over your golden dataset, generates the output with your current system, passes both the output and the expected answer to the judge model, and collects a score (0 or 1). Average these scores to get a metric like '87% factual accuracy'. If the score drops after a change, you revert.

Eval run = generate outputs → grade with judge → average score → ship or revert.

You can write judges for multiple dimensions: tone ('is this polite?'), hallucination ('is every claim in the retrieved context?'), safety ('did it refuse this harmful query?'). Run the eval suite on every prompt change, model swap, or chunking change. Treat it like CI/CD unit tests — a score drop blocks the deploy.

Write separate judges for accuracy, tone, safety. Run on every pipeline change.

Code examples

Define the golden dataset

test_set = [
    {
        'input': 'What is the refund policy?',
        'expected_fact': 'Refunds are allowed within 30 days of purchase.'
    },
    {
        'input': 'Can I buy a car?',
        'expected_fact': 'The system must refuse — we only sell software.'
    },
]

The expected value is the core fact or behavior, not a rigid string. The judge will check whether the model's actual output contains or satisfies this expected fact.

LLM-as-a-judge grading function

from openai import OpenAI
client = OpenAI()

def grade_output(question: str, expected: str, actual: str) -> int:
    prompt = f"""You are an impartial judge.
Question: {question}
Expected Fact: {expected}
Actual Model Answer: {actual}

Does the Actual Answer contain the Expected Fact?
Respond with exactly 1 for Yes, or 0 for No."""

    resp = client.chat.completions.create(
        model='gpt-4o',        # judge must be capable — don't use a small model
        messages=[{'role': 'user', 'content': prompt}],
        temperature=0.0        # deterministic grading
    )
    return int(resp.choices[0].message.content.strip())

The judge evaluates semantic meaning, not exact characters. temperature=0 makes grading deterministic. Always use the most capable model available as the judge — a weak judge produces unreliable scores.

The eval loop

def run_evals(test_set: list[dict], agent_func) -> float:
    scores = []
    for item in test_set:
        actual = agent_func(item['input'])      # run the system under test
        score = grade_output(item['input'],      # grade the output
                             item['expected_fact'],
                             actual)
        scores.append(score)

    accuracy = sum(scores) / len(scores)
    print(f'Eval result: {accuracy * 100:.1f}% accuracy')
    return accuracy

This loop is the heartbeat of AI engineering. Run it before shipping any prompt change. If accuracy drops from 95% to 80%, you don't ship. Over time, add more test cases as you discover edge cases in production.

Key points

Common mistakes

Glossary

golden dataset
A curated, static list of test inputs and the expected correct behavior — used as a measuring stick to score your system before each deployment.
LLM-as-a-judge
A technique where a powerful LLM grades the outputs of another model using a structured grading prompt, replacing slow human grading.
regression
When a change meant to improve the AI causes it to fail at tasks it previously handled correctly — often invisible without an automated eval suite.

Recall questions

Questions & answers

Your team's LLM evaluation pipeline produces flaky results: the same model output gets a pass on one run and a fail on the next. What are two likely causes and how would you fix them?

First: the judge LLM's temperature is greater than 0, causing non-deterministic grading. Fix: set temperature=0. Second: the judge prompt is ambiguous, causing the judge to apply inconsistent criteria. Fix: simplify to strict binary output (1 or 0) with clear, specific grading rubrics.

Approach: Identify that flakiness comes from non-determinism in the judge. Temperature and prompt clarity are the two primary levers.

You are building an eval to check if your RAG bot is hallucinating. How would you structure the LLM-as-a-judge prompt?

Pass the retrieved context chunks and the model's generated answer to the judge. Ask: 'Is every claim in the Generated Answer fully supported by the Retrieved Context? Answer 1 for Yes, 0 for No.' This evaluates faithfulness — no golden expected answer is needed, only the context-answer pair.

Approach: Show that hallucination evaluation compares the retrieved context against the generated output — not against a pre-written expected answer.

Continue learning

Previous: RAG Evaluation

Return to AI Engineering Roadmap