Module 1 of 2 · Lesson 3 of 3
Lesson · Module 1 · GenAI System Design
Evaluating output you cannot unit-test
Build an evaluation plan for open-ended output: what to measure offline, how to use human and model judges without fooling yourself, and what to watch after launch.
Builds on
A unit test asserts that a function returns one expected value. A language model asked to explain a leave policy can phrase a correct answer in thousands of ways, and a wrong answer can look almost identical to a right one. Equality checks fail in both directions: they reject good answers that are worded differently, and they pass nothing about whether a fluent answer is true.
Evaluation is the work of replacing that missing test with evidence. It decides whether a prompt change, a different model or a new retrieval setting is an improvement, and it keeps deciding after launch. This lesson builds an evaluation plan in the order a candidate would want to state it: what good means, what data to measure on, which measurement tools to use and where each one misleads, and what to watch once real users arrive. The running example is still the internal help assistant from the first lesson with the retrieval pipeline from the second.
Start from what a bad answer costs#
Quality isn't one number. For the help assistant, a wrong answer about expense limits is an annoyance, while a wrong answer about a legal obligation is a real problem. Writing the failure costs down first decides which dimensions need the strictest checks. A workable set for this assistant:
- Faithfulness. Every claim is supported by the retrieved passages. This is the property grounding exists to provide.
- Answer relevance. The response addresses the question that was asked.
- Correct citation. The cited passage is one that actually supports the claim.
- Appropriate refusal. When the corpus has no answer, the assistant says so and doesn't improvise.
- Format. Output follows the structure downstream code expects.
Latency and cost are constraints measured alongside these, not extra quality dimensions. Separate the dimensions that are hard requirements from those that are preferences, because a release rule such as "no regression on faithfulness, and no more than a small cost increase" is much easier to apply than a single blended score.
Build the evaluation set#
The set of inputs you measure on decides what the numbers mean. Three sources are common, and they have different biases:
| Source | Strength | Weakness |
|---|---|---|
| Sampled real traffic, with privacy handling | Matches what users actually ask | Doesn't exist before launch, and may contain sensitive content that needs scrubbing and access control |
| Questions generated from the documents | Cheap and broad | Tends to resemble the documents' wording, so it is easier than real questions |
| Hand-written hard cases | Targets known failure modes | Small, and reflects the author's imagination |
A useful set mixes them and deliberately includes the awkward cases: questions the corpus can't answer, questions that need two documents, ambiguous questions, near-duplicate policies, and inputs that try to override the instructions. Label each example against a written rubric so that two reviewers would agree, and keep a portion of the set aside that nobody tunes against.
How large does the set need to be? Enough to see the difference you care about. If an example either passes or fails, then with n independent examples and an observed pass rate p, a rough 95 percent interval under the normal approximation is:
margin ~ 1.96 * sqrt(p * (1 - p) / n)
p = 0.80, n = 100: 1.96 * sqrt(0.16 / 100) ~ 0.078 (about 7.8 points either way)
p = 0.80, n = 400: 1.96 * sqrt(0.16 / 400) ~ 0.039 (about 3.9 points either way)
With a hundred examples, one estimate of 80 percent and another of 84 percent lie within each other's noise, so that gap alone doesn't show one system is better. This is an approximation that assumes independent examples and moderate pass rates. Comparing two system versions on the same examples, and looking at the examples where they disagree, is considerably more sensitive than comparing two separate intervals. The model computes a probability distribution over the next token, and the decoding strategy then either samples from it or selects deterministically. When the deployment samples, a single run is one draw, so repeat runs for comparisons that matter and report the spread. Switching to deterministic decoding during evaluation removes that variation where the system allows it, but it then measures the deterministic configuration and not the sampling behavior production users see.
Choose measurement tools, and know where each one lies#
Programmatic checks are deterministic and cheap enough to run on every example and every build: the output parses against the expected schema, every cited identifier exists in the retrieved set, a required field is present, the length is within limits. They can't tell whether the content is right, but they catch a large share of plain breakage.
Reference-based metrics compare output with a known answer. They are strong where a single right answer exists, such as extracting a date or a category, where exact match is meaningful. For open-ended text, metrics based on word overlap reward similar wording and not correctness, so a correct paraphrase can score badly and a fluent wrong answer can score well.
Human review is the reference standard for open-ended output, and it's slow and costly. It only works with a rubric, and the reviewers' own consistency should be measured, for example by having two people label a shared sample and checking how often they agree. Low agreement is a signal to investigate, not a diagnosis in itself. Check whether the rubric is unclear, whether the disagreements cluster on ambiguous cases, whether the reviewers need calibration and where else the disagreement comes from. Measuring consistency is better than assuming the reviewers are careless.
Model judges scale review by asking a language model to grade output against the rubric. The research behind this practice is encouraging and also specific about limits. One study found that strong judges agreed with human preferences over 80 percent of the time on its benchmarks, about the level of agreement among humans, and also documented position bias, verbosity bias, a tendency to favor the judge's own outputs, and limited reasoning ability (Zheng and others, 2023). That result was measured on that study's tasks with the model that was strongest at the time, so agreement on your task has to be measured and not assumed. Practices that follow from the documented biases:
- Calibrate the judge: have people label a sample drawn from your own data, compare, and look at where the judge disagrees.
- For pairwise comparisons, evaluate each pair in both orders and report the order-sensitivity rate, meaning the share of pairs whose verdict changes when the order swaps. Treat a flipped verdict as a sign that the judge is uncertain about that pair, not as a result to discard. Send those pairs to human review, and keep them in the reported results as their own category instead of excluding them or counting only the stable verdicts. A high order-sensitivity rate is itself a finding about the judge or the rubric.
- Give the judge the source passages when asking about faithfulness, since it can't verify claims it can't see.
- Prefer a narrow rubric question ("is each claim supported by the passages?") to a broad one ("is this a good answer?").
- Re-check the judge whenever the judge model, its version or the rubric changes.
No single tool covers all five dimensions from the start of this lesson. Read across a row to see which tools can check that dimension:
| Dimension | Programmatic check | Reference-based metric | Model judge | Human review |
|---|---|---|---|---|
| Faithfulness | No, it can't read meaning | Weak, since overlap with a reference answer isn't support from the passages | Yes, when it sees the passages and has been calibrated | Yes, the reference standard |
| Answer relevance | No | Weak, since it rewards wording close to the reference | Yes, with a narrow rubric question | Yes |
| Correct citation | Partly: every cited identifier exists in the retrieved set | Partly: cited identifiers match the passages labelled as needed, though another passage may also support the claim | Yes, given the cited passage and the claim | Yes |
| Appropriate refusal | Partly, when declines have a fixed form | Yes, when each example is labelled answerable or not | Yes, for free-form declines | Yes |
| Format | Yes, by parsing against the schema | Rarely needed | Unnecessary when a parser can decide | Unnecessary when a parser can decide |
The rows explain why plans layer the tools: deterministic checks on every example and every build, a calibrated judge for the dimensions that need reading, and people on a sample to keep the judge honest.
Measure the parts, not only the whole#
An end-to-end score tells you something is wrong without saying where. The pipeline from the previous lesson has two stages that fail differently, so measure each:
retrieval: question-level recall@k = questions with every needed passage in the top k results
/ questions that have an answer in the corpus
passage-level recall@k = needed passages found in the top k results
/ needed passages, counted over the same questions
generation: faithfulness and relevance, given the passages that were retrieved
The two retrieval figures use different units, so report which one you mean. A question that needs two documents counts as a success at question level only when both are in the top k. If just one is retrieved, the question fails at question level and contributes one of its two passages at passage level. Question-level recall is the stricter figure and usually the closer match to whether the generator has what it needs. Passage-level recall shows how much of the needed evidence is being found.
If question-level recall@k is low, improving the prompt won't help. If question-level recall is high and faithfulness is low, the problem is in generation or prompt assembly. Questions with no answer in the corpus belong in a separate slice, scored on whether the assistant declines.
Watch the system after launch#
Offline evaluation decides whether a change should ship. It can't tell you how real traffic behaves, because production inputs drift away from any fixed set: new products, new vocabulary, new kinds of questions. Online signals fill the gap, and each has a trap:
- Explicit feedback, such as thumbs up or down, is direct, but few users give it and it skews toward strong reactions.
- Behavioral signals, such as users rephrasing the question, regenerating the answer or escalating to a person, are plentiful but ambiguous. A rephrase can mean the answer failed or that the user wanted more detail.
- Sampled review of production traffic by people, or by a calibrated judge, gives interpretable quality estimates and is also how new failure cases reach the evaluation set.
- Guardrail and safety trigger rates, along with latency and cost per request, can shift for reasons unrelated to answer quality, so they're watched in their own right.
- Input drift, for example a rising share of questions about a topic the corpus barely covers, signals a coverage gap before users complain.
Online comparisons between two versions need enough traffic to see the effect you care about, and the arithmetic from the earlier interval applies to them as well. Prefer changes that can be rolled back quickly, which the failure-handling lesson returns to.
How evaluation itself fails#
- Overfitting to the set. Prompts tuned until the evaluation set passes can stop generalizing. The held-out slice exists to expose this.
- Leakage. Evaluation examples copied into few-shot prompts or into the retrieval corpus inflate scores.
- A stale set. A set built at launch ages as the product, the corpus and the users change. Schedule refreshes from sampled traffic.
- Moving baselines. A provider updating a hosted model, or a judge model changing, shifts scores with no change on your side. Record the exact model and prompt versions with each result, and keep a fixed reference run for comparison.
- A metric becoming the goal. A team that is rewarded for a judge score can find prompts that please the judge and not the user. Keep human samples in the loop to catch it.
Using this in a design discussion#
An answer that stands out names the evaluation plan before the architecture is finished. It states what a bad answer costs, the dimensions that follow, where the data comes from, which checks are automatic and which need people, how the judge is calibrated, and what is monitored after launch.
Sample wording for a candidate: "I'd define faithfulness and appropriate refusal as the hard requirements, build a set from sampled questions plus hand-written hard cases, and score retrieval and generation separately. I'd calibrate any model judge against human labels from our own data before trusting it, and after launch I'd sample real traffic for review and feed the failures back into the set."
Measuring quality is what makes the later trade-offs in this Track decidable. A faster, cheaper configuration is only an improvement if the evaluation says quality held, and the design problem that closes this Module, an incident handoff summary, needs grounding and evaluation together.
Related Knowledge
Practice
Problem
Design a live incident summary that an on-call engineer can trustDesign a system that condenses chat, alerts and deploy events from a live incident into a handoff summary, and decide how to tell whether the summary can be trusted.
MediumPractice for this part of Reasoning about a model request: cost, knowledge and quality · Builds on this lesson
Related Problems
Related Lessons
Plan for wrong, unsafe and unavailable output: layered checks, graceful degradation, where a person belongs in the loop, and how to roll back a model or prompt change.
A decision lesson for senior and staff-level discussions: which kind of change should carry a quality improvement, what evidence the next step needs, and how to defend the choice under questioning.