Skip to content

Knowledge · Concept

Comparing two system versions on an evaluation set

How to put an interval on a pass rate, compare two versions scored on the same cases with a paired analysis, and plan how many cases a comparison needs. Covers the assumptions that make a measured gain misleading.

5 sections1 table

Fast understanding

Definition

An evaluation set is a sample of the inputs you care about, so a pass rate measured on it is an estimate with sampling error, not the true rate. Comparing two versions of a system means asking whether the gap between their measured rates is larger than that error. Two ideas carry most of the weight. A rate needs an interval around it, and the right interval depends on the sample size and the rate. And when both versions answer the same cases, the comparison should be made case by case, which removes the variation that comes from some cases being harder than others.

The ideas apply wherever a pass or fail outcome is counted: an offline evaluation set, a held-out slice, a canary that compares a new prompt with the current one. Evaluating output you can't unit-test introduces the basic margin, and Containing failure applies it to a canary. This entry collects the method in one place and extends it to the cases those lessons only point at.

Explanation

How it works

One rate. For n independent cases with an observed pass rate p, the usual normal-approximation margin at 95 percent is 1.96 x sqrt(p x (1 - p) / n). It rests on the central limit theorem, which Miller applies to evaluation scores (Miller, 2024, Section 2.1). It behaves badly for small n and for rates near 0 or 1: the interval can extend past 100 percent, or collapse to zero width when no failures were seen. Brown, Cai and DasGupta show that the standard interval's coverage is erratic, and recommend the Wilson interval or the Jeffreys interval for small n and the Agresti-Coull interval for larger n (Brown and others, 2001). The Wilson interval has a closed form, with z = 1.96:

center = (p + z^2 / (2n)) / (1 + z^2 / n)
half   = z x sqrt(p(1 - p) / n + z^2 / (4 n^2)) / (1 + z^2 / n)
interval = center - half to center + half

With 19 passes in 20, the normal approximation gives 85.4 to 104.5 percent, which exceeds 100. The Wilson interval gives 76.4 to 99.1 percent. With 0 failures in 50 cases, the normal approximation gives a zero-width interval at 0 percent, while the Wilson upper bound for the failure rate is about 7.1 percent. Zero observed failures in 50 cases is compatible with a failure rate of several percent.

Two versions on the same cases. Sort every case into one of four cells. Let b be the number that passed only before the change, and c the number that passed only after it.

Passes afterFails after
Passes beforepasses bothb: lost by the change
Fails beforec: gained by the changefails both

Only b and c say anything about the difference. Cases that pass both or fail both are the same for either version. The difference in pass rate is (c - b) / n. Each case contributes +1, 0 or -1 to the difference, so its variance per case is (b + c) / n - ((c - b) / n)^2, and its standard error is the square root of that divided by the square root of n. Miller shows that the paired variance is the unpaired variance minus twice the covariance between the two versions' scores, and that scores on the same questions are usually positively correlated, so pairing reduces noise at no extra cost (Miller, 2024, Section 4.2).

For a significance test on the same table, McNemar's test asks whether the b + c disagreements split more unevenly than a fair coin would. The exact form is a two-sided binomial test with p = 0.5 on b + c trials. Dietterich compared five approximate tests for comparing two learning algorithms and found that McNemar's test had low Type I error, and that for algorithms that can be run only once it was the only one of the tests with acceptable Type I error (Dietterich, 1998). That finding is about comparing classifiers on a fixed test set, which is the situation of a frozen evaluation set.

How many cases. For a paired comparison, the number of cases needed to detect a difference of delta with power 0.8 at a two-sided 5 percent level is about 7.85 x Var / delta^2, where Var is the per-case variance of the paired difference, close to the share of cases on which the versions disagree. Miller gives the general planning formula (Miller, 2024, Section 5). Assume 15 percent of cases disagree and you want to detect a 3-point difference: about 7.85 x 0.149 / 0.0009, or roughly 1,300 cases. Comparing two independent samples, with both versions passing at about 75 percent, needs roughly 3,300 cases per version, or about 6,600 in total, for a similar detectable difference, because the shared case difficulty is no longer cancelled. The independent figure is 7.85 x (0.75 x 0.25 + 0.75 x 0.25) / 0.0009, about 3,270 per version. The paired figure counts cases that both versions answer, so 1,300 paired cases means 2,600 model answers to score, while the independent design needs 6,600 answers. Both figures assume a 3-point difference, a two-sided 5 percent level and 80 percent power, and the exact requirement changes with the assumed rates. The paired figure also depends on the disagreement rate, which you do not know before the experiment. Estimate it from a pilot run.

Deeper reference

Example

A constructed example. A team compares a current prompt with a candidate on 200 held-out cases. The counts are assumed.

passes both = 132     only before (b) = 8     only after (c) = 22     fails both = 38
before: 140 / 200 = 70.0%      after: 154 / 200 = 77.0%      difference = +7.0 points

Treating the two rates as independent samples gives standard errors of 3.2 and 3.0 points and a standard error for the difference of 4.4 points. The 95 percent interval is -1.6 to +15.6 points, which includes zero, so the gap alone does not show the candidate is better.

Using the pairing, the per-case variance is 30 / 200 - 0.07^2 = 0.1451, and the standard error of the difference is sqrt(0.1451 / 200) = 2.7 points. That gives a normal-approximation 95 percent confidence interval of +1.7 to +12.3 points for the difference. The exact McNemar test is a separate calculation on the same table: for 22 against 8 disagreements it gives a two-sided p-value of about 0.016. The interval says how large the difference could plausibly be, and the test says whether the split of disagreements is unlikely under a fair coin. The same data support a claim in one analysis and not in the other, because the unpaired calculation ignores that both versions faced the same cases, so it cannot cancel the difficulty they share.

What the paired result does not say is how large the improvement is in production or whether it is worth shipping. The interval is wide, from under 2 points to over 12. The assumptions are listed in the failure modes below: independent cases, a set that resembles real traffic, and a comparison decided before the results were seen.

Failure modes

  • Cases that are not independent. If several cases come from one document, one ticket thread or one customer, the effective sample is smaller than n and the standard error computed from n is too small. Miller discusses clustered questions and shows that clustered standard errors can be substantially larger than the naive ones (Miller, 2024, Section 2.2). Group such cases and compute the interval over groups.
  • A set that does not resemble traffic. The interval describes the population the cases were drawn from. A set built from the corpus's own wording measures performance on questions that read like the corpus.
  • Choosing the comparison after seeing the data. A set used to decide what to change cannot also give an unbiased test of that change. Keep a held-out slice that nobody tunes against. A held-out set can be reused, but each time its results influence a decision, the risk of overfitting to it grows and unadjusted intervals and p-values stop being valid, so account for that or add fresh cases.
  • Many comparisons. If you test 20 slices or metrics at the 5 percent level and nothing differs, the chance that at least one looks significant is 1 - 0.95^20, about 64 percent (assuming independent tests). Choosing the best of several candidates on the same cases has the same effect: the winner's measured gain is biased upward. Decide the primary comparison in advance, correct for the number of candidates or comparisons, and label the rest as exploratory.
  • Repeat runs treated as more cases. When decoding is sampled, running each case several times reduces the sampling noise within a case, with diminishing returns, but it does not add new cases. Miller warns that a pooled standard error over all repeated answers is inconsistent (Miller, 2024, Section 3.1). Average within each case, then analyse across cases.
  • Noisy or biased labels. If a model judge produces the pass or fail label, its errors add noise to every rate, and a bias in the judge is not reduced by more cases. Check the judge against human labels first.
  • Statistical significance mistaken for value. A gain can be real and still too small to justify a change in cost or risk. Set the smallest difference worth acting on before the run, and design the set size around it.

Interview considerations

Name the smallest improvement that would change the decision, then the number of cases needed to see it, and say that both versions will be scored on the same cases and compared per case. Say that nobody tunes against the held-out slice, and that comparing several candidates on it, or reusing it across rounds, calls for accounting for selection and multiple testing, or for fresh cases. Mention that cases drawn from the same source are treated as a group. If the interviewer pushes on sample size, the honest answer is that it depends on how often the two versions disagree, which a pilot run estimates.

Tip

Sample wording for a candidate: "I'd run both versions on the same held-out cases and look at the cases where they disagree. With 22 cases the new version fixes and 8 it breaks, that's a clear lean, and I'd put an interval on the difference instead of comparing two separate pass rates. Before running, I'd decide the smallest gain that matters and size the set from the disagreement rate in a pilot."