Concept
Comparing two system versions on an evaluation set
How to put an interval on a pass rate, compare two versions scored on the same cases with a paired analysis, and plan how many cases a comparison needs. Covers the assumptions that make a measured gain misleading.
Topics: evaluationstatisticsconfidence-intervals