The evaluation set is the cheapest artefact in an AI project. It is also the one most often skipped, and its absence is what turns every subsequent decision into an argument about impressions.
The rule is simple and slightly uncomfortable: build it before you build the system, from real users, during discovery. It cannot be reconstructed convincingly afterwards, because by then everyone has seen what the system does and the questions drift towards what it handles well.
What it is
A fixed set of cases, each with an input and a verified correct answer, that runs on every change. Not a test suite in the unit-test sense — the outputs are not deterministic — but a scored benchmark that says whether a change made things better or merely different.
Around a hundred cases is a workable start. Ten is theatre. A thousand is a project of its own and rarely necessary at the beginning.
Where the questions come from
From the people who will use the system. This is not a detail.
Engineers, asked to write test questions, write questions the system can answer — the failure is unconscious and universal. Users write what they will actually send: badly phrased, missing context, containing a false premise, or with no correct answer at all.
Good sources, in order of value:
- The existing queue. If the system is replacing or assisting a manual process, that process has a history. Real tickets, real emails, real queries.
- The people who answer them today, asked what they get asked.
- The awkward cases the same people remember, which are disproportionately valuable.
Coverage matters more than count
A hundred cases that are all the easy path is a hundred cases that will never tell you anything. Four categories, all necessary:
- The common path — what most queries look like. Establishes the baseline.
- Real edge cases — the ones that actually occur, not the ones someone imagined. The unusual account type, the query spanning two policies, the exception the process has always handled by hand.
- Cases where the answer is "I don't know" or "this needs a human". These are the most commonly missing and the most diagnostic, because a system that always answers is a system that will confidently answer wrongly.
- Cases with a false premise. A user asking about a product you do not offer. The correct behaviour is to say so, not to help.
What to measure
Not one number. A retrieval system is a pipeline, and a single end-to-end accuracy score tells you something broke without saying where.
- Retrieval hit-rate — did the right passage come back at all? Measured with generation removed from the loop. In our audits, four out of five failures happen before the model sees the context, which is why this metric is the one worth instrumenting first.
- Answer accuracy against the verified answer.
- Citation fidelity — does the cited source actually support the sentence it is attached to? A well-formed answer citing an unrelated document is worse than a refusal, because it survives review.
- Refusal correctness — did it decline the cases it should have declined?
Grading, honestly
Exact-match scoring works for structured extraction and not much else. For open answers there are three options, and each has a cost:
Human grading is the most accurate and does not scale. Useful for establishing the baseline and for periodic audits of whatever automated method you adopt.
Model-as-judge scales and introduces its own bias — it tends to favour answers that resemble what it would have produced. Usable, provided it is itself validated against human grades on a sample rather than trusted outright.
Structural checks — did it cite a source, did the number match, did it stay within the allowed set — catch a surprising share of failures cheaply and should be the first layer regardless.
Using it as a gate
The point is not to have the set. It is to run it on every model change, prompt change, retrieval change or index rebuild, and to block a release that regresses.
That is what makes model choice replaceable. When a provider deprecates a version or a cheaper option appears, the question becomes a measurement rather than a debate — and teams without an evaluation set answer it by upgrading and hoping.
The uncomfortable part
Building this set will tell you, early, that the system is worse than the demo suggested. That is the artefact working correctly.
The alternative is finding out in production, from a user, about a case nobody had thought to try — at which point the cost of the discovery is considerably higher and the conversation is with someone who has lost confidence in the system.