How to measure an AI agent before it ships
A demo proves an AI agent can work once. An eval harness proves how often it works and what it does when it fails. Here is the smallest one worth building.
Most AI features are shipped on the strength of a demo. Someone types three questions, the agent answers all three, and the room agrees it is ready. Then it meets real users, who ask the fourth question. Nobody in the room can say what happens next, because nobody measured it.
How do you measure an AI agent before it ships? You build an eval harness. This is a fixed, versioned set of questions with expected answers, an automated grader, and a run record. The harness reports the hallucination rate and the refusal rate together, which prevents gaming the metrics.
The gap is not model quality. The gap is that a demo measures whether the agent can be right. Production cares about how often it is right, and what it does when it is not.
In this article
- Why demos are not enough
- What the harness has to contain
- The one number worth publishing
- What this costs
- Finding questions without traffic
- Managing untrusted LLM judges
- Reading the number over time
- Surviving model deprecations
A demo is a sample size of three
The illusion of the happy path
The reason a demo feels convincing is that you wrote the questions. You know the catalogue, so you ask about products that exist. You know the tone, so you phrase things the way the prompt expects. Every question you think of is drawn from the same distribution as the examples you built against.
Real users ask the fourth question
Real users are not drawn from that distribution. They ask about the product you discontinued last year, or ask in two languages in one sentence. They paste an order number with a typo in it. Most of these are not adversarial. They are just outside the handful of paths anyone thought to try.
So the first thing an eval harness gives you is not a score. It is a set of questions you did not choose.
What the harness has to contain
A useful harness is smaller than people expect. It has four parts.
The four components
- A fixed question set, versioned in the repo. This is not generated fresh each run. It is fixed, so a score this week is comparable to a score last week. Start at fifty questions. Pull them from real logs the moment you have real logs, or write them deliberately if you do not.
- An expected answer, or an expected refusal. This is the part teams skip, but it makes the harness worth something. Someone has to decide what a correct response looks like. This includes questions where the correct response is "I don't have that information." An agent that answers everything is just an agent whose failures you have not found yet.
- A grader. For factual lookups, use an exact or fuzzy match against the source record. For prose, use an LLM judge with a rubric. Spot-check its verdicts against human judgement, and know its agreement rate before trusting its number.
- A run record. Every run writes its results with the commit, model version, prompt version, and date. The number by itself is trivia. The number over time is what you actually manage.
The one number worth publishing
The three states of an answer
If a retrieval-grounded agent produces an answer, it is in one of three states. It can be grounded in a real source record. It can be correctly declining. Or it can be asserting something that is not in the source data.
The third state is the hallucination rate, and it is the number worth stating out loud.
Of N questions in the eval set, the agent asserted a fact not present in the grounding data X times. Rate: X/N.
Honesty in metrics
Two things make this honest rather than decorative. It counts assertions against source data, not vibes. This means you can only compute it if the agent's grounding is inspectable. It is also stated alongside N, because 2% over fifty questions and 2% over five thousand are not the same claim.
The refusal rate
There is a second number that matters just as much and gets published less often. That is the refusal rate. An agent can drive its hallucination rate to zero by declining everything.
Reported together, the pair describes a real trade-off. Reported alone, either one is marketing.
What this costs
Time and effort
A first harness is roughly a day of work. Half of it is writing questions, and half of it is wiring the runner. It is genuinely boring.
Surviving a model swap
It is also the only artefact that survives a model swap. When the provider deprecates your model with sixty days' notice, the harness is what tells you whether the replacement is better or worse. It does this in an afternoon instead of a fortnight of anecdotes.
Where the questions come from before you have traffic
Deliberate creation
The objection to a fixed question set is always the same: we do not know what users will ask yet. That is true, and it is not a reason to skip the harness. It is a reason to write the first fifty questions deliberately rather than waiting for the data.
Four sources before launch
- The support inbox. If the process you are automating already exists in human form, these are real questions in real phrasing. They are also free.
- The source data. Walk the catalogue, policy document, or schema. Write a question for each region, especially the thin regions where retrieval will fail first.
- The boundary of scope. Write questions the agent should decline. This is the half of the set nobody writes, but it catches the worst failures.
- Adversarial phrasing. Ask the same question badly, in two languages, or with a typo in the identifier. Ask with a false assumption baked in.
Retiring old questions
Fifty questions written this way take an afternoon. The moment you have real logs, you replace the invented ones with real ones. Keep the set at a fixed size, retiring questions the system has passed twenty times running in favour of ones it has never seen.
What to do about the grader you do not trust
The theatre of grading
Grading is where most harnesses quietly become theatre. Exact match works for a lookup with one right answer. Nothing else is that clean. The moment the agent produces prose, teams reach for an LLM judge and stop asking whether the judge is any good.
Calibrating the judge
Hold the judge to the standard you hold the agent. Take thirty graded examples, have a human grade them independently, and compute the agreement rate between the two.
If the judge agrees with the human eight times in ten, your reported score carries roughly that much confidence and no more. This is usable, as long as you say so.
Tightening the rubric
If it agrees six times in ten, the judge is noise wearing a number. The fix is a tighter rubric. Do not ask "is this answer good". Ask "does this answer assert any fact not present in the supplied context, yes or no".
Narrow, binary, checkable questions produce judges that agree with humans. Broad qualitative ones do not. Re-measure the agreement rate whenever you change the judge model. A grader is a dependency like any other, and it drifts.
Reading the number over time, not the number
The comparative value
A single eval run tells you almost nothing. Its value is comparative. The first run establishes a baseline, which is usually worse than the team expected. That is the harness working.
From then on, the run record is what turns a debate into an observation.
Turning debates into observations
Someone changes the chunk size and the score moves three points. That is a fact rather than an opinion. Someone swaps the reranker and the hallucination rate falls but the refusal rate doubles. That is a trade-off surfaced before a user found it.
Warning patterns
Two patterns are worth watching for specifically:
- Climbing refusal rates. A score improves while the refusal rate climbs. The agent is not getting better; it is getting more evasive.
- Widening variance. A score holds steady while variance across reruns widens. This usually means temperature or retrieval non-determinism has crept in. The single number is hiding a system that now answers the same question two different ways.
Store the raw per-question results, not just the aggregate. The aggregate tells you something moved. Only the per-question diff tells you what.
Why this survives the model you are using today
Handling deprecations
Providers deprecate models on their own schedule, and they give you less notice than you would like. When that notice arrives, a team without a harness has one option. They swap the model string, poke at it by hand for a week, and ship on a feeling.
A team with a harness runs fifty questions against the candidate model in an afternoon. They read the per-question diff, and know exactly which behaviours regressed and whether the prompt needs adjusting.
Testing the untestable
The same applies to every other change that would otherwise be untestable. This includes a new embedding model, a reranker, a prompt rewrite, a chunking strategy, or a cheaper model for a subset of traffic. Each of these is a question about whether quality held. The harness is the only thing in the stack that can answer it without a fortnight of anecdotes.
The cost argument
This is also what makes the cost argument straightforward. The harness is roughly a day to build. It pays for itself the first time it stops a regression reaching production. It pays for itself again every time it replaces a week of manual re-testing with an afternoon.
Conclusion
For the diagnostic side of this—what to do once the harness tells you the agent is inventing things—we have written separately about why a RAG chatbot hallucinates. Building the harness is the first thing we do in an AI reliability evals engagement.
The reason we build one into every engagement is not rigour for its own sake. It is that "it works" is not a claim anyone can check. But "it answered 47 of 50, declined 2, and got 1 wrong — here is the one" is.