The defect that reaches a user is fluent, plausible and wrong, and no suite raises a flag on it. What production measurement catches, and what late discovery costs.
A crashing agent is a good day. The failure that reaches users arrives on time, in fluent English, well formed and wrong, and nothing in a log tells it apart from the correct answers around it. That is the defect an automated suite is least equipped to see, which is why "we test it" does not answer a client asking how you would know.
The cost of finding out late is concrete. On agents running over live ERP data, five tools were quietly truncating their own responses, and a planner caught three wrong weekly figures the agent had rolled up itself instead of leaving to the query. Both survived a green suite. Both were caught by a person reading production output closely enough to notice a number that looked off.
One suite reported 43 of 43 passing while a tool underneath it was completely broken. The suite was not faulty; no case exercised that tool. An audit of the same system later found that most of its tools were never exercised by a live test.
Structural regressions — the wrong data basis, a dropped field, SQL that no longer parses, a tool pointed at a stale table — are what a suite is good at. A fluent wrong answer is what it cannot reach, because a suite only asks what somebody thought to write down. Green means the cases that exist passed, not that the right ones exist.
The version of this that fails treats acceptance testing as a milestone: run it, sign it, move on. The failure it catches does not stop being possible on sign-off day. Every defect of that class on our record surfaced when somebody put a real question to a live system, never when CI ran.
A finance question-answering agent makes the point from the other direction. A head-to-head comparison of two deployed builds ran seven scenarios against each. The compute layer was correct every time on both builds, and three scenarios still failed. Every error sat in the sentence describing the number, where nothing testing the arithmetic was positioned to catch it.
Order the layers by what each costs to run, so the cheap ones have spoken before the expensive ones start:
Two rules about building the cases matter more than the layering. Assert invariants, not pinned values: an exact expected count against a daily extract is a failure scheduled in advance, and a suite that cries wolf on a calendar is one people learn to skim. And score a refusal as a pass: an agent declining a question it cannot answer correctly is doing what you asked.
A gate reporting 89 of 89 says something a coloured light cannot: how large the population it checked actually was. From the same measurement pass, a sandbox build cleared 93 of 93, and a wider gate over that build returned 55 of 55. Those are separate facts about coverage rather than one status. Publish the denominator and a reader can see whether a gate quietly got smaller.
Telemetry belongs somewhere the client can query it. On a search platform we routed Cloud Logging into BigQuery so query volume, latency and zero-result rate are things the client reads directly — observability they own, not a report they wait for.
On the ERP agents, live traffic runs at a median of 35 seconds and a 95th percentile of 58, at 2.4 tool calls per turn; its two heaviest tools take 13 to 14 seconds apiece and scan around a gigabyte every time they run. Those are measured numbers rather than tuned ones, and deliberately so: we chose accuracy on the agreed scenarios ahead of speed, and the same report sets out what would be traded away if that priority ever changed.
Measurement only catches what somebody thought to measure. Neither the truncating tools nor the wrong weekly figures were raised by an instrument. Attention caught both, and the fixes were structural: the roll-up moved into SQL, so one pooled series comes back per request.
And a gate that exists and passes is not the same claim as a gate that runs. Whether the evaluation suite ran ahead of a given promotion still came down to somebody remembering — a wiring gap in the instrumentation, recorded as open at handover rather than closed on paper. That gap is where the cost lives, measured in how long a wrong number sits in front of a real user before anyone happens to look.
Get in touch
Tell us what system the answer lives in and who needs it. We'll reply with a view on whether it's a two-week assessment, a five-week pilot, or something else.