Unit tests, data invariants, business prompts and a faithfulness check each catch a different failure — and a demo can pass every one of them while a real question still breaks it.
Every agent we've shipped passes a demo before it fails a real question. That gap is why we run four evaluation layers before anything reaches a pilot, each one catching a defect class the others miss.
On the shortage agent we run in production over live SAP data, that stack is 190 unit tests, 25 data invariants checked against production, 40 business prompts, and a faithfulness check on top — and even that combination didn't catch everything a real planner would.
190 offline tests run in seconds with no cloud dependency. They confirm the code does what the code is supposed to do — a query is well-formed, a function returns the right shape. They say nothing about whether the answer is the right answer.
25 invariants run against production data and assert properties that survive daily change — an extract is no more than a few days old — rather than pinning an exact value that breaks the moment the data moves. Assert the shape of the truth, not a snapshot of it.
40 prompts drawn from how planners actually ask, not how an engineer would phrase a query. This layer catches an agent that answers a technically different question than the one that was asked — correct SQL, wrong scope.
The gap none of the first three layers close: the model narrating a number that isn't the one it retrieved. On this engagement, a faithfulness check caught exactly that — the underlying data was right, the sentence describing it wasn't — and the fix was to pin the model version rather than let it float, plus a verbatim rule for how a retrieved number gets restated.
All four layers passing is a strong signal, not a guarantee. The defect class that matters most — a fluent, plausible, wrong answer to a question nobody scripted — is still caught by a real person asking a real question in pilot, not by any automated suite. Automated evaluation and pilot feedback catch different problems; run both, and don't call either one optional.
Tell us what system the answer lives in and who needs it. We'll reply with a view on whether it's a two-week assessment, a five-week pilot, or something else.