A model score is not a system score, and treating the two as interchangeable is a governance failure, not a technical footnote.
In February 2024, a Canadian small-claims tribunal ordered Air Canada to pay a grieving passenger roughly CAD $812 after its website chatbot invented a bereavement-fare refund policy that did not exist. The airline’s defense — that it could not be held responsible for what its own chatbot said — was rejected by the tribunal member, who found the position hard to take seriously. A month earlier, a UK parcel firm, DPD, had to disable the AI layer of its support chat after a routine system update stripped its guardrails and it began swearing at a customer and writing poems about how useless it was. And through the spring of 2024, journalists at The Markup documented that New York City’s official small-business chatbot, built on Microsoft Azure AI services and trained on thousands of city web pages, was routinely telling business owners they could legally pocket workers’ tips, refuse tenants with housing vouchers, and go cash-only — all things the underlying law in New York City explicitly prohibits.
None of these are stories about a weak model. The underlying language models in all three cases were, by the standards of the day, competent generalists. What failed was everything wrapped around the model: the absence of a policy-grounding constraint, a guardrail architecture that a routine deploy could silently strip, a scope boundary that was never enforced. Each system likely looked fine on whatever pre-launch evaluation got it approved. None of that evaluation told anyone what would happen when a real customer, in a real conversation, hit the seam between the model and everything else.
This is the core problem with using benchmark scores, offline evaluation sets, and vendor-reported metrics as evidence that an AI system is “good enough” to deploy: AI quality is not a property of the model. It is an emergent property of the entire production system — prompts, retrieval, orchestration, tools, latency, context management, upstream data, user behavior, and the business process the output feeds into. Evaluating the model in isolation from that system doesn’t produce a conservative estimate of real-world performance. It produces a different measurement of a different thing, dressed up to look like an answer to the question anyone actually cares about.
The benchmark itself is shakier than the slide suggests
Before getting to the system layer, it’s worth being honest about how much weight even a clean model-level benchmark score can bear. Frontier models now routinely score 88–93% on MMLU, a gap so narrow between models with visibly different production behavior that the benchmark has largely stopped discriminating between them. Two separate forces are eroding the signal. The first is data contamination: because benchmarks are published on the open web and models train on enormous scraped corpora, overlap between training data and test questions is common even without intent. OpenAI’s own technical documentation for GPT-4 noted that BIG-bench — a benchmark suite that embeds a unique “canary” string specifically so it can be filtered out of training data — had still been ingested and had to be excluded from evaluation, because the filtering mechanism didn’t survive contact with a web-scale crawl. The second is Goodhart’s law in its plainest form: once a benchmark becomes a target that developers optimize against, it stops functioning as an unbiased measure of the underlying capability it was built to represent. Researchers who tested twenty proposed contamination-mitigation strategies across ten models and five benchmarks found none of them meaningfully improved resistance to gaming without also changing what the benchmark measured.
The practical implication for anyone approving a deployment is blunt: a public leaderboard number tells you almost nothing about how a model will behave on your organization’s actual queries, because nobody has been training against your workload. The more defensible move — and the one increasingly recommended in the applied-evaluation literature — is a small, private evaluation set built from real tickets, contracts, or transcripts specific to your domain, refreshed periodically so it can’t itself be memorized. Even twenty to fifty well-chosen questions from your own production data will tell you more about deployment readiness than another point of MMLU.
That’s the model layer, and it’s already less solid than most procurement decks imply. But even a perfectly clean, uncontaminated, representative model score would only answer one narrow question. It says nothing about the four or five other layers a production AI system actually depends on.
Five different questions, one number
Organizations routinely collapse several distinct evaluation questions into a single reported figure, and the collapse is where false confidence gets manufactured.
Model evaluation asks whether the underlying model, in isolation, performs a bounded task well against a fixed, known input distribution — the MMLU-and-friends layer. It tells you about the model’s reasoning and knowledge, not about anything it will encounter downstream.
System evaluation asks whether the assembled pipeline — retrieval, prompts, tools, memory, guardrails — produces correct behavior end to end, including the failure interactions between components that never show up when each component is tested alone.
Task evaluation asks whether a specific real-world task, defined by what the user actually needed, was completed correctly. A chatbot can generate a fluent, well-grounded, policy-compliant answer to the wrong question and still fail the task.
User-experience evaluation asks whether the interaction itself was usable: was it fast enough, did it repeat itself, did it know when to hand off to a human, did the conversation feel coherent across turns.
Business-process evaluation asks whether the system’s use, in aggregate, moved the metric the organization actually cares about — resolved calls, reduced escalations, accurate compliance advice — rather than merely producing plausible-looking transcripts.
A model can pass the first of these cleanly and fail all four of the others. That is exactly what happened at Air Canada, at DPD, and at NYC MyCity.
Where the pipeline actually breaks
Walk a typical LLM application end to end and the failure points map cleanly onto the layers above.
Speech recognition, if the application takes voice input, is often the first and least examined layer. Benchmark word-error-rate figures — the low single digits reported on clean, native-accent corpora like Switchboard or LibriSpeech — bear little relationship to what a call center actually experiences. Studies comparing accented and cross-corpus speech consistently show error rates two to four times higher than the clean-benchmark figure: one widely cited accent corpus reports roughly 20% average WER across international English varieties against under 3% on standard read speech, and research into racial disparities in commercial ASR systems has documented comparably large gaps between demographic groups on the same systems. Worse, blended WER hides exactly the errors that matter most operationally: a misheard filler word costs nothing, but a misheard account number, policy name, or amount is the difference between a resolved call and a compliance incident. This is why entity-level recall, not aggregate WER, is the metric that actually predicts downstream harm.
Retrieval is the next layer, and it’s where most RAG-based applications quietly fail. Bad chunking splits a policy clause across two retrieved passages so neither contains the full condition. A stale index serves last quarter’s fee schedule. The evaluation research on this layer has matured quickly — frameworks like RAGAS decompose the question into faithfulness, answer relevance, and context relevance rather than one blended score, and later work (ARES, among others) has shown that different automated judges can disagree with each other by tens of percentage points on the same context-relevance question, which is itself a reminder that even “system-level” metrics need their own scrutiny before they’re trusted.
Generation is where a model can be fluent and confident while being ungrounded — precisely what happened to Air Canada’s chatbot, which produced a coherent, plausible-sounding refund policy with no connection to the airline’s actual terms. This is the layer where faithfulness or groundedness metrics — does every claim in the answer trace back to retrieved or verified source material — matter far more than a generic quality score.
Agent orchestration and tool use introduce their own failure surface: wrong tool selected, right tool called with malformed arguments, or no tool called when one was needed. Standardized benchmarking of tool-calling agents shows a consistent pattern — accuracy that looks strong on single-turn, single-tool tasks drops measurably once an agent has to maintain state and make sequential decisions across a multi-turn conversation, which is the setting almost every real deployment actually operates in.
Guardrails and prompt-layer controls are a related but distinct failure surface, and DPD’s incident is the cleanest public example: an unrelated system update silently removed constraints on tone and topic, and nothing in the deployment pipeline caught the regression before a customer did. The model didn’t change. The scaffolding around it did, invisibly.
User experience covers everything from repetitive phrasing to poor escalation logic — and this is where NYC’s MyCity bot failed most consequentially. The problem wasn’t that it occasionally gave a wrong answer; it was that it had no mechanism for recognizing when a question exceeded its competence and needed to defer to a human or a verified source, so it answered legal questions about labor and housing law with the same unearned confidence it used for anything else.
Business outcome, finally, is where all of the above either resolves into something the organization can defend or becomes a viral screenshot, a regulatory inquiry, or a tribunal ruling.
Decomposition, not aggregation
The practical response to this is to stop asking “is the AI good” and start asking “which layer, if any, is failing, and what evidence proves it.” This isn’t a novel idea invented for this article — it’s already implicit in the standards and frameworks most governance teams already reference, even if implementation rarely follows through.
ISO/IEC 25059, the AI-specific extension to the ISO/IEC 25010 software quality model, exists precisely because conventional software quality characteristics weren’t built for systems with probabilistic outputs, learned behavior, and data dependence. It doesn’t carry certifiable requirements of its own — you can’t “comply” with it — but it does something arguably more useful for this problem: it gives organizations a shared vocabulary of sub-characteristics (functional adaptability, user controllability, transparency, intervenability, and more) to specify and evaluate separately, rather than folding everything into one “performance” line item.
NIST’s AI Risk Management Framework makes the same point from the risk-management side. Its Measure function exists specifically to convert qualitative concerns — “this might be biased,” “this might be unreliable” — into quantitative, trackable evidence that can substantiate a go/no-go decision, feed EU AI Act conformity documentation, or satisfy ISO/IEC 42001’s monitoring and measurement clause. Practitioners who work with the framework regularly note that Measure is the function organizations most often fail to operationalize in practice: it’s common to see a governance policy (Govern) and a system inventory (Map) without any corresponding measurement infrastructure, leaving Manage to mean “whatever happens after an incident.”
The frameworks already assume decomposition. The gap is operational: most organizations still produce one dashboard number, usually derived from a pre-launch evaluation set the vendor or internal team built, and treat it as though it answers all five questions above at once.
| Layer | Example failure | Possible metric / evidence |
|---|---|---|
| Speech-to-text | Misheard account number, name, or amount | Word error rate; entity error rate / entity recall |
| Retrieval | Wrong, incomplete, or stale evidence retrieved | Recall@K; context precision/recall |
| Generation (LLM) | Fluent but unsupported or fabricated answer | Faithfulness / groundedness score |
| Agent orchestration | Wrong tool called, malformed arguments, or no call made | Tool-selection accuracy (single-turn and multi-turn) |
| Guardrails / prompt layer | Persona, scope, or policy constraint bypassed | Guardrail regression pass rate, re-run after every deploy |
| UX / conversation | Repetition, unnatural turn-taking, no escalation | Repetition rate, latency, escalation precision/recall |
| Business outcome | Customer misled, issue unresolved, law misstated | First-contact resolution rate, complaint/appeal rate |
What an independent reviewer should actually ask for
Each row in that table implies an owner, and an internal auditor or independent reviewer should be able to name one for each: data or ML engineering for the speech and retrieval layers, model risk or MLOps for groundedness, platform engineering for orchestration and guardrail regression, product for UX metrics, and the business or risk owner for the outcome metric, benchmarked against the pre-AI baseline rather than in isolation.
The evidence a reviewer should ask for at each layer is specific and reproducible, not a policy document asserting that testing occurred. For retrieval, that’s a Recall@K trend measured against a refreshed evaluation set, not a one-time snapshot from launch. For the LLM layer, it’s a faithfulness score computed on a rolling sample of real, answered production queries — not the static set the system was tuned against, which tells you how well it was tuned, not how well it currently performs. For guardrails, it’s the regression suite output from the most recent deploy: the exact artifact that, had it existed and been re-run, would have caught DPD’s failure before a customer did rather than after it went viral. For UX, it’s escalation precision and recall segmented by intent category, not a single blended satisfaction score that averages away the categories where the system is actually failing.
This is the distinction between documentation and evidence that matters most in practice. A policy stating that the system “is evaluated for accuracy and reliability” is documentation. A dated log showing the guardrail regression suite ran against the current deployment and what it caught is evidence. Only one of those would hold up if a regulator, a court, or a journalist asked for it — which, as three separate 2024 incidents demonstrate, is not a hypothetical.
The question that actually matters
If a manager asks “is our AI performing well,” the honest response is that this is at least five questions wearing one coat, and a single benchmark score or vendor spec sheet answers none of them convincingly. What would justify answering yes is a decomposed set of evidence — one line per layer in the table above, drawn from current production traffic rather than a pre-launch test set, refreshed on a defined cadence, with a named owner for each number and a visible record of what happened the last time one of those numbers moved. Anything short of that is a plausible-sounding answer. It is not one that would survive the next Air Canada tribunal, the next DPD screenshot, or the next Markup investigation.
Leave a Reply