Why Do LLM Applications Fail in Production?

Every LLM demo works. That's the problem.

In November 2023, Jake Moffatt’s grandmother died. He went to Air Canada’s website, opened the chatbot, and asked about bereavement fares. The chatbot told him he could book a regular ticket and apply for a discount within 90 days. He did. Air Canada refused the refund (its actual policy required the application before travel) and pointed out that the chatbot had been wrong. A British Columbia tribunal disagreed, ruled the airline liable for negligent misrepresentation, and awarded Moffatt $650.88 plus interest (Moffatt v. Air Canada, 2024 BCCRT 149).

The chatbot, presumably, had passed its demo. Someone, somewhere, had typed “can I get a bereavement fare?” and gotten a clean, plausible-looking answer. That demo cost Air Canada a lawsuit, a news cycle, and the chatbot itself, which was quietly removed from the site within weeks.

Two months later, in January 2024, a former engineer named Chris Bakke convinced a Chevrolet of Watsonville chatbot to sell him a 2024 Tahoe for one dollar (”no takesies backsies”) by injecting the instruction directly into a customer-service prompt. That same week, a frustrated DPD customer in the UK got the company’s chatbot to write a poem calling DPD “the worst delivery firm in the world” and to swear, on company letterhead, after a routine system update knocked out its safety guardrails.

All three systems had launched. All three had been demoed. All three were gone within days.

The pattern these incidents reveal is the dominant story of the 2024–2026 LLM build cycle: demos work, production breaks, and the failure is almost never the model itself. It is the retrieval pipeline, the tool-calling layer, the evaluation harness, the memory store, the orchestration graph, and the observability that nobody built because the demo didn’t need it. LLM engineering, in other words, has stopped being a prompt engineering discipline and started being a distributed systems discipline. Most teams haven’t caught up.

This post is a tour of why.

Why demos lie

A demo is a fundamentally different system from a production deployment, even when the model weights are identical. Demos run on inputs the builder implicitly curated: questions they thought to test, documents uploaded yesterday, short context windows, a single user, no concurrency. Latency looks fine. Cost looks fine. Accuracy looks fine because the bad questions never made the script.

Production undoes every assumption in the first hour. Real users ask malformed, contradictory, multi-intent questions. Retrieval indices drift as documents are added, deprecated, and silently re-chunked. Tool APIs return new error codes. Sessions get long enough that the system prompt and the user’s first message are 8,000 tokens apart. Sub-agents call each other. Caches go cold. A spike in concurrent requests flips the latency distribution from snappy to cascading timeout.

The temptation, when something breaks, is to blame the model. But model-level reasoning is rarely the right level of analysis. A 95%-accurate model wired into a system with no recovery path produces a system that fails 5% of the time spectacularly, because nothing catches the failure. The failure is the system, not the prediction.

The retrieval illusion

Retrieval-augmented generation is the most common architectural pattern in production LLM systems and the one that fails most quietly. Barnett et al., in the most-cited industry paper on the subject, identified seven failure points based on three real deployments: missing content (the answer isn’t in the corpus, but the model answers anyway), missed top-K, not in context, not extracted, wrong format, incorrect specificity, and incomplete answers. None are model failures. All feel like model failures to a user.

Chunking alone is a graveyard of demo-time decisions. Fixed-size chunking with overlap performs adequately on FAQ-shaped corpora and disastrously on technical documents where a concept spans pages. Semantic chunking helps but introduces its own failure: chunks coherent in isolation that lose the cross-references giving them meaning. Databricks benchmarks on long-context RAG show retrieval quality varying by an order of magnitude across chunking strategies on the same corpus.

The most insidious failure is hallucinated grounding: the model receives retrieved context, ignores or misuses it, and produces an answer that looks grounded, confident, specific, structure, but whose factual content is invented. RAGAS-style faithfulness metrics catch this by asking a separate model whether each claim in the answer is supported by the retrieved context. Teams that don’t run faithfulness evaluations discover the problem when a customer or journalist quotes the chatbot back to them.

RAG is not a feature you turn on. It is a pipeline with at least six independently failing components, each with its own evaluation question.

The long-context mirage

The other place teams hide retrieval failure is in the context window. Frontier models advertise 200K, 1M, and 10M token windows. The argument: stuff the whole knowledge base into context and skip the retriever.

Liu et al. (TACL 2024) dismantled this with one experiment. They placed a relevant document at varying positions inside a long context and measured recall. The result is the now-famous U-shaped curve: models recall information at the beginning and end of the context reliably, and lose it in the middle almost completely. On multi-document QA, performance dropped more than 30% when the relevant document moved from position 1 to the middle.

Vendors responded with the Needle-in-a-Haystack eval, scored near 100%, and declared victory. But NIAH measures string recall, not reasoning over distributed evidence. Chroma’s “context rot” research shows every frontier model tested — GPT-4.1, Claude Opus 4, Gemini 2.5 Pro, Qwen3-235B — degrades measurably as context length grows, especially on tasks that require integrating information from multiple positions.

Long context is not a substitute for retrieval. It substitutes for some retrieval — the kind where a single prominent fact must be located. For synthesizing across many fragments, a long window makes things worse: it dilutes signal and increases latency and cost without buying reliability. Mature teams treat the context window as a working set, not a database.

Tools that lie quietly

Retrieval is one way an LLM connects to the outside world. Tool calling is the other, and it has its own characteristic failure: tools fail silently while the model continues confidently.

The Berkeley Function Calling Leaderboard found that frontier models do well on single, well-formed calls and poorly when memory, dynamic decisions, or long-horizon reasoning enter the picture. τ-bench (Yao et al., 2024), which tests agents in retail and airline domains with simulated users, found GPT-4o succeeds on fewer than 50% of full tasks. Worse, the consistency metric — pass^k, the probability of succeeding on all k runs — collapses; pass^8 in retail is below 25%. The agent isn’t unreliable in the boring sense of being wrong; it’s unreliable in the dangerous sense of being inconsistent. Evaluation runs on Tuesday don’t predict Wednesday’s behavior.

A tool call goes wrong four ways. The model invokes the wrong tool. It invokes the right tool with malformed arguments. The tool succeeds but returns an unexpected payload, which the model misinterprets. Or the tool fails, returns an error, and the model — trained to be helpful — papers over it with a confident invented answer. This last failure is the silent one: clean response, successful LLM call in the logs, and only an integration test comparing the model’s claim to the tool’s actual return reveals the lie.

Schema drift compounds it. Fields rename, enums add values, required becomes optional. An agent prompted on the old schema keeps calling tools the old way, and modern API gateways forgive it by silently coercing or ignoring fields. The agent looks correct; the system is wrong. Teams that run schema-contract tests on every deploy catch this. Teams that don’t ship a steadily widening gap between what the agent thinks it’s doing and what’s actually happening.

The multi-agent tax

If a single agent fails 5% of the time, two agents in series fail roughly 10%, and the math gets worse. Multi-agent systems, the default architecture for ambitious LLM products in 2024, multiply both the surface area for failure and the difficulty of debugging it.

Cemri et al., in “Why Do Multi-Agent LLM Systems Fail?”, analyzed more than 200 tasks across seven popular frameworks and produced the Multi-Agent System Failure Taxonomy. Fourteen distinct failure modes across three categories: specification (the agents got the wrong job), inter-agent misalignment (they disagree about what they’re doing), and task verification (no one checks whether the work is right). Built from 1,600 annotated traces, the most striking finding is that failures are not primarily prompt-quality problems. They are protocol problems: what information is shared between agents, who owns what state, how disagreements are resolved.

This sets up the public debate of the year. Cognition’s «Don’t Build Multi-Agents» argued sub-agents lose context about each other’s work; a single agent with good memory beats a swarm. Anthropic’s «Building Effective Agents» countered that workflows of simple, composable agents are robust, and that orchestrator-worker patterns outperform monolithic agents. Both are right, and both agree on the underlying point: multi-agent reliability is determined by memory and message protocol, not the agents. If the orchestration layer doesn’t aggressively manage context, deduplicate state, verify hand-offs, and detect non-progress, no amount of model intelligence rescues the system.

The reliability arithmetic is brutal. Five agents at 99% per-step reliability, sequential, give ~95% system reliability. Ten steps at 99% give 90.4%. Most production agent loops have far more than ten steps and far less than 99% per-step. The only way out is to design for recovery: checkpoints, verifiers, retries with different prompts, fallbacks to a simpler path, not for peak performance.

The evaluation mirage

The single most common mistake teams make when shipping LLM products is assuming their evaluation tells them what their users will experience. It almost never does, for three structurally different reasons.

• LLM-as-judge bias. GPT-4 matches human preferences with about 80% agreement — the same as inter-human agreement, and the result that launched a thousand judge-based eval pipelines. The same paper documented systematic biases: position bias (Vicuna-13B’s win-rate over ChatGPT moved from 2.5% to 82.5% depending on order), length bias (longer wins regardless of quality), and self-preference bias (models prefer their own family’s outputs, confirmed by subsequent work even after controlling for quality).

• Benchmark contamination. Mirzadeh et al. (Apple, 2024), in GSM-Symbolic, took the standard GSM8K math benchmark and generated symbolically-equivalent variants by changing names and numbers. Performance dropped 0.3% (GPT-4o) to 9.2% (Mistral-7B), and statistical analysis suggests the original test set is in essentially every frontier model’s training data. The benchmark is measuring memorization as much as reasoning.

• Evaluation-production drift. Your eval set was hand-written by a few engineers in a sprint six months ago. Your real users ask different questions, in different distributions, with different ambiguity profiles, in different languages. You cannot evaluate a moving target with a static set. The fix is unglamorous: continuously sample real production traffic, label a slice, and treat that as the eval set.

Latency, cost, and the compounding loop

Every LLM system has an economic shape that looks fine in a demo and grotesque in production. Demos involve one prompt, one response, one user. Production agents involve many prompts per task, retries, sub-agent calls, multi-step planning, reflection, verification, all compounding.

Internal numbers shared by Anthropic and several agent platforms in 2024–2025 put agent token consumption at roughly 4× chat usage and multi-agent at 15× or more. Cost-aware orchestration research suggests maximally accurate configurations are 4–10× more expensive than balanced ones, and most products do not need maximum accuracy on every call. A demo runs ten agent loops a day; production runs ten thousand, and the bill arrives at month-end.

The under-appreciated lever is caching. Prompt caching, retrieval caching, and tool-result caching can turn a 30-second cold path into a 300ms warm path — a 100× improvement on the request that matters most, the second-and-subsequent identical or near-identical request. Most teams discover caching after their first surprise invoice.

Memory and the ghost in the context window

Memory is the part of agent design demos cannot test, because demos are short. Once a session lasts hours, or a user has multiple sessions over weeks, two failure surfaces appear: forgetting and contamination.

Forgetting is simpler. Naive conversational memory, a list of past turns truncated to fit the window, drops information in the order it arrived, regardless of importance. The system “remembers” the user said I prefer dark mode in turn 3 only as long as turn 3 fits. Smarter implementations summarize older turns, but summarization is lossy: it preserves the structure of the conversation while losing the specifics that turn out to matter.

Contamination is more dangerous. If memory is writeable from user input, directly or indirectly through a tool that ingests user-provided documents, memory becomes an injection vector. The MINJA family of memory injection attacks demonstrated in 2024–2025 that adversarial inputs can plant instructions in long-term memory that persist across sessions, with reported injection success rates above 95% in lab conditions. Multi-agent systems with shared memory are especially exposed: a single contaminated entry propagates through every agent that reads from the store.

The deeper issue is that LLM memory has no native concept of trust or provenance. Every byte that ends up in the context window has equal weight on the next prediction, regardless of whether it came from a system prompt, a verified database, a tool’s structured output, or a user’s pasted text. Mature memory systems impose this structure externally: tagged provenance per entry, untrusted-content sandboxes, and policies about which memory classes can ever drive privileged actions.

Observability, or the discipline most teams skip

The common thread through every failure mode above is that the failure is invisible without instrumentation. A retrieval miss looks like a hallucination. A tool error looks like a reasoning failure. A context-window degradation looks like the model “getting dumber” over a long conversation. A multi-agent miscoordination looks like the agent “having a bad day.” Without traces — full, structured records of every prompt, every tool call, every retrieved document, every model decision — every one of these is unattributable.

LLM observability is not log aggregation. It is a discipline with three layers.

• Request tracing captures every component invocation in a single linked trace, so an engineer can see, for one user-visible failure, what prompt was sent, what was retrieved, what tools ran, what they returned, and how many tokens were spent.

• Eval telemetry runs automated correctness, faithfulness, and policy checks on a sampled fraction of production traffic, dashboarding results so drift surfaces as a metric movement rather than a customer complaint.

• Failure forensics keeps enough metadata to replay an incident, debug it offline, and ship a regression test that prevents recurrence.

The mature LLM platforms LangSmith, Helicone, Arize Phoenix, the model providers’ own eval tooling all converge on roughly this shape. The teams shipping reliable agents treat their LLM stack the way infrastructure teams treat distributed systems: with traces, SLOs, error budgets, and a runbook.

Reframing: LLM engineering is distributed systems engineering

The throughline of the last three years of production LLM experience is that the disciplines that matter are the ones distributed-systems engineers already know. Retrieval is a database problem. Tool calling is an RPC problem. Multi-agent orchestration is a workflow-engine problem. Memory is a state-management problem. Evaluation is a monitoring problem. Latency and cost are capacity-planning problems. The model is one component in a system that is mostly not the model.

This reframing has a few non-obvious consequences.

• The first is that hiring shifts: the engineers who ship reliable LLM products are not primarily prompt engineers; they are infrastructure engineers who happen to work with a probabilistic component.

• The second is that architecture shifts: the most reliable systems we’ve seen in production are also the simplest, with as few agentic loops as the task tolerates and as much determinism as can be wrung out of the deterministic parts. Anthropic’s “Building Effective Agents” guidance, distilled, is use a workflow until you can’t.

• The third is that evaluation shifts: from leaderboard scores to production-derived test sets, with traces as the primary artifact, not benchmarks.

The teams that lead the next phase of LLM product building will be the ones that take this seriously. They will treat their LLM stack as a stochastic distributed system that occasionally calls a model. They will build observability before they build features. They will evaluate on real traffic. They will design for recovery, not for peak. They will choose simpler agentic graphs over more sophisticated ones. They will accept that the model is going to be wrong sometimes, and they will architect the system to notice, recover, and improve when it is.

That is the work.

References

Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). “Seven Failure Points When Engineering a Retrieval Augmented Generation System.” CAIN 2024 (3rd International Conference on AI Engineering – Software Engineering for AI). arXiv:2401.05856.

Bakke, C. (2024). “Chevrolet of Watsonville chatbot incident.” Documented at AI Incident Database, Incident #622, January 2024.

Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., & Stoica, I. (2025). “Why Do Multi-Agent LLM Systems Fail?” arXiv:2503.13657.

Chroma Research. (2024). “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Technical report.

Cognition. (2024). “Don’t Build Multi-Agents.” Cognition Labs blog post.

Databricks. (2024). “Long-Context RAG Performance of LLMs.” Databricks engineering blog.

Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). “RAGAS: Automated Evaluation of Retrieval Augmented Generation.” EACL 2024 Demo Track. arXiv:2309.15217.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics, Vol. 12. arXiv:2307.03172.

Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024). “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models.” Apple Machine Learning Research. arXiv:2410.05229.

Moffatt v. Air Canada. (2024). 2024 BCCRT 149. British Columbia Civil Resolution Tribunal, February 14, 2024.

Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., & Gonzalez, J. E. (2025). “The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models.” Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267:48371–48392.

TIME Magazine. (2024). “DPD AI Chatbot Swears, Calls Itself ‘Useless.’” January 2024 reporting on the DPD chatbot incident.

Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.” arXiv:2406.12045.

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023 (Datasets and Benchmarks Track). arXiv:2306.05685.

Anthropic. (2024). “Building Effective Agents.” Anthropic research blog, December 2024.