Why Your AI Agents Keep Failing (It Is Not the Model)

The first time one of my agents failed in production, I did what almost everyone does. I blamed the model. I swapped it for a bigger one. The failure came back the next day, in a slightly different shape, and I spent another afternoon convinced the weights were the problem. They were not. The model had done exactly what I asked. The problem was what I had put in front of it.

Context engineering is the practice of deciding which information enters an AI model's context window at each step of an agent's run, and in what form. It matters because most agent failures come from what the model was given, not from the model itself. It differs from prompt engineering, which shapes how a single instruction is written rather than managing the full set of tokens across a multi-step task.

That distinction sounds academic until you have shipped an agent and watched it break in ways no prompt tweak can fix. This post is about where agents actually fail, backed by a study of more than a thousand real bug reports, and about the four operations you can use to fix the layer that produces most of those failures.

Table of Contents#

• What Is Context Engineering and Why Does It Replace Prompt Engineering?
• Why Do Most Agent Failures Start Before the Model Runs?
• How Does the Lost in the Middle Effect Break Production Agents?
• What Are the Four Operations of Context Engineering?
• How Do You Apply WRITE, SELECT, COMPRESS, and ISOLATE in Practice?
• What Does Context Engineering Actually Improve in Production?
• FAQ

What Is Context Engineering and Why Does It Replace Prompt Engineering?#

For a couple of years the industry talked about prompt engineering as if it were the whole game. Write the right instruction, get the right output. That framing worked when the interaction was a single request and a single response. It stops working the moment you build an agent that runs for fifty turns, calls tools, reads their results, and accumulates a history that grows with every step.

Tobi Lütke, the CEO of Shopify, put the shift plainly in a post on June 19, 2025: "I really like the term 'context engineering' over prompt engineering. It describes the core skill better: the art of providing all the context for the task to be plausibly solvable by the LLM." The key word there is plausibly. The model can only solve what the context makes solvable. If the right information is absent, buried, or drowned out by noise, no amount of clever instruction phrasing rescues it.

Andrej Karpathy expanded on this a few days later, on June 25, 2025: "Context engineering is the delicate art and science of filling the context window... Too little or of the wrong form and the LLM doesn't have the right context for optimal performance. Too much or too irrelevant and the LLM costs might go up and performance might come down." Both failure modes are real. Starving the model and flooding the model produce different symptoms, but both are context problems, not model problems.

This is why I now think of prompt engineering as a subset of context engineering. Writing the instruction is one decision among many. The larger job is managing everything the model sees at every point in the run. I have written before about how Anthropic frames this same discipline for production agents, and the framing there matches what I keep seeing in my own systems.

Why Do Most Agent Failures Start Before the Model Runs?#

Most agent failures start before the model runs because the failure lives in the pipeline that assembles context, not in the inference step itself. According to Islam et al. in "When Agents Fail: A Comprehensive Study of Bugs in LLM Agents" (arXiv:2601.15232), a study of 1,187 real bug reports drawn from Stack Overflow, GitHub, and Hugging Face across seven major LLM frameworks, 58% of all agent bugs lived in the agent core and context layer. Not in the model weights. The paper breaks the remaining failures into tool integration at 21%, planning at 13%, and memory at 8%. So the single largest category of production bugs is the layer that decides what the model gets to see. When people say their agent is unreliable, they usually mean their context pipeline is unreliable, and they have misattributed the cause to the model because the model is the visible part.

The clearest example I know of is n8n version 2.6.3, released in February 2026. It broke function calling across both OpenAI and Anthropic at the same time. A single vendor bug is one thing. A bug that hits two independent model providers simultaneously tells you the problem is not the model at all. The cause was a tool schema cache that had gone stale after an upgrade. The model never ran. The context pipeline failed before inference even started, and the symptom looked like the models had forgotten how to call functions.

Google's AI Overviews gave the internet a more famous version of the same lesson in 2024, when the system recommended adding glue to pizza to keep the cheese from sliding off. The model did not invent that. Retrieval upstream had pulled a Reddit joke thread into the context, and the model followed its instructions faithfully on top of bad source documents. The failure was in selection, the step that chose which documents to retrieve, not in generation.

Here is a failure from my own work that made the pattern concrete. A database migration agent inserted 127 soft-deleted records that it had been explicitly told to skip. The skip instruction was right there in the original prompt. But by turn 50, that instruction sat buried under 160,000 tokens of accumulated history, and roughly 60% of those tokens were irrelevant tool results from steps that had already completed. The instruction was still technically in context. It had just been diluted into irrelevance.

How Does the Lost in the Middle Effect Break Production Agents?#

The lost in the middle effect breaks production agents because model attention does not distribute evenly across a long context, and information in the middle of a large input gets systematically underweighted. Liu et al. documented this in "Lost in the Middle: How Language Models Use Long Contexts," published in TACL 2024. Their finding is specific and unsettling. When the document containing the answer sits at position 10 in a 20-document context instead of position 1, accuracy drops by more than 30%. Same model, same question, same documents. The only thing that changed was where the relevant information sat in the sequence. Models attend most strongly to the beginning and end of their context and let the middle sag. This is why the strength of a primary instruction depends not just on its wording but on its position, and why a correct instruction can effectively vanish inside a long enough context window.

This effect compounds with length in a way that is easy to underestimate. There is a widely cited degradation pattern with GPT-4o where accuracy fell from 99.3% to 69.7% as the context grew. Same model. Same prompt. The only variable was more input. A thirty-point accuracy collapse driven entirely by context volume is not a rounding error. It is the difference between a system you can trust and one you cannot.

Put the two findings together and you get the core mechanic behind my migration agent bug. The skip instruction was correct, but it had drifted toward the middle of a bloated context, and the context had grown large enough to degrade attention on its own. Two independent failure modes stacked on top of each other, and neither one had anything to do with the quality of the model. This is the same class of problem I dug into when writing about why sending less context cut our LLM costs and raised accuracy at the same time. Less really can be more, and the reason is mechanical, not aesthetic.

What Are the Four Operations of Context Engineering?#

If the problem is what the model sees, the solution is a set of levers for controlling what the model sees. Lance Martin, in his June 23, 2025 write-up on context engineering for LangChain, organizes these into four operations: WRITE, SELECT, COMPRESS, and ISOLATE. I find this framing useful because each operation maps to a specific failure mode, so when an agent breaks you can reason about which lever was missing.

WRITE means saving information outside the context window, in scratchpads or long-term memory, so the model does not have to carry everything at once. Anthropic's multi-agent researcher does exactly this. When it approaches the 200,000 token limit, it saves its plan to memory rather than letting the plan get crowded out by later work.

SELECT means pulling the relevant information into context at the moment it is needed, through retrieval, semantic search, or tool selection. When you are managing a large collection of tools, careful selection produces up to a threefold improvement in tool selection accuracy compared to loading everything at once.

COMPRESS means retaining only the tokens that carry signal and dropping the rest. JetBrains reported in 2025 that observation masking over agent trajectories of 250-plus turns reduced costs by 52% while improving solve rates by 2.6%, and did so more effectively than LLM summarization.

ISOLATE means splitting work across separate context windows, through multi-agent designs or sandboxing. HuggingFace's deep researcher keeps token-heavy objects as variables in a sandbox and never passes them through the model at all.

Here is how the four operations map to the failures above.

How Do You Apply WRITE, SELECT, COMPRESS, and ISOLATE in Practice?#

The operations are only useful if you can see which one you need. The way I do this now is to trace a failing run and ask which lever was missing at the point of failure. That requires instrumentation, which is its own topic. If your agent has no tracing, you are guessing, and I have written separately about the observability gap that makes agents impossible to debug. Assume you have traces. Here is how the diagnosis goes.

My migration agent failed on COMPRESS. Sixty percent of its context was completed tool results with no forward value. The fix was observation masking: once a step finished, its verbose tool output got replaced with a short summary of the outcome, and the skip instruction rose back to the top of what mattered. The JetBrains result suggests this is not a niche trick. Masking beat summarization on both cost and solve rate, which surprised me, because summarization feels more sophisticated. It turns out that dropping noise cleanly is often better than paraphrasing it.

The n8n incident was a SELECT failure at the schema layer. A stale cache selected the wrong tool definitions. The fix there is cache invalidation on upgrade, plus a validation step that checks the schema the model receives against the live tool signature before the run starts. Boring plumbing. It is exactly the kind of boring plumbing that the Islam et al. study says accounts for the plurality of production bugs.

The Google Overviews failure was a SELECT problem too, one level up. Retrieval pulled a joke thread. The fix is source filtering and reranking so that low-authority documents do not reach the context in the first place. The model was never going to save you from a bad document if the document was sitting right there in its input.

WRITE and ISOLATE come into play as agents get longer and more parallel. WRITE is what keeps a long-running agent from losing its own plan. ISOLATE is what keeps a token-heavy subtask from polluting the main context, and it is the reason multi-agent designs can outperform a single overloaded context window. I dug into the tradeoffs of that approach in my breakdown of when an agent swarm actually earns its cost, because isolation is not free and it is easy to over-apply.

The point is that each lever answers a specific failure. You do not apply all four blindly. You trace the run, find the layer that failed, and reach for the operation that governs that layer.

What Does Context Engineering Actually Improve in Production?#

The reason to care about all of this is that the gains are large and they do not require touching the model. Applying all four operations together has been shown to move task success from 41% to 73% on the same underlying model. That is a thirty-two point swing produced entirely by managing context better. No fine-tuning, no larger model, no new weights. Just better decisions about what the model sees and when.

Anthropic frames the target cleanly in its guide to effective context engineering for AI agents: the goal is to "find the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome." I keep that sentence pinned because it inverts the instinct most of us start with. The instinct is to give the model more, on the theory that more information is safer. The evidence says the opposite. More context degrades attention, dilutes instructions, raises cost, and increases the surface area for the kind of stale-cache and bad-document failures that the Islam et al. study counted. Smallest and highest-signal, not largest and most complete.

There is a deeper reason this matters for how you spend your time. If 58% of bugs live in the context layer, then most of your reliability work is context work, and most of your model-swapping is misdirected effort. When I stopped reaching for a bigger model and started tracing what my agents were actually being shown, the failures became tractable. They stopped looking like mysterious model misbehavior and started looking like ordinary engineering problems: a cache that was not invalidated, a document that should have been filtered, a history that should have been compressed. Those are problems you can fix.

The model is rarely the thing that failed. The context in front of it is. Once you internalize that, the whole discipline of building reliable agents reorganizes itself around a single question at every step: what does the model need to see right now, and what is just noise?

FAQ#

What is context engineering in simple terms?#

Context engineering is the practice of controlling exactly what information an AI model sees at each step of a task, and in what form. Instead of writing one clever instruction, you manage the full set of tokens the model receives across a multi-step run. The goal, in Anthropic's words, is the smallest set of high-signal tokens that make the desired outcome likely. It is the discipline that determines whether an agent stays reliable as its context grows.

Is context engineering the same as prompt engineering?#

No. Prompt engineering shapes how a single instruction is written. Context engineering manages everything the model sees across an entire multi-step run, including tool results, retrieved documents, memory, and history. Prompt engineering is best understood as one small part of context engineering. Tobi Lütke and Andrej Karpathy both argued in mid-2025 that context engineering describes the core skill more accurately, because filling the context window well is what actually determines whether a task is solvable.

Why do most AI agent failures happen in the context layer?#

According to Islam et al. (arXiv:2601.15232), a study of 1,187 real bug reports across seven frameworks, 58% of all agent bugs live in the agent core and context layer, not in model weights. Tool integration accounts for 21%, planning for 13%, and memory for 8%. The context pipeline is where information gets assembled, cached, retrieved, and truncated, so it is where the most things can go wrong before the model ever runs.

What is the lost in the middle effect?#

The lost in the middle effect, documented by Liu et al. in TACL 2024, is the tendency of language models to underweight information placed in the middle of a long context. When the answer document sits at position 10 in a 20-document context instead of position 1, accuracy drops by more than 30%. Models attend most strongly to the start and end of their input, so a correct instruction can effectively disappear if it drifts into a long context's middle.

Does adding more context make an agent more reliable?#

Usually the opposite. There is a documented GPT-4o pattern where accuracy fell from 99.3% to 69.7% as context grew, with the same model and the same prompt. More input degrades attention, dilutes key instructions, raises cost, and widens the surface area for failures like stale caches and bad retrieved documents. The evidence favors the smallest high-signal context over the largest complete one.

How much can context engineering improve results?#

Applying all four operations, WRITE, SELECT, COMPRESS, and ISOLATE, has been shown to move task success from 41% to 73% on the same underlying model, with no fine-tuning. On the compression lever specifically, JetBrains reported in 2025 that observation masking over 250-plus turn trajectories cut costs by 52% while improving solve rates by 2.6%. The gains come from managing context, not from changing the model.

Where should I start if my agent is unreliable?#

Start with tracing, so you can see what the model was actually shown at the point of failure. Then map the failure to one of the four operations: a bloated history is a COMPRESS problem, a wrong retrieved document or stale tool schema is a SELECT problem, a lost plan is a WRITE problem, and a token-heavy subtask polluting the main context is an ISOLATE problem. Fix the layer that failed rather than swapping the model.