I have been staring at a customer support agent returning completely wrong shipping status information for two hours before I realized the model was not the problem at all. The retrieval was working. The embeddings were fine. The answer just happened to be based on order data from six hours ago, before the warehouse flagged the package for address verification. The agent told a customer their order was in transit. It was sitting in a distribution center. Both things were true at different points in time. Only one of them was true right now.
That experience is what pushed me to really think hard about what "production RAG" actually means. And the conclusion I kept arriving at is that most production RAG failures are not failures of the model, the embeddings, or even the retrieval logic. They are failures of the data architecture underneath. The model reasons correctly over the data it receives. The data it receives is wrong.
Table of Contents#
- The demo gap that nobody talks about
- A concrete example: "Why is my order late?"
- Failure mode 1: Stale state
- Failure mode 2: Slow retrieval
- Failure mode 3: Fragmented memory
- Failure mode 4: Disconnected tools
- Why better retrieval does not fix any of this
- What a context layer actually needs
- Series: The Context Layer Problem
- FAQ
The Demo Gap That Nobody Talks About#
Every RAG demo I have ever seen follows the same structure. There is a clean PDF, a small vector index, a well-formed question, and a good answer. The demo works because it was designed to work. The document set is small, static, and curated. The query is within distribution. Nobody is asking ambiguous questions about live operational data.
The demo proves your embedding pipeline is functional. It proves absolutely nothing about production.
At production time, the agent is not working against a curated document corpus. It is working against live operational systems that change constantly, that live in multiple places with different schemas and different authentication models, and that need to be queried together to produce an accurate answer. The gap between those two scenarios is not a matter of scale. It is a matter of architecture.
I have seen teams spend weeks tuning their embedding model, experimenting with different chunking strategies, and adding rerankers before they realize the issue was never retrieval quality. The issue was that the data they were retrieving was stale, fragmented across three disconnected sources, or simply structured in a way that semantic search was never going to handle correctly. No embedding model bridges that gap.
A Concrete Example: "Why Is My Order Late?"#
This is the scenario I keep returning to because it exposes every failure mode at once.
A customer support agent receives the question: "Why is my order late?"
To answer that accurately, the agent needs access to at minimum five data sources simultaneously. The customer database tells you who this person is and what their account standing is. The order management system tells you what was ordered, when it was placed, and what the current fulfillment status is. The shipping provider API tells you where the package physically is right now and whether there are any carrier delays. The ticketing system tells you whether this customer has already opened a case about this specific order. The policy documentation tells you what the SLA commitments are and what compensation thresholds apply to delayed orders.
That is not a retrieval quality problem. That is a data architecture problem.
If you approach this with naive RAG, you have roughly two options. You can ingest all five sources as text chunks into a single vector store, or you can query one source and treat the result as complete. Neither works. A vector search over embedded shipping API responses does not give you structured tracking state. An embedding of yesterday's order record does not reflect this morning's fulfillment event. And the agent has no mechanism to understand that these five sources need to be joined, that answering "why is my order late for customer X" requires correlating a customer ID across multiple systems, not running a semantic similarity search.
This is the scenario that breaks production agents. Not adversarial prompts. Not hallucination. An ordinary customer support question requiring real-time, multi-source, relationship-aware data retrieval.
Failure Mode 1: Stale State#
Order status changes when the warehouse scans a package. A support ticket opens when the customer calls. Inventory adjusts when another order ships. These are not edge cases. They are the normal operating rhythm of any e-commerce system.
Naive RAG has no sync mechanism. It retrieves from whatever was indexed at the last crawl. If your pipeline re-indexes every fifteen minutes, the agent makes decisions based on data that is up to fifteen minutes out of date. For inventory levels, pricing, and order status, that is not stale. That is wrong.
The shipping delay scenario I opened with is exactly this. The agent told the customer their order was "in transit" because the indexed copy reflected the status from six hours ago, before the address verification flag was added. The embedding model had no way to know the data was stale. The agent had no way to know. The retrieval pipeline just returned the most semantically relevant chunk, which happened to contain outdated information.
Faster re-indexing helps at the margins. Re-indexing every five minutes instead of every fifteen minutes narrows the stale window but does not close it. The real fix is continuous data sync via change data capture, where individual change events propagate into the retrieval layer as they are committed to the source system, not on a schedule. That is a fundamentally different architecture from periodic re-indexing, not a tuned version of the same one.
Failure Mode 2: Slow Retrieval#
Agentic RAG loops are sequential by default. The agent retrieves context, reasons over it, decides it needs more context, retrieves again, reasons some more, decides to call a tool, processes the result, and eventually produces an answer. Each hop adds latency.
At 800ms per retrieval call, which is not unreasonable for a vector search over a large corpus with an embedding round-trip, a three-hop retrieval chain adds 2.4 seconds before the agent has produced its first reasoning token. Add a tool call to the shipping API at another 400ms, add a retrieval for policy documentation at another 800ms, and you are looking at 3.6 seconds of retrieval latency alone. Under real query volumes with hundreds of concurrent sessions, this compounds into queue depth problems, timeout errors, and degraded response quality as agents truncate their reasoning chains to meet SLA thresholds.
The temptation is to parallelize retrieval calls. That helps. But the deeper issue is that a good context layer should return structured data and semantic search results together in a single query, reducing the total number of hops required rather than just running the same number of hops concurrently.
Failure Mode 3: Fragmented Memory#
The customer contacted support three weeks ago about a different order. During that interaction, the agent learned something useful: this customer always ships to a work address on Tuesdays, and the carrier reliably fails to deliver to that address before 9am. That pattern is directly relevant to today's question about why an order that was shipped on a Tuesday has not been received.
The agent does not know this. Every conversation starts cold. There is no persistent memory layer in a naive RAG architecture. No mechanism for an agent to record what it learned in session N and retrieve it in session N+47. User preferences are re-discovered on every conversation. Resolved failure patterns are not stored anywhere the agent can reach. Cross-session context simply does not exist.
This is not just inefficient. It produces genuinely worse outcomes. An agent with access to resolved-case history, user preference signals, and prior interaction context answers the "why is my order late" question with more precision than one treating every query as its first. Fragmented memory is an architecture gap and it requires purpose-built memory infrastructure, not a larger vector index.
I wrote more about the different ways agents can handle memory across episodic, semantic, and procedural layers. The short version is that most teams building RAG pipelines today have essentially no agent memory beyond the current context window.
Failure Mode 4: Disconnected Tools#
The support agent needs the customer database, the order management system, the shipping API, the ticketing system, and the policy documentation. Those are four structured operational systems and one unstructured document corpus. They live in separate services with separate schemas, separate authentication models, and separate query interfaces.
At agent execution time, joining data across these sources means the agent calls each one individually, receives heterogeneous responses with different structures, and performs the join in its reasoning context. Under controlled conditions with predictable queries, this sometimes works. Under real query variance, with ambiguous order references, multiple open tickets, and partial shipping data from the carrier, it fails unpredictably. The agent misses a join condition, retrieves incomplete data from one source, and reasons to a confident but wrong answer.
The observable failure here is subtle. The agent does not error. It produces a response. The response is confident. It is wrong because it is missing half the data it needed to be right. This is one of the failure patterns I find hardest to catch in testing because the agent is doing exactly what you asked it to do with the information it was able to retrieve.
A unified query interface across structured and vector sources is not a quality-of-life improvement. It is what makes multi-source agentic reasoning reliable at all. When you are evaluating production agent failures, it is worth reading through how the agent observability gap makes these kinds of disconnected-tool failures particularly hard to diagnose.
Why Better Retrieval Does Not Fix Any of This#
The response to production RAG failures starting in 2024 was a wave of retrieval improvements. Rerankers. Query rewriting. HyDE. Hybrid dense-sparse search. Late interaction models. These are real techniques. Reranking surfaces more relevant chunks from the same index. Query rewriting handles reformulations that would otherwise miss. Hybrid search improves recall on sparse queries. All of this is genuine progress.
None of it fixes stale state. None of it fixes disconnected sources. None of it gives your agent persistent memory across sessions.
That is not a criticism of retrieval research. These techniques are solving the problem they were designed to solve: retrieval quality from a static corpus. The problem is that production agents are not operating against static corpora. Applying a reranker to stale data returns the most relevant stale result. Running query rewriting against five disconnected sources still requires five separate round-trips and a join in the reasoning context.
The "RAG is dead" discourse that circulated in late 2024 was equally unhelpful in the other direction. It conflated the failure of naive RAG architectures with a failure of retrieval-augmented generation as a concept. Retrieval is not the problem. The problem is treating retrieval as an afterthought, a vector store bolted onto an LLM, rather than as a first-class infrastructure concern that must handle freshness, latency, memory, and multi-source unification.
The agent framework explosion gave teams more tools to orchestrate retrieval. But orchestration without a coherent context layer underneath is just better-organized access to the same broken data.
What a Context Layer Actually Needs#
After thinking through these failure modes, I arrived at four properties that a real context layer for production agents has to have. Not nice-to-haves. Not configuration options. Requirements.
First, always-fresh data via change data capture from operational databases. Not periodic re-indexing on a schedule. Continuous ingestion so the data the agent retrieves reflects the current state of the source systems, not the state from fifteen minutes ago or an hour ago.
Second, fast structured and vector query together in a single round-trip. Not a vector store for semantic search plus a separate database for structured lookups plus a third system for relational queries. A unified layer that returns structured and semantic results together, which is what eliminates the multi-hop chains that make agentic latency unacceptable at scale.
Third, persistent agent memory with session context stored under a configurable TTL, and a mechanism to promote extracted preferences and learned patterns to long-term storage that survives across conversations. Agents need to write what they learn and retrieve it later. The context window boundary is not the right scope for memory.
Fourth, a unified tool interface. One query entry point that abstracts across all the data sources the agent needs, so the agent is not performing ad-hoc joins in its reasoning context under live query variance.
Redis Iris, announced recently by Redis, is one of the more architecturally serious attempts to build exactly this. It is designed specifically for the fast-changing operational data problem, not document retrieval, not batch analytics, but live agent context at production latency requirements. That is what Part 2 gets into.
The relevant context from agentic LLM workflow patterns is that the retrieval step is typically the most failure-prone part of any agentic pipeline. The patterns people use to work around retrieval failures, adding retry logic, adding fallback sources, broadening queries, are all treating symptoms. The underlying architecture is the cause.
Series: The Context Layer Problem#
This post is the first of four. The failure modes above set up the rest of the series.
Part 2 breaks down how Redis Iris actually works, component by component, with honest tradeoffs on each piece. Part 3 compares Redis Iris against Pinecone Nexus and naive RAG across three concrete scenarios, with a decision framework built around the one axis that actually matters. Part 4 is the honest builder verdict with the 5-question checklist I use to decide whether this architecture fits a given situation.
This Post is Part of a Series#
The Context Layer Problem is a 4-part series on why retrieval fails in production AI systems and what to do about it.
- Part 1: Why Your RAG Pipeline Fails in Production — the 4 runtime failure modes
- Part 2: How Redis Iris Actually Works — RDI, Context Retriever, Memory, LangCache
- Part 3: Redis Iris vs. Pinecone Nexus vs. Naive RAG — decision framework
- Part 4: Should You Actually Use Redis Iris? — honest builder verdict
FAQ#
What are the most common production RAG failures?#
The four I see consistently are stale state, where indexed data no longer reflects reality in the source systems; slow retrieval, where multi-hop agentic chains accumulate enough latency to cause timeout errors and degraded response quality; fragmented memory, where agents have no persistent context across sessions and re-discover everything from scratch on each conversation; and disconnected tools, where the agent must join data from multiple heterogeneous sources in its reasoning context under real query variance. Retrieval quality problems like low precision or missing relevant chunks are real, but they are less common as a root cause than these four architecture problems.
Does adding a reranker fix RAG production failures?#
A reranker improves retrieval quality from a fixed index. It does not fix stale data, it does not give your agent cross-session memory, and it does not unify your retrieval surface across multiple data sources. If your failure mode is low precision retrieval from a correct, fresh, unified index, a reranker helps. If your failure mode is any of the four I described above, a reranker makes no difference.
How stale is too stale for a production RAG system?#
It depends entirely on the data. For a policy document that changes quarterly, data that is a week old is fine. For order status in an active e-commerce system, data that is fifteen minutes old can produce wrong answers that damage customer trust. There is no universal threshold. The question to ask is: what is the worst outcome if my agent answers based on data from two hours ago? If the answer is "the customer gets incorrect information about a time-sensitive situation," your re-indexing cadence is not the right solution.
Why can't I just query my production database directly from the agent?#
Two reasons. First, agents at production scale can generate query volumes that no transactional database was built to handle. A human analyst might run fifty queries in a workday. An agentic pipeline running in parallel across many sessions can generate that volume in seconds. Second, relational schemas are optimized for write consistency, not agent-time reads. A normalized schema with orders, line items, products, and customers across multiple tables requires the agent to construct joins, resolve foreign keys, and reconstruct business objects under latency pressure. The agent does not need normalized correctness. It needs a pre-joined, pre-indexed representation shaped for fast reads.
What is change data capture and why does it matter for RAG?#
Change data capture tracks individual write events at the transaction log level in a source database, and streams them into a downstream system as they are committed. For RAG, this means the retrieval layer can stay synchronized with the source system continuously rather than on a periodic re-index schedule. An order status update propagates to the retrieval layer in seconds, not on the next cron job run. For data that changes frequently, CDC is the difference between an agent that sees current reality and one that operates on a stale snapshot.
Is naive RAG always wrong for production?#
No. For static document corpora, internal knowledge bases, FAQ bots, and single-source retrieval over content that changes infrequently, naive RAG is the right tool. Adding CDC infrastructure and entity model maintenance to a quarterly-updated document corpus is waste, not sophistication. The failure of naive RAG is specific: it falls apart when the data is fast-changing, multi-source, or requires relationship-aware retrieval. For use cases that do not hit those walls, it is entirely appropriate.
How does agent memory relate to retrieval failures?#
Most production RAG architectures treat the current context window as the only memory the agent has. Every conversation starts cold. The agent cannot retrieve insights from prior sessions, cannot access learned patterns about a specific user, and cannot build on context that was established in a different conversation. This makes agents structurally less useful over time than they should be, because they never accumulate knowledge about the users and situations they encounter. The memory problem is adjacent to but distinct from retrieval quality. You can have excellent retrieval from external sources and still have no memory of prior interactions. Both matter for production quality.