Every few months someone asks the same question in a planning meeting: “Models can read a million tokens now. Why are we still running a vector database?”
It’s a fair question, and in 2026 we finally have data to answer it. The short answer is that the naive RAG of 2023 is dead: fixed chunks, one embedding model, top-k by similarity, and hope. Retrieval itself is not. It grew up, and the reasons we still need it are different from what most people assume.
This issue covers the four things that changed, the five places RAG still breaks in production, and a short audit you can run on your own system.
The Big Idea: Four things that changed
1. Long context is now a real alternative, and it can be more accurate
The honest starting point is that stuffing documents into the prompt works, and often works better than a mediocre retriever.
In a June 2026 comparison on the same document-grounded task, Hamilton et al. found that long-context prompting reached the highest correctness, 73.1% against 65.4% for semantic RAG, but at 26 times the per-query token cost.
That tells you what the real trade-off is. Retrieval doesn’t win on raw accuracy when everything fits in the window. You keep retrieval for four reasons: cost, corpus size, freshness and permissions.
A practical rule from Anthropic: if your knowledge base is under about 200,000 tokens, roughly 500 pages, you can often put the whole thing in the prompt and skip RAG entirely.
2. Bigger windows are not better windows
The catch with long context is that “fits in the window” is not the same as “the model uses it well.”
Researchers at Chroma call this “context rot.” Even on deliberately simple tasks, model performance degrades as input length increases, often in surprising and uneven ways. Earlier, the well-known Lost in the Middle study showed that models use information at the start and end of a long input best, and do significantly worse when the relevant passage sits in the middle.
So the 2026 pattern is not “RAG or long context.” It’s retrieval to narrow the field, then a generous window to reason across what’s left.
3. Hybrid retrieval with context became the baseline
Pure vector search is no longer a serious production default. Embeddings are good at meaning and weak at exact matches: product codes, error IDs, names, clause numbers. Keyword search (BM25) is the opposite. Production systems run both and merge the results.
The biggest easy upgrade of the last two years is Contextual Retrieval. Before indexing, you prepend a short, model-generated note to each chunk explaining where it sits in its document. A chunk that says “revenue grew 3%” becomes a chunk that says which company and which quarter. In Anthropic’s tests:
Contextual embeddings cut retrieval failures by 35%
Adding contextual keyword search cut them by 49%
Adding a reranker on top cut them by 67%
The one-time cost was about $1.02 per million document tokens, using prompt caching so each document is only paid for once.
4. Retrieval became a tool the agent calls
The newest shift is architectural. Instead of retrieving once before the model answers, agents now decide when to search, what to search for, and whether the results are good enough.
Anthropic’s guidance on context engineering describes this as “just in time” context: keep lightweight references such as file paths, IDs and links in the prompt, and let the agent load details when it needs them. The goal is “the smallest possible set of high-signal tokens” that gets the job done.
This is powerful for open-ended, multi-step questions. It is also slower, more expensive and harder to debug, so treat it as the last step up, not the first.
Where production RAG still breaks
Despite all of the progress above, many RAG failures still happen before the model writes a single word. They fall into five buckets.
1. Parsing. Tables get flattened into word soup, scanned PDFs come through as nothing, and headings disappear, so chunks lose their structure. If a human can’t read your parsed text, the model can’t either. Use a layout-aware parser and keep tables as Markdown.
2. Retrieval recall. Teams blame the model for wrong answers when the right document was never retrieved in the first place. If the correct chunk isn’t in the top results, no prompt will save you. Measure this directly (see the audit below) before touching the prompt.
3. Permissions. Retrieval can surface documents the user isn’t allowed to see. Filtering after generation is too late, because the model has already read the text. Store access metadata with every chunk and filter before anything reaches the model.
4. Freshness and conflicting sources. Old policies, outdated docs and duplicate versions compete with current ones. The model will cheerfully cite the 2023 version. Re-index incrementally, store dates, and prefer the newest source when two conflict.
5. No evaluation. This is the root cause behind most of the others. Without a test set, every change is a guess, and every regression reaches users first.
The key point in the diagram: fix from left to right. A parsing failure caps retrieval quality, and a retrieval failure caps answer quality. Tuning the prompt while the parser is broken is wasted effort.
How to use it: a 30-minute RAG audit
Run this on any RAG system you own or inherit.
Collect 20 real questions that users actually asked. For each one, note which document or chunk should answer it.
Check retrieval alone. For each question, does the right chunk appear in the top 10 results? This single number tells you more than any answer-quality score.
Read 5 parsed documents as plain text. Look for broken tables, missing headings and garbled numbers.
Ask 3 questions your documents can’t answer. A healthy system says “not found.” An unhealthy one makes something up.
Test one permission boundary. Log in as a restricted user and ask about restricted content.
Write down your baseline, then change one thing at a time.
Here’s a minimal way to measure step 2:
def retrieval_hit_rate(eval_set, retrieve, k=10):
"""eval_set: [{"question": str, "relevant_ids": set[str]}, ...]
retrieve: your search function, returning chunks with a "chunk_id"."""
hits = 0
for item in eval_set:
results = retrieve(item["question"], k=k)
found = {r["chunk_id"] for r in results}
if found & item["relevant_ids"]:
hits += 1
return hits / len(eval_set)
# Example: print(f"Hit rate @10: {retrieval_hit_rate(eval_set, search):.0%}")
As a rule of thumb, if your hit rate at 10 is below about 80%, fix retrieval before anything else. Add keyword search, contextualize your chunks, and add a reranker, in that order.
The Filter
Four resources worth your time this week:
The Token Tax of Epistemic Accuracy: the clearest recent comparison of RAG against long context, framed as a cost and accuracy trade-off rather than a winner.
Context Rot: why a million-token window doesn’t mean a million useful tokens.
Introducing Contextual Retrieval: the highest-return retrieval upgrade most teams haven’t made yet, with the exact prompt.
Effective context engineering for AI agents: how retrieval changes when an agent, not a pipeline, decides what to load.
One question
What’s the most common failure in your RAG system right now: parsing, retrieval, permissions, stale data, or something I didn’t list? Reply and tell me. I read every response, and the best answers will shape a future Build Log.
New here? Subscribe free and get the RAG and AI Agents Cheat Sheet: a decision tree for choosing between RAG and long context, production defaults, real benchmark numbers and copy-paste prompt templates.
If this was useful, forward it to one person who is building with AI.