I spent a weekend last month ripping out a retrieval pipeline I had built six months earlier. The feature was a support-ticket triage bot that pulled relevant docs from a vector database, stuffed the top matches into the prompt, and asked Claude to draft a response. The whole thing worked, but the plumbing around it was becoming a second job. Re-indexing whenever docs changed. Tuning the chunk size every time a new doc type landed. Debugging why a clearly relevant doc ranked fourth instead of first.
The replacement took me a Sunday afternoon. I dropped the vector DB entirely, concatenated all 180,000 tokens of support docs into a single system prompt, enabled prompt caching, and sent every ticket to Claude with the full doc set in context. Quality went up. Latency went up too, but not as much as I expected. Cost went down once caching kicked in. The whole pipeline fit in about 60 lines of code.
That success made me cocky. The next week I tried the same thing on our codebase assistant, which searches across 400,000 tokens of source code to answer developer questions. I yanked out the retrieval, stuffed the whole repo into context, and waited for the same magic to happen.
It did not. Quality got worse. Costs tripled. Users complained. I spent the next week quietly putting RAG back.
The lesson was that long context does not kill RAG. It changes the shape of the decision. This post is the framework I wish I had before I threw out the first pipeline and the mental model that kept me from making the same mistake a third time.
Why This Question Suddenly Matters
Two years ago nobody asked whether to use RAG. You used RAG because context windows were 8k or 16k tokens and anything useful would not fit. The question was only which vector DB, which embedding model, and which chunking strategy.
That world is gone. Here is what the current landscape looks like in early 2026:
- Claude Opus 4.7 and Sonnet 4.6 ship with a 1 million token context window
- Gemini 2.5 Pro offers 2 million tokens
- GPT-5 sits at 400,000 tokens
- Prompt caching on Anthropic, OpenAI, and Google cuts repeat-token costs by 75 to 90 percent
- Token prices have fallen roughly 60 percent year over year
Each of those shifts moves the calculus. A 200,000 token doc set that would have cost dollars per query in 2023 costs cents in 2026, especially if the tokens hit a warm cache. The “too expensive to be worth it” wall that made RAG mandatory has moved a long way out.
But long context is not free, and the cost curve is not the only thing that matters. There is a quality story, a latency story, and a developer experience story, and each of them cuts differently depending on what you are actually trying to do.
The Case For Stuffing The Window
Let me steelman long context first, because it is the approach I reflexively underestimate and I want to be honest about where it wins.
Simplicity is a real feature. A RAG pipeline has at least five moving parts. An embedding model, a vector store, a retriever, a reranker, and the prompt template that stitches everything together. Each one has its own failure modes. Each one adds ops work when it breaks. Stuffing the window replaces all five with “put the data in the prompt.” That simplicity is worth something, and the something is more than pride. It is fewer bugs, faster changes, and less time staring at the retrieval logs wondering why your system returned the wrong chunk.
Quality is often better when relevance is ambiguous. RAG is great when you know exactly which doc has the answer. It is bad when the answer requires piecing together information from multiple docs that do not obviously match the query. Long context lets the model do the cross-referencing itself, which is often what humans want when they ask a question. My support-ticket bot was in exactly this category. The right answer frequently drew from three or four docs where none of them contained the exact phrase the user had typed.
Prompt caching makes the cost palatable. Without caching, running a 200,000 token prompt on every request would be a budget disaster. With caching, the static portion of the prompt pays full price once, gets read from cache on subsequent requests at a fraction of the cost, and refreshes its TTL every time it is hit. The math on a stable doc set changes from “this is too expensive” to “this is a rounding error once traffic is steady.” LLM cost optimization in 2026 increasingly lives or dies on whether you actually understand how caching works on your provider.
No training time, no re-indexing. When your docs change, you update the prompt. There is no embedding to regenerate, no index to rebuild, no stale-data debugging. For docs that change frequently, this is a significant operational win.
Better attention patterns than you expect. The “lost in the middle” problem was a 2023 concern that has mostly been engineered around in the frontier models. Claude, GPT, and Gemini all handle mid-context retrieval reasonably well now. Still not perfect. Still worth structuring your prompt so important information is near the top or bottom. But the crippling 2023-era decay is not what it was.
The Case For Sticking With RAG
Now let me steelman RAG, because it still wins in entire categories of problem and “just use long context” is a meme that sometimes leads developers into bad architectural calls.
Long context costs scale linearly with your data. Every query pays for the entire doc set even if the answer is in one paragraph. At 200,000 tokens, this is cheap with caching. At 2 million tokens, it is not. At 20 million tokens, which is roughly where any real enterprise knowledge base lives, it is architecturally impossible. RAG stays viable at any scale because it only pays for what it retrieves.
Latency is real and it adds up. A 200,000 token prompt to Claude Opus 4.7 takes roughly 6 to 12 seconds to return a response, even with caching. A RAG setup that retrieves 4,000 tokens and sends them to the same model returns in 2 to 4 seconds. For a batch job this does not matter. For a user-facing chatbot where the user is waiting, every second counts.
Attention degrades at the edges even now. Even with the improvements since 2023, model accuracy on needle-in-haystack retrieval tasks still drops 10 to 20 percent between 50k and 500k tokens. If you care about catching every relevant detail in the docs, RAG plus a reranker is still more reliable than stuffing everything in context.
Dynamic data is hard to stuff. If your data changes per query, per user, or per session, you cannot get the caching discount. That removes the biggest cost advantage of long context. Multi-tenant apps where every user has different data access are a classic RAG use case and long context does not change that.
Citations are easier. RAG pipelines can return the source of every fact because they know which chunk they retrieved. With long context you either trust the model to cite correctly, which it sometimes does not, or you build a separate post-processing step that tries to map claims back to positions in the prompt. The RAG approach is architecturally cleaner for any use case where trust and attribution matter.
The Real Decision Framework
After a couple of months of running both patterns in production, here is the framework I actually use when someone asks whether a new feature should be RAG or long context.
Size
If your total data fits in 500,000 tokens, long context is on the table. If it fits in 200,000 tokens, long context is often the better default. Over 1 million tokens, RAG is probably mandatory. Over 10 million tokens, it is definitely mandatory.
Stability
If your data is the same for every user and changes rarely, long context plus prompt caching is a strong default. If your data is per-user, per-tenant, or per-session, caching breaks and RAG becomes more attractive. If your data changes multiple times per day, the operational cost of re-indexing can still tilt the scales toward long context even for medium-sized corpora.
Latency requirements
For user-facing chat where response speed matters, RAG is almost always faster. For batch processing, async workflows, or use cases where a 10-second response is acceptable, long context is fine. If you are building something that plugs into a durable workflow engine anyway, the latency hit of long context may not matter.
Query pattern
If most queries are narrow lookups with clear keywords, RAG works great. If queries often require synthesizing across multiple documents or making inferences that span the corpus, long context usually produces better answers.
Citation requirements
If you need to cite specific sources in every response, RAG is the cleaner path. If citations are nice-to-have or the response just needs to be useful, long context is fine.
What I Actually Built
Let me get concrete about the two features I mentioned at the top, because they map cleanly onto this framework and they are the exact kinds of decisions you will make on your own features.
Support ticket triage: long context won
The support docs were about 180,000 tokens total. They changed maybe once a week when the docs team pushed an update. Every user asked similar questions against the same doc set. Most tickets required piecing information from three or four docs. Users did not care if a response took 8 seconds instead of 3, because they had already submitted a ticket and were waiting for a reply by email.
This is long-context paradise. Data size is small enough to fit comfortably in a prompt. Stability is high so caching works. Latency requirements are loose. Queries require synthesis, which is where long context beats retrieval. Citations are nice but not required.
Switching to long context dropped maintenance overhead to near zero. Re-indexing was eliminated. Chunk-size tuning was eliminated. The retriever, which had been my biggest source of bugs, was just gone. Quality went up because the model could pull context from anywhere in the doc set instead of being stuck with whatever the retriever handed it. Costs dropped once caching was active because the system prompt had a 90 percent cache hit rate.
Codebase assistant: RAG won, and it was not close
The codebase was around 400,000 tokens, which still fits in a context window. The problem was that code is not like docs. Users asked questions like “how does billing work” or “where is the webhook handler” that require finding specific files. Response time mattered because developers lose patience fast when their tools are slow. And the whole corpus was not stable. Half a dozen files changed per day, which meant the cache was permanently cold for anything touching active development.
I tried long context anyway because it worked so well the first time. The results were miserable. Quality was lower because the model would drift into tangentially related code instead of answering the specific question. Latency hit 15 seconds on queries that had been 3 seconds under RAG. Costs tripled because the cache was never warm. And I could not cite file paths cleanly because the model would describe code instead of pointing at it.
I rebuilt RAG with a better retriever, added a reranker, and shipped the whole thing at roughly 3-second response time with precise file citations. The RAG version is still in production. I have not tried to replace it again.
The Hybrid Pattern That Actually Ships
The honest production pattern in 2026 is not “pick one.” It is “use RAG to narrow the search space, then use long context to let the model reason across what you retrieved.”
Instead of retrieving 4,000 tokens and hoping they contain the answer, you retrieve 100,000 tokens and let the model sort through them. Instead of a dozen painfully chunked paragraphs, you retrieve 10 full documents at their natural boundaries. Instead of a top-1 match that might be wrong, you retrieve a top-20 and let the model figure out which ones matter.
This pattern gets the best of both. You keep the cost advantage of not paying for your entire corpus on every query. You keep the latency advantage because you are still sending 100,000 tokens instead of a million. And you get the quality advantage of long context because the model has enough room to actually think about the problem instead of being handed the bare minimum.
The catch is that the retriever matters less than it used to. Getting the top-3 exactly right was critical when the prompt could only hold the top-3. Getting a reasonable top-20 is much easier than getting a perfect top-3, and the model will do the final filtering for you. That shifts the ops story too. You tune your retriever for recall, not precision, and you let the model handle the rest.
Most of the production AI features I have built in the last six months use this pattern. RAG for the first filter, long context for the reasoning. It is not as clean as “we use Pinecone” or “we just stuff the window,” but it reflects what actually works.
Prompt Caching Is The Quiet Unlock
I want to be blunt about something. None of the long context math works without prompt caching, and a lot of developers still treat caching as an optimization they will get around to later. Treat it as mandatory.
Without caching, a 200,000 token prompt at Claude Opus 4.7 input rates runs about 60 cents per query. At 10,000 queries per day, that is 6,000 dollars per day, which is an insane number for a support bot. With caching on the stable system prompt, the same workflow drops to around 6 to 8 cents per query once the cache is warm. The math only works at the second number.
The catch is that caching has rules and developers keep getting them wrong. The prompt prefix must be byte-for-byte identical across requests. You place cache breakpoints explicitly. You reset the cache TTL on every hit so it stays warm under steady traffic. You structure your prompt with the long stable portion first and the short per-query portion last. If any of these are off, your cache hit rate drops to near zero and your beautiful long-context architecture becomes a money fire.
This is also why long context works great for feature-level prompts with shared context and terribly for use cases where every prompt is different. Multi-tenant apps where every user has their own data are not a good fit because you need a separate cache entry per tenant. Single-tenant features or features with a shared doc set are the sweet spot.
If you are weighing long context and you are not sure whether caching will work for your use case, assume the answer is no until you have read the provider’s caching docs and tested the hit rate. Context engineering is real work and caching is a big part of it.
When Neither Is Enough
There is one more category worth naming because I see developers reach for the wrong tool in it all the time. If your use case involves an agent that needs to perform multi-step reasoning, call tools, and maintain state across turns, neither RAG nor long context is your main problem. AI agent memory and state persistence is a separate concern that neither approach solves on its own.
An agent that calls five tools, pulls back results, reasons about them, and calls more tools does not have a retrieval problem. It has a working-memory problem. You can combine either RAG or long context with agent memory, but swapping one retrieval pattern for another will not fix an agent that is losing track of what it decided three turns ago.
If you find yourself asking whether to use RAG or long context for an agent, the question before that one is probably how you are managing the agent’s state, and the retrieval choice will fall out of that answer rather than driving it.
What To Build This Week
If you already have RAG in production and it works, do not rip it out. The temptation is real because the long context numbers look sexy and the simplification story is appealing. But “it works” is worth a lot, and long context is not free to adopt correctly. The right time to move is when you are already making structural changes, not on a random Tuesday.
If you are starting a new feature that involves feeding documents to an LLM, default to long context first. Measure the cost with caching enabled. Measure the latency on your target hardware. If the numbers work, you have saved yourself a month of retrieval plumbing. If they do not, you know exactly what constraints are pushing you toward RAG and you can design the pipeline around those specific needs instead of building something generic.
If you are somewhere in between, the hybrid pattern is usually the right answer. Retrieve a wide net, let the model reason over it, and iterate from there. This is the pattern that handles the most use cases with the least architectural commitment, and it is the one I reach for first when someone asks me to scope a new AI feature.
The short version is that long context did not kill RAG. It changed what RAG is for. RAG used to be about fitting anything into a tiny window. Now it is about deciding which chunk of a very large corpus is worth paying attention to, and letting the model handle the rest. The decision is more nuanced than it was three years ago, which is mostly good news. You have more options, and the right option is less often “whatever we built in 2023 and never revisited.”
The one decision you should not make is to ignore the question. Running RAG pipelines you no longer need is a tax. Stuffing windows when you should be retrieving is a different tax. Go look at your current AI features, pick the one that has grown the most awkward, and ask which of these two patterns would handle it better today. The answer might be what you already have. Often it is not.