Cache-Augmented Generation (CAG): When You Can Skip Retrieval Entirely

Retrieval-Augmented Generation (RAG) has become the default architecture for grounding large language models in external knowledge. But RAG carries real overhead: it needs a retrieval system, adds latency at query time, and can fail when retrieval surfaces the wrong documents. Cache-Augmented Generation (CAG) offers an alternative for a specific but common situation — when your knowledge base is small enough to fit inside the model's context window. Instead of retrieving documents on every query, CAG loads the knowledge once, caches the model's processed state, and reuses it. This article explains what CAG is, how it works, and when it beats RAG.

The Idea in One Sentence

CAG preloads all relevant knowledge into the model's context ahead of time, precomputes and stores the model's internal key-value cache for that knowledge, and then answers queries directly against it — eliminating the retrieval step entirely.

Quick Refresher: How RAG Works

In a typical RAG pipeline, each user query triggers the following:

The query is embedded and used to search a vector database (or other index).

The most relevant documents are retrieved.

Those documents are inserted into the prompt alongside the query.

The model generates an answer grounded in the retrieved text.

This works well for large knowledge bases, but it introduces a retrieval component that must be built, tuned, and maintained — and every query pays the cost of a search plus the risk that retrieval returns irrelevant or incomplete context.

How CAG Works

CAG takes advantage of two developments: models with long context windows, and the transformer's key-value (KV) cache. Here's the flow:

Step 1: Preload the Knowledge

Rather than storing documents in an external index, CAG places the entire knowledge base — or the relevant subset — directly into the model's context window. This assumes the knowledge is small enough to fit (which modern long-context models make increasingly feasible).

Step 2: Precompute the KV Cache

When a transformer processes text, it computes internal key and value tensors for each token — the "KV cache" — that represent its understanding of that text. CAG runs the knowledge through the model once and saves this KV cache. This is the computationally expensive part, and it's done only once, ahead of time (offline).

Step 3: Answer Queries Against the Cache

When a user asks a question, CAG loads the precomputed KV cache (so the model doesn't have to reprocess the whole knowledge base) and simply appends the query. The model generates an answer as if it had just read all the knowledge — because, in effect, its cached state already reflects that it has. No retrieval, no vector search, no per-query document lookup.

Why This Is Faster

The key insight is that the heavy lifting — processing the knowledge base — happens once and is cached, not repeated on every query. In RAG, each query pays for retrieval plus processing the retrieved context. In CAG, the knowledge is already "baked into" the reusable cache, so each query only pays to process the short query itself and generate an answer. For repeated queries against a stable knowledge base, this can be substantially faster and simpler.

Advantages of CAG

No retrieval infrastructure — there's no vector database, no embedding pipeline, no retriever to build, tune, or maintain.

Lower per-query latency — skipping retrieval and reusing the cache removes steps from the critical path.

No retrieval errors — because all knowledge is present, there's no risk of the retriever missing a relevant document or fetching the wrong one.

Full context awareness — the model sees the entire knowledge base at once, so it can reason across all of it rather than only the top-k retrieved chunks.

Simplicity — the overall architecture is much simpler to reason about and operate.

Limitations of CAG

CAG is not a universal replacement for RAG. Its constraints are real:

Context window ceiling — the entire knowledge base must fit within the model's context window. For large corpora (millions of documents), this is impossible.

Cost of large contexts — even when knowledge fits, very long contexts can be expensive to process and may degrade model performance ("lost in the middle" effects, where models attend less well to information buried in long contexts).

Static knowledge assumption — the cache reflects a fixed snapshot. If the knowledge changes frequently, the cache must be recomputed, eroding the efficiency gains.

Memory footprint — storing KV caches consumes memory, and larger knowledge bases mean larger caches.

CAG vs. RAG: How to Choose

The decision comes down mainly to the size and volatility of your knowledge base.

Choose CAG when:

Your knowledge base is small and stable (e.g., a product manual, a policy handbook, a fixed FAQ, a single contract).

Low latency and simplicity matter.

You want to avoid building and maintaining retrieval infrastructure.

The full knowledge comfortably fits in context.

Choose RAG when:

Your knowledge base is large — too big to fit in any context window.

Knowledge changes frequently and must stay current.

You need to scale to many documents or dynamic sources.

You want to retrieve only the most relevant slices rather than loading everything.

Consider a hybrid when your needs fall in between: use RAG to narrow a large corpus down to a relevant, bounded subset, then load and cache that subset with CAG for fast, full-context answering within a session.

A Concrete Example

Imagine a customer-support bot for a single software product with a 40-page manual. With RAG, every user question triggers a vector search over chunks of the manual, and the bot answers from whatever chunks come back — occasionally missing relevant sections. With CAG, the entire manual is loaded once, its KV cache is precomputed, and every question is answered against the complete manual instantly, with no chance of retrieval missing a section. Here, CAG is the simpler, faster, more reliable choice. But if that same company had 40,000 product manuals, CAG couldn't fit them all, and RAG would be the right tool.

Conclusion

Cache-Augmented Generation reframes a question we've come to take for granted: do we actually need to retrieve? When a knowledge base is small and stable enough to fit in a model's context window, the answer is often no. By preloading knowledge, precomputing the KV cache, and reusing it across queries, CAG delivers grounded answers with lower latency, no retrieval errors, and dramatically simpler architecture. It doesn't replace RAG — large and fast-changing corpora still demand retrieval — but it's a powerful option for the many real-world cases where all the knowledge you need can simply be kept in context. As context windows keep growing, expect CAG to become an increasingly practical choice.