Source-linked AI summary
Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
Brian J Chan, Chao-Ting Chen, Jui-Hung Cheng, Hen-Hsen Huang
TL;DR
RAG adds latency, document-selection risks, and system complexity, while long-context models can hold relevant knowledge in a preloaded context. This paper proposes CAG, which precomputes a KV cache for use with queries, and reports that it can outperform RAG in efficiency and accuracy for manageable knowledge bases.
Problem
RAG integrates external knowledge but adds real-time retrieval latency, document-selection errors, and system complexity.
Source: §1 · p. 1Method
CAG preloads relevant documents into a long-context LLM and precomputes a KV cache for answering queries without runtime retrieval.
Source: §1 · p. 1Results
Experiments identify scenarios where long-context LLMs outperform traditional RAG in efficiency and accuracy, especially with manageable knowledge bases.
Source: §1 · p. 2Takeaways & Limitations
CAG may serve as a streamlined, effective alternative to RAG when the documents or knowledge base are limited and manageable.
Source: §1 · p. 2CAG requires loading all relevant documents into the model’s context and becomes impractical for significantly larger datasets.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Retrieval-augmented generation (RAG) has gained traction as a powerful approach for enhancing language models by integrating external knowledge sources. However, RAG introduces challenges such as retrieval latency, potential errors in document selection, and increased system complexity. With the advent of large language models (LLMs) featuring significantly extended context windows, this paper proposes an alternative paradigm, cache-augmented generation (CAG) that bypasses real-time retrieval. Our method involves preloading all relevant resources, especially when the documents or knowledge for retrieval are of a limited and manageable size, into the LLM's extended context and caching its runtime parameters. During inference, the model utilizes these preloaded parameters to answer queries without additional retrieval steps. Comparative analyses reveal that CAG eliminates retrieval latency and minimizes retrieval errors while maintaining context relevance. Performance evaluations across multiple benchmarks highlight scenarios where long-context LLMs either outperform or complement traditional RAG pipelines. These findings suggest that, for certain applications, particularly those with a constrained knowledge base, CAG provide a streamlined and efficient alternative to RAG, achieving comparable or superior results with reduced complexity.
1 Introduction
The introduction presents cache-augmented generation (CAG) as a retrieval-free alternative that preloads knowledge for long-context LLMs. It argues that CAG can outperform RAG in efficiency and accuracy for manageable knowledge bases.
1 Introduction
- RAG adds real-time retrieval latency, risks of document-selection errors, and system complexity with corresponding maintenance overhead.Source: §1 · p. 1
- CAG preloads relevant documents and precomputes the LLM’s KV cache, enabling runtime answers without additional retrieval.The introduction says this mitigates retrieval latency and errors while simplifying the system.Source: §1 · p. 1
- A 32K–64K context window is described as sufficient for sources such as internal documentation, FAQs, customer-support logs, and domain-specific databases.Source: §1 · p. 1
- Experiments identify scenarios where long-context LLMs outperform traditional RAG in both efficiency and accuracy, especially with manageable knowledge bases.The introduction frames CAG as a streamlined alternative for these applications.Source: §1 · p. 2
2 Methodology
The methodology preloads a curated knowledge collection into an LLM’s extended context and computes a reusable KV cache. At inference, the model uses that cache with the query; appended query tokens can later be truncated to reset the cache.
2 Methodology
- CAG divides retrieval-free knowledge integration into external-knowledge preloading, inference, and cache reset.Source: §2 · p. 2
- The curated documents are formatted to fit the model’s extended context and processed once to produce a KV cache that can be stored for later use.The cost of processing the collection is incurred only once, regardless of subsequent query count.Source: §2 · p. 2
- During inference, the precomputed KV cache is loaded with the user’s query, and the LLM uses the cached context to generate a response.Source: §2 · p. 2
- Because the knowledge is preloaded, inference avoids retrieval latency and reduces risks of errors or omissions from dynamic retrieval.The combined document-and-query prompt is described as supporting a unified understanding.Source: §2 · p. 2
- To reset the cache between sessions, the system truncates newly appended query tokens rather than reloading the entire cache from disk.The paper says this permits rapid reinitialization.Source: §2 · p. 2
3 Experiments
Experiments compare CAG with sparse and dense RAG on SQuAD and HotPotQA at varying reference-text lengths. CAG generally leads on BERTScore and avoids retrieval, while its performance edge narrows as data grows.
3.1 Experimental Setup
- Experiments use SQuAD and HotPotQA test sets with varying reference-text lengths; longer references make retrieval more challenging, particularly for RAG.The datasets provide single-passage and multi-hop QA challenges, respectively.Source: §3.1 · p. 3, Table 1 · p. 3
3.2 Baseline Systems
- The sparse baseline uses BM25 to retrieve top-k passages, while the dense baseline selects semantically aligned passages using embeddings.RAG baselines tested top-1, top-3, top-5, and top-10 passages.Source: §3.2 · pp. 3–4
3.3 Results
- CAG achieved the highest BERTScore in most cases, while preloading the full test reference avoids errors from incomplete or irrelevant RAG passages.The comparison is reported in Table 2; BERTScore is the measure used for answer similarity.Source: §3.1 · p. 3, Table 2 · p. 4, §3.3 · p. 4
- CAG eliminates retrieval time and remains more efficient than dense RAG, although generation time grows with knowledge size.Preloading the KV cache avoids processing the reference text on the fly.Source: Table 3 · p. 4, §3.3 · pp. 4–5, Figure 2 · p. 5
- As data size rises, CAG’s performance gap over RAG narrows; sparse RAG beating dense RAG suggests these datasets may not require deeper semantic retrieval.The authors also relate the narrowing gap to possible long-context degradation [6].Source: §3.3 · p. 4
4 Conclusion
The paper argues for reconsidering traditional RAG workflows and identifies hybrid preloading plus selective retrieval as a possible approach. Such a hybrid could balance efficiency with flexibility when both context completeness and adaptability matter.
4 Conclusion
- A hybrid could preload a foundation context and retrieve only for edge cases or highly specific queries.The authors present this as a potential approach to balance preloading efficiency with retrieval flexibility.Source: §4 · p. 5
- The proposed hybrid is intended for situations where context completeness and adaptability are both important.Source: §4 · p. 5
- The authors call for rethinking traditional RAG workflows as long-context LLMs evolve.Source: §4 · p. 5
Limitations
CAG requires loading all relevant documents into the model context, so the authors frame it for manageable knowledge sources rather than significantly larger datasets.
Limitations
- CAG requires loading all relevant documents into the model’s context.Source: Limitations · p. 5
- The approach is suited to manageable sources such as small-company internal knowledge bases, FAQs, and call centers.Source: Limitations · p. 5
- The authors state that the approach becomes impractical for significantly larger datasets.Source: Limitations · p. 5
http://arxiv.org/ps/2412.15605v2
The passage identifies an available PNG figure asset.
http://arxiv.org/ps/2412.15605v2
- The figure asset is named sample-franklin.png and is available in PNG format.Source: http://arxiv.org/ps/2412.15605v2 · p. 7