PaperlayerView exact revision on arXiv

Source-linked AI summary

CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG

Gyuwan Kim, Cheoneum Park, Tao Yang

arXiv:2608.07458v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Chunk-level KV-cache reuse can leave long-context RAG with coarse, redundant, and noisy representations under tight latency budgets. CoinRAG composes offline-computed, query-relevant nugget KV caches with chunk context, consistently outperforming the baselines with a 5.3% average F1 improvement under a 100 ms P99 budget.

  • Problem

    Chunk-level KV-cache granularity can introduce noise and redundancy, limiting accurate RAG responses under low-latency budgets.

  • Method

    CoinRAG extracts query-relevant nuggets through two-stage retrieval and composes their precomputed KV slices with contextual chunk representations for online inference.

  • Results

    CoinRAG consistently outperforms Standard RAG and Standard CAG, averaging a 5.3% F1 improvement (41.7 vs. 39.6) across three LongBench datasets under a 100 ms P99 latency budget.

  • Takeaways & Limitations

    CoinRAG establishes a new Pareto frontier for effective long-context RAG inference under an interactive low-latency SLA budget.

  • Takeaways & Limitations

    CoinRAG incurs corpus-scaled offline storage and computation costs and assumes documents change infrequently enough for one-time or infrequent preprocessing.

Abstract

from arXiv · show

Recent optimization studies on Retrieval-Augmented Generation (RAG) have exploited chunk-level KV cache reuse to avoid processing long retrieved contexts for higher efficiency, while significant information redundancy and noise still remain in the coarse-grained chunks. This paper optimizes the Pareto frontier under low prefill latency constraints while maximizing accuracy by proposing CoinRAG (Contextualized Information Nugget KV Cache Reuse for Long-Context RAG). The name metaphorically reflects our core mechanism: much like assembling small tokens (or "coins") to accumulate a larger value, CoinRAG compositionally reuses offline-computed, fine-grained nugget caches to form a learned contextual representation efficiently in a more semantically relevant but compact manner. Specifically, instead of full-chunk encoding, CoinRAG identifies query-relevant semantic units within retrieved chunks through two-stage retrieval and seamlessly assembles their sliced KV representations with a chunk-level context. Extensive evaluations on LongBench multi-hop question answering tasks demonstrate that CoinRAG significantly reduces operational costs and outperforms the other baselines with a new Pareto frontier and an average 5.3% relative improvement in answer quality (F1) under a standard fast prefill latency budget.

1 Introduction

CoinRAG targets RAG’s high prefill latency and computational redundancy by selecting fine-grained, query-relevant information nuggets and composing contextual KV caches. On LongBench multi-hop QA, it improves answer quality under a P99 latency budget and remains competitive without a latency limit.

  • RAG improves factual consistency and reduces hallucination without model retraining, but coarse document or chunk processing causes expensive inference.
  • CoinRAG maximizes RAG accuracy under a low-latency budget by using slim, fine-grained information extracted from text chunks to reduce noisy-context evidence loss.Its framework performs offline text-span nugget extraction, nugget-aware fine-tuning, and two-stage online retrieval.
  • CoinRAG consistently outperforms Standard RAG and Standard CAG, delivering 5.3% (41.7 vs. 39.6) answer quality improvement averaged over 3 datasets under the P99 latency 100 ms budget.
  • 5.2% 3-dataset average F1 score improvement remains when the latency limit is removed, with CoinRAG outperforming 2 out of 3 datasets.

2 Proposed Method: CoinRAG

CoinRAG improves long-context RAG efficiency and accuracy by replacing coarse chunk-level reuse with query-driven, fine-grained nugget KV caches. It extracts grounded spans offline, retrieves relevant nuggets online, and composes their contextualized caches into a compact prefix representation.

  • Motivation: CoinRAG targets the noise and redundancy of coarse chunk-level KV caches by representing each chunk as essential information nuggets.Its design seeks higher information density under a low-latency budget.
  • Offline Cache Preparation: CoinRAG encodes every chunk once offline, then slices precomputed KV representations for query-relevant textual units instead of re-encoding retrieved text online.This shifts context encoding offline and bypasses full-text re-encoding during inference.
  • Contextualized Nugget Caches: CoinRAG slices each nugget from its source chunk cache, preserving document-level context rather than encoding an isolated text snippet.Boundary indices [s_i, e_i] identify the cache slices C_bi[s_i : e_i].
  • Query-Driven Retrieval: Two-stage retrieval first selects top-k_c chunks and then ranks candidate nuggets within them by embedding similarity to the query.The resulting nuggets are represented by source-chunk identifiers and contiguous boundary token indices.
  • Cache Composition: Position rotation and order-preserving contiguous alignment compose heterogeneous nugget caches with the system prompt while masking unretrieved text to reduce context length and GPU memory.The method addresses RoPE mismatches between nuggets from different chunks and token positions.
  • Nugget-Aware Fine-Tuning: Nugget-aware fine-tuning aligns training with test-time non-contiguous cache synthesis by constructing prefixes from retrieved nuggets and minimizing answer-token cross-entropy.This mitigates the structural gap caused by training on continuous sequences and inference on composed cached segments.

3 Comparison to Prior RAG Paradigms

CoinRAG is positioned against standard RAG, chunk-level cache-augmented generation, cross-chunk cache-recomputation methods, and nugget-based RAG. It combines offline, fine-grained nugget caching with contextual reconstruction while differing from online, context-free nugget construction.

  • Standard RAG: Standard RAG processes full retrieved chunks and the query online, causing severe prefill latency over long sequences.Its cache is computed as KVStandardRAG = KVM(p ⊕tchunk ⊕q).
  • Cache-Augmented Generation: TurboRAG represents chunk-level CAG methods that precompute and reuse coarse full-chunk KV caches to reduce time to first token.Standard CAG bypasses online encoding but loads full-chunk representations into GPU memory at inference.
  • CoinRAG: CoinRAG slices independently offline-encoded chunks into nuggets, compiling compact contextual caches while preserving document-level semantic grounding and reducing latency.This optimized cache-slicing pipeline builds on Standard CAG’s cached full-chunk representations.
  • Cross-Chunk Cache Methods: CacheBlend restores cross-chunk attention by selectively recomputing a small subset of cached tokens, while KVLink uses trainable link tokens between chunks.Both methods partially recompute or augment cached KV states online rather than relying purely on chunk-level caching.
  • Nugget-Based RAG: CoinRAG forms query-independent nuggets offline and selects them through online two-stage retrieval, whereas GINGER and Crucible construct query-specific nuggets online with an LLM and omit surrounding context.CoinRAG nuggets remain text spans from the original document and recover their source chunk’s broader context during inference.

4 Experiments

CoinRAG is evaluated on three LongBench multi-document QA benchmarks against Standard RAG, TurboRAG, CacheBlend, and KVLink using F1 accuracy and TTFT efficiency. Under latency and context-length constraints, CoinRAG establishes stronger accuracy–efficiency trade-offs, while ablations validate its contextual encoding, two-stage retrieval, position alignment, and nugget-aware fine-tuning.

  • Evaluation setup: CoinRAG is compared with Standard RAG, TurboRAG, CacheBlend, and KVLink using token-level F1 and P99 TTFT latency.TTFT measures prefix-context construction and query processing during prefill, which dominates inference time for short answers.
  • Evaluation setup: Experiments cover HotpotQA, 2WikiMQA, and MuSiQue, which require cross-document aggregation and reasoning.Each dataset provides a task-specific document corpus for retrieval.
  • Latency-constrained results: 5.3% higher F1 and 1.84× shorter contextual token length make CoinRAG superior to TurboRAG averaged across three datasets under a P99 100 ms budget.CoinRAG outperforms Standard RAG, CacheBlend, TurboRAG, and KVLink on all three datasets under this constraint.
  • Latency-constrained results: 5.2% average F1 improvement (42.7 vs. 40.6) and 6.8× shorter average length distinguish CoinRAG from TurboRAG without a latency limit.CoinRAG outperforms TurboRAG on two of three datasets, although slim KV representations can miss some cross-chunk interactions.
  • Context-length results: 10.1× shorter context than Standard RAG is achieved by CoinRAG even without a length limit, while it attains higher F1 at the same context-length limit.The compact representation is attributed to nugget-level KV cache reuse.
  • Ablation studies: Nugget-aware fine-tuning improves peak F1 by +11.3, +6.3, and +6.4 points on HotpotQA, 2WikiMQA, and MuSiQue, respectively.The improvement addresses position-embedding shifts from stitching non-contiguous nugget spans; contextual encoding, two-stage retrieval, and position alignment also improve performance, especially under tight budgets.

5 Conclusion

CoinRAG is a lightweight long-context RAG framework that delivers interactive responses through contextual KV nugget cache reuse. Its offline nugget extraction, nugget-aware finetuning, two-stage retrieval, and contextualized cache composition improve F1 under latency constraints and without them.

  • Technical contributions: CoinRAG uses offline text-span information nuggets, nugget-aware finetuning, two-stage query-specific retrieval, and contextualized KV cache composition.These components form its slim representation for efficient long-context RAG responses.
  • Empirical results: 5.3% average F1 gain: under a standard 100 ms P99 latency budget, CoinRAG beats TurboRAG by 4.7%, 0.5%, and 14.6% across 3 datasets.TurboRAG is identified as the best competitor in this comparison.
  • Empirical results: 5.2% F-1 improvement or more: without a latency limit, CoinRAG still outperforms the baselines on average across 3 datasets.The result indicates that removing noise and unnecessary context outweighs losses from missing some interaction.

Limitations

CoinRAG’s efficiency gains entail offline resource costs, model-specific cache coupling, retrieval-dependent answer quality, and chunk-local attention. Its evaluation also excludes inter-query KV-cache reuse under a limited online-cache assumption.

  • Systemic constraints: Offline pre-encoding accelerates online prefill but incurs corpus-scaled storage, memory-mapped I/O or hierarchical caching needs, and one-time nugget-aware fine-tuning costs.These costs increase with corpus size and require supporting infrastructure.
  • Systemic constraints: The precomputed cache is tied to a specific model checkpoint, so backbone updates or architectural changes require documents to be re-encoded.Stored KV representations depend on transformer weights and positional-embedding topologies.
  • Systemic constraints: Answer quality remains bounded by chunk- and nugget-level retrieval recall because missed canonical evidence spans cannot be synthesized downstream.Task-aware retrievers or cross-encoder rerankers are proposed to mitigate this bottleneck.
  • Systemic constraints: Nuggets from different chunks do not attend to each other during encoding, limiting cross-chunk interactions in CoinRAG’s chunk-granular composition.This limitation is shared with TurboRAG-style chunk-level caching; CacheBlend addresses cross-chunk attention and is identified for future investigation in CoinRAG.
  • Evaluation assumptions: The evaluation assumes limited online-cache capacity and therefore excludes inter-query KV-cache reuse when consecutive queries share retrieved documents.Assessing this reuse requires a traffic dataset containing query appearance order, while standard RAG could reduce TTFT with large cache capacity.

A Future Work · B Answer Generation Prompt

CoinRAG is evaluated on multi-hop, multi-document QA because it stresses cross-chunk retrieval and cache reuse, while remaining limited to single-shot question answering. Its answer-generation prompt combines a static pre-encoded instruction prefix with dynamically mapped nugget caches and an appended question.

  • A Future Work: CoinRAG evaluates HotpotQA, 2WikiMQA, and MuSiQue because their queries require synthesizing evidence scattered across multiple retrieved chunks.These benchmarks directly stress the cross-chunk retrieval and cache reuse behavior targeted by CoinRAG.
  • A Future Work: The evaluation focuses on single-shot multi-document question answering rather than genuine multi-turn interaction.
  • A Future Work: Prompt injection remains a broader RAG security concern because retrieved documents may contain malicious content.CoinRAG’s offline-computed nugget representations allow anomaly detectors and sanitizers to be applied before online inference.
  • A Future Work: Adversarial retrieved spans can still expose answer generation to cached malicious content despite offline-stage mitigations.
  • B Answer Generation Prompt: Figure 5 specifies a prompt template that instructs the base language model to generate a short answer from retrieved nuggets.The instruction block enforces a three-tier execution priority, beginning with verbatim extraction when the correct answer appears explicitly in the clues.
  • B Answer Generation Prompt: The core instruction is pre-encoded once as a static system prefix, while dynamic nugget sequences use precomputed full-context KV cache slots.
  • B Answer Generation Prompt: The target question is appended at the very end of the prompt sequence after dynamic nugget clues are mapped to cache slots.

C Offline Nugget Extraction: Prompt, Example, and Characteristics

CoinRAG uses GPT-4o-mini and a structured prompt to extract passage-specific nuggets offline, with candidate counts determined by each passage’s LLM output. An example illustrates the extraction workflow, while empirical statistics characterize nugget compression across training and evaluation splits.

  • Offline nugget extraction: GPT-4o-mini generates an initial list of passage-specific nuggets using CoinRAG’s structured offline-extraction prompt.Figure 6 specifies the system prompt template and constraints used throughout offline processing.
  • Offline nugget extraction: Approximately 7.1 candidate nuggets are proposed per passage on average, with K determined by the LLM output rather than fixed in advance.This corresponds to Algorithm 1, Line 1.
  • Extraction example: Figure 7 demonstrates step-by-step extraction from a raw LongBench HotpotQA passage into structured factual nuggets.The example uses passage_idx=7, “Grania: She-King of the Irish Seas”.
  • Nugget characteristics: Empirical statistics for IRCoT training splits and LongBench evaluation splits characterize query-, passage-, and nugget-level extraction behavior.Table 4 reports these operational characteristics and density statistics for each dataset.
  • Nugget characteristics: The offline pipeline consistently compresses diffuse document contexts into compact, atomic nuggets across both training and evaluation splits.The passage states that mean nugget length is far shorter than mean passage length, but the supplied excerpt does not provide the numerical values.

D Steps of Online Inference: Example

CoinRAG’s online inference pipeline uses four stages to answer a LongBench HotpotQA multi-hop question. It retrieves chunks, selects fine-grained nuggets, assembles cached KV segments, and generates an answer conditioned on the optimized context cache.

  • Online processing pipeline: Fine-grained information nuggets are then retrieved exclusively within the pre-selected chunk pool to remove uninformative tokens while maintaining factual relevance.
  • Online processing pipeline: A dense embedding model first retrieves the top-kc candidate text chunks from the external corpus.
  • Online processing pipeline: The corresponding nugget KV representations are fetched from the precomputed document cache and stitched together with position alignment into the final prompt prefix.
  • Online processing pipeline: Finally, the language model processes the online query conditioned on the optimized context cache to synthesize the correct answer.

E Evaluation Setup Details

CoinRAG’s evaluation setup combines two-stage nugget retrieval, selective KV-cache loading, and one-time nugget-aware fine-tuning. Experiments sweep nugget counts and CacheBlend recomputation ratios under standardized corpus-level preprocessing and concise QA prompting.

  • Chunk and Nugget Retrieval: CoinRAG first retrieves candidate chunks using the identical k_c range, then sweeps the selected nugget count k across 12 values from 1 to 200.The tested values are k ∈{1, 2, 3, 4, 5, 10, 20, 30, 50, 100, 150, 200}.
  • Chunk and Nugget Retrieval: Offline extraction uses GPT-4o-mini with structured prompting and accepts fuzzy-matched candidates only when similarity to the source span exceeds τ = 0.7.Extracted nuggets are atomic, fine-grained spans rather than full sentences.
  • KV Caching and Loading: Every corpus chunk is pre-encoded once into full-context KV representations, while CacheBlend sweeps recomputation ratio r from 0.0 to 1.0 in 0.1 intervals.CoinRAG stores only tokens belonging to at least one extracted nugget, reducing disk footprint relative to full-chunk or document caching.
  • Nugget-Aware Fine-tuning: Nugget-aware fine-tuning runs for one epoch on 277,280 instances from HotpotQA, 2WikiMQA, and MuSiQue with effective batch size 16.Extraction, cache construction, and fine-tuning are one-time costs per corpus and backbone checkpoint, rather than per query.
  • Generation and Prompting: The QA prompt requests concise answers grounded in retrieved evidence, and retrieved items are concatenated in ranked top-1-to-top-k order for the prefix context cache.The base model is Qwen2-7B-Instruct, selected for native flexible position rotation via RoPE.

F Hardware Infrastructure and I/O Management · G Fine-Tuning Hyperparameters

CoinRAG’s efficiency evaluation uses a specified single-node hardware and storage setup, with offline extraction and cache construction treated as one-time corpus-preparation costs. Nugget-aware fine-tuning uses a fixed, memory-efficient configuration designed to avoid out-of-memory errors under strict sequence-length constraints.

  • F Hardware Infrastructure and I/O Management: Experiments run on one node with 64 physical CPU cores, 128 threads, 188 GB memory, an RTX PRO 6000 with 96 GB VRAM, and an L40S with 48 GB VRAM.Training uses the RTX PRO 6000, while evaluation and latency benchmarks use the L40S.
  • F Hardware Infrastructure and I/O Management: Evaluation and latency benchmarks run on the L40S in bfloat16, while nugget extraction uses the GPT-4o-mini API offline with concurrency limited to 100.Precomputed KV caches reside on a 3.84 TB Samsung PM9A3 local NVMe SSD and load dynamically into GPU memory.
  • F Hardware Infrastructure and I/O Management: $265–325 is the one-time GPT-4o-mini Batch API cost to extract 6.08M nuggets from 950,221 training passages at 99.9% validity.Extraction for the three evaluation corpora costs under $2 and yields 277K nuggets from 5,681 passages.
  • F Hardware Infrastructure and I/O Management: Approximately 30% of disk footprint could be reduced by retaining only nugget-covered positions, which span approximately 70% of chunk tokens after deduplication.Both cache construction and this storage optimization are one-time corpus-preparation costs rather than per-query costs.
  • F Hardware Infrastructure and I/O Management: 140 GPU-hours are required for one epoch of nugget-aware fine-tuning over 277,280 examples and 17,330 optimizer steps with gradient accumulation of 16.Training runs on one RTX PRO 6000 and reaches a peak memory footprint of 75.8 GB; this cost is incurred once per backbone checkpoint.
  • G Fine-Tuning Hyperparameters: The fine-tuning pipeline uses a fixed hyperparameter configuration with gradient checkpointing and memory-efficient 8-bit Paged AdamW to prevent out-of-memory errors under strict sequence-length constraints.The complete optimization configuration is specified in Table 5.

H Successful and Failed Inference Examples

CoinRAG’s cache-stitching pipeline supports robust semantic inference and knowledge recovery in compact representations, but aggressive nugget slicing can amplify entity biases or omit linking paths. Despite these failures, CoinRAG maintains strong average F1 and outperforms baselines under a targeted latency budget.

  • Successful cases: CoinRAG’s cache-stitching pipeline enables robust semantic inference and knowledge recovery within a compressed footprint.The paper categorizes positive HotpotQA examples into three success types.
  • Successful cases: Pleiospilos was correctly identified as the flowering plant over Pueraria.The prediction matches the gold answer in the example.
  • Failure modes: Aggressive nugget slicing can amplify surface entity biases or omit contiguous linking paths during retrieval and alignment.These failure trajectories provide insights for future optimizations.
  • Overall evaluation: CoinRAG still delivers strong average F1 on three test datasets and outperforms baselines under a targeted latency budget.The evaluation result is stated despite the illustrated failed cases.
  • Failure modes: Gary L. Bennett’s mission question was answered incorrectly as Jupiter instead of the gold answer, Sun, despite retrieved evidence including Ulysses’ primary Sun mission.This illustrates the “Misled despite full evidence” failure type.
Loading 2608.07458v1…