CoinRAG: Contextualized Information Nugget KV Cache Reuse for Long-Context RAG

1 Introduction

• CoinRAG adopts a P99 Time-to-First-Token budget under 100 ms, reflecting interactive service targets; lowering latency also enables higher throughput.Source: §1 · p. 1
• Standard RAG remains costly in TTFT even with KV reuse, so limiting retrieval to meet a low-latency SLA can hurt accuracy; CoinRAG uses slim nuggets to reduce noisy-context evidence loss.The approach extracts nuggets from chunks in advance rather than retrieving and encoding entire chunks at inference time.Source: §1 · pp. 1–2
• CoinRAG extracts text-span nuggets offline, fine-tunes nugget awareness, retrieves relevant nuggets in two stages, and composes contextualized KV caches for nuggets from the same chunks.Source: §1 · p. 2
• Under a P99 latency budget of 100 ms, CoinRAG improves answer quality by 5.3% (41.7 vs. 39.6) averaged over three LongBench datasets, outperforming Standard RAG and Standard CAG.The evaluation uses multi-hop question-answering benchmarks.Source: §1 · p. 2
• Without a latency limit, CoinRAG achieves a 5.2% improvement in three-dataset average F1 and outperforms on two of three datasets.Source: §1 · p. 2

2 Proposed Method: CoinRAG

• The method targets higher RAG accuracy under low-latency constraints by replacing coarse chunk representations with essential content and query-driven nugget selection.The design aims to reduce noise and redundancy while avoiding attention dilution.Source: §2 · p. 2

2.1 Problem Formulation

• CoinRAG encodes every chunk once offline, then composes query-relevant cached spans into a prefix before encoding the online query.The resulting context cache concatenates the composed prefix with newly computed query representations.Source: §2.1 · p. 2

2.2 Offline Nugget Extraction

• Offline nugget extraction uses an LLM to propose spans, matches them exactly or fuzzily to source text, and drops candidates below a similarity threshold.Accepted nuggets are recorded as start and end token positions, enabling later cache slicing.Source: §2.2 · pp. 2–3

2.3 Two-Stage Online Nugget Retrieval

• CoinRAG retrieves top-k chunks densely, selects top-k embedded nuggets within them, and slices their precomputed KV caches to preserve full-chunk contextual grounding.The selected spans retain the KV states they had when encoded in their source chunks, unlike isolated nugget re-encoding.Source: §2.3 · p. 3

2.4 Contextualized KV Cache Composition with Position Alignment

• CoinRAG aligns nugget caches by rotating their positional indices, preserving document order while packing spans contiguously and masking unretrieved text.Continuous positions reduce context length, GPU memory footprints, and decoding time without changing the model architecture.Source: §2.4 · pp. 3–4

2.5 Nugget-Aware Fine-tuning

• Nugget-aware fine-tuning aligns training with inference by composing retrieved-nugget prefix caches for query-answer examples and minimizing answer-token cross-entropy.Training examples include ground-truth and distractor documents.Source: §2.5 · p. 4

3 Comparison to Prior RAG Paradigms

• Standard RAG processes the full retrieved text and query online for every request, causing severe prefill latency over long sequences.Source: §3 · p. 4

3.1 Cache-Augmented Generation

• Standard CAG caches full retrieved chunks offline; TurboRAG (Lu et al., 2025) and Block-attention (Ma et al., 2025) reuse chunk-level KV caches to reduce TTFT.Preloading the entire corpus, as in Chan et al. (2025), can cause severe memory bloat.Source: §3.1 · pp. 4–5
• CoinRAG encodes nuggets offline and uses cache slicing to assemble compact context while preserving Standard CAG’s document-level semantic grounding.It operates at finer granularity than full-chunk caching, reducing latency.Source: §3.1 · p. 5
• CacheBlend (Yao et al., 2025) selectively recomputes tokens to restore cross-attention, while KVLink (Yang et al., 2025) uses link tokens to restore cross-chunk self-attention.Source: §3.1 · p. 5

3.2 Nugget-based RAG

• Nuggets began as assessor-enumerated essential information for IR/QA (Voorhees, 2004), with AutoNuggetizer (Pradeep et al., 2025) automating their construction for generation systems.GINGER (Łajewska and Balog, 2025) and Crucible (Dietz et al., 2026) extend nugget use to generation.Source: §3.2 · p. 5
• CoinRAG extracts nuggets offline and query-independently, then makes retrieval query-specific in two stages; GINGER and Crucible instead build standalone nuggets online with an LLM.Because they do not precompute KV caches and incur repeated LLM-call prefill overhead, GINGER and Crucible are excluded from the empirical comparison.Source: §3.2 · p. 5

4.1 Experimental Setup

• Experiments compare CoinRAG with Standard RAG, TurboRAG, CacheBlend, and KVLink on HotpotQA, 2WikiMQA, and MuSiQue using token-level F1 and P99 TTFT.They also report prefill length as a proxy for KV-cache memory use.Source: §4.1 · pp. 5–6

4.2 Main Results

• At a 100 ms P99 latency budget, CoinRAG outperforms all four baselines on all three datasets, averaging 5.3% higher F1 than TurboRAG with a 1.84× shorter contextual token length.Standard RAG is limited to retrieving one document under this budget; TurboRAG is the strongest other baseline.Source: Figure 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7
• Under equal prefill-length limits, CoinRAG achieves higher F1 than other baselines, and its context remains up to 10.1× shorter than Standard RAG with no length limit.The smaller active KV cache reduces runtime GPU memory use and can allow more concurrent requests, increasing serving throughput.Source: Figure 3 · p. 7, §4.2 · p. 7
• With no latency limit, CoinRAG beats TurboRAG on two of three datasets, with a 5.2% average F1 improvement (42.7 vs. 40.6) and a 6.8× shorter average length.At 116 ms, CoinRAG still leads all baselines; at higher budgets, some baselines catch up or surpass it on individual datasets, while CoinRAG retains an average advantage exceeding 5%.Source: §4.2 · p. 7

4.3 Ablation Studies

• Contextualized nugget encoding raises peak F1 over isolated encoding by 6.3, 4.9, and 3.9 points on the three datasets.The cached KV representations carry contextual information beyond the isolated nugget span.Source: §4.3 · pp. 7–8
• Two-stage chunk-to-nugget retrieval yields 9.6–17.5% higher peak F1 than one-stage retrieval across all three datasets, with fewer selected nuggets at peak.The fewer selected nuggets shorten active context and lower TTFT; restricting candidates to nuggets within retrieved chunks improves retrieval quality.Source: §4.3 · p. 8
• Position alignment improves F1 by 3.0–8.5% across datasets under a tight 75 ms P99 budget, but is comparable to no alignment from 100 ms onward.The ablation leaves each nugget at its original position within its source chunk.Source: §4.3 · p. 8
• Nugget-aware fine-tuning improves peak F1 by 11.3 points on HotpotQA, 6.3 on 2WikiMQA, and 6.4 on MuSiQue.It calibrates the model against position-embedding shifts from stitching non-contiguous nugget spans, bridging the training-inference alignment gap.Source: §4.3 · p. 8

5 Conclusion

• CoinRAG combines offline text-span nugget extraction, nugget-aware finetuning, two-stage query-specific retrieval, and contextualized KV cache composition.The framework uses a slim representation for effective and interactive long-context RAG responses.Source: §5 · p. 8
• At a standard 100 ms P99 latency budget, CoinRAG beats best competitor TurboRAG in F1 by 4.7%, 0.5%, and 14.6% across three datasets, averaging 5.3%.The conclusion also reports a new Pareto frontier for effective long-context RAG inference under an interactive TTFT SLA budget.Source: §5 · p. 8
• Without a latency limit, CoinRAG outperforms baselines by an average 5.2% F-1 improvement or more across three datasets, as noise removal outweighs missed interactions.Position alignment is most beneficial under tight P99 latency budgets, such as below 75 ms, and remains comparable to no alignment beyond that budget.Source: §5 · p. 8

Limitations

• Static KV pre-encoding reduces online prefill time but adds corpus-scaled storage and offline computation costs, including one-time nugget-aware fine-tuning.The authors assume documents change infrequently, making one-time or infrequent offline computation acceptable; storage may require memory-mapped I/O or hierarchical caching.Source: Limitations · p. 9
• Cached KV representations are tied to the model checkpoint, so updating the backbone or changing its architecture requires re-encoding documents.The coupling arises from dependence on transformer weights and positional embedding topologies.Source: Limitations · p. 9
• Answer quality is bounded by chunk- and nugget-level retrieval recall; missed canonical evidence spans prevent the downstream model from synthesizing the correct answer.Task-aware fine-tuned retrievers or cross-encoder rerankers are proposed as potential ways to mitigate this retrieval bottleneck.Source: Limitations · p. 9
• Nuggets from different chunks do not attend to one another during encoding, and integrating cross-chunk attention techniques into CoinRAG remains future work.The authors compare CacheBlend, which addresses cross-chunk attention in KV-cache reuse, and note that the limitation is shared with TurboRAG-style chunk-level caching.Source: Limitations · p. 9
• The evaluation assumes limited online-cache capacity and excludes inter-query KV reuse; evaluating that reuse requires traffic data with query appearance order.The authors describe this assumption as consistent with prior TurboRAG, KVLink, and CacheBlend studies and fair for comparison; with larger cache capacity, standard RAG could reduce TTFT through document overlap, while its no-latency-limit F1 in Table 3 remains unchanged.Source: Limitations · p. 9

A Future Work

• CoinRAG is evaluated on HotpotQA, 2WikiMQA, and MuSiQue, which require synthesizing evidence across retrieved chunks and test cross-chunk retrieval and cache reuse.The evaluation focuses on single-shot multi-document question answering.Source: A Future Work · p. 12
• Extending evaluation to conversational or repeated-query benchmarks could test offline cache reuse’s amortized benefit when turns retrieve overlapping document contexts.The paper identifies this extension as a way to broaden applicability beyond its single-shot setting.Source: A Future Work · p. 12
• Offline nugget caching permits standard prompt-injection mitigations, such as anomaly detectors and sanitizers, to run without affecting online latency.Prompt injection is identified as a broader RAG security concern (Greshake et al., 2023).Source: A Future Work · p. 12
• CoinRAG may still attend to cached malicious content when an adversarial span is retrieved, so the paper leaves systematic study of nugget-cache susceptibility to future work.The passage notes that offline mitigations do not eliminate this residual risk.Source: A Future Work · p. 12

B Answer Generation Prompt

• The answer-generation prompt requires verbatim extraction of explicit answers, then multi-hop reasoning grounded in retrieved context, and uses parametric knowledge only if both fail.The instruction block is injected into the base language model to generate a short answer from retrieved nuggets.Source: B Answer Generation Prompt · p. 12
• CoinRAG pre-encodes the static system prefix once, maps dynamic nugget sequences to precomputed full-context KV cache slots, and appends the question last.This describes the prompt’s separation of static instructions from dynamic clues.Source: B Answer Generation Prompt · p. 12

• GPT-4o-mini proposes an average of approximately 7.1 candidate nuggets per passage, with the count determined by the model output rather than a fixed setting.The structured prompt template and constraints are specified in Figure 6.Source: C Offline Nugget Extraction: Prompt,… · p. 12
• Figure 7 illustrates extraction on a HotpotQA text segment, transforming a raw paragraph about “Grania: She-King of the Irish Seas” into structured factual nuggets.The example uses passage_idx=7.Source: C Offline Nugget Extraction: Prompt,… · p. 12
• Across three datasets, mean nugget length is far shorter than mean passage length, while adjacent nugget overlap remains low in IRCoT training and LongBench evaluation splits.The statistics cover query-, passage-, and nugget-level characteristics.Source: C Offline Nugget Extraction: Prompt,… · p. 12

D Steps of Online Inference: Example

• Figure 8 illustrates the end-to-end execution flow with a multi-hop question from LongBench HotpotQA.Source: D Steps of Online Inference: Example · p. 12
• Dense retrieval selects top-kc candidate chunks, and nugget retrieval then runs exclusively within that pool to filter uninformative tokens while retaining factual relevance.Source: D Steps of Online Inference: Example · p. 13
• Selected nuggets’ precomputed KV representations are fetched from the document cache and stitched with position alignment, avoiding online text re-encoding.The cached segments form a seamless decoding prefix.Source: D Steps of Online Inference: Example · pp. 13–14
• The language model processes the online query conditioned on the optimized context cache to synthesize the answer.Source: D Steps of Online Inference: Example · p. 14

Chunk and Nugget Retrieval

• GPT-4o-mini extracts fuzzy-matched nuggets (τ=0.7), and BGE-M3 retrieves 512-token chunks and scores nugget relevance; sweeps vary chunk kc and selected-nugget k.Extracted nuggets are sub-sentence clauses or phrases; CoinRAG sweeps kc over {1, 2, 3, 4, 5, 10, 20, 30, 50} and k over {1, 2, 3, 4, 5, 10, 20, 30, 50, 100, 150, 200}.Source: E Evaluation Setup Details · p. 14, Figure 7 · p. 15

KV Caching and Loading

• Each corpus chunk is pre-encoded once as a full-context KV representation for selective inference loading; CacheBlend sweeps recomputation ratio r from 0 to 1 in 0.1 increments.CoinRAG stores only tokens belonging to at least one extracted nugget, reducing disk footprint relative to methods that cache full chunks or documents.Source: E Evaluation Setup Details · p. 14

Nugget-Aware Fine-tuning

• Nugget-aware fine-tuning uses one epoch over 277,280 training instances from HotpotQA, 2WikiMQA, and MuSiQue, with effective batch size 16.Nugget extraction, cache construction, and fine-tuning are one-time costs per corpus and backbone checkpoint, not per query.Source: E Evaluation Setup Details · p. 14

Generation and Prompting

• Qwen2-7B-Instruct supports flexible RoPE position rotation; an explicit QA prompt requests concise evidence-based answers, and retrieved items are concatenated in ranked order.The full CoinRAG prompt template is provided in Appendix B.Source: E Evaluation Setup Details · pp. 14–15

F Hardware Infrastructure and I/O Management

• The setup assumes a static or infrequently updated corpus, so reported offline costs are incurred once; incremental updates could re-encode only affected chunks..Source: F Hardware Infrastructure and I/O… · p. 15
• Evaluation and latency benchmarks run in bfloat16 on an NVIDIA L40S, while training uses an RTX PRO 6000 and nugget extraction uses the GPT-4o-mini API.The single node has dual AMD EPYC 9354 processors, 188 GB system memory, and a local 3.84 TB NVMe SSD for dynamically loaded KV caches.Source: F Hardware Infrastructure and I/O… · pp. 15, 17
• Each 512-token Qwen2-7B-Instruct chunk cache occupies approximately 28 MB, but CoinRAG transfers less by slicing only required nugget token ranges.Chunk caches are stored as independent files and asynchronously loaded with a per-document thread pool.Source: F Hardware Infrastructure and I/O… · p. 17
• $265–325 for 6.08M training-corpus nuggets and under $2 for 277K evaluation-corpus nuggets are one-time extraction costs.The training corpus contains 950,221 unique passages, and the extracted nuggets have 99.9% validity.Source: F Hardware Infrastructure and I/O… · p. 17
• Under 30 minutes builds the three evaluation-corpus KV stores, which occupy 146 GB for HotpotQA, 75 GB for 2WikiMQA, and 173 GB for MuSiQue.Storage scales at approximately 27–28 MB per chunk; nugget spans cover about 70% of chunk tokens, so retaining only those positions could reduce footprint by roughly 30%.Source: F Hardware Infrastructure and I/O… · p. 17
• 140 GPU-hours on one RTX PRO 6000 fine-tune Qwen2-7B-Instruct for one epoch over 277,280 examples and 17,330 optimizer steps.Gradient accumulation is 16, peak memory is 75.8 GB, and this is a one-time cost per backbone checkpoint.Source: F Hardware Infrastructure and I/O… · p. 17

G Fine-Tuning Hyperparameters

• Fine-tuning uses a fixed hyperparameter configuration, gradient checkpointing, and memory-efficient 8-bit Paged AdamW to prevent out-of-memory errors under sequence-length constraints.Table 5 provides the hyperparameter specifications for the nugget-aware fine-tuning pipeline.Source: G Fine-Tuning Hyperparameters · p. 17, Table 5 · p. 17

H Successful and Failed Inference Examples

• Success cases illustrate cache stitching enabling robust semantic inference and knowledge recovery within a compressed footprint.Figure 9 categorizes three HotpotQA success-case types.Source: H Successful and Failed Inference… · pp. 17–18, Figure 9 · p. 18
• CoinRAG delivers a strong average F1-score across three test datasets and outperforms baselines under a targeted latency budget.The section presents this evaluation result despite the failed examples discussed alongside it.Source: H Successful and Failed Inference… · p. 18
• Failed cases show that aggressive nugget slicing can amplify surface entity biases or omit contiguous linking paths during retrieval and alignment.Figure 10 categorizes three HotpotQA failure types.Source: H Successful and Failed Inference… · p. 18, Figure 10 · p. 19