PaperlayerView exact revision on arXiv

Source-linked AI summary

Cartridges++: KV Cache Compression without Off-Context Derailment

Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni

arXiv:2609.35621v1cs.LG

TL;DR

Long-context KV caches are costly to store and serve, while CKV evaluations largely test document-related queries rather than unrelated ones. This paper measures off-context capability loss in Cartridges and introduces data mixing and relevance routing; both reduce that loss while retaining document utility.

  • Problem

    CKV evaluations largely focus on document-related queries, leaving open how reusable compressed memories affect behavior on unrelated queries where native attention can suppress irrelevant-context interactions.

    Source: §1 · p. 2

  • Method

    The paper evaluates off-context capability preservation and introduces training-time data mixing and inference-time relevance routing for document-specific Cartridges.

    Source: §1 · p. 2

  • Results

    Cartridges have strong context fidelity but greater off-context interference than full context, while both Cartridges++ variants improve capability preservation and retain document-utility gains across four long-context benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Reusable memories should be evaluated both for the information they preserve and for the interference they introduce when that information is irrelevant.

    Source: §8 · p. 10

    Off-context evaluation focuses on TinyMMLU, MMLU-Pro, and IFEval; mixing can reduce fidelity or leave residual interference, while routing depends on accurate relevance decisions and re-prefilling.

    Source: §8 · p. 10

Abstract

from arXiv · show

Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons. Compressed KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time. Methods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, Cartridges have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs. While existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, on-context queries, we investigate the crucial deployment question of whether they can handle off-context queries, something the native KV representation is particularly good at, thanks to the mechanics of attention. We observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions. We propose Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The router variant decides at inference time whether the query should use the learned long-context memory, while the data-mixing variant allocates a small fraction of training Q/As to queries outside the reference long document. Our study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training.

1 Introduction

The introduction argues that testing document utility alone misses how compressed KV memories affect unrelated queries. It identifies Cartridges’ off-context interference and proposes data mixing and relevance routing to retain document utility while recovering capability preservation.

1 Introduction

  • Figure 1 contrasts context fidelity and capability preservation, illustrating that Cartridges++ variants retain fidelity while improving general capabilities.The figure aggregates results across long-context datasets and depicts both an on-context query and an unrelated Saturn question.Source: Figure 1 · p. 2
  • Existing CKV evaluations largely test what compression preserves from the source document, leaving behavior on unrelated deployment queries untested.The paper argues that native attention can efficiently turn off interactions with unrelated context, a property that learned CKVs, particularly Cartridges, lose.Source: §1 · p. 2
  • Cartridges provide stronger context fidelity but cause greater off-context interference than full context, while heuristic eviction methods show more limited degradation.The paper names the desired property capability preservation: an irrelevant memory should not substantially degrade the underlying LLM relative to operating without context.Source: §1 · p. 2
  • Cartridges++ combines training-time data mixing with an inference-time relevance router to mitigate interference while preserving document-dependent utility.The introduction reports improvement in capability preservation across four long-context benchmarks while retaining document-utility gains.Source: §1 · pp. 2–3
  • The paper evaluates reusable, document-specific CKVs on both context fidelity and off-context capability preservation, including knowledge, instruction following, and context contamination.Its two-sided evaluation measures whether memories retain relevant document utility and avoid degrading behavior on unrelated queries.Source: §1 · p. 2

2 Related Work

Related work covers training-free KV eviction and learned context representations, while noting that prior compression evaluations largely focus on context-internal behavior. The paper studies off-context behavior and adapts selective activation and data mixing to document-specific KV memories.

2 Related Work

  • Training-free eviction compresses by selecting or combining native KV states; methods include H2O, StreamingLLM [Xiao et al., 2024], SnapKV [Li et al., 2024], and KVzip [Kim et al., 2025].PyramidKV and PyramidInfer allocate non-uniform layer budgets [Cai et al., 2025; Yang et al., 2024]; standard formulations of these methods and SnapKV use downstream queries for selection.Source: §2 · p. 3
  • Learned compressors either amortize an encoder across documents or optimize document-specific representations; Cartridges optimize key and value tensors, while Attention Matching fits biases and values to reconstruct full-cache attention.Cartridges operate in the frozen LLM without an encoder or model-weight updates; Cartridges at Scale extends the approach to document collections.Source: §2 · p. 3
  • Prior evaluations of compression failures largely examine context-internal losses, whereas this work tests context-external behavior across knowledge, instruction following, and contamination.The paper uses full-context baselines to distinguish ordinary distraction from interference induced by optimized KV state, and applies post-hoc routing and no-context-teacher data mixing.Source: §2 · p. 3

3 Problem Formalization

The section defines compressed KV memories and formalizes two requirements for reusable document-specific memories: preserving document-relevant behavior and limiting changes to behavior when the document is irrelevant.

3.1 KV cache and compression

  • Compression reduces generation cost from O(L) to O(m), where the compressed cache has m positions for a document of length L.The compression factor is ρ = m/L; the full cache footprint grows with document length and model depth.Source: §3.1 · p. 4

3.2 Distillation-based compression: Cartridges

  • Cartridges optimize document-specific KV tensors while keeping the LLM frozen, using synthetic traces from document subcontexts for teacher–student context distillation.The full-context model is the teacher, and the same frozen model conditioned on the Cartridge is the student.Source: §3.2 · p. 4
  • A trained compressed cache is loaded as a prefix and reused across queries without access to the source document.Source: §3.2 · p. 4

3.3 Two-sided requirements on a reusable memory

  • Reusable memories must serve document-related queries that concern information from the document and off-context queries that do not require it.Source: §3.3 · p. 4
  • R+ measures context fidelity, while R− measures how much the memory perturbs the model’s original behavior on queries that do not require the document.Full-context interference provides a reference for off-context perturbation.Source: §3.3 · p. 4
  • A well-behaved reusable memory should keep both R+ and R− small, with R− at minimum not substantially exceeding full-context interference.Source: §3.3 · p. 4
  • Cartridge self-study supervises document-grounded behavior but does not explicitly constrain R−, and the paper reports that this leads to interference.Source: §3.3 · pp. 4–5

4 Mitigating Memory Interference

The section addresses Cartridges’ one-sided training objective with two complementary mitigations: mixed training examples or inference-time relevance routing.

Mitigation strategies

  • The two interventions target either Cartridge training or its use, and a lightweight prompting baseline is reported separately because it does not eliminate interference.Source: §4 · p. 5

Training-time mitigation: data mixing

  • CD++ mixes self-study examples with a fraction γ of general-purpose off-context examples, using the no-Cartridge model as teacher for those examples.γ is the fraction of supervised assistant tokens contributed by the off-context pool; γ = 0 and γ = 1 recover self-study-only and off-context-only supervision.Source: §4 · p. 5
  • Data mixing regularizes the memory without adding a new loss or changing LLM weights, with γ controlling the trade-off between context fidelity and capability preservation.Source: §4 · p. 5

Inference-time mitigation: relevance routing

  • CR++ uses a router to activate an already-trained Cartridge for relevant queries and otherwise generates with the no-context model.A rejected query has the Cartridge removed, recovering no-context behavior without retraining the memory or LLM.Source: §4 · p. 5
  • The router uses incoming-query embeddings, with a kNN router as the main variant and its threshold calibrated on held-out off-context negatives.The approach incurs a lightweight relevance decision before generation.Source: §4 · p. 5

5 Benchmarking Beyond Context Fidelity

The benchmarking framework evaluates context fidelity alongside capability preservation, using document-relevant and off-context queries to assess reusable compressed memories.

Evaluation framework

  • The framework operationalizes context fidelity and capability preservation independently of any particular dataset.It compares compressed memories with full-context and no-context reference conditions.Source: §5 · p. 5

Evaluation axes

  • Capability preservation covers general knowledge, instruction following, and context interference, complementing context fidelity on document-relevant queries.Source: §5 · pp. 5–6
  • Instruction-following preservation tests whether users’ objectively verifiable constraints are satisfied despite an irrelevant memory.Source: §5 · p. 6
  • Context interference measures the fraction of unrelated-query responses whose rationale fabricates or attributes a quotation to the document, or claims it supplies the answer.The judge is independent of answer correctness; merely mentioning or dismissing the document does not count.Source: §5 · p. 6
  • Table 1 summarizes the benchmarks used to instantiate context-fidelity and capability-preservation axes; compare its coverage across those two requirements.Source: §5 · p. 5, Table 1 · p. 6

6 Experimental Setup

The experiments span four long-context datasets, multiple model families and compression methods, and benchmarks for off-context capability preservation.

Long-context datasets

  • The four long-context datasets range from approximately 4K to 140K tokens and cover research-paper QA, fictional patient records, Kalamang–English translation, and narratives.The benchmarks use task-native metrics: multiple-choice accuracy, corpus chrF, or reference-answer log-perplexity, as appropriate.Source: §6 · p. 6

Capability-preservation benchmarks

  • Capability-preservation evaluation uses TinyMMLU and MMLU-Pro for general knowledge, IFEval for instruction following, and a judge for context interference.Each query is paired with an unrelated cached document; Dolci-Instruct-SFT is used for mitigation training and calibration, not off-context evaluation.Source: §6 · p. 6

Models and compression methods

  • Experiments use Qwen3-4B, Qwen3-4B-Instruct-2507, and Gemma-4-E4B-it, comparing Cartridges with Attention Matching, eviction methods, and matched-length text summaries.Compression factors range from 0.001 to 0.20; Gemma compression applies only to global-attention cache slots.Source: §6 · pp. 6–7

Cartridges++ implementation details

  • Data mixing is evaluated at γ values of 0.01, 0.05, and 0.10, while the main per-document kNN router uses k = 10 and a threshold calibrated at α = 0.05.Source: §6 · p. 7

Compute cost of mitigation strategies

  • Data mixing adds no inference-time cost and has training cost comparable to standard Cartridges; routing adds a kNN lookup for rejected requests or an extra query prefill for accepted ones.The extra prefill does not repeat the long-document prefill.Source: §6 · p. 7

7 Results

Cartridges retain stronger on-context performance than eviction methods but can worsen off-context capabilities and context interference. Cartridges++ mitigations recover off-context behavior while largely preserving context fidelity, with gains transferring across models and settings.

Compression trade-off

  • Cartridges preserve substantially stronger on-context performance than eviction methods, especially under extreme compression, but worsen general capabilities and increase context contamination beyond retaining the full document.Eviction methods that retain native KV states sacrifice context fidelity but remain substantially closer to no-context behavior off-context.Source: §7 · pp. 7–8, Figure 3 · p. 8

Data mixing

  • At 1% mixing, general knowledge and instruction following improve and context interference falls; low-to-moderate mixing preserves context fidelity, while aggressive mixing degrades on-context performance.The study uses 5% mixing as a practical trade-off for CD++; gains also appear at 1%.Source: Figure 4 · p. 8, §7 · p. 8

Relevance routing

  • The kNN router accepts over 90% of on-context queries, and an external MiniLM encoder also routes strongly, showing that post-hoc routing need not rely on backbone representations.Figure 5 reports routing accuracy on three long-context datasets with Qwen3-4B-Instruct-2507.Source: §7 · p. 8, Figure 5 · p. 8

Overall mitigation results

  • Cartridges++ extends the trade-off frontier: 5% data mixing reduces off-context degradation with small or no on-context cost, while routing brings off-context accuracy close to the no-context reference and largely retains context fidelity.The reported comparison aggregates results across contexts and compression factors; routing also improves capability preservation when applied to Attention Matching, while Cartridges++ retains higher context fidelity with comparable capability preservation.Source: Figure 6 · p. 9, §7 · p. 9

Transfer across models and contexts

  • The improvements transfer to Gemma-4-E4B-it and to shorter QuALITY-20 contexts, where both mitigations remain effective.Gemma-4 results show preserved Cartridge context fidelity with substantially improved capability preservation.Source: Figure 7 · p. 9, §7 · p. 9

Additional analyses

  • Both mitigations preserve their benefits in multi-turn interactions without additional collapse beyond single-turn behavior; longer training does not resolve the original degradation, while data mixing retains its gains at later checkpoints.Routing in the multi-turn evaluation uses per-turn relevance labels.Source: §7 · p. 9

8 Conclusion

The conclusion frames irrelevant-memory interference as a cost of learned KV memories and presents data mixing and relevance routing as mitigations. The accompanying material documents AI use, reproducibility resources, and related work.

8 Conclusion

  • Cartridges can substantially perturb the model when stored documents are irrelevant, while Cartridges++ mitigations reduce degradation and retain context fidelity across the studied model families and context lengths.Data mixing can trade some document specialization for capability preservation; routing selectively activates the learned memory for relevant queries.Source: §8 · p. 10
  • Off-context evaluation covers TinyMMLU, MMLU-Pro, and IFEval, leaving additional potential failure modes for future characterization.Data mixing may reduce context fidelity or leave residual interference on poorly covered requests; routing depends on accurate relevance decisions and requires re-prefilling accepted queries.Source: §8 · p. 10

AI use statement

  • The authors report using generative AI for software development, qualitative analysis, figure creation, editing, and literature identification, with outputs reviewed by the authors.They report replicating judge analysis with two additional judges and checking figures against underlying results.Source: AI use statement · p. 11

Reproducibility statement

  • The reproducibility statement points to official model, dataset, and benchmark releases, method implementations, and appendices documenting baselines and evaluation protocols.It identifies Sections 5 and 6 and Appendix B as sources for protocol and benchmark details.Source: Reproducibility statement · p. 11

A Additional Related Work

  • Related work spans cache precision and dimensionality reduction, reusable learned encoders, document-specific memories, and selective activation; this study focuses on directly optimized KV states and their off-context interference.The paper distinguishes its training-time no-context-teacher data mixing and post-hoc CKV router from related memory approaches.Source: A Additional Related Work · p. 16

B.1 Extended benchmark details

The appendices specify benchmark populations and metrics, implementation choices for Cartridges++ and baselines, position conventions, and the protocol used to judge context interference.

B.1 Extended benchmark details

  • Four long-context settings span approximately 8K to 138K tokens, with task-specific metrics; off-context capability is assessed using TinyMMLU, MMLU-Pro, and IFEval.QuALITY uses 357 questions across 20 articles; LongHealth-10 uses 200 questions with eight generations each; QASPER-16 has 78 questions; MTOB uses 50 translation sentences.Source: §B.1 · p. 17, Table 2 · p. 17

B.2 Implementation details: Cartridges++

  • Cartridges use frozen-model self-study traces and chunk-conditioned teacher distributions, while data mixing adds no-context Dolci supervision and routing classifies queries using representations extracted without the Cartridge.The main router uses a fixed cosine kNN-10 recipe, with its activation threshold calibrated on held-out Dolci negatives rather than capability-preservation benchmarks.Source: §B.2 · pp. 18–19

B.3 Implementation details: baselines

  • Reusable eviction baselines adapt selection to a fixed document-only cache, while Attention Matching fits attention biases and values and text summaries replace the document within the same token budget.SnapKV and PyramidKV use document-prefill queries; H2O scores the full prefill once, then freezes the selected cache.Source: §B.3 · pp. 20–21

B.4 Position and sink conventions

  • Cartridge queries begin immediately after the compact prefix, whereas evicted caches retain original rotary positions; shifting the Cartridge query to the original context end lowers interference only alongside most on-context gain.The positional ablation moves the query and answer while leaving the Cartridge at the prefix.Source: §B.4 · p. 21, Figure 8 · p. 21

B.5 Judge rubric and scoring protocol

  • Context interference measures whether an off-context rationale is influenced by the irrelevant resident document, not whether its answer is correct.The primary rate counts a response as interfered when it fabricates a quote or answers the question from context; the judge input omits the document, gold answer, and parsed answer identity.Source: §B.5 · p. 22, Figure 9 · p. 22, Figure 10 · p. 23

C.1 Cartridges++: data mixing

Data mixing recovers off-context behavior while preserving document utility at modest mixing levels, whereas explicit relevance prompting does not reliably restore no-context behavior. The routing analysis compares acceptance of relevant queries with rejection of off-context queries.

C.1 Cartridges++: data mixing

  • Even 1% Dolci supervision visibly improves off-context accuracy and interference, while 5–10% generally moves capability measures toward no-context references and very large mixtures can reduce document utility.At γ = 0.10, held-out Dolci loss falls while self-study convergence matches the standard recipe; γ = 1 prioritizes Dolci and does not learn the document.Source: §C.1 · p. 24, Figure 11 · p. 24, Figure 13 · p. 26

C.2 Cartridges++: routing

  • Layer-18 features give a consistently strong router acceptance–rejection balance; mean and ramp pooling are similar, while kNN is stronger than LR on English QA and LR is stronger on MTOB.The main router fixes k = 10 as an intermediate choice; MiniLM is a competitive backbone-independent alternative.Source: §C.2 · p. 24, Figure 14 · p. 27

C.3 Prompting baseline

  • An explicit relevance instruction does not reliably improve off-context accuracy or IFEval and yields no consistent source-task gain, though it reduces judged interference in some cells.It does not recover no-context behavior.Source: §C.3 · p. 26, Figure 15 · p. 27

D Effect of Training Duration

The duration and per-benchmark analyses show that standard Cartridges retain degraded off-context behavior beyond their initial training pass, while data mixing preserves an off-context advantage. Across datasets, context fidelity alone does not capture the differing capability-preservation profiles.

D Effect of Training Duration

  • After the first pass, standard Cartridges gain little source-task performance and their off-context accuracy and interference plateau at degraded levels rather than recovering.The duration sweep indicates this failure is not an artifact of the chosen number of optimization steps.Source: D Effect of Training Duration · p. 28, Figure 16 · p. 28
  • At matched update counts, 5–10% data mixing keeps off-context performance near its initial level and prevents the large interference increase of self-study-only training.Mixed variants show similar early source-task convergence, and longer optimization neither creates nor erases their main benefit.Source: D Effect of Training Duration · p. 28

E.1 Per-benchmark overview

  • Across datasets, Cartridges are strongest on longer source tasks, but capability preservation does not reliably improve with a larger cache, and eviction methods remain comparatively clustered off-context.On MTOB, some off-context metrics deteriorate as compression factor increases; the per-benchmark figures break down these patterns by dataset and metric.Source: §E.1 · p. 29, Figure 17 · p. 29, Figure 18 · p. 30

E.2 Results in another model family: Gemma

Gemma results reproduce the Cartridge trade-off, while data mixing and routing mitigate off-context losses; these findings also extend to shorter contexts and generated-answer evaluation. Off-context degradation varies by subject, and accuracy losses do not necessarily track judged interference.

E.2 Results in another model family: Gemma

  • On Gemma-4-E4B-it, standard Cartridges improve source-task performance but reduce off-context knowledge and instruction following; both Cartridges++ mitigations succeed in this setup.This reproduces the qualitative asymmetry seen in the Qwen family, including on an architecture with local/sliding-window and global attention.Source: §E.2 · p. 30, Figure 20 · p. 31, Figure 21 · p. 32
  • Gemma data mixing recovers much of the off-context loss while keeping the source-task curve close to standard Cartridges.Mixtures are compared at identical completed-update boundaries; the preservation–specialization trade-off transfers across model families.Source: §E.2 · p. 32, Figure 24 · p. 33
  • Gemma routing uses mean-pooled layer-18 features with cosine kNN-10; the MiniLM router is backbone-agnostic and needs no feature-extractor retraining across Gemma and Qwen.The routing recipe was selected after repeating depth and pooling ablations with Gemma representations.Source: §E.2 · pp. 32–33, Figure 23 · p. 33

E.3 Smaller contexts: QuALITY

  • At roughly 8K tokens, QuALITY-20 does not consistently favor learned memory on-context, yet Cartridges still reduce off-context accuracy and instruction following and increase judged interference.Data mixing and kNN routing bring off-context accuracy and interference toward the no-context reference while keeping Cartridge on-context accuracy.Source: §E.3 · p. 34, Figure 25 · p. 34

E.4 QASPER: generated-answer F1

  • Generated token F1 on the same 78 QASPER questions supports the teacher-forced perplexity conclusion: Cartridges retain a large source-task advantage at high compression and remain among the strongest methods as budget grows.The two metrics need not induce identical method orderings at every compression factor.Source: §E.4 · pp. 34–35, Figure 26 · p. 35

E.5 Fine-grained MMLU-Pro categories

  • At 20% compression with a LongHealth Cartridge, the largest accuracy losses are in law and other, while philosophy has the largest interference increase.Across QASPER and MTOB, interference and accuracy losses also vary by category; subjects with the largest accuracy losses need not have the most judged interference.Source: §E.5 · p. 35, Figure 27 · p. 36
  • At 5% compression, example Cartridge responses include context contamination in correct and incorrect MMLU-Pro answers and in IFEval responses that pass evaluated constraints.The first four example cards attribute text absent from the resident document; the bottom cards show patient-record material replacing requested biography or SAP-procedure content.Source: §E.5 · p. 35, Figure 28 · p. 37

F Additional Failure Case: Failure to Abstain

On mismatched questions, Cartridges rarely abstain even when other methods do; their interference pattern persists across judges. In multi-turn use, mixing and routing preserve off-context capability, while supplementary baseline analyses examine reusable key selection and text summaries.

F Additional Failure Case: Failure to Abstain

  • On mismatched questions, Cartridges abstain on at most 10%, versus 36–98% for full context, no context, and eviction methods, even at equal or near-equal accuracy.The memory lacks evidence for these questions, and the generation prompt neither mentions abstention nor offers an abstain option.Source: F Additional Failure Case: Failure to… · p. 38, Figure 29 · p. 38

G Preserved Capability: Multi-Turn Interaction

  • Across repeated relevant turns, Cartridges show no additional multi-turn collapse; mixing improves off-context turns and routing selects the clean or Cartridge-prefilled path independently each turn.The routing results use clean per-turn re-prefilling.Source: G Preserved Capability: Multi-Turn… · p. 40, Figure 31 · p. 40

H Cross-Judge Agreement on Context Interference

I Additional Baseline Analyses

  • SnapKV and PyramidKV show similar compression curves whether retained keys are selected using offline self-study questions or all document-token queries during prefill.The main experiments use context prefill because it is query-independent and pools evidence from the full document.Source: I Additional Baseline Analyses · p. 44, Figure 34 · p. 44
  • The text-summary baseline matches the compressed-KV token budget; the study compares prose and itemized prompts and selects Concise Summary for short QuALITY articles and Itemized Record for long contexts.Prompt analyses relate generated summary length to on-context performance.Source: I Additional Baseline Analyses · p. 44, Figure 35 · p. 45, Figure 36 · p. 45

Related papers

Loading 2609.35621v1…