PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Should a Document Live: Context, Representations, or Parameters?

Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias, Adrià de Gispert

arXiv:2609.17346v1cs.CLcs.AI

TL;DR

LLMs need efficient ways to inject knowledge absent from pretraining, but context, representation, and parameter approaches involve different trade-offs. The paper conducts a controlled comparison across five knowledge-intensive benchmarks and finds that Cartridges are generally most accurate, while forgetting and composition remain important constraints.

  • Problem

    Existing evaluations often test recall of pretrained Wikipedia knowledge and cover limited methods or only single-document settings.

  • Method

    The paper compares representation-based KV-cache methods and parametric fine-tuning methods across five knowledge-intensive benchmarks, storage budgets, retrieval settings, and control benchmarks.

  • Results

    Cartridges are the most accurate injection method across nearly every storage budget and the only method matching ICL in multi-document retrieval, but they incur forgetting.

  • Takeaways & Limitations

    No method dominates every axis: Cartridges favor accuracy and document composition, while parametric adapters offer fixed-size, cheap serving and less forgetting.

  • Takeaways & Limitations

    The study uses two mid-sized models and a single retrieve-then-compose pipeline, so larger models and more sophisticated composition schemes remain untested.

Abstract

from arXiv · show

To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.

1 Introduction

The paper asks where new knowledge should live—context, representations, or parameters—through a controlled comparison of injection methods and trade-offs across settings.

  • Prior evaluations often use pretrained Wikipedia knowledge, testing recall rather than whether methods encode genuinely new information.
  • The study compares Cartridges and Compaction with LoRA, MLP adapters, and full fine-tuning across five knowledge-intensive tasks and single- or multi-document settings.
  • The analysis measures how adapter size affects performance and catastrophic forgetting, alongside storage and runtime trade-offs.
  • Adapter composition across retrieved documents is possible under certain conditions but remains challenging.

2 Related work

Related work places knowledge injection in context, compressed representations, or model parameters, but prior comparisons remain mixed and narrowly scoped.

  • Representation-based methods include compressed vectors or KV caches, while parametric methods include adapters and other trainable parameter updates.
  • Knowledge injection methods encode documents once into compact adapters, using either compressed KV-cache prefixes or model parameters.
  • Existing comparisons report mixed conclusions and omit important settings or method families, including supervised fine-tuning and multi-document composition.

3 Methodology

The methodology evaluates context-based, representation-based, and parametric adaptation under no-context, oracle single-document, and retrieved multi-document conditions.

  • The study compares context, Cartridges, Compaction, LoRA, MLP adapters, and full fine-tuning as knowledge-injection methods.
  • The no-context baseline measures answers available from pretraining alone, while ICL places the document directly in the context window.
  • Cartridges train a KV-cache prefix, whereas Compaction compresses documents by matching the attention assigned to the full document.
  • In the oracle setting, only the gold document is encoded to isolate single-document knowledge injection from retrieval.
  • In the multi-document setting, retrieved chunks are composed by concatenating KV caches for representation methods or averaging adapter weights for parametric methods.

4 Experimental Settings

Experiments use two instruction-tuned models, five knowledge-intensive benchmarks, controlled adapter sizes, retrieval settings, and KL-divergence distillation from an in-context teacher.

  • Qwen3-8B is the main model, with validation on Gemma-3-12B in the appendix.
  • The experiments evaluate adaptation methods on five knowledge-intensive benchmarks using dataset-specific metrics and document statistics.
  • Training uses the same LLM-generated Self-Study dataset for all methods.
  • The objective minimizes KL divergence between an in-context teacher and the adapted student model.
  • For multi-document evaluation, chunks are retrieved without query reformulation and concatenated in retrieval order for ICL and representation-based methods.
  • Adapter sizes are compared at 2×, 10×, 20×, 50×, and 100× compression, matching LoRA and MLP footprints to KV-cache storage where possible.

5 Experimental Results

In single-document oracle evaluation, representation-based methods lead at fixed size, while Cartridges remain robust under compression and support multi-document composition better than alternatives.

  • 5.1 Single Oracle document: 73.4 average score makes Compaction strongest at fixed size, matching ICL's 73.6 and exceeding Cartridges, LoRA, MLP adapters, and full fine-tuning.The corresponding averages are 70.6 for Cartridges, 64.5 for LoRA, 62.2 for MLP adapters, and 59.8 for full fine-tuning.
  • 5.1 Single Oracle document: Cartridges stay nearly flat from 2× to 100× compression and dominate storage-matched parametric methods on every single-document dataset.Examples include LongHealth 81.1 → 77.3, QuALITY 78.6 → 76.4, and TechQA 75.8 → 76.9.
  • 5.1 Single Oracle document: Compaction degrades steeply with compression, falling on FinQA from 66.4 at 2× to 19.3 at 20× and on LongHealth from 87.7 to 46.5 at 100×.At high compression, parametric methods overtake Compaction.
  • 5.2 Multiple retrieved documents: Cartridges maintain or improve scores as retrieved-document count rises, whereas Compaction and merged parametric adapters degrade monotonically.For example, Cartridges rise on LongHealth from 70.8 to 83.2 between k=1 and k=10, while Compaction drops from 72.9 to 56.3.
  • 5.2 Multiple retrieved documents: Jointly trained parametric adapters outperform merged adapters once a distractor is added, but composition remains ineffective for Compaction and parametric injection.At k=3, Joint versus merging is 13.0 versus 4.9 on FinQA and 40.0 versus 23.5 on TechQA.
  • 5.2 Multiple retrieved documents: Larger adapters do not solve composition: LoRA and MLP scores converge by k=10, while Compaction is further harmed by compression.On LongHealth, Compaction falls from 56.3 at 2× to 37.2 at 50×; LoRA and MLP converge to around 5 on FinQA and 40–49 on LongHealth.

6 Analysis

The analysis measures catastrophic forgetting and inference cost after knowledge injection. Cartridges and wider MLP adapters can reduce general capabilities, while parametric methods are cheapest per query.

  • Catastrophic forgetting: Cartridges degrade control-benchmark performance by 6% on average, with HumanEval falling by 16 points at high compression.Compaction stays at the base model across compression rates.
  • Catastrophic forgetting: LoRA preserves base-model capabilities at every rank, whereas MLP adapters degrade beyond a moderate bottleneck.On GSM8K, MLP performance falls from approximately 92 at the smallest bottleneck to approximately 60 at the largest.
  • Catastrophic forgetting: The low-rank constraint, rather than parameter count alone, is identified as explaining LoRA's greater robustness at matched parameter count.Large MLP adapters require a lower learning rate to avoid a steeper collapse, including GSM8K falling to 24 at 10^-4.
  • Cost analysis: Every injection method is cheaper per query than ICL, with parametric methods cheapest at inference and independent of document length.ICL repeatedly preprefills retrieved documents, costing about 1k tokens on FinQA to 11k on LongHealth for the document alone.

7 Discussion

The discussion identifies clear trade-offs among injection methods: Cartridges lead in accuracy and composition, while low-rank methods reduce forgetting and full fine-tuning performs worst. Catastrophic forgetting appears linked more to full-rank updates than to storage type or parameter count alone.

  • Which method to use: Cartridges are most accurate across settings and remain effective for multi-document composition, whereas parametric adapters trail at matched storage.Joint fine-tuning is the best available parametric alternative but still lags composed Cartridges, especially on FinQA.
  • Which method to use: Full fine-tuning is the least accurate and most expensive approach, while forgetting more than any low-rank adapter.
  • Composition is not straightforward: Independent artifact composition remains difficult: merging parametric adapters or concatenating independently compressed caches can collapse with distractors.Mixed training enables Cartridges to concatenate and perform well, while KV-cache reuse and joint compression are proposed routes for improving composability.
  • On catastrophic forgetting: Only Cartridges forget among representation methods, suggesting catastrophic forgetting does not inherently arise from compressed KV caches.
  • On catastrophic forgetting: Full-rank updates appear to drive forgetting: full-rank MLP adapters and full fine-tuning forget, whereas low-rank constraints act as a regularizer.The discussion suggests subspace constraints for parametric methods intended to preserve general ability.

8 Conclusion

The paper presents a controlled comparison of representation-based and parametric knowledge injection across five knowledge-intensive benchmarks and single- and multi-document settings. Cartridges lead in accuracy and multi-document use, but incur forgetting, while other methods offer different cost, storage, or compositionality trade-offs.

  • Cartridges are the most accurate injection method at nearly every storage budget and the only method extending to multiple retrieved documents.They incur mild forgetting on control benchmarks.
  • Compaction matches the in-context oracle at low compression but degrades quickly as compression increases.
  • Parametric adapters trail Cartridges at matched storage and cannot be composed across documents, but remain fixed-size and cheap to serve.
  • Composing independently trained per-document artifacts remains the main open problem, with compressed-cache reuse identified as a promising representation-based direction.

Limitations

The study’s conclusions are bounded by its model, task, composition, and training-data choices. In particular, it tests only two mid-sized instruction-tuned models and one retrieve-then-compose pipeline.

  • The study evaluates only Qwen3-8B and Gemma-3-12B, so it does not establish whether the same trade-offs hold for larger models.
  • The evaluation covers knowledge-intensive tasks but not skill acquisition or improvement, such as reasoning or coding.
  • Multi-document findings reflect a single retrieve-then-compose pipeline using mean-merging for parametric adapters and prompt concatenation for representation methods.More sophisticated reranking, filtering, routing, retrieval, or merging could improve composition.
  • All methods use the same self-study synthetic data, so relative performance depends on that data’s quality and on the generators producing it.

Ethical Considerations

The ethical discussion emphasizes risks from storing injected knowledge in weights or KV caches and from adapting models that may hallucinate or forget general capabilities. The work is a controlled comparison, not a deployed system, and uses public datasets described as free of personal data.

  • Weights and KV caches are harder to inspect or update than explicitly retrieved passages, and adapted models can still hallucinate.
  • Catastrophic forgetting is safety-relevant because injecting new knowledge can silently erode a model’s general capabilities.
  • The study is a controlled comparison rather than a deployed system, limiting the direct scope of its deployment implications.

C Gemma-3-12B Results

On Gemma-3-12B, Compaction at 2× nearly matches the in-context oracle, while LoRA and trained Cartridges perform worse; the latter is attributed to catastrophic forgetting.

  • 65.2 vs. 68.9: Compaction at 2× nearly matches the in-context oracle, while LoRA scores 50.8 on average.
  • 37.3: trained Cartridges underperform on Gemma-3-12B and remain below Compaction across datasets.The paper attributes this pattern to catastrophic forgetting.
  • Distillation outperforms next-token prediction for LoRA on every dataset, averaging nearly 4 points higher.The largest gains occur on reasoning-heavy datasets.

G Self-Study Data Synthesis Details

The pipeline synthesizes self-study data from diverse prompts, then evaluates retrieval-augmented generation using fixed document chunking, dense embeddings, and top-k retrieval. Retrieval quality is consistently high, while effective context length is slightly below nominal due to overlap and shorter final chunks.

  • Question Generation: The question generator produces diverse factual, comparative, reasoning, and detail-oriented questions, while varied seed types expose the cartridge to multiple interaction patterns.Questions are generated in batches and include corpus-specific details such as names, dates, identifiers, and numerical values.
  • Sampling Strategy: Documents are sampled proportionally to length, assigning longer documents higher weights because they are assumed to contain more facts.The method replaces uniform random chunk sampling with proportional-to-length sampling to balance document coverage.
  • Sampling Rounds and Temperatures: 40,000 training examples per dataset are generated through four independent 10,000-sample rounds using question, structuring, summarization, and use_case prompts.The creative seed type is excluded from training because it produces open-ended prompts less suited to distillation.
  • Document Indexing: Documents are chunked with fixed-size windows and overlap, embedded into 1024-dimensional vectors, and retrieved using cosine similarity.The supplied indexing description specifies chunk size C = 512 tokens and 10% overlap; the effective-token table concerns a 1024-token configuration.
  • Retrieval: At evaluation, the system retrieves top-K chunks for K ∈{1, 3, 5, 10}, concatenates them in score order, and prepends them to the question as reader context.The pipeline retrieves K = 10 once and slices the result for smaller K values; shorter final chunks and overlap make effective context length slightly below nominal k × 1024.
  • Retrieval Quality: 95% or higher Recall@10 across four datasets indicates relevant documents are usually retrieved, making chunking-related information loss the main remaining concern.The reported datasets are LongHealth, QuALITY, FinQA, and TechQA.
Loading 2609.17346v1…