PaperlayerView exact revision on arXiv

Source-linked AI summary

Doc-to-LoRA: Learning to Instantly Internalize Contexts

Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert Tjarko Lange

arXiv:2602.15902v1cs.CLcs.AI

TL;DR

Long-context prompting is effective but costly, and repeatedly applying context distillation is impractical for changing prompts. D2L meta-learns a hypernetwork that generates LoRA adapters in one forward pass, achieving near-perfect long-context needle retrieval and better efficiency than standard context distillation on QA tasks. Its scope remains bounded by expensive meta-training, target-model-specific retraining, and a performance gap from in-context learning.

  • Problem

    Long prompts make inference memory-intensive and slow, while per-prompt context distillation incurs substantial training cost and latency.

  • Method

    D2L meta-learns a hypernetwork that maps an unseen context to LoRA adapter parameters for a frozen target LLM in a single forward pass.

  • Results

    D2L achieves near-perfect zero-shot needle-in-a-haystack accuracy beyond 4× the target LLM’s native context window and outperforms context distillation on QA under limited update budgets.

  • Takeaways & Limitations

    D2L enables subsequent queries to use internalized context without re-consuming the original prompt, reducing update latency and memory usage.

  • Takeaways & Limitations

    D2L requires an expensive meta-training phase, must be retrained for a new target LLM, and still has a performance gap from in-context learning.

Abstract

from arXiv · show

Long input sequences are central to in-context learning, document understanding, and multi-step reasoning of Large Language Models (LLMs). However, the quadratic attention cost of Transformers makes inference memory-intensive and slow. While context distillation (CD) can transfer information into model parameters, per-prompt distillation is impractical due to training costs and latency. To address these limitations, we propose Doc-to-LoRA (D2L), a lightweight hypernetwork that meta-learns to perform approximate CD within a single forward pass. Given an unseen prompt, D2L generates a LoRA adapter for a target LLM, enabling subsequent queries to be answered without re-consuming the original context, reducing latency and KV-cache memory consumption during inference of the target LLM. On a long-context needle-in-a-haystack task, D2L successfully learns to map contexts into adapters that store the needle information, achieving near-perfect zero-shot accuracy at sequence lengths exceeding the target LLM's native context window by more than 4x. On real-world QA datasets with limited compute, D2L outperforms standard CD while significantly reducing peak memory consumption and update latency. We envision that D2L can facilitate rapid adaptation of LLMs, opening up the possibility of frequent knowledge updates and personalized chat behavior.

1. Introduction

The paper targets the transient, memory-intensive nature of in-context learning and the cost of repeatedly internalizing changing information with context distillation. D2L meta-learns a hypernetwork that converts a context into a reusable LoRA adapter in one forward pass, with strong long-context and QA results.

  • Motivation: ICL is effective but transient and memory-intensive, while long prompts increase latency and memory through quadratic attention and KV-cache growth.Generation quality typically degrades at longer context lengths.
  • Motivation: Repeated context distillation is slow when information sources change, limiting scalable customization for practitioners.Standard supervised finetuning also requires task-specific data and risks overfitting when data is scarce.
  • D2L approach: D2L meta-learns a hypernetwork that maps a context to a context-specific LoRA adapter, allowing later queries without re-consuming the context.A single inexpensive forward pass executes the learned distillation process, reducing latency and KV-cache size.
  • D2L approach: D2L uses a Perceiver-based architecture and chunking to process contexts longer than the target model’s context window and produce higher-rank adapters.The design maps variable-length inputs to fixed-shape outputs while expanding adapter rank across chunks.
  • Results: Near-perfect zero-shot accuracy is achieved for needle information at lengths up to 4× the base model’s maximum length.This result is reported for the synthetic Needle-in-a-Haystack task.
  • Results: Under limited update budgets, D2L outperforms context distillation while reducing update latency and memory usage, and generalizes to long-document QA.The paper also reports zero-shot transfer of visual information from a VLM to a text-based LLM for image classification.

2. Preliminaries

Context distillation transfers behavior induced by an in-context prompt into model parameters using the same LLM as teacher and student. The paper distinguishes single-query distillation from a more robust multi-query formulation that supports unseen downstream queries.

  • Context Distillation: Context distillation trains a student without the context to imitate responses from the same LLM acting as a context-aware teacher.The teacher samples responses with access to context c, while the student trains using only query x.
  • Context Distillation: The context-distillation objective uses sampled teacher responses as targets for training context-specific model parameters.The context-specific parameters are initialized from the original parameters.
  • Context Distillation: Context distillation can internalize new knowledge or behaviors without the expensive data-gathering steps required by supervised finetuning.Once internalized, the model can use the resulting parameters for later inference.
  • Defining Internalization: Query-dependent distillation relies on a single query x and risks overfitting because its learning signal comes from one triplet.The paper introduces query-independent distillation as a more robust alternative.
  • Defining Internalization: Query-independent distillation uses multiple queries and self-generated responses to form a small context-specific dataset.Relevant questions can be generated by prompting an LLM about context c.
  • Defining Internalization: Successful internalization means the model accesses information from context through internalized parameters as if the context were still provided.The paper defines internalization through the optimization of the query-independent objective.

3. Meta-Learning Context Distillation

D2L meta-trains a hypernetwork to map contexts to LoRA updates for a frozen target model, amortizing context-distillation work into one forward pass. Its Perceiver-style architecture handles variable-length contexts, while chunking composes adapters for long inputs.

  • Meta-learning context distillation: D2L maps a context to LoRA adapter parameters that modify a frozen base model, producing a context-internalized model.The hypernetwork Hϕ generates the parameter update ΔWc for each context.
  • Meta-learning context distillation: Meta-training optimizes one hypernetwork to generalize the context-to-parameter mapping across contexts rather than training a separate model for each context.The learned mapping approximates context-conditioned distillation for unseen contexts.
  • Meta-learning context distillation: After training, one hypernetwork forward pass generates internalized parameters and amortizes query generation and backpropagation used by traditional context distillation.The mapping is intended to serve as a generic context-to-parameter transformation.
  • D2L Architecture: For each target layer, D2L feeds frozen-LLM activations into a shared hypernetwork that outputs low-rank LoRA matrices.The layer update is represented as ΔWl = BlAl.
  • D2L Architecture: The Perceiver-style hypernetwork maps variable-length activations to a fixed number of latent queries corresponding to the LoRA rank.Cross-attention from learnable latent queries to layer activations produces the latent vectors used by output heads.
  • D2L Architecture: D2L processes long contexts in contiguous chunks and concatenates per-chunk adapters along the rank dimension, yielding total rank r · K.This integrates information across chunks without changing the hypernetwork’s output shape and can also produce compressed KV cache.

4. Implanting Synthetic Needle-in-a-Haystack Information

D2L is evaluated on synthetic needle-in-a-haystack retrieval, where it must internalize a hidden number into a LoRA adapter. It preserves retrieval accuracy beyond the base model’s context limit while greatly reducing inference memory.

  • Task setup: The NIAH task inserts a sentence defining a special 4-digit number into distractor text, which the model must retrieve from a query.The base model is gemma-2-2b-it with an 8K-token context length.
  • Task setup: D2L maps the context into a LoRA adapter that stores the needle value, while the base model receives only the query during evaluation.Inputs are segmented into equal-sized chunks, with 1024 tokens as the maximum chunk size.
  • Retrieval results: Perfect accuracy is maintained through 8K tokens, and retrieval remains close to perfect up to 40 chunks, or 40K tokens, after the base model degrades beyond its limit.This demonstrates generalization across both chunk size and total chunk count.
  • Efficiency results: At 128K tokens, the base model requires more than 12 GB of additional memory, whereas internalized knowledge uses less than 50 MB regardless of haystack length.The comparison concerns additional memory required to generate a response.

5. Experiments

Experiments compare D2L with context-distillation and other baselines on reading comprehension, long-document QA, and visual-information transfer. D2L combines strong internalization with lower update and inference costs, including zero-shot generalization to longer documents and another modality.

  • Experimental setup: D2L is evaluated on six real-world QA benchmarks using generated data and LoRA adapters, alongside CD, T2L, base-model, and in-context baselines.The main task metric is word-level ROUGE-L F1, reported relative to the base model with direct context access.
  • Reading comprehension: 82.5% relative performance to the ICL upper bound is achieved on SQuAD, while D2L outperforms all in-parameter baselines across SQuAD, DROP, and ROPES.Its SQuAD performance is roughly comparable to LLMLingua-2 at 40% of the original context length, but without retaining the context.
  • Reading comprehension: Less than 1 second is needed for D2L internalization, compared with around 40 seconds for oracle CD and more than 100 seconds for vanilla CD.Vanilla CD also incurs query-generation overhead, while T2L updates instantly but does not effectively internalize knowledge.
  • Reading comprehension: Less than 2 GB of VRAM is used by D2L and oracle CD during updates, whereas CD with generated queries uses more than 40 GB.The comparison is reported across the three reading-comprehension benchmarks.
  • Long-document QA: D2L internalizes long-context documents without explicit long-sequence training, reaches the oracle-CD upper bound on 2WikiMultihopQA, and maintains sub-second internalization with iterative LoRA generation.With five queries, CD uses up to 79 GB of VRAM; D2L uses twice less memory than the oracle update.
  • Long-document QA: During response generation, ICL requires around 1 GB of VRAM, while all in-parameter knowledge methods use less than 100 MB.D2L also answers more accurately than the CD baseline in this long-document setting.
  • Visual information transfer: A text-only target model reaches 75.03% accuracy using visual information internalized from a VLM, despite D2L and the target LLM having seen only text during training.Using a VLM as context encoder negatively impacts text-based QA but enables visual transfer into the target model’s parameters.

6. Analyses

The analyses test D2L under extreme query-internalization and varying CD query budgets. D2L remains effective with sub-second internalization, while vanilla CD improves with more queries but remains slower and less effective under practical budgets.

  • Zero-Shot Query Internalization: D2L achieves 0.587 ROUGE-L recall when internalizing queries, outperforming the no-context baseline at 0.185.In this swapped configuration, performance is lower than usual and outputs are often verbose, reducing precision.
  • Interpretation: D2L may emulate distillation over many queries because training across millions of contexts exposes it to broader query distributions than any single sample.The authors also note that this may reflect a specialized, query-related bias.
  • Varying Query Budgets: Vanilla CD improves from 0.506 with 20 queries to 0.650 with 100 queries on a 100-sample SQuAD subset.The improvement remains below D2L’s 0.866 result.
  • Varying Query Budgets: D2L reaches 0.866 on the same SQuAD subset without per-sample query generation or backpropagation.Its result is substantially closer to the oracle CD upper bound.
  • Latency: CD with 100 queries takes more than 10 minutes per sample, whereas D2L internalizes knowledge in the sub-second regime.The comparison motivates D2L for settings requiring responsive updates.

7. Related Work

Related work uses hypernetworks and context distillation to adapt language models or compress prompts. D2L differs by meta-training a hypernetwork with the CD objective to generate context-specific LoRA adapters.

  • Hypernetwork-Based Adaptation: Prior hypernetwork methods adapt LLMs on the fly for different use cases, including compressing examples or task instructions into prefix or gist tokens.The cited examples include MEND and Gisting.
  • Generative Adapter: Generative Adapter trains its hypernetwork with next-token prediction on ground-truth tokens, whereas D2L uses the CD objective and generated queries with self-responses.D2L outputs a context-specific LoRA adapter.

A. NIAH experiment details

The NIAH experiment uses a simplified D2L architecture and controlled synthetic training setup to evaluate needle internalization. Training uses 640K randomly chunked samples and runs for one epoch on a single H200 GPU.

  • Architecture: NIAH maps gemma-2-2b-it activations from layer 6 to LoRA adapters for all layers.The per-layer output heads share the same inputs, and the experiment uses cross-entropy on ground-truth tokens instead of self-distillation.
  • Training Data: Training samples contain 32 to 256 tokens, with random chunking into 1, 2, or 3–8 chunks.The chunk probabilities are 50%, 12%, and 37.5%, respectively.
  • Compute: D2L is trained for 1 epoch with learning rate 4 × 10^-5 on a single H200 GPU.Meta-training takes around 3 hours, and all evaluations use a single H200 GPU.

B. Main Experiments Details

Figure 5 describes the training-data length distribution and distinguishes the contexts consumed by D2L from the query-response inputs used by the base model for meta-training.

  • Training Data: D2L consumes contexts, while the base model consumes queries and responses to compute the meta-training loss.The total count measures unique context-query-response triplets.
  • Training Data: The original contexts before query generation contain roughly 900M tokens in total.This figure reports the aggregate token count before generating queries.

B.1. Meta-Training Data Generation Pipeline

D2L’s meta-training corpus combines large-scale educational web text with passage-grounded QA data, while its hypernetwork converts layer activations into LoRA parameters. Training uses staged chunking, and data ablations assess robustness to removing QA examples.

  • Data construction: The corpus contains approximately 900 million FineWeb-Edu tokens and 3.2 million unique contexts after adding filtered PwC, SQuAD, ROPES, and DROP passages.FineWeb-Edu samples are paired with 10 context-grounded queries generated by gemma-3-12b-it.
  • Evaluation setup: Evaluation uses at most 500 test samples per dataset because context distillation has high overhead latency.All measurements are performed on a single H200 GPU.
  • Hypernetwork: The hypernetwork uses a Perceiver-style cross-attention encoder over per-layer token activations followed by output heads that generate LoRA matrices.The architecture is described as two modules: an encoder and LoRA output heads.
  • Training procedure: D2L is trained in two stages, first producing one chunk for 80K gradient steps before introducing randomly chunked inputs.The first stage emphasizes information internalization before compositionality across generated LoRAs.
  • Data ablation: Removing QA-task samples leaves overall performance comparable and still consistently better than low-budget CD across datasets, although DROP performance declines substantially.The ablation suggests robustness to training-data format, with task-specific sensitivity on DROP.

C.2. Ablating Training Objective

The ablations examine training loss, LoRA rank, knowledge interference, and alternative parameterizations. KL-based distillation and higher-rank adapters improve selected results, while unrelated prompts can interfere with the base model’s knowledge.

  • Training objective: KL distillation reaches a 0.385 recall score, compared with 0.235 for the next-token-prediction variant.The paper attributes the difference plausibly to KL matching the teacher’s full response distribution.
  • LoRA rank: Rank-16 LoRA outperforms rank-8 LoRA by significant margins on SQuAD and DROP, while performing similarly on ROPES.The experiment increases generated-LoRA rank from 8 to 16 to probe scalability with more training compute.
  • Knowledge interference: Replacing contexts with unrelated assistant or distracting prompts significantly reduces D2L performance relative to the base model and CD.The setup tests whether internalized information interferes when queries are unrelated to the internalized context.
  • Evaluation scope: The reported evaluations cover performance, update memory, and latency across QASPER, MultiFieldQA, and SQuAD settings.The supplied table and figure captions identify these evaluation axes and datasets without providing their numerical rows.
  • Alternative parameterization: The generated-KV-cache variant achieves near-perfect retrieval accuracy up to 8K tokens, after which performance degrades gracefully.Directly generated keys degrade earlier, around 4K tokens, whereas normalization and RoPE sustain accuracy longer.

E. Results with Other Models

D2L generalizes across Mistral-7B-Instruct-v0.2 and Qwen3-4B-Instruct-2507, retaining strong internalization with speed and memory advantages. However, long-context updating remains difficult for Mistral, and evaluation uses recall because its responses are verbose.

  • Model coverage: D2L is evaluated with Mistral-7B-Instruct-v0.2 and Qwen3-4B-Instruct-2507 as target language models.These experiments test generality across model families and sizes.
  • Evaluation metric: For Mistral and Qwen3, ROUGE-L recall is reported instead of ROUGE-L F1 because verbose responses lower precision and F1.The models remain verbose despite instructions to output only the answer.
  • Results: Across Figures 10–13, D2L internalizes new information while outperforming in-parameter baselines with speed and memory efficiency.The paper describes the overall finding as consistent with results using gemma-2-2b-it.
  • Results: The results across model families and sizes indicate robustness and generality of the proposed method.This is the paper’s stated cross-model conclusion.
  • Limitation: All internalization methods struggle to update long-context information for Mistral-7B-Instruct-v0.2.The authors provide no explanation and leave investigation of this phenomenon for future work.

G. Conclusion

D2L emulates context distillation through a single hypernetwork forward pass, enabling rapid context internalization and reuse. It extends effective context length and reduces update costs, but still requires expensive meta-training and retraining for each target LLM.

  • Conclusion: D2L emulates context distillation in a single hypernetwork forward pass, amortizing query generation and backpropagation into meta-training.The method is presented as an approach to instant and inexpensive knowledge internalization.
  • Conclusion: On synthetic NIAH, D2L internalizes needle information and extends the base model’s effective context window beyond 4× its original length.The conclusion identifies this as the principal long-context result.
  • Conclusion: On real-world QA tasks, D2L outperforms traditional CD under limited query budgets while reducing internalization memory usage and latency.The conclusion frames this as the main practical QA outcome.
  • Conclusion: D2L can zero-shot internalize visual information from a VLM context encoder, giving the target LLM basic visual understanding through internalized information.This result extends the method beyond text-only context internalization.
  • Limitations: D2L still requires a single expensive meta-training phase and retraining the hypernetwork for a new target LLM.A complete gemma-2-2b-it meta-learning run takes around 5 days on 8 H200 GPUs.
  • Limitations: The method’s current scope is limited to LoRA parameterization, while the paper notes possible performance, efficiency, and forgetting benefits from alternatives.The discussion also notes a performance gap between ICL and in-parameter knowledge methods.
Loading 2602.15902v1…