Encode Once, Answer Many: Internalizing Context via Parametric Flow

1 Introduction

• Long-context prompting can degrade accuracy and raises latency and memory costs, yet compression leaves knowledge external and requires re-supplying it for each query.The cited work describes lost-in-the-middle behavior, weak long in-context learning, and context rot as related effects.Source: §1 · pp. 1–2
• Fine-tuning internalizes knowledge but requires slow, hyperparameter-sensitive optimization for every new context, limiting continually changing or user-specific settings.The optimization must be repeated whenever the knowledge changes.Source: §1 · p. 2
• CLIK maps a context to a jointly generated LoRA in one forward pass using cross-attention, then answers from the query alone without the original context.The generator models dependencies across adapter layers and modules rather than emitting independent fragments.Source: §1 · pp. 2–3
• Because the adapter depends only on the context, it can be generated once and reused across queries, amortizing the cost of reading the document.The introduction characterizes this as combining the immediacy of in-context learning with the persistence of parameterized knowledge.Source: §1 · p. 3
• CLIK substantially outperforms the no-context base model and prior hypernetwork-based generators on single- and multi-hop document QA, but remains behind full-context access and transfers to multi-turn dialogue without specific training.The generated adapter’s context-dependent information also concentrates in a compact, specific LoRA substructure.Source: §1 · p. 3

2.1 Context Internalization

• Training-free methods keep knowledge outside model parameters and re-supply it at inference, whereas optimization-based methods internalize it but require a separate loop for each context.CLIK amortizes that per-context optimization into a learned mapping.Source: §2.1 · p. 3

2.2 Hypernetworks

• Prior LLM hypernetworks generate LoRA adapters from task descriptions or contexts, but fragment-wise designs ignore dependencies; CLIK instead generates the full adapter jointly through cross-attention.The passage contrasts CLIK with Doc-to-LoRA’s per-layer independent hypernetwork and SHINE’s learned memory-token input to a subsequent hypernetwork.Source: §2.2 · p. 3
• The overview depicts distillation from an in-context teacher to a query-only student, with the frozen backbone shared and only the parameter generator trainable.The objective is temperature-scaled KL divergence between next-token distributions over shared answer tokens.Source: Figure 2 · p. 4
• Context-to-LoRA generation is presented as one instance of a broader weight-space generation trend spanning other modalities, tasks, and agentic settings.Examples include image editing, video-anomaly detection, and internalizing video or codebases into LoRA parameters.Source: §2.2 · p. 4

3 Method

• CLIK maps each context to a low-rank adapter for the frozen language model, which answers questions using a context-free prompt.The parameter generator is the only trainable component.Source: §3 · p. 4

3.1 Training Data Construction

• FineWeb-Edu passages provide general-domain contexts, with synthetic question–answer pairs teaching the generator what to internalize and how to recall it.The paper reports that FineWeb-Edu surpasses SQuAD, MS MARCO, and TransMLA in its ablation.Source: §3.1 · p. 5

3.2 Context Encoding

• The context encoder combines hidden states from all backbone layers using learned weights, retaining token-level positions for conditioning the generator.The ablation reports that this learned mixture outperforms using the last layer, a middle layer, or a uniform mean.Source: §3.2 · p. 5

3.3 Parameter Generator Pipeline

• Adapter factors become uniform-shaped weight tokens; factorized attention exchanges information across layers and within each layer, then cross-attends to context states.Factorized attention avoids full attention over the flattened adapter array, which the paper describes as impractical at this scale.Source: §3.3 · p. 6
• Training distills the in-context teacher’s next-token distributions into the adapter-based student using temperature-scaled KL divergence over answer tokens.Only the generator is updated; the teacher is treated as a fixed target.Source: §3.3 · p. 6
• The generator produces the adapter in one pass, injects it into the frozen backbone, and answers from the question alone; each context’s adapter is reused across its queries.The context is consumed by the generator and is not re-read by the backbone.Source: §3.3 · p. 7

4.1 Experimental Settings

• Experiments use a frozen Qwen3-4B-Instruct backbone and evaluate six single- and multi-hop QA datasets, with context removed at inference except for Base + Context.CLIK generates rank-16 LoRA updates for gate, up, and down projections across 36 layers using a 12-block generator; training uses FineWeb-Edu 10B contexts, five generated question–answer pairs per context, and two epochs. Baselines include context-internalization methods retrained on the same data and adapter targets.Source: §4.1 · pp. 7–8

4.2 Main Results

• With the format instruction, CLIK is the best context-free method on all six datasets, ahead of SHINE (Liu et al., 2026) and Doc-to-LoRA (Charakorn et al., 2026), but behind Base + Context on every dataset.Without the instruction, scores are strongly affected by output format, so CLIK sometimes exceeding Base + Context is not evidence of matching direct context access. CLIK was trained without a format instruction, making that protocol out of distribution for the generator.Source: Table 1 · p. 8, §4.2 · p. 9

4.3 Controlled Comparison at Matched Scale

• At matched generator scale, SHINE-L still trails CLIK, while half-size CLIK-S beats the larger D2L and SHINE comparators; CLIK-B uses the least training compute and has higher average F1 than every baseline.Matching parameter counts does not match compute: CLIK requires several times Doc-to-LoRA’s GPU-hours at the same number of steps, while halving CLIK’s depth also halves its generator compute.Source: Table 3 · p. 10, §4.3 · p. 10

4.4 Context Following of the Generated Adapter

• Adding in-domain SQuAD, DROP, and ROPES training data yields only modest gains, largest on DROP; FineWeb-Edu-only training already transfers to these evaluation tasks.The added supervision mainly adapts answer format, while the generator already handles the extractive and short-reasoning answers in SQuAD and ROPES well.Source: Table 4 · p. 11, §4.4 · p. 11
• On counterfactual probes, CLIK follows edited and novel context facts more reliably than Doc-to-LoRA (Charakorn et al., 2026) and SHINE (Liu et al., 2026), while retaining specificity and unrelated-query performance.CLIK answers paired nonce-fact documents with each document’s own value, has the highest generator accuracy on novel facts, and follows edited facts more often than the backbone prior under strong prior conflict. Across generators, wrong-context swaps stay close to Base accuracy and general knowledge on unrelated queries is preserved.Source: Table 5 · p. 11, §4.4 · p. 12

4.5 Multi-turn Evaluation

• Dynamic KV cache and dynamic LoRA both improve multi-turn results over single-turn without multi-turn-specific training, but neither has history-independent per-turn cost.Dynamic KV cache keeps the initial-context LoRA fixed while the backbone prompt accumulates history; dynamic LoRA regenerates parameters from the initial context plus the history at each turn, so its input cost grows with turns.Source: Figure 3 · p. 12, §4.5 · p. 13
• The exploratory recurrent update performs worse than even single-turn because the trained text-to-weight generator is not equipped to condition on prior adapter weights.The prior adapter’s flattened factor chunks lack semantic organization and canonical layer or position identity; learning a weight-to-weight update pathway is left to future work.Source: §4.5 · p. 13, Table 6 · p. 14

4.6 Cost and Efficiency

• CLIK’s per-context adapter replaces the context KV cache and is several times smaller for long documents, while CLIK-B reduces per-context state by more than an order of magnitude.The rank-16 CLIK adapter is comparable in size to the KV cache for a context of a few hundred tokens; only for short contexts is it larger than that cache, and the adapter must be loaded and switched per request.Source: §4.6 · p. 14
• After one-time internalization, each further question costs only adapter-based TTFT, while the adapter’s smaller state reduces memory and speeds loading compared with the context KV cache it replaces.CLIK internalization costs more than prefilling, though the extra cost is small for CLIK-B; batches with distinct contexts decode faster with lower peak memory.Source: §4.6 · pp. 14–15

4.7 Ablation Study on Hyper-parameters

• The best cross-attention input is a learnable mixture of hidden states across layers, the learning rate peaks at an intermediate value, and FineWeb-Edu yields the best result among tested training corpora.SQuAD training produces the lowest F1, while larger and more general sources help progressively; the authors avoid specializing training to a fixed short-answer format.Source: §4.7 · p. 15

4.8 Ablation of the Generator Architecture

• Removing across-layer and within-layer weight-token attention reduces performance, with joint removal especially harmful and the independent-fragment generator falling to Doc-to-LoRA’s level.Within-layer attention removal causes a larger drop than removing across-layer attention; the loss is concentrated in precision and is largest on HotpotQA.Source: §4.8 · p. 16

4.9 Analysis of LoRA Patterns

• Training strengthens the updates and develops a division of labor: gate and up projections become more consistent across contexts, while down projections become more context-specific, especially in deeper layers.The down-projection cross-context cosine shifts lower from early to late training, whereas gate and up cosine increases slightly.Source: §4.9 · pp. 16–17, Figure 4 · p. 17, Figure 5 · p. 17
• Factor interventions show that swapping downA across contexts leaves performance essentially unchanged, while swapping downB nearly collapses it; the authors caution this does not prove knowledge intrinsically resides in downB.The downB intervention can combine factors from different coordinate systems; the constructive CLIK-B test instead uses one shared coordinate system.Source: §4.9 · p. 18
• CLIK-B retains most of CLIK’s performance, outperforms the Table 1 baselines under format instruction, and exceeds D2L-L and SHINE-L in average F1 on Table 3.The remaining gap is largest on DROP and multi-hop datasets, indicating that CLIK’s gate, up, and downA factors also contribute context-specific information.Source: §4.9 · p. 18

5 Conclusion

• CLIK generates context-conditioned LoRA parameters in one forward pass, then answers downstream queries without re-reading the original context or test-time training.A Transformer uses cross-attention from context to adapter weights.Source: §5 · p. 18
• On document question answering, CLIK improves over the no-context base model and compares favorably with hypernetwork-based methods, while remaining behind a model that reads context directly.It avoids per-task fine-tuning; counterfactual probes indicate answers draw on internalized context rather than only the backbone prior.Source: §5 · pp. 18–19
• In multi-turn dialogue, CLIK improves over its single-turn baseline without additional training.Source: §5 · p. 19
• The study primarily evaluates document question answering and does not examine broader task types, longer contexts, or larger backbones.It also does not model retaining or reconciling knowledge across successive updates.Source: §5 · p. 19

A Parameter Generator Configuration

• The adapter covers the gate, up, and down feed-forward projections in all 36 layers, using rank 16 and scale α = 32.Each context produces about 21.2M adapter parameters (40.5 MiB in BF16); CLIK-B generates only down-projection B factors, reducing this to 1,474,560 parameters (2.8 MiB), or 14.4× fewer.Source: A Parameter Generator Configuration · p. 22
• The generator conditions on learned ELMo-weighted, all-layer backbone states projected from dimension 2560 to generator width 2048.Source: A Parameter Generator Configuration · p. 22
• A 12-block Transformer uses across-layer, within-layer, and cross-attention to generate adapter factors from weight tokens.Tokens are initialized from learned layer and within-layer position embeddings; two output heads project A and B tokens back to weight space, and the B head starts at zero.Source: A Parameter Generator Configuration · p. 22
• The generator has 1.01B parameters, including 604M in attention sublayers and about 403M in feed-forward sublayers.Its size is the quantity matched against baselines in the controlled comparison.Source: A Parameter Generator Configuration · p. 22

B Data Generation Configuration

• A fixed system prompt frames generation as reading-comprehension question generation, while a user template supplies the passage and formatting rules.Source: B Data Generation Configuration · p. 22
• The generated question–answer pairs leave answer length and response style unconstrained, following the model’s native in-context behavior.Source: B Data Generation Configuration · p. 22
• Each FineWeb-Edu passage is paired with several self-generated question–answer pairs, and each pair becomes a separate supervision target conditioned on that passage.Source: B Data Generation Configuration · p. 22

C Evaluation Configuration

• The backbone never sees the source passage or an in-context exemplar at evaluation; required knowledge comes through the generated LoRA adapter.Source: C Evaluation Configuration · p. 23
• The evaluation compares questions alone with questions wrapped in an answer-format instruction, which is used only at test time.Training prompts contain the question alone, making the instruction-wrapped protocol out of distribution for the generator.Source: C Evaluation Configuration · p. 23
• Both protocols use the backbone chat template and greedy decoding, with at most 256 new tokens.Source: C Evaluation Configuration · p. 23

D Re-implemented Baselines

• The comparison shares a frozen Qwen3-4B-Instruct backbone, training data, and generated adapter setup, while each baseline retains its architecture and objective.The controlled scale comparison varies generator scale while keeping each method’s objective and other settings fixed.Source: D Re-implemented Baselines · pp. 23–24
• Question-generation instructions require specific, answerable, fact-seeking questions that avoid overlap and vary their forms.The prompts also prohibit hallucinated information and discourage giving away too much of an answer in the question.Source: D Re-implemented Baselines · p. 24
• Answer-generation instructions request detailed answers grounded in the context, while evaluation prompts require outputting only the answer.The answer instructions call for comprehensive details and prohibit phrases such as “according to the context.”Source: D Re-implemented Baselines · pp. 24–25
• Doc-to-LoRA uses context distillation with a KL objective and an ℓ1 adapter penalty, whereas SHINE uses next-token prediction on answers.SHINE maps 2560-dimensional memory states to adapter slices, using 231 states per layer and discarding 1,536 surplus outputs.Source: D Re-implemented Baselines · p. 24

E Training and Inference Cost

• Inference measurements fix every query and answer at 32 tokens; the backbone has 4.02B parameters and a 144 KiB-per-token KV cache.These controls apply to all inference measurements, including multi-turn latency.Source: E Training and Inference Cost · p. 25
• CLIK training uses 3.9×10^4 GPU-hours (25.5 days), compared with 1.30×10^4 for Doc-to-LoRA, 1.36×10^4 for SHINE, and 1.03×10^4 for CLIK-B.The training set contains 9.67M passages truncated to 2048 tokens, with five QA pairs per passage over two epochs, totaling 96.7M examples.Source: E Training and Inference Cost · p. 25
• For internalized methods, the context prefill is counted once; SnapKV’s cache scoring is also counted once, but a new question selects a cache again.The new-question latency is the reused-state time-to-first-token for internalized methods, versus internalization time plus time-to-first-token for SnapKV.Source: E Training and Inference Cost · p. 25
• Inference memory accounting includes backbone and generator or compressor weights, per-context states, KV cache, workspace, and transient internalization activations.The appendix also lists architecture-specific settings for the re-implemented baselines.Source: E Training and Inference Cost · p. 25, Table 14 · p. 26

F Construction of the Counterfactual Probes

• Paired counterfactual documents differ only in one answer-bearing entity, number, or date replaced by distinct nonce values, while the question stays fixed.Each document is internalized separately, and both adapters receive the identical question.Source: F Construction of the Counterfactual… · p. 26
• Novel-fact probes use short passages about fictional entities and verify that the base model answers their fact-seeking questions at chance level.Source: F Construction of the Counterfactual… · p. 26
• Strong-prior-conflict probes edit a passage to contradict a fact the base model answers correctly, then distinguish answers matching the edited value from those matching its prior.Source: F Construction of the Counterfactual… · p. 26
• Wrong-context swaps pair each question with a random other passage’s adapter, while unrelated-query tests compare general-knowledge accuracy under a random passage adapter with base accuracy.Source: F Construction of the Counterfactual… · p. 26

G Frequently Asked Questions

• CLIK generates adapter weights by letting each weight token cross-attend to the full context, unlike SHINE’s generator, which reads a fixed-length context-compressing token sequence.The passage identifies this full-context representation as the change that raises CLIK accuracy over SHINE.Source: G Frequently Asked Questions · p. 26
• CLIK-B shares gate and up factors across contexts and uses per-context Bdown, a split that lowers accuracy outside question answering; the compared Doc-to-LoRA instead generates all three factors.CLIK-B is retained as a compact variant for question answering, while the published down-projection-only Doc-to-LoRA follows context poorly.Source: G Frequently Asked Questions · pp. 26–27
• The synthesized training QA pairs do not overlap evaluation questions and answers, and the FineWeb training passages do not overlap evaluation contexts.The cited comparison distinguishes this paper’s training mixture from the Doc-to-LoRA paper’s mixture, which included benchmark training splits.Source: G Frequently Asked Questions · p. 27
• SnapKV decodes at 9.0k tokens per second versus CLIK’s 8.3k, but it selects a cache again for each new question while CLIK reuses its adapter.The stated new-question latency includes internalization plus time-to-first-token for SnapKV, but only time-to-first-token for CLIK.Source: G Frequently Asked Questions · p. 27
• All baselines use the same long-answer training pairs and scoring protocol, and short-answer evaluations add a format instruction at test time.The training data prompt requests detailed, comprehensive answers; the short-answer instruction is not included during training.Source: G Frequently Asked Questions · p. 27
• Without the short-answer instruction, CLIK’s precision and recall stay close to Base + Context; with it, CLIK remains behind on precision.Under the short-answer instruction, precision mainly measures whether the output follows that instruction.Source: G Frequently Asked Questions · p. 27

• CLIK agrees with the in-context reference on Q1–Q4 but answers Q5 incorrectly, while the closed-book Base often adds unsupported or incorrect details.The passage is already largely familiar to the backbone, so this comparison illustrates answer behavior rather than knowledge acquisition.Source: H Qualitative Comparison · p. 27
• The closed-book Base answers several questions correctly from prior knowledge but adds spurious or incorrect details, including in Q2.Source: H Qualitative Comparison · p. 27
• CLIK's answer to Q5 is “Super Bowl 50” rather than the ground-truth “Super Bowl.”Source: H Qualitative Comparison · p. 27

I Multi-turn Dialogue Example

• On the separate SQuAD comparison, CLIK matches the in-context reference on Q1–Q4 but errs on Q5; the closed-book Base often adds unsupported or incorrect details.The authors describe this as output behavior rather than knowledge acquisition because the backbone already knows much of the passage.Source: I Multi-turn Dialogue Example · p. 29
• In the dynamic KV-cache example, the five outputs give Denver Broncos, Carolina Panthers, Levi's Stadium in Santa Clara, California, Denver Broncos, and Gold.Source: I Multi-turn Dialogue Example · p. 30
• Dynamic KV cache keeps the adapter generated from the initial context fixed and accumulates prior question–answer history in the model input.The base model's KV cache carries conversational state as turns proceed.Source: I Multi-turn Dialogue Example · pp. 28, 30
• In the dynamic-LoRA example, the five outputs include Santa Clara, California for Q3 and Denver Broncos for Q4, with gold for Q5.Source: I Multi-turn Dialogue Example · p. 30
• Dynamic LoRA discards the KV cache, keeps the model input to the current question, and regenerates an adapter from the context expanded with each latest question–answer pair.The parameter-generator context grows from c0 to c1 and onward across turns.Source: I Multi-turn Dialogue Example · pp. 28, 30