PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Evolving Context Parameterization for Large Language Models

Xiaobing Shi, Zherui Li, Yiming Jiang, Kun Wang, Yufei Guo

arXiv:2609.14168v1cs.CLcs.AI

TL;DR

Continual updates can leave valid and invalid information together in parameterized LLM states, a setting existing methods do not explicitly address. PLUME combines a global update representation with query-activated memory evidence, and improves average benchmark results on MUSE-Bench.

  • Problem

    Existing context parameterization methods typically assume static contexts and do not explicitly address evolving information validity, although continual updates require incorporating new information while preserving unaffected information.

    Source: §1 · p. 1

  • Method

    PLUME combines a global update representation with query-activated memory evidence and adaptively integrates their predictions during decoding.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    29.91%↑ ROUGE-L Recall (44.46 for D2L to 57.76) and 54.86%↑ LLM-as-a-Judge (34.49 to 53.41) averaged across MUSE-Bench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support coordinating global updates with selective evidence access to build reliable, reusable parameterized memory.

    Source: §7 · p. 9

    PLUME's performance is bounded by the underlying parameterization's lossy compression, which may not faithfully preserve long or information-dense contexts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Context parameterization enables large language models (LLMs) to internalize contexts into reusable model parameters, avoiding repeated processing across subsequent queries. However, existing methods typically assume static contexts and lack explicit mechanisms for distinguishing validity states under continual updates. To study this real-world scenario, we formalized the Memory Updating with Sequential Evolution (MUSE) task and constructed MUSE-bench to evaluate update incorporation and unaffected-information preservation. The resulting challenge requires preserving the global state while adjusting the contribution of memory evidence. Motivated by this, we proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding. Comprehensive evaluation on MUSE-bench demonstrated PLUME's effectiveness in sequential evolution settings, yielding relative improvements of 29.9% in average ROUGE-L Recall and 54.9% in LLM-as-a-Judge. Our codes are available at: https://github.com/xiaobingshi-LLM/PLUME.

1 Introduction

Continual updates make it difficult for parameterized contexts to incorporate new information while preserving unaffected knowledge and avoiding reliance on invalid information. The paper introduces MUSE and MUSE-Bench to study this setting and proposes PLUME, which combines a global update representation with query-specific memory evidence.

1 Introduction

  • Continual updates can leave information valid, supplemented, replaced, or invalidated, so parameterized states must incorporate updates while preserving unaffected information.Combining information with different validity states can lead to reliance on invalid information or inconsistent responses.Source: §1 · p. 1
  • MUSE-Bench evaluates continual updates in which valid and invalid information coexist, unlike prior work focused on conflicts between external context and model priors.It is constructed from five datasets covering question answering and reasoning; parameterizing the full context as one unified representation substantially degraded performance.Source: §1 · p. 2
  • PLUME adaptively integrates a global update representation with query-specific memory evidence, using memory evidence to target corrections when global predictions underweight valid information.Memory evidence can be unreliable under phrasing divergence and capture only partial information, so PLUME uses it as a complement to the global representation.Source: §1 · p. 2
  • 29.91%↑ ROUGE-L Recall (44.46 for D2L to 57.76) and 54.86%↑ LLM-as-a-Judge (34.49 to 53.41) averaged across MUSE-Bench.PLUME consistently outperformed baselines across models and task settings.Source: §1 · p. 2

2 Background

Context parameterization internalizes context into reusable model parameters, avoiding repeated context processing at inference. The section describes context distillation and Doc-to-LoRA as approaches to this goal, while noting their update-related costs and limits.

2 Background

  • Context parameterization internalizes context into model parameters decoupled from per-query inference, motivated by repeated-processing costs in long-context and retrieval-augmented methods.Those methods can also underutilize evidence away from the input boundaries.Source: §2 · p. 2
  • Context distillation trains a context-specific parameter state to reproduce a teacher distribution using queries, allowing later answers without explicitly supplying the context.It requires context-specific distillation data and iterative optimization for each new context, making repeated updates costly.Source: §2 · pp. 2–3
  • Doc-to-LoRA uses a hypernetwork to map context to a LoRA weight delta, generating a reusable context-specific adapter in one forward pass.It reduces parameterization cost for unseen contexts but does not explicitly model continual updates to existing parameterized states.Source: §2 · p. 3

3 MUSE-Bench

MUSE-Bench evaluates whether parameterized context states incorporate updates while preserving unaffected information. It formalizes sequential evolution, constructs context histories with update transitions, and assesses answer quality and update robustness.

3 MUSE-Bench

  • MUSE-Bench is designed to jointly evaluate incorporation of currently valid information and preservation of unaffected information under continual updates.It addresses a gap left by evaluations focused on static context internalization or targeted factual modifications.Source: §3 · p. 3

3.1 Motivation and Task Formulation

  • MUSE models sequential evolution as recursive context-state transitions and evaluates update-affected and unchanged queries after each transition, with the target model receiving only the query.Prior evaluations insufficiently assessed continual update incorporation alongside preservation of unaffected information.Source: §3.1 · p. 3

3.2 Benchmark Construction

  • MUSE-Bench contains five datasets, approximately 3,000 contexts, and 13,000 question-answer pairs, built by updating target and dependent facts while retaining the original context and questions.Candidate samples were generated with GPT-series models and underwent 26 rounds of quality verification, including checks that updated and unaffected answers remained supported.Source: §3.2 · pp. 3–4

3.3 Evaluation Protocol

  • The evaluation combines answer coverage, semantic equivalence, final-answer agreement, and locality to assess answer quality and preservation of unaffected information.Metrics are computed per query and aggregated within each dataset; GPT-5.6-Sol judges semantic equivalence and final-answer agreement.Source: §3.3 · p. 4

4 Method

PLUME combines a global update representation, query-activated memory evidence, and adaptive decoding for context parameterization under continual updates. Its components preserve broad context while allowing query-relevant local evidence to influence generation.

4 Method

  • PLUME combines a global update representation, query-activated memory evidence, and adaptive decoding for context parameterization under continual updates.The method is introduced as Parametric LoRA Updates with Memory Evidence.Source: §4 · p. 4

4.1 Global Update Representation

  • The global update representation combines the full-context parameter state with a weighted parameter shift from the pre-update state, retaining broad context while emphasizing changes.The full-context component supports queries with information unaffected by updates; the shift alone may omit background needed to answer them.Source: §4.1 · pp. 4–5

4.2 Query-Activated Memory Evidence

  • Query-activated memory evidence selects a temporally latest unit among candidates with near-highest lexical relevance scores, providing a local parameter view that complements compressed context.The hypernetwork receives the selected evidence text; query dependence arises through evidence selection.Source: §4.2 · p. 5

4.3 Adaptive Memory Evidence Decoding

  • Adaptive decoding increases memory evidence’s gating weight with its divergence from the global prediction, while retaining the global prediction as the decoding backbone.The gate saturates at λmax, bounding the evidence contribution; lower λt makes token selection more governed by the global prediction.Source: §4.3 · p. 5

5 Experiments

The experiments evaluate PLUME on MUSE-Bench, cross-task generalization, efficiency and information retention, component ablations, and representative cases. Across these settings, PLUME improves answer quality and locality while retaining the efficiency benefits of reusable parameterized contexts.

5.1 Experimental Setup

  • The evaluation uses five MUSE-Bench datasets and Qwen3 and Gemma2 backbones, with GSM8K and CRUXEval for cross-task generalization.Comparisons include Base Model variants, D2L, CD, AnyEdit, and CD and D2L oracles; metrics cover answer quality, locality, latency, and peak memory.Source: §5.1 · p. 6

5.2 Main Results

  • PLUME raises average ROUGE-L Recall and LLM-as-a-Judge over D2L from 44.46/34.49 to 57.76/53.41 across five datasets.On longer, more challenging datasets, improvements also persist; recall drops only 2.09 points from 3 to 20 updates.Source: §5.2 · p. 6

5.3 Efficiency and Information Retention

  • PLUME matched lightweight methods in update efficiency, added modest generation overhead, and achieved the strongest parameterized-method locality, exceeding D2L and CD by 10.25 and 20.30 points.Figure 3 reports locality, where higher scores indicate better preservation of unaffected information.Source: §5.3 · pp. 6–7, Figure 3 · p. 7

5.4 Generalization and Overall Comparison

  • On GSM8K, PLUME improves average ROUGE-L Recall and LLM-as-a-Judge from 23.18/16.42 to 45.39/41.47; on CRUXEval, from 39.48/13.63 to 62.31/50.62.Figure 4 shows balanced improvements across evaluation dimensions, alongside strong efficiency and information retention.Source: Table 2 · p. 7, §5.4 · p. 7, Figure 4 · p. 7

5.5 Ablation Study

  • PLUME’s average normalized ROUGE-L Recall of 81.36 exceeds 57.28 without the global update representation and 63.95 without memory evidence.For normalized Locality, PLUME scores 87.37, exceeding those ablations by 11.03 and 4.80 points; Figure 5 compares the components.Source: Figure 5 · p. 7, §5.5 · p. 7

5.6 Case Study

  • In a representative update case, D2L gives an incorrect answer by conflating validity states, whereas PLUME follows the currently valid update.The case includes valid, invalidated, and unaffected information in the same parameterized state.Source: Figure 6 · p. 8, §5.6 · p. 8

6 Related Work

Prior work includes methods for reusing internalized contexts and for editing model knowledge, but those lines do not address validity changes within parameterized contexts. MUSE-Bench and PLUME target cases where valid and invalidated information is jointly internalized in model parameters.

Context Internalization

  • Context distillation and D2L internalize contexts into reusable parameter representations, but assume static, internally consistent contexts.RAG and long-context models instead supply context at inference time, incurring repeated overhead.Source: §6 · p. 8

Knowledge Editing and Continual Updates

  • Knowledge editing changes model facts without full retraining, while prior sequential-editing work examines interference and preservation of prior knowledge.This paper instead studies continual updates to parameterized contexts with factual dependencies and unaffected information.Source: §6 · p. 8

Knowledge Conflicts

  • Prior conflict studies examine parametric knowledge versus external context or conflicts among retrieved contexts; here, valid and invalidated information is already jointly internalized in parameters.MUSE-Bench and PLUME are designed for this setting, without direct access to the original context.Source: §6 · p. 8

7 Conclusion

The paper introduces MUSE and MUSE-Bench for studying context parameterization under continual updates, evaluating update incorporation alongside preservation of unaffected knowledge. It presents training-free PLUME, which coordinates global updates and query-activated memory evidence, and reports gains across datasets and backbones.

7 Conclusion

  • MUSE and MUSE-Bench jointly evaluate incorporating updates and preserving unaffected knowledge in continual context parameterization.The benchmark frames the setting addressed by the paper’s training-free PLUME method.Source: §7 · p. 8
  • PLUME combines a global update representation with query-activated memory evidence through adaptive decoding, with ablations supporting their complementary roles.The findings support coordinating global updates with selective evidence access for reliable, reusable parameterized memory.Source: §7 · pp. 8–9

Limitations

PLUME’s limitations arise from its reliance on D2L’s context compression and its training-free design. The paper also reports how LLMs were used and identifies the model backbones and hypernetworks used in evaluation.

Limitations

  • D2L’s lossy context parameterization may fail to preserve detail in long, information-dense contexts, bounding PLUME’s performance by the underlying representation’s fidelity and capacity.Because PLUME is training-free, it also cannot directly improve the hypernetwork’s ability to distinguish valid updates from invalidated information.Source: Limitations · p. 9

A LLM Usage Statement

  • LLMs assisted with manuscript editing, data construction, quality verification, and automatic evaluation, but not with core research ideas or methodology design.Source: A LLM Usage Statement · p. 12

B.1 Models

  • Qwen3-4B-Instruct-2507 was the primary backbone; Gemma-2-2B-it provided a smaller, distinct-family robustness check, and both corresponding pretrained D2L hypernetworks remained frozen during evaluation.Source: §B.1 · p. 12

B.2 Benchmark and Datasets

MUSE-Bench combines five context-update datasets, with two additional reasoning datasets used to test generalization beyond factual question answering. The evaluation compares parameterized methods with context-free, full-context, and upper-bound references.

B.2 Benchmark and Datasets

  • MUSE-Bench contains 2,944 contexts and 12,932 QA pairs, with a weighted mean full-history length of 3,811.9 tokens across its five datasets.The reported per-dataset averages range from 328.3 tokens for SQuAD to 18,539.8 for 2WikiMultihopQA.Source: §B.2 · pp. 12, 14, Table 3 · p. 14
  • The five MUSE-Bench datasets cover factual QA and reasoning; GSM8K and CRUXEval add mathematical and code-reasoning tasks under the same context-update evaluation protocol.The annotated updates preserve original context and question structure while modifying answer-relevant information and dependent facts.Source: §B.2 · pp. 12–13

B.3 Baselines

  • The baselines include a query-only model, a full-context oracle, D2L context parameterization, supervised Context Distillation, and query-specific or gold-context oracle references.The oracle references use query-specific information unavailable to non-oracle methods, so they are upper bounds rather than directly comparable deployment settings.Source: §B.3 · pp. 14–15
  • AnyEdit applies AlphaEdit-ARE parameter edits to MLP down-projection layers 4–8, processing context-history chunks of at most 8,192 tokens.The setup uses 50-token edit windows without overlap and 25 gradient steps per window.Source: §B.3 · p. 15

B.4 Evaluation

The evaluation combines answer-quality metrics with locality and efficiency measures, and specifies aggregation, implementation, and inference choices. It also defines exact LoRA composition and PLUME’s memory-evidence settings.

B.4 Evaluation

  • Across datasets, the reported overall scores use a macro-average of per-dataset metrics so datasets with more QA pairs do not dominate.Source: §B.4 · p. 16
  • ROUGE-L Recall is the primary lexical metric because fluent model answers are often longer than span-style references, while Precision and F1 provide complementary diagnostics.Recall measures reference-answer coverage; Precision measures how much of the output is reference-supported.Source: §B.4 · p. 16
  • Locality is mean ROUGE-L Recall on queries unaffected by an update; its SQuAD evaluation uses 704 unaffected queries from 50 contexts.A higher score indicates better preservation of answers supported by unchanged facts.Source: §B.4 · p. 16
  • LLM-as-a-Judge scores an answer as 1 only when it passes both semantic-equivalence and final-answer-agreement rubrics; otherwise, it scores 0.Source: §B.4 · pp. 16–17

B.5.1 Exact LoRA Composition and Rank

  • The LoRA factors for global composition are concatenated exactly, without dense-weight merging or recompression; two rank-8 inputs yield factorization width 16.The implemented width is not necessarily the effective matrix’s algebraic rank.Source: §B.5.1 · p. 17

C Hyperparameters

  • Contexts exceeding 8,192 tokens are divided into near-equal contiguous chunks, and reported D2L parameterization uses Iterative mode with rank-8 chunk adapters.PLUME uses the query to activate memory evidence, but does not pass the query to the D2L hypernetwork.Source: C Hyperparameters · p. 17
  • PLUME separately parameterizes the old and full contexts, constructs a global update adapter, activates one memory unit using relevance and a recency margin, and adaptively fuses predictions.The reported hyperparameters include α = 1, β = 0.75, δ = 0.5, λmax = 1, and τ = 0.3.Source: C Hyperparameters · pp. 17–18

D More Experiments

  • The supplementary experiments discuss PLUME’s hyperparameter sensitivity and computational efficiency; experiments ran on two NVIDIA A800 GPUs using bfloat16.Source: §B.5 · p. 17, D More Experiments · p. 18

D.1 Additional Analysis of the Main Results

PLUME’s reported advantages span answer quality, locality, update and generation latency, and memory, with component analysis assigning complementary roles to its global and query-activated representations. Additional analyses report performance across accumulated updates and broader task formats, alongside candidate-rank and normalization analyses.

D.1 Additional Analysis of the Main Results

  • ROUGE-L Recall was 79.27 after 3 updates and 77.18 after 20, a gradual decline of 2.09 points as updates accumulated.The intermediate reported values are 78.72, 78.05, and 77.74 after 5, 10, and 15 updates, respectively.Source: §D.1 · p. 18, Figure 12 · p. 19
  • PLUME’s gains on GSM8K and CRUXEval extend beyond extractive or document-QA formats to broader continual context-update scenarios.The compared tasks involve multi-step arithmetic and deterministic program reasoning, respectively.Source: §D.1 · pp. 18–19
  • PLUME’s advantage spans ROUGE-L Recall, LLM-as-a-Judge, update and generation latency, and Locality, with global state and local evidence serving complementary roles.The global update representation preserves the full context state while emphasizing the latest change; query-activated evidence provides a local view that reduces interference from irrelevant information.Source: §D.1 · p. 19
  • The radar chart normalizes dataset-specific axes separately and orients metrics so larger values are better.The axes comprise answer-quality and Locality metrics alongside latency and memory metrics.Source: §D.1 · p. 19
  • Token-level ROUGE-L Recall rises sharply from top-1 to top-5 candidate tokens, then begins to saturate through top-10.The supplied figure caption describes this candidate-rank pattern without giving numerical values.Source: Figure 13 · p. 19
  • 81.36 was PLUME’s average normalized ROUGE-L Recall, compared with 57.28 for the memory-only and 63.95 for the global-only variants.Scores are normalized against the Base Model w/ Context (oracle) on the same dataset and backbone; the reported average uses two normalized Recall scores.Source: §D.1 · p. 19

D.2 Candidate-Rank Analysis

The analyses examine candidate ranking, parameter sensitivity, efficiency measurement, and the task and annotation rules for sequential updates. Together, they specify how the benchmark evaluates revision adoption while retaining unaffected information.

D.2 Candidate-Rank Analysis

  • Token-level ROUGE-L Recall rises from 45.90% at top-1 to 71.79% at top-5, then increases more gradually to 75.86% at top-10.This pattern indicates that much of the currently valid answer remains among high-ranked candidates but is often outranked by interfering candidates.Source: §D.2 · p. 19

D.3 Sensitivity to β

  • ROUGE-L Recall and Locality rise from 71.94 and 78.63 at β = 0.125 to 80.65 and 84.24 at β = 0.75, then fall to 70.62 and 75.48 at β = 3.The selected coefficient strengthens the latest-update direction within the full-context representation rather than replacing it with query-local evidence.Source: §D.3 · pp. 19–20, Figure 14 · p. 20

D.4 Sensitivity to τ

  • At τ = 0.3, ROUGE-L Recall reaches 80.65 and Locality reaches 84.24; Recall is lower at τ = 0.2, τ = 0.4, and τ = 1.The neighboring Recall values are 80.10 at τ = 0.2 and 80.24 at τ = 0.4; at τ = 1, Recall/Locality are 78.77/81.26.Source: §D.4 · p. 20, Figure 15 · p. 20

D.5 Sensitivity to λmax

  • At λmax = 1, Recall and Locality reach 80.65 and 84.24; at λmax = 5, they fall to 79.21 and 82.52.The authors report that allowing local evidence to dominate does not improve the model and erodes preservation supplied by the global context state.Source: §D.5 · p. 20, Figure 16 · p. 20

D.6 Efficiency Measurement

  • The efficiency protocol separates context-update time and memory from query-generation time and memory, measuring each phase independently.Tables 7 and 8 report update and generation efficiency on SQuAD and 2Wiki; lower latency and memory values are better.Source: §D.6 · pp. 20–21, Table 7 · p. 21, Table 8 · p. 21

E Data Annotation Guidelines

  • MUSE annotation treats a context and all its associated questions together, retaining chronological history while later information supersedes only overlapping earlier details.Information outside the update scope remains valid.Source: E Data Annotation Guidelines · p. 22, §E.2 · p. 23

E.1 Task Semantics

  • MUSE separates queries whose answers change under an update from those that remain valid, evaluating both against the latest evidence in the full history.Success requires adopting revised information for Qupd while preserving unaffected answers for Qkeep.Source: §E.1 · p. 22

E.2 MUSE-Bench

  • Benchmark construction preserves the original context, appends a fixed update notice, and revises coupled evidence so obsolete answers are no longer supported while unaffected answers remain valid.Candidate examples are checked for answer support, obsolete evidence paths, dependent-fact consistency, and preservation of unaffected answers.Source: §E.2 · pp. 23–24

E.3 Human Evaluation of Main Results

On a sampled evaluation subset, the automatic metrics were close to human assessment, with LLM-as-a-Judge more closely aligned than ROUGE-L Recall. The accompanying material specifies cross-task data construction, PLUME’s inference workflow, the authors’ interpretation and future directions, and the judge prompts.

E.3 Human Evaluation of Main Results

  • LLM-as-a-Judge scored 72.86 versus human evaluation’s 73.22, a 0.36-point difference; ROUGE-L Recall scored 78.17, 4.95 points from human evaluation.These scores are on a 0–100 scale and were obtained from 50 contexts containing 1,037 question–answer pairs.Source: §E.3 · p. 24

E.4 Cross-Task Data Construction

  • Cross-task data construction changes numerical conditions for GSM8K and program state or computation for CRUXEval while preserving each task’s reasoning format.Revised problems are recomputed for consistency, and revised CRUXEval programs must be executable with reference outputs derived from the revised program and input.Source: §E.4 · pp. 24–25

F PLUME Algorithm

  • PLUME constructs a global update LoRA once per context history, then activates query-relevant memory evidence and adaptively combines the two token distributions during decoding.The workflow selects the latest unit within a lexical-relevance margin and weights evidence predictions according to predictive divergence.Source: F PLUME Algorithm · p. 25, I Prompts · p. 26

G Discussion

  • The discussion interprets PLUME as global-first: the full-context representation supplies a stable state, while activated evidence offers targeted correction when query predictions disagree.Candidate-rank analysis suggests valid answers may remain among high-probability candidates despite not ranking first; global amplification and adaptive decoding address complementary parts of that competition.Source: G Discussion · p. 25

H Future Work

  • Future work will explore context parameterization beyond D2L, including reducing information loss in long contexts and across successive updates.The authors also propose combining PLUME’s training-free mechanisms with update-aware objectives for hypernetwork training and evidence routing.Source: H Future Work · p. 25

I Prompts

  • The prompts define D2 as semantic equivalence with required information preserved, and D3 as agreement of the committed final answer including required qualifiers and relations.Figures 17 and 18 show the respective prompts.Source: I Prompts · p. 25, Figure 17 · p. 27, Figure 18 · p. 28
Loading 2609.14168v1…