PaperlayerView exact revision on arXiv

Source-linked AI summary

Cortex: Consolidating Context into Parametric Memory

Jintian Zhang, Feng He, Xinnian Liang, Ziwen Xu, Haoming Xu, Haojing Huang, Shuangzhi Wu, Ningyu Zhang

arXiv:2610.20399v1cs.CLcs.AIcs.CVcs.LGcs.MA

TL;DR

Context can improve language-model performance, but gains may be lost after context removal; this motivates consolidating them while preserving non-target capabilities. Cortex anticipates context use with prospective queries and distills context-conditioned behavior into mergeable FFN updates. Across evaluated settings it lies on the empirical consolidation–retention Pareto frontier in most cases, with 88.9% CR and 99.8% RR after 50 sequential factual edits on a 14B model.

  • Problem

    Context can improve model performance, but those gains do not automatically persist after the supporting context is removed.

    Source: §1 · p. 2

  • Method

    Cortex generates prospective queries to anticipate context use and distills context-conditioned outputs into mergeable low-rank or structural FFN updates.

    Source: §1 · p. 3, §7 · p. 20

  • Results

    Cortex lies on the empirical CR–RR Pareto frontier in most evaluated settings; after 50 sequential factual edits on a 14B model, Cortex-Structural reaches 88.9% CR and 99.8% RR.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Memory Consolidation provides a shared objective for context-specified editing, skill learning, steering, and suppression while requiring non-target capability preservation.

    Source: §1 · pp. 2–3

    Folding structural expansion into a standard FFN simplifies deployment but does not reduce parameter count; pruning or distillation is needed to reduce deployed capacity.

    Source: §6.3 · p. 20

Abstract

from arXiv · show

Context can improve large language model performance, but these gains may be lost after context removal. Inspired by biological systems consolidation, we formulate Memory Consolidation (MC) as internalizing these gains while preserving non-target capabilities, connecting context-specified editing, skill learning, and behavioral steering under a common objective. We introduce a five-task-family testbed measuring consolidation and retention under single, batch, and sequential updates. We propose Cortex, which generates prospective queries to anticipate uses of context and distills context-conditioned outputs into mergeable low-rank or structural feed-forward network updates. Across 4B--120B dense and mixture-of-experts backbones, including both text-only and multimodal models, Cortex lies on the empirical consolidation--retention Pareto frontier in most evaluated settings. After 50 sequential factual edits on a 14B model, Cortex-Structural recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance.

1 Introduction

Context can improve model behavior, but those gains do not automatically persist when supporting context is removed. The paper frames Memory Consolidation as internalizing those gains while preserving non-target capabilities, then presents Cortex and a testbed for evaluating that objective.

1 Introduction

  • In-context gains can disappear when context is removed, while external-memory approaches depend on retrieving and supplying relevant content at inference.The paper motivates parameter updates as a complementary way to encode useful experience in parametric memory.Source: §1 · p. 2
  • Memory Consolidation internalizes context-induced gains while preserving initial performance outside the intended change, and sequential updates must retain earlier valid learning.The formulation unifies context-specified changes in knowledge, skills, and behavior under a persistence objective.Source: §1 · p. 2
  • The testbed covers five task families and measures consolidation and retention for single, batch, and sequential updates.It spans agent experience consolidation, novel API learning, persona steering, knowledge suppression, and factual editing; CR measures recovered context-induced gain and RR measures retained original performance on non-target queries.Source: §1 · pp. 2–3
  • Cortex generates prospective queries for anticipated context use and uses context-conditioned responses as learning targets without accessing held-out evaluation data.A context-conditioned teacher supervises a context-free student through output-distribution KL and hidden-state MSE alignment.Source: §1 · pp. 2–3
  • Cortex confines updates to FFNs, using low-rank Synaptic updates or Structural additions of nonlinear feature channels while preserving existing weight blocks.For pretrained MoEs, shared expansion gives every token a direct path through added parameters while router parameters remain fixed; both variants fold into standard FFN/MoE layers.Source: §1 · p. 3
  • 88.9% CR and 99.8% RR after 50 sequential factual edits on a 14B model; Cortex-Structural exceeds AlphaEdit by 66.2 and 3.8 percentage points, respectively.Across 4B–120B dense and MoE backbones, Cortex lies on the empirical CR–RR Pareto frontier in most evaluated settings. Ablations report improved CR and RR from FFN-only updates, reduced forgetting from structural expansion even without replay, higher CR from shared MoE expansion, and higher RR with modestly reduced CR from hidden-state alignment.Source: §1 · p. 3

2 Problem Formulation

The formulation distinguishes context-induced memory failures from capability failures and specifies what must change, what must remain stable, and how those outcomes are assessed.

2 Problem Formulation

  • Target queries Q+_C should change according to context C, while retain queries Q-_C should remain unchanged; the sets may overlap under different scoring functions.The formulation describes performance with or without context using a higher-is-better scoring function.Source: §2 · p. 3
  • A substantially positive context gap marks a memory failure: performance benefits from context but loses that useful effect when context is removed.A near-zero gap when context-free performance is poor may instead reflect a capability failure; memory consolidation targets memory failures.Source: §2 · p. 3
  • Memory consolidation updates model M using non-trivial context C to recover context-induced gains without C while preserving non-target performance.Evaluation excludes its queries and answers from rehearsal construction and gradient-based training; sequential evaluation covers still-valid memories after each step.Source: §2 · p. 4
  • CR measures recovered context-induced gain and RR measures retained baseline performance; either can exceed one, and interpretation requires a substantial gap and positive retain baseline.CR = 1 denotes full measured-gain recovery, while RR = 1 denotes preservation of the baseline retain score.Source: §2 · p. 4

3 Cortex: Consolidating Context into Parametric Memory

Cortex targets recovery of context-induced gains and preservation of non-target capabilities by combining prospective rehearsal, FFN updates, and context-to-memory distillation.

3 Cortex: Consolidating Context into Parametric Memory

  • Cortex supports single, batch, and sequential consolidation, using the frozen pre-update model for rehearsal and teacher supervision while training only newly introduced FFN parameters.Earlier writes remain frozen during each sequential update.Source: §3 · p. 4, §3.1 · p. 5

3.1 What to Consolidate: Prospective Rehearsal

  • Prospective rehearsal uses queries that anticipate future uses of context and the model’s context-conditioned responses as learning targets, without using held-out evaluation data.For agent tasks, training-environment queries and successful offline trajectories replace model-generated queries.Source: §3.1 · p. 4

3.2 Where to Consolidate: FFN Memory and Plasticity

  • Synaptic writes add rank-at-most-r updates to frozen existing projections, while Structural writes add nonlinear channels and keep original weight blocks fixed.When k = r in a dense SwiGLU layer, Structural trains fewer parameters but increases deployed intermediate width.Source: §3.2 · pp. 5–6
  • For MoEs, Cortex’s shared Structural branch gives every token a path through the added parameters and can fold into every expert when routing weights are normalized.Router parameters remain frozen, while upstream representation changes can still alter routing decisions.Source: §3.2 · p. 6

3.3 How to Write: Context-to-Memory Distillation

  • Context-to-memory distillation trains a student without source context to match a frozen teacher using output-distribution KL and post-FFN representation MSE on aligned positions.Optional query replay adds earlier queries with responses regenerated by the preceding model, constraining changes to previously consolidated behavior.Source: §3.3 · pp. 6–7

4 Memory Consolidation Testbed

The Memory Consolidation Testbed measures recovery of context-induced gains and preservation of performance outside the intended change across five task families and multiple update settings.

4 Memory Consolidation Testbed

  • The testbed spans procedural, behavioral, and declarative memories; addition, revision, and suppression; and local or global intended effects.Its five families are agent experience consolidation, novel API learning, persona steering, knowledge suppression, and factual editing.Source: §4 · p. 7, Figure 4 · p. 8

4.1 Design Principles and Evaluation Protocol

  • Contexts are manually constructed to elicit substantial context gaps, while held-out evaluation queries and answers are excluded from context construction.Single, batch, and sequential protocols differ in how contexts are incorporated; past source contexts cannot be revisited during sequential updates.Source: §4.1 · p. 8

4.2 Task Instantiations

  • Agent experience consolidation tests whether summaries of successful strategies improve later decisions, while novel API learning tests package use after documentation removal.Agent evaluation uses disjoint episodes; API performance is measured on 150 executable problems spanning API use and debugging.Source: §4.2 · p. 9
  • Persona steering measures persistence of behavioral tendencies after instruction removal; Power-seeking fluency is not broad capability retention, and Psychopathy has no reported RR.Power-seeking concept adherence and fluency are scored separately.Source: §4.2 · p. 10
  • Knowledge suppression tests entity-specific answering restrictions using target-answer overlap, but reduced overlap on its probes does not establish parameter-level erasure.Retention is measured on neighbor probes and BBH, with supplementary MMLU and SimpleQA results reported separately.Source: §4.2 · p. 10
  • Factual editing evaluates efficacy, paraphrase generalization, and portability, while locality and broader benchmarks contribute to retention.Batch and continual editing additionally assess general-capability retention on MMLU, SimpleQA, and a backbone-specific mathematics benchmark.Source: §4.2 · p. 10

5 Experiments

Experiments span five task families and 4B–120B dense and MoE backbones, with single, batch, and sequential updates. Cortex reaches the empirical CR–RR Pareto frontier across all 15 Figure 5 panels, and Structural recovers 88.9% CR after 50 edits while retaining 99.8% RR.

5.1 Single and Batch Consolidation

  • At least one Cortex variant lies on the empirical CR–RR Pareto frontier in all 15 Figure 5 panels.The panels cover task–backbone settings; CR is paired with task-specific RR measures.Source: §5.1 · p. 10, Figure 5 · p. 11
  • Cortex exceeds distillation and specialized editing baselines in Qwen batch settings, while hypernetwork and test-time-training baselines show weaker batch recovery or low CR despite high RR.For Qwen2.5 batch code, Cortex reaches 66–74% CR with 98–99% RR; on Qwen batch editing, its 80–91% CR exceeds the strongest FT-M, MEMIT, or AlphaEdit result on each backbone.Source: §5.1 · pp. 11–12

5.2 Continual Consolidation

  • After 50 factual edits, Structural reaches 88.9% CR and 99.8% RR, versus 28.5% CR for the strongest baseline by final CR, ROME.Structural adds 0.74M parameters per edit; Synaptic reaches 80.6% CR and 97.2% RR with 2.73M new parameters per edit.Source: Figure 6 · p. 13, §5.2 · p. 13
  • Structural and Synaptic lose 2.4 and 5.2 CR points from immediate post-edit to final checkpoint, while retaining 97.2% and 92.6% mean CR for the first ten facts after 40 later edits.Structural also has higher portability than Synaptic at the final checkpoint, 49.4% versus 30.7%, despite nearly identical efficacy.Source: §5.2 · pp. 13–14

5.3 Ablations and Analysis

  • LoRA controls using Cortex data do not account for Structural’s gains, and matching the teacher distribution outperforms fitting generated answers alone in one comparison.On Qwen2.5-14B batch code, Structural exceeds LoRA-KL by 23.0 CR points; on GPT-OSS-120B batch code, LoRA-KL exceeds LoRA-CE by 16.6 points.Source: §5.3 · p. 15
  • FFN-only updates improve CR and RR over attention-adapting LoRA, while replay keeps both variants above 96% accuracy after 25 edits; without replay, Structural retains 40.5% and Synaptic 8.0%.The no-replay variants had similar initial acquisition, showing that later updates can interfere with earlier predictions even when earlier parameters are frozen.Source: §5.3 · p. 15
  • Hidden-state alignment’s retention benefit depends on update type, while shared MoE expansions improve CR per trainable weight and give every token a direct path through added parameters.At matched total weight budget, sharing improves CR by 4.6 points on ALFWorld and 10.3 on Code, with RR about one point lower; 32 of 768 layer–expert branches receive less than 1% of mean exposure in every seed on ALFWorld.Source: §5.3 · pp. 15–16
  • For the tested continual sequence, Structural k = 1 attains its highest mean CR at the smallest swept capacity: 88.9% CR with 0.74M added parameters per edit.Increasing trainable parameters does not consistently improve performance under the same optimization recipe.Source: §5.3 · p. 16

6 Related Work

Memory Consolidation concerns whether context-induced behavioral changes persist after context removal while unrelated capabilities are retained. Cortex is distinguished from external-memory access, task-specific adaptation, and related update methods by how it derives learning targets, where it writes, and how it updates.

6.1 Memory Consolidation and Neighboring Problems

  • Parametric memory alone is not consolidation: experience must persist as a change available through the model’s own computation in future interactions.Retrieval-augmented generation and agent memory instead access or store external information, while context compression reduces the cost of presenting it.Source: §6.1 · p. 16
  • MC treats context-specified editing, suppression, steering, and skill learning as one problem requiring persistent gains after context removal and retention of unrelated behavior.Unlike task-specific methods, MC leaves construction of learning data from context to the method.Source: §6.1 · pp. 16–17

6.2 Cortex and Related Approaches

  • Cortex uses prospective queries and context-conditioned responses to rehearse anticipated uses, then distills behavior into FFN updates that fold into standard layers without a separate recall mechanism.Its structural MoE extension is shared across tokens with router parameters frozen; its training trajectories are fixed offline trajectories from the context-conditioned pre-update model.Source: §6.2 · pp. 17–18
  • Cortex’s prospective rehearsal extends beyond source-text reproduction, while its FFN-restricted output and representation matching shares transfer objectives with DCD but uses truncated forward KL and post-FFN residual MSE.Cortex optimizes each context’s update directly with rehearsal data and teacher signals, unlike learned update generators that amortize adaptation through shared training.Source: §6.2 · pp. 17–18

6.3 Continual Updates and Memory Architectures

  • In continual editing, Cortex replays historical queries and uses pre-update and initial-model teachers to anchor accumulated memories and capabilities outside intended changes.Freezing earlier parameters alone did not preserve earlier edits in the reported setting, whereas query replay substantially improved retention.Source: §6.3 · p. 18
  • The paper frames deployment-specific learning as complementary to centralized training, while leaving extraction and validation of experience from interaction as a separate problem.Whether consolidation helps with indirect associations remains open; persistent storage does not guarantee successful recall or composition.Source: §6.3 · p. 19
  • Persistent parametric memory motivates distributed integration, growable capacity, and native access, but folding Structural expansion into an FFN does not reduce its parameter count.Pruning or distillation, resource costs, experience validation, and control over retained or shared user-specific information remain open concerns.Source: §6.3 · pp. 19–20

7 Conclusion and Future Work

The paper concludes that Memory Consolidation unifies several context-specified behavioral changes, and that Cortex performs strongly across evaluated model scales and architectures. It also characterizes FFN updates as corrections to context-conditioned behavior and identifies broader continual-learning and efficiency questions for future work.

7 Conclusion and Future Work

  • Memory Consolidation formalizes internalizing context-induced gains while preserving non-target capabilities, unifying editing, skill learning, steering, and suppression.Cortex uses prospective rehearsal and mergeable FFN updates within a five-task testbed.Source: §7 · p. 20
  • Cortex lies on the empirical consolidation–retention Pareto frontier in most evaluated settings across model scales and architectures.The conclusion identifies cross-task continual and large-batch consolidation, pruning and merging structural expansions, and hypernetworks that predict structural parameters as future work.Source: §7 · p. 20

A FFN-local context correction and shared MoE expansion

  • Cortex learns an FFN correction for context removal by aligning post-FFN residual states, with exact matching characterized on finite sets of aligned computation states.The paper distinguishes this representation rationale from performance on unseen states, which depends on learned approximation and rehearsal coverage.Source: A FFN-local context correction and… · p. 30, §A.1 · p. 30

A.1 Context correction and hidden-state supervision

  • Intermediate-state alignment supervises contextual effects to be consolidated, while output KL supervises predictive behavior.The fixed-feature result identifies the closest representable correction as ∆P, where P = Φ†Φ, attained by Ds = ∆Φ†.Source: §A.1 · pp. 30–31

A.2 Finite-state realizability of Structural plasticity

The appendix characterizes when Structural expansions can interpolate finite sets of corrections, relates Synaptic updates to shared feature mappings, and analyzes shared MoE expansions’ gradient connectivity. It also points readers to the experimental protocols and task-level materials.

A.2 Finite-state realizability of Structural plasticity

  • A Structural expansion can interpolate arbitrary corrections on a finite set when its inputs satisfy the theorem’s nonzero and pairwise nonidentical, non-opposite conditions and its width is at least the number of states.The final-FFN corollary transfers exact matching to teacher residual states and logits under its stated architectural conditions; performance on unseen states depends on approximation and rehearsal coverage.Source: §A.2 · pp. 31–32

A.3 Relation to Synaptic plasticity

  • A shared rank-limited down-projection update exists exactly when the correction matrix is representable from the shared feature matrix, and the minimum-norm construction is L = ∆H†.This analyzes the down-only Synaptic subclass; Structural instead adds feature channels while retaining existing weight blocks.Source: §A.3 · pp. 32–33

A.4 Shared Structural expansion in MoE layers

  • A shared MoE expansion pools training signals into one parameter set without the multiplicative routing factor applied to independent expansions.This characterizes gradient connectivity, while the resulting consolidation performance is evaluated empirically; frozen routers may still produce different routes when upstream hidden states change.Source: §A.4 · p. 33

B Experimental Details and Additional Results

  • Appendix B documents single and batch protocols, continual-learning protocols and trajectories, controlled comparisons by research question, and locations of complete score tables and evaluation prompts.The materials are organized across Appendices B.1–B.3 and the respective task appendices.Source: B Experimental Details and Additional… · p. 33

B.1 Single and Batch Consolidation

The single- and batch-consolidation experiments vary task-specific backbones, training data, update modules, and baselines. Evaluation uses shared task-level protocols and reports consolidation and retention from task aggregates.

B.1 Single and Batch Consolidation

  • Single edits consolidate one fact, while batch edits combine 50 facts; API learning uses separate adapters per package or one model for all 15 packages.Persona settings use one adapter per persona; RWKU uses a joint inhibition context for ten entities, and agent consolidation uses experience contexts and successful offline trajectories.Source: §B.1 · p. 34
  • The evaluated backbones and baseline coverage are task-dependent, as summarized in Table 2.The listed backbone abbreviations include Qwen, Gemma, and GPT-OSS models.Source: Table 2 · p. 34, §B.1 · p. 34
  • For API learning and factual editing comparisons with SDCD and SELF-PARAM, Cortex uses 28 rehearsal pairs per package and one pair per fact.Continual consolidation uses 16 generic examples per update; single and batch consolidation additionally use four fixed generic examples for retention.Source: §B.1 · p. 34
  • The experiments vary training data, update location, and supervision signal as separate choices.Synaptic targets FFN projections, LoRA-All additionally targets attention projections, and Structural adds an FFN branch while freezing the original FFN; SFT and context-distillation controls replace the objective with CE or KL.Source: §B.1 · p. 34
  • Baseline hyperparameters and training duration are selected using held-out consolidation metrics, without using retention scores for model selection.Trainable baselines are retrained with the selected configuration before test evaluation; MemP instead uses fixed external memory and retrieval.Source: §B.1 · pp. 34–35
  • Task-level scores are aggregated before CR/RR normalization, with repeated attempts aggregated within each question or episode.Editing consolidation averages efficacy, rephrasing, and portability; locality contributes to retention, and retention datasets receive equal weight after scaling.Source: §B.1 · pp. 36, 38, Table 4 · p. 37

B.2 Continual Consolidation

The continual protocol updates a model in a fixed sequence of 50 factual edits, using a pre-update model to generate current rehearsal examples and distillation targets. The reported comparisons track acquisition, later preservation, and general-capability retention across updates.

B.2 Continual Consolidation

  • At each update, the frozen pre-update model generates current context-conditioned rehearsal examples and distillation targets, while earlier adapters remain active and frozen.Only the new adapter is optimized; evaluation uses accumulated adapters without source contexts or replay-buffer access.Source: §B.2 · p. 39
  • Question–answer screening rejects examples outside the edit target or explicitly referring to the supplied context, with up to ten retries before random selection.Replayed examples do not receive this additional screening.Source: §B.2 · p. 39
  • The continual evaluation measures editing facets after every update and general retention every five updates on fixed question subsets.The consolidation aggregate averages efficacy, rephrasing, and portability; retention averages locality, MMLU, SimpleQA, and GSM8K with equal dataset weight.Source: §B.2 · pp. 39–40, Table 6 · p. 40
  • Structural loses 2.4 CR points from acquisition to the final checkpoint, compared with 5.2 points for Synaptic, across the same 50 facts.The comparison uses the main configurations, Structural k = 1 and Synaptic r = 1; Table 7 summarizes both.Source: §B.2 · p. 41, Table 7 · p. 41
  • Fact-level matrices show immediate acquisition on the diagonal and follow each fact through later edits in its column.Figures 10 and 12 show the Cortex capacity trajectories; gray cells mark facts not yet edited.Source: §B.2 · p. 41, Figure 10 · p. 44, Figure 12 · p. 46

B.3 Ablations and Mechanistic Analysis

The ablation protocol varies rehearsal data, update location, loss, and parameter sharing across consolidation settings. Batch interventions use controlled training objectives and configurations, with evaluation scores aggregated before CR/RR normalization.

B.3 Ablations and Mechanistic Analysis

  • Controlled comparisons vary data, update location, loss, and parameter sharing in single, batch, and continual consolidation.Data controls use cached rehearsal examples in batch settings and online construction with query replay in continual settings.Source: §B.3 · p. 48
  • Batch training configurations use teacher-top-1,024 KL at temperature two, with CE disabled and MSE enabled or disabled by configuration.The notation o selects answer-prediction positions for KL and aligned answer-token hidden states for MSE; oi also includes aligned non-context input positions.Source: §B.3 · p. 48, Table 8 · p. 50
  • For top-Kv KL, the teacher distribution is normalized over its 1,024 highest-logit vocabulary entries, while the student distribution is normalized over the full vocabulary.Hidden-state MSE supervises aligned post-FFN states at all transformer layers.Source: §B.3 · p. 48
  • Structural branches use a zero-initialized down projection, while Synaptic and LoRA-All use default PEFT LoRA initialization with zero initial adapter output.Structural width k counts added nonlinear channels, not LoRA rank.Source: §B.3 · p. 48
  • Batch interventions use a cosine learning-rate schedule with 10% warmup, BF16, gradient checkpointing, and learning rate 5 × 10^-4.Microbatch size, data-parallel world size, sequence limit, epochs, and ZeRO stage vary by configuration.Source: §B.3 · p. 48, Table 8 · p. 50

B.3.1 RQ1: Rehearsal Data and Supervision

The ablations test how rehearsal data and supervision, as well as FFN-only versus attention-inclusive updates, affect consolidation and retention. Their findings include task-dependent benefits from the writing mechanism and a better continual consolidation–retention balance for restricted updates.

B.3.1 RQ1: Rehearsal Data and Supervision

  • The supervision controls compare LoRA-All with CE or KL on cached rehearsal examples in batch settings and online data construction with query replay in continual settings.The continual LoRA-All configuration uses rank four and generates its own rehearsal answers and teacher targets at each update.Source: §B.3.1 · p. 51
  • On continual editing, KL raises LoRA-All RR from 82.3% with CE to 87.0%, while portability falls from 33.8% at acquisition to 19.1% after all edits.The continual controls use online data construction and query replay; Figures 15 and 16 show editing and retention trajectories.Source: §B.3.1 · p. 51, Figure 15 · p. 52, Figure 16 · p. 53
  • On Qwen2.5 batch editing, LoRA-CE reaches 91.9% CR, versus 90.8% for Synaptic and 88.6% for Structural.For Qwen2.5 batch code, Structural exceeds LoRA-KL by 23.0 CR points.Source: §B.3.1 · p. 51, Figure 14 · p. 51

B.3.2 RQ2: FFN-Only Versus Attention-Inclusive Updates

  • The batch attention comparison holds rank and optimization fixed while changing adapted modules; LoRA-All adds attention projections to Synaptic FFN updates.At rank four, the configurations add 10.9117M and 17.2032M matrix weights per edit, respectively.Source: §B.3.2 · p. 54
  • Under continual KL, FFN-only updates reach 79.6% CR and 91.0% RR, compared with 75.8% CR and 87.0% RR when attention is also updated.The same ordering appears with KL+MSE and in the batch editing comparison.Source: §B.3.2 · p. 54, Figure 19 · p. 56, Figure 20 · p. 57

B.3.3 RQ3: Historical Query Replay

Historical rehearsal helps preserve edited knowledge across subsequent updates, while hidden-state alignment improves retention with effects on consolidation that depend on the update parameterization.

B.3.3 RQ3: Historical Query Replay

  • The replay comparison tests query replay, fixed-QA replay, and no historical replay while earlier adapters remain active and frozen during each new adapter’s training.Query replay regenerates answers to stored questions; fixed-QA replay retains answer strings while recomputing teacher logits and hidden states.Source: §B.3.3 · p. 58
  • With query replay, final CR is 88.9% for Structural and 80.6% for Synaptic, versus 45.9% and 6.6% without replay; fixed-QA reaches 88.1% and 81.3%.The comparison covers Structural k = 1 and Synaptic r = 1 under KL+MSE.Source: §B.3.3 · p. 58, Figure 21 · p. 59
  • After 25 subsequent edits, mean efficacy/rephrase accuracy on the first 25 facts is 96.1% for Structural and 97.6% for Synaptic with replay, versus 40.5% and 8.0% without it.Following the same facts at each age isolates changes to previously acquired knowledge; Structural retains more old knowledge than Synaptic without replay.Source: §B.3.3 · p. 58, Figure 22 · p. 60, Figure 24 · p. 61

B.3.4 RQ4: Hidden-State Alignment

  • In continual editing, adding MSE changes CR/RR by −5.0/+5.8 points for LoRA-All, −4.0/+2.7 for Synaptic, and +1.3/+3.6 for Structural.All three parameterizations gain retention, while Structural also improves consolidation; the batch results identify dataset-level contributors to aggregate changes.Source: §B.3.4 · p. 62

B.3.5 RQ5: Shared MoE Capacity and Training Exposure

Shared MoE expansions improve target performance at matched total expansion weights and use fewer added weights at matched active capacity. In the capacity sweep, the smallest update capacity in each family has the highest mean CR.

B.3.5 RQ5: Shared MoE Capacity and Training Exposure

  • The shared alternative sends every token through one added branch, whereas independent expansions attach separate branches to experts and only selected expansions contribute to each token.The comparison matches either total added matrix weights or active added width; budgets are per layer and exclude biases.Source: §B.3.5 · p. 64, Table 9 · p. 64
  • Training monitors channel displacement between window endpoints and cumulative valid-token routing exposure, normalized within each layer and seed.The heatmaps mark layer–expert pairs below 1% of their layer’s mean exposure in every seed; 32 of 768 pairs meet that criterion in the main ALFWorld heatmap.Source: §B.3.5 · pp. 64–65, Figure 30 · p. 66, Figure 31 · p. 67
  • At matched total expansion weights, Shared reaches mean target pass@1 of 90.05% on ALFWorld and 67.11% on Code, versus 86.07% and 62.48% for Unshared.At matched active expansion weights, Shared reaches 87.73% and 61.81%, respectively, using one eighth as many added matrix weights.Source: §B.3.5 · p. 65, Figure 28 · p. 65, Figure 29 · p. 66

B.3.6 RQ6: Update Capacity

  • Mean CR falls from 88.9% to 84.3% across Structural capacity endpoints and from 80.6% to 75.6% across Synaptic endpoints; each family’s smallest capacity has its highest mean CR.Structural also achieves higher CR and RR at nearby cross-family budgets.Source: §B.3.6 · p. 68, Figure 32 · p. 68

B.3.7 Additional Controls: Answer Source and Supervision Scope

The controls vary rehearsal-answer source and supervision scope. Self-generated answers outperform externally generated answers in the reported editing comparison, while aligned input-and-output supervision modestly raises CR and slightly lowers RR in continual Structural editing.

B.3.7 Additional Controls: Answer Source and Supervision Scope

  • Replacing self-generated answers with Gemini 3 Pro answers reduces mean efficacy from 89.5% to 21.4% and MMLU accuracy from 75.8% to 15.9% across training seeds.The comparison is GPT-OSS-20B Synaptic editing; both conditions obtain distillation targets by prefilling the same target backbone.Source: §B.3.7 · p. 69, Figure 33 · p. 69, Figure 34 · p. 70
  • In continual Structural k = 16 editing, aligned input-and-output supervision changes final CR/RR from 84.27/98.98% to 86.63/98.80% versus output-only supervision.KL and MSE both change jointly; the additional supervised positions are query tokens, excluding the teacher-only context.Source: §B.3.7 · p. 69, Figure 37 · p. 73, Figure 38 · p. 73

C Detailed Results of Single, Batch, and Continual Consolidation

The appendix indexes task- and setting-specific results and describes how CR/RR and table variability are reported across the detailed evaluations.

C Detailed Results of Single, Batch, and Continual Consolidation

  • The index links result tables to settings and backbones for persona steering, knowledge suppression, API learning, factual editing, and agent-experience consolidation.The listed settings include single, batch, and continual evaluations.Source: C Detailed Results of Single, Batch,… · p. 74
  • CR and RR use unrounded task-level scores and the corresponding Vanilla and Context references, with component scores averaged before normalization and equal weight across retention datasets.For factual editing, efficacy, generalization, and portability contribute to the target score, while locality contributes to retention.Source: C Detailed Results of Single, Batch,… · p. 74
  • The Code tables cover batch and single consolidation, and the ALFWorld and τ 2-Bench tables report agent-experience consolidation results.The Code captions specify target P@K and evaluation-generation counts; the τ 2-Bench caption states that each method is evaluated on 100 tasks.Source: Table 10 · p. 75, Table 11 · p. 76, Table 12 · p. 77
  • Separate tables report batch model editing, continual editing on 50 facts, and single editing for Qwen3, Qwen2.5, and GPT-OSS.The single-editing tables evaluate retention through Locality only.Source: Table 13 · p. 78, Table 14 · p. 79, Table 15 · p. 80
  • The remaining tables report personality-steering results for Qwen3 and Gemma3 and for Qwen2.5 and GPT-OSS, plus unlearning results.Their captions specify standard-deviation reporting and units for the personality-steering metrics.Source: Table 17 · p. 82, Table 18 · p. 83, Table 20 · p. 84

D Evaluation Prompts

This appendix describes evaluation prompts and procedures across tasks, including task interfaces, context supplied to baselines, response preparation, and scoring or grading rules. It also specifies unlearning and BBH prompt formats, including restricted-list answer checks and the omission of BBH’s few-shot chain-of-thought field.

D Evaluation Prompts

  • Prompt templates use runtime substitutions and literal boxed-format requirements, with roles stated separately and model-specific chat serialization omitted.Context instructions for context-based baselines are separate from evaluation questions.Source: D Evaluation Prompts · p. 85

D.1 Evaluation Size and Repeated Sampling

  • Each sampled Edit or SimpleQA answer receives one grading decision, while Power Seeking uses separate behavior and fluency grading calls.A repeat means one sampled answer per query or one seeded episode per interactive task, not a judge retry. Table 21 summarizes query and sampling counts; the accompanying text gives task-specific scopes, including 1,140 MMLU questions and 1,539 RWKU queries.Source: §D.1 · p. 85, Table 21 · p. 85

D.2.1 Edit: Answer Correctness; D.2.2 SimpleQA: Three-Way Factual Grading

  • Edit correctness requires the gold answer’s important information without contradictions, whereas SimpleQA adds a NOT_ATTEMPTED label for refusals, uncertainty, and overly vague answers.SimpleQA marks incorrect, incomplete, contradictory, or falsely augmented answers INCORRECT; Edit uses a two-way correct/incorrect rubric.Source: §D.2.1 · pp. 85–86, §D.2.2 · pp. 86–87

D.2.3 Power Seeking: Behavior and Fluency

  • Power Seeking is graded in independent calls for behavioral intent and fluency, with the behavioral rubric distinguishing unconstrained power seeking from ethically constrained, neutral, or rejecting responses.Behavior receives a 1–4 score, uninterpretable responses are capped at 2, and the quality-adjusted score is capped at 2 when fluency is zero; otherwise it equals the behavioral score.Source: §D.2.3 · pp. 87–88

D.2.4 Candidate Text Supplied to the Judge

  • Judge inputs preserve Edit answer text but extract SimpleQA answers by preferring boxed answers, then answer spans, then final-answer lines, and finally the last nonempty line.Recognized generation delimiters and terminal padding are handled during cleanup, and raw responses, judge messages, and outputs are retained for inspection.Source: §D.2.4 · p. 88

D.3 General-Capability Evaluation Templates; D.3.1 MMLU; D.3.2 GSM8K; D.3.3 AIME 2026; D.3.4 MATH500; D.3.5 SimpleQA; D.3.6 LiveCodeBench

  • These templates are user messages applied to the loaded checkpoint without reinserting the knowledge used to train it; task prompts specify step-by-step reasoning and boxed answer formats.Source: §D.3 · p. 88, §D.3.1 · p. 88, §D.3.2 · p. 89

D.4.1 Editing: Single and Batch

  • ZsRE evaluates each fact with four separate probes: efficacy and rephrasing use the edited answer, locality uses the unchanged answer, and portability uses the corresponding reference answer.Single- and batch-edit evaluations differ in which facts are internalized or supplied in context, not in the wording of these queries.Source: §D.4.1 · p. 90

D.4.2 Code-API: Single and Batch

  • Code-API evaluation uses stored dataset questions for 150 implementation and repair tasks, while reference implementations and executable tests are withheld from generation prompts and used for scoring.Single-package checkpoints are evaluated on that package’s questions; batch checkpoints are evaluated across packages.Source: §D.4.2 · pp. 90–91

D.4.3 Unlearning: Forget and Neighbor Probes

  • RWKU level-1 probes use a completion instruction and level-2 probes use a short-answer instruction; neighbor probes assess related-knowledge preservation separately from MMLU and SimpleQA retention.Both probe formats require the last response line to contain a boxed answer.Source: §D.4.3 · pp. 91–92

D.4.4 Power Seeking; D.4.5 Psychopathy

  • Power-seeking probes pass the target question verbatim while placing behavioral instructions separately; psychopathy probes request boxed yes/no answers scored against dataset behavior matches, not an LLM rubric.Source: §D.4.4 · p. 92, §D.4.5 · p. 92

D.5.1 ALFWorld

  • ALFWorld checkpoint evaluation begins with an environment observation and no system message, then appends each assistant response and user feedback; the separate baseline interface can add a task-instruction system message.If no action is extracted, the evaluator returns feedback indicating that no valid action was found.Source: §D.5.1 · pp. 92–93

D.5.2 τ 2-Bench

  • Tau 2-Bench agents receive a domain policy, tool schemas, and evolving user dialogue; the Harmony interface preserves assistant and tool-call history and separates tool calls from user-facing messages.The policy setting selects a basic or model-specific expanded instruction.Source: §D.5.2 · pp. 93–94

D.6.1 Editing

  • Editing context instructions do not replace dataset questions: Qwen2.5/GPT-OSS prioritize updated facts, while Qwen3 frames them as absolute truths and inserts relevant facts singly or in batches.The surrounding prompt template is fixed; inserted facts or documentation form the knowledge payload.Source: §D.6 · p. 94, §D.6.1 · pp. 94–95

D.6.2 Code-API

  • Code-API context uses one package document for the single-package oracle baseline or concatenates 15 package documents in loader order for batch evaluation, leaving the task question unchanged.Documents are separated by newlines.Source: §D.6.2 · p. 95

D.6.3 Steering

  • Steering baselines supply system prompts that favor power-seeking choices or a Machiavellian, self-interested persona with diminished empathy.The power-seeking prompt prioritizes power, control, leverage, or resources; the psychopathy prompt prioritizes self-interest over helping others.Source: §D.6.3 · pp. 95–96

D.6.4 Unlearning

  • RWKU unlearning baselines use model-specific system instructions to feign ignorance of listed people while directing models to retain normal capabilities on other topics.The Qwen3-4B-Thinking instruction explicitly says other-topic performance remains intact.Source: §D.6.4 · pp. 96–97
  • The unlearning protocol first retrieves the factual answer, then checks whether the question’s subject or answer matches a restricted-list entry, including aliases and indirect identifiers.The prompt frames this check as necessary before suppressing knowledge at the final output layer.Source: §D.6.4 · p. 98
  • A match requires the exact refusal “I don't have information about that person”; questions about related entities not on the list remain allowed.The prompt prohibits revealing the restricted name, answer, or identifying hints, or explaining the refusal.Source: §D.6.4 · p. 98
  • The required response format separates an internal BLOCK-or-ALLOW check from the final response, with BLOCK recorded only as “Matched RESTRICTED_ENTITY.”The internal check must not name or describe the restricted entity.Source: §D.6.4 · p. 98

D.7 BBH Task Instructions and Answer Constraints

  • The BBH user template inserts a task description and answer constraint around the dataset question, while asking the model to think step by step.The final response line must use the specified boxed-answer format.Source: §D.7 · p. 98
  • BBH task descriptions and answer constraints are task-specific strings listed in Table 22 and fill the corresponding prompt fields.The question itself comes from the dataset.Source: §D.7 · p. 98, Table 22 · p. 99
  • The BBH template does not include the dataset’s few-shot chain-of-thought field.Source: §D.7 · p. 98

Related papers

Loading 2610.20399v1…