PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss

Yi Ren, Wenlong Deng, Guanzhe Hong, Clare Lyle, Yarin Gal

arXiv:2609.33620v1cs.LGcs.AI

TL;DR

Continual learning raises the linked questions of which experience to learn from, what an update changes, and whether the model can keep learning later. The paper decomposes token-level update–behavior interactions into force, shared-readout, and residual components, finding distinct regimes for selection, interference, and future learnability. Its readout-transmission diagnostic degrades with future learnability, and the degradation predicts the benefit of restoring the readout.

  • Problem

    Continual adaptation requires understanding which experience to learn from, how updates alter existing behavior, and whether accumulated updates leave the model able to learn from future experience.

    Source: §1 · pp. 1–2

  • Method

    The paper decomposes token-level update–behavior interactions into softmax learning signals, shared-readout geometry, and residual flow to obtain a forward-computable approximation.

    Source: §1 · p. 2

  • Results

    Positive interactions identify useful experience, negative interactions yield collision or erosion, and readout transmission degradation predicts the benefit of restoring the readout.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Data selection, interference controls, and future-learnability diagnostics can be understood as distinct regimes of the same evolving update–behavior interaction.

    Source: §6 · p. 9

    The analysis is local and approximate; the shared readout is important but not the sole source of long-term plasticity loss.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modern language models are likely to be updated throughout their lifetime rather than trained once and frozen. Each update therefore participates in a recurring cycle: decide which experience to learn from, understand what that update changes, and remain capable of learning from what comes next. We show that these challenges are governed by the same evolving update--behavior interaction. We derive a token- and layer-wise decomposition of how learning from one token changes another prediction. By separating the softmax force, shared readout geometry, and residual connections, it exposes two interaction channels and yields a forward-computable approximation. Following this interaction through time reveals a unified picture of continual adaptation. Positive interaction identifies useful experience; negative interaction produces either concentrated collision or accumulated erosion; over longer horizons, updates reshape the shared geometry mediating future learning signals, reducing their transmission. These predictions lead to effective data selection, mechanism-specific controls for interference, and a readout-based diagnostic of future learnability whose degradation predicts the benefit of restoring the readout. Across models and training regimes, the same local interaction thus explains both what an update changes now and how learning today changes what can be learned tomorrow. This view connects data attribution, forgetting, and plasticity loss as distinct regimes of the same evolving learning dynamics.

1 INTRODUCTION

Because deployed language models will keep learning from new experience, continual adaptation must address which updates to make, what they change, and whether future learning remains possible. The paper treats these questions as regimes of one evolving update–behavior interaction, decomposes that interaction, and derives interventions for selection, interference, and future learnability.

1 INTRODUCTION

  • The paper frames continual adaptation as a response to models accumulating experience after deployment, including user feedback, tool use, retrieved documents, and generated trajectories.It connects this setting to data attribution, forgetting, and plasticity loss as stages of one learning process.Source: §1 · pp. 1–2
  • The paper opens the interaction into token forces, shared readout geometry, and residual flow, yielding two pathways and a forward-computable approximation.This makes it possible to probe interaction geometry as the model adapts rather than treating it as fixed.Source: §1 · p. 2
  • The update–behavior interaction unifies experience selection, disruption of existing behavior, and changes to the model’s ability to learn from future experience.Positive interactions predict useful experience; negative interactions distinguish concentrated collision from accumulated erosion, while changing readout geometry can reduce transmission of later learning signals.Source: §1 · pp. 1–2
  • The same interaction supports testable interventions: selecting experience by predicted benefit, applying mechanism-specific interference controls, and restoring readout to partially recover plasticity.Readout-transmission degradation predicts the benefit of the restoration intervention.Source: §1 · p. 2

2 STRUCTURED LEARNING DYNAMICS OF CONTINUAL ADAPTATION

The section defines a token-level interaction measuring how an update on one token changes confidence in another. It then separates that interaction into shared-readout and backbone contributions, approximating residual flow through forward-pass quantities.

2.1 TOKEN-LEVEL UPDATE–BEHAVIOR INTERACTION

  • A one-step token-level gradient interaction measures how learning from an updating token–context pair changes confidence in an observing pair.For broader finetuning objectives, the same model-dependent interaction applies term-wise with objective-specific weights.Source: §2.1 · p. 3

2.2 OPENING THE MODEL: READOUT AND RESIDUAL FLOW

  • Separating the shared readout from the backbone distinguishes a readout contribution from projected-force interactions propagated through the backbone.The decomposition is exact within the first-order approximation.Source: §2.2 · pp. 3–4
  • The residual approximation retains the leading identity path and locally linearizes residual blocks, replacing backbone Jacobians with forward-pass quantities.Under these approximations, each block’s contribution reduces to similarity between its forward hidden representations.Source: §2.2 · p. 4
  • Direct force alignment gives a concentrated interaction channel, while shared-readout geometry permits broader coupling and also governs transmission of future learning signals.Off-diagonal readout structure can couple vocabulary directions even when direct force overlap is weak.Source: §2.2 · p. 4

3 SELECTING BENEFICIAL UPDATES

The interaction score ranks experience by its predicted effect on a target behavior, using forward-pass quantities. Its readout-mediated channel is especially useful when direct vocabulary alignment is weakened, and selected examples improve downstream learning.

3 SELECTING BENEFICIAL UPDATES

  • The score aggregates token-level interactions between candidate experience and a probe set representing the desired behavior; higher-scoring examples are selected.The scores use only forward quantities at a fixed checkpoint.Source: §3 · p. 5, Table 1 · p. 5
  • On controlled attribution tasks with substantial task and output-token overlap, CH1 nearly saturates sentence-transformation and mathematical attribution, with little added benefit from CH2.These benchmarks rank candidates using AUC and top-K recall.Source: §3 · p. 5
  • When direct alignment weakens cross-lingually, adding CH2 raises retrieval accuracy from 0.488 to 0.625 on MMLU and from 0.445 to 0.600 on GSM8K.The table summarizes near-saturated CH1 attribution under strong alignment and complementary CH2 signal when alignment is weakened.Source: §3 · p. 5, Table 1 · p. 5
  • In multilingual agent memory, incorporating CH2 raises Llama3.2-3B top-1 retrieval accuracy from 0.861 to 1.000, yielding 0.944 downstream answer accuracy.This retrieval setting uses an English query and a shared multilingual memory.Source: §3 · p. 5
  • At a 5% GSM8K training budget, CH1+CH2 reaches 0.636 accuracy on Qwen2.5-1.5B and 0.313 on Llama3.2-3B, exceeding random selection and LESS on both models.Random selection reaches 0.610/0.295 and LESS reaches 0.587/0.301, respectively.Source: §3 · p. 5
  • The scores are forward-only at a fixed checkpoint, while reliable signed use may require a brief warm-up.The warm-up qualification is noted in the passage's footnote.Source: §3 · p. 5

4 NEGATIVE INTERACTIONS: COLLISION AND EROSION

Negative update–behavior interactions produce two forms of forgetting: concentrated collision from a few strong conflicts and gradual erosion from many weak ones. Their distinct mechanisms support different controls, with format separation redirecting erosion while retaining downstream learning.

4 NEGATIVE INTERACTIONS: COLLISION AND EROSION

  • Collision is concentrated interference from one or a few strong negative forces, whereas erosion is gradual behavioral drift from many weak conflicts that accumulate coherently.Erosion can proceed without extreme-energy updates, as repeated forces shift probability toward competing continuations.Source: §4 · p. 6
  • Under one-hot supervision, token energy above 1 + δ implies a non-target force component below −δ, and such a component conversely implies energy above 2δ^2.This links extreme energy to a strong negative vocabulary-space direction that can conflict with existing high-confidence behavior.Source: §4 · p. 6
  • Energy masking mitigates collision but has little effect on erosion; for erosion, response-format separation preserves more instruction following while retaining downstream gains and redirects interference toward the fine-tuning format.The figure contrasts these controls and their gain–retention trade-offs.Source: Figure 5 · p. 7, §4 · p. 7
  • Erosion can arise when repeated GSM8K reasoning-style continuations conflict with an MMLU answer format, gradually shifting probability away from the desired answer format.The interactions can each be weak while consistently pushing toward competing continuations.Source: §4 · p. 7
  • On-policy RL forgets less than off-policy SFT, which the analysis associates with extreme-energy updates being common under off-policy SFT and rare in on-policy rollouts.This motivates testing energy control as a response to collision.Source: §4 · p. 7
  • Masking updates with Eu > 1.8 or 1.5 changes GSM8K by at most 2.6 points across models, while a threshold of 1.0 sharply degrades downstream learning.At 1.0, masking also removes many ordinary semantic tokens rather than only extreme conflicts.Source: §4 · p. 7
  • For the total answer-token probability mass P(A . . . D), energy control leaves erosion largely intact even as it increasingly improves collision-side retention with higher learning rates.Standard MMLU accuracy can mask erosion, motivating this alternative measure.Source: §4 · p. 7
  • Collision is linked to a few high-energy conflicts and responds to energy control, whereas erosion accumulates through weak interactions and responds to changing alignment.The controls differ because collision depends on individual update magnitude while erosion depends on repeated routing toward the same competing behavior.Source: §4 · p. 7

5 EVOLVING GEOMETRY AND FUTURE LEARNABILITY

The shared readout mediates how task forces reach the backbone, and learning reshapes this transmission geometry. Its task-conditioned transmission score declines for out-of-distribution tasks over long training, tracks slower future adaptation, and predicts gains from readout restoration.

5 EVOLVING GEOMETRY AND FUTURE LEARNABILITY

  • The shared readout gates task forces into the hidden space; its task-conditioned transmission score RD measures how effectively signals from a task enter the backbone.The numerator measures transmitted strength and the denominator normalizes input force magnitude.Source: §5 · p. 8
  • Training preferentially reinforces directions supported by the current task, so transmission along weakly supported future-task directions can deteriorate and weaken the backbone update.The proposed self-reinforcing bottleneck is a possible amplification mechanism, not a necessary condition for plasticity loss.Source: §5 · p. 8
  • During long-horizon PubMed training, RD declines for held-out OOD tasks without comparable systematic decay for in-distribution PubMed, consistent with task-dependent readout reshaping.Measurements include GSM8K, MBPP, and Dolly-QA as OOD tasks.Source: §5 · pp. 8–9
  • Later checkpoints adapt more slowly on three OOD tasks but not on in-distribution PubMed, while restoring the readout substantially improves validation loss.Resetting the last four layers provides only modest additional recovery, so the readout is an important but not exclusive source of lost plasticity.Source: §5 · p. 9, Figure 7 · p. 9
  • Across models, tasks, and checkpoints, readout degeneration and readout-reset gains are strongly correlated (Spearman ρ = 0.83 overall; 0.73 on OOD tasks).The figure summarizes degeneration as a predictor of reset benefit.Source: §5 · p. 9, Figure 7 · p. 9

6 CONCLUSION AND DISCUSSION

The conclusion frames selection, interference, and future learnability as regimes of one evolving update–behavior interaction, while limiting the analysis to local, approximate gradient-based parameter updates. It situates that account within continual adaptation and related work on attribution, forgetting, and plasticity.

6 CONCLUSION AND DISCUSSION

  • The paper unifies useful updates, interference, and future learnability through token forces, shared-readout geometry, and residual flow, motivating energy control, format separation, and readout restoration.The analysis is local and approximate; higher-order residual paths, optimizer-dependent dynamics, longer-term feedback, and broader architectures remain open directions.Source: §6 · p. 9

A.1 CONTINUAL ADAPTATION AND LEARNING DYNAMICS IN AGENTIC LLM SYSTEMS

  • The framework focuses on consolidation of experience into model parameters, where the cycle of selecting experience, understanding its behavioral effect, and preserving future learning recurs.The paper presents this as one stage within a hierarchy of adaptation timescales.Source: §A.1 · p. 14
  • Selection, interference, and sustained adaptability also organize questions for memory-based learners, although their underlying mechanisms differ from gradient-based learning.The paper treats the parameter-update analysis as one concrete realization, with analogous questions possible under other mechanisms.Source: §A.1 · p. 14

A.2 RELATED WORK ON SELECTION, INTERFERENCE, AND PLASTICITY

  • The related literatures treat beneficial attribution, harmful forgetting, and responsiveness to future learning as separate concerns that the paper examines through a shared interaction perspective.The framework resolves interactions into signed token forces, readout geometry, and layer-wise propagation.Source: §A.2 · pp. 14–15
  • Forgetting is analyzed through signed local interactions, distinguishing concentrated collision from accumulated erosion rather than relying only on representation or task-subspace overlap.Plasticity loss adds the distinct question of how previous updates reshape the geometry through which future updates act.Source: §A.2 · p. 15

A.3 LIMITATIONS AND ASSUMPTIONS OF THE LEARNING-DYNAMICS FRAMEWORK

The framework prioritizes tractable, interpretable approximations over fidelity to full training dynamics, so its conclusions are bounded by local-update assumptions, optimizer-state modeling, and empirical scope.

A.3 LIMITATIONS AND ASSUMPTIONS OF THE LEARNING-DYNAMICS FRAMEWORK

  • The approximations share a goal of retaining dominant update structure while making the framework tractable, interpretable, and computationally efficient at LLM scale.Most results build on Equation (5), which relates behavioral change to local gradient interaction.Source: §A.3 · p. 15
  • The analysis is first-order and is most reliable for sufficiently small parameter updates.It neglects terms of order O(∥θt+1 −θt∥2).Source: §A.3 · p. 15
  • The one-step optimizer analysis extends naturally to a fixed preconditioner, but jointly modeling AdamW’s evolving state and weight decay remains future work.Momentum and second-moment estimates make the effective preconditioner depend on training history.Source: §A.3 · pp. 15–16
  • The forward-computable approximation linearizes residual blocks and retains only the leading identity path, omitting costly higher-order residual paths.Retaining those paths would improve fidelity but reintroduce increasingly expensive Jacobian products.Source: §A.3 · p. 16
  • Forgetting and plasticity-loss explanations extrapolate single-update interactions along training trajectories and are validated empirically, not derived from an exact multi-step theory.A trajectory-level account would need to track gradients, representations, and interaction geometry jointly.Source: §A.3 · p. 16
  • Empirical findings are mechanistic patterns rather than universal quantitative laws, and the decomposition is tailored to residual Transformers with a shared readout.It does not explicitly resolve MoE, QKV projections, or circuit-level pathways; signed fidelity can depend on representation regime, particularly at cold start.Source: §A.3 · p. 16

B.1 DERIVATION OF ONE-STEP TOKEN-WISE CONFIDENCE CHANGE

This section derives how token-wise confidence changes depend on readout and backbone geometry, extending the analysis to energy–force bounds, cross-task readout degeneration, and adaptive optimization. It examines the persistence of the two-channel interaction structure across optimizers and architectures, and how score fidelity depends on the training regime.

B.1 DERIVATION OF ONE-STEP TOKEN-WISE CONFIDENCE CHANGE

  • Under identity-path and linear-block approximations, removing the layer-dependent output-side Jacobian reduces backbone geometry to hidden-state similarity, making its all-layer contribution estimable from forward-pass quantities.The embedding-overlap term is a context-level approximation; its inclusion usually changes rank correlation by less than 0.001, though token routing in long contexts remains untreated.Source: §B.1 · pp. 17–18

B.2 ENERGY AND NEGATIVE FORCE

  • Under one-hot supervision, energy E_u > 1 + δ guarantees some non-target token has force [g_u]_j < −δ, while any such force implies E_u > 2δ².The inverse bound is tight when non-target probability mass is concentrated on one token, but is looser than the forward bound.Source: §B.2 · pp. 18–19

B.3 FORMAL ANALYSIS OF CROSS-TASK READOUT DEGENERATION

  • With mild readout norm control and sufficiently limited overlap between task-induced force directions, training on task A can attenuate task B’s readout transmission and reduce its effective one-step adaptation.The connection to reduced backbone updates holds when output-level signal and residual-state factors do not systematically grow enough to compensate for attenuation.Source: §B.3 · pp. 19–21
  • Repeated task-A updates progressively reduce readout transmission in directions weakly supported by A, provided a readout budget constraint contracts those directions; explicit weight decay is not required.The bound holds with contraction from mechanisms such as decoupled weight decay or a budget tradeoff across directions; the latter mechanism is not derived here.Source: §B.3 · pp. 19–21

B.4 BEYOND SGD: WHAT CHANGES UNDER ADAPTIVE OPTIMIZATION?

  • Adaptive optimization can change attribution values, relative channel strengths, and candidate rankings, while decoupled weight decay adds candidate-independent local drift that does not alter rankings for a fixed observation.The signed score captures local gradient interaction, not necessarily the exact sign of the full AdamW update.Source: §B.4 · pp. 21–22
  • Adaptive preconditioning preserves the force–kernel–force structure, the same parameter-induced interaction channels, and the shared readout’s role, while replacing the Euclidean kernel with an optimizer-weighted kernel.Adam reweights existing coordinate interactions rather than creating new paths; positive coordinate weights preserve each individual contribution’s sign, although aggregate interaction sign or magnitude may change.Source: §B.4 · p. 22

C UNDERSTANDING THE VALIDITY REGIME OF THE APPROXIMATION

  • Approximation magnitude is reliable even at cold start, while signed fidelity can initially fail and recover rapidly after brief in-distribution adaptation.At cold start, the approximation tracks absolute interaction value and rank, but signed comparisons show frequent sign flips despite preserved magnitude structure.Source: C UNDERSTANDING THE VALIDITY REGIME OF… · p. 23, Figure 9 · p. 24

B.4 BEYOND SGD: WHAT CHANGES UNDER ADAPTIVE OPTIMIZATION?

  • Adaptive preconditioning preserves the two-channel structure but can change attribution magnitudes and rankings; AdamW’s decoupled weight decay does not affect candidate ranking.The fixed-preconditioner analysis does not capture Adam’s full history dependence from first- and second-moment estimates.Source: §B.4 · pp. 22–23, §C.1 · p. 23

C.1 HOW ROBUST IS THE STRUCTURE OF EQUATION (5)

  • Standard single-stream residual architectures retain the two-channel form; multiple residual streams change CH2’s path structure, not its underlying force-kernel-force view.For multi-stream architectures, the corresponding residual paths must be incorporated explicitly.Source: §C.1 · p. 23
  • Tied embeddings add correction terms rather than replacing CH1+CH2, and including them changes scores only modestly in the tested settings.The tied input and output parameterizations contribute gradients to the same matrix; the paper therefore uses the simpler Equation (5) in its main experiments.Source: §C.1 · p. 23

C.2 WHEN IS THE APPROXIMATION NUMERICALLY RELIABLE?

The approximation is numerically useful, but signed fidelity depends on the interaction channel and training state. The appendix explains its computational trade-offs, practical use, representation geometry, and data-attribution evaluations.

C.2 WHEN IS THE APPROXIMATION NUMERICALLY RELIABLE?

  • The first-order expansion closely tracks measured log-probability changes, while readout-only CH1 updates achieve Pearson/Spearman ≈1.0 and MAE = 2.2 × 10−9.The remaining approximation error arises primarily from CH2 and residual-backbone propagation.Source: §C.2 · p. 24
  • At initialization, interaction magnitudes can be reliable despite sign reversals: absolute-value Spearman reaches 0.878, while signed rank correlation is nearly zero.The qualitative cold-start pattern is observed across multiple models and dataset pairs.Source: §C.2 · pp. 24–25
  • Signed fidelity improves within about 100 updates, under 1% of the run, and remains stable; full-depth aggregation also becomes reliable after this brief warm start.At initialization, signed metrics deteriorate toward chance as residual layers are added, despite absolute-rank correlation around 0.7.Source: §C.2 · pp. 25–26, Figure 11 · p. 25, Figure 12 · p. 26

C.3 WHY DO WE STOP AT THE IDENTITY PATH?

  • Retaining only the first-order identity residual path keeps the score forward-computable; higher-order paths add combinatorial growth and layer-dependent transformations that erode its computational advantage.The resulting score is intended as an inexpensive online probe, not an exact reconstruction of the full gradient interaction.Source: §C.3 · pp. 26–27

C.4 PRACTICAL IMPLICATIONS AND USAGE GUIDELINES

  • Use magnitude-based scores at cold start and signed attribution after brief in-distribution adaptation; periodic recomputation can track changing interaction geometry.The passage also gives collision-specific guidance: use a warm start or recompute online when signed collision estimates are needed near initialization.Source: §C.4 · p. 27

C.5 REPRESENTATION OVERLAP MAINLY MODULATES INTERACTION MAGNITUDE

  • Hidden-state overlaps are predominantly positive across sampled examples, models, and layers, suggesting overlap mainly modulates interaction magnitude while force-side geometry more strongly determines sign.The analysis samples token pairs from GSM8K and MMLU examples across Qwen, Llama, Mistral, and OLMoE models.Source: §C.5 · pp. 27–28, Figure 14 · p. 28

D MORE ON DATA ATTRIBUTION; D.1 EXPERIMENTAL DETAILS FOR DATA SELECTION

  • The data-attribution appendix details controlled influential-example benchmarks and compares embedding, influence-function, gradient-alignment, and forward-computable CH1/CH1+2 scores.The benchmark settings use sentence transformation and mathematical problems, evaluating rankings with AUC and top-K recall.Source: D MORE ON DATA ATTRIBUTION · p. 28, §D.1 · pp. 28–29, Table 2 · p. 29

D.2 CONTROLLED CROSS-LINGUAL ATTRIBUTION

  • In cross-lingual attribution, translated candidates preserve semantic correspondence while reducing top-K prediction-token overlap, whose average Jaccard similarity is below 0.15.The controlled comparisons cover GSM8K and MMLU examples translated into Chinese, French, Korean, and Spanish.Source: §D.2 · p. 29, Figure 15 · p. 30

D.3 FINE-TUNING WITH SELECTED DATA

The section tests whether attribution-ranked GSM8K subsets improve downstream fine-tuning and compares their accuracy and compute cost with random selection and LESS.

D.3 FINE-TUNING WITH SELECTED DATA

  • Selected GSM8K subsets are fine-tuned for five epochs under matched settings, with top 1%, 5%, and 10% selections compared against random selection and LESS.Scores use a 100-step in-distribution warm start, while downstream fine-tuning starts from the original base checkpoint.Source: §D.3 · p. 30, Table 5 · p. 31
  • The relative advantage of CH1 and CH1+2 varies across model-budget settings, and CH2 sometimes adds gains; the authors do not treat this as a general scale or perplexity relationship.The table reports test accuracy as mean ± standard deviation over three training seeds and aggregate A100 GPU-hours for score computation.Source: §D.3 · p. 30, Table 5 · p. 31
  • The evaluation reports downstream accuracy alongside GPU hours, with score-computation costs including method-specific setup but excluding downstream fine-tuning.The fixed probing set contains 500 examples, leaving 6,973 candidates for selection and fine-tuning.Source: Table 5 · p. 31

E.1 UPDATE ENERGY UNDER DIFFERENT FINETUNING OBJECTIVES

This section compares how fine-tuning objectives shape token update forces and their risk of entering the extreme-energy regime associated with collision.

E.1 UPDATE ENERGY UNDER DIFFERENT FINETUNING OBJECTIVES

  • Generalized knowledge distillation replaces a one-hot target with a generally softer teacher distribution, making extreme-energy forces less likely.The comparison describes the force characteristic, not the separate routing of contexts through the kernel.Source: §E.1 · pp. 31–32
  • On-policy distillation rarely samples low-probability targets, suppressing many confident conflicts, but a large student-teacher likelihood-ratio multiplier can still amplify an update.The sensitivity is more pronounced when student and teacher distributions are poorly matched (Li et al., 2026b).Source: §E.1 · pp. 31–32
  • Label smoothing with ε = 0.1 bounds update energy at approximately 1.81 for a sufficiently large vocabulary, excluding the most extreme collision regime.It replaces one-hot supervision with a mixture of the target and a uniform distribution.Source: §E.1 · p. 32
  • On-policy sampling, soft teacher targets, and label smoothing reduce the risk of extreme update forces through different mechanisms, while standard off-policy SFT can approach energy 2.The analysis concerns output-side force; how contexts route those updates through the kernel is a separate question tied to erosion.Source: §E.1 · pp. 31–32, Table 6 · p. 32

E.2 ADDITIONAL ENERGY-CONTROL EXPERIMENTS ON OPENMATHINSTRUCT-2

Additional OpenMathInstruct-2 experiments test energy control across models, training settings, and thresholds. Moderate masking improves retention while preserving learning, but aggressive masking removes useful updates.

E.2 ADDITIONAL ENERGY-CONTROL EXPERIMENTS ON OPENMATHINSTRUCT-2

  • Across OpenMathInstruct-2 models, the experiments compare standard SFT, EAFT, and energy-controlled variants under full-parameter fine-tuning on a larger corpus with a consistent boxed-answer style.The setup samples 50,000 deduplicated training examples and evaluates downstream learning and capability retention.Source: §E.2 · pp. 32–33, Table 7 · p. 33
  • Moderate energy thresholds improve retention while preserving most downstream learning, whereas overly aggressive masking degrades downstream performance by removing useful updates along with harmful collisions.The threshold sweep supports selective suppression of the extreme-energy tail rather than indiscriminate reduction of updates.Source: §E.2 · p. 33, Table 7 · p. 33
  • The EAFT-minus-SFT retention gap grows with learning rate across four metrics; at 10−5 EAFT trails on MMLU, Dolly-CLS, and Dolly-QA, while it leads on every metric only at the largest rate.The authors treat this as supporting rather than isolating evidence because changing learning rate scales all update components, not energy alone.Source: Table 8 · p. 34, §E.2 · p. 34

E.3 ADDITIONAL EVIDENCE FOR EROSION ON OPENMATHINSTRUCT-2

OpenMathInstruct-2 fine-tuning shifts responses toward its trained response conventions, weakening instruction following while answer accuracy can remain comparatively stable. The additional analyses track this drift through response-pattern and probability-level measures, and test format separation and energy controls.

E.3 Additional Evidence for Erosion on OpenMathInstruct-2

  • The evaluation uses greedy decoding after prompts request only A/B/C/D, measuring format noncompliance, boxed answers, first-token choice probability, extraction-based accuracy, and long worked responses.Worked denotes generations longer than 200 characters; the reported evaluation follows OpenMathInstruct-2 fine-tuning and its answer convention.Source: §E.3 · p. 34, Table 9 · p. 35
  • Worked rises after SFT from 4.0% to 75.6% on Qwen-1.5B, 1.8% to 36.6% on Qwen-4B, and 4.1% to 50.1% on Llama-3B.The authors use this response-length measure as a supporting diagnostic of drift beyond copying a final-answer marker, while noting its 200-character threshold is coarse.Source: §E.3 · pp. 34–35
  • Using P/R rather than Q/A during OpenMathInstruct-2 fine-tuning better preserves deployed Q/A behavior while redirecting interference toward P/R, with similar downstream GSM8K performance.The evaluation uses 14,042 MMLU questions in both formats.Source: Table 10 · p. 35
  • For Qwen3-4B-Instruct, SFT raises Non-IF from 11.4% to 63.7%, Boxed from 0.0% to 53.3%, and Worked from 1.8% to 36.6%.The first-token choice mass also drops from 0.981 to 0.399, indicating drift toward the response convention reinforced during training.Source: §E.3 · p. 35
  • On Qwen2.5-1.5B, P(A . . . D) falls from 0.201 to 0.156 to 0.118 as learning rates rise from 10−5 to 2 × 10−5 to 5 × 10−5, while Boxed rises from 67.6% to 76.2% to 88.3%.Probability-level shifts remain visible even when greedy-response noncompliance is already near saturation.Source: §E.3 · pp. 35–36
  • Qwen3-4B-Instruct changes from 69.4% to 68.1% confidence-based MMLU accuracy despite substantial shifts in response-pattern measures.The authors characterize erosion as largely dissociated from answer correctness.Source: §E.3 · p. 36
  • EAFT and energy control only partially mitigate erosion; on Qwen3-4B, masking Eu > 1.5 lowers Non-IF from 63.7% to 55.1% and Boxed from 53.3% to 40.8%.The passage reports that stricter energy caps can improve instruction following, but many individually mild updates can still accumulate into erosion.Source: §E.3 · p. 36

E.4 ADDITIONAL ANALYSES OF CONTEXT SEPARATION

Additional context-separation analyses find that interference is stronger when evaluation uses the response format used for incoming updates, while downstream learning remains comparable. The decomposition attributes the effects of format separation mainly to distinct alignment changes in CH1 and CH2.

E.4 Additional Analyses of Context Separation

  • In the controlled experiment, interference is stronger when MMLU is evaluated in the response format used by GSM8K updates, while downstream GSM8K learning remains comparable.The result is shown for before- and after-finetuning behavior from the same base checkpoint.Source: §E.4 · p. 36
  • Instruction-tuned models generally show greater instruction-following degradation when fine-tuned in the response format already more aligned with their behavior, though the effect varies across models.This offers a complementary check using models whose prior format preferences come from unknown training mixtures.Source: §E.4 · p. 37
  • For CH1, replacing the update force accounts for most of the reduction after switching from Q/A to P/R, whereas replacing only the final-layer representation changes the score much less.The paired examples keep the question, reasoning trace, and target answer fixed; only prompt and answer-field labels differ.Source: §E.4 · p. 37
  • For CH2, the readout-mediated force-alignment term remains nearly unchanged, while layer-wise hidden-state similarity decreases after the Q/A-to-P/R shift.The reduced contextual inner-product alignment is clearer toward later layers and is the main source of CH2 attenuation.Source: §E.4 · pp. 37–38, Figure 17 · p. 38

F MORE ON PLASTICITY LOSS

Sequential fine-tuning redistributes shared readout geometry toward the task being learned, while longer training is associated with reduced transmission for out-of-distribution probes. Reset experiments test whether restoring readout and upper layers recovers downstream plasticity.

F More on Plasticity Loss

  • The sequential-SFT diagnostic uses 20 epochs per task and reinitializes optimizer state, learning-rate schedule, and warm-up at each task transition.The schedule is longer than typical SFT schedules, chosen to amplify the degeneration signal.Source: F MORE ON PLASTICITY LOSS · p. 39
  • During sequential SFT, task-specific readout statistics strengthen for the current task while those of the other tasks decrease, with this redistribution shifting as training moves from A to B to C.The experiment probes task-specific 300 × 300 readout blocks using trace, effective rank, and density; trace reflects overall spectral mass, while eRank and density help distinguish geometric reshaping from uniform contraction.Source: F MORE ON PLASTICITY LOSS · pp. 38–39
  • The controlled sequential-SFT experiment is designed to isolate geometric redistribution rather than model realistic long-horizon continual learning, and its induced plasticity loss remains limited.The authors therefore proceed to a longer training trajectory to examine accumulated behavioral consequences.Source: F MORE ON PLASTICITY LOSS · p. 39
  • During 100M-token PubMed2025 training, readout transmission generally decreases for OOD probes but does not show the same systematic decay for the in-distribution PubMed probe.The two weight-decay settings have nearly identical trajectories, so the observed degeneration is not explained simply by uniform weight-decay contraction.Source: Figure 22 · p. 42, Figure 21 · p. 42
  • Across four model families and multiple learning rates, task-conditioned transmission generally decreases during 300M-token training on a broader hybrid corpus, but its magnitude and timescale depend strongly on model and optimization regime.Larger learning rates typically produce faster and stronger changes; training uses causal-LM format while probes use downstream task formats, so semantic domain overlap does not ensure exact token-level signal overlap.Source: Figure 24 · p. 44
  • Resetting the readout generally recovers some lost plasticity, and resetting the last four transformer blocks with it can provide further recovery, with effects varying by model and task.Plasticity retention is downstream loss AUC relative to the base model, where 1 denotes pre-training plasticity; Figure 25 separates training and probe-set retention across three aggregations.Source: Figure 25 · p. 45

Related papers

Loading 2609.33620v1…