PaperlayerView exact revision on arXiv

Source-linked AI summary

A Step Towards Forgetting: Optimiser History and the Loss of Answer Mass

Valeria Ruscio, Seth Nabarro, Keiran Thompson

arXiv:2610.03940v1cs.CLcs.AI

TL;DR

The paper asks how a language model can lose probability mass on previously learned answers even when its current gradient acts to preserve them. It separates old-task loss into answer-set confusion and leakage, and examines Adam’s current and stored-gradient contributions through decompositions, interventions and finite-update integration. Across the tested model families, answer mass declines while discrimination usually improves, with history and current-gradient effects opposing each other.

  • Problem

    Old-task loss can combine weaker discrimination among learned answers with probability leaking outside their set, obscuring distinct changes in retained behaviour.

    Source: §1 · p. 1

  • Method

    The paper separates confusion from leakage and evaluates Adam’s history and current channels against each component, using matched-history interventions and finite-update integration.

    Source: §1 · p. 2

  • Results

    Across three model families, answer-set probability falls in every trajectory while discrimination among old answers usually improves; accumulated history favours leakage while the current gradient opposes it.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Assessing both loss components reveals that recovery of old-answer mass and discrimination among old alternatives can move in opposite directions.

    Source: §8 · p. 11

    The empirical scope is sequential classification with single-token answers, a fixed task order and model-specific learning rates; extension to open-ended generation remains untested.

    Source: §8 · p. 11

Abstract

from arXiv · show

During fine-tuning, a language model can assign less probability to previously learned answers even when the current gradient acts to preserve that probability. With momentum, each update also carries gradients computed at earlier model states, and these stored contributions can push the model in the opposite direction. We investigate how this optimiser memory contributes to forgetting by separating old-task loss into confusion among its answers and leakage of probability outside the answer set. Across three language-model families, answer mass consistently declines while discrimination among old answers usually improves: the model becomes less likely to produce answers that it can still distinguish correctly. Decomposing Adam updates reveals opposing contributions to this loss of answer mass. Over training, accumulated history favours leakage, while the current gradient opposes it. Resolving history by age shows that the harmful contributions come mainly from older gradients of the new task, whereas recent gradients tend to protect the old answers. Changes in history's effect are dominated by its orientation relative to the old-task gradient. Interventions that reset momentum while matching the initial update norm establish that stored history affects retention, with state-dependent immediate effects and lower final old-task loss over longer Adam continuations, mainly through recovered answer mass. Finally, integration along finite updates shows that most sampled large loss increases are captured by local projections, while curvature along history amplifies some events. Together, these findings reveal how an optimiser's memory can erode learned behaviour even as its current gradient acts to preserve it.

1 Introduction

Old-task forgetting can reflect either weaker discrimination among old answers or probability leaking outside their set, so aggregate loss may conceal opposing behavioural changes. The paper separates these effects and examines how current gradients and stored optimiser history contribute to them.

1 Introduction

  • The old-task loss separates confusion among old answers from leakage of probability outside the answer set, which can change in opposite directions.A model can improve its relative preference for the correct answer while assigning it less probability overall.Source: §1 · pp. 1–2
  • With momentum, updates combine current minibatch gradients with earlier gradients formed at different model states, whose effects on old-task loss can have opposing signs.The first-order effect depends on the projection of the applied displacement onto the old-task gradient.Source: §1 · p. 1
  • Across three model families, answer-set probability falls in every trajectory while discrimination among old answers usually improves.The paper also reports that history favours leakage over training while the current gradient opposes it; older new-task gradients mainly supply harmful leakage contributions.Source: §1 · p. 2
  • Matched-displacement interventions show that changing stored history changes retention, with immediate effects depending on state and longer-term benefit depending on new-task learning.Finite-update integration finds that initial projections capture most sampled large loss increases, while curvature along history amplifies some events.Source: §1 · p. 2

2 Related work

Prior continual-learning methods often protect earlier behaviour by constraining parameter changes or shaping new updates. This paper instead evaluates the optimiser’s actual displacement against separate old-task loss components, connecting output-level retention changes to optimiser geometry.

2 Related work

  • Unlike methods that constrain updates using old-task sensitivity, this work evaluates the optimiser’s actual displacement against separate gradients for confusion and answer mass.The cited approaches include elastic weight consolidation [Kirkpatrick et al., 2017], Synaptic Intelligence [Zenke et al., 2017], Gradient Episodic Memory and Orthogonal Gradient Descent [Lopez-Paz and Ranzato, 2017; Farajtabar et al., 2020].Source: §2 · p. 2
  • Stored-gradient methods vary in how they use optimiser memory, while this paper evaluates stored gradients by their alignment with a specific old-task loss component.The related methods include MoFO [Chen et al., 2024], AdEMAMix [Pagliardini et al., 2024], cautious optimisers [Liang et al., 2024] and work on sequential-transfer interference (Frati et al., 2025).Source: §2 · p. 2
  • The paper’s finite-path analysis addresses how curvature changes the accuracy of local loss predictions, a topic also connected to edge-of-stability and forgetting research.The cited work includes Cohen et al. (2021) and Li et al. (2024).Source: §2 · p. 2
  • The confusion–leakage factorisation makes explicit how old answers can retain their relative ordering while losing probability to other vocabulary tokens.Figure 1 illustrates a combined update that raises leakage while reducing confusion, and the path integrals separate history, current and mixed curvature contributions.Source: §2 · p. 3, Figure 1 · p. 3

3 The geometry of retained behaviour

The section defines old-task retention through two loss components and decomposes Adam’s update into history and current channels. It then describes how to attribute local changes to these channels and to gradients of different ages.

3.1 Two ways to lose an answer

  • Old-task loss is evaluated on a fixed population, and its full-vocabulary likelihood separates confusion within the old answer set from leakage outside it.The two components have counterparts in restricted-answer and full-vocabulary accuracy, respectively.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 Two sources of an update

  • Adam’s displacement is split into history, current-gradient and reconstruction-residual channels, with the history and current channels sharing adaptive scaling.Projecting each channel onto each loss-component gradient yields four local contributions; positive entries indicate local loss increases, and row and column sums summarize behaviour and channel effects.Source: §3.2 · p. 4

3.3 The age and orientation of history

  • The age decomposition groups active stored-gradient contributions into 1–4, 5–16 and 17-or-more-update bins, retaining their native magnitudes and widths.The sign of each contribution depends on the stored gradient, current preconditioner and current leakage sensitivity.Source: §3.3 · pp. 4–5
  • Changes in a channel’s projection are separated into channel motion and task-gradient motion, then into orientation and norm changes.The attribution uses shares of summed absolute scalar contributions to avoid cancellation across updates; a nested decomposition also considers adaptive scaling.Source: §3.3 · p. 5

4 Experimental setting

The experiments use sequential single-token text classification across three model families, with the optimiser state carried from the old task into the new one. Behavioural endpoints, replay-based attributions, matched-history interventions and finite-path measurements use distinct populations and protocols.

4 Experimental setting

  • The study trains Pythia-410M, SmolLM2-135M and Qwen3-0.6B-Base on SST-2 followed by AG News, with three seeds and two answer-codebook conditions per model.Each task has 1,024 training, 32 validation and 128 test texts; models train for 600 old-task updates then 128 new-task updates with Adam state carried across the boundary.Source: §4 · p. 5
  • Behavioural endpoints use all 128 test texts, whereas history replay uses a fixed 32-text test subset plus all 32 validation texts across 1,664 updates.The replay covers seven Pythia and SmolLM2 trajectories and all six Qwen trajectories; the former were the first completed and are not balanced across conditions.Source: §4 · p. 5
  • History interventions compare branches sharing weights, minibatches and all optimiser state except history, with the first modified displacement norm matched.Continuation branches at updates 8, 32 and 64 train for another 32 steps; finite-path measurements use adaptive integration and independent derivative checks.Source: §4 · p. 5
  • Leakage increases in all 18 trajectories after 128 new-task updates, while conditional confusion decreases in 16.Figure 2 reports the two components on the full 128-text test population; bars show components and black markers their sum, with shared and disjoint codebooks paired within seed.Source: Figure 2 · p. 6
  • Supporting experiments on association learning, forecasting and activation interventions are described in the appendices.These are listed as supporting experiments beyond the principal sequential classification setup.Source: §4 · p. 6

5 History and current learning act on answer mass in opposite directions

Across trajectories, old-answer mass falls even as conditional discrimination usually improves. Adam’s accumulated history favours leakage while the current gradient opposes it; older new-task gradients and changing history orientation account for important parts of this pattern.

5.1 Answer mass falls while conditional predictions improve

  • In all 18 trajectories, leakage increases, while confusion decreases in 16; two SmolLM2 runs improve total old-task loss despite losing answer mass.Full-vocabulary accuracy falls in all 12 Pythia and SmolLM2 runs, including the two with improved total loss.Source: §5.1 · p. 6

5.2 A consistent sign pattern within the update

  • Across all 13 replays, history and the current gradient have opposite-sign cumulative leakage projections: history favours leakage, while the current channel favours recovery.Their combined local effect can favour or oppose leakage, and accumulated residuals can reverse the endpoint account in individual replays.Source: §5.2 · pp. 6–7
  • History’s effect on confusion varies across models, and replay-subset results can differ from full-test endpoints.History’s confusion projections are harmful in three Pythia replays and beneficial in four SmolLM2 replays; confusion rises in three Qwen replay subsets but falls at all six full-test endpoints.Source: §5.2 · p. 7

5.3 The positive history contributions come from older new-task gradients

  • Older task-B gradients favour leakage in every replay, whereas ages 1–4 oppose it in 12 of 13; the task-A moment contributes comparatively little.The positive contributions arise mainly from older gradients accumulated while learning the new task, so resetting only at the task boundary would leave later contributions intact.Source: §5.3 · p. 7

5.4 Orientation dominates changes in history’s local effect

  • Orientation accounts for 77.19–83.97% of changes in history’s total-loss projection and is the largest term in every trajectory.History motion accounts for 59.10–75.56% of its absolute two-term attribution; adaptive-denominator changes contribute 1.80–3.23% within the nested decomposition, whose shares use separate normalisations.Source: §5.4 · pp. 7–8

6 Changing history changes retention

Resetting Adam’s stored first moment while matching the initial displacement changes retention. Immediate effects depend on the intervention state, while longer continuations lower final old-task loss across the tested Adam trajectories, mainly through recovered answer mass.

6 Changing history changes retention

  • The intervention changes the stored first moment while holding the starting model and minibatch fixed, then matches the reset update’s norm to the original displacement.This isolates the immediate effect of changing update direction at the original movement scale.Source: §6 · pp. 8–9
  • At a fixed displacement scale, the immediate retention effect depends on intervention state: a matched reset helps in one SmolLM2 seed, while preserving history helps in another.Momentum SGD also shows a substantial matched-reset effect.Source: §6 · p. 9
  • After 32-step continuations, block-matched resets lower final old-task loss in all six Adam model-and-seed trajectories, mainly through improved leakage in Pythia.Pythia’s leakage improves by 0.227 nats while confusion worsens by 0.100, for a net old-task-loss improvement of 0.127 nats; SmolLM2 improves on both components on average.Source: §6 · p. 9
  • At equal new-task validation loss, Pythia’s mean old-task-loss difference is +0.003580 nats at target 0.75 and −0.025831 nats at target 1.0.Norm matching constrains only the first displacement; branches then evolve independently, and reset benefit depends on new-task progress.Source: §6 · p. 9, Table 4 · p. 9

7 From local projections to finite loss changes

Finite-update analysis tests how well local projections explain loss changes along the full parameter displacement. Most sampled large increases are first-order dominated, but curvature along history amplifies some paths.

7 From local projections to finite loss changes

  • Finite-path integration assigns loss change to fixed channels while the task gradient evolves; the weighted path Hessian separates diagonal, mixed, and remainder contributions.The mixed term can oppose diagonal contributions; directional contractions avoid computing the full Hessian.Source: §7 · p. 10, §7.1 · p. 10

7.2 Most sampled large increases are first-order dominated

  • 17 of 19 accepted large total-loss increases in the uniform Pythia and SmolLM2 sample were first-order dominated, as were both uniform and all five validation-selected Qwen increases.No accepted natural-component path reversed a favourable initial direction.Source: Table 5 · p. 10, §7.2 · p. 10

7.3 History curvature can outweigh protective coupling

  • At update 9 of Pythia seed 53, positive history curvature was the largest contribution under both codebooks, while mixed curvature was protective but did not offset it.The independently integrated full-displacement curvature agrees with each remainder to about 2.8 × 10^-5 nats.Source: Table 6 · p. 11, §7.3 · p. 11
  • The two Qwen paths contain mixed curvature of both signs, but none meets the materiality criterion; one disjoint-answer path’s mixed term contributes about 7.7% of its loss increase.At the shared-answer path, MA < 0 while RA > 0.Source: §7.3 · p. 11
  • In the highlighted OLMo step, the initial projection favoured lower old-task loss, but the finite-step remainder reversed its direction.The integrated history contribution was +0.309796 nats and the current channel was −0.021920 nats.Source: §7.3 · p. 11

8 Discussion and Limitations

Optimiser history affects both old-answer production and ranking, so recovery in total loss can conceal different component-level trade-offs. The findings are bounded by the tested settings and by limits on causal interpretation and event prevalence.

8 Discussion and Limitations

  • Pythia reset branches recover answer mass despite worse conditional discrimination, while two natural SmolLM2 runs improve total loss despite losing answer mass.Assessing both loss components connects an intervention to the behaviour it preserves.Source: §8 · p. 11
  • The experiments cover sequential classification with single-token answers, a fixed task order, and model-specific learning rates; extension to open-ended generation remains untested.Paired codebooks and seeds sharing evaluation examples also mean runs are not fully independent.Source: §8 · p. 11
  • The age decomposition identifies which active gradient-age groups supply positive leakage projections, but does not isolate staleness as their cause.Its measured contributions include decay, magnitude, and bin width; orientation shares also depend on parameterisation and normalisation.Source: §8 · p. 11
  • Finite-path event prevalence is limited by the trajectories that pass numerical checks, sampling, and acceptance criteria.Accepted paths resolve channel interactions, but do not establish how common each event is in the population.Source: §8 · pp. 11–12

9 Conclusion

The conclusion links loss of old-answer mass to stored new-task gradients, their changing orientation, and their interaction with the current update. Supporting analyses use activation interventions and finite-path decompositions to examine individual updates and channel contributions.

9 Conclusion

  • In sequential-task experiments, old-answer probability can fall while discrimination improves; Adam’s history projections favour leakage while current-gradient projections oppose it.Older new-task gradients supply positive leakage contributions, and matched-state interventions connect stored optimiser information to answer-mass recovery.Source: §9 · p. 12

A Supporting registry and activation analyses

  • The supporting association-learning study follows individual optimiser updates and tests endpoint behaviour with activation interventions, controlled soft targets, and held-out prompt templates.For OLMo, the analysis fits a fixed activation-orientation map using calibration prompts and evaluates it on other entity identities.Source: A Supporting registry and activation… · p. 15, §A.1 · p. 15

A.2 A reversal along the update path

  • In the highlighted OLMo reversal, both channels initially point toward lower old-task loss, but the loss rises later along the displacement and accumulated nonlinear contribution outweighs the favourable start.Integrated history contributes most of the damage while the current-gradient channel remains beneficial overall.Source: §A.2 · pp. 15–16

A.3 Channel contributions across events

  • Across selected audited events, the dominant channel varies by model and event; Qwen3 also shows history either driving damage or protecting retention, depending on the update.In the selected Pythia, Qwen2.5 Instruct, and SmolLM2 events, history dominates, the current channel dominates, and the channels contribute comparably, respectively.Source: §A.3 · p. 16
  • A Pythia text-study path also reverses a favourable initial slope, while selected paths show both harmful and protective integrated history.The audited channel contributions reconstruct the total change along the selected old-task paths.Source: §A.3 · p. 16, Table 7 · p. 16

A.4 Activation rescue depends strongly on the model state

The appendix reports a strong but state-specific activation rescue in OLMo, then documents the association-learning, prediction, reproducibility, and event-analysis procedures used across the broader study.

A.4 Activation rescue depends strongly on the model state

  • At the highlighted OLMo endpoint, the fitted rotation recovers most of the old-task loss increase on held-out entities, restores two-answer accuracy, and outperforms the tested controls.Most held-out entities improve, and the new task benefits as well; the figure contrasts fixed channel states and endpoint changes.Source: §A.4 · p. 16, Figure 5 · p. 17
  • The fitted rotation improves history-only and full-update states but worsens the pre-update state, while effects elsewhere range from partial recovery to deterioration.Across other OLMo states, loss can improve without changing accuracy; tested Pythia rotations have much smaller endpoint effects, and some text-study effects are small or mixed.Source: §A.4 · pp. 16–17

A.5 Sequential association learning

  • Each association-learning registry assigns red and blue target probabilities to 32 entities, with disjoint prompt templates across splits but shared entity identities.The loss uses probabilities normalized over the full vocabulary, so it reflects both probability outside the two answers and their relative ordering.Source: §A.5 · p. 17
  • The sequential-learning setup trains task A for 600 updates and task B for 128, preserving Adam moments and its step counter across the task boundary.The model suite includes OLMo, Pythia, Qwen, SmolLM2, and Qwen3 checkpoints; all receive the same plain-text prompts.Source: §A.5 · pp. 17–18

C Prediction methods and scores

  • The registry predictors expand from loss features to projection, static geometry, and temporal vector features, with separate fitted predictors for each forecast lead.The feature sets contain 5, 6, 25, and 45 features; predictors use information available before the target update.Source: §A.6 · p. 18, C Prediction methods and scores · p. 20

A.6 Frozen prediction across checkpoints

  • The frozen-prediction analysis compares pre-update diagnostics with advance-warning forecasts, using OLMo-fitted predictors on other checkpoints and requiring retained evaluation data for old-task gradients.The forecasting timestamp precedes the previous update’s completion for registry predictors; the completed-update analysis uses a different timestamp.Source: §A.6 · p. 18

B Reproducibility

  • The reproducibility appendix records model revisions, fixed data and minibatch seeds, evaluation populations, and numerical settings for the experiments.All recorded registry-suite checkpoints use three seeds, 600 acquisition updates, and 128 second-task updates.Source: B Reproducibility · pp. 18–19

D Retention and selected events

  • In selected Qwen3 paths, the same sign reversal can accompany very different old-task loss changes, and task B’s change also differs across the two updates.At update 3, old-task loss changes by +0.000293927; at update 59, it changes by +0.075012094.Source: D Retention and selected events · p. 23

E Numerical integration and curvature audits

The numerical audits accept most sampled paths but leave some curvature measurements unresolved; the activation intervention shows that a striking rescue is not uniform across states and sites.

E Numerical integration and curvature audits

  • 32 of 36 task-specific path decompositions pass both integration and slope criteria; the four failures come from two updates and are excluded from mechanistic examples.One OLMo update reaches its refinement cap without passing quadrature, while one Pythia update fails the slope checks.Source: E Numerical integration and curvature… · p. 24, Table 17 · p. 24
  • Several OLMo curvature measurements remain unresolved: only five of eight standard records pass directional finite-difference checks, and two refined new-task measurements fail stability checks.Both refined old-task measurements at seed 2/update 11 resolve, but its new-task measurements do not.Source: E Numerical integration and curvature… · p. 24, Table 19 · p. 25
  • At the two refined sites of update 21, old-task curvature is approximately −127.073 and −127.871, while new-task directional curvature is approximately +20.523 and +27.130.The local contrast is compatible with either sign of the full loss change, which depends on the entire path.Source: E Numerical integration and curvature… · p. 24

F Activation intervention, controls, and topology

  • The rotation intervention patches the final-token residual output while holding downstream weights fixed, and the geometric checks show unchanged sampled Betti curves despite a short interval between sampling thresholds.The interval calculations retain H0 and H1 intervals; the highlighted event’s explicit interval matchings have small endpoint discrepancies.Source: F Activation intervention, controls,… · pp. 25–26
  • The highlighted layer-0 rotation rescue is accompanied by smaller, absent, or harmful effects elsewhere, and a mean-shift control beats the fitted rotation at one tested site.These endpoint comparisons limit generalization from the strongest selected case.Source: F Activation intervention, controls,… · p. 26, Table 20 · p. 26

G Text-task forecasts and interventions

The text-task studies document old-task impairment and mixed intervention outcomes, then examine matched-history continuations, path contributions, and rotation replications.

G Text-task forecasts and interventions

  • In Pythia seed 11, old-task restricted-answer accuracy falls from 82.8% after acquisition to 68.8% during task-B training, while old-task NLL rises from 0.598 to 2.909.The final full-vocabulary accuracy is 53.1%.Source: G Text-task forecasts and interventions · p. 26
  • The projection policy’s old-task loss lies within the matched random-schedule range in both seeds, so these comparisons do not show an advantage over the controls.The reported ranges are [3.485, 7.185] and [1.125, 3.269].Source: G Text-task forecasts and interventions · p. 27
  • Norm-matched reset branches have state-dependent effects: at seed 12/update 9, reset improves both final losses, while at seed 13/update 6, preserved and reset history produce different immediate A/B changes.The matching constrains only the first displacement; optimizer states evolve independently afterward.Source: G Text-task forecasts and interventions · p. 27, §G.1 · p. 28, Table 24 · p. 28

G.1 Detailed text paths, history branches, and rotation replication

H Optimiser calibration, confirmation, and numerical checks

The optimiser appendix documents calibration and confirmation conditions, immediate norm-matched probes, and pilot runs; its history reconstruction reports age-related leakage patterns on a limited trajectory subset.

H Optimiser calibration, confirmation, and numerical checks

  • The main optimiser confirmations cover two models, four optimizers, and three seeds, with one main run failing the old-task acquisition gate and a second failure in the Muon appendix confirmation.All six label controls pass; the main-confirmation count alone would miss the appendix failure.Source: H Optimiser calibration, confirmation,… · p. 28, Table 26 · p. 29
  • Calibration selects among fixed learning-rate candidates using an acquisition gate and a validation objective combining new-task loss with excess old-task loss.The gate requires at least 0.1 nat validation improvement and restricted-answer accuracy of at least 0.6.Source: H Optimiser calibration, confirmation,… · pp. 28–29
  • Across optimiser runs, same-update projection AUROC for Pythia ranges from 0.939 to 0.995, but sparse events and within-trajectory dependence constrain interpretation.The β1 = 0 forecast sequences contain 29, nine, and four events; combined features do not consistently surpass the latest-loss score.Source: H Optimiser calibration, confirmation,… · p. 29, Figure 6 · p. 30
  • The immediate reset probes pass all recorded state and norm-group audits, while 22 of 24 fixed-path curvature audits pass and none of the audited paths reverses a favourable initial direction into a harmful endpoint.The path audit covers selected fixed steps only.Source: H Optimiser calibration, confirmation,… · pp. 29–30

Appendix optimiser pilots.

  • The blocked-Shampoo hybrid’s single-seed pilot acquires the old task and ends with lower old-task loss, whereas the FP32-Muon hybrid fails the acquisition gate.The Shampoo pilot ends at A loss .370598 from .536864; Muon worsens old-task validation loss by 2.639396 nats.Source: Appendix optimiser pilots · pp. 30–31

I History reconstruction and cross-update geometry

  • The history replay uses matched 32-example test populations, and the age analysis reports a repeated signed pattern of leakage across recorded bins.Its seven Pythia and SmolLM2 trajectories are the first completed subset rather than a balanced sample of conditions.Source: I History reconstruction and… · pp. 31–32
  • The cross-time geometry analysis distinguishes changes in orientation, gradient magnitude, and channel magnitude, while its table reports absolute-contribution shares and median consecutive-update cosines.The table’s shares follow separate or nested attribution conventions.Source: I History reconstruction and… · p. 31, Table 29 · p. 32

J History interventions and forecasting

This section examines how resetting optimiser history and other interventions relate to old-task retention across model states and training conditions, alongside forecasting comparisons of recent validation-loss changes and geometric features. It also covers variable effects of step-reduction and activation interventions and paired-codebook results for disjoint labels, including changes in leakage, total loss, and restricted-answer accuracy.

J History interventions and forecasting

  • Across all six acquired Adam trajectories, block-matched resets improve final old-task loss on average; for Pythia, leakage improves by 0.227013 nats while confusion worsens by 0.099843 nats.Comparisons matched on new-task acquisition remain mixed: Pythia’s mean A-loss difference is +0.003580 at B-validation loss .75 and −0.025831 at 1.0.Source: J History interventions and forecasting · p. 32
  • The combined predictor beats the loss-history model in all eight trajectories by mean AUROC .074943 at the .05-nat threshold, but the latest loss increase beats it in seven.The combined model also improves Brier score and log loss in seven of eight trajectories; at horizon seven, its AUROC difference from the loss-only model is −.001966.Source: J History interventions and forecasting · pp. 32–33

K Event-level loss and intervention measurements

  • Large reversals occur at four reported updates, and all 24 trajectories end above their post-acquisition old-task loss anchor; at OLMo update 20, a layer-0 rotation raises loss by +0.044355.The reported reversals are OLMo seed 1 updates 21 and 76, Pythia-410M seed 3 update 18, and Qwen2.5 Instruct seed 1 update 96.Source: K Event-level loss and intervention… · pp. 33–34

L.1 Adam and momentum SGD

  • Adam’s first-moment history is separated from the current gradient, and grouping history contributions by age yields the bins used to analyse their variation across updates.For momentum SGD, the history channel is −η_tµz_(t−1) and the current-gradient channel is −η_tb_t; plain SGD has no history channel.Source: §L.1 · p. 35, §L.4 · p. 36

L.2 The straight-path identity and its curvature remainder

  • The finite-step analysis decomposes endpoint loss change into an initial slope and curvature remainder; positive curvature at one sampled point alone cannot determine the weighted curvature integral.A magnitude–orientation decomposition separates changes in gradient size, orientation, and their interaction, which sum to the same remainder.Source: §L.2 · p. 35, §L.3 · p. 35

M.1 Sequential association learning

  • The studies cover registry and text-task settings, with matched-state optimizer probes and a natural-history replay; forecasts use only information available before the next update.The registry measures 3,072 updates across eight checkpoints, while the natural-history replay covers seven of twelve text trajectories and separates its measurement population from source-trajectory test evaluation.Source: §M.1 · p. 36, §M.2 · p. 36, §M.3 · p. 37

N.1 Local geometry is informative, while forecasting follows temporal persistence

  • In text trajectories, the latest validation-loss change ranks next-event deterioration more accurately than the preceding-update projection in each seed; in every acquisition-passed Adam or momentum-SGD trajectory, it outperforms that projection.For the matched-start study, the combined predictor beats its loss-history baseline in all eight trajectories, yet the latest loss increase exceeds the combined model in seven.Source: §N.1 · pp. 38–39

N.2 History interventions change retention beyond displacement size

  • Reset effects depend on state: a matched reset improves final old-task loss across acquired Adam trajectories, while selected immediate probes show opposite retention effects; full registry trajectories also forget under plain SGD.In the SmolLM2 momentum-SGD probe, reset improves both tasks, whereas in an Adam probe preserved history produces greater improvement on both; the 32-update reset benefit is heterogeneous across forks and acquisition-matched comparisons.Source: §N.2 · p. 39

N.3 Forgetting usually begins in a locally unfavourable direction

  • A selected OLMo event ended with higher old-task loss despite favourable initial projections: integrated history contributed +0.309796 nats while the current gradient contributed −0.021920.Other audited paths were dominated by history, the current gradient, or comparable contributions; either channel could be harmful or protective at a selected state.Source: §N.3 · p. 40

N.4 Step-reduction policies produce variable trade-offs between retention and acquisition

  • Projection-triggered step reductions had mixed effects across two Pythia text seeds, and final old-task loss stayed within the matched-action-count random-schedule range in each seed.Loss-history triggers and a lower constant learning rate remained competitive; intervention timing and cost to new-task acquisition also matter when evaluating the policy.Source: §N.4 · p. 40

N.5 Probability leakage can dominate the loss of retained behaviour

  • In the disjoint-label control, full-vocabulary accuracy fell to zero despite correct old answers being selected for most examples when restricted to the old answer set.Answer-mass leakage dominated old-task loss, so the correct answer could still outrank its designated alternative while probability shifted elsewhere in the vocabulary.Source: §N.5 · p. 40

N.6 Activation interventions delimit the frame interpretation

  • A fitted activation rotation restored two-answer accuracy at one selected OLMo event while preserving activation norms, covariance spectrum, and distance-based persistence.The map helped history-only and full-update states but harmed the pre-update state; benefits at other OLMo sites and in Pythia were smaller, variable, or absent.Source: §N.6 · p. 40

P Finite-path sampling and numerical validation

P.1 Curvature of the paired amplification paths

  • In both paired amplification cases, history–history supplied the largest positive curvature contribution, while mixed terms were negative.These terms describe fixed directions along the actual full displacement, not the different path that training with history alone would induce.Source: §P.1 · p. 43

P.2 Functional accuracy and paired codebooks

  • Disjoint codebooks produced greater leakage and total-loss deterioration in all six pairs, with higher restricted accuracy in all six and lower full-vocabulary accuracy in five.The acquired old-task weights and selected examples were identical within each pair; the codebook, instructions, and new-task token identities changed together.Source: §P.2 · p. 43

P.3 Operational event classification

  • The operational classifier labels a path as nonlinear amplification when its remainder exceeds 0.01 nats and at least 25% of the observed increase, before testing first-order dominance.These are operational thresholds, not significance tests; the separate mixed-material flag requires an absolute mixed term of at least 0.01 nats and 25% of the observed increase.Source: §P.3 · p. 44

P.4 Qwen3 finite-path measurements

In the sampled Qwen3 paths, large total-loss increases were first-order dominated and associated with leakage, while curvature terms could reinforce or protect. The related work connects this analysis to task retention, gradient interference, loss geometry, answer competition, and representation change.

P.4 Qwen3 finite-path measurements

  • All seven sampled large increases in total loss were first-order dominated and showed large leakage increases; no selected test path had a large confusion increase.The samples were selected in different ways and do not estimate a common prevalence across models.Source: §P.4 · p. 45, Table 45 · p. 45
  • Mixed curvature can reinforce or protect: one disjoint-answer example contributed about 7.7% of the observed increase, while another shared-answer example had negative mixed curvature.First-order dominance can coexist with nonlinear terms of either sign; the full remainder in the shared-answer example remained positive but below the amplification cutoff.Source: Table 44 · p. 45, §P.4 · p. 46, Table 46 · p. 46
  • Across 36 held-out component records, the largest absolute difference between the full-displacement curvature integral and R was 4.99 × 10−6 nats.The largest absolute recorded residual in the direction sums was 2.37 × 10−9 nats.Source: §P.4 · p. 46

Q Extended related work

  • The analysis combines task-gradient constraints, optimiser memory, and path attribution to evaluate stored directions against components of retained behaviour.This combination is presented as the basis for evaluating stored directions against specific retained-behaviour components.Source: Q Extended related work · p. 47

Q.1 Retaining behaviour through parameter and functional constraints

  • Prior retention approaches connect behaviour to parameter importance, learning history, or preserved predictions; this analysis interprets stored gradients through their projections onto old-task loss components.Resolving history by origin and age distinguishes stored gradients that favour leakage from those that oppose it.Source: §Q.1 · p. 47

Q.2 Gradient interference and the direction of the applied update

  • Unlike methods that align task gradients, channel projections evaluate the signed applied displacement—including stored first moment and adaptive scaling—against total, confusion, and leakage gradients.Finite-path integration tracks how these sensitivities evolve between recorded parameter iterates.Source: §Q.2 · p. 47

Q.3 Momentum, gradient age, and retention

  • The related-work discussion covers momentum filtering, gradient age, and task-relative alignment, including MoFO, AdEMAMix, cautious optimisers, and optimiser-dependent task interference.Source: §Q.3 · p. 47

Q.4 Loss geometry and finite-path attribution

  • Finite-path attribution evaluates curvature along the applied update and separates diagonal from mixed Taylor-remainder contributions across history, current, and residual channels.The cited work relates loss geometry to curvature and finite learning rates; the path-attribution connection integrates channel derivatives along successive parameter iterates, whose integrals sum to the loss change when decomposition and integration are exact.Source: §Q.4 · pp. 47–48

Related papers

Loading 2610.03940v1…