PaperlayerView exact revision on arXiv

Source-linked AI summary

When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

Martin Hofmann, Patrick Mäder

arXiv:2610.09621v1cs.LG

TL;DR

The paper asks whether training history predicts future learning beyond the current state. It tests this with a response probe and a screen forecasting final run error. History did not improve prediction of probe response, but helped forecast end-of-run error early in a run, while little of the run had been observed.

  • Problem

    Prior work shows that training paths shape later learning, but whether history predicts future learning beyond a measurement of the current state remains open.

    Source: §1 · p. 1

  • Method

    The paper compares history-based predictors with checkpoint measurements for a probe of response to further training and forecasts of final error from partial runs.

    Source: §1 · p. 2

  • Results

    History did not improve prediction of response to further training, but helped forecast final run error early and was no better than current validation error after a fifth of the run.

    Source: §6 · pp. 7–8

  • Takeaways & Limitations

    In the tested settings, history was informative only while the current state was not yet informative about the target.

    Source: §8 · p. 9

    The evidence is exploratory and limited to small networks, synthetic or small image tasks, short horizons, six test histories, and 13 designed forecasting families.

    Source: §7 · p. 9

Abstract

from arXiv · show

Networks that behave alike now can still learn differently when training continues. Work on loss of plasticity and critical periods shows that the path to a state shapes what follows; it does not show whether the path carries information that a measurement of the state itself misses. We ask when the training history of a network predicts its future learning better than its current state. In a main study, small multilayer perceptrons were trained under three history regimes (42 histories), and future learning was measured at four checkpoints by a short probe: a copy of the network trained for 100 updates on a new task. Before the prediction result was read, the protocol checked the probe. It responded monotonically to a function-preserving rescaling of hidden units, repeated measurements agreed (intraclass correlation 0.940, [0.903, 0.997], in the least reliable class, mean of three repeats), and a re-initialisation of units was visible directly after it but not 100 to 200 updates later. A history state of at most four dimensions did not improve on a calibrated model of the current state (gain -21.4%, 90% interval [-91.9, 8.1]; required in advance: 10%). A companion screen on 1,560 synthetic regression runs asked the same question for a target further away, the final error of the run. There, history models forecast better than the current validation error after 12 of up to 240 epochs (compact state 30.3%, [15.8, 39.4], a contextual comparison) and were not distinguishable from it after 48. In both studies the history was informative only while the current state was not yet informative about the target; this reading was formed after the results.

1 Introduction

Prior work shows that training paths shape later learning, but leaves open whether history predicts future learning beyond a measurement of the current state. The paper tests this with a short response probe and a forecast of end-of-run error.

1 Introduction

  • The central question is whether training history predicts future learning beyond information already available in the current state.The distinction matters because checkpoint-only monitoring can be applied to stored models, whereas history-dependent monitoring must be carried through the run.Source: §1 · p. 1
  • The study compares history information with the current state for two targets at different distances: response to further training and final run error.The near target is measured by a probe; the far target is forecast after observing a growing portion of a run.Source: §1 · p. 2
  • The near-target test required a history state of at most four dimensions to improve held-out prediction error by at least 10% over the checkpoint-only baseline.The criterion also required a bootstrap lower bound above zero and a mean gain of at least 5% when a history regime was held out.Source: §1 · p. 2
  • The main study checked that the probe registered known manipulations and produced repeatable values before interpreting its prediction result.The paper also reports a forecasting screen that varies how much of the run has been observed.Source: §1 · p. 2

2 Background

Prior work documents effects of training paths on later behavior and uses partial training runs to forecast outcomes. This paper addresses the less-tested comparison with calibrated measurements of the current state.

The path and the state

  • Continued training can reduce learning of new targets, and early training conditions can have lasting effects [Lyle et al., 2023].Prior work also reports worse generalization for warm-started than freshly initialized networks (Ash and Adams, 2020).Source: §2 · p. 2
  • Earlier studies compare training paths and propose state diagnostics, but do not test whether path summaries predict future learning beyond those diagnostics.This identifies the comparison addressed by the paper.Source: §2 · p. 2

Forecasting from a partial run

  • Learning-curve extrapolation supports early stopping in hyperparameter optimization and architecture search [Swersky et al., 2014].Other predictors use weights, gradients, and training telemetry, but are rarely compared with calibrated current-state predictors under whole-family holdouts.Source: §2 · p. 2

Measuring before interpreting a null

  • The paper uses repeatability, positive controls, and dose series to validate measurement before interpreting a null prediction result.Repeatability is quantified with intraclass correlation.Source: §2 · p. 2

3 Main study: design

The main study compares history-based predictors with checkpoint measurements using a probe of future learning. It specifies the network histories, probe, predictors, validation checks, and decision criteria before the production run.

3 Main study: design

  • At four anchor-task checkpoints, a copy receives 100 updates on a new task; history models add past events to checkpoint diagnostics, while the baseline uses diagnostics alone.The configuration was frozen before execution.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Networks and histories

  • The study uses two-hidden-layer perceptrons trained on synthetic tasks, with 12-task histories in repetitive, diverse, or conflicting regimes followed by a shared anchor phase.Histories vary by initialization and training batches; six additional re-initialization histories serve only as a probe control.Source: §3.1 · pp. 2–3

3.2 The probe

  • The probe records loss reduction after a checkpoint copy trains for 100 updates on a challenge task, measuring response to further training rather than final run performance.Challenges are near, conflicting, or far from the anchor direction; one instance is held out for testing and another measurement is repeated independently.Source: §3.2 · p. 3

3.3 Predictors

  • Predictors share a decoder: the checkpoint-only baseline uses 748 diagnostic features, while the compact history model encodes events in a recurrent state of one, two, or four dimensions.A 32-unit gated recurrent unit is the reference without the dimension limit; neither regime labels nor probe responses are inputs.Source: §3.3 · p. 4

3.4 Checking the probe

  • Probe validity is checked by rescaling hidden units without changing the network function and by comparing response maps from unchanged and re-initialized states.The unchanged-state comparison estimates the noise floor using the same or an independent stream of training batches.Source: §3.4 · p. 4

3.5 Criteria and protocol

  • The fixed criteria require repeatability and a positive-control effect before testing whether compact history improves prediction over the baseline.Prediction requires at least 10% gain with a 90% bootstrap lower bound above zero and at least 5% mean gain under a held-out regime; the protocol was exploratory and locally version-controlled.Source: §3.5 · p. 4

4 Main study: results

In the 42-history main study, the probe’s first-stage reliability criterion was not fully met, while a revised positive control passed. A history model did not improve prediction of probe response over the calibrated current-state model.

4.1 Stage 1: the probe before revision

  • The initial probe check found conflict-response reliability of 0.781 [0.656, 0.988], below 0.80, so the fixed-order conclusion was neither supported nor rejected.The positive control stayed below its threshold at both tested doses.Source: §4.1 · p. 5

4.2 Stage 2: dose response and repeats

  • In the second-stage control, the median effect rose from 0.208 to 0.684 to 1.447 [1.192, 1.490], and the criteria were met.The three-repeat mean reliability was 0.940 [0.903, 0.997].Source: §4.2 · p. 5
  • The re-initialisation control separated histories directly after re-initialisation but not 100 or 200 updates later, so its registered prediction was not met.At 1500 updates the effect was 0.39 [0.16, 0.73], while at steps 1300 and 1400 it was −0.06 and −0.05.Source: §4.2 · p. 5

4.3 The history did not improve on the current state

  • The compact history state’s gain was −21.4% [−91.9, 8.1] over B1, below the required 10%; its held-out-regime mean gain was −80.3%.The GRU’s gain was −23.8% [−96.6, 30.0].Source: §4.3 · p. 6
  • The GRU fit training histories better than B1 but performed worse on validation, pointing to overfitting with 24 training histories.The fixed-summary extension stayed slightly worse on test (gain −0.9%, [−2.4, 1.3]); six test histories could resolve only large gains.Source: §4.3 · p. 6

5 Forecasting screen: a target further away

This screen tests whether training history predicts final run error better as more of a run becomes observable. History helped early relative to current validation error, but was not distinguishable from it after 48 epochs.

5 Forecasting screen: a target further away

  • The screen used 1,560 regression runs from 13 function families and forecast final test error after observing 12, 24, or 48 epochs.Each family was held out in turn, with forecasters trained on runs from the other twelve.Source: §5 · p. 6
  • The intervals are 90% bootstrap intervals over 13 families, without correction for multiple comparisons.The planned comparison with current validation error was contextual.Source: §5 · p. 7
  • After 12 epochs, the compact state improved over current validation error by 30.3% [15.8, 39.4]; after 24 epochs, the gain was 20.9% [5.3, 30.9].The 12-epoch compact-state gain over the snapshot model was 13.1% [2.4, 23.7].Source: §5 · p. 7
  • At 48 epochs, the Transformer gained 15.1% [0.8, 28.4] over the snapshot model, but only 8/13 families met the breadth criterion.At that epoch, compact-state gain over current validation error was 0.8% [−19.8, 15.2], not distinguishable from it.Source: §5 · p. 7

6 Discussion

The response probe was validated, but neither a compact history summary nor an unrestricted or regularised history model improved prediction of the near target over the current state. History helped forecast the run’s final error early, and the discussion proposes that its value depends on target distance and timing, while marking that connection as post hoc.

The near target

  • The compact history summary did not beat the current-state model for the response target; the probe’s validation makes this null interpretable.The unrestricted GRU overfit, and a regularised summary added nothing.Source: §6 · p. 7

The far target

  • After 12 epochs, history models improved forecasts over current validation error; by 48 epochs, that comparison was not distinguishable.Figure 4 reports the forecasting screen; after 12 epochs the compact state’s improvement was 30.3% [15.8, 39.4], while at 48 epochs the second registered threshold was not met.Source: Figure 4 · p. 8, §6 · p. 8

The path and the state

  • In these studies, history was informative while the current state was not yet informative about the target; this connection was formed after the results.For the far target, history helped early in the run; for the near target, the probe read the response to the next 100 updates.Source: §6 · pp. 7–8

Scope and future work

  • The findings do not establish general state sufficiency or apply to large models or language modelling.The discussion also notes that the probe no longer separated re-initialisation histories 100 to 200 updates later, and proposes testing immediate post-task probes and per-history task sequences.Source: §6 · p. 8

7 Limitations

The evidence is exploratory and limited in sample size, registration, reliability, and task scope.

Sample size

  • The prediction test used six test histories and three task sequences, while the forecasting screen used 13 designed families.The paper characterises the combined evidence as exploratory.Source: §7 · p. 9

Order and registration

  • Protocols were frozen in local version control rather than registered externally, and the second measurement stage followed knowledge of the first prediction results.The second-stage plan was fixed before new data and changed no prediction criterion, fitted model, or prediction.Source: §7 · p. 9

Reliability

  • The frozen reliability criterion for a single measurement was not met for one class; the value above criterion was reported for the mean of three repeats.Source: §7 · p. 9

Scope

  • The studies cover small networks, synthetic or small image tasks, and short horizons; post hoc screen elements are marked in table 3.Source: §7 · p. 9

8 Conclusion

The conclusion is that compact history summaries did not improve near-target response prediction, while history helped forecast final error only early in a run.

Near target

  • A compact history summary did not predict response to further training better than a calibrated current-state model read through a previously checked probe.Source: §8 · p. 9

Far target

  • History helped forecast a run’s final error only while little of the run had been observed.Source: §8 · p. 9

Interpretation

  • In the tested settings, history was informative only while the current state was not yet informative about the target.This connection between the studies was formed after the results.Source: §8 · p. 9

Use of AI-assisted tools

The paper reports the use of AI assistants and documents protocol choices, measurement checks, baselines, and diagnostic analyses for the main study.

Use of AI-assisted tools

  • AI coding and writing assistants supported research code, run orchestration, analysis, literature lookup, and drafting; the authors retain responsibility for the final work.Source: Use of AI-assisted tools · p. 9

A Protocol history and deviations

  • Production used the final protocol versions, with all 42 main-study histories and 900 reserved responses analysed from completed records.For the forecasting screen, the reported version used a final test set never observed during training; overflow cells were reported without clipping or reruns.Source: A Protocol history and deviations · p. 14

B Further details of the main study

  • The fixed criteria included known-answer and repeatability checks, transfer, compression, and a positive control with function preservation verified to 10^-6 output tolerance.The criteria also included nonlinear and task-geometry baselines and stability of response-map dimension.Source: B Further details of the main study · p. 14
  • Neural predictors used a fixed learning-rate and weight-decay grid, validation-based early stopping, and five initialisations; test values were not used for refitting.State dimension was selected as the smallest within one standard error of the best validation error.Source: B Further details of the main study · p. 14
  • At γ = 0.5, the control effect was positive in 5/6 histories; at γ = 2, it was 0.753 after the first update and 0.044 at step 100.The second-stage production control was positive in 9/12, 11/12, and 11/12 histories across its three doses.Source: B Further details of the main study · p. 14
  • The third-repeat reliability was 0.940 [0.903, 0.997] across 24 checkpoints; the two-repeat value was a projection, not a criterion pass.The single-measurement criterion was not met for one class.Source: B Further details of the main study · p. 14, Figure 5 · p. 15
  • The task-geometry baseline error was 1.387, between B1 and history models, while adding the regime label to B1 yielded a −0.03% gain.The nonlinear baseline error was 2.149, worse than B1, making that comparison uninformative.Source: B Further details of the main study · p. 14
  • Of 670 pairs close in current-state features, all but one were excluded because predicted responses differed; the remaining pair’s measured responses were close.The diagnostic sought at least eight pairs with close state features and predictions but clearly different measured responses.Source: B Further details of the main study · p. 15

C Further results of the forecasting screen

With 200 training runs, history models forecast better at epochs 12 and 24 but not 48; reproducibility materials describe checks on generated results and analysis records.

C Further results of the forecasting screen

  • With 200 training runs, history models keep their advantage at epochs 12 and 24 but are worse than the current value at epoch 48.With 100 runs, snapshot and Transformer models produced extreme forecasts in two held-out families; with all families available for training, the compact state was no better than the snapshot model.Source: C Further results of the forecasting… · p. 15
  • The forecasting screen used designed families, unmatched forecaster capacities, and a fixed stopping rule rather than convergence for runs reaching the epoch limit.The intervals describe stability across the designed set; 832 runs reached the epoch limit.Source: C Further results of the forecasting… · p. 15

D Reproducibility

  • The reproducibility audit checks source hashes, defined macros, citation entries, placeholders, and typed decimal values against design constants.The paper reports that result numbers, tables, and figures are generated from source result records and scripts.Source: D Reproducibility · pp. 15–16
  • The paper records frozen configurations, commits, analysis-file hashes, and the forecasting screen’s bootstrap seed and draw count.The forecasting screen used seed 13032026 with 20,000 bootstrap draws.Source: D Reproducibility · p. 16

E.1 CIFAR-100 subsets with two architectures

In the CIFAR-100 screens, history-based forecasting did not consistently outperform calibrated or snapshot current-state predictors. Results were bounded by a small number of held-out blocks and, in the class-incremental screen, nearly constant outcomes.

E.1 CIFAR-100 subsets with two architectures

  • The CIFAR-100 screen used 72 ResNet-18 and VGG-11 runs across 12 tasks, forecasting final test accuracy after 6, 12, or 24 epochs.A task block was held out; the reported reservoir was the best of ten arms on test folds, and calibrated current-accuracy baselines were added after the first analysis.Source: §E.1 · pp. 16–17
  • Reservoir errors were 0.0684, 0.0661, and 0.0496 at epochs 6, 12, and 24, versus 0.0669, 0.0492, and 0.0374 for quadratic current-accuracy prediction.The reservoir was no better than the quadratic predictor at any horizon; it beat linear calibration only at epoch 6 and beat an order-free trace summary in all six comparisons.Source: §E.1 · p. 17, Table 4 · p. 17, Figure 6 · p. 17
  • The CIFAR-100 forecasting screen was limited to three held-out blocks from one image corpus, with fixed reservoir settings and a post hoc baseline.The design had no estimate of uncertainty because only three blocks were held out.Source: §E.1 · pp. 16–17

E.2 Class-incremental CIFAR-100

  • The class-incremental screen used 56 runs across seven training regimes, forecasting five outcomes after up to 20 updates on each new class.It held out a complete class order, leaving four units for inference; the design, including multiple-test correction, was fixed before the runs.Source: §E.2 · pp. 17–18
  • In the class-incremental screen, the snapshot model had the lowest mean error through 10 updates and calibrated current values did at 20; no difference survived multiple-test correction.The GRU beat calibrated current values by at most 0.06 points through 10 updates but never beat the snapshot model; the compact state was worse than calibrated values at every horizon.Source: §E.2 · p. 18, Table 5 · p. 18
  • Nearly constant class-incremental outcomes left little power to distinguish forecasters, and low forecast errors largely reflected calibration.Across 5,544 class stages, most new-class accuracies were 100% and most old-class accuracies were zero; replay delayed but did not prevent the collapse.Source: §E.2 · p. 18
Loading 2610.09621v1…