PaperlayerView exact revision on arXiv

Source-linked AI summary

Sequential Pretraining Favors Large Models

Mohnish Harwani, Yujia Zheng

arXiv:2610.09611v1cs.LG

TL;DR

The paper asks whether large models’ advantages reflect only representational power or also robustness to adverse training effects such as primacy bias. It introduces Exposure Therapy, which exposes models early to samples from later data distributions, and reports improved late-data and overall pretraining performance up to the billion-parameter scale.

  • Problem

    The paper asks whether large models’ advantages come only from representing more training structure or also from robustness to early data that impairs later learning, a concern when pretraining presents heterogeneous data sequentially.

    Source: §1 · pp. 1–2

  • Method

    Exposure Therapy intermittently replaces ordinary updates with samples from later data distributions, which remain in the later training corpus.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    Exposure Therapy improves learning on late data distributions and overall foundation-model performance up to the billion-parameter scale, with negligible extra computational cost.

    Source: §1 · p. 2, §5.1 · p. 8

  • Takeaways & Limitations

    Some benefits of large models may arise from robustness to adverse training effects, and improved training algorithms can recover part of those advantages in smaller models.

    Source: §1 · p. 2, §7 · p. 8

    Pretraining experiments use only two data distributions, and computational cost prevents rigorous hyperparameter tuning for Exposure Therapy in foundation models.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

1 Introduction

The introduction frames data order as a constraint in pretraining and asks whether large models benefit from robustness to training effects as well as representational power. It defines primacy bias and introduces Exposure Therapy (ET) to improve learning from later data distributions.

1 Introduction

  • The paper identifies a gap in work on data order: prior studies mostly examine retaining learned information, not how order and capacity jointly affect learning new information.Large models’ capability advantages have motivated scaling despite substantial training and inference costs [Wei et al., 2022, Huang et al., 2026].Source: §1 · pp. 1–2
  • Large models may owe some advantages to robustness against training biases, not only to learning more representative solutions.The paper defines primacy bias as early distributions persistently impairing the fit of later ones.Source: §1 · p. 2
  • Pretraining often presents heterogeneous data sequentially rather than jointly, making data order a constraint rather than a design choice.Later stages may emphasize specialized domains such as code, mathematics, tool use, and agentic behavior.Source: §1 · pp. 1–2
  • ET sparsely replaces training updates with samples from distributions that models would otherwise encounter later, while retaining those samples in later training.The method aims to improve learning of new distributions and overall foundation-model performance, with negligible extra computational cost, up to the billion-parameter scale.Source: §1 · p. 2

2 Preliminaries

The paper measures how early exposure affects later target fitting by comparing training orders that share the same target-dominated continuation. It interprets a persistent performance gap as primacy sensitivity.

2 Preliminaries

  • The setup compares target-first training with OOD-first training, then gives both conditions the same target-dominated Phase 2 trajectory without resetting parameters, optimizer state, or schedule.OOD refers to a distribution distinct from the target being evaluated.Source: §2 · p. 3
  • Primacy sensitivity is the persistent target-performance gap between the target-first and OOD-first conditions.A positive value indicates that early OOD exposure persistently impairs target fit; the experiments approximate the limit by training Phase 2 until target performance converges.Source: §2 · p. 3
  • The experiments extend the controlled scale analysis to pretraining and introduce Exposure Therapy to improve robustness without increasing parameter count.Source: §2 · p. 3

3 The Data Order-Capacity Relationship

Controlled vision experiments isolate how training history and model capacity jointly affect learning new distributions. The results connect smaller models’ sensitivity to greater reuse of early-recruited units and the difficulty of adapting capacity later in training.

3 The Data Order-Capacity Relationship

  • The vision experiments isolate primacy bias for mechanistic analysis rather than replicating pretraining dynamics; the paper then extends the analysis to sequential foundation-model pretraining.Source: §3 · p. 3

3.1 Experimental Setup: Isolating Training-History Sensitivity

  • The controlled vision experiments vary MLP width on MNIST, Fashion-MNIST, and KMNIST, using real inputs with fixed random labels for the OOD phase.During Phase 2, every fourth update comes from a separately sampled OOD set; models train until target performance converges.Source: §3.1 · p. 3

3.2 Primacy bias decreases with model scale

  • Primacy sensitivity falls sharply with MLP scale: small OOD-first models underperform target-first models, while sufficiently overparameterized models become nearly insensitive to data order.Figure 2 compares final target performance under the two training orders across datasets and seeds.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 Representational overlap decreases with model scale

  • Small models show both stronger downstream impairment and greater reuse of units recruited during Phase 1; both decrease as scale increases.The pattern is qualitatively consistent across activation thresholds.Source: §3.3 · p. 4
  • Large models can circumvent capacity used for early distributions, while small models often must repurpose existing capacity, which can be difficult after plasticity declines.This makes adaptability after early learning relevant even when a smaller model has enough parameters to fit the target in isolation.Source: §3.3 · p. 4

4 Primacy Bias in Foundation Model Pretraining

The section examines how sequential pretraining can make capacity-constrained models less able to fit later distributions. Across 100M–1B models, training on another distribution first raises final English loss relative to English-first training.

4 Primacy Bias in Foundation Model Pretraining

  • Figure 3 links declining primacy sensitivity with less overlap between units recruited during early OOD training and those recruited during later target training.The correspondence supports the hypothesis that larger models route downstream learning through a less-overlapping set of units.Source: Figure 3 · p. 5
  • Pretraining often presents heterogeneous domains sequentially, so later code, mathematics, tool-use, or reasoning distributions must be learned after earlier data have shaped the model.This makes the impairment especially consequential when later data emphasize desirable capabilities.Source: §4 · pp. 4–5

4.1 Sequential-Domain Pretraining Protocol

  • At each scale, models train on English or one of four OOD corpora in the first half, then share a majority-English second phase; data and optimization budgets grow with model size.The setup tests sensitivity when models remain capacity-constrained relative to their training problems.Source: §4.1 · p. 5
  • Across 100M, 500M, and 1B models, OOD-first training produces substantially higher final English loss than English-first training.The magnitude of impairment varies across OOD corpora.Source: §4.1 · p. 5, Figure 4 · p. 6

5 Exposure Therapy

Exposure Therapy (ET) intermittently directs learning toward distributions that will receive substantial training later. In pretraining tests, it reduces later English-loss impairment and improves aggregate performance, while less overparameterized models require stronger exposure.

5 Exposure Therapy

  • ET intermittently replaces ordinary updates with examples from a distribution that would otherwise receive substantial training later, while leaving those data in the later corpus.Exposure during the highly plastic period is intended to reserve some capacity for later distributions.Source: §5 · p. 6
  • ET is applied throughout training because the duration of the high-plasticity period is generally unknown, avoiding the need to estimate when that period ends.The method is motivated by allocating structure during the early critical learning period.Source: §5 · p. 6
  • Across Yahoo Answers, Yelp, and Emotion, the exposure dose needed to recover 90% of target-first performance increases as model capacity decreases.The most capacity-constrained settings require 20%–30% exposure.Source: §5 · p. 6, Figure 5 · p. 7

5.1 Exposure Therapy in Foundation Model Pretraining

  • In the pretraining experiments, the exposure set is 10% of the target corpus, the exposure rate is 30%, and exposure updates remain active in both phases.The evaluation includes 1B models across code, mathematics, German, and Finnish OOD corpora, with FineWeb and TinyStories target distributions.Source: §5.1 · pp. 6–7
  • At 1B parameters, ET reduces subsequent English-loss impairment across tested OOD and English target distributions and lowers aggregate test loss across tested conditions.The aggregate comparison covers both the Phase-1 OOD corpus and the English target distribution.Source: Figure 6 · p. 7, §5.1 · p. 8

6 Limitations

The paper’s mechanistic evidence and pretraining evaluations have scope and setup limitations.

6 Limitations

  • Mechanistic evidence is limited to MLPs, where feature allocation is better understood, and why parameters are avoided or difficult to repurpose remains unresolved.The precise effect behind this avoidance is left for future work.Source: §6 · p. 8
  • Pretraining experiments use only two distributions and fixed ET hyperparameters, without rigorous identification of how to tune them for pretraining.Toy language models are used to study how suitable exposure strength varies with scale, but exact hyperparameters might not transfer.Source: §6 · p. 8
  • The main pretraining setup assumes some Phase-2 OOD signal, within-distribution shuffling, and useful OOD data before English; ablations show ET also helps when these assumptions are relaxed, sometimes less strongly.The authors note that the findings may not extend to all forms of pretraining.Source: §6 · p. 8

7 Conclusion

The paper identifies primacy bias as persistent impairment in fitting late data after early exposure, and finds that overparameterized models can avoid heavily recruited early parameters. In sequential pretraining, Exposure Therapy (ET) introduces late data early to improve capacity allocation and recover some benefits of scale algorithmically.

7 Conclusion

  • Overparameterized models can circumvent parameters heavily recruited for early distributions, which may be difficult to repurpose as plasticity declines.The paper presents this as a mechanism for greater robustness to primacy bias.Source: §7 · p. 8
  • Primacy bias is persistent impairment in fitting late data due to exposure to early distributions.The conclusion examines this effect under sequential pretraining, where data distributions are encountered over time.Source: §7 · p. 8
  • Exposure Therapy introduces a subset of late data early in pretraining to promote more efficient capacity allocation across distributions.The paper reports that ET improves performance on late distributions and overall capability up to the billion-parameter scale, with negligible extra computational cost.Source: §7 · p. 8

A Primacy Sensitivity in Toy Experiments

The toy experiments compare target-first and OOD-first training while controlling the shared training procedure. Their setup uses a fixed random-label corruption for early OOD learning and shared target-dominated training afterward.

A Primacy Sensitivity in Toy Experiments

  • Paired target-first and OOD-first runs use the same model scale, random seed, architecture, initialization distribution, optimizer, and total updates.They differ only in the Phase-1 objective, and parameters, optimizer state, and learning-rate state continue across the phase boundary.Source: A Primacy Sensitivity in Toy Experiments · p. 12
  • The toy OOD objective pairs real inputs with fixed random targets, corrupting the input–target mapping rather than the input marginal.In OOD-first runs, all Phase-1 updates draw from fixed corruption set A; target-first runs instead use the target task.Source: A Primacy Sensitivity in Toy Experiments · p. 12
  • During Phase 2, both conditions use the same schedule: one OOD update for every three target updates.OOD updates use separately sampled corruption set B with newly generated random targets; examples are sampled with replacement within each source.Source: A Primacy Sensitivity in Toy Experiments · p. 12
  • Vision experiments use two-hidden-layer MLPs on MNIST, Fashion-MNIST, and KMNIST, with separate output heads and shared hidden representations.Training runs for 30,000 Phase-1 updates and 120,000 Phase-2 updates; the authors report this duration is appropriate for convergence.Source: A Primacy Sensitivity in Toy Experiments · p. 12
  • Toy models use Adam and cross-entropy loss with a constant learning rate.The passage does not give the rate value.Source: A Primacy Sensitivity in Toy Experiments · p. 12

B OOD-Set Accuracy Across the Full Parameter Ladder

Small models can fit the early OOD mapping during Phase 1 yet show much lower accuracy on it after target-dominated Phase 2. The overlap analysis compares hidden units recruited by OOD and target distributions across phases.

B OOD-Set Accuracy Across the Full Parameter Ladder

  • Small models’ accuracy on the original Phase-1 OOD set drops much lower after target-dominated Phase 2, despite high Phase-1 accuracy.Figure 7 tracks parameter count horizontally and accuracy on the same OOD examples vertically; pale and dark bars mark the ends of Phases 1 and 2, respectively.Source: Figure 7 · p. 12, B OOD-Set Accuracy Across the Full… · p. 12

C Overlap Mechanism

  • The overlap analysis compares usable neurons before and after Phase 2 to examine how early learning can limit capacity for target distributions.Activation snapshots are taken at each phase’s end using probe data from the OOD and target distributions.Source: C Overlap Mechanism · p. 13
  • A unit’s activation score is its mean absolute post-activation response on the corresponding phase’s probe set.The units are post-ReLU hidden activations in MLPs.Source: C Overlap Mechanism · p. 13
  • The analysis selects each phase’s top-f units and measures their cross-phase overlap with a Jaccard index, averaged across hidden layers.Top-f selection and overlap are computed independently within each hidden layer.Source: C Overlap Mechanism · p. 13

D Pretraining Experimental Details

The pretraining experiments compare English-first and OOD-first byte-level Transformers across three model scales and several English–OOD corpus pairs. ET intermittently replaces scheduled updates with batches from a fixed set of late-training examples, without adding data or optimization steps.

D Pretraining Experimental Details

  • The pretraining experiments use byte-level causal Transformers at approximately 100M, 500M, and 1B parameters, comparing English-first and OOD-first training across two English and four OOD corpora.The OOD corpora are code, mathematics, German, and Finnish; English corpora are FineWeb and TinyStories.Source: D Pretraining Experimental Details · pp. 13–14
  • Models use a 256-token byte vocabulary, sequence length 512, and seven-layer causal Transformer architecture.Model width is scaled to obtain the three parameter counts; the feed-forward activation is GeLU, with no dropout.Source: D Pretraining Experimental Details · p. 13
  • Training has two equal-length phases, with total optimization steps increasing by model scale; only Phase 1 differs between English-first and OOD-first conditions.Phase 2 uses the same fixed 75% English and 25% OOD mixture, implemented as three English updates per OOD update.Source: D Pretraining Experimental Details · pp. 13–14
  • Evaluation reports held-out English loss and aggregate loss over held-out English and OOD data.The evaluation uses 100,000 bytes of withheld random byte-sequences, split evenly between the distributions seen during training.Source: D Pretraining Experimental Details · p. 14

E Exposure Therapy Details

  • ET fixes an exposure set of s distinct late-training examples and, at each step, replaces an ordinary update with an exposure update at rate a.Exposure updates sample batches from the set independently; they replace rather than supplement ordinary updates, so total optimization steps are unchanged and no data are added or withheld.Source: E Exposure Therapy Details · pp. 14–15
  • ET’s exposure-set size estimates how much capacity to reserve for late training, while its exposure rate controls how strongly training reinforces that reservation.The authors note that smaller models may need a more accurate target-distribution estimate and more frequent reinforcement.Source: E Exposure Therapy Details · p. 15

F Exposure Therapy and Model Scale Experimental Details

The section defines a relative accuracy measure for Exposure Therapy and a minimum tested dose threshold, then uses that threshold to compare dosage needs across model scales.

F Exposure Therapy and Model Scale Experimental Details

  • M_ET compares Exposure Therapy accuracy with target-first accuracy: 0.75 means 75% of the reference, 1 matches it, and values above 1 exceed it.The comparison is defined for a model with N parameters and ET dosage a and exposure-set size c.Source: F Exposure Therapy and Model Scale… · p. 15
  • The minimum ET dose is the smallest tested dosage that reaches 90% of target-first accuracy.It is equivalently the smallest dosage for which M_ET is at least 0.9.Source: F Exposure Therapy and Model Scale… · p. 15
  • The experiments examine how the minimum required ET dose changes across model scales; the reported details concern toy language models.Section 5 demonstrates that exposure dosage decreases as models become more overparameterized.Source: F Exposure Therapy and Model Scale… · p. 15

G Exposure Therapy Pretraining at 100M and 500M Scales

ET improves sequential pretraining outcomes across model scales and reverse-order settings. Ablations find benefits with shuffled Phase-2 mixtures, while target-only late training produces distribution-dependent effects.

H Ablation Experiments Under Varying Pretraining Conditions

G Exposure Therapy Pretraining at 100M and 500M Scales

  • Across 100M, 500M, and 1B models, ET reduces English-loss impairment and aggregate loss relative to unregularized OOD-first training.The evaluation uses FineWeb and TinyStories as English targets, with code, mathematics, German, and Finnish as OOD distributions.Source: G Exposure Therapy Pretraining at 100M… · p. 15, Figure 8 · p. 16

H.1 Reverse Order Pretraining

  • When broad English data comes first, ET generally improves late specialized-distribution learning and consistently improves aggregate performance across model scales.The reverse-order experiments include code, mathematics, German, and Finnish; the figure compares target-first, English-first, and English-first with ET.Source: §H.1 · pp. 15–16, Figure 9 · p. 17

H.2 Persistent Effects to Retain Early Structure

  • The experiments mix a minority of OOD data into late learning to provide a weak signal for retaining earlier structure.The authors note that naive training over two distributions can fail to retain early structure even at billion-parameter scale in these small experiments.Source: §H.2 · pp. 16–17
  • In target-only late training, ET also improves retention of earlier structure, reducing catastrophic forgetting while improving late learning.The authors leave further analysis of ET's effect on catastrophic forgetting to future work.Source: §H.2 · pp. 17–18
  • With shuffled Phase-2 mixtures, ET reduces target and aggregate loss; with target-only late training, it helps German and Finnish more than code and mathematics.Under target-only late training, code and mathematics show little target-loss change, while aggregate loss falls across all four distributions.Source: §H.2 · p. 17, Figure 10 · p. 18
Loading 2610.09611v1…