PaperlayerView exact revision on arXiv

Source-linked AI summary

Replay on Demand: An Emergent Curriculum for Balancing Adaptation and Forgetting in Continued Pretraining

Lukas Thede, Shengzhuang Chen, Stefan Winzeck, Matthias Bethge, Zeynep Akata, Jonathan Richard Schwarz

arXiv:2609.40089v1cs.LG

TL;DR

Continued pretraining must adapt language models to new domains while retaining previously acquired capabilities, but fixed replay allocates training independently of actual retention needs. Replay on Demand (RoD) lets adaptation and replay examples compete using the model’s learning and forgetting signals. Across domains, model families, and scales, it reaches or improves upon fixed replay’s adaptation–forgetting frontier without a predefined replay ratio.

  • Problem

    Continued pretraining can degrade pretrained capabilities, while fixed replay allocates compute between adaptation and retention before the model’s actual retention needs are known.

    Source: §1 · p. 1

  • Method

    RoD jointly ranks adaptation examples by remaining learning potential and replay examples by observed forgetting, letting both compete for a shared training budget.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Across domains, model families, and scales, RoD reaches or improves upon the adaptation–forgetting frontier of fixed replay without a predefined replay ratio.

    Source: §5 · p. 9

  • Takeaways & Limitations

    RoD targets vulnerable sources and increases replay as forgetting emerges, supporting a strong stability–plasticity trade-off without prescribing replay in advance.

    Source: §5 · p. 10

    RoD’s advantage over fixed replay is smaller for task-based forgetting, which the authors attribute to proxy replay loss being less aligned with the evaluated capabilities.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilities. Replay can mitigate this trade-off, but fixed replay mixtures allocate training independently of the model's actual retention needs. We introduce Replay on Demand (RoD), which instead derives the replay allocation from the model's learning dynamics. RoD jointly prioritizes adaptation samples by their remaining learning potential and replay samples by their observed forgetting. Their competition for a shared training budget yields an online curriculum that determines what to train on at each step. Across models, scales, and adaptation domains, RoD reaches or improves upon the adaptation-forgetting frontier of tuned fixed-replay baselines and model merging without prescribing a replay allocation in advance. Replay concentrates on sources that are more vulnerable to forgetting and dynamically increases and redistributes as forgetting emerges during training. Together, our results show that replay can be allocated online from the model's evolving state, targeting what is needed, when it is needed.

1 Introduction

Continued pretraining must balance adaptation to a new distribution with retaining pretrained capabilities, but fixed replay must allocate compute before actual retention needs are known. RoD uses the model’s learning and forgetting dynamics to allocate replay, reaching or improving upon tuned fixed-replay and model-merging frontiers across the evaluated settings.

1 Introduction

  • Continued pretraining adapts models to new distributions but can degrade capabilities acquired during pretraining, creating an adaptation–retention trade-off.Replay can mitigate forgetting, but under a fixed budget it displaces adaptation data, while maintaining both requires additional compute.Source: §1 · p. 1
  • Fixed replay mixtures can spend compute on well-retained knowledge while leaving more vulnerable knowledge underprotected, even as retention needs change during training.The forgetting is heterogeneous across capabilities and parts of the pretraining distribution.Source: §1 · p. 2
  • RoD jointly ranks adaptation examples by remaining learning potential and replay examples by forgetting, allowing both to compete for a shared budget and dynamically changing replay’s share and composition.As adaptation learning potential decreases and forgetting emerges, replay becomes more competitive.Source: §1 · p. 2, Figure 1 · p. 2
  • Across legal and German continued pretraining, Nemotron and Qwen models, and 4B–35B scales, RoD reaches or improves upon tuned fixed-replay and model-merging adaptation–forgetting frontiers without a prescribed replay allocation.RoD directs replay toward more-forgotten sources and increases and redistributes it as retention needs emerge.Source: §1 · p. 2

2 Related Work

Prior work studies replay, adaptive mixtures, and model-dependent data selection in continued pretraining and continual learning. RoD’s distinction is to jointly allocate compute between adaptation and retention from the model’s evolving state.

2 Related Work

  • Continued pretraining adapts pretrained language models to new data while aiming to preserve prior capabilities; replay is common, and allocation of compute between adaptation and retention is a central design choice.Related approaches include source–target mixture scaling laws, adaptive mixtures, and replay selection or scheduling based on utility or learning state.Source: §2 · p. 2
  • Reducible-loss methods use model-state signals to select data, but the cited approaches do not jointly allocate compute between adaptation and retention.RHO prioritizes examples by reducible loss relative to a reference model (Mindermann et al., 2022), while CSReL applies reducible-loss selection to replay examples.Source: §2 · p. 3
  • RoD jointly determines replay amount and composition by competing adaptation and replay examples, unlike mixture-optimization or dedicated replay-selection methods.Its adaptation reference estimates learnability, while the pretrained model measures degradation.Source: §2 · p. 3

3 Method

RoD scores adaptation and replay examples using source-specific reducible losses, then selects the highest-scoring candidates across both sources under one training budget. This makes replay amount and composition respond to the model’s evolving learning and forgetting state.

3 Method

  • RoD is an online data-selection method that dynamically allocates training compute between adaptation and replay by scoring both sources and letting candidates compete for a shared budget.The selection determines both how much and what to replay throughout training.Source: §3 · p. 3

3.1 Problem Setup

  • Rather than fixing the adaptation–replay fraction in advance, RoD selects k examples from the union of both candidate sets according to current utility.The setup defines θ0 as the pretrained model, θt as the model after t updates, and uses token-normalized language-modeling loss.Source: §3.1 · p. 3

3.2 Source-Specific Reducible Loss

  • RoD uses one reducible-loss formulation with source-specific references: an adaptation specialist estimates remaining learning potential, while θ0 measures replay degradation.Adaptation scores decrease as examples are learned; replay scores are positive where performance has degraded since the start of adaptation.Source: §3.2 · pp. 3–4
  • The source-specific construction approximates the original RHO objective separately for adaptation and replay, rather than using one reference trained on a mixed adaptation–retention objective.The paper points to section A.1 for the full derivation and assumptions.Source: §3.2 · p. 4

3.3 Joint Adaptation and Replay Selection

  • RoD selects the globally highest-scoring k candidates; under a local marginal-utility approximation, replay share emerges from relative demand rather than a predefined ratio.The method’s interpretation depends on the stated local approximation and its assumptions.Source: §3.3 · p. 4
  • A larger candidate multiplier gives more candidates a chance to compete at greater scoring cost; reference losses can be cached, but current-model scoring requires an inference pass each step.Experiments balance the candidate pools and use m = 2 by default; the selected batch then receives a standard training pass.Source: §3.3 · p. 4
  • Replay scores start at zero when adaptation begins, then rising degradation makes replay more competitive and shifts selection toward more-forgotten examples.The method adapts both replay share and replay composition as learning and forgetting change.Source: §3.3 · p. 4

4 Experiments

Experiments compare RoD with no-replay, fixed-replay, and model-merging baselines across legal and German continued pretraining in Nemotron and Qwen models. They assess adaptation alongside distribution- and capability-level forgetting.

4.1 RoD improves the adaptation–forgetting frontier

  • Across native- and proxy-replay settings, RoD reaches or improves upon the adaptation–forgetting frontier without presetting replay ratios.Source: Figure 2 · p. 5, §4.1 · pp. 5–6
  • On Legal, RoD cuts validation-loss forgetting 43% versus 10% fixed replay (0.125 to 0.071); on German, it adapts more strongly while closely matching 32% replay retention.Source: §4.1 · pp. 5–6
  • On Qwen3.5-9B Legal, RoD matches 10% fixed-replay adaptation loss (0.988) and reduces validation-loss forgetting from 0.075 to 0.031; on German, it matches the 17% replay trade-off.Source: §4.1 · p. 6
  • With proxy replay, task-based forgetting gains over fixed replay are smaller, but RoD reduces forgetting versus no-replay by 60% on Legal Qwen3.5-9B and 43% and 46% on German Qwen3.5-9B and 4B.Source: §4.1 · p. 6

4.2 RoD learns a demand-driven replay curriculum

  • On German Nemotron-12B, RoD reduces the vulnerability–residual-forgetting slope to 0.39 at 24% replay, versus 0.93 and 0.78 at fixed 17% and 32%; vulnerable categories receive up to 2.1× their prevalence.Source: Figure 3 · p. 6, §4.2 · p. 7
  • Replay starts absent, grows as forgetting emerges to 19–32% late in training, and its category composition shifts across model–domain settings.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 Joint adaptation–replay competition drives RoD

  • Joint competition improves adaptation and substantially reduces forgetting at the same average replay share; fixed-budget selection alone explains only part of RoD’s gains.The trade-off and allocation stabilize when unconstrained; m = 2 is the smallest candidate multiplier that leaves replay unconstrained.Source: §4.3 · p. 8, Figure 5 · p. 8

4.4 RoD’s model-dependent components transfer across model scales

  • A Qwen3.5-4B reference recovers native Qwen3.5-9B RoD’s trade-off, while smaller-model curricula remain competitive for targets up to 35B and scale gaps up to 8.75×.Source: §4.4 · p. 9, Table 1 · p. 9

5 Conclusion

RoD reaches or improves upon fixed replay’s adaptation–forgetting frontier without a predefined replay ratio, while adapting replay to evolving retention needs. Ablations and cross-scale results point to joint competition as the main source of gains and offer a path to larger models.

5 Conclusion

  • Across domains, model families, and scales, RoD reaches or improves upon the adaptation–forgetting frontier of fixed replay without a predefined replay ratio.RoD targets vulnerable sources and increases replay as forgetting emerges.Source: §5 · pp. 9–10
  • Ablations attribute RoD’s gains primarily to joint adaptation–replay competition rather than selection within either stream alone.Source: §5 · p. 10
  • Smaller adaptation references recover standard RoD’s trade-off, and transferred curricula remain effective across an 8.75× scale gap.Source: §5 · p. 10
  • The results support replaying what is needed when it is needed, with cross-scale results offering practical paths for applying this principle to larger models.Source: §5 · p. 10

AI use statement

The authors used generative AI for specified implementation, verification, mathematical-material integration, and manuscript-refinement tasks. They state that authors reviewed and verified AI-assisted outputs and take responsibility for the final work.

AI use statement

  • LLMs implemented functions from author-provided specifications, improved code efficiency, checked experimental configurations and outputs, and prepared author-provided mathematical material for the manuscript.Source: AI use statement · p. 10
  • The authors say AI was not used to formulate hypotheses, develop conceptual frameworks, independently derive or prove claims, design methods or experiments, or independently interpret results.Source: AI use statement · p. 10
  • Generative AI refined scientific figures based on author-specified changes and improved the language, clarity, and consistency of author-written drafts.Source: AI use statement · p. 10
  • The authors reviewed and verified AI-assisted outputs and accept responsibility for the final content, claims, code, and artifacts.Source: AI use statement · p. 10

Ethics statement

The work studies continued pretraining methods using existing language models and datasets for research, without human subjects or collection of personal or sensitive data.

Ethics statement

  • The study concerns continued pretraining methods for language models and uses existing models and datasets for research.Source: Ethics statement · p. 10
  • The authors report no human subjects and no collection of personal or sensitive data.Source: Ethics statement · p. 10
  • The authors identify no methodology-specific ethical concerns beyond those generally associated with developing and adapting large language models, and report following the ICLR Code of Ethics.Source: Ethics statement · p. 10

Reproducibility statement

The paper reports experiments using public datasets and open-weight models, and provides appendix material deriving RoD’s source-specific scores and interpreting joint selection, including the effect of proxy replay.

Reproducibility statement

  • Experiments use publicly available datasets and open-weight language models, with implementation and reproduction code provided in a repository.The paper also documents models, datasets, preprocessing, training configuration, baselines, evaluation, compute, and random seeds.Source: Reproducibility statement · p. 10

A Additional Method Details

  • Appendix A provides further motivation for RoD’s source-specific reducible-loss scores and joint selection, along with the complete training algorithm and implementation details.Source: A Additional Method Details · p. 15

A.1 Deriving Source-Specific Reducible Loss from RHO

  • RoD uses an adaptation-specialist reference for adaptation and the pretrained model for replay, avoiding a single reference that already encodes an adaptation–retention trade-off.The resulting scores measure excess loss relative to a source-appropriate reference.Source: §A.1 · pp. 15–16

A.2 Joint Selection as Resource Allocation

  • The k-th largest score acts as a shadow price for a training slot, so replay share emerges from the relative adaptation and replay scores rather than a prescribed ratio.RoD uses unweighted score comparison as a parameter-free default.Source: §A.2 · p. 17

A.3 Interpretation under Proxy Replay

  • With proxy replay, the pretrained model may not be converged on that distribution, so the replay score can reward fitting the proxy as well as returning toward the pretrained model.This offers one explanation for weaker coupling between proxy validation-loss retention and task-based retention; a self-distillation alternative is not used because it requires storing or recomputing token-level pretrained predictions.Source: §A.3 · pp. 17–18

A.4 RoD Algorithm and Implementation

RoD selects adaptation and replay examples jointly using model-relative losses, while its reference models estimate remaining learning potential and forgetting. The experiments span multiple model families, scales, and adaptation corpora, with proxy replay used when original pretraining data are unavailable.

A.4 RoD Algorithm and Implementation

  • Each step scores adaptation and replay candidates against separate reference losses, then selects the top k jointly for standard language-model training.Reference losses are cached offline; RHO affects data selection, not example weights.Source: §A.4 · pp. 18–19

A.5 Adaptation Reference Construction

  • The adaptation reference is a converged adaptation-only specialist whose loss estimates the learning potential remaining for each adaptation example.It uses warmup, constant learning rate until validation-loss convergence, then 0.5 epochs of cosine decay; the same model is the no-replay baseline.Source: §A.5 · p. 19

B Extended Experimental Setup

  • The experiments test generalization across Legal and German domains, model families, and scales, and transfer smaller-model curricula to larger targets.Curriculum transfer avoids online selection at target-model scale.Source: B Extended Experimental Setup · pp. 19–20

B.1 Models

  • Primary experiments use Nemotron-Nano-12B-v2 and Qwen3.5-9B, with Qwen3.5-4B and larger 30B- and 35B-class targets in scale-transfer experiments.Qwen’s original pretraining data are unavailable, so its replay uses the Nemotron general corpus as a proxy.Source: §B.1 · p. 20, §B.2 · p. 20

B.2 Adaptation and Replay Data

  • Legal adaptation uses a legal corpus, while German adaptation spans nine sources; replay uses the Nemotron general corpus, directly for Nemotron and as a proxy for Qwen.RoD balances the candidate stream before joint selection, so corpus proportions do not impose a minimum replay share.Source: §B.2 · pp. 20–21

B.3 Training Setup

Training uses a shared optimization setup, while baselines vary how replay is allocated or combine models after training. Evaluation measures adaptation and retention through held-out losses and capability tasks.

B.3 Training Setup

  • All primary settings use a fixed learning-rate schedule and trained-token-matched budgets, with RoD considering twice the trained batch size when selecting examples.The candidate multiplier is m = 2; larger-model curriculum transfer reuses the selected batch sequence without online target-scale selection.Source: §B.3 · p. 21

B.4 Baselines

  • Baselines include no-replay CPT, fixed replay ratios, and post-hoc interpolation of pretrained and adaptation-model parameters.Fixed-replay settings are swept at matched token budgets; max-replay runs use approximately twice the budget and are treated as higher-budget references.Source: §B.4 · pp. 21–22

B.5 Evaluation

  • Evaluation measures adaptation with held-out adaptation loss and forgetting with held-out general-data loss, supplemented by multiple-choice results across four capability groups.Forgetting metrics cap improvements at zero so gains in one source or capability cannot offset degradation elsewhere.Source: §B.5 · pp. 22–23

C.1 Additional Adaptation–Forgetting Results

Even after fixed-replay CPT is trained to convergence, RoD at the matched 1× compute budget remains on or improves upon its adaptation–forgetting frontier. Additional training can narrow fixed replay’s adaptation gap but does not yield a better frontier than RoD.

C.1 Additional Adaptation–Forgetting Results

  • Continuing RoD about 20% beyond its main budget improves neither adaptation nor retention in Nemotron-12B German and Legal settings.The authors report RoD has already converged at the 1× budget.Source: §C.1 · p. 23
  • RoD at 1× compute remains on or improves upon the converged fixed-replay frontier, although fixed-replay training uses up to 2.27× the matched budget.The convergence comparison covers German adaptation; extending fixed-replay training primarily improves adaptation.Source: §C.1 · p. 23, Figure 6 · p. 24
  • Additional fixed-replay training narrows its adaptation gap but does not produce a better adaptation–forgetting frontier than RoD despite greater compute.The result indicates the matched-budget comparison does not alone account for RoD’s favorable trade-off.Source: §C.1 · pp. 23–24

C.2 Proxy Replay and Task-Based Forgetting

The analyses show that RoD’s replay responds to changing forgetting signals and redistributes protection toward vulnerable sources. With proxy replay, however, validation-loss retention can correspond less closely to capability retention, especially for reasoning-intensive capabilities.

C.2 Proxy Replay and Task-Based Forgetting

  • With Qwen’s proxy replay, RoD substantially mitigates task-based forgetting versus no-replay CPT, but its remaining gap to fixed replay is clearest in reasoning and mathematics.The proxy distribution is less aligned with Qwen’s original pretraining data, making validation-loss changes a noisier signal of capability retention.Source: §C.2 · p. 25, Figure 7 · p. 25

C.3.1 RHO Loss Dynamics

  • Across settings, adaptation scores fall as learning proceeds while replay scores rise with forgetting, dynamically increasing replay’s competitiveness in selection.Replay-share shifts coincide with data-pass transitions and learning-rate annealing.Source: §C.3.1 · pp. 25–26, Figure 8 · p. 26

C.3.2 Forgetting Profiles

  • RoD produces flatter source-level forgetting profiles than fixed replay, providing stronger relative protection to sources most vulnerable under no-replay CPT.For German Nemotron-12B, the fitted slope is 0.39 with RoD versus 0.93/0.78 under fixed replay.Source: §C.3.2 · pp. 26–27, Figure 9 · p. 27

C.3.3 Replay Allocation

  • RoD’s replay allocation is positively associated with source vulnerability across settings, oversampling highly vulnerable sources and often undersampling stable ones.Reported correlations are r = 0.88 for German Nemotron-12B, r = 0.66 for Qwen3.5-9B, r = 0.45 for Qwen3.5-4B, and r = 0.76 for Legal Nemotron-12B.Source: §C.3.3 · p. 27, Figure 10 · p. 28

D Limitations and Conceptual Considerations

The section qualifies RoD’s computational costs, its dependence on replay coverage, and the limits of interpreting replay loss as capability retention. It also describes how RoD handles trade-off preferences without a preset replay allocation.

D.1 Computational Requirements

  • RoD adds online candidate scoring and offline reference construction, while smaller-model curriculum transfer can decouple model-dependent computation from target scale.Experiments use a candidate multiplier of 2, with little benefit from larger pools; a 4B adaptation reference recovers a similar trade-off for a 9B target.Source: §D.1 · p. 28

D.2 Relative Weighting of Adaptation and Replay

  • Direct comparison in token-normalized loss space lets adaptation and replay compete without a preset preference; relative weighting for a chosen operating point remains future work.Source: §D.2 · p. 29

D.3 Requirements on the Replay Distribution

  • RoD can prioritize only knowledge covered by replay; Qwen proxy replay reduced forgetting, but mismatched data weakened validation-loss alignment, especially for reasoning-intensive capabilities.Source: §D.3 · p. 29

D.4 Interpretation and Scope of the Replay Signal

  • Base-relative loss is task-agnostic, but sequence-loss increases need not equal capability loss; beyond-budget RoD did not improve adaptation or retention.With proxy replay, the loss signal corresponded less closely to task-based forgetting, yet RoD substantially reduced task-based forgetting relative to no-replay CPT.Source: §D.4 · p. 29, Table 4 · p. 31

Related papers

Loading 2609.40089v1…