PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Review: Spaced Repetition for Continual Pre-Training of Language Models

Alankar Atreya, Devesh Batra, Yoages Kumar Mantri, Geremy Bantug, Greig A Cowan, Raad Khraishi

arXiv:2608.17530v1cs.AIcs.LG

TL;DR

Continual pre-training must learn new information without erasing old knowledge, but uniform replay ignores differences in examples’ retention difficulty. SRT uses SuperMemo-2-based adaptive review scheduling to select examples for retention and consolidation, preserving broad benchmark performance while recovering old knowledge and maintaining new-knowledge acquisition.

  • Problem

    Continual pre-training can catastrophically forget old knowledge, while uniform replay ignores heterogeneous retention difficulty when selecting historical examples.

  • Method

    Spaced Repetition Training uses per-example SuperMemo-2 review states and perplexity-based recall quality to schedule old and new examples without changing the model, objective, or optimizer.

  • Results

    SRT preserves broad benchmark performance while recovering old-knowledge accuracy and matching or exceeding new-knowledge acquisition; scheduling alone yields a 23.8 percentage-point gap in old-knowledge accuracy on TinyLlama Wikipedia.

  • Takeaways & Limitations

    Adaptive review timing is an architecture-agnostic scheduling layer that complements existing continual-pretraining recipes and improves the stability-plasticity trade-off.

  • Takeaways & Limitations

    Language experiments use two Llama-family models and English-only evaluation, while temporal QA benchmarks are in-domain and broad-capability results use one training run per condition.

Abstract

from arXiv · show

Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.

1 Introduction

Continual pre-training can erase prior knowledge and broad capabilities, while conventional replay overlooks differences in how quickly examples are forgotten. Spaced Repetition Training reframes replay as sample-level adaptive review scheduling and mitigates forgetting while preserving new-knowledge acquisition and broader performance.

  • Continual pre-training updates can cause catastrophic forgetting of previously learned knowledge and degrade multitask reasoning and factual recall.
  • Most replay methods choose a global historical-data mixture and sample old examples uniformly or with fixed curricula, ignoring heterogeneous retention difficulty.
  • SRT treats continual pre-training as adaptive review scheduling, deciding which examples to review at each step rather than only how much history to replay.
  • SRT maintains per-example SuperMemo-2 state, converts perplexity into recall quality, and reviews difficult examples sooner while lengthening gaps for retained examples.The scheduler applies symmetrically to historical examples for retention and new examples for consolidation without changing the model, loss, or optimizer.
  • SRT mitigates catastrophic forgetting more effectively than naive continual pre-training and uniform replay while maintaining new-knowledge acquisition and preserving broad benchmark performance.The reported findings come from temporally separated Wikipedia and code corpora; vision and tabular experiments suggest the scheduling principle generalises beyond language.

2 Related Work

Related work frames continual pre-training around catastrophic forgetting, replay-based mitigation, and domain-specific adaptation challenges. Human-inspired replay methods motivate SRT’s use of per-example review scheduling and perplexity-based recall quality.

  • Catastrophic forgetting: Catastrophic forgetting overwrites representations needed for earlier tasks; mitigation strategies include regularization, architectural isolation or expansion, and replay.Replay interleaves previous examples with new data.
  • Continual pre-training of LLMs: Continual pre-training can improve target-domain performance while shifting models away from prior capabilities, motivating studies of rewarming, data mixtures, replay ratios, curricula, and temporal benchmarks.Continual updates can also degrade established multitask language-understanding and factual-recall benchmarks.
  • Human-inspired replay: SuperMemo-2 schedules expanding inter-review intervals from recall quality and per-item state, while newer neural methods use self-synthesized rehearsal, Leitner-style queues, or forgetting-curve replay.SRT instead maintains SuperMemo-2 state for individual training examples and derives recall quality from per-example perplexity in the standard forward pass.

3 Method

SRT treats continual pre-training as adaptive review scheduling: it preserves old knowledge while acquiring new information by deciding which examples to revisit at each step. It uses SuperMemo-2 review states and training-time perplexity, without changing the model, objective, or optimizer.

  • Problem formulation: SRT frames continual pre-training as balancing evaluation risks on historical and incoming corpora under a stability-plasticity trade-off.The weighting parameter λ controls this trade-off.
  • Problem formulation: The scheduler selects specific examples, not merely an old/new batch ratio, because retention difficulty varies across both corpora.This contrasts with naive training on only new data and replay methods that sample historical examples without adaptive selection.
  • Review scheduling: SRT assigns every old and new example SuperMemo-2 state, including ease factor, successful-review count, inter-review interval, and next due step.Initialization uses Ei = 2.5, ni = 0, and Ii = 1, with staggered initial due times.
  • Review scheduling: At each step, SRT allocates ⌊ρB⌋ slots to due old examples and remaining slots to due new examples, releasing unused capacity when one due set is empty.Both streams are scheduler-selected, and ρ controls the target old-exposure fraction.
  • Recall and updates: Per-example perplexity from the standard forward pass is converted into recall quality q ∈ {1, 2, 3, 4, 5} to update review intervals.If q < 3, SRT treats the example as forgotten and schedules immediate re-review; otherwise, it increases the interval under SuperMemo-2.
  • Recall and updates: SRT computes recall quality before the optimizer step and retains the standard next-token loss, architecture, and parameterization unchanged.The scheduler can replace perplexity with another confidence-based signal for non-generative models.

4 Experimental Setup

The experiments use temporally separated Wikipedia and code corpora to distinguish old-knowledge retention from new-knowledge acquisition. SRT is evaluated across two model scales against continual-learning baselines using source-grounded QA and broad capability benchmarks.

  • Datasets: Temporally separated old and new corpora enable evaluation of retained knowledge versus acquired knowledge across encyclopedic text and source code.Wikipedia uses pre- and post-cutoff information aligned by article title, while code uses repositories separated by creation and commit timestamps.
  • Datasets: Wikipedia contains 31,729 aligned examples in each split, while code contains 22,055 old and 70,643 new examples.These corpus sizes are reported for the two domains’ old and new training data.
  • Models and training: SRT is evaluated with TinyLlama-1.1B-Chat and Llama-3.2-3B-Instruct under a shared continual-pretraining recipe and fixed hyperparameters across conditions.The recipe uses next-token cross-entropy, AdamW, linear learning-rate scheduling, gradient clipping, and bfloat16 where supported.
  • Baselines: The comparison includes the original checkpoint, naive CPT, Uniform Replay with a 20/80 old/new ratio, and PPL-Prioritised replay.These conditions isolate no replay, uniform sampling, and difficulty-aware replay against SRT’s adaptive scheduling.
  • Evaluation: Source-grounded QA uses matched old and new MMLU-style splits of 500 questions per domain, reporting mean accuracy and bootstrap standard deviation from 10,000 resamples.Questions are generated from source passages in Dold and Dnew and manually checked for answerability and label support.
  • Evaluation: Broad capability is measured with MMLU, BBH, GSM8K, and PIQA using the EleutherAI LM Evaluation Harness for standardized few-shot evaluation.The benchmarks cover multitask knowledge and reasoning, hard reasoning, mathematical reasoning, and physical commonsense.

5 Results

SRT consistently improves retention while preserving or improving new-knowledge acquisition, outperforming naive continual pre-training and uniform replay across temporal QA and broad capability benchmarks. The same adaptive scheduling principle also transfers to vision and tabular data, with modest computational overhead.

  • 5.1 Temporal QA: Across model scales, SRT recovers retention while preserving or improving new-knowledge accuracy, whereas naive CPT trades old knowledge for new knowledge.On TinyLlama, CPT raises Wikipedia new accuracy from 13.1% to 17.0% while old accuracy falls from 54.3% to 11.7%.
  • 5.1 Temporal QA: 23.8 percentage points: the uniform-replay-to-SRT old-Wikipedia accuracy gap on TinyLlama, versus 5.4 points on Llama-3.2-3B-Instruct.Both methods use the same 20/80 exposure cap and pool structure, so the difference is attributed to within-pool scheduling.
  • 5.1 Temporal QA: SRT achieves the best combined score for every model-domain pair, restoring Wikipedia old accuracy to or above baseline while improving both old and new code-QA accuracy relative to CPT.Uniform replay remains consistently behind SRT on code QA.
  • 5.2 Broad capability benchmarks: 38.8-point decline: CPT reduces Llama-3.2-3B-Instruct GSM8K accuracy from 77.6% to 38.8%, while uniform replay further drops BBH to 8.4% and GSM8K to 6.8%.SRT preserves base-model performance on broad capability benchmarks at this scale.
  • Cross-modal evidence: 91.6%, 65.8%, 53.1%, and 53.3%: SRT’s highest overall accuracies on MNIST, Fashion-MNIST, CIFAR-10, and Wine, respectively.Replacing perplexity with predicted-class confidence produces the same qualitative scheduling pattern.
  • Computational overhead: 14.7%: SRT’s wall-clock overhead relative to CPT, corresponding to a 14.6% throughput reduction; its extra cost over uniform replay is 3.5%.The overhead comes from an additional forward pass for recall-quality computation and SuperMemo-2 state updates.

6 Ablations

Ablations identify a 20/80 old/new replay split and default SM-2 settings as the strongest configurations, while extreme budgets and threshold scaling degrade performance. The results show that both exposure allocation and review-state calibration materially affect SRT.

  • Replay budget: The 20/80 old/new replay ratio performs best, reaching 49.6% old and 20.0% new accuracy.Old-heavy and new-heavy schedules reduce the budget available for consolidation and adaptation, weakening either new-data acquisition or old-knowledge retention.
  • Perplexity-to-quality threshold scaling: Default threshold scaling α = 1.0 is best, reaching 22.6% combined accuracy.Stricter and more relaxed thresholds perform worse; over-relaxed thresholds can assign high quality to poorly retained examples and stop scheduling them for review.
  • Initial ease factor: The default initial ease factor E0 = 2.5 achieves the best 22.6% combined accuracy.Smaller values over-schedule already retained examples, whereas larger values expand intervals too quickly and allow fragile examples to go unrevised.

7 Discussion

The discussion argues that SRT helps when finite training budgets must preserve heterogeneous old knowledge, because within-pool scheduling—not exposure alone—is the active ingredient. It also warns that uniform replay can damage reasoning and positions SRT as an architecture- and objective-preserving scheduling layer that can complement other methods.

  • Core finding: SRT improves continual pre-training when old and new knowledge share a finite budget and retention difficulty varies across examples.The discussion identifies these conditions as the specific claim supported by the results.
  • Scheduling mechanism: Uniform Replay and SRT share the same 20/80 exposure cap and pool structure, isolating within-pool selection as uniform versus scheduled.This comparison supports scheduling as the active ingredient rather than exposure alone.
  • Reasoning effects: Replay drops BBH from a CPT baseline of 44.3% to 8.4% and GSM8K from 38.8% to 6.8%, while SRT preserves both close to the original model.The discussion presents uniform replay as potentially worse than naive CPT on reasoning capabilities.
  • Limitations: The authors do not establish the mechanism behind uniform replay’s reasoning damage and caution that poorly designed replay may disrupt continual updates.They offer interference dynamics only as a possible explanation requiring representational analysis beyond the work’s scope.
  • Scope and integration: SRT is a scheduling layer that leaves architecture, training objective, and replay budget unchanged, so it can combine with rewarming, mixture selection, parameter-efficient tuning, or synthetic rehearsal.Its contribution is presented as orthogonal to these methods.

8 Conclusion

The conclusion presents Spaced Repetition Training as an adaptive, cognitively inspired approach to continual pre-training that schedules per-example review using SuperMemo-2 and perplexity-derived recall quality. Across Wikipedia and code benchmarks, SRT improves retention while preserving new-knowledge acquisition and broader capabilities.

  • Method: SRT schedules per-example review with SuperMemo-2 and perplexity-derived recall quality, rather than treating continual pre-training solely as a mixture problem.The method revisits examples the model currently struggles to retain instead of examples it has already learned.
  • Results: Across temporally grounded Wikipedia and code QA benchmarks at two model scales, SRT recovers old-knowledge accuracy discarded by naive continual pre-training while matching or exceeding new-knowledge acquisition.This demonstrates improved stability-plasticity trade-offs without sacrificing learning of new information.
  • Results: At larger scale, SRT preserves broad capability benchmark performance that naive continual pre-training and uniform replay degrade, supporting adaptive review timing as architecture-agnostic and effective.The conclusion characterizes adaptive review timing as a simple mechanism for improving continual pre-training stability and plasticity.

Limitations

The evaluation is limited to two Llama-family models and English data, while temporal QA benchmarks provide in-domain measurements tied to the study’s specific corpora. Behavior at larger scales, across model families, and in multilingual settings remains open.

  • Model and evaluation scope: Evaluation uses TinyLlama-1.1B and Llama-3.2-3B-Instruct, both from the Llama family, and is restricted to English.Behavior at larger scales, across different model families, and in multilingual settings remains open.
  • Benchmark scope: The temporal QA benchmarks are constructed for this work and should be interpreted as in-domain measurements of retention and acquisition on the specific corpora.

Ethics and Reproducibility … A.3 Evaluation Question-Answering Sets

The paper documents ethical and reproducibility constraints while detailing temporally separated Wikipedia and code corpora and manually checked, held-out QA evaluations for retention and acquisition. These procedures support comparable public-data replication without releasing proprietary artifacts.

  • Ethics and Reproducibility: Continual updates may introduce factual errors, preserve outdated information, or reinforce unsafe code patterns, so source-grounded QA accuracy is not deployment reliability.Replay buffers and SRT state must also respect retention limits, deletion requests, and data-governance obligations.
  • Ethics and Reproducibility: Exact benchmark artifacts, code, logs, and QA items remain proprietary, but the reported algorithm, mappings, corpora criteria, hyperparameters, protocol, and results support comparable public-data replication.Benchmark generation used LLM assistance with manual checks.
  • A Corpora Construction Details: The appendix specifies construction details for Wikipedia and code corpora and the domain-specific question-answering evaluation sets.These materials cover the corpora described in Section 4.1.
  • A.1 Wikipedia Corpora: Wikipedia corpora use timestamped English snapshots selected around model pre-training windows, including November 2023 and January 2024 for Llama-3.2-3B-Instruct.The TinyLlama snapshots bracket its mid-2023 pre-training data assembly window, with a later snapshot capturing post-cutoff revisions.
  • A.1 Wikipedia Corpora: Sentence-level sequence diffs define Dnew from inserted sentences and updated modified spans, reducing unchanged text while preserving localized factual updates absent from corresponding Dold articles.For Llama-3.2-3B-Instruct, candidate articles are stratified by topic and change magnitude for balanced domain coverage.
  • A.2 Code Corpora: Code corpora use GitHub repository creation and commit timestamps, with old repositories predating January 2022 and new repositories created after July 2024 to prevent temporal leakage.The repositories span thirteen programming languages and represent new artifacts rather than aligned revisions.
  • A.3 Evaluation Question-Answering Sets: Each domain has 500-question-per-split held-out multiple-choice QA benchmarks, with four choices for Wikipedia questions and questions disjoint from training corpora.Manual checks verify answerability and source support; Dold measures retention, while Dnew measures post-cutoff acquisition.

B Training Details … C.4 Model Architectures

The appendix fixes shared training settings across continual pre-training conditions and extends SRT to class-incremental vision and tabular benchmarks. These auxiliary experiments adapt recall quality to classification confidence and compare SRT with EWC using simple PyTorch convolutional and feedforward models.

  • B Training Details: The same hyperparameters are used for TinyLlama-1.1B-Chat and Llama-3.2-3B-Instruct across CPT, Uniform Replay, and SRT, isolating replay-strategy effects.Training and evaluation hyperparameters are summarized in Table 9.
  • B Training Details: An effective batch size of 512 is maintained across model scales using bfloat16 mixed-precision training, gradient checkpointing, and a per-device batch size of 32 with 16 accumulation steps.Training consumes one pass over Dnew at the configured replay ratio; exact step counts depend on corpus size.
  • C Auxiliary Vision and Tabular Experiments: SRT is evaluated beyond language on class-incremental vision and tabular benchmarks, where its architecture-agnostic scheduling mechanism provides an auxiliary generalization check.The auxiliary experiments cover small vision and tabular classification benchmarks.
  • C.1 Datasets and Class-Incremental Setup: The auxiliary datasets are MNIST, Fashion-MNIST, CIFAR-10, and UCI Wine, each split into old classes for base training and new classes introduced sequentially.Image datasets use semantically confusable class assignments to induce stronger interference between old and new representations.
  • C.2 Recall Quality for Non-Generative Models: For classification models, SRT replaces perplexity with predicted probability pθ(yi | xi), assigning longer review intervals to higher confidence in the correct label.Low confidence corresponds to low recall quality and shorter review intervals.
  • C.3 Baseline: Elastic Weight Consolidation: The vision and tabular experiments compare SRT with Elastic Weight Consolidation, which regularizes updates to parameters important for previously learned tasks.EWC estimates parameter importance from the diagonal Fisher Information Matrix computed on old data.
  • C.3 Baseline: Elastic Weight Consolidation: EWC is especially practical for the small fully connected and convolutional vision models because Fisher computation has low overhead, unlike the language-model setting.Gradient-level regularization is not the focus of the main language-model experiments because it is significantly more expensive.
  • C.4 Model Architectures: All auxiliary models are PyTorch feedforward or convolutional networks with ReLU activations, trained using cross-entropy loss and AdamW.MNIST and Fashion-MNIST use two convolution-pooling blocks plus two fully connected layers; CIFAR-10 adds dropout in the classifier head, while Wine uses a three-layer MLP over 13 features.

C.5 Results · D External Validation on TemporalWiki

SRT improves retention in class-incremental vision and tabular experiments and reduces catastrophic forgetting on the external TemporalWiki benchmark. These results suggest its scheduling mechanism generalizes beyond language when paired with an appropriate recall-quality signal, while TemporalWiki supports only relative comparisons.

  • C.5 Results: Naive CPT combines near-total old-class forgetting with high new-class accuracy, reproducing catastrophic forgetting across vision datasets.This mirrors the failure mode observed at language scale.
  • C.5 Results: EWC partially mitigates forgetting on MNIST and Wine but performs poorly on Fashion-MNIST and CIFAR-10.The passage attributes this pattern to semantic similarity between old and new classes, though the supplied text is truncated.
  • C.5 Results: Auxiliary experiments suggest SRT’s scheduling mechanism generalizes beyond text to non-generative classification settings.These experiments are auxiliary rather than central evidence for the paper’s language-model continual-pretraining claim.
  • C.5 Results: In non-generative classification, SRT derives recall quality from predicted-class confidence rather than perplexity.This adapts the recall signal while preserving the underlying scheduling principle.
  • D External Validation on TemporalWiki: TemporalWiki evaluates factual knowledge with object-token perplexity on Changed and Unchanged subject-relation-object probes, where lower perplexity indicates better retention.Results are reported as category means for Base, CPT, and SRT on Llama-3.2-3B-Instruct.
  • D External Validation on TemporalWiki: 2785 vs. 2112 average: SRT limits TemporalWiki perplexity degradation to roughly 30% above base, whereas CPT nearly triples perplexity on both categories.This externally corroborates SRT’s retention advantage and CPT’s catastrophic forgetting.
  • D External Validation on TemporalWiki: TemporalWiki perplexity values are tokenizer-dependent and snapshots predate Llama-3.2-3B-Instruct’s knowledge cutoff, so only relative ordering is meaningful.Both Changed and Unchanged categories therefore represent pre-cutoff knowledge for these models.
  • C.5 Results: SRT achieves the highest overall accuracy on all four vision and tabular datasets, averaged over 10 random seeds.Naive CPT loses essentially all old-class accuracy on vision datasets, while EWC only partially mitigates forgetting on MNIST and Wine and performs poorly elsewhere.
Loading 2608.17530v1…