PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Sewon Min

arXiv:2610.15965v1cs.LGcs.CL

TL;DR

With high-quality training data finite, pretraining increasingly requires repeated exposure to the same data. This paper tunes batch size and repetition-aware recipes, then separates GPU availability from training time; smaller batches and recipe changes improve repeated-data training, and fewer GPUs can outperform greater parallelism under a deadline.

  • Problem

    Pretraining often assumes a single pass over abundant data, although high-quality human-written data are finite and repeated-data training is increasingly needed.

    Source: §1 · p. 1

  • Method

    The paper jointly studies batch size, repetitions, and repetition-aware recipes, then maps attainable performance under separate GPU and training-time constraints.

    Source: §6 · p. 11

  • Results

    Repeated-data training favors smaller batches and adapted recipes, while under a fixed deadline smaller batches using fewer GPUs can achieve lower validation loss than larger batches using more GPUs.

    Source: §1 · pp. 1–2, §6 · p. 11

  • Takeaways & Limitations

    Data-constrained pretraining requires jointly selecting repetitions, batch size, compute parallelism, and training recipe.

    Source: §6 · p. 11

    Results cover 153M–1.5B-parameter models and 0.333B–3B-token data pools, and may depend on architecture, optimizer, schedule, context length, and training data.

    Source: §6 · p. 11

Abstract

from arXiv · show

We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.

1 INTRODUCTION

With unique training data finite, pretraining must reuse data, so the paper studies how training recipes and resource allocation affect repeated-data performance. It finds that smaller batches and repetition-aware recipes improve attainable loss, while GPU and time constraints must be considered separately.

1 INTRODUCTION

  • Finite high-quality human-written data increasingly makes repeated training necessary, unlike the single-pass assumption of much pretraining.The study simplifies repeated data by uniformly repeating a fixed dataset.Source: §1 · p. 1
  • Jointly tuning batch size, learning rate, weight decay, and repetitions reveals a U-shaped validation-loss relationship, favoring smaller batches than conventional pretraining.Batch tuning extends beneficial repetitions from 8 to 20 for a 1.5B model and from 40 to 112 for a 153M model; small-batch gradient noise likely regularizes repeated-data overfitting.Source: §1 · pp. 1–2
  • Repacking documents each epoch, tying input and output embeddings, and applying positive weight decay to shared embeddings improve validation loss and make more repetitions beneficial.The best checkpoint shifts from epoch 20 to 32 for the 1.5B model and from 112 to 240 for the 153M model.Source: §1 · pp. 1–2
  • Even under a fixed deadline, smaller batches using fewer GPUs can achieve lower validation loss than larger batches using more GPUs.The paper cautions that maximizing parallelism may therefore be suboptimal for repeated-data training.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper concludes that data-constrained pretraining requires changing both training recipes and compute allocation.Smaller-than-typical batches may require long sequential training; tighter deadlines favor larger batches, while latency-oriented parallelism can trade more GPUs for faster optimizer steps.Source: §1 · p. 2

2 RELATED WORK

Prior scaling work largely assumes fresh data are effectively unlimited, while repeated-data studies have paid less attention to batch size. This paper jointly tunes recipe choices for fixed data and finds that repeated-data training changes the batch-size tradeoff.

2 RELATED WORK

  • Conventional scaling laws select compute-optimal model–data allocations while effectively assuming an unlimited supply of fresh data (Kaplan et al., 2020; Hoffmann et al., 2022).Finite, slowly growing high-quality data has motivated repeated-data pretraining (Muennighoff et al., 2023; Xue et al., 2023).Source: §2 · p. 2
  • At fixed compute, repeated training on filtered data can outperform a 10× larger superset with appropriate regularization, such as weight-decay tuning (Fang et al., 2025).Source: §2 · p. 3
  • Prior infinite-compute, finite-data studies emphasize regularization, but fix batch size in their repeated-data experiments (Kim et al., 2026; Xu et al., 2026).Kim et al. (2026) study stronger regularization, while Xu et al. (2026) add regularization methods and an interaction-aware scaling law.Source: §2 · p. 3
  • Unlike conventional batch-size reasoning, where larger batches expose parallelism and smaller batches can be more token-efficient, repeated-data training can favor smaller batches even under a fixed deadline.The paper fixes the unique-data pool and jointly tunes learning rate, weight decay, and repetitions for each batch size.Source: §2 · p. 3

3 BATCH SIZE SHAPES THE REPEATED-DATA FRONTIER

Jointly tuning batch size and recipe parameters reveals an interior optimum for repeated-data training. Across three model sizes, validation CE is minimized near B = 64, smaller than batches predicted by single-pass scaling laws.

3.1 EXPERIMENTAL SETUP

  • The setup fixes model size and a 1B-token unique-data pool, tunes learning rate and weight decay across each batch size, and evaluates next-token CE on a fixed validation split.The study uses decoder-only Transformers at 153M, 474M, and 1.5B parameters; each batch size is compared over its own training horizon.Source: §3.1 · pp. 3–4

3.2 RESULTS

  • For the 1.5B model, validation CE falls from 3.124 at B = 512 to 2.977 at B = 64; all three model sizes show a U-curve minimized near B = 64.The batch-size sweep is on the 1B-token pool, with learning rate, weight decay, and epoch count tuned for each model-size and batch-size pair.Source: Table 1 · p. 4, Figure 2 · p. 4, §3.2 · p. 4
  • At the 1.5B model’s best observed repeated-data loss, single-pass batch-size laws predict batches 4.7× and 2.9× larger than the selected 0.262M-token batch.At 1.8 × 10^20 FLOPs, the laws predict 1.233M and 0.747M tokens, respectively.Source: §3.2 · p. 4
  • For B = 512, training loss eventually falls below B = 64 while validation loss remains worse and rises; for B = 16, lower training loss accompanies higher validation loss.The authors interpret the large-batch pattern as consistent with weaker stochastic regularization and the small-batch pattern as overfitting from additional optimizer steps.Source: §3.2 · p. 5, Figure 3 · p. 5
  • Weight-decay tuning matters more for large batches and severe data constraints; at B = 64, the epoch-1 choice of 0.3 remains optimal through saturation.Source: §3.2 · p. 5

4 TRAINING CHOICES SHIFT THE REPEATED-DATA FRONTIER

Repeated-data training benefits from adapting both the data presentation and model recipe, improving validation loss and HellaSwag BPB across model sizes and batch sizes. The finalized recipe also benefits from larger unique-data pools, where the best observed loss improves and selected configurations shift toward larger batches and later epochs.

4.1 CONTROLLED RECIPE ABLATIONS

  • REPACK improves validation CE over SHUFFLE at B = 64 and B = 512, while FIXED underperforms SHUFFLE and has a larger train-validation gap at B = 512.REPACK dynamically repacks documents into new fixed-length sequences, creating fresh sequences and batches across epochs.Source: §4.1 · p. 6
  • With REPACK fixed, weight tying alone yields little to no improvement, while positive embedding decay consistently improves validation loss; combining them gives the best results.The combined recipe improves loss from 2.971 to 2.946 at B = 64 and from 3.064 to 3.030 at B = 512, versus untied embeddings without embedding decay.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2 COMPLETE RESULTS WITH THE REPETITION-TUNED RECIPE

  • The repetition-aware recipe improves validation loss and HellaSwag BPB across model sizes and batch sizes, while extending training horizons and shifting optimal batches for smaller models.Selected epochs move from 20 to 32 for 1.5B, 32 to 80 for 474M, and 112 to 240 for 153M; optimal batch size shifts from 64 to 128 for the 474M and 153M models.Source: §4.2 · p. 7, Figure 6 · p. 7
  • The repetition-aware recipe combines REPACK, tied embeddings, and positive embedding decay; these recipe choices materially affect attainable loss and should be explicit design decisions rather than inherited defaults.The surveyed open-source stacks use these ingredients inconsistently and often in combinations different from the best-performing configuration.Source: §4 · p. 5, §4.2 · p. 7
  • With more unique data, the best observed validation loss improves and selected configurations shift toward larger batches and later epochs.The comparison covers 0.333B-, 1B-, and 3B-token pools.Source: §4.2 · p. 8, Figure 7 · p. 8

5 SCALING ACROSS GPU AND TIME BUDGETS

Because smaller batches can require long sequential training, the paper treats GPU availability and time as separate constraints. Its results show that the preferred batch and recipe depend on deadlines and GPU budgets, and that the repetition-aware recipe improves the joint GPU-time frontier.

5.1 MOTIVATION: WHY SEPARATE GPUS AND TIME?

  • The unconstrained operating point can be impractical under deadlines: the 70B/10T-token extrapolation projects 22.6 years of training, despite an estimated 6,144 H100s being potentially attainable for large labs.The extrapolation is illustrative rather than conclusive and uses an extreme scale beyond the measured model and data sizes.Source: §5.1 · pp. 8–9

5.2 TIME-CONSTRAINED SCALING

  • Short deadlines tend to favor larger batches, while longer budgets progressively favor smaller batches until the unconstrained best batch B⋆ is reached.The pattern holds across the three model sizes and the training recipes, epoch counts, and batch sizes examined.Source: §5.2 · p. 9, Figure 8 · p. 9
  • Under a 32K-step deadline, a small batch using 16 GPUs achieves lower loss than a larger batch using 64 GPUs, although the larger batch is best at 8K steps.Thus, increasing global batch size to use more available GPUs can be suboptimal under a fixed time budget.Source: §5.2 · p. 9
  • For the 153M and 474M models, ORIGINAL may be preferred under very short deadlines, but the repetition-aware recipe becomes optimal later; it is optimal across the observed horizon for 1.5B.The difference is described as consistent but minor, likely because the recipe delays overfitting and increases beneficial repetitions.Source: §5.2 · pp. 9–10
  • Latency-oriented parallelism can shorten training without changing global batch size; the simulated H100 model projects 2.7× faster 1.5B training using 8× more GPUs when microbatch size falls from 8 to 1.At extrapolated 70B scale, 8× more model parallelism projects a 3.75× speedup; these are illustrative simulations, not end-to-end measurements.Source: §5.2 · p. 10

5.3 JOINT GPU-TIME-CONSTRAINED SCALING

  • When GPU and time budgets both bind, stopping epoch, batch size, and compute-parallelism configuration must be selected jointly; gradient accumulation offers another way to trade time against GPU availability.The paper frames GPUs and time as distinct resource constraints because small batches can require longer training.Source: §5 · p. 8, §5.1 · p. 9, §5.3 · p. 10
  • Additional GPUs shift selected configurations toward greater concurrency, while more time favors smaller batches; with more time, extra GPUs yield marginal gains and can eventually worsen performance.This pattern is reported for both ORIGINAL and repetition-aware recipes on the 1B-token pool.Source: §5.3 · pp. 10–11, Figure 9 · p. 10
  • Repetition-aware training uniformly improves the joint GPU-time frontier after each recipe is independently optimized for the available resources.The frontier covers GPU counts and optimizer-step-equivalent deadlines on the 1B-token pool; beyond the best observed unconstrained loss, additional GPUs or time do not improve it.Source: Figure 9 · p. 10, §5.3 · p. 11

6 CONCLUSION AND LIMITATIONS

The paper studies repeated-data pretraining and shows that batch size and repetition-aware recipe choices affect the attainable frontier, while GPU and time constraints require joint resource decisions. Its findings are limited to specific model and data scales and may depend on the architecture, optimizer and schedule, context length, and training data.

6 CONCLUSION AND LIMITATIONS

  • The paper improves the repeated-data frontier with batch-size optimization and a recipe combining dynamic repacking, tied input/output embeddings, and positive weight decay on shared embeddings.It then maps GPU- and time-constrained frontiers across 153M–1.5B-parameter models and 0.333B–3B-token data pools.Source: §6 · p. 11
  • The findings cover 153M–1.5B models and 0.333B–3B-token pools, and may depend on dense architecture, AdamW with WSD, context length, and the filtered DCLM-Baseline corpus.The 70B/10T example is an illustrative projection, not a rigorously validated estimate; extending the findings to MoE remains an important direction.Source: §6 · p. 11

REPRODUCIBILITY STATEMENT

The statement says code and exact commands will be released upon acceptance and points to §E for figure- and table-specific hyperparameters.

REPRODUCIBILITY STATEMENT

  • Code and exact commands are slated for release upon acceptance, and §E reports the hyperparameters needed to reproduce every figure and table.The listed hyperparameters are learning rate, weight decay, and batch size.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 MODEL ARCHITECTURES

The appendix specifies the model architectures, data and training setup, evaluation metric, and robustness checks used in the experiments.

A.1 MODEL ARCHITECTURES

  • Experiments use dense decoder-only Transformers with nominal sizes of 153M, 474M, and 1.5B, a 4,096-token context, and the listed common architecture components.After weight tying, trainable parameter counts are approximately 102M, 372M, and 1.3B, while the paper retains the nominal labels.Source: §A.1 · p. 15, Table 2 · p. 15

A.2 DATA AND TRAINING DETAILS

  • Training text comes from DCLM-Baseline, with documents split before packing and a 100M-token validation set.Sequence indices are reshuffled each epoch in §3; the later default repacks documents into new sequences each epoch.Source: §A.2 · p. 15
  • Training uses AdamW and a warmup–stable–decay schedule, with batch size, learning rate, and weight decay jointly tuned.The search covers the stated batch-size, learning-rate, and weight-decay grids; learning rates tuned on the 1B-token pool are transferred to the 0.333B- and 3B-token pools.Source: §A.2 · p. 15

A.3 EVALUATION DETAILS

  • The primary evaluation metric is next-token cross-entropy on a fixed, document-disjoint DCLM-Baseline validation split.Downstream-task evaluation is discussed separately in §B.3.Source: §A.3 · p. 15

B.1 ROBUSTNESS TO RANDOM SEEDS

  • Across three independently sampled 1B-token pools, validation-CE standard deviations range from 0.00021 to 0.00318, and the optimal batch size is 64 in every pool.The variation across pools is reported as substantially smaller than differences between batch configurations.Source: §B.1 · p. 16, Table 3 · p. 16

B.2 COMPARISON WITH UNIQUE-DATA TRAINING

The section compares repeated-data training with unique-data training, examines downstream trends across batch sizes, and surveys open-source training defaults.

B.2 COMPARISON WITH UNIQUE-DATA TRAINING

  • With batch tuning, repeated-data training matches ∼8× more unique data at B = 64 and ∼16× at B = 1024; without tuning B, it matches ∼8× at B = 1024.Figure 10 compares the best repeated-data endpoints with unique-data curves.Source: Figure 10 · p. 16, §B.2 · p. 16
  • The roughly four-epoch similarity between unique- and repeated-data training depends on batch size: larger batches preserve it for more epochs but fewer steps, while smaller batches do so for more steps but fewer epochs.The passage attributes the earlier observation to Muennighoff et al. (2023).Source: §B.2 · p. 16

B.3 DOWNSTREAM TASK GENERALIZATION

  • Across model sizes, overly large batches perform poorly, while the strongest downstream results tend to occur at intermediate batch sizes rather than at the smallest sizes.Downstream accuracy curves are more variable and not always convex; where BPB curves are convex, their preferred batch size is consistently near B = 64.Source: Figure 11 · p. 17, §B.3 · p. 18

C.1 COMPARISON WITH EXISTING OPEN-SOURCE DEFAULTS

  • A survey of nine representative open-source training stacks finds substantial variation in defaults, often differing from the paper’s best-performing repetition-aware recipe.The authors recommend treating these configuration choices as explicit design decisions rather than automatically inheriting defaults.Source: §C.1 · p. 19, Table 4 · p. 19

C.2 FAILED ATTEMPTS OR HIGHER-RISK RECIPES: SMALL-BATCH NOISE AT LARGE BATCH

The section tests alternatives to small-batch gradient noise and reports DiLoCo’s mixed validation-loss and downstream-accuracy results; it also gives illustrative 70B/10T resource projections from fitted curves.

C.2 FAILED ATTEMPTS OR HIGHER-RISK RECIPES: SMALL-BATCH NOISE AT LARGE BATCH

  • Batch warmdown and branch-train-merge do not improve the training-time-validation-loss frontier, while structured gradient noise has recurrent spikes and no stable trajectory.Periodic local optimization was the only tested approach described as promising; the reported comparison uses DiLoCo rather than LocalSGD.Source: §C.2 · p. 19
  • DiLoCo improves HellaSwag accuracy by approximately 1.7–3.1% over the stronger matched-step baseline, but does not consistently improve validation CE.The authors regard the method as promising but not conclusive and do not use it in the remaining experiments.Source: §C.2 · pp. 19–20, Table 5 · p. 20

D.1 ILLUSTRATIVE 70B/10T REPETITION AND BATCH EXTRAPOLATION

  • The epoch projection fits validation-loss curves from the tuned repetition-aware trajectories across a 3 × 3 grid and estimates a practical stopping epoch within 0.001 loss of the fitted minimum.The curve model separates useful learning from degradation with repetition; the fit weights grid cells equally and checkpoints by local spacing in log E. Training for the 153M model on 3B tokens stopped at epoch 640 while loss was still improving, so that cell uses a one-sided penalty.Source: §D.1 · p. 21
  • The fitted epoch curve gives E⋆(70B, 10T) = 214.2, rounded to 214 epochs, with multiplicative RMS error 1.011× on the nine model-derived targets.The target-scale projection is not validated on measurements at that scale.Source: §D.1 · p. 21
  • The batch-size fit projects approximately 1,536 sequences for 70B/10T, using twelve observations because nine were considered too few for a reliable fit.The batch-size fit has multiplicative RMS error 1.29× and log-space R2 = 0.768.Source: §D.1 · p. 22
  • The illustrative 70B model trained on 10T unique tokens is projected to require about 340.1 million optimizer steps, 6,144 H100 GPUs, and 22.6 years under the conventional configuration.The estimate uses rounded projections of E⋆ = 214 and B⋆ = 1,536 and is explicitly an extreme extrapolation.Source: §D.1 · pp. 21–22

D.2 FROM THROUGHPUT-FIRST TO LATENCY-FIRST COMPUTE PARALLELISM

The section contrasts throughput-first and latency-first GPU configurations at a fixed global batch, then shows how batch-size choices vary with GPU count and deadline.

D.2 FROM THROUGHPUT-FIRST TO LATENCY-FIRST COMPUTE PARALLELISM

  • Throughput-first selects the feasible configuration with highest throughput per GPU, whereas latency-first minimizes optimizer-step time while preserving the same global batch and respecting resource constraints.Latency-first may assign additional GPUs to tensor or pipeline parallelism when the reduction in exposed data-parallel communication exceeds added communication and scheduling overhead.Source: §D.2 · p. 22
  • The latency comparisons use a simulated, communication-aware H100 model and are fitted projections, not end-to-end measurements.Implementing and benchmarking the full microbatch and DP/TP/PP design space is beyond the authors’ scope.Source: §D.2 · p. 22
  • Latency-first configurations yield 1.67×–3.75× shorter optimizer steps across model sizes at the cost of more GPUs; the 70B configuration is 3.75× faster with 8× more GPUs.The projections indicate that additional model parallelism eventually stops reducing latency because of communication and other overheads.Source: §D.2 · p. 22, Table 6 · p. 23

D.3 SUPPLEMENTARY MATERIALS FOR TIME–GPU-CONSTRAINED SCALING

  • Figure 12 encodes the lowest-validation-CE observed batch size for each exact GPU count and deadline, with microbatch size fixed; its black boundary marks where the unconstrained globally optimal loss is reached.Beyond that boundary, additional time or GPUs do not improve validation loss.Source: Figure 12 · p. 23, §D.3 · p. 23

E EXPERIMENT CONFIGURATIONS FOR SECTIONS 3 AND 4

The appendix documents experiment configurations and the assumptions, timing model, and selection process used to estimate parallelism performance. Its fitted latency objective selects TP8/PP4, while additional pipeline stages can add overhead without reducing compute time.

E EXPERIMENT CONFIGURATIONS FOR SECTIONS 3 AND 4

  • The experiment-configuration appendix reports B, LR, WD, and E for runs, defining them as global batch size, peak learning rate, AdamW weight decay, and training passes including terminal WSD decay.Tables document batch sweeps, selected checkpoints across training horizons, loader comparisons, and weight-tying and embedding-decay comparisons.Source: E EXPERIMENT CONFIGURATIONS FOR… · p. 24, Table 7 · p. 24, Table 8 · p. 24

F PARALLELISM AND STEP-TIME ESTIMATION

  • The timing setup models dense decoder-only Transformers at sequence length 4,096 and compares throughput-first and latency-first configurations under memory, divisibility, and topology constraints.Both policies fix the global batch at 1,024 sequences and gradient accumulation at 1; latency-first fixes microbatch size at 1 and searches tensor- and pipeline-parallel degrees.Source: F PARALLELISM AND STEP-TIME ESTIMATION · p. 26
  • Configuration-dependent effective GPU rates account for tensor-parallel collectives and reduced kernel efficiency as local matrices and pipeline stages become smaller.The single-microbatch pipeline critical path is modeled from stage time; its alternate form is exact only for balanced stages.Source: F PARALLELISM AND STEP-TIME ESTIMATION · p. 26
  • The step-time model combines data-parallel communication, pipeline and compute time, optimizer time, and fixed framework costs, with communication overlap limited by available compute.Pipeline splitting is not cost-free: activation traffic grows with pipeline degree, and smaller stages can lower utilization.Source: F PARALLELISM AND STEP-TIME ESTIMATION · p. 26
  • The fitted latency objective selects TP8/PP4; PP8 is slower because it adds stage-efficiency losses and pipeline overhead without reducing compute after PP4 reaches the compute floor.The estimates are projections, not measurements of every configuration.Source: F PARALLELISM AND STEP-TIME ESTIMATION · pp. 26–27
Loading 2610.15965v1…