PaperlayerView exact revision on arXiv

Source-linked AI summary

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges

arXiv:2609.08966v1cs.AIcs.CL

TL;DR

The paper asks whether checkpoints that score best before downstream training also produce the best final models. It compares checkpoints through a full 30B MoE training stack and finds that intermediate rankings can reverse, while later-stage quality is associated with solution density. The results support selecting checkpoints based on performance after the remaining pipeline, within the tested scope.

  • Problem

    Intermediate checkpoint selection assumes that lower loss or higher benchmark scores remain predictive after later training, but the paper asks whether that assumption holds across a multi-stage pipeline.

  • Method

    The study compares CONSTANT, COOLDOWN, and MERGE checkpoints across a 30B MoE pipeline with pretraining, mid-training, long-context adaptation, and SFT, including learning-rate sweeps.

  • Results

    Subsequent training reverses checkpoint rankings; retuning SFT raises COOLDOWN’s aggregate from 0.247 to 0.294 but does not resolve its stopping failure.

  • Takeaways & Limitations

    Checkpoint selection should target performance after the remaining training pipeline rather than current scores alone.

  • Takeaways & Limitations

    The results come from one 30B MoE family with one seed per setup, and solution density is measured on only two tasks without establishing causation.

Abstract

from arXiv · show

Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.

1 Introduction

Checkpoint selection for modern multi-stage training asks which source will produce the best final model, not merely which scores best immediately. In a 30B MoE pipeline, later training can reverse intermediate checkpoint rankings, motivating evaluation of downstream compatibility and solution density.

  • Modern language-model training spans multiple stages, so checkpoint selection must target the best final model rather than the best current score.
  • Executing every training stage for every candidate is expensive, making intermediate evaluation an attractive but consequential selection strategy.
  • Subsequent training can reverse the ranking of intermediate checkpoints.
  • CONSTANT and COOLDOWN share the first 6.7T of 7.5T tokens, yet their ordering reverses after identical mid-training, long-context adaptation, and SFT.
  • The paper connects checkpoint reversal to solution density and examines whether intermediate rankings predict final rankings.

2 Experimental Design

The experiments compare checkpoints and learning-rate schedules in a 30B MoE trained through pretraining, mid-training, long-context adaptation, and conversational SFT. The design includes checkpoint merging and a nine-configuration mid/long learning-rate sweep, with stage-specific aggregate evaluations.

  • The 30B MoE uses 7.5T-token pretraining followed by 100B-token mid-training, 100B-token long-context adaptation, and 10B-token conversational SFT.
  • CONSTANT and COOLDOWN share 6.7T tokens, while COOLDOWN decays its learning rate to 10% of maximum during the final 800B tokens.
  • MERGE combines 20 equally spaced CONSTANT checkpoints over a 600B-token trailing window without additional training.
  • The learning-rate sweep tests nine combinations of mid-training and long-context peak-rate factors, including schedules with cosine decay across stages.
  • The (1, 1) mid/long schedule achieved the highest post-SFT aggregate among the nine schedules and was reused for CONSTANT and MERGE.
  • Evaluations use completion benchmarks through long-context adaptation and six chat-formatted benchmarks after SFT, with aggregate scores formed from cluster means or an unweighted mean.

3 Checkpoint Rankings Change Across the Training Stack

Pretraining and mid-training scores do not reliably determine post-SFT checkpoint quality across sources, while long-context scores become more predictive within the learning-rate sweep. However, matched long-context scores still fail to predict the CONSTANT–COOLDOWN reversal.

  • 3.1 Pretraining scores do not determine post-SFT scores: COOLDOWN improves CONSTANT’s train loss, validation loss, and pretraining aggregate, yet CONSTANT leads COOLDOWN by 0.113 after the identical downstream pipeline.
  • 3.1 Pretraining scores do not determine post-SFT scores: MERGE has the highest aggregate before and after downstream training, but its final advantage over CONSTANT is only 0.003.
  • 3.1 Pretraining scores do not determine post-SFT scores: The post-SFT benchmark profile shows that COOLDOWN is not uniformly worse, while excluding HumanEval+ reduces the aggregate gap to 0.016.
  • 3.3 Long-context rankings stabilize within a sweep, not across sources: Across eleven trajectories, mid-training correlates weakly with post-SFT aggregate, with Pearson r = 0.473 and Spearman ρ = 0.482.
  • 3.3 Long-context rankings stabilize within a sweep, not across sources: After long-context adaptation, correlations rise to Pearson r = 0.884 and Spearman ρ = 0.964, while the relationship becomes almost monotone.
  • 3.3 Long-context rankings stabilize within a sweep, not across sources: Within COOLDOWN trajectories, long-context correlation reaches r = 0.935 and ρ = 0.950, but matched scores still fail to predict source ordering: 0.637 versus 0.636 precedes a 0.113 post-SFT gap.
  • 3.3 Long-context rankings stabilize within a sweep, not across sources: The authors do not claim that long-context adaptation causally creates predictiveness because later checkpoints are also closer to the final model.

4 What Checkpoint Scores Miss

The paper examines why intermediate checkpoint scores can miss properties relevant to subsequent training, using solution density as a local robustness probe. Perturbation results align with downstream trainability, while retuning SFT does not remove COOLDOWN’s stopping failure.

  • Solution density: Solution density measures whether nearby perturbed parameters retain benchmark performance, providing a local-geometry view of checkpoint quality.The probe evaluates an external capability rather than pretraining loss around the checkpoint.
  • Solution density: At τ = 0.90 on GSM8K, solution density is 27% for CONSTANT, 13% for MERGE, and 0% for COOLDOWN.The separation persists at σϵ = 0.001.
  • Solution density: The perturbation ordering matches downstream trainability: CONSTANT and MERGE finish at 0.360 and 0.363, while COOLDOWN reaches 0.247.The authors hypothesize that local robustness helps checkpoints tolerate parameter displacement during subsequent optimization.
  • Retuning SFT: COOLDOWN’s largest deficit is HumanEval+ at 4.9%, versus 64.6% for CONSTANT and 66.5% for MERGE.Raw outputs also show repetitive responses reaching the token cap on nearly all AIME and GPQA examples.
  • Retuning SFT: Changing the SFT factor from 1/3 to 1 raises the aggregate from 0.247 to 0.294 but leaves 100% of AIME and 99% of GPQA responses at the cap.No tested rate jointly resolves the stopping failure.
  • Retuning SFT: Sweeping all three SFT factors across 27 COOLDOWN checkpoints leaves every checkpoint below fixed-recipe CONSTANT and MERGE checkpoints.Using only the first generated code block raises HumanEval+ from 7.3% to 61.0%, indicating latent code capability despite repeated later blocks.

5 Related Work

Prior work shows that pretraining metrics need not predict downstream adaptation, and relates learning-rate schedules and local geometry to later trainability. This paper extends that work by tracking checkpoint rankings across multiple training stages.

  • Checkpoint quality and downstream adaptability: Matched pretraining loss can correspond to substantially different downstream transfer performance, while learning-rate decay can improve pretraining metrics but harm continued training and SFT.These findings motivate evaluating checkpoints beyond their current pretraining scores.

6 Conclusion

The conclusion is that checkpoint quality depends on the remaining training pipeline rather than current scores alone. The evidence connects downstream adaptability with solution density while preserving important scope and causal limitations.

  • Conclusion: COOLDOWN has better pretraining metrics than CONSTANT but performs worse after the downstream pipeline.Checkpoint rankings can reverse during later training.
  • Conclusion: Checkpoint selection should target performance after the remaining pipeline, not the current score alone.Intermediate aggregates do not anticipate the reversal between CONSTANT and COOLDOWN.
  • Limitations: The study covers one 30B MoE family with one seed per setup, does not test every settings combination, and measures solution density on only two tasks.The authors also note that solution density does not establish causation and that COOLDOWN’s stopping failure enlarges the observed gap.
  • Solution density: Flatness and solution density probe the same local parameter neighborhood with different functions and are not equivalent measures.Pretraining-loss flatness evaluates loss changes, whereas solution density evaluates preservation of an external capability.
  • Solution density: CONSTANT and MERGE have more performance-retaining neighborhoods than COOLDOWN, consistent with a connection between local geometry and subsequent adaptability.The paper uses solution density as a diagnostic rather than a causal explanation.

B.1 Architecture

The model is a roughly 30B-parameter decoder-only MoE Transformer, pretrained with next-token prediction before downstream checkpoint construction and training stages.

  • Architecture: The model has 50 Transformer layers: two dense layers followed by 48 mixture-of-experts layers.It has approximately 30B total parameters and activates approximately 3B parameters per token.
  • Pretraining: Pretraining uses causal next-token prediction on a diverse English- and German-containing corpus packed into 4,096-token sequences.
  • MERGE construction: MERGE forms a weighted average of a trailing window of CONSTANT checkpoints, with weights increasing linearly from oldest to newest.The oldest checkpoint receives weight 1/210 and the newest receives weight 20/210.
  • MERGE construction: Although checkpoint weights rise linearly, cumulative update coefficients decline from q1 = 1 to q20 = 20/210 ≈ 0.095.This attenuates later updates after training without following a cosine-cooldown optimization trajectory.

C Evaluation Suites and Aggregate Construction

The evaluation framework uses stage-specific benchmark suites and aggregates scores by cluster before SFT, while directly averaging six benchmark values after SFT.

  • Evaluation suites: Pre, Mid, and Long use completion-style suites, while SFT uses chat-formatted prompts and stage-specific adapters.Mid adds long-context evaluation, and Long uses larger code and long-context suites.
  • Aggregate construction: Pre, Mid, and Long first average scores within clusters and then weight cluster means equally; SFT directly averages six benchmark values.
  • Aggregate construction: Pre averages three cluster means, whereas Mid and Long average four cluster means.
  • Aggregate construction: Individual pre-SFT benchmarks do not have equal final weight because cluster means, rather than raw score counts, determine the aggregate.HumanEval alone forms the Code EN cluster and receives one third of the aggregate rather than one ninth.
  • Aggregate construction: The SFT aggregate is an unweighted mean of six benchmark values, with IFBench contributing one value formed from loose and strict prompt-level scores.Example-level correct counts are never pooled across benchmarks.

D Complete Training Trajectories

The trajectory analysis evaluates intermediate aggregates against post-SFT outcomes across eleven training trajectories, finding much stronger rank association after long-context adaptation than after mid-training.

  • Trajectory analysis: All eleven training trajectories enter the reported correlations, whose post-SFT outcomes are represented by aggregate scores.
  • Correlation results: After mid-training, Pearson coefficients have 95% confidence intervals of [−0.177, 0.836], compared with [0.605, 0.970] after long-context adaptation.
  • Correlation results: Within the nine COOLDOWN trajectories, mid-training correlations are r = 0.448 and ρ = 0.450, while long-context correlations are r = 0.935 and ρ = 0.950.The mid-training tests have p = 0.226 and p = 0.224; the long-context tests have p < 0.001 for both coefficients.

E Stopping and Repetition Audit

The stopping audit finds extensive repetition in COOLDOWN outputs, with the pathology producing different penalties across evaluation contracts and persisting after increasing the SFT learning-rate factor.

  • Audit design: The audit covers 294 IFBench prompts, 240 AIME generations, and 198 GPQA Diamond prompts per checkpoint under a 32,768-token cap.
  • Stopping behavior: COOLDOWN is substantially less likely to terminate than CONSTANT and MERGE, with median zlib-to-raw ratios of 0.019, 0.044, and 0.027 across IFBench, AIME, and GPQA.The corresponding CONSTANT ratios are 0.288, 0.310, and 0.059; MERGE ratios are 0.299, 0.328, and 0.066.
  • SFT-rate diagnostic: Changing COOLDOWN from SFT factor 1/3 to factor 1 does not repair the behavior; compression ratios remain 0.022, 0.053, and 0.031.
  • SFT-rate diagnostic: On the fixed (1, 1) upstream trajectory, increasing the SFT factor from 1/3 to 1 raises the post-SFT aggregate from 0.247 to 0.294.This is the highest recorded complete COOLDOWN result, while its original HumanEval+ score is 0.067.
  • SFT-rate diagnostic: Lower SFT learning rates improve HumanEval+ on average while reducing reasoning and overall aggregates, so no tested factor dominates all recorded capabilities.
  • Qualitative examples: One GPQA response repeats a sentence 2,012 times, reaches the cap mid-word, and is marked invalid despite stating the correct option earlier.
  • Qualitative examples: The same stopping failure produces different measured penalties under whole-response, compact-answer, and final-block evaluators.

F HumanEval+ Re-execution Audit

The HumanEval+ audit isolates extraction as the changed scoring variable and finds that COOLDOWN’s apparent weakness reflects long, repetitive responses with passing early programs but failing final programs. A fixed-recipe COOLDOWN checkpoint cannot be rescored because its raw generations are unavailable.

  • Audit design: The audit compares alternative fenced-code extraction rules on identical generations and re-executes programs in the original 164 network-disabled HumanEval+ test suites.Re-executing the final block reproduces every original verdict, so extraction is the only changed scoring variable.
  • Audit findings: Every gain from first-block extraction comes from responses with a passing early program and a failing final program.No last-block success becomes a first-block failure for either audited COOLDOWN cap.
  • Audit findings: Increasing the output cap does not repair COOLDOWN’s behavior: first-block accuracy remains near 60% while generations continue to contain thousands of repeated blocks.The observed issue therefore persists under a larger generation budget.
  • Scope boundary: The fixed-recipe COOLDOWN SFT-1/3 checkpoint cannot receive corrected first-block scoring because its retained artifact contains aggregate scores but no raw generations.Its original HumanEval+ score is 8/164 (4.88%), but the corrected score is not transferred to the primary source comparison.
Loading 2609.08966v1…