PaperlayerView exact revision on arXiv

Source-linked AI summary

From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning

Yupeng Chang, Wenxuan Zhang, Yuan Wu

arXiv:2609.36569v1cs.LGcs.AI

TL;DR

Checkpoint-selection comparisons at a fixed validation size do not distinguish gains from more validation data, gains over NLL selection, and gains over retaining the final checkpoint. The paper varies validation budget while holding training trajectories, candidate checkpoints, and independent test items fixed. More validation data improve generation-based selection and these rules outperform matched NLL, but gains over the final checkpoint remain unresolved.

  • Problem

    Fixed-budget comparisons do not establish whether more validation data improve checkpoint selection, whether a rule beats validation-loss selection, or whether it beats retaining the final checkpoint.

    Source: §1 · p. 1

  • Method

    The study varies nested validation budgets while holding completed trajectories, candidate checkpoints, and independent test items fixed, comparing generated accuracy, checkpoint agreement, and matched NLL.

    Source: §1 · p. 2

  • Results

    +0.32 pp for generated-accuracy selection and +0.29 pp for checkpoint agreement as the mathematical validation budget rises from 32 to 305–313 examples; gains over the final checkpoint remain unresolved.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Benefiting from additional validation data, outperforming NLL selection, and outperforming the final checkpoint are distinct empirical claims requiring separate evidence.

    Source: §1 · p. 3, §5 · p. 9

    The reported bootstrap intervals are conditional on observed configurations and item pools, so dependence from shared model families or datasets may remain.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fixed-budget comparisons do not by themselves distinguish three empirical claims: whether more validation data improve checkpoint selection, whether a selection rule outperforms validation-loss selection, and whether it improves over simply retaining the final checkpoint. We therefore treat checkpoint selection as a finite-information decision problem. Holding completed training trajectories, candidate checkpoints, and independent test items fixed, we vary the validation budget and separately measure improvement from additional validation data, gain over negative log-likelihood (NLL) selection, and gain over the final checkpoint. Across 60 mathematical SFT trajectories and 19 configurations, increasing the validation budget from 32 to 305-313 examples raises independent-test accuracy by 0.32 percentage points (pp) for generated-accuracy selection and 0.29 pp for checkpoint agreement, with 95% configuration-bootstrap CIs of [0.10, 0.56] and [0.11, 0.50], respectively. At the full validation budget, the two generation-based rules outperform matched NLL selection by 0.71 and 0.85 pp, respectively, while their gains over the final checkpoint remain unresolved. A cross-domain replication on 12 newly trained Commonsense trajectories shows the same qualitative separation: increasing the validation budget from 32 to 1,024 questions improves generated-accuracy and checkpoint-agreement selection by 0.87 and 0.27 pp, while gains over the final checkpoint again remain unresolved. Together, these results show that benefiting from more validation data, outperforming NLL selection, and outperforming the final checkpoint are distinct empirical claims that require separate evidence.

1 INTRODUCTION

SFT creates multiple checkpoints but typically retains only one, and visible checkpoint variation does not establish that finite validation data can reliably identify a better one. This section frames selection as a finite-information problem and separates validation-budget gains, gains over NLL selection, and gains over retaining the final checkpoint.

1 INTRODUCTION

  • A median 3.5 pp best-to-worst accuracy spread within an original-grid trajectory shows checkpoint variation, but finite-data selection may not reliably recover it.Fixed-budget comparisons also leave unresolved whether beating validation-loss selection means beating the final checkpoint.Source: §1 · p. 1
  • The paper treats checkpoint selection as a finite-information decision and measures how selected-checkpoint performance on independent test items changes with validation budget.It distinguishes gains from more validation data, gains over validation-loss selection, and gains over retaining the final checkpoint.Source: §1 · p. 2
  • Prior work reports divergence between validation loss and downstream generation quality and finite-sample selection bias, but fixed-size SFT comparisons do not measure budget response or gains over the final checkpoint.The cited prior work includes (Ouyang et al., 2022; Zhou et al., 2023), (Kaur et al., 2025; Ruan et al., 2025), and (Cawley & Talbot, 2010).Source: §1 · p. 2
  • Controlled paired comparisons hold completed trajectories, candidate checkpoints, and independent test items fixed while varying nested validation subsets and comparing generated accuracy, checkpoint agreement, and matched NLL.The primary comparisons evaluate selected checkpoints only on items excluded from selection; a matched-half analysis measures same-item optimism.Source: §1 · p. 2
  • +0.32 pp for generated-accuracy selection and +0.29 pp for checkpoint agreement as the mathematical validation budget rises from 32 to 305–313 examples; Commonsense gains are +0.87 and +0.27 pp, while gains over the final checkpoint remain unresolved.The mathematical results span 60 trajectories and 19 configurations; the Commonsense replication uses 12 new trajectories.Source: §1 · pp. 2–3
  • The contributions frame checkpoint selection around validation budget and distinguish recoverable gains from visible within-trajectory variation.The paper also presents the separate measurement framework and reports that additional validation can improve generation-based selection while gains over the final checkpoint remain unresolved.Source: §1 · p. 3

2 MEASUREMENT DESIGN

The measurement design treats checkpoint selection as a finite-information decision, holding trajectories, candidate checkpoints, and independent test items fixed while varying validation data. It defines distinct selection signals and separates validation-budget gains, gains over matched NLL, and gains over the final checkpoint, with independent evaluation.

2 MEASUREMENT DESIGN

  • The design varies validation budget within completed trajectories while keeping candidate checkpoints and independent test items fixed, isolating validation-data value from early stopping or additional training.Source: §2 · p. 3

2.1 PROBLEM FORMULATION AND SELECTION SIGNALS

  • Generated validation accuracy uses prompts and labels, while checkpoint agreement uses prompts alone and scores how often a checkpoint matches the plurality answer.Checkpoint agreement is transductive because it uses evaluation prompts, so it receives the same sample-splitting treatment.Source: §2.1 · p. 3, §2.3 · p. 4

2.2 MEASURING SELECTION GAINS

  • Validation-budget gains, gains over matched NLL, and gains over the final checkpoint are measured separately because one comparison does not establish the others.The design holds candidate checkpoints and test items fixed while varying nested validation subsets.Source: §2 · p. 3, §2.2 · p. 4

2.3 INDEPENDENT EVALUATION AND STATISTICAL INFERENCE

  • Primary comparisons select on validation data and evaluate frozen checkpoint choices on an independent pool; bootstrap intervals are conditional on observed configurations and item pools.Intervals including zero are treated as inconclusive, not evidence of equivalence.Source: §2.3 · pp. 4–5

3 EXPERIMENTS AND ANALYSIS

Across mathematical SFT analyses, the paper separates validation-budget improvement, recovery over matched NLL, and gains over the final checkpoint; the first two are supported while the third remains unresolved. The analyses also examine same-item optimism and configuration-level variation.

3 EXPERIMENTS AND ANALYSIS

  • The study uses completed SFT trajectories with candidate checkpoints and test items held fixed while nested validation subsets vary; the original grid is nonfactorial and concentrated on small-data mathematical SFT.The mathematical budget analysis uses 60 trajectories across 19 configurations, while the separately analyzed Commonsense replication adds 12 newly trained trajectories.Source: §3 · pp. 5–6, §3.1 · p. 6

3.1 VALIDATION-BUDGET RESPONSE

  • From 32 examples to full validation budget, independent-test accuracy improves +0.32 pp with generated-accuracy selection and +0.29 pp with checkpoint agreement; at full budget, they beat same-budget matched NLL by +0.71 pp and +0.85 pp, respectively.Gains over the final checkpoint remain unresolved: +0.01 pp [−0.33, +0.35] and +0.15 pp [−0.08, +0.37], respectively.Source: Table 1 · p. 6, §3.1 · p. 6
  • In 12 newly trained Commonsense trajectories, increasing the budget from 32 to 1,024 questions improves accuracy by +0.87 pp for generated accuracy and +0.27 pp for checkpoint agreement.At full budget, both rules outperform matched NLL, while gains over the final checkpoint remain unresolved.Source: §3.1 · p. 6, Table 1 · p. 6

3.2 RECOVERY OVER MATCHED NLL AND GAINS OVER THE FINAL CHECKPOINT

  • Across 81 trajectories and 26 configurations, generated-accuracy selection and checkpoint agreement gain +0.74 pp and +0.90 pp over matched NLL, but neither establishes a gain over the final checkpoint.The paired comparisons use the same trajectories, test items, and bootstrap resamples.Source: §3.2 · p. 7, Table 2 · p. 7

3.3 APPARENT AND INDEPENDENTLY EVALUATED GAINS

  • The accuracy-maximizing rule gains +1.33 pp on selection-half items but +0.05 pp on complementary items, a paired difference of +1.29 pp.These gaps measure same-item optimism: apparent gains can be larger on selection items than on unseen items.Source: §3.3 · p. 7, Table 2 · p. 7

3.4 SCOPE AND BOUNDARY EVIDENCE

  • Pooled results hide substantial configuration-level variation: at full validation budget, generated-accuracy gains over the final checkpoint were positive in 9 of 19 configurations and negative in 10; checkpoint-agreement gains were positive in 12 and negative in 7.These signs are descriptive rather than configuration-level significance tests.Source: §3.4 · p. 8

4 RELATED WORK

Prior work motivates generation-based checkpoint selection because validation loss may diverge from downstream generation quality, while finite-sample selection raises uncertainty concerns. Related methods address neighboring objectives, but this paper focuses on which SFT checkpoint to retain.

4 RELATED WORK

  • Validation loss can diverge from downstream generation quality during SFT, motivating reward-model or generation-based alternatives and studies of validation-selection uncertainty.Prior work also reports different ID/OOD tradeoffs for validation-perplexity and validation-accuracy selection.Source: §4 · p. 9
  • Finite-sample model selection, early stopping, and evaluation uncertainty motivate attention to selection bias, validation uncertainty, and stability-aware checkpoint selection.This paper varies validation budget while holding trajectories, candidate checkpoints, and test items fixed, and uses additional analyses to probe same-item optimism and robustness.Source: §4 · p. 9
  • Agreement-based prediction is closely related to the paper’s agreement signal, while verifier supervision, weight averaging, and self-consistency are adjacent objectives or controls rather than checkpoint-selection methods.The paper’s scope is selecting which SFT checkpoint to retain.Source: §4 · p. 9

5 CONCLUSION

The conclusion separates validation-budget gains, gains over matched NLL, and gains over the final checkpoint, which remain unresolved in the studied settings. It recommends reporting these quantities separately and treating the observed best checkpoint as a data-dependent estimate.

5 CONCLUSION

  • Across mathematical SFT trajectories, more validation data improve generation-based selection and these rules outperform matched NLL, but gains over the final checkpoint remain unresolved; Commonsense preserves this separation.Matched-half analysis shows that reusing selection items can overstate apparent gains.Source: §5 · p. 9
  • Report validation-budget response, matched-budget baseline comparisons, and final-checkpoint comparisons as distinct quantities, with choices frozen before independent evaluation and uncertainty reported at the appropriate unit.In the studied settings, the observed best checkpoint is a data-dependent estimate, not automatically a reliable target.Source: §5 · p. 9
  • The evidence does not establish a universal selector or sufficient validation size; it provides a protocol for making checkpoint-selection claims more precisely.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The paper documents its selection, training, measurement, and uncertainty procedures and provides an artifact with records and scripts for verifying reported analyses.

REPRODUCIBILITY STATEMENT

  • Sections 2–3 and Appendices A–D document selection rules, checkpoint grids, validation budgets, uncertainty procedures, and training and measurement protocols.The artifact includes frozen prediction and scoring records, validation plans, checkpoint-choice records, analysis code, and verification scripts for reported tables and figures.Source: REPRODUCIBILITY STATEMENT · p. 10

USE OF LARGE LANGUAGE MODELS

The paper reports how large language models contributed to research and manuscript work, and describes appendix analyses and their evidence hierarchy. Table 4 summarizes the populations and protocols associated with selection-gain analyses.

USE OF LARGE LANGUAGE MODELS

  • Generative AI tools supported methodology and experimental-design feedback, code development, result interpretation and presentation, manuscript editing, and figure preparation; authors reviewed the work and take responsibility.Source: USE OF LARGE LANGUAGE MODELS · p. 10

Appendix

  • Table 4 covers analyses of selection gain whose populations overlap, so its rows are not independent replications; the first three analyses are post-hoc reanalyses.The GSM8K new-pool extension freezes its protocol before inference, while Commonsense uses newly trained trajectories and freezes choices before independent-test evaluation.Source: Table 4 · p. 15

A SUPPORTING ANALYSES

A.1 TASK-LEVEL STRATA

The section reports task-stratified gains descriptively and examines how finite validation budgets, information reuse, and NLL scoring shape checkpoint-selection comparisons.

A.1 TASK-LEVEL STRATA

  • Task-stratum gains use the 81-trajectory, 26-configuration population, and the strata are descriptive rather than independent confirmations.The nonfactorial grid and small subgroup counts preclude interpreting strata as independent replications or causal task/model-family effects.Source: Table 5 · p. 15, §A.1 · p. 15

A.2 MATCHED-HALF SAME-ITEM OPTIMISM

  • Positive Br measures how much larger the apparent selection gain is on the finite selection items than on complementary items at the same size.The matched-half comparison averages split directions and partitions within each trajectory; Br is an information-reuse diagnostic, not an independent replication or population-oracle claim.Source: §A.2 · pp. 15–16

A.3 COMPLETE-RESPONSE NLL CONTROL

  • Complete-response and matched 512-token NLL select the same checkpoint in every mathematical trajectory and have the same paired gain over the final checkpoint.The comparison holds validation examples and checkpoint grids fixed across 60 trajectories.Source: §A.3 · p. 16

A.4 VALIDATION BUDGET AND INDEPENDENT SELECTION GAINS

  • The pooled budget curves summarize heterogeneous configuration-level effects but do not identify a sufficient validation size or a causal difference between generation-based rules.The curves measure adding examples from observed pools, not a sufficient budget, equivalence margin, or causal comparison.Source: §A.4 · p. 16, Table 7 · p. 17
  • Nested subsets of 32, 64, 128, 256, and full-pool examples compare independent-test gain over the final checkpoint with gain over matched NLL.Full pools contain 305–313 examples; repeated subsets come from fixed pools, and intervals condition on those pools.Source: §A.4 · p. 16, Table 7 · p. 17

A.5 CROSS-DOMAIN COMMONSENSE REPLICATION WITH PROTOCOL FROZEN BEFORE TRAINING

The section tests checkpoint-selection findings across Commonsense and several mathematical-data sensitivities, while keeping training trajectories or checkpoint choices fixed where specified. The results distinguish budget gains and NLL comparisons from unresolved superiority over the final checkpoint.

A.5 CROSS-DOMAIN COMMONSENSE REPLICATION WITH PROTOCOL FROZEN BEFORE TRAINING

  • At 1,024 Commonsense questions, increasing the budget improves generated-accuracy selection by +0.87 pp and checkpoint agreement by +0.27 pp, while gains over the final checkpoint remain unresolved.At full budget, the rules exceed matched NLL by +1.05 pp and +1.08 pp, respectively; their final-checkpoint gains are +0.09 pp and +0.12 pp, both unresolved.Source: §A.5 · p. 18

A.6 LARGER SOURCE-QUESTION GSM8K VALIDATION POOL

  • In the larger GSM8K pool, generated accuracy’s full-minus-32 gain is +0.40 pp [+0.06, +0.66], while agreement’s +0.10 pp [−0.29, +0.31] remains unresolved.Both rules’ full-budget contrasts against the final checkpoint and matched NLL remain unresolved; the extension reuses existing trajectories and test predictions.Source: §A.6 · p. 18

A.7 ONE VARIANT PER ORIGINAL QUESTION

  • With one variant per original question, the pooled budget benefit and advantage over matched NLL persist, without establishing superiority to the final checkpoint.The full endpoint has 228–244 unique questions; this sensitivity changes question weighting and does not isolate a causal effect of deduplication.Source: §A.7 · p. 19, Table 10 · p. 19

A.8 FROZEN-CHOICE TEST-QUESTION SENSITIVITY

  • Across separate configuration and test-question sensitivity views, full-minus-32 gains and contrasts against matched NLL remain positive, while superiority to the final checkpoint remains unresolved.The two interval families condition on different fixed quantities and are neither combined into a joint interval nor independent replications.Source: §A.8 · p. 20, Table 11 · p. 20

B CHECKPOINT-GRID SENSITIVITY

Checkpoint-grid thinning preserves the qualitative finding that matched NLL performs below the final checkpoint in the tested trajectories. The section also documents the training and evaluation configuration behind the primary grid.

B CHECKPOINT-GRID SENSITIVITY

  • Keeping every second, third, or fourth checkpoint leaves the paired NLL gain over the final checkpoint negative: −0.84, −0.77, and −0.57 pp, respectively.Mean grid size falls from 13.84 to 7.60, 5.60, and 4.80; this sensitivity uses existing trajectories, not an independent replication.Source: B CHECKPOINT-GRID SENSITIVITY · p. 21

C.1 TRAINING, CHECKPOINTS, AND EVALUATION DATA

  • The 24 primary configurations use response-masked cross-entropy with AdamW, linear learning-rate decay, BF16, zero weight decay, and a 5% warmup ratio.LoRA uses peak learning rate 2 × 10−4 and default rank 8; full fine-tuning uses 2 × 10−5.Source: §C.1 · p. 21, Table 12 · p. 21
  • Primary seeds are 42–44, with 45–47 added for Llama MATH; retained checkpoints require proxy and task measurements and a final checkpoint.Partially evaluated trajectories are excluded rather than assigned a shorter effective budget.Source: §C.1 · p. 21
  • The eight Commonsense evaluation tasks contribute 500 items each, so pooled accuracy is an equal-weight task average and one task-group score per trajectory.The listed tasks include ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, SocialIQA, and WinoGrande.Source: §C.1 · pp. 21–22

C.2 TRAINING AND VALIDATION SOURCE RECORDS

The source-record audit reconstructs training–validation separation under recorded identifiers, while noting limits on historical provenance and semantic separation. The evaluation audit reports no detected overlap under its specified checks.

C.2 TRAINING AND VALIDATION SOURCE RECORDS

  • The reconstructed source splits have zero original-question-group and exact-query intersections between training and validation under the recorded identifiers.The audit assigns whole source-question groups deterministically; Table 13 reports records and distinct groups, not historical per-run consumed-record manifests.Source: §C.2 · p. 22, Table 13 · p. 22
  • Run-level provenance is unavailable for 18 imported trajectories, and the historical Mistral–MATH source was not recovered, limiting certification across all runs.The reconstruction establishes zero overlap only where the source split can be reconstructed.Source: §C.2 · p. 22
  • Exact source grouping does not establish semantic separation between rewritten questions or absence of base-model pretraining exposure.The source audit’s scope is bounded by recorded identifiers and available artifacts.Source: §C.2 · p. 22

C.3 TEACHER-FORCED SCORING AND EVALUATION SEPARATION

  • The frozen overlap audit flags none of 5,819 evaluation questions using exact, normalized, and character 13-gram checks at a 0.8 threshold.A planted positive control recovers intended duplicates, but the checks do not rule out semantic duplicates or pretraining exposure.Source: §C.3 · p. 22

D LIMITATIONS

The evidence is limited to fixed-budget, greedy-decoded SFT in a nonfactorial model/task grid, so the findings support distinctions among selection claims within these settings rather than a universal ranking or threshold.

D LIMITATIONS

  • The experiments cover fixed-budget, greedy-decoded SFT up to 14B across a nonfactorial model/task grid.The cross-domain Commonsense replication and larger-pool GSM8K analysis broaden the evidence but do not cover all architectures, tasks, or training regimes.Source: D LIMITATIONS · p. 22
  • The Commonsense replication and larger-pool GSM8K analysis broaden the evidence without covering all architectures, tasks, or training regimes.Source: D LIMITATIONS · p. 22
  • The results distinguish validation-budget improvement, gain over matched NLL, and gain over the final checkpoint within these settings, not a universal selector ranking or validation-size threshold.Source: D LIMITATIONS · p. 22
Loading 2609.36569v1…