PaperlayerView exact revision on arXiv

Source-linked AI summary

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T. Ash, Akshay Krishnamurthy

arXiv:2608.25756v1cs.LGcs.AI

TL;DR

Existing SFT pipelines may produce checkpoints that score well on local objectives but are not optimal initializations for RL because coverage and single-sample accuracy can diverge. The paper introduces TailSFT, which filters already-fit sequences and uses initial-policy-relative progress to target under-modeled responses. On OLMo-3 7B, TailSFT improves large-K pass@16 and subsequently improves pass@1 after GRPO, motivating stage-aware checkpoint evaluation.

  • Problem

    The paper asks whether SFT can be modified to produce better initializations for subsequent RL when cross-entropy and useful response coverage are misaligned.

  • Method

    TailSFT is a sequence-level filtering method that removes already-fit sequences and measures progress relative to the initial policy to focus training on the distributional tail.

  • Results

    TailSFT improves pass@16 by up to 16.8% on coding and 3.1% on math, and improves final pass@1 after GRPO by up to 3.9%.

  • Takeaways & Limitations

    Intermediate checkpoints can be evaluated by how effectively they support subsequent training, with high large-k pass@k potentially more valuable than lower cross-entropy for SFT-to-RL.

  • Takeaways & Limitations

    In the homogeneous graph task, the initial-policy reference provides no useful distinction across examples, making TailSFT slightly weaker than other filtering variants.

Abstract

from arXiv · show

Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.

1 Introduction

The paper asks whether SFT checkpoints should be optimized as initializations for later RL rather than only for their local objective. It proposes TailSFT, which filters already-fit sequences to improve coverage and reports stronger post-RL performance.

  • Why coverage matters: Large-K pass@K measures whether repeated sampling exposes RL to reward-bearing responses, which can matter more than single-sample accuracy.The coverage principle connects this property to the learning signal available during RL.
  • Motivation and method: Standard cross-entropy can shift probability away from useful responses that are already represented by the initial model, so local SFT quality and coverage may diverge.A model with worse cross-entropy can nevertheless have better coverage.
  • Motivation and method: TailSFT filters already-fit sequences, directing SFT toward under-modeled responses and prioritizing coverage over low cross-entropy.It is designed as a lightweight, sequence-level drop-in objective for SFT.
  • Results: 16.8% on coding and 3.1% on math are TailSFT’s maximum absolute pass@16 gains on OLMo-3 7B.These results are reported on standard coding and math tasks.
  • Results: 3.9% is TailSFT’s maximum absolute improvement in final pass@1 after GRPO.The reported result links higher-coverage SFT checkpoints with improved post-RL performance.
  • Diagnostic and broader view: A lightweight coverage-ratio diagnostic uses the base model and one standard SFT run to identify settings where TailSFT is likely to improve coverage.The paper presents a sufficient condition for TailSFT to improve coverage.
  • Diagnostic and broader view: The broader implication is that intermediate checkpoints should be judged by how well they support subsequent training, not only by their current-stage objective.For SFT-to-RL, high large-k pass@k can be more valuable than lower cross-entropy when the criteria conflict.

2 Preliminaries on Coverage

The preliminaries formalize coverage as the accessibility of rewarding responses through repeated sampling. Large-K pass@K serves as an empirical proxy for this property and for the reward signal available to RL.

  • Coverage concept: Coverage describes how readily a model can sample reward-bearing responses during repeated rollouts.The initial probability of rewarding responses determines the chance that RL receives useful learning signal.
  • Coverage concept: For a prompt with rewarding-response probability pπ(x), K independent rollouts contain at least one rewarding response with probability 1 − (1 − pπ(x))^K.If all rollouts receive zero reward, no gradient is received for that prompt.
  • Coverage profile: The coverage profile measures the data-policy probability mass that the model underweights by at least a factor of N; smaller values indicate better coverage.The scale N corresponds to the sampling budget needed to reach that mass.
  • Coverage profile: When Cov_N(πD∥π) ≤ 1/2, K ≥ 2N log(1/ε) samples suffice for Best-of-K to approach the data policy’s expected reward within Cov_N(πD∥π) + ε.A comparable worst-case necessity result relates N and K.
  • Empirical measure: For binary rewards, pass@K(π) = E_x∼µ[1 − (1 − pπ(x))^K] equals the expected reward of Best-of-K sampling.At K comparable to the RL rollout budget, it measures how often the policy exposes RL to reward-bearing responses.

3 TailSFT: Supervised Fine-Tuning for Coverage

TailSFT filters sequence-level SFT updates to preserve coverage, using progress relative to the initial policy to target under-modeled responses. Controlled experiments and theory support filtering, while also identifying limits of the simple graph setting and absolute thresholds.

  • TailSFT design: Standard SFT keeps increasing every training response’s likelihood, whereas coverage improves by increasing probability on responses that remain insufficiently reachable.Once a response is adequately covered at a sampling scale, further likelihood increases do not improve coverage at that scale.
  • TailSFT design: TailSFT redirects updates away from responses whose losses have decreased most and toward responses that remain under-modeled.Filtering uses length-normalized loss for selection, while optimization retains token-averaged cross-entropy on retained sequences.
  • Filtering criteria: TailSFT measures progress relative to the initial policy, unlike absolute filtering’s common threshold and quantile filtering’s batch-relative rule.The adaptive stopping point reflects the probability initially assigned to each response.
  • Controlled experiment: Standard SFT attains the best cross-entropy and pass@1, while all three filtering methods achieve substantially higher pass@8 on graph navigation.The result demonstrates a trade-off between local SFT metrics and repeated-sampling coverage.
  • Controlled experiment: The graph task provides evidence that filtering already-fit examples improves coverage, but TailSFT is slightly weaker there because homogeneous prompts make the initial-policy reference uninformative.Pretraining imperfections add noise to the offset score without useful distinguishing signal in this setting.
  • Theoretical analysis: In the theoretical setting, the target preserves the initial policy’s relative preferences among rewarding responses, although the initial model may overallocate total probability to unrewarding responses.This motivates using the initial policy to preserve useful distributional structure.
  • Theoretical analysis: Offset filtering can be tuned to achieve coverage no worse than standard SFT or the best absolute threshold, and the inequality can be strict.The theorem formalizes the advantage of using initial-policy-relative progress.
  • Theoretical analysis: The initial policy contains distributional information that finite-sample standard SFT may overwrite, whereas absolute filtering pushes observed responses toward a common threshold.Offset filtering is designed to retain this initial structure.

4 Language Model Experiments

TailSFT improves large-K coverage across most language-model evaluations and uses a coverage-ratio diagnostic to identify settings where it is most likely to help. Higher-coverage TailSFT initializations consistently improve post-GRPO pass@1 and can accelerate early learning.

  • 4.1 SFT Results: Across 18 dataset–benchmark pairs, TailSFT improves pass@16 in 15 cases, while pass@1 changes are mixed.The largest gain is 16.79% on CruxEval-O after SFT on BigCode; OMEGA-500 is essentially unchanged at −0.20%.
  • 4.2 Coverage Ratio Diagnostic: TailSFT filters already-fit responses to preserve under-modeled, base-reachable responses during SFT.The coverage ratio is computed from the base model and a single standard SFT run; ρ16 > 1 means estimated coverage loss exceeds gain.
  • 4.3 GRPO Results: TailSFT improves post-GRPO pass@1 in every matched comparison, with absolute gains ranging from 1.21% to 3.93%.In coding experiments, TailSFT starts with lower pre-RL pass@1 but finishes higher after RL than standard SFT.
  • 4.3 GRPO Results: TailSFT runs can increase early reward up to 2.5× faster, retain higher post-RL pass@16 in four of five comparisons, and tie in the fifth.The TailSFT runs begin with lower reward, but the gap closes quickly during GRPO.

5 Discussion

The discussion frames TailSFT as a lightweight, stage-aware alternative to independently optimizing each training phase. It targets response coverage before RL while leaving the two-stage pipeline and RL algorithm unchanged.

  • Stage-aware model development: TailSFT is a drop-in SFT replacement that filters already-fit sequences to improve coverage before RL.Its design combines theoretical analysis and controlled empirical validation.
  • Data selection and diversity-preserving SFT: Unlike data-selection methods that preserve the full-data objective, TailSFT changes the effective SFT objective to improve coverage and post-RL performance.It uses loss reduction relative to the initial policy and masks the most-improved sequences online.
  • Coupling supervised and reinforcement training: TailSFT leaves the two-stage pipeline and RL algorithm unchanged, unlike methods that couple supervised and reinforcement objectives.Related approaches alter when or how SFT and RL objectives are combined.
  • Coverage and test-time compute: TailSFT operationalizes the coverage principle as an online SFT rule, improving useful coverage before RL rather than relying solely on RL-stage exploration.The paper situates this approach among work on coverage, test-time compute, sharpening, and latent capabilities.
  • Stage-aware model development: TailSFT extends stage-aware evaluation from diagnosing post-RL proxies to modifying SFT and validating the initialization through matched RL runs.The discussion emphasizes targeting the full training trajectory rather than fitting each stage independently.

B Theoretical Analysis: TailSFT in the expert conditioning setting

The analysis models SFT as expert conditioning and compares standard empirical risk minimization with TailSFT's absolute and relative clipping variants through coverage-aware optimization. Its theorem establishes that offset clipping is never worse than ERM, can dominate it, while absolute clipping can be worse than ERM.

  • Setup: Expert conditioning defines the downstream policy by conditioning the pretrained distribution on a response subset S.The simplified setting uses response distributions without prompts, with π⋆(y) proportional to πref(y) for y in S.
  • Objectives: TailSFT ignores examples whose model probability exceeds an absolute threshold or a reference-scaled threshold.Absolute clipping ignores yi when π(yi) ≥ α; relative, or offset, clipping ignores it when π(yi) ≥ βπref(yi).
  • Objectives: The optimization minimizes forward KL to πref among policies minimizing the selected loss, preserving coverage through mode-covering behavior.The losses considered are ERM, absolute clipping, and offset clipping.
  • Theorem: Offset clipping is always preferred to empirical risk minimization across reference policies, subsets, and datasets.The theorem's first claim also holds when restricted to parameter regimes where absolute-clipping zero loss is achievable.
  • Theorem: Offset clipping can dominate, whereas absolute clipping can be worse than ERM in suitable reference-policy and expert-subset instances.These comparisons hold in the theorem's stated large-response-space or high-probability regimes.

B.1 Proof of Theorem B.1

The proof characterizes KL-projected clipped minimizers and compares their coverage of expert-supported responses. It shows why offset clipping can preserve unseen support while absolute clipping and ERM may leave it uncovered.

  • Proof strategy: The proof first characterizes solutions to the KL-projection problem, then analyzes ERM, absolute clipping, and offset clipping coverage.The structure of each optimizer supplies the comparisons used in Theorem B.1.
  • Optimizer structure: KL projection under capped lower bounds has the form π(y) = min(fy, λp(y)), with λ selected by normalization.This structural form underlies the clipped-loss analysis.
  • Offset clipping: When N ≥ 1/πref(S), choosing β = 1 makes πref a zero-loss offset-clipping minimizer with zero coverage under the stated criterion.The result describes a regime where the reference policy itself is retained.
  • Offset clipping: Offset clipping can preserve coverage of observed expert actions while assigning only λπref(y) to unobserved actions.When zero loss is feasible, observed actions receive at least βπref(y), whereas unobserved actions remain reference-scaled.
  • ERM: ERM assigns zero mass to unobserved expert actions, yielding coverage at least πref(S \ ˆS)/πref(S).This follows because ERM equals the empirical distribution over sampled responses.

C.1 Graph Reasoning Task and Data

The graph reasoning benchmark uses layered directed acyclic graphs whose valid paths encode responses, separating easy pretraining navigation from a difficult deterministic SFT path-selection rule. This controlled construction tests whether filtering fit examples improves coverage.

  • Task: The path-following task abstracts reasoning as selecting a valid source-to-target path in a directed acyclic graph.The task is designed as a minimal and flexible setting for studying TailSFT interventions.
  • Graph construction: Each graph has 10 layers, with edges only between consecutive layers and valid paths traversing passable nodes.The source and target occupy the first and final layers, while intermediate layers contain four vertices each.
  • Graph construction: Graphs vary by k layers containing two passable nodes, producing 2^k valid source-to-target paths.The remaining layers contain exactly one passable node, and the locations of branching layers are randomized.
  • Data: Pretraining samples valid paths uniformly, teaching generic graph navigation, while SFT deterministically selects one path using a shard-dependent global rule.The shard is computed from parity information across graph layers.
  • Difficulty: The SFT policy is challenging because matching it requires identifying global graph structure rather than only local passability rules.Reward is one only when the model's path matches the SFT-selected path.

C.2 Training Details

The synthetic graph-navigation experiments use a GPT2-style model with fixed pretraining and SFT training schedules, while Figures 5 and 6 report absolute- and quantile-clipping sweeps.

  • Training setup: The synthetic graph-navigation setup uses a GPT2-style transformer with 4 heads, 6 blocks, and 384-dimensional embeddings.Pretraining uses 256,000 fixed graphs, while SFT generates new graphs online.
  • Training schedule: Pretraining runs for 200k steps and SFT runs for 50k steps, with evaluation every 200 steps.The optimizer is Adam with learning rates of 1 × 10−4 for pretraining and 5 × 10−6 for SFT.
  • Hyperparameter sweeps: Figures 5 and 6 detail TailSFT hyperparameter sweeps for absolute clipping and quantile clipping, respectively.Both figures concern the synthetic graph-navigation experiment.

C.3 Additional Experimental Results

Additional graph-navigation results compare clipping variants across hyperparameter sweeps, showing that quantile-based methods are more robust than absolute clipping.

  • Sweep behavior: Absolute clipping is more sensitive to its hyperparameter than quantile and offset-quantile clipping.Absolute thresholds require tracking how per-example cross-entropy distributions evolve during training.
  • Sweep behavior: Quantile-based clipping is easier to set because thresholds are defined per batch and is comparatively robust.This applies to both loss quantiles and offset-loss quantiles.
  • Sweep behavior: For quantile variants, thresholds that are too small recover standard SFT, while thresholds that are too large slow convergence without changing pass@8.Large clipping parameters retain focus on harder examples but reduce computational efficiency.
  • Reported analyses: Figures 5–7 provide detailed sweeps for absolute, quantile, and offset-quantile TailSFT variants.The figures cover the three clipping choices in the synthetic graph-navigation experiment.

D.1 SFT

TailSFT modifies supervised fine-tuning by filtering already-fit sequences relative to the initial policy, directing training toward under-modeled responses while preserving coverage.

  • Method: TailSFT applies the same sequence-level filtering algorithm to math and code, with domain-specific data, prompts, grids, and evaluations.The shared procedure is described separately from domain-specific details.
  • Method: TailSFT drops already-fit examples by comparing each current loss with its base-model initialization loss.Both losses are mean, length-normalized cross-entropies over target tokens.
  • Method: The method filters examples with the most negative loss margins, while standard token-averaged cross-entropy is optimized on retained sequences.The selection mask is computed on each selection batch across all data-parallel ranks.
  • Method: The filter fraction can be fixed or scheduled, including a 0→f ramp that increases filtering over training.Standard SFT corresponds to f = 0.
  • Findings: Filtering usually matches or improves pass@16 coverage, with its main cost being a small pass@1 decrease.TailSFT can recover coverage above the base model when standard SFT lowers it below baseline.
  • Findings: Pass@16 gains are distributed across many problems rather than concentrated in a small subset.Table 5 decomposes gains into paired per-problem changes.

D.1.1 Code

The code experiments evaluate TailSFT across several decontaminated instruction datasets and benchmarks, emphasizing pass@16 coverage and its relationship to standard SFT.

  • Training details: Code SFT uses sequence length 2048, an effective batch size of 32, and learning rates searched over {1, 2, 3} × 10−5.Runs use no weight decay, per-rank microbatch 1, and four gradient-accumulation steps.
  • Datasets: The code datasets are decontaminated and format-converted before training, with Table 6 summarizing dataset row counts.Magicoder retains 33,817 training rows after filtering and token-length removal; BigCode retains 49,260.
  • Coverage results: On OCI, filtering improves MBPP+ coverage by +1.15 pp but reduces HumanEval+ coverage by −2.74 pp.The OCI HumanEval+ coverage ratio is ρ16 = 0.23, outside the diagnostic’s ρ16 > 1 regime.
  • Coverage results: TailSFT’s largest coverage gains occur where standard SFT reduced coverage below the base model, including CruxEval-I.For BigCode, standard SFT falls from 70.5 to 61.7; for Magicoder, it falls to 59.8.
  • Evaluation: Evaluation samples 16 generations per problem at temperature 1.0 and top-p = 1.0 across three independent seed sets.The protocol follows the OLMES EvalPlus configuration used in the OLMo-3 code suite.
  • Evaluation: Pass@16 measures the fraction of documents with at least one correct completion, whereas pass@1 is the mean per-document pass rate.Both metrics are computed from 16 samples rather than a best-of-k maximum.
  • Coverage results: TailSFT’s pass@k advantage generally grows with k, reaching +16.79 on CruxEval-O for BigCode at k=16.HumanEval+ and LiveCodeBench are exceptions where TailSFT trails at every k.

D.1.2 Math

The math experiments evaluate TailSFT filtering on OLMo-3 7B across data construction, hyperparameter selection, pass@k coverage, and subsequent GRPO initialization. Results emphasize larger-k coverage gains, especially on harder problems, with a retained post-GRPO pass@1 advantage.

  • Setup: Math SFT uses a 350k-example OpenMathInstruct-2 subset, two training epochs, and a fixed learning rate of 3 × 10−5 across filtering arms.The selected filtering configuration is the 0→0.5 ramp.
  • SFT results: Filtering gains concentrate on harder MATH-500 levels, while no MATH-500 pass@16 cell drops by more than 1.8 pp.Easy levels are near saturation and therefore obscure where filtering acts.
  • SFT results: AIME pass@16 improves across completed rows, whereas the frozen OMEGA-500 arm is essentially tied with no filtering.The reported benchmark pattern indicates gains are not uniform across every evaluation setting.
  • Coverage: AIME’s TailSFT-minus-Standard pass@16 gap grows from +0.37 to +3.07, while MATH-500 Level 5 grows from +0.36 to +2.74 as k increases.OMEGA-500 remains flat because the two models are already matched.
  • GRPO: TailSFT initialization retains about +2 points of post-GRPO pass@1 on MATH-500 Level 5.The comparison uses matched Standard and TailSFT initializations.

D.2.2 Code

The code section describes matched Standard and TailSFT initializations for GRPO and evaluates their training dynamics and coverage-related metrics. TailSFT recovers quickly during GRPO and its early reward can increase up to roughly 2.5× faster.

  • GRPO setup: Code GRPO compares matched no-filter and filtered SFT checkpoints, differing only in how their initializations were trained.The shared procedure varies by dataset-specific data, reward, hyperparameter grid, and evaluation.
  • GRPO setup: Code GRPO trains on MBPP+ with unit-test rewards, searches rollout group size n ∈{2, 4} and actor learning rate, and evaluates 378 test problems.Checkpoint selection uses the highest validation pass rate on a held-out 100-problem split.
  • Training dynamics: Early TailSFT reward increases reach roughly 2.5× faster in some settings.This is reported as a training-efficiency result rather than a final evaluation metric.
  • Coverage diagnostic: The coverage ratio ρ16 uses a base-reachable set defined by empirical base pass@16, while coverage gain is mean TailSFT-minus-Standard pass@16 on that set.The plotted quantities are defined over problems that are neither effectively unsolved nor saturated by the base model.
  • Training dynamics: TailSFT reward begins below Standard but recovers Standard’s starting reward within a small fraction of training and rises at least as fast thereafter.The comparison is reported across the four main-table settings.
Loading 2608.25756v1…