PaperlayerView exact revision on arXiv

Source-linked AI summary

Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, Hao Peng

arXiv:2602.01058v2cs.LGcs.AIcs.CL

TL;DR

Offline SFT performance is an unreliable proxy for the quality of an initialization after online RL because SFT data follow a behavior policy while RL follows the target policy’s rollouts. PEAR corrects this mismatch by reweighting offline losses with importance weights, and it consistently improves post-RL reasoning performance across games, math benchmarks, and model families. The method is most applicable when behavior-policy token probabilities can be obtained or adequately approximated.

  • Problem

    Stronger offline SFT checkpoints can underperform weaker ones after identical RL, showing that offline accuracy is an unreliable proxy for post-RL performance.

  • Method

    PEAR reweights each offline loss using target-to-behavior likelihood ratios, with token-wise, block-wise, and sequence-level variants that correct the offline-to-online distribution mismatch.

  • Results

    PEAR consistently improves post-RL performance over strong baselines, achieving up to +30 pp Pass@8 on AIME-25 after online RL.

  • Takeaways & Limitations

    Offline training should prepare the policy for the RL that follows by prioritizing continuations reproducible under the target policy.

  • Takeaways & Limitations

    PEAR requires behavior-policy token log-probabilities, which may be impractical for curated data or data produced by a closed-source teacher.

Abstract

from arXiv · show

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We attribute this to a mismatch typical in current SFT-RL pipelines: the distribution that generates the offline SFT data can differ substantially from the policy optimized during online RL, which learns from its own rollouts. We propose PEAR (Policy Evaluation-inspired Algorithm for Offline Learning Loss Re-weighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen 2.5 and 3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass at 8 gains up to a 14.6 percent on AIME2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

1. Introduction

Reasoning-model post-training commonly combines offline SFT with online RL, but offline gains alone are an unreliable guide to final post-RL performance. PEAR addresses the behavior–target distribution mismatch by reweighting offline losses and consistently improves post-RL results across reasoning tasks and models.

  • Motivation: Offline objectives optimized for SFT can produce gains that shrink, disappear, or reverse after identical RL training.This makes offline performance a misleading proxy for the quality of an RL initialization.
  • Motivation: Offline SFT trains on behavior-policy data, whereas online RL learns from rollouts generated by the evolving target policy.The resulting distribution mismatch must be corrected for an effective offline-to-online transition.
  • Method: PEAR reweights offline losses with target-to-behavior likelihood ratios so training better reflects continuations that online RL will revisit.The method includes sequence-level, token-wise suffix-ratio, block-wise, and negative-data variants.
  • Results: +25 pp absolute Pass@1 on SynLogic logic games was achieved by PEAR over canonical SFT, from 13.1% to 38.3%.This comparison used Qwen3-1.7B-Base on the same data and an SFT–RL pipeline varying only the SFT objective.
  • Results: +30 pp Pass@8 on AIME-25 was achieved by PEAR over canonical SFT, from 5% to 35%.This comparison used DeepSeek-Distill-Qwen-1.5B.

2. Offline Performance May Not Entail Online

Offline performance does not reliably predict post-RL performance because uniform SFT supervision follows behavior-policy data while RL follows target-policy rollouts. PEAR uses off-policy likelihood ratios to align offline loss weighting with continuations compatible with the policy optimized online.

  • Offline ≠ Online: Offline objectives that outperform standard SFT can still lose their advantage or become worst after identical RL training.For Qwen3-1.7B-Base, TopLogP had the best offline scores but the worst post-RL performance among the compared objectives.
  • Why Uniform Loss Is Misaligned: Offline SFT uses behavior-policy prefixes, whereas online RL samples trajectories from the evolving target policy, creating an occupancy mismatch.This mismatch can harm the subsequent online phase.
  • Why Uniform Loss Is Misaligned: Long-form reasoning amplifies early distribution differences, allowing offline data to over-represent continuations that are dead ends under the target policy.Uniformly training on such tokens can reinforce transitions that RL rarely revisits.
  • Off-Policy Evaluation: Off-policy evaluation corrects behavior–target shifts with likelihood ratios over entire trajectories or continuation suffixes.These estimators motivate sequence- and continuation-level compatibility weights for offline training.
  • Off-Policy Evaluation: PEAR down-weights logged continuations unlikely under the target policy and up-weights continuations that remain plausible.Figure 2 illustrates this by reducing the weight of an over-represented A→B→C continuation after token A.

3. Method

PEAR reweights the per-token offline training loss using importance weights to make SFT a stronger initialization for subsequent online RL. It supports sequence-, token-, and block-level weighting, with block weighting trading granularity for stability.

  • Core objective: PEAR modifies only the weighting of each token’s offline loss while retaining the underlying SFT/NLL or KD/forward-KL objective.The weights are computed from offline-data likelihood information and applied as fixed coefficients.
  • Core objective: PEAR computes importance weights from token likelihood ratios between the target policy πθ and behavior policy πβ, then applies them to the offline loss.The procedure computes token log-likelihood ratios, aggregates them into weights, stabilizes them, and weights each token’s loss.
  • Weighting variants: Sequence-level weighting applies one global importance score uniformly to every token in a trajectory.The score represents the sequence’s relative likelihood under πθ versus πβ.
  • Weighting variants: Token-level weighting uses a suffix importance ratio to assess the plausibility of each continuation under the model after the current token.A discount factor γ controls variance over long horizons.
  • Weighting variants: Block-level weighting assigns the same discounted continuation weight to tokens within each contiguous block.It reduces the effective length of multiplicative importance weights, trading granularity for improved stability; B = 1 recovers token-level PEAR.
  • Negative examples: For verified failures, PEAR can add a trajectory-weighted repulsive term that uses gradient ascent to push the model away from negative responses.The negative examples are represented as D− and receive sequence-level weights.

4. Experiments

Controlled experiments evaluate PEAR across verifiable reasoning games and mathematical reasoning, comparing SFT objectives under the same downstream RL procedure. PEAR variants consistently improve post-RL performance, including when paired with knowledge distillation and when behavior policies are approximated.

  • Experimental setup: PEAR and its variants consistently improve post-RL performance over strong baselines across reasoning games and mathematical benchmarks.The study uses controlled SFT–RL pipelines that vary only the offline objective.
  • Results: PEAR improves post-RL Pass@1 across model sizes under a fixed RL budget, even when its offline performance does not exceed standard SFT.PEAR is designed to shape the initialization for online RL rather than maximize offline scores in isolation.
  • Compatibility with distillation: PEAR’s weighting also improves KL-based knowledge distillation with minimal additional overhead by reusing information already computed during distillation.The same suffix-likelihood-ratio weighting is used for the default token-level form.
  • PEAR variants: Sequence-, token-, and block-level PEAR variants all outperform standard SFT, with sequence-level weighting also proving effective despite its simplicity.The variants apply importance weighting at different granularities.
  • Robustness and extensions: PEAR remains effective when offline and online domains differ, when negative trajectories are mixed, and when a proxy or ensemble approximates the behavior policy.A reasonably aligned proxy can approach white-box PEAR performance, while negative-trajectory mixing can provide additional RL gains.
  • Learning dynamics: PEAR updates are more aligned with GRPO updates and produce the smallest online-to-offline parameter drift among the compared initializations.The analysis uses gradient principal angles and average NSS drift.

5. Related Works

Related work studies SFT and RL dynamics, offline policy optimization, parameter drift, and modifications to SFT objectives. PEAR differs by targeting the offline-to-online transition through loss reweighting based on policy compatibility.

  • SFT and RL dynamics: Prior work finds that SFT can overfit and degrade out-of-distribution performance, whereas on-policy RL more often improves generalization across distribution shifts.Some studies also report that RL can partially undo SFT-induced drift.
  • SFT and RL dynamics: Recent analyses examine RL post-training through on-policy sampling bias and parameter-space dynamics, including spectral structure and patterns inherited from pretraining.NSS measures relative drift of a singular-value spectrum after training.
  • Offline RL for language models: Offline RL work treats language-model responses as logged decision trajectories or adds offline mechanisms to online RL, whereas PEAR focuses on bridging the conventional SFT-plus-RL pipeline.These approaches apply policy-optimization techniques to logged responses or semi-offline data.
  • Modifications to SFT: Other SFT modifications reduce overfitting or capability loss through probability-based objectives, entropy regularization, reweighting, gating, or importance-weighted stabilization.Examples include iw-SFT, DFT, AFT, Proximal-SFT, and OPC-SFT.

6. Conclusion

Post-RL performance can reverse the ranking suggested by SFT alone because offline SFT and online RL optimize under different policy distributions. PEAR addresses this mismatch by reweighting offline continuations toward those compatible with the current policy.

  • Stronger offline performance is an unreliable proxy for post-RL performance, with identical RL producing substantial rank reversals.
  • Offline SFT imitates logged continuations, whereas online RL updates on trajectories sampled from the evolving policy.
  • PEAR down-weights logged continuations implausible under the current policy and up-weights continuations that remain plausible.
  • PEAR consistently improves post-RL accuracy across verifiable reasoning games and math benchmarks, with up to 30 percentage points Pass@8 gain on AIME-2025.

A. Black-Box PEAR: Proxy and Ensemble Behavior Policies

PEAR remains effective when the behavior policy is approximated rather than accessed exactly. Proxy quality matters: stronger token-level agreement with the data generator corresponds to better post-RL performance, while clipping changes produce only mild variation.

  • PEAR assumes exact behavior-policy probabilities in its main form, but remains effective with approximate proxies for curated, scraped, or closed-source data.
  • Single-model proxies and proxy ensembles provide practical ways to estimate the behavior policy without exact access.
  • Every proxy variant substantially beats SFT, while the strongest Qwen3-32B proxy matches white-box PEAR on SynLogic.
  • Higher Pearson correlation between proxy and true behavior-model token probabilities corresponds to better post-RL performance.
  • Variation across clipping ranges is mild, with pre-RL Pass@32 averaging 47.5% and Pass@64 averaging 52.19%.

C. Block-Size Ablation

Intermediate block sizes provide the best trade-off between fine-grained importance signals and multiplicative variance. PEAR’s benefits also extend beyond reasoning tasks and complement methods operating at the RL stage.

  • C. Block-Size Ablation: Intermediate block sizes B ∈{4, 8} balance fine-grained signal against multiplicative variance and outperform both token- and sequence-level extremes.
  • C. Block-Size Ablation: Token-level weighting yields lower Pass@1, while sequence-level weighting is weakest on Pass@8 because it loses within-sequence variation.
  • PEAR improves instruction-following performance by +20.3 pp on IF-Eval and +12.9 pp on IF-Bench.
  • Generalization to RLHF with learned reward models is left as a future step.
  • PEAR and CHORD target different pipeline stages: PEAR replaces SFT before RL, whereas CHORD mixes supervised and policy-gradient updates during RL.

F. Does PEAR Suppress Useful Low-Probability Trajectories?

PEAR’s reweighting does not appear to suppress useful low-probability content: reasoning patterns, required formats, and cross-domain transfer are preserved. Its mismatch correction also transfers from GRPO to DAPO, while diagnostic metrics characterize model and gradient drift.

  • Reasoning patterns: PEAR preserves comparable usage of standard reasoning markers and does not collapse reasoning patterns.
  • Answer formats: PEAR-initialized models reliably produce required answer formats such as \boxed{} without observable degradation in format compliance.
  • Cross-domain transfer: PEAR gains transfer across task distributions: SynLogic offline training followed by Enigmata RL still outperforms SFT initialization.
  • RLVR algorithm transfer: PEAR +DAPO outperforms SFT+DAPO by +18 Pass@1 and +8.75 Pass@8, indicating mismatch correction is not tied to GRPO.
  • Diagnostics: Forward KL measures divergence between the base and offline-trained model distributions using a 269-pair SynLogic calibration set.
  • Diagnostics: Normalized Spectrum Shift measures parameter-space drift through normalized distances between singular-value spaces.
  • Diagnostics: Gradient-rotation analysis compares offline and online update directions using SVD-derived subspace angles and macro-averages them across linear modules.

H.5. Results

The results examine how offline updates and objective reweighting relate to online performance. The section presents importance-weighted and difficulty-aware formulations, alongside diagnostics and hyperparameter settings.

  • Results: IS-SFT produces comparable forward KL divergence and much lower update sparsity than SFT methods.It also causes greater offline-stage updates and spectrum drift against the base model, followed by smaller online-stage updates.
  • Method: PEAR's suffix-ratio formulation reweights logged continuations so offline expectations reflect continuations more likely under the target policy than the behavior policy.The change-of-measure interpretation requires behavior-policy support wherever the target policy has positive probability.
  • Method: PEAR estimates token-level credit by combining terminal feedback with suffix importance weights under the target policy.The return estimate propagates feedback backward and discounts continuations unlikely under on-policy rollouts.
  • Method: TALR reweights token-level negative log-likelihood using an exponential function of token difficulty and computes the batch loss as a token mean.The method uses stop-gradient weights and a fixed weight floor of 0.01 in all experiments.
  • Method: The objective family includes NLL, −p, hard-thresholded NLL, and higher-power prior-leaning variants.The family is parameterized as fα(p) = 1−p^α, with α →0 recovering NLL.

L. Discussion: Should Offline Training Match Online RL Characteristics In The Two-Stage Process?

Conservative update diagnostics do not necessarily identify SFT checkpoints that prepare models well for downstream RL. The discussion argues that post-RL performance depends on moving toward behaviors favored by on-policy returns, not merely minimizing drift.

  • Discussion: Smaller KL, sparser updates, and smaller representation rotation primarily measure conservatism rather than accurate mismatch correction.These proxies can make an update look stable without showing that it matches downstream RL needs.
  • Discussion: KL penalties reduce drift without improving performance over vanilla SFT, whereas PEAR performs better despite more aggressive KL drift and denser updates.The comparison indicates that update magnitude alone is not the relevant downstream objective.
  • Discussion: The relevant direction for downstream RL is toward behaviors that improve expected return under on-policy rollouts, even when this requires nontrivial deviation from the base model.A stable-looking checkpoint may still lack the inductive biases, coverage, or credit-assignment structure needed for efficient RL.

M. Table For Offline vs Online Metrics (Pass@1 and Pass@8)

Offline and online pass-rate rankings are not consistently aligned across learning objectives and model settings. The section compares these metrics for Qwen3-1.7B-Base and Qwen3-4B-Base on SynLogic.

  • Offline versus online metrics: Offline and online performances do not have a consistent ranking across learning objectives.Figure 17 shows intersecting line segments for the evaluated Qwen3-1.7B-Base and Qwen3-4B-Base settings.
  • Qwen3-1.7B-Base: Table 12 reports offline and online pass rates for Qwen3-1.7B-Base with different learning objectives on SynLogic.The table is the direct comparison for the 1.7B model.
  • Qwen3-4B-Base: Table 13 reports offline and online pass rates for Qwen3-4B-Base with different learning objectives on SynLogic.The table extends the offline-versus-online comparison to the 4B model.
Loading 2602.01058v2…