PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, Xu Yang

arXiv:2508.05629v3cs.LG

TL;DR

SFT often generalizes less effectively than RL, and the paper attributes this gap to an ill-posed implicit reward in standard SFT gradients. It introduces DFT, which reweights the SFT loss by token probability to stabilize updates, and reports improved generalization across challenging tasks, alongside a factual-learning limitation.

  • Problem

    Standard SFT can have limited generalization because its sparse, inverse-probability reward produces disproportionately large gradients and training instability.

  • Method

    DFT dynamically reweights the SFT loss using token probability, canceling inverse-probability distortion while remaining implementable as a simple reweighted SFT loss.

  • Results

    DFT consistently improves over standard SFT across models and challenging mathematical reasoning tasks, with gains extending to code generation and multimodal reasoning.

  • Takeaways & Limitations

    DFT offers a simple, resource-efficient SFT alternative that improves gradient stability and generalization and can also perform competitively in offline RL settings.

  • Takeaways & Limitations

    DFT can hinder factual learning when the model lacks sufficient prior knowledge, whereas SFT remains more reliable for absorbing new factual information.

Abstract

from arXiv · show

In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.

1 INTRODUCTION

SFT is efficient and widely used but often generalizes less effectively than RL, motivating methods that improve SFT itself when only positive demonstrations are available. The paper analyzes SFT’s implicit reward structure, proposes DFT to stabilize token-level updates, and reports broader gains across tasks and models.

  • Motivation: SFT efficiently imitates expert demonstrations but often exhibits limited generalization compared with RL.SFT remains especially important when datasets contain positive demonstrations without negative samples or a reward model.
  • Analysis: SFT gradients can encode a sparse, inverse-probability reward that becomes excessively large for low-probability expert actions.The resulting reward structure is associated with unstable optimization.
  • Method: DFT rescales the SFT objective at each token by its probability, producing more stable and uniformly weighted updates.This is presented as a one-line modification that makes the gradient behave more like an RL-style update.
  • Results: DFT improves performance and generalization across models, scales, data sizes, mathematical reasoning, code generation, and multimodal reasoning.On Qwen-2.5-Math fine-tuned with NuminaMath-CoT, its gains are several times larger than standard SFT, which can degrade on difficult benchmarks.
  • Results: DFT outperforms offline RL methods and is competitive with or superior to online RL methods on specified math tasks.Unlike those RL methods, DFT requires neither a reference model nor large batch sizes.
  • Analysis: DFT selectively raises or lowers token probabilities, increasing the proportion of less strongly fitted tokens and suggesting improved regularization.Traditional SFT instead pushes probabilities uniformly toward the training set.
  • Contributions: The paper combines a theoretical account of SFT as policy-gradient learning with a practical one-line modification that improves SFT generalization.Its contributions are explicitly described as both theoretical and practical.

2 RELATED WORK

Related work combines SFT and RL, directly optimizes preferences, or analyzes their shared formalism. This paper’s objective design also contrasts with Focal Loss by downweighting poorly classified samples to encourage generalization.

  • Hybrid and Preference Methods: Hybrid methods commonly use SFT pretraining followed by RL refinement, while newer approaches interleave SFT and RL updates.DPO instead optimizes policies directly on preference data without reward modeling.
  • Hybrid and Preference Methods: NFT models incorrect generations with an implicit negative policy to enable self-improvement without explicit feedback.These approaches are discussed alongside preference-optimization methods that combine imitation and reinforcement signals.
  • Unified Formalisms: Theoretical work reframes SFT and RL through reward-weighted SFT, implicit rewards, lower bounds, and the balance of positive and negative feedback.These analyses motivate adjustments such as smaller learning rates or importance weighting.
  • Objective Design: DFT uses −p log(p), downweighting poorly classified samples in contrast to Focal Loss’s −(1 − p)^γ log(p).The paper presents this inversion as a response to overfitting and memorization in current LLM training.
  • Objective Design: Focal Loss emphasizes hard cases by downweighting well-classified samples, whereas DFT deliberately downweights poorly classified samples to encourage generalization.The comparison highlights opposite weighting philosophies.

3 METHOD

The method interprets SFT gradients through an RL lens, identifying sparse, inverse-probability rewards as a source of instability and limited generalization. DFT corrects this distortion by probability-based reweighting, producing uniform expert-trajectory rewards and more stable updates without additional sampling or reward models.

  • SFT and RL Gradient Expression: SFT can be rewritten as an on-policy policy gradient with a sparse exact-match reward and an importance weight of 1/πθ.The importance weight compares the expert distribution with the model distribution.
  • SFT and RL Gradient Expression: Low model probability for expert responses makes the inverse-probability weight excessively large, causing disproportionate gradients and training instability.The sparse exact-match reward can also overfit rare exact-match samples.
  • Proposed Method: DFT multiplies the reward by the corrective policy probability, neutralizing inverse-probability weighting and yielding a uniformly weighted expert-trajectory reward.The corrected loss is implemented as a simple reweighted SFT objective, with stop-gradient applied to the scaling term.
  • Proposed Method: Token-level importance sampling is used in the final DFT loss because trajectory-level weights can induce numerical instability.This token-level treatment follows a practice adopted in PPO.
  • Proposed Method: The corrected reward avoids concentration on low-probability reference tokens, supporting more stable updates and improved generalization without extra sampling or reward models.The resulting uniform reward is compared with verification-based reward approaches that assign uniform reward to correct samples.

4 EXPERIMENTS

Across mathematical reasoning, offline RL, code generation, and multimodal benchmarks, DFT generally improves performance, convergence, and generalization over SFT, while showing a factual-knowledge limitation.

  • Mathematical reasoning: +15.66 points: DFT’s average gain over the Qwen2.5-Math-1.5B base model exceeds SFT’s +2.09-point gain by over 5.9×.Across other model families and sizes, DFT also produces larger average gains than SFT.
  • Mathematical reasoning: DFT improves difficult mathematical reasoning benchmarks where SFT can degrade performance, including Olympiad Bench, AIME24, and AMC23.For Qwen2.5-Math-1.5B on Olympiad Bench, accuracy rises from 15.88 base and 12.63 with SFT to 27.08 with DFT.
  • Mathematical reasoning: DFT converges faster, surpasses SFT’s best final accuracy within 10–20 steps on some benchmarks, and reaches peak performance within 120 steps on most benchmarks.The results attribute these learning-efficiency gains to dynamic reweighting producing more informative gradient updates.
  • Cross-domain evaluation: DFT also improves code-generation and multimodal reasoning results, including Qwen2.5-Coder-7B MultiPL-E accuracy of 62.3 and MathVerse accuracy of 37.54.On Qwen2.5-Coder-7B, DFT surpasses SFT by +12.8, +11.0, and +4.7 points on HumanEval, HumanEval+, and MultiPL-E respectively.
  • Limitations: DFT can underperform SFT on factual knowledge: Natural Questions accuracy increases from 31.24% to 36.62% with SFT but falls to 30.14% with DFT.The paper relates this limitation to DFT reinforcing existing beliefs when the model lacks sufficient factual knowledge.

5 CONCLUSION

The paper interprets SFT’s limited generalization through an ill-posed implicit reward and introduces DFT, which reweights token losses by token probability. Experiments report improved generalization over SFT and competitive offline-RL performance, while the evaluation scope and task suitability remain limited.

  • 5 CONCLUSION: DFT reweights the SFT loss using token probability to improve gradient stability and generalization.The method is presented as a lightweight change intended to bridge SFT and RL.
  • 5 CONCLUSION: DFT consistently improves over standard SFT across models and challenging mathematical reasoning tasks.The conclusion reports this as an empirical pattern across the evaluated settings.
  • 5 CONCLUSION: DFT also outperforms several online and offline RL baselines in adapted offline-RL scenarios.The conclusion characterizes this as evidence of broader applicability.
  • 5 CONCLUSION: The evaluation covers mathematical reasoning and code generation but not broader task categories or larger-scale LLMs.These boundaries are explicitly identified as areas for future exploration.
  • 5 CONCLUSION: Conventional SFT remains more efficient for acquiring factual knowledge, while DFT may underweight hard or underrepresented examples.The paper does not claim that DFT universally outperforms SFT.

ETHICS STATEMENT

The study follows the ICLR Code of Ethics and uses publicly available datasets without human subjects, personally identifiable information, or proprietary data.

  • ETHICS STATEMENT: The work adheres to the ICLR Code of Ethics.The authors report no conflicts of interest, legal compliance issues, or sponsorship-related influences.
  • ETHICS STATEMENT: The study does not involve human subjects, personally identifiable information, or proprietary data.All datasets used are described as publicly available and documented in the appendix.
  • ETHICS STATEMENT: DFT modifies gradient computation without introducing capabilities beyond standard large language model capabilities.The authors report no method-specific potential risks related to bias, fairness, or security.
  • ETHICS STATEMENT: DFT may inherit biases present in the underlying data or model.The authors identify safeguards for these scenarios as a possible direction for future research.

REPRODUCIBILITY STATEMENT

The authors describe reproducibility measures covering public datasets, training configurations, evaluation settings, and formal derivations of theoretical claims.

  • REPRODUCIBILITY STATEMENT: All experimental datasets are publicly available and properly cited in the main text and appendix.The reproducibility statement identifies dataset availability and citation as part of its documentation practice.
  • REPRODUCIBILITY STATEMENT: Training configurations and evaluation settings are described in Section 4 and Appendices A.5–A.6.The listed configurations include model architectures, hyperparameters, and optimizers.
  • REPRODUCIBILITY STATEMENT: The equivalence between SFT and policy gradient is formally derived in Appendix A.2.The statement identifies this derivation as part of the reproducibility documentation.

A.1 USAGE OF LLM

The authors used LLMs only as writing assistants for manuscript clarity, coherence, and presentation; the conceptual and experimental contributions remained original.

  • A.1 USAGE OF LLM: LLMs were used primarily to refine and polish the manuscript.Their stated role was limited to improving clarity, coherence, and presentation.
  • A.1 USAGE OF LLM: The authors state that all conceptual and experimental contributions remain original.

A.2 DETAILED DERIVATION OF EQUATION (5)

The derivation rewrites the expert-demonstration expectation by inserting the model distribution as an importance weight, yielding an expectation under the model policy. This expresses the SFT gradient as an on-policy policy gradient with importance sampling.

  • The expert-demonstration expectation is expanded into a summation over all possible outputs y.
  • Inserting πθ(y | x) introduces importance weights that compare the expert distribution with the model distribution.
  • The summation is rewritten as an expectation under the policy distribution y ∼ πθ(· | x).
  • The resulting formulation shows that the SFT gradient is an on-policy policy gradient with importance sampling.

A.3 DISCUSSIONS AND INSIGHTS

DFT changes cross-entropy’s probability-dependent gradient scaling by directly maximizing target-token probability, producing uniformly scaled updates. This avoids excessively large updates for unlikely targets and is associated with more stable optimization and stronger generalization.

  • DFT’s gradient is mathematically equivalent to directly maximizing target-token probability rather than its log-probability.
  • Cross-entropy scales target updates by 1/πθ, whereas DFT uses a uniform factor of 1 despite sharing the same gradient direction.
  • By avoiding oversized gradients on unlikely expert tokens, DFT provides more conservative and stable updates.
  • From the RL perspective, DFT assigns uniformly 1 reward across expert trajectories, treating correct references equally.
  • DFT trades aggressive fitting of rare tokens for improved stability and calibration while maintaining alignment with the pre-training distribution.

A.4 COMPARISION WITH CONCURRENT WORK IW-SFT

Across concurrent-work comparisons, additional datasets, parameter-efficient tuning, and hyperparameter tests, DFT generally outperforms SFT or iw-SFT and remains effective under varied training conditions. Its gains extend to high-quality data and LoRA settings, while batch size is relatively non-dominant.

  • A.4 COMPARISION WITH CONCURRENT WORK IW-SFT: DFT achieves higher average accuracy than iw-SFT for LLaMA-3.2-3B (+2.39), LLaMA-3.1-8B (+4.15), DeepSeekMath-7B (+3.34), and Qwen2.5-Math-1.5B (+1.30).
  • A.4 COMPARISION WITH CONCURRENT WORK IW-SFT: iw-SFT’s advantage on Qwen2.5-Math-7B (+2.45) is inconsistent across datasets, and it can underperform standard SFT on several benchmarks.
  • A.4 COMPARISION WITH CONCURRENT WORK IW-SFT: In offline RL, DFT exceeds iw-SFT’s overall average by +3.57 points, while iw-SFT improves over SFT by only +1.58.
  • A.4 COMPARISION WITH CONCURRENT WORK IW-SFT: On OpenR1-Math-220k, SFT gains +13.24 average-accuracy points over the base model, while DFT adds +9.03 for a total improvement of +22.27.
  • A.4 COMPARISION WITH CONCURRENT WORK IW-SFT: Under LoRA, DFT raises Qwen2.5-Math-1.5B average accuracy from 15.92 base and 16.87 SFT to 32.90.
  • A.7 TRAINING HYPER-PARAMETERS ABLATION: DFT consistently outperforms SFT across the tested learning rates, while intermediate rates 1e-4 and 5e-5 perform best for both methods.
  • A.7 TRAINING HYPER-PARAMETERS ABLATION: Both methods remain relatively stable when batch size ranges from 32 to 256, with no consistent accuracy trend favoring larger or smaller batches.
Loading 2508.05629v3…