Source-linked AI summary
Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
Bo Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng, Yunhua Zhou, Xipeng Qiu
TL;DR
The theoretical relationship between SFT and preference optimization in LLM post-training has been underexplored. The paper uses implicit reward learning to connect the methods and reports gains from reducing SFT learning rates and using alternative objectives.
Problem
Although preference learning commonly follows SFT, their relationship has been understood mainly through empirical observations, leaving limited theoretical grounding for how the stages relate.
Source: §1 · pp. 1–2Method
The paper derives a unified implicit-reward framework in which SFT is a special case, then extends the logits-to-Q-function relationship to SFT and derives alternative f-divergence objectives.
Source: §1 · p. 2, §3.2 · p. 4Results
Up to 25% relative improvement and 6% absolute win-rate gain followed reduced SFT learning rates, while alternative objectives yielded up to 4% absolute win-rate gains after DPO.
Source: §1 · p. 3Takeaways & Limitations
The authors conclude that SFT stabilizes random implicit rewards, linking theoretical understanding to practical strategies for post-training.
Source: §6 · p. 10The effectiveness of the explored KL-divergence approaches remains unknown: numerical instability persisted, and the authors did not design specialized operators for them.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.
1 Introduction
The introduction frames SFT and preference optimization as post-training approaches whose theoretical relationship has been underexplored. It presents implicit reward learning as a unifying framework, with learning-rate reduction, KL-preserving alternatives, and an SFT logits-to-Q-function connection as key insights.
1 Introduction
- Post-training learns from expert demonstrations or environmental signals, and preference learning commonly follows SFT with methods such as DPO; the theoretical relationship between these stages remains underexplored.The introduction says this relationship was understood predominantly through empirical observations, with SFT often treated as a warm-up step.Source: §1 · pp. 1–2
- The authors show that SFT learns an implicit reward and, like DPO, operates within the same optimal policy-reward subspace.They present SFT as a special case of implicit reward learning.Source: §1 · pp. 2–3
- In conventional SFT, the KL term is constant with respect to the policy and contributes no gradient, so it does not constrain model updates.The authors propose reducing the SFT learning rate to mitigate the absent KL constraint.Source: §1 · pp. 2–3
- Alternative SFT objectives derived from other f-divergences preserve the KL term during optimization and improve post-DPO performance.The reported absolute win-rate improvement is up to 4%.Source: §1 · pp. 2–3
- Reducing the SFT learning rate yielded up to 25% relative improvement and a 6% absolute win-rate increase; alternative objectives yielded up to 4% absolute win-rate gains after DPO.The reported 6% increase is from 15.6% to 21.5%.Source: §1 · p. 3
- The paper extends the logits-to-Q-function relationship from preference learning to SFT, with indirect experimental evidence supporting the extension.The formulation is described as enabling efficient estimation of state values under the model’s implicit reward.Source: §1 · pp. 2–3
2 Related Work
The related work situates the paper among inverse reinforcement learning, implicit-reward methods for LLMs, and broader analyses of post-training theory. It distinguishes the paper’s focus on relationships between post-training objectives, especially in sequential SFT and preference learning.
2.1 Inverse Reinforcement Learning
- Inverse reinforcement learning assumes that an expert is optimal under some reward function; this work uses closed-form solutions rather than inner-loop optimization.It also uses a more general f-divergence formulation and does not introduce negative samples [20].Source: §2.1 · p. 3
2.2 Implicit Reward in LLMs
2.3 Post-Training Theory Analysis for LLMs
- Prior analyses connect implicit rewards with reward models and the generation-verification gap [28], or examine learning dynamics across training phases [14]; this work focuses on relationships between post-training objectives.Source: §2.3 · p. 3
3 Unified View between SFT and DPO
The section frames SFT and preference learning within a shared implicit-reward view. It develops distribution matching as a basis for that connection, then relates SFT logits to expected returns.
3.1 Distribution Matching in Post-training
- Post-training distribution matching minimizes an f-divergence between expert and policy state-action distributions, using KL regularization against the base model instead of entropy.The stated motivation is to preserve the base model’s intrinsic properties and natural-language priors.Source: §3.1 · p. 4
3.2 Imitation Learning as Implicit Reward Discovery
- The distribution-matching objective is equivalent to first learning an optimal policy under an arbitrary reward, then optimizing a function of that reward.The latter objective has a closed-form solution, which the authors leverage in their approach.Source: §3.2 · p. 4
- Common SFT is a special case of implicit reward discovery and searches the same optimal policy-reward subspace as DPO.The derivation selects total variation as the f-divergence and treats the process as modeling reward implicit in expert demonstrations.Source: §3.2 · p. 5
- In conventional SFT, the KL term is constant during gradient updates and does not constrain them; reducing the learning rate mitigates this limitation.Alternative f-divergence objectives can preserve the KL term, though many involve logarithmic or exponential operations that may cause numerical instability.Source: §3.2 · p. 5
3.3 Model Maintains Intrinsic Expected Return Estimation During SFT
- During SFT, language-model logits correspond to Q-values for the learned implicit reward, and the state-only offset does not affect action rankings.The Q-value comprises the implicit reward plus the discounted expected next-state value.Source: §3.3 · pp. 5–6
- Under the value-dominance assumption, log-sum-exp of model logits can serve as a value function instead of Monte Carlo sampling for other divergence formats.The assumption states that, for most state pairs, differences in C(s) are smaller than differences in value.Source: §3.3 · p. 6
4 Empirical Study
The empirical study examines instruction-following post-training, testing how SFT learning rates and imitation objectives affect post-DPO results, and evaluating logits and implicit rewards as measures of state quality and reward alignment.
4.1 Basic Experiment Setting
- Experiments use Llama3-8B and Mistral-7B, UltraChat-200K for SFT, and Ultra-feedback for DPO, evaluating with AlpacaEval2, Arena Hard, and MT-bench.Some experiments use AlpacaEval2 as the representative benchmark to limit API costs.Source: §4.1 · p. 6
4.2 Small Learning Rate SFT Leads to Better post-DPO Results
- Because conventional SFT's KL term contributes no gradient, the authors reduce SFT learning rates from 2 × 10^-5 to 5 × 10^-6 for Llama3-Base and 1 × 10^-6 for Mistral.They keep the following DPO and SimPO hyperparameters the same as in the 2 × 10^-5 configuration.Source: §4.2 · pp. 6–7
- The reported SimPO results show 20% relative (5% absolute) improvement for Llama3-8B and 25% relative (6% absolute) for Mistral after DPO with smaller SFT learning rates.The comparison is reported in Table 1.Source: §4.2 · p. 7, Table 1 · p. 7
4.3 Other Forms of Imitation Loss Behave Better in Sequential Training
- Pearson χ2's squared probability-difference term acts as a KL constraint, while Squared Hellinger's probability-difference coefficient modulates the gradient update size.The paper selects these objectives in part to avoid numerical stability issues associated with logarithmic and exponential calculations.Source: §4.3 · p. 7
- Pearson χ2 and Squared Hellinger yield weaker SFT but better post-DPO results than total-variance SFT for both Mistral and Llama.Table 3 reports the downstream results for the different training targets and their post-DPO checkpoints.Source: §4.3 · p. 7, Table 3 · p. 8
4.4 Value and Reward in SFT
- LLM value rankings are positively correlated across all tested settings, approaching 1 across four dataset shards and remaining positive across changes in training data or model prior.The rankings are calculated over step-wise reasoning states from MATH-500 trajectories.Source: §4.4 · p. 8, Figure 2 · p. 9
- Lower DPO loss, indicating closer implicit-reward alignment with the environment, positively correlates with model performance on AlpacaEval 2.Figure 2's middle panel pairs DPO loss with the corresponding AlpacaEval2 result.Source: §4.4 · pp. 8–9, Figure 2 · p. 9
4.5 Reward Stabilization through SFT
- V(s0) rises rapidly and converges during SFT; post-DPO performance follows a consistent trend, and SFT reaches a reasonable reward range by 150 steps.The paper describes subsequent SFT steps as focusing on more refined modeling.Source: §4.5 · p. 9, Figure 3 · p. 9
5 Broader Impacts, Limitations and Future Work
The paper discusses philosophical questions raised by implicit rewards, reports limits in experimenting with additional divergence functions, and proposes multi-objective SFT–DPO learning as future work.
Broader Impacts: Philosophical Dimension
- SFT’s implicit rewards raise philosophical questions about whether LLMs have preset environmental awareness and exhibit consciousness-like properties.The paper frames these as open inquiries, not conclusions about LLM consciousness.Source: §5 · p. 9
Limitations
- KL- and JS-based alternatives were not experimentally explored because logarithmic and exponential calculations caused numerical instability and extremely large losses.Small-value clipping prevented NaNs, but losses could still reach 6e7; specialized operators were not designed, leaving effectiveness unknown.Source: §5 · p. 9
Future Work: SFT and DPO multi-object learning
- Because both SFT and DPO model implicit rewards, the authors propose exploring multi-objective learning instead of sequential training.Appendix D describes failed implementation attempts offered as input to future work.Source: §5 · p. 9
6 Conclusion
The conclusion presents implicit reward learning as a unifying account of SFT and preference learning, then summarizes learning-rate and alternative-divergence approaches, and extends the logits-to-Q-function relationship to SFT.
6 Conclusion
- Conventional SFT is a special case of implicit reward learning using total variation divergence, but lacks a KL term.The authors present implicit reward learning as a unifying view connecting SFT and preference learning in LLM post-training.Source: §6 · p. 10
- Reducing learning rates significantly improves model performance, while alternative f-divergence objectives that preserve the KL term show additional gains.Source: §6 · p. 10
- The authors extend DPO’s logits-to-Q-function mapping to SFT and report mathematical derivations and experimental validation, including SFT’s role in stabilizing random implicit rewards.Source: §6 · p. 10
A.1 Inverse Reinforcement Learning and f-Divergence
The section reviews inverse reinforcement learning and f-divergence work, then distinguishes this paper’s post-training formulation and focus on sequential SFT and DPO.
A.1 Inverse Reinforcement Learning and f-Divergence
- Inverse reinforcement learning recovers a reward under which expert behavior is optimal; cited advances include maximum entropy, GAIL, and f-divergence formulations.The review cites for IRL, for maximum entropy, [43] for GAIL, and [20] and [44] for f-divergence approaches.Source: §A.1 · p. 15
- Prior approaches avoid explicit reward learning through Q-functions, reverse-KL imitation with entropy regularization, or f-divergence formulations for out-of-distribution generation.These approaches are cited as [15], [16],, and [21], respectively.Source: §A.1 · p. 15
- This work modifies maximum-entropy distribution matching for post-training, enabling closed-form solutions without inner-loop optimization, and uses general f-divergence without negative samples.It contrasts its formulation with [15] [16] on Bellman-equation use and on divergence generality and negative samples.Source: §A.1 · p. 15
B Theoretical Proofs
The appendix states the maximum-entropy fixed-point basis for its proofs, derives an equivalent reward-based view of f-divergence distribution matching, and identifies SFT as the total-variation special case.
B Theoretical Proofs
- The proof section invokes maximum-entropy RL fixed-point equations for the optimal policy and its corresponding value function.Source: B Theoretical Proofs · p. 15
B.1 Non-adversarial Imitation: Finding Reward
- Theorem 3 equates minimizing f-divergence between expert and policy state-action distributions with learning an optimal policy under an arbitrary reward, then optimizing a function of that reward.The proof rewrites the objective using a convex conjugate and substitutes −r; it exchanges min-max order at a saddle point and applies a closed-form maximization.Source: §B.1 · pp. 15–16
- The proof’s maximization over the latter part of the expression has a closed-form solution.Source: §B.1 · p. 16
B.2 SFT is a Special Case
- With total variation distance, the conjugate is f*(t) = t and the training objective reduces to the MLE term.Throughout training, the reward maintains equation (4) and the policy remains optimal for the latter parts of the objective.Source: §B.2 · p. 16
B.3 SFT Maintains Intrinsic Expected Return Estimation
The section derives that SFT logits estimate the Q-function of the learned implicit reward, up to a state-dependent offset. It also introduces a table comparing training targets derived from different f-divergences.
B.3 SFT Maintains Intrinsic Expected Return Estimation
- During SFT, logits correspond to the learned implicit-reward Q-function: the reward plus the discounted expected value of the next state.The theorem gives Q(s, a) = r̂(s, a) + γE[V̂_r(s_{t+1})].Source: §B.3 · p. 17
- The logits map linearly to Q-values with an offset C(s) that depends only on the state.The authors hypothesize that C(s) remains numerically similar across states and say the experimental section provides supporting evidence.Source: §B.3 · p. 17
C Different f-Divergence Leads to Different Loss Format
- Table 4 lists conjugate functions and derived training targets for different f-divergences, using ΔV for Vπ(s0) − Vπ(st).Source: Table 4 · p. 18
D Multi-Object
The section explores multi-objective training that combines SFT with a constraint on DPO accuracy. In the reported setting, later SFT stages show a performance drop after DPO, while the authors leave other domains unexplored.
D Multi-Object
- The proposed multi-objective formulation minimizes SFT loss while constraining chosen-versus-rejected accuracy to remain within δ of a target.The authors use a Lagrangian multiplier method for this formulation.Source: D Multi-Object · p. 18
- The model is trained directly on UltraFeedback, with Figure 4 presenting training metrics from one run.Source: D Multi-Object · p. 18, Figure 4 · p. 19
- Different tested growth and decay coefficients for λ all led to poor downstream results; the authors attribute this to increased probability mass in negative regions as the SFT coefficient rises.They say this diminishes the squeezing effect and also validate the conflict through interleaved SFT and DPO experiments.Source: D Multi-Object · p. 19
- After each DPO stage, subsequent SFT training has a significant performance drop, indicating conflicting objectives in this experimental setting.Table 5 reports Win Rate and LC Win Rate percentages after each stage; the supplied passages do not give specific values.Source: D Multi-Object · p. 19, Table 5 · p. 19
- The authors leave other domains untested and hypothesize better results where expert trajectories align more closely with SFT’s assumption that training data represents optimal behavior.Source: D Multi-Object · p. 19