PaperlayerView exact revision on arXiv

Source-linked AI summary

UFT: Unifying Supervised and Reinforcement Fine-Tuning

Mingyang Liu, Gabriele Farina, Asuman Ozdaglar

arXiv:2505.16984v2cs.LGcs.CL

TL;DR

Existing post-training methods trade off supervised learning and reinforcement-based exploration, motivating a unified approach. UFT combines supervision with reward optimization through hints, generally outperforming SFT and RFT while reducing reasoning-length sample complexity from exponential to polynomial. The paper's theoretical setup assumes a deterministic optimal policy for its analysis.

  • Problem

    SFT and RFT provide different strengths for post-training, motivating a method that combines supervised feedback with reinforcement-based exploration.

  • Method

    UFT unifies SFT and RFT by combining reward optimization with log-likelihood maximization on partial-solution hints.

  • Results

    UFT generally outperforms SFT and RFT, and its sample complexity depends polynomially rather than exponentially on reasoning length.

  • Takeaways & Limitations

    Unified training enables models to explore and learn from existing solutions within one fine-tuning paradigm.

  • Takeaways & Limitations

    The theoretical analysis assumes a deterministic optimal policy, with one such policy selected for the model.

Abstract

from arXiv · show

Post-training has demonstrated its importance in enhancing the reasoning capabilities of large language models (LLMs). The primary post-training methods can be categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). SFT is efficient and well-suited for small language models, but it may lead to overfitting and limit the reasoning abilities of larger models. In contrast, RFT generally yields better generalization but depends heavily on the strength of the base model. To address the limitations of SFT and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process. UFT enables the model to effectively explore solutions while incorporating informative supervision signals, bridging the gap between memorizing and thinking underlying existing methods. Notably, UFT outperforms both SFT and RFT in general, regardless of model sizes. Furthermore, we theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for the first time that unified training can exponentially accelerate convergence on long-horizon reasoning tasks.

1 Introduction

LLM fine-tuning parallels human learning through supervised memorization and reinforcement-based exploration, but existing training regimes typically separate these processes and favor different model capacities.

  • SFT memorizes collected reasoning traces by maximizing their log-likelihood, whereas RFT explores reasoning space using verifier feedback on final answers.
  • Prior work usually applies either SFT or RFT throughout fine-tuning, or applies RFT only after SFT completes.
  • When the LLM is weak, SFT typically works better because verifier-based reinforcement learning provides sparse rewards for discovering correct answers.

SFT RFT

UFT unifies supervised feedback and reinforcement learning so models can explore solutions while learning from partial solutions. Across model scales and tasks, it generally outperforms SFT and RFT, while its theory gives polynomial rather than exponential dependence on reasoning length.

  • SFT RFT: SFT uses full solution annotations, RFT uses none, and SFT-RFT discards annotations after its initial supervised stage.UFT instead combines reward optimization with supervised learning on hints, allowing exploration and learning during the same training process.
  • SFT RFT: UFT raises the performance ceiling imposed by pretraining capacity by integrating hints into reinforcement learning as supervision.Unlike R3, which primarily uses hints as exploration aids, UFT also treats them as part of the supervision signal.
  • SFT RFT: UFT generally outperforms competing algorithms across Countdown, MATH, and Knights and Knaves tasks evaluated on Qwen2.5 models.Additional evaluations cover Qwen2.5 and Llama3.2 models across multiple reasoning tasks, with UFT consistently outperforming previous methods.
  • SFT RFT: UFT reduces sample-complexity dependence on reasoning length from exponential under RFT to polynomial.The theoretical analysis identifies this as an exponential improvement over standard RFT.
  • SFT RFT: UFT integrates reward optimization with log-likelihood maximization on partial solutions, smoothly transitioning from supervised to reinforcement learning.The framework is designed to address the trade-off between SFT's memorization and RFT's generalization.

2 Preliminaries

The paper models problem solving as traversal through a finite branching search tree, where complete root-to-leaf paths represent solutions. Policies select child branches, and value functions measure expected terminal reward from each state.

  • Search Tree: A problem-solving process is represented as a search tree of height H and branching factor B, with each root-to-leaf path corresponding to a complete solution.Each non-leaf node has B children representing different next tokens or sentences.
  • Search Tree: The tree contains one root, expands by a factor of B at each level, and assigns terminal rewards R(s) in [0, 1] to leaf nodes.The level sets S_h organize nodes by height, while the total node count follows from repeated branching.
  • Policies and Values: A policy π assigns probabilities over the B child actions at each non-leaf state and induces a trajectory through the tree.Transitions are determined by the selected action's child-state mapping.
  • Policies and Values: The value function V^π(s) is the expected reward obtained by sampling a trajectory from state s according to policy π.The paper writes the root value as the expected reward from following π starting at the root.
  • Optimal Policy: An optimal deterministic policy π* follows a root-to-leaf path with the highest expected reward, denoted V* at the root.The paper selects one deterministic optimal policy among those achieving the maximum root value.

3 Unified Fine-Tuning (UFT)

UFT combines hint-guided exploration with supervised feedback, gradually reducing hints while optimizing reward and hint likelihood. This design addresses sparse rewards and combines RFT’s exploration with SFT’s knowledge acquisition.

  • 3.2 Objective Function Modification: UFT adds a log-likelihood term for hints to reward optimization, enabling simultaneous exploration and learning from informative supervision.The objective also includes a KL-divergence term relative to a reference policy.
  • 3.1 Exploration with Hint: UFT guides trajectory sampling with a partial-solution hint concatenated to the problem description, increasing the frequency of correct-answer exploration.The hint is a slice of the full solution used during sampling.
  • 3.1.1 Hint Length Sampling: Cosine annealing gradually reduces the expected hint length, avoiding abrupt distribution shifts while aligning training with zero-hint evaluation.The hint length l is sampled from Binomial(L, p), with E[l] = p · L.
  • 3.1.1 Hint Length Sampling: UFT’s training curve is smoother and converges faster than staged reinforcement learning, although both methods use zero hints after step 300.This comparison is reported for Qwen2.5-0.5B.
  • 3.1.1 Hint Length Sampling: RFT with cosine hint scheduling outperforms R3 but remains worse than SFT-RFT, and for Llama-3.2-1B it is worse than SFT alone.This motivates adding a log-likelihood term to the objective.
  • 3.2 Objective Function Modification: The objective interpolates between RFT and SFT: with zero hint proportion it becomes RFT, while the fully supervised setting degenerates to SFT.This unification lets the training process transition between reinforcement and supervised learning.
  • 3.2 Objective Function Modification: Adding the log-likelihood term makes UFT match SFT-RFT performance for small models and raises RFT’s ceiling by enabling new knowledge acquisition.The reported comparison is based on Figure 5.

4 Theoretical Justification

The theoretical analysis shows that RFT requires exponentially many samples in reasoning-tree height under sparse rewards, whereas UFT achieves polynomial dependence using hints. This establishes an exponential sample-complexity improvement for long-horizon reasoning.

  • RFT Lower Bound: RFT’s lower-bound sample complexity grows exponentially with tree height H, requiring exploration of at least B^H/K nodes when K correct solutions exist.The lower bound targets a 50% pass@1 success rate.
  • RFT Lower Bound: The lower-bound construction uses deterministic search-tree transitions and problems whose correct solutions cannot be distinguished before sufficient nodes are explored.This differs from traditional reinforcement-learning lower bounds based on stochastic transitions.
  • RFT Lower Bound: When rewards are sparse and K is constant, learning the optimal policy takes iterations exponential in the reasoning-tree height.This formalizes why long reasoning is difficult for RFT.
  • UFT Improvement: UFT reduces the dependence on reasoning length from B^H to H^5 in the stated theorem, an exponential improvement enabled by hints.The result applies to the theorem’s 50% pass@1 setting, whose success threshold changes sample complexity only by a constant factor.

5 Experiments

Across model sizes and reasoning tasks, UFT adapts between memorization and generalization while generally outperforming prior fine-tuning methods. It also improves exploration efficiency and reduces rollout cost relative to RFT.

  • Overall results: UFT matches SFT for small models and matches or exceeds RFT for larger models, adapting to model capacity across tasks.For small models, RFT rarely finds correct answers; for larger models, UFT generalizes as well as RFT while SFT-based methods overfit.
  • Exploration and memorization: UFT finds the correct answer at every timestep for Qwen2.5-0.5B on Logic, whereas RFT rarely explores it.The comparison concerns cumulative average success in exploring the correct answer during training.
  • Exploration and memorization: UFT outperforms R3 by shifting hint lengths toward zero and maximizing hint log-likelihood, helping small models encode solution information.Its performance proximity to SFT-RFT supports memorization when the model lacks enough initial capacity to solve the task.
  • Generalization across models: UFT significantly improves Llama-3.2, with Llama-3.2-1B surpassing Llama-3.2-3B after 250 RFT steps on Countdown.The comparison is based on the behavior of Llama-3.2-1B/3B across Countdown, MATH, and Logic.
  • Knowledge acquisition: UFT introduces new knowledge, while RFT primarily helps models use existing knowledge.This distinction is presented as a contribution of UFT relative to RFT.
  • Computational cost: UFT is faster than RFT because hint-guided rollouts begin from partial solutions rather than from scratch.Starting from hints reduces rollout cost during training.

6 Conclusion and Limitations

The paper concludes that UFT unifies supervised and reinforcement fine-tuning, combining memorization and exploration across model scales. Its evidence is limited to human-annotated solutions, GRPO, and models below the 70B scale.

  • Conclusion: UFT unifies SFT and RFT, combining supervised learning from hints with reinforcement learning in one fine-tuning framework.The paper presents this integration as its central framework and contribution.
  • Conclusion: UFT generally outperforms both SFT and RFT; small models tend to memorize, while large models generalize.The paper also theoretically claims an exponential speed-up over RFT.
  • Limitations: The experiments use only human-annotated solutions and GRPO as the reinforcement learning algorithm.These choices define the principal methodological scope of the reported evidence.
  • Limitations: The experiments do not use state-of-the-art 70B-scale models because of computational constraints, although additional evidence supports benefits at larger settings.The paper identifies broader model-scale evaluation as a future direction.

B.3 Ablation Study

The ablation studies examine hint length, the supervised-term coefficient β, hint-phase length, and hint-length distributions. The reported findings emphasize how β affects learning from hints, especially for smaller models.

  • Hint length: The hint-length ablation tests L = 4/5/6 solution pieces with uniformly sampled hint lengths on Qwen2.5-0.5B MATH(3,4,5).MATH is used because its solutions are relatively longer than Countdown solutions.
  • β ablation: A higher β amplifies the supervised log-likelihood term and helps the model learn more from hints.The main experiments use β = 0.001, the VERL default.
  • β ablation: For larger models, varying β produces minimal performance changes, likely because their pretrained priors are stronger.This observation is reported for Qwen2.5-3B on Countdown.
  • Additional ablations: Additional ablations evaluate hint-phase length and hint-length distribution on Countdown.The supplied table captions identify these as separate studies for Qwen2.5-0.5B.

B.4 Generalization to Larger Models

The larger-model generalization results include Qwen2.5-1.5B and indicate that UFT can remain useful when a task exceeds the model’s reasoning capacity. This is especially evident on Logic.

  • Generalization to larger models: Although Qwen2.5-1.5B has enough capacity for Countdown and MATH, it remains insufficient on Logic, where UFT substantially outperforms RFT.The paper therefore anticipates UFT may help larger models on extremely hard tasks.
  • Generalization to larger models: The broader response comparison covers Qwen2.5-0.5B and Qwen2.5-3B trained by different algorithms.The supplied figure caption identifies the models and comparison scope but does not state a specific outcome.

B.5 Additional Results

Across model sizes, UFT combines strong performance from SFT and RFT: it excels on smaller models and closely matches RFT on larger models. The results average accuracy across Countdown, MATH(3,4,5), and Logic.

  • UFT responses align better with solutions than RFT for Qwen2.5-0.5B and show longer reasoning with verification for Qwen2.5-3B.
  • The reported average accuracy covers models trained on Countdown, MATH(3,4,5), and Logic.
  • 9.45% accuracy makes UFT outperform SFT-RFT and RFT for Qwen2.5-0.5B.SFT-RFT reaches 7.28%, while RFT reaches 3.25%.
  • 30.93% accuracy lets UFT closely match RFT’s 32.15% for Qwen2.5-3B.SFT-RFT is substantially lower at 17.34%.
  • UFT matches or exceeds SFT when smaller models rely on memorization and benefits similarly to RFT when larger models require generalization.

C Proof of Theorem 4.2

Theorem 4.2 establishes a lower bound for reinforcement fine-tuning on branching search trees. Under sparse rewards, discovering an optimal policy can require exploration exponential in the tree height, motivating UFT’s theoretical analysis.

  • Theorem 4.2 considers trees with height H and branching factor B, targeting 50% pass@1 success.
  • With multiple correct leaves K, any algorithm must explore at least B^H/(4K) nodes in S_H.
  • When K is constant and rewards are sparse, learning the optimal policy takes iterations exponential in H.This lower bound is used to explain why long reasoning is difficult.
  • The proof constructs possible trees whose correct-solution leaves are selected uniformly, then analyzes leaf queries over T explorations.

E Proof of Theorem 4.3

Theorem 4.3 analyzes UFT through a theoretically specified algorithm that samples hints, estimates action values, updates the policy, and selects the best intermediate policy. The formal result provides a pass@1 guarantee under a condition on β and bounds exploration during training.

  • The proof combines concentration bounds, regret decomposition, and conversion from expected reward to success rate.
  • The training loop samples hint lengths with full support, evaluates actions across subsequent levels, and performs repeated updates over T steps.
Loading 2505.16984v2…