PaperlayerView exact revision on arXiv

Source-linked AI summary

SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, Dongbin Zhao

arXiv:2506.19767v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how to integrate SFT and RL effectively for LLM reasoning. It analyzes their policy effects and entropy dynamics, then proposes single-stage SRFT with entropy-aware weighting. SRFT achieves 59.1% average accuracy, outperforming zero-RL baselines by 9.0% on reasoning tasks and 10.9% on OOD benchmarks.

  • Problem

    The effective balance between SFT’s demonstration learning and RL’s exploration remains unclear because sequential or overly weighted approaches face overfitting, exploration, and optimization challenges.

  • Method

    SRFT is a single-stage method that simultaneously applies SFT and RL to demonstrations and self-exploration rollouts using entropy-aware weighting.

  • Results

    59.1% average accuracy was achieved, with improvements of 9.0% over zero-RL baselines on reasoning tasks and 10.9% on out-of-distribution benchmarks.

  • Takeaways & Limitations

    SFT makes coarse-grained global policy adjustments, RL makes fine-grained selective optimizations, and entropy provides an indicator for integrating them.

  • Takeaways & Limitations

    The method uses relatively simple exponential entropy weighting and assumes access to high-quality demonstrations.

Abstract

from arXiv · show

Large language models (LLMs) have achieved remarkable progress in reasoning tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) remains a fundamental challenge. Through comprehensive analysis of token distributions, learning dynamics, and integration mechanisms from entropy-based perspectives, we reveal key differences between these paradigms: SFT induces coarse-grained global changes to LLM policy distributions, while RL performs fine-grained selective optimizations, with entropy serving as a critical indicator of training effectiveness. Building on these observations, we propose Supervised Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach simultaneously applies SFT and RL to directly optimize the LLM using demonstrations and self-exploration rollouts rather than through two-stage sequential methods. Extensive experiments show that SRFT achieves 59.1% average accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning benchmarks and 10.9% on three out-of-distribution benchmarks.

1 INTRODUCTION

SFT and RL present complementary but difficult-to-balance approaches to reasoning fine-tuning: SFT broadly follows demonstrations, whereas RL selectively explores policy improvements. SRFT unifies them in one entropy-aware stage and improves benchmark performance.

  • SFT can overfit demonstrations, while RL may be sample inefficient, explore poorly, or collapse toward repetitive suboptimal outputs.
  • SFT guides policies toward demonstration distributions, whereas RL explores improved solutions near the base policy, but either approach alone can be limited.
  • SFT makes coarse-grained global policy changes, while RL performs fine-grained selective modifications.
  • Entropy dynamics indicate how to balance SFT and RL during training.
  • SRFT combines demonstrations and self-exploration in a single stage using entropy-aware weighting mechanisms.
  • 59.1% accuracy was achieved across five mathematical reasoning benchmarks, while improvements over zero-RL baselines were 9.0% on reasoning tasks and 10.9% on OOD tasks.

2 PRELIMINARIES

SFT trains a language-model policy to reproduce target responses, while RL models token generation as sequential decision-making and updates the policy using reward-based objectives.

  • SFT minimizes negative log-likelihood so the policy maximizes the conditional probability of target responses given input prompts.
  • In SFT notation, x_i is an input prompt and y_i is its target response generated by the behavior policy.
  • RL represents token generation as a Markov Decision Process, with states formed from the prompt and tokens generated so far.
  • Rewards are often sparse and delayed until sequence completion, complicating credit assignment across generated tokens.
  • GRPO generates response groups for each prompt, normalizes rewards into relative advantages, and applies clipped updates without a learned value function.

3 ANALYSIS OF SFT AND RL IN LLM REASONING

The analysis contrasts SFT and RL through token distributions, learning dynamics, and integration order. It finds that SFT changes policies broadly, RL modifies them selectively, and single-stage integration is more effective than unfavorable sequential orderings.

  • 3.1.1 EFFECTS ON TOKEN DISTRIBUTIONS: SFT substantially changes probabilities across entire response sequences, whereas RL selectively modifies a small subset of tokens.RL leaves numerical content and mathematical proof statements largely unchanged.
  • 3.1.1 EFFECTS ON TOKEN DISTRIBUTIONS: SFT sharpens distributions by increasing target-token probabilities and decreasing probabilities for other vocabulary tokens.This produces more deterministic outputs.
  • 3.1.2 VISUALIZATION OF LEARNING DYNAMICS: All fine-tuning paradigms improve performance while moving away from the base-model probability space.SFT makes larger distribution changes than RL, while RL remains nearer its initialization neighborhood.
  • 3.2.1 SEQUENTIAL INTEGRATION ANALYSIS: Applying SFT after RL consistently produces suboptimal performance, and a KL-divergence constraint yields only limited improvements.This ordering is described as fundamentally incompatible, motivating entropy-based analysis.
  • 3.2.1 SEQUENTIAL INTEGRATION ANALYSIS: RL-after-SFT is more suitable for reasoning than RL-after-SFT reversal, while entropy dynamics help explain the asymmetry.After RL, entropy becomes low; subsequent SFT sharply increases it before a gradual decline, with a plateau after approximately 90 steps.
  • 3.2.2 SINGLE-STAGE INTEGRATION ANALYSIS: Single-stage SFT+RL combines both objectives and achieves superior training efficiency compared with sequential SFT→RL approaches.Unified optimization leverages demonstrations while preserving supervised knowledge-distillation benefits, and both dataset-utilization methods outperform pure RL across performance metrics.

4 METHOD

SRFT unifies supervised fine-tuning and reinforcement learning in one stage, combining demonstration learning with self-generated exploration through entropy-aware weighting. It uses SFT for coarse behavior-policy approximation and RL for fine-grained policy refinement.

  • SRFT applies SFT and RL in a single stage to both demonstration data and self-generated trial-and-error data.The method combines coarse-grained behavior-policy approximation with fine-grained policy refinement.
  • Learning from demonstrations: Demonstrations support SFT-based approximation of the expert behavior policy and off-policy RL-based fine-grained learning.Demonstrations are incorporated into heterogeneous batches alongside on-policy rollout groups.
  • Learning from demonstrations: Expert demonstrations typically receive higher rewards, increasing group advantages and promoting optimistic exploration in the policy.
  • Entropy-aware integration: wSFT = 0.5 ∗stop_grad(exp(−H(πθ))) reduces the SFT loss influence when policy entropy is high.The weighting mitigates performance degradation from distribution mismatch while preserving behavior-policy approximation.
  • Learning from self-exploration: Self-exploration learning decomposes binary-reward RL into positive-response likelihood maximization and negative-response likelihood minimization.Positive samples are generated on-policy, while negative samples reduce probability mass assigned to incorrect responses.
  • Entropy-aware integration: SRFT combines demonstrations and self-exploration rollouts with two entropy-aware weighting mechanisms to maintain stable training dynamics.

5 EXPERIMENTS.

Experiments evaluate SRFT on mathematical reasoning and out-of-distribution benchmarks using Qwen2.5-Math-7B. SRFT achieves strong benchmark performance, faster improvement than RL, stable entropy, and component-sensitive results.

  • Experimental Setup: Training uses OpenR1-Math-46k-8192 demonstrations and rollout prompts, with eight trajectories per prompt and 500 training steps.
  • Evaluation Setup: The evaluation covers AIME24, AMC, Minerva, OlympiadBench, and MATH500, using avg@32 for AIME24 and AMC and pass@1 for the other three datasets.
  • Reasoning Benchmark Performance: SRFT achieves an average score of 59.1 across five competition-level mathematical reasoning benchmarks, exceeding the best RL baseline by +9.0 points.It also improves over SFT methods by +4.8 points and SFT+RL methods by +3.4 points.
  • Out-of-Distribution Generalization: SRFT achieves an average score of 62.5 on out-of-distribution benchmarks, outperforming the best baseline by +4.7 points.
  • Training Dynamics: SRFT improves performance faster than RL while both methods show increasing training rewards.
  • Training Dynamics: RL generates more concise responses on challenging data, whereas SRFT progressively lengthens responses and maintains more stable entropy.The paper associates SRFT’s stable entropy with continued exploration during training.
  • Ablation Study: Removing the entropy-aware SFT weighting mechanism causes a -4.0 point performance drop.

6 RELATED WORK

Related work positions RL as a way to extend reasoning beyond SFT and studies how supervised foundations and reinforcement learning interact. Existing work explores both sequential and synergistic combinations.

  • Reinforcement Learning for LLM Reasoning: RL methods have improved mathematical reasoning and complex problem-solving beyond the limitations attributed to SFT alone.
  • Integrating SFT and RL: Research on integrating SFT and RL examines how reasoning chains establish an initial policy foundation that RL can subsequently optimize.

7 CONCLUSION

SRFT unifies SFT and RL in a single stage using entropy-aware weighting, while achieving strong reasoning and OOD benchmark performance. The paper also identifies simple entropy weighting and reliance on high-quality demonstrations as limitations.

  • 59.1% average accuracy is achieved by SRFT, with improvements of 9.0% on reasoning tasks and 10.9% on out-of-distribution benchmarks over zero-RL baselines.These results cover five reasoning benchmarks and three out-of-distribution benchmarks.
  • SRFT unifies SFT and RL through entropy-aware weighting in a single-stage approach.The method is designed to optimize reasoning using both fine-tuning paradigms together.
  • The current method uses relatively simple exponential entropy-weighting functions, leaving adaptive scheduling and multi-timescale analysis for future work.The limitation concerns the simplicity of the entropy-control mechanism.
  • SRFT assumes access to high-quality demonstrations, while training with imperfect demonstrations remains an open direction.

A MORE EXPERIMENTAL RESULTS

The experiments examine SRFT’s token-probability changes and reinforcement-learning behavior through visualizations and controlled gradient-truncation tests. The supplied passages characterize SRFT as balancing SFT and RL while targeting high-entropy tokens.

  • SRFT produces moderate token-probability changes, balancing SFT and RL while preserving the model’s base abilities.The visualization links this balance to enhanced reasoning capabilities.
  • Figure A1 visualizes the token probability distribution of SRFT after training.
  • RL predominantly targets tokens with high-entropy distributions during fine-tuning.Controlled experiments truncate gradients for high-probability tokens to validate this observation.
  • Figure A2 reports performance for RL with gradient clipping applied to low-entropy tokens.

B EXPERIMENTAL DETAILS

The experimental setup combines specified SFT, RL, and SRFT training configurations with evaluations across eight benchmarks. Training uses OpenR1-Math-220k reasoning traces and 64 A100 GPUs, while evaluation uses VLLM and task-specific metrics.

  • Training: SFT trains for 3 epochs with batch size 128, learning rate 5×10−6, and 10% linear-schedule warmup.KL-regularized SFT adds divergence from the base model with λ = 0.2.
  • Training: RL and SRFT use RL training settings, including 500 steps, 8 rollouts per prompt, and a fixed learning rate of 1 × 10−6.The setup also increases RoPE theta from 10,000 to 40,000 and the window size to 16,384.
  • Evaluation: Evaluations use temperature 0.6 and maximum generation length 8192, with avg@32 for AIME24 and AMC and pass@1 for the other three datasets.Generated solutions are checked with Math-Verify and OAT-Grader.
  • Templates: Figure A3 presents chat templates for SFT, RL, and SRFT training paradigms.
  • Ablations: Template ablation indicates that the design guides reasoning while maintaining consistency across mathematical domains.
  • Ablations: Table A1 reports template-ablation results on mathematical reasoning benchmarks.
  • Training Dataset: OpenR1-Math-220k contains 220,000 math problems with two to four DeepSeek-R1 reasoning traces per problem.The traces are tool-verified so that each problem has at least one correct reasoning path.
  • Evaluation: The study evaluates eight benchmarks divided into mathematical reasoning and out-of-distribution categories.

C.2.1 MATHEMATICAL REASONING BENCHMARKS

The mathematical reasoning evaluation includes competition, curated, advanced, and olympiad-level benchmarks spanning varied difficulty and mathematical domains. These datasets assess step-by-step problem solving, multi-step reasoning, symbolic manipulation, and mathematical insight.

  • AIME24 contains 2024 American Invitational Mathematics Examination problems requiring advanced mathematical reasoning and integer answers from 0 to 999.The benchmark includes 15 problems across algebra, geometry, trigonometry, number theory, probability, and combinatorics.
  • AMC contains 83 problems from AMC12 2022 and 2023, covering elementary algebra, geometry, number theory, probability, and related topics.
  • MATH500 is a 500-question subset of Hendrycks’ 2021 MATH dataset designed as a clean evaluation set avoiding potential data contamination.
  • Minerva evaluates advanced mathematical problem solving across high-school to undergraduate domains, including multi-step reasoning and symbolic manipulation.
  • Olympiad benchmarks contain competition-level mathematics organized across 33+ sub-domains and 10+ difficulty levels.The problems require mathematical insight, creativity, and advanced problem-solving ability.

D SFT GRADIENT DERIVATION

The derivation shows how SFT gradients raise target-token probabilities while reducing non-target probabilities across the vocabulary. It also formalizes model space and reference-model distances for analyzing training dynamics.

  • SFT Gradient Derivation: The target-token gradient depends on model prediction probabilities across the entire vocabulary because softmax normalization affects all tokens.The derivation expresses the log-probability gradient using the model’s predictions for all vocabulary tokens.
  • SFT Gradient Mechanism: SFT gradients increase the target token’s probability and decrease every non-target token’s probability at each time step.The decrease for each non-target token is proportional to its current probability.
  • SFT Gradient Mechanism: SFT produces broad, coarse-grained changes that systematically sharpen probability distributions toward training targets.This conclusion is presented as confirmation of empirical observations.
  • Model Space Formalization: The model space consists of conditional probability distributions over token sequences, represented as an infinite-dimensional vector space.Each dimension corresponds to a token’s conditional probability at a position given the prompt and preceding tokens.
  • Model Space Formalization: Distances to reference models provide a tractable way to measure and visualize model dynamics in a projected subspace.The distance quantifies differences in assigned sequence probabilities.

E.2 EXPERIMENTAL SETUP

The experimental setup constructs reference-response data from diverse mathematical reasoning benchmarks and tracks model checkpoints through a reference-model distance framework. These distances produce trajectories for comparing optimization dynamics across training paradigms.

  • Reference-Response Dataset: 1,024 response sequences per reference model are generated under identical prompts drawn from five mathematical reasoning benchmarks.The benchmarks include AIME24, Minerva, Olympiad, AMC, and MATH-500.
  • Reference Models: Three reference models define the projection basis, including DeepSeek-R1 for state-of-the-art reasoning performance and QwQ-32B as a high-performing model.The passage identifies these models as two components of the selected reference set.
  • Model-Dynamics Tracking: Training checkpoints are evaluated by their probability assignments over reference responses, yielding trajectories that compare SFT and RL optimization dynamics.The trajectories are used to examine model evolution over time, including convergence behaviors.
Loading 2506.19767v1…