Source-linked AI summary
Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, Ivan Titov
TL;DR
SFT and RFT have complementary strengths, but existing post-training methods treat them as distinct paradigms. Prefix-RFT combines demonstration prefixes with reward-guided exploration, and the paper reports improved performance over standalone and parallel mixed-policy baselines on math reasoning.
Problem
SFT imitates demonstrations but may generalize problematically, while RFT can learn unexpected behaviors and is sensitive to the initial policy; the paper asks how to integrate their complementary training signals.
Source: §1 · p. 1Method
Results
On math reasoning, Prefix-RFT outperforms standalone SFT and RFT as well as parallel mixed-policy methods.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that integrating SFT and RFT is a valuable direction and reports robustness across model scales, families, and demonstration quantities and qualities.
Source: §8 · p. 9Identifying optimal hyperparameters is challenging for unified training frameworks and remains future work.
Source: §5.1 · p. 7
Abstract
from arXiv · showhide
Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (1) SFT excels at mimicking demonstration data, but can lead to problematic generalization as a form of behavior cloning. (2) Conversely, RFT can significantly enhance a model's performance but is prone to learning unexpected behaviors, and its performance is sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a test bed, we empirically demonstrate that Prefix-RFT is simple yet effective. Not only does it surpass the performance of standalone SFT and RFT, but it also outperforms parallel mixed-policy RFT methods. Our analysis highlights the complementary nature of SFT and RFT, validating that Prefix-RFT effectively harmonizes them. Further ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data.
1. Introduction
SFT and RFT offer complementary strengths but distinct limitations, motivating a framework that combines demonstrations with reward-guided exploration. The paper introduces Prefix-RFT and reports stronger math-reasoning performance than standalone and mixed-policy alternatives.
1. Introduction
- SFT mimics curated demonstrations but, as behavioral cloning, may lead to problematic generalization and robustness; RFT can improve capabilities but may produce unexpected behaviors and depends on initial-policy strength.The paper also notes that RFT’s sparse rewards complicate credit assignment in multi-step tasks.Source: §1 · p. 1
- SFT supplies dense supervision that can expand a model’s knowledge, while RFT supplies goal-oriented optimization toward robust problem-solving.The paper frames these roles as complementary and asks how to integrate SFT process supervision with RFT optimization.Source: §1 · p. 1
- Prefix-RFT samples a demonstration prefix, generates its continuation with the policy, and trains on that hybrid trajectory alongside standard rollouts.The prefix guides exploration while leaving the model flexibility to find a better continuation than the demonstration.Source: §1 · pp. 1–2
- On math reasoning, Prefix-RFT outperforms naïve SFT, RFT, two-stage SFT-then-RFT, and parallel mixed-policy methods.The paper also reports validation across model scales and families and varying demonstration quantities and qualities.Source: §1 · p. 2
2. A Unified View on SFT and RFT
The section presents SFT and RFT as sharing a gradient-based optimization structure despite their different training signals. It then frames a hybrid objective that learns from both exploration and offline demonstrations.
2. A Unified View on SFT and RFT
- SFT uses offline expert demonstrations, with a low-variance gradient that pushes the model toward the expert data distribution.The section describes the SFT objective as imitation of demonstrations from an offline expert policy.Source: §2 · p. 2
- Policy-gradient RFT generates on-policy rollouts, collects rewards, and weights token updates by estimated advantage.For value-model-free methods such as GRPO, the estimated advantage is usually the same for every token in a response.Source: §2 · p. 2
- PPO-style RFT reuses samples from an older policy for multiple updates, using importance sampling and clipping to address distribution shift and limit large policy changes.Its gradient applies a dynamic, per-token weight to the log probability.Source: §2 · p. 2
- SFT, policy-gradient RFT, and PPO all update sequence log probabilities, differing in the sequences and weights used to shape those updates.SFT weights expert sequences implicitly by 1; policy-gradient RFT uses estimated advantages, while PPO adds dynamic per-token weights and clipping.Source: §2 · p. 2
- The hybrid objective allows responses from online rollouts, offline demonstrations, or both, and separates exploration tokens from imitation tokens with assigned weights.Figure 1 depicts a demonstration prefix guiding an online continuation, which is mixed with other online rollouts for RFT-style training.Source: §2 · p. 3, Figure 1 · p. 3
3. Prefix Reinforcement Fine-Tuning
Prefix-RFT combines SFT’s demonstration guidance with RFT’s exploration by using demonstration prefixes and policy-generated continuations in reinforcement training. It aims to retain SFT’s reliable learning direction while avoiding full-sequence imitation.
Prefix Reinforcement Fine-Tuning
- Prefix-RFT replaces one standard on-policy rollout with a demonstration prefix followed by a policy-generated continuation, and trains on that hybrid trajectory alongside standard rollouts.The total rollout budget remains the same as in standard RFT.Source: §3 · p. 3
- The full hybrid trajectory’s advantage determines whether its offline prefix is reinforced, providing partial guidance while the policy explores the continuation.The same PPO ratio and clipping mechanism penalize large updates from demonstration data.Source: §3 · p. 3
Entropy-based Clipping for Constrained Update on Demonstrations
- Entropy-based clipping retains only top-k% high-entropy demonstration tokens, filtering out low-entropy tokens that may add little signal or induce sharp overwrite updates.The retained prefix tokens are reinforced according to the trajectory-level advantage, not because entropy identifies uniquely correct tokens.Source: §3 · pp. 3–4
Controlling Prefix Length with Cosine Decay Scheduler
- A cosine-decay scheduler lowers the minimum sampled prefix-length fraction over training, mitigating the bias toward early demonstration tokens and adding a curriculum schedule.The authors note that some skills, such as drawing conclusions or summarizing, tend to occur near the end of sequences.Source: §3 · p. 4
Related Works
- Prefix-RFT differs from LUFFY’s mixing of offline data with on-policy rollouts, UFT’s weighted SFT loss on prefixes, and ReLIFT’s staged interleaving of SFT and RFT.The paper characterizes its approach as practical, simple, and easy to integrate into existing RFT pipelines.Source: §3 · p. 4
4. Main Experiments
On math and general reasoning benchmarks, Prefix-RFT performs strongly against the compared baselines, and the experiments examine its results across models and sampling budgets. The study uses math reasoning because reliable, inexpensive verifiers are available.
Experiment Settings and Baselines
- The experiments use about 46,000 length-filtered OpenR1-Math-220K problems with DeepSeek-R1 demonstrations, primarily training Qwen2.5-Math-7B.Math reasoning is selected as the test setting because reliable and inexpensive verifiers are available.Source: §4 · p. 4
Main Results
- In the main comparison, SFT and RFT perform similarly, SFT contributes more on AIME25, and RFT benefits more on AMC and MATH500; joint RFT with SFT loss is counterproductive.ReLIFT does not necessarily outperform the two-stage SFT+RFT baseline despite using less demonstration data.Source: §4 · p. 4
Results on more models
- On Qwen2.5-Math-1.5B, Prefix-RFT achieves an average score of 41.1, exceeding LUFFY and conventional SFT and RFT; a similar trend is reported for two other models.The other tested models are LLaMA-3.1-8B and Qwen3-1.7B-base.Source: §4 · pp. 4–5
Main Results
- Prefix-RFT significantly outperforms the baselines across six math benchmarks and three general-domain reasoning tasks, with performance comparable to concurrent work LUFFY.Table 1 reports the main math and general-reasoning benchmark results for Qwen2.5-Math-7B.Source: §4 · p. 4, Table 1 · p. 5
Pass@k results
- At Pass@2048, Prefix-RFT improves over the base model by 6.67 percentage points on both AIME24 and AIME25, and is the only method reported to raise the performance upper bound.The reported comparison evaluates performance across the sampling spectrum.Source: §4 · p. 5
5. Analysis
The analysis treats SFT and RFT as complementary, then examines whether Prefix-RFT combines their strengths and shifts reliance from demonstrations toward model-generated reasoning.
5.1. Does Prefix-RFT bridge SFT and RFT?
- SFT reaches Best@16 0.96 but Avg@16 0.52, while RFT reaches Avg@16 0.61, showing different strengths in finding and robustly solving solutions.SFT’s higher Best@16 reflects finding solution paths, while its lower Avg@16 reflects less reliable solving.Source: §5.1 · p. 6
- On Trainhard, Best@16 is 0.85 for SFT versus 0.14 for RFT; RFT’s demonstration loss rises from 0.67 to 0.81 during training.Best@16 converges faster than Avg@16, which the authors interpret as a discover-and-gradually-reinforce strategy.Source: §5.1 · p. 6
- Prefix-RFT achieves higher Avg@16 and Best@16 scores and lower demonstration loss than RFT, with gains on problems RFT could not solve.The analysis describes these findings as raising RFT’s upper bound.Source: §5.1 · p. 6
5.2. Advantage-driven updates induce dynamic transition between SFT and RFT
- For both model sizes, the positive reward gap between prefix-started rollouts and overall training gradually narrows, indicating declining reliance on prefixes.The 7B model’s prefix-started reward approaches 1.0 within its first 100 training steps.Source: §5.2 · p. 7, Figure 4 · p. 7
- Harder problems show greater demonstration-loss reduction, while easier problems show marginal reduction, indicating example-level shifts toward self-exploration.Difficulty is measured by mean solution accuracy; lower accuracy means a harder problem.Source: §5.2 · p. 7, Figure 5 · p. 7
6. Ablation Studies
The ablations compare entropy-based prefix-token selection and clipping with a cosine-decay scheduler against uniform sampling.
6. Ablation Studies
- Approximately 50% benchmark performance, the highest training reward, and the shortest 2k–2.5k-token responses are achieved by top 20% after 300 steps.Top 20% selects the 20% of prefix tokens with highest entropy.Source: §6 · p. 8
- Relaxing clipping or selecting random or low-entropy tokens reduces performance; bottom 20% harms both training reward and benchmark performance.The authors report that merely constraining update ratios is insufficient; selecting high-entropy tokens matters.Source: §6 · p. 8
- Uniform scheduling degrades performance but remains above naive SFT and RFT, and its prefix-to-overall reward gap widens rather than narrows.The comparison is against the cosine-decay schedule, whose reward gap narrows during training.Source: §6 · p. 8
7. Additional Results and Discussions
Additional experiments test Prefix-RFT with fewer or lower-quality demonstrations, on code generation, and under further methodological ablations.
7. Additional Results and Discussions
- 40.8 to 37.6 is the reported performance change when demonstrations are reduced from 45k to 0.45k samples, a 99% reduction.The 0.45k setting uses 1% of the training data; Prefix-RFT variants still outperform SFT and RFT baselines under the tested constraints.Source: §7 · p. 8, Table 2 · p. 9
- Performance is nearly identical with 1.5B- and 32B-generated demonstrations, but challenging benchmarks such as AIME are more affected by data constraints.The authors note that high-quality, large-scale demonstrations remain beneficial for top-difficulty problems.Source: §7 · pp. 8–9
- Prefix-RFT consistently improves over RFT on MBPP+, HumanEval+, and the Coder1 test split in a preliminary code-generation experiment.The experiment uses Qwen3-1.7B-Base and GPT-5.4-generated demonstrations.Source: §7 · p. 9, Table 3 · p. 9
- Prefix loss is crucial, and dynamic advantage-based weighting outperforms static weighting and gains from reinforcing high-quality prefixes rather than entropy clipping alone.Using prefixes only as guidance degrades performance to the standard RFT level.Source: §7 · p. 9
8. Conclusion
Prefix-RFT combines offline demonstration prefixes with reinforcement fine-tuning and is evaluated across model scales, families, and demonstration conditions. The paper reports limits to its current evidence: experiments focus mainly on verifiable reasoning, while open-ended generation, noisy rewards, and systematic demonstration selection remain future work.
8. Conclusion
- Prefix-RFT blends SFT and RFT by sampling prefixes from offline demonstrations as hints.The conclusion says the method was tested across model scales and families and with varying demonstration quantities and qualities.Source: §8 · p. 9
- The experiments primarily cover verifiable reasoning with reliable outcome rewards; broader open-ended generation and noisy-reward settings remain future work.Code-generation results suggest transfer to another verifier-based domain, while systematic selection among multiple demonstrations is also left for future study.Source: §8 · p. 9
Conflict of Interest Disclosure
An author is affiliated with Qwen Team, Alibaba Group; publicly available Qwen-series models are evaluated under the same protocols as the other models.
Conflict of Interest Disclosure
- An author is affiliated with Qwen Team, Alibaba Group, and publicly available Qwen-series models are evaluated under the same protocols as other models.Source: Conflict of Interest Disclosure · p. 10
Impact Statement
The authors state that the work aims to advance LLM post-training and identify no societal consequences requiring specific discussion.
Impact Statement
- The authors identify no societal consequences that they feel require specific highlighting.Source: Impact Statement · p. 10
A.1. Related Works
The related-work discussion situates Prefix-RFT among approaches combining offline data and online reinforcement learning, and the appendix evaluates it on math benchmarks and additional reasoning tasks. Results report gains across sampling budgets and model configurations, while ablations and design analyses examine prefix updates, clipping, and scheduling.
A.1. Related Works
- Prior work combines offline data and online RL through two-stage SFT-then-RFT, sampled offline states, or other integration strategies; the optimal way to join SFT and RFT remains open.The comparison describes UFT's static prefix weighting, LUFFY's full-trace mixing and policy reshaping, and ReLIFT's staged treatment of difficult problems.Source: §A.1 · p. 13
A.2. Evaluation and experiment hyperparameters
- Evaluation covers six math benchmarks, reporting avg@32 for AIME 2024, AIME 2025, and AMC, and pass@1 for Minerva, OlympiadBench, and MATH-500.ARC-c, GPQA-diamond, and MMLU-Pro are also used to test general reasoning; most hyperparameters follow Yan et al. (2025), with the method trained for 500 steps using Dr.GRPO.Source: §A.2 · p. 13
A.3. More experiment results
- Prefix-RFT scores 41.1 on Qwen2.5-Math-1.5B and is reported to outperform baselines across Qwen2.5-Math-1.5B, LLaMA-3.1-8B, and Qwen3-1.7B-Base.Five inference runs also show consistent wins over LUFFY; on AIME 2025 the reported margin is +4.07% (25.13 vs 21.06).Source: §A.3 · p. 14, Table 5 · p. 14, Table 6 · p. 15
- 76.7% Pass@2048 on AIME 2025 exceeds both the Base model and RFT, each at 70.0%.The reported advantage holds across sampling budgets from k = 1 to k = 2048.Source: §A.3 · p. 14, Table 7 · p. 15
- 45.4 vs 45.5 for Freeze Prefix and standard RFT shows similar performance; Update All reaches 45.7 with rollout-length explosion, while dynamic weighting scores 51.8 versus 43.8 for static weighting.Applying top-20% clipping to on-policy RFT rollouts scores 43.8, similar to naive RFT; the authors attribute the performance leap to prefix learning rather than clipping alone.Source: §A.3 · pp. 14–15, Table 8 · p. 16
- 33.8 for the DR-PO variant, which samples all eight rollouts from one prefix but excludes prefixes from the loss, is attributed possibly to train–inference distribution shift.Unclipped prefix updates produce gradient norms nearly double those of frozen-prefix and standard RFT updates, despite prefixes making up only 5%–10% of batch tokens.Source: §A.3 · pp. 14–15
A.4. Principled Design and Management of Hyperparameters
- Entropy-based clipping targets high-entropy prefix tokens as a conservative off-policy filter, and retaining 20% preserves most effective prefix-side gradient contribution.The text cautions that entropy selection is not a correctness oracle; nearly all lowest-entropy tokens already match the demonstration and contribute little target-logit gradient.Source: §A.4 · p. 16, Table 9 · p. 16, Table 10 · p. 16
- Cosine decay starts with long prefixes and moves to short ones, exposing expert actions at different positions while transitioning from SFT-like guidance toward RFT-like exploration.The authors describe the schedule as addressing position bias and creating a training curriculum; monitored rollout length can guide adjustments to clipping strength or prefix length.Source: §A.4 · pp. 16–17