Source-linked AI summary
Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, Bin Cui, Wentao Zhang
TL;DR
The paper studies how to combine RL and SFT to improve reasoning and OOD generalization while reducing dependence on expensive demonstrations. It analyzes their complementary strengths and introduces ReLIFT, which interleaves RL with fine-tuning on hardest questions; ReLIFT achieves state-of-the-art results with less demonstration data and reduced training time.
Problem
RL primarily reinforces existing behaviors, while SFT can acquire new knowledge but depends on substantial high-quality demonstrations and may generalize less effectively OOD.
Method
ReLIFT identifies hardest examples during RL, obtains high-quality Chain-of-Thought solutions, adds them to a fine-tuning buffer, and interleaves RL with fine-tuning.
Results
ReLIFT achieves state-of-the-art results across five challenging math benchmarks and one OOD benchmark, including 52.6% accuracy with Qwen2.5-Math-7B.
Takeaways & Limitations
RL and SFT are complementary: RL is more effective for easier questions, while SFT is crucial for solving challenging questions, enabling ReLIFT to use less demonstration data.
Takeaways & Limitations
The experiments use DeepSeek-R1 as the source of high-quality data and do not include human-annotation experiments because of resource constraints.
Abstract
from arXiv · showhide
Recent advances in large language model (LLM) reasoning have shown that sophisticated behaviors such as planning and self-reflection can emerge through reinforcement learning (RL). However, despite these successes, RL in its current form remains insufficient to induce capabilities that exceed the limitations of the base model, as it is primarily optimized based on existing knowledge of the model rather than facilitating the acquisition of new information. To address this limitation, we employ supervised fine-tuning (SFT) to learn what RL cannot, which enables the incorporation of new knowledge and reasoning patterns by leveraging high-quality demonstration data. We analyze the training dynamics of RL and SFT for LLM reasoning and find that RL excels at maintaining and improving performance on questions within the model's original capabilities, while SFT is more effective at enabling progress on questions beyond the current scope of the model. Motivated by the complementary strengths of RL and SFT, we introduce a novel training approach, \textbf{ReLIFT} (\textbf{Re}inforcement \textbf{L}earning \textbf{I}nterleaved with Online \textbf{F}ine-\textbf{T}uning). In ReLIFT, the model is primarily trained using RL, but when it encounters challenging questions, high-quality solutions are collected for fine-tuning, and the training process alternates between RL and fine-tuning to enhance the model's reasoning abilities. ReLIFT achieves an average improvement of over +5.2 points across five competition-level benchmarks and one out-of-distribution benchmark compared to other zero-RL models. Furthermore, we demonstrate that ReLIFT outperforms both RL and SFT while using only 13\% of the detailed demonstration data, highlighting its scalability. These results provide compelling evidence that ReLIFT overcomes the fundamental limitations of RL and underscores the significant potential.
1 INTRODUCTION
RL and SFT have complementary roles in LLM reasoning: RL refines questions within existing capabilities, while SFT helps address more challenging questions. ReLIFT combines them by identifying hard examples during RL and using them for targeted fine-tuning.
- RL reinforces existing behaviors and may not foster capabilities beyond the base model’s inherent limitations.
- SFT incorporates new knowledge and reasoning patterns through high-quality demonstrations but depends on substantial data and may generalize less effectively to OOD scenarios.
- RL improves easier questions more effectively, whereas SFT is more beneficial for the hardest questions; SFT can also degrade some previously correct simpler answers.
- ReLIFT identifies challenging examples from RL rollouts, obtains high-quality solutions, filters incorrect answers, and periodically fine-tunes on the resulting buffer.
- ReLIFT achieves 52.6% accuracy with Qwen2.5-Math-7B and outperforms pure SFT, pure RL, and combined baselines across five math benchmarks and one OOD benchmark.
2 REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE FINE-TUNING
The analysis finds complementary training dynamics: RL preserves and improves performance on easier questions, while SFT is more effective for the hardest questions. ReLIFT interleaves RL with targeted fine-tuning on challenging examples collected during rollouts.
- RL VS. SFT: TRAINING DYNAMICS ACROSS QUESTION DIFFICULTY LEVELS: Hardest questions are items with zero correct answers out of eight, representing problems beyond the model’s current capabilities.
- RL VS. SFT: TRAINING DYNAMICS ACROSS QUESTION DIFFICULTY LEVELS: RL outperforms SFT on Easy and Medium questions, while SFT performs better on Hardest questions.SFT can reduce accuracy on some questions the initial model solved, whereas RL consistently improves accuracy across difficulty levels.
- REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE FINE-TUNING: ReLIFT identifies hardest examples during RL rollouts, obtains and filters high-quality Chain-of-Thought solutions, and stores reliable pairs in a fine-tuning buffer.A question is considered particularly challenging when its rollout accuracy is zero; incorrect solutions are filtered out.
- REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE FINE-TUNING: When the fine-tuning buffer reaches threshold M, ReLIFT samples a batch of hard question-solution pairs and performs one fine-tuning step.The fine-tuning step minimizes cross-entropy loss, with entropy regularization added to avoid overly constraining exploration.
- REINFORCEMENT LEARNING INTERLEAVED WITH ONLINE FINE-TUNING: ReLIFT adaptively alternates RL and fine-tuning, applying fine-tuning more frequently early and prioritizing RL as the model improves.The approach is intended to combine RL’s strength on existing abilities with SFT’s benefit for challenging questions.
3 EXPERIMENTAL SETUP
The experiments use filtered mathematical reasoning data and evaluate ReLIFT against SFT, RL, and combined-training baselines on five established benchmarks. The setup also includes an out-of-distribution evaluation component.
- Dataset Construction: The training set contains 46k prompts and high-quality demonstrations after filtering the default 94k OpenR1-Math-220k prompt subset.Generations longer than 8192 tokens or verified incorrect by Math-verify are removed, producing OpenR1-Math-46k-8192.
- Evaluation: Evaluation covers AIME 2024, AIME 2025, AMC, OlympiadBench, and MATH500, using avg@32 for the smaller test sets and avg@8 for OlympiadBench and MATH500.
- Baselines: ReLIFT is benchmarked with Qwen2.5-Math-7B against SFT, GRPO, multiple RL baselines, and methods combining SFT and RL.The combined baselines include RL with SFT loss, LUFFY, and SFT then RL.
4 EXPERIMENTAL RESULTS
ReLIFT achieves strong benchmark performance while using less training time and demonstration data, and its gains depend on selectively scheduled fine-tuning of difficult examples. The method also generalizes across multiple base models and benefits from a specific entropy coefficient.
- Main Results: 52.6% overall accuracy establishes ReLIFT as state of the art across five competition-level math benchmarks and one OOD benchmark.It achieves the best or second-best performance on every individual benchmark.
- Main Results: ReLIFT outperforms pure SFT, pure RL, and combined RL–SFT methods while producing more concise responses.The compared methods include RL with SFT loss, SFT then RL, and LUFFY.
- Main Results: Less training time and fewer demonstration samples accompany ReLIFT’s best performance.The resource comparison reports lower requirements while retaining the strongest results.
- Ablation Study: ReLIFT’s strategic schedule and focus on the hardest questions outperform all-step, uniform, and random fine-tuning schedules.The all-step variant quickly collapses, while uniform and random schedules yield lower accuracy and longer responses.
- Ablation Study: α = 1 × 10^-4 gives the highest reported accuracy of 52.6, while deviations in either direction significantly reduce performance.This coefficient is used consistently in the experiments.
- Generalization: ReLIFT consistently improves results across three additional base models and six benchmarks, maintaining a top-two ranking with acceptable response length.The tested models include Qwen2.5-Math-1.5B, Qwen2.5-7B, and Llama3.1-8B.
5 RELATED WORKS
Prior work shows that RLVR can improve reasoning but may reinforce existing capabilities rather than create genuinely new ones. Existing RL–SFT combinations remain constrained by demonstration quality and by the knowledge available to the SFT-trained model.
- Reinforcement Learning: RLVR has produced substantial gains in complex reasoning, including direct application to base models without intermediate SFT.Examples include OpenAI o-series, DeepSeek-R1, and Kimi k-series.
- Reinforcement Learning: RLVR-trained models can underperform their base models on pass@k at higher k values, indicating limited expansion of reasoning capabilities.This supports the claim that RLVR may not acquire genuinely new reasoning abilities.
- Hybrid Methods: Sequential SFT-then-RL methods depend on high-quality demonstrations, while later RL remains confined to knowledge present in the SFT-trained model.This makes acquisition of genuinely new information difficult.
6 DISCUSSION AND CONCLUSION
The paper concludes that RL and SFT have complementary strengths: RL is effective for easier questions, while SFT is crucial for harder ones. ReLIFT interleaves the two and achieves strong results with less demonstration data, while several implementation and scaling boundaries remain.
- Discussion: The experiments use DeepSeek-R1 as the source of high-quality data for reproducibility and cost-effectiveness, without experiments using human annotation.The authors state that the core mechanism is source-agnostic and reserve human-annotation experiments for future research.
- Discussion: Offline-collected data is used experimentally, although asynchronous external collection is described as technically feasible during RL updates.Feasibility depends on the data generation rate matching the rate of hard-problem identification.
- Conclusion: RL is more effective for easier questions, whereas SFT is crucial for solving challenging questions.This division motivates interleaving the two training approaches.
- Conclusion: ReLIFT achieves state-of-the-art results with significantly less demonstration data, reduced training time, and more concise solutions.The paper identifies scaling to larger models and improved RL–SFT coordination as future work.
B MAIN EXPERIMENT DETAILS
The experiments use standardized RL, SFT, hybrid-training, evaluation, reward, and prompting configurations across the reported model settings. Additional implementation choices adapt data, prompts, and response limits to the evaluated architecture.
- RL Implementation: RL uses eight rollouts per batch, a rollout batch size of 128, an update batch size of 64, and a learning rate of 1 × 10^-6.Rollout temperature is 1.0 for exploration, with KL, length normalization, and standard-error normalization omitted.
- SFT Implementation: SFT uses OpenR1-Math-46k-8192, batch size 64, learning rate 5 × 10^-5, and three training epochs.The configuration was selected through a small-scale hyperparameter search.
- Hybrid Baselines: RL with SFT loss combines seven on-policy samples with one off-policy SFT sample per prompt.The remaining setup follows the RL experiments.
- Hybrid Baselines: SFT-then-RL trains SFT for three epochs before running the RL phase for 300 steps.Both phases use the configurations specified for their standalone implementations.
- ReLIFT Implementation: ReLIFT uses a learning rate of 1 × 10^-6 for both RL and interleaved fine-tuning.Its other configurations follow the previously described RL and fine-tuning settings.
- Evaluation: Evaluation uses VLLM with temperature 0.6 and a maximum of 8192 tokens, with results verified by MATH-VERIFY.Baseline results are independently reproduced for methods combining SFT and RL.
- Reward Function: The reward function assigns reward 1 when the answer is correct.All training experiments use the same reward function.
- Model-Specific Settings: Llama3.1-8B uses 11,000 selected OpenR1 samples, a 2 × 10^-5 SFT learning rate, and a 2048-token maximum response length.The samples contain correct DeepSeek-R1 solutions shorter than 2,048 tokens.
D DETAILED EXPERIMENT RESULTS
The ablation study shows that ReLIFT’s effectiveness depends on both when fine-tuning occurs and which data are selected for it.
- Table 4 reports results for ReLIFT(all), ReLIFT(uniform), and ReLIFT(random).
- ReLIFT’s ablation results indicate that scheduling and fine-tuning-data selection are both crucial for success.The study compares ReLIFT(all), ReLIFT(uniform), and ReLIFT(random) settings.
E REASONING BEHAIVERS
Keyword analysis finds distinct problem-solving patterns across RL, SFT, and ReLIFT: SFT summarizes more, ReLIFT rethinks less, and ReLIFT plans more.
- Keyword counts are normalized by total token counts to compare responses with different lengths.
- Summarizing: SFT uses notably more summary keywords, while ReLIFT uses only slightly more than RL and both remain far below SFT.
- Rethinking: ReLIFT uses significantly fewer rethinking or re-evaluation keywords than the other models.The paper associates this pattern with greater efficiency and confidence in the initial approach.
- Planning: ReLIFT shows a substantial increase in planning keywords, indicating a more strategic problem-solving approach.The analysis describes this as a more robust and sophisticated approach to planning.
F CASE STUDY
AIME25 Problem 3 illustrates that ReLIFT produces a concise correct solution, whereas RL misses a valid combination and SFT uses excessive analysis.
- ReLIFT identifies all three valid combinations while producing a more concise response.The combinations are (6, 2, 1), (5, 3, 1), and (4, 3, 2).
- RL fails to include one possible combination in its response.Its response is marked incorrect and has length 1269 tokens.
- SFT obtains the three combinations but requires excessive analysis.Its response is marked correct and has length 6636 tokens.
- The problem asks for the remainder of the number of valid flavor assignments modulo 1000.Nine distinct players choose chocolate, vanilla, or strawberry, with chocolate counts greater than vanilla counts greater than strawberry counts.
G MORE OOD RESULTS
ReLIFT generalizes across four diverse out-of-distribution benchmarks, achieving the highest average score and ranking first or second on every benchmark.
- The OOD evaluation covers ARC-Challenge, GPQA-Diamond, MMLU-Redux, and HumanEval across STEM, coding, and broader domains.
- Table 5 reports overall performance across the four benchmarks for different models.
- ReLIFT achieves the highest average score and ranks first or second on every evaluated OOD benchmark.
- ReLIFT surpasses RL even though RL performs strongly on ARC-Challenge and MMLU-Redux.
H ABLATION STUDY ON BUFFER SIZE AND DIFFICULTY THRESHOLD
The ablation examines how ReLIFT’s difficulty threshold Q and fine-tuning buffer size M affect performance. Extreme settings degrade results, while several configurations achieve robust near-peak performance.
- Hyperparameter roles: Q gates retention of incorrect or uncertain rollout samples for SFT, while M controls the fine-tuning buffer size.The study evaluates these hyperparameters over 400 training steps on five math benchmarks.
- Suboptimal settings: 0.392 and 0.403 are the degraded performance values observed with an overly small buffer or relaxed Q combined with small M.The problematic settings include M = 32 and Q = 1/4 with M = 64.
- Robust configurations: Approximately 0.49 ± 0.003 is achieved by (M = 64, Q = 0), (M = 128, Q = 1/8), and (M = 256, Q = 1/4).These configurations balance data selection and fine-tuning capacity.
- Recommended configuration: (M = 64, Q = 0) is recommended because it reaches near-optimal performance while requiring the minimum demonstration data.The Q = 0 condition also minimizes gating-function complexity.
- Training dynamics: Training curves for different M and Q combinations show that extreme hyperparameter settings produce suboptimal results.The curves are presented in Figure 9.