Source-linked AI summary
AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
TL;DR
The paper asks how SFT data scaling and RL interact when building strong math and code reasoning models. It constructs stronger SFT initializations, studies RL initialization and sampling temperature, and applies a staged curriculum. The resulting AceReason-Nemotron-1.1 7B achieves state-of-the-art performance among Qwen2.5-7B-based reasoning models.
Problem
The paper studies the limited systematic evidence on how SFT initialization and RL training jointly affect reasoning-model performance.
Method
The authors scale SFT data along prompt and response axes, then apply staged on-policy RL with curated verification-based math and code data.
Results
AceReason-Nemotron-1.1 7B achieves state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks.
Takeaways & Limitations
Stronger SFT foundations remain advantageous after effective RL, while appropriate temperature tuning and staged training narrow initialization gaps and improve reasoning.
Abstract
from arXiv · showhide
In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We begin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scaling the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL: (i) Does a stronger SFT model consistently lead to better final performance after large-scale RL training? (ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization? Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the performance gap between initial SFT models narrows significantly throughout the RL process. Leveraging a strong SFT foundation and insights into the synergistic interplay between SFT and RL, our AceReason-Nemotron-1.1 7B model significantly outperforms AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks, thereby demonstrating the effectiveness of our post-training recipe. We release the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B
1. Introduction
The paper studies how supervised fine-tuning and reinforcement learning can be integrated to build stronger math and code reasoning models. It scales SFT data, analyzes RL initialization and temperature, and reports strong performance from the resulting recipe.
- Long chain-of-thought reasoning acquired through large-scale RL has become a major driver of progress in frontier language models.
- The work examines the synergy between SFT and RL through training-dynamics analyses and an integrated post-training approach.
- Scaling prompts, responses per prompt, and training epochs improves reasoning performance, with prompt scaling producing larger gains and improvements plateauing around the fifth or sixth epoch.
- Stronger SFT models remain better after large-scale RL, while their performance gap narrows during training.
- Temperature selection during RL is important for balancing exploration and exploitation for a given SFT initialization.
- AceReason-Nemotron-1.1 7B achieves record-high performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks.
2. Related Work
Prior work advances math and code reasoning through SFT distillation and verification-based RL, but systematic analysis of how SFT and RL interact remains limited. The paper’s training pipeline combines these approaches across staged math and code RL.
- Verification-based RL uses external verifiers to reward outputs whose correctness can be checked against oracle answers.
- GRPO became popular for math and code reasoning because of its simplicity, robustness, and effectiveness.
- SFT distillation methods train smaller models on synthetic reasoning samples generated by frontier reasoning models such as DeepSeek-R1 or QwQ.
- Technical reports often lack a systematic study of the interplay and integration between SFT and RL.
- The AceReason-Nemotron 1.1 pipeline combines math and code SFT with staged math-only RL, code-only RL, and a final math-only RL stage.
- The pipeline uses progressively longer response lengths for its staged math RL, covering 8K, 16K, and 24K tokens.
3. Method
The method builds curated math and code SFT datasets, scales prompts and responses, and applies on-policy GRPO through a staged math-code-math RL curriculum. Data quality, verification, and response-length scheduling are central design choices.
- Training pipeline: The training pipeline begins with math and code SFT, followed by staged math-only RL, code-only RL, and a final math-only RL stage.
- SFT data curation: The SFT data collection uses deduplicated math and coding prompts from multiple high-quality datasets, followed by decontamination against test samples.
- SFT data curation: The final SFT dataset contains 247K math prompts and 136K code prompts after filtering simpler samples to balance difficulty levels.
- SFT data scaling: SFT scaling increases unique prompts for broader problem coverage and responses per prompt for diverse reasoning paths.
- SFT data scaling: Seven SFT datasets scale from 36K samples in v1 to 2.2M samples in v7 while maintaining similar response-length distributions.
- RL training: The RL procedure applies on-policy GRPO with 8 or 16 rollouts per question, a single policy-gradient update, token-level advantages, and no KL-divergence term.
- RL data curation: Rule-based verification computes reward scores, while RL data quality depends on suitable prompt difficulty, accurate answers, and reliable coding test cases.
- RL curriculum: Math RL extends response lengths from 8K to 16K and 24K, followed by code RL at 24K and 32K and final math RL at 32K.
4. Evaluation
Scaling SFT data and training, followed by carefully configured multistage RL, improves math and code reasoning. RL gains persist from strong SFT initializations, narrow some starting gaps, and depend on balancing exploration with exploitation.
- Main results: RL produces substantial gains across math and code: 10.6% on AIME24, 16.4% on AIME25, 8.4% on LiveCodeBench v5, and 8.3% on v6.The same RL method and training data are applied to the strong AceReason-Nemotron-1.1-7B SFT model.
- SFT data scaling: 8% AIME25 improvement, from 41.3 to 49.3, follows further response-per-prompt scaling from SFT v5 to v7.The final dataset maintains a similar number of prompts while increasing the number of responses per prompt.
- SFT data scaling: Scaling prompts and responses per prompt significantly improves model performance, with expanding unique prompts producing the larger effect.The fitted coefficients are a=4.831 for unique prompts and b=2.635 for responses per prompt, with R^2=0.989.
- SFT training: Performance improves through the fifth SFT epoch and plateaus around the fifth to sixth, while v7 consistently outperforms v6.The v7 dataset contains nearly twice as many responses per prompt on average, and multi-epoch training benefits both versions.
5. Conclusion
The study finds that scaling SFT data improves reasoning, while effective RL narrows initial model differences and produces strong math and code performance. The final model reaches 63.2% on AIME25 and 52.8% on LiveCodeBench V5.
- Both increasing unique prompts and responses per prompt substantially improve reasoning, with prompt scaling producing larger gains.
- SFT performance improves through the fifth epoch, then plateaus around the fifth or sixth epoch.
- RL significantly reduces performance gaps between initially different SFT models, although its first stage may not immediately improve results.
- Math-only RL benefits even strong coding SFT models and can further improve coding performance.
- 63.2% on AIME25 and 52.8% on LiveCodeBench V5 are achieved by the final 7B model.
Math
The section instructs the reader to provide a final answer enclosed in a boxed format.
- The final answer must be placed inside a box.
No Starter Code (Python)
The section instructs the reader to write Python code for solving the problem and use a specified code-block format.
- Write Python code to solve the problem.
- Place the solution in the specified Python code-block format.
Has Starter Code
The section addresses problems with starter code and requires solving from the provided function header using the specified solution format.
- Solve the problem starting with the provided function header.
- Use the starter code and place the solution in the specified Python code-block format.
B. RL training from different SFT models on AIME25
Figure 14 tracks AIME25 accuracy from different SFT initializations through four math-only RL stages, showing that training narrows some but not all performance gaps.
- RL training progression: AIME25 accuracy is reported at SFT initialization and after Math-Only RL Stages 1–4, extending response lengths from 8K to 32K.Step-0 reflects initial SFT checkpoints; later values are final accuracies at each training stage.
- Comparing SFT initializations: The final gap between RL models initialized from SFT-7B-v5 and SFT-7B-v7 narrows after training.The observation is consistent with results shown in Figure 7.
- Comparing SFT initializations: A substantial performance gap remains between RL initialized from SFT-7B-v7 and DeepSeek-R1-Distill-Qwen-7B.The result suggests RL has limited ability to close gaps when initial SFT quality differs substantially.
C. Pass@k Accuracy on Math-Only RL Models
The paper evaluates pass@k for two SFT models and their math-only RL-trained versions using repeated random sampling, finding gains for both RL models.
- Evaluation procedure: Pass@k is evaluated on AIME24, AIME25, LiveCodeBench V5, and V6 by repeatedly averaging scores from randomly selected outputs.The procedure generates 256 outputs per math sample and 128 per coding sample, repeats random selection 100 times, and averages the results.
- RL effects: Math-only RL improves pass@k for both evaluated SFT models, consistent with the paper’s broader observations.The comparison covers two SFT models and their subsequent math-only RL-trained versions.
- RL effects: RL from the weaker SFT v5 produces greater pass@k improvements than RL from the stronger SFT model.The paper states that this mirrors the pass@1 results.
D. Problem-Level Solving Rates on Math-Only RL Models
Problem-level comparisons show that math-only RL improves coding transfer as well as math performance, enabling models to solve some problems missed by their initial SFT checkpoints.
- Cross-benchmark comparison: Problem-level solving rates compare SFT models with their math-only RL-trained versions across AIME24, AIME25, LiveCodeBench V5, and V6.Accuracy is averaged over 256 outputs for AIME24 and AIME25 and 128 outputs for LiveCodeBench V5 and V6.
- Cross-domain transfer: Despite training solely on math tasks, the RL model performs well on coding benchmarks.The comparison is presented for both the SFT v7 and SFT v5 starting points.
- Cross-domain transfer: Math-only RL enables the model to solve some coding problems that the initial SFT model could not.The passage describes this as occurring alongside the coding-benchmark performance comparison.