Source-linked AI summary
A Practical Two-Stage Recipe for Mathematical LLMs: Maximizing Accuracy with SFT and Efficiency with Reinforcement Learning
Hiroshi Yoshihara, Taiki Yamaguchi, Yuichi Inoue
TL;DR
Mathematical LLM training lacks a systematic way to combine SFT and RL for both accuracy and efficiency. The paper proposes extended SFT followed by GRPO, finding that this sequence raises accuracy while GRPO reduces token usage, with strong results on challenging benchmarks including leak-free AIMO.
Problem
It remains unclear how to combine SFT and RL to maximize mathematical reasoning accuracy and efficiency.
Method
The paper applies extended SFT first, then GRPO with rewards designed to preserve accuracy and improve token efficiency.
Results
The recipe improves accuracy and token efficiency across AIME and MATH-500 evaluations and achieves top-tier performance on challenging benchmarks including AIMO.
Takeaways & Limitations
SFT and GRPO serve complementary roles: SFT pushes accuracy toward its limit, while GRPO makes high-performing mathematical reasoners more efficient.
Takeaways & Limitations
The recipe shows limited gains on the most challenging problems whose initial accuracy is low.
Abstract
from arXiv · showhide
Enhancing the mathematical reasoning of Large Language Models (LLMs) is a pivotal challenge in advancing AI capabilities. While Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are the dominant training paradigms, a systematic methodology for combining them to maximize both accuracy and efficiency remains largely unexplored. This paper introduces a practical and effective training recipe that strategically integrates extended SFT with RL from online inference (GRPO). We posit that these methods play complementary, not competing, roles: a prolonged SFT phase first pushes the model's accuracy to its limits, after which a GRPO phase dramatically improves token efficiency while preserving this peak performance. Our experiments reveal that extending SFT for as many as 10 epochs is crucial for performance breakthroughs, and that the primary role of GRPO in this framework is to optimize solution length. The efficacy of our recipe is rigorously validated through top-tier performance on challenging benchmarks, including a high rank among over 2,200 teams in the strictly leak-free AI Mathematical Olympiad (AIMO). This work provides the community with a battle-tested blueprint for developing state-of-the-art mathematical reasoners that are both exceptionally accurate and practically efficient. To ensure full reproducibility and empower future research, we will open-source our entire framework, including all code, model checkpoints, and training configurations at https://github.com/analokmaus/kaggle-aimo2-fast-math-r1.
1. Introduction
The paper addresses the unclear relationship between SFT and RL by proposing a sequential recipe that uses extended SFT for accuracy and GRPO for efficiency. Evaluations on challenging benchmarks, including leak-free AIMO, support the recipe’s practical effectiveness and reproducibility.
- The paper proposes sequentially applying extended SFT to establish a strong accuracy baseline, then GRPO to improve efficiency without compromising accuracy.
- 10 SFT epochs consistently and significantly improve problem-solving accuracy despite possible temporary dips during early epochs.
- GRPO primarily improves token efficiency after SFT by shortening generated solutions while maintaining or slightly improving high accuracy.
- The recipe achieves top-tier performance on AIME 2024, AIME 2025, and the strictly leak-free AIMO benchmark.
- The authors plan to release model weights, source code, datasets, checkpoints, and training configurations to support reproducibility.
2. Related Work
Prior work improves reasoning through scaling, inference-time computation, and post-training, while efficiency research addresses the cost of verbose reasoning. This paper distinguishes its approach by assigning accuracy and efficiency to sequential SFT and GRPO stages.
- Inference-time scaling can improve reasoning performance by allocating more computation through additional tokens or candidate solutions.
- Post-training research uses supervised learning and reinforcement learning to improve the reasoning abilities of LLMs.
- Chain-of-Thought prompting can produce excessively verbose rationales, creating computational overhead and latency.
- The proposed recipe decouples objectives by using extended SFT to maximize accuracy and GRPO to make the resulting model more token-efficient.
3. Methods
The method uses a two-stage pipeline: intensive SFT on a curated high-difficulty dataset, followed by GRPO designed to preserve accuracy while reducing solution length.
- 3. Methods: The training pipeline first applies intensive SFT and then GRPO to enhance token efficiency while preserving reasoning accuracy.
- The SFT dataset combines selected OpenR1 Math and openr1 hard examples emphasizing long, difficult, or moderately accurate reference solutions.
- Deduplication and shortest-correct-generation selection produce a dataset of 7,900 problem-solution trace-answer triplets.
- SFT is extended to 10 epochs, using full-parameter training with 24,000-token sequences and packing enabled.
- 3.2. Stage 2: GRPO for Enhanced Token Efficiency: GRPO follows SFT because SFT can increase accuracy while also producing longer, redundant reasoning traces.
- 3.2. Stage 2: GRPO for Enhanced Token Efficiency: GRPO combines format, cosine-similarity, and length-penalty rewards to enforce structure, provide continuous feedback, and discourage verbosity.
4. Experiments
The experiments evaluate the recipe on AIMO, AIME 2024, AIME 2025, and MATH-500, measuring pass@1 across repeated sampling runs. AIMO supplies a controlled, leak-free test environment for assessing generalization.
- The experiments analyze the combined method’s effects on accuracy and efficiency before evaluating the best model in the competitive AIMO setting.
- 4.1. Benchmarks: The evaluation covers leak-free AIMO, AIME 2024, AIME 2025, and MATH-500, with performance reported as pass@1 averaged over 64 sampling runs.
- 4.1. Benchmarks: AIMO uses 50 public problems for in-competition monitoring and 50 private problems for final assessment, limiting exposure to test cases.
4.2. Models
The study focuses on 14B models for AIMO while testing whether the recipe generalizes across 1.5B, 7B, and 14B DeepSeek-R1-Distill-Qwen models.
- 4.2. Models: The experiments target 14B models because of AIMO competition constraints, while also evaluating 1.5B and 7B variants.All base models are DeepSeek-R1-Distill-Qwen models.
- 4.2. Models: Table 2 reports mean Pass@1 and mean output token length on MATH500.
- 4.2. Models: Figure 1 compares SFT performance on AIME 2024 and 2025 using mean Pass@1 accuracy and mean token length per training epoch.
4.3. Results on AIME
On AIME 2024 and 2025, the proposed 10-epoch SFT followed by RL improves accuracy and token efficiency across nearly all model sizes, with larger gains at larger scales.
- 4.3. Results on AIME: 10-epoch SFT followed by RL improves accuracy and token efficiency over original models across nearly all model sizes on AIME 2024 and 2025.Accuracy gains become more pronounced as model size increases; direct RL can improve efficiency but reduce 14B accuracy.
- 4.3. Results on AIME: Figure 2 compares mean Pass@1 accuracy against mean token length across reward-function combinations and truncated token budgets from 8k to 32k.
- 4.3. Results on AIME: For the 1.5B model, 10-epoch SFT does not produce a substantial accuracy increase.
4.4. Results on MATH-500
On MATH-500, the recipe follows the same pattern as on AIME: extensive SFT raises accuracy and token length, while subsequent RL raises accuracy and reduces token length.
- 4.4. Results on MATH-500: On MATH-500, 10-epoch SFT increases both accuracy and mean token length, while subsequent RL further improves accuracy and reduces token length.The pattern holds despite MATH-500’s lower difficulty and lower inference-token requirements relative to AIME.
4.5. The Impact of Extensive SFT
The impact analysis finds that extended SFT is needed for stable, effective RL training, although the strongest gains are concentrated on problems that are not initially the most difficult.
- 4.5. The Impact of Extensive SFT: Accuracy improves as SFT epochs increase, whereas one epoch sharply increases average token length while reducing accuracy.The authors therefore identify prolonged SFT as crucial for enabling stable and effective subsequent RL training.
- 4.5. The Impact of Extensive SFT: Figure 3 shows per-problem changes in mean pass@1 and token length from the original model to the proposed recipe.
4.6. Ablation on Reward Functions
The reward ablation shows that length penalties reduce inference tokens, while cosine rewards slightly improve accuracy over binary accuracy rewards; the final recipe combines both.
- 4.6. Ablation on Reward Functions: Length penalties reduce the average number of inference tokens, while cosine rewards yield slightly higher accuracy than binary accuracy rewards.The comparison evaluates binary accuracy, cosine, and length-penalty configurations, with format reward used throughout.
- 4.6. Ablation on Reward Functions: The final training recipe combines cosine and length-penalty rewards to balance token efficiency and accuracy.
- 4.7. Analysis of Per-Problem Performance: Per-problem results show simultaneous improvements in mean pass@1 and solution length for most questions.The recipe maintains or improves accuracy on problems where the baseline approached 100%.
- 4.7. Analysis of Per-Problem Performance: The most challenging problems show limited performance gains, leaving them as a key challenge for future work.
4.8. Final Performance on the AIMO Benchmark
On the strictly leak-free AIMO benchmark, the model achieved strong public and private scores, ranking 4th and 8th respectively among 2,212 teams.
- 4.8. Final Performance on the AIMO Benchmark: 29/50 on the public set ranked the model 4th, while 28/50 on the private set ranked it 8th among 2,212 competing teams.
5. Conclusion
The paper concludes that extended SFT and subsequent GRPO play complementary roles: SFT maximizes mathematical accuracy, while GRPO improves token efficiency without compromising that performance. Evaluations on challenging benchmarks, especially the strictly leak-free AIMO competition, support the recipe’s practical effectiveness.
- 5. Conclusion: Extended SFT pushes accuracy toward its limit, while subsequent GRPO improves token efficiency without compromising peak performance.
- 5. Conclusion: The sequential SFT-then-GRPO strategy offers a practical pathway to models that combine high accuracy with efficiency.
- 5. Conclusion: The recipe achieved top-tier performance on AIME and MATH and a high rank in the competitive, strictly leak-free AIMO benchmark.The authors present these results as evidence of robustness and real-world effectiveness.