Source-linked AI summary
AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
TL;DR
Reasoning-model RL recipes and data-curation details remain insufficiently specified, while prior work often favors distillation for smaller models. This paper studies staged RL from strong distilled 7B and 14B models using curated verifiable data and finds gains across math and code, including cross-domain transfer with little math degradation after code RL.
Problem
Frontier RL recipes and data-curation strategies are incompletely reported, and prior studies often find distillation more effective than RL for smaller models.
Method
The paper trains strong distilled 7B and 14B models with math-only RL followed by code-only RL, curated prompts, and verification-based rewards.
Results
RL substantially improves math and code reasoning, with math-only RL transferring to coding and later code-only RL adding coding gains with minimal math degradation.
Takeaways & Limitations
RL elicits capabilities acquired through pretraining and fine-tuning while extending performance to previously unsolvable problems.
Abstract
from arXiv · showhide
Despite recent progress in large-scale reinforcement learning (RL) for reasoning, the training recipe for building high-performing reasoning models remains elusive. Key implementation details of frontier models, such as DeepSeek-R1, including data curation strategies and RL training recipe, are often omitted. Moreover, recent research indicates distillation remains more effective than RL for smaller models. In this work, we demonstrate that large-scale RL can significantly enhance the reasoning capabilities of strong, small- and mid-sized models, achieving results that surpass those of state-of-the-art distillation-based models. We systematically study the RL training process through extensive ablations and propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts. Notably, we find that math-only RL not only significantly enhances the performance of strong distilled models on math benchmarks (e.g., +14.6% / +17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks (e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition, extended code-only RL iterations further improve performance on code benchmarks with minimal or no degradation in math results. We develop a robust data curation pipeline to collect challenging prompts with high-quality, verifiable answers and test cases to enable verification-based RL across both domains. Finally, we identify key experimental insights, including curriculum learning with progressively increasing response lengths and the stabilizing effect of on-policy parameter updates. We find that RL not only elicits the foundational reasoning capabilities acquired during pretraining and supervised fine-tuning (e.g., distillation), but also pushes the limits of the model's reasoning ability, enabling it to solve problems that were previously unsolvable.
1. Introduction
The paper addresses missing RL recipes and the belief that distillation is better for smaller models by showing that large-scale RL can substantially improve strong 7B and 14B distilled models. Its staged math-then-code approach yields cross-domain gains and supports further reasoning improvements.
- Motivation and contribution: Large-scale RL substantially enhances reasoning in strong small- and mid-sized distilled models, matching distillation-based results at 7B and surpassing them at 14B.The starting models are DeepSeek-R1-Qwen-Distilled-7B/14B.
- Motivation and contribution: The proposed recipe trains first on math-only prompts and then on code-only prompts.The separate stages were motivated partly by the substantially longer verification time for code.
- Motivation and contribution: +14.6% / +17.2% on AIME 2025 for 7B / 14B models after math-only RL.The same passage reports cross-domain LiveCodeBench gains of +6.8% / +5.8%.
- Motivation and contribution: +6.8% / +5.8% on LiveCodeBench v5 for 7B / 14B models after math-only RL.Math-only RL therefore improves code reasoning as well as math performance.
- Motivation and contribution: Extended code-only RL causes minimal or no degradation on math reasoning tasks.For the 7B model, the reported AIME 2024 / 2025 changes are +1.0% / -0.8%.
- Motivation and contribution: The work develops a systematic data-curation recipe for high-quality math problems and coding tasks with verifiable answers or test cases.These resources support verification-based RL across both domains.
2. Related Work
Prior work established RL and distillation as routes to reasoning, but reported results vary with model size, initialization, and domain. The paper positions its contribution against missing reproducible recipes and evidence favoring distillation for smaller models.
- Reasoning-model training: Reasoning models have been developed through pretraining, supervised fine-tuning, and RL across math and code domains.Rule-based verification supports deterministic math answers and code execution against predefined test cases.
- Reasoning-model training: Large-scale RL became prominent through OpenAI o1 and DeepSeek-R1, particularly with rule-based verification.Math uses formatted deterministic answers, while code uses compilation and execution against test cases.
- Open training questions: Subsequent work introduced data-curation methods and GRPO-stabilization techniques such as progressive response-length increases, clip-higher, and overlong filtering.These efforts respond to omitted implementation details in frontier-model reports.
- Open training questions: Prior reports found distillation more effective than RL for small and mid-sized models, recommending RL mainly for the largest models.The paper instead investigates RL initialized from strong distilled models.
3. Method
The method combines verification-based GRPO with curated math and code data, strict on-policy updates, and staged curricula that increase response length and problem difficulty. Sequential math-only and code-only RL improves reasoning across both domains, including cross-domain coding gains from math RL.
- RL framework: GRPO samples rollout groups, assigns rule-based rewards against oracle answers, and uses normalized group rewards for token-level policy updates.The method avoids a separate value-function model and uses a token-level loss variant.
- RL framework: Strict on-policy training performs one gradient update after each rollout group, stabilizing training and preventing entropy collapse.The current and data-collection policies therefore match, making the importance weight equal to 1.
- Data curation: Math data combines DeepScaler and NuminaMath, filters contamination and unsuitable questions, and retains problems with verified majority-voted solutions.Questions requiring fewer than 2,000 reasoning tokens are removed, while 2,000–4,000-token problems are downsampled.
- Training process: The training curriculum extends response lengths from 8K to 32K tokens and increases prompt difficulty at later stages.The pipeline identifies progressive length extension and increasingly difficult prompts as efficiency and reliability strategies.
- Results: Math-only RL raises LiveCodeBench v5 to 44.4% (+6.8%) for 7B and 58.9% (+5.8%) for 14B.The 14B result surpasses DeepCoder-14B at 57.9%, and gains occur across coding topics.
- Results: Math-only SFT can yield poor performance on code benchmarks, contrasting with the cross-domain coding gains observed after math-only RL.The table caption explicitly frames this contrast.
- Data curation: Code data spans competitive-programming formats and topics, while filtering incompatible problems and adding edge-case test cases to reduce false rewards.The verifier grants positive reward only when generated code passes all test cases.
- Training process: The pipeline applies math-only RL before code-only RL, with stage-specific difficulty ranges, response lengths, temperatures, and rollout counts.The curriculum is designed for models of different scales and continues until policy convergence.
4. Evaluation
Evaluation shows that sequential math-only and code-only RL substantially improves reasoning across math and coding benchmarks, with gains extending to harder previously unsolved problems. Ablations further indicate that response-length expansion, math-RL initialization, and reliable verification shape training effectiveness.
- Main Results: RL improves both math and coding performance over the initial distilled SFT models at 7B and 14B scales.The 7B model gains 14.5% on AIME 2024, 14.6% on AIME 2025, 14.2% on LiveCodeBench v5, and 8% on v6; the 14B model also surpasses larger distilled SFT models.
- Main Results: 14B accuracy rises from 69.7%/50.2% to 78.6%/67.4% on AIME24/25 and from 53.1%/47.9% to 61.1%/54.9% on LiveCodeBench v5/v6.These results surpass significantly larger DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B models.
- Training Dynamics: Within 500 RL steps, extending responses to 16K increases average length from 4K to around 6.5K tokens while AIME24 accuracy rises from 55% to 62%.Both response length and accuracy then plateau; extending to 24K and 32K shows a similar trend.
- Sequential RL: Initializing code-only RL from math-RL checkpoints yields a substantially better starting point and significantly higher final LiveCodeBench accuracy.Sequential training therefore preserves the cross-domain benefit of math-RL while enabling further coding gains.
- Cross-Domain Effects: Math-only RL improves coding performance across all problem topics, with especially strong gains on Math, Counting, and Combinatorics.The topic-level analysis compares the initial SFT models with models after math-only RL and after subsequent code-only RL.
- Hard Problems: RL extends problem-solving beyond the distilled model's prior boundary, adding 30 and 23 solvable coding problems on LiveCodeBench v5 and v6 and three AIME24 problems with zero baseline solve rate.The improvement includes both higher accuracy on easier problems and gains on challenging problems where the SFT model has lower than 20% accuracy.
5. Conclusion
Large-scale RL substantially improves reasoning in strong small- and mid-sized SFT models through staged math-then-code training. The approach supports verification-based RL and can extend performance to previously unsolvable problems.
- Large-scale RL substantially enhances reasoning capabilities in strong, small- and mid-sized SFT models.
- Math-only RL precedes code-only RL, with code training further improving code benchmarks while minimally affecting math results.
- A robust curation pipeline collects challenging prompts, verifiable answers, and test cases for verification-based RL across math and code.
- RL both elicits reasoning acquired during pretraining and SFT and pushes models toward solving previously unsolvable problems.
Math
Math prompts ask the model to reason step by step and place its final answer inside boxed formatting.
- Math solutions should include step-by-step reasoning and place the final answer within boxed formatting.
No Starter Code (Python)
Python problems without starter code require a complete solution formatted as Python code.
- Write Python code to solve the problem and place the complete solution in the specified Python code-block format.
Has Starter Code
Python problems with starter code require solving from the provided function header and returning code in the specified format.
- Begin the solution from the provided function header.
- Place the resulting Python solution in the specified Python code-block format.
A.2. Example of AceReason-Nemotron-14B response to simple query
The example response interprets the query as counting the three “r” letters in “strawberry,” then computes that three strawberries provide nine “r”s. The surrounding evaluation material reports improved AIME accuracy during math RL and compares AceReason with DeepSeek-R1-Distill.
- The conversational response to “hi” is friendly and open-ended, asking how assistance can be provided.
- The model interprets “r’s” as the letter “r” and counts three occurrences in “strawberry.”
- Three strawberries are calculated as sufficient because each contributes three “r”s toward the requested nine.
- Stage 2 (16K) math RL produces a large AIME2025 performance improvement within 300 steps.
- AceReason’s AIME2024 and AIME2025 accuracy distributions improve over the distilled SFT model across 64 generations.
- On AIME24 and AIME25, the AceReason-14B model solves one additional problem, with larger gains on problems having higher initial accuracy.