Source-linked AI summary
Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision
Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Armstrong, Stephen Ge, Edgar Minasyan, Feng Chen, Igor Gitman
TL;DR
Existing mathematical reasoning datasets provide limited diversity in reasoning styles, depths, tool use, and problem domains. Nemotron-Math constructs a 7.5M-trace corpus from multi-mode, Python-augmented generation over AoPS and StackExchange-Math problems, and supports efficient long-context fine-tuning. It reports stronger supervision and broad benchmark performance, including 100% maj@16 accuracy on AIME24/25 with high reasoning and Python TIR.
Problem
Existing mathematical reasoning datasets only partially capture diverse reasoning styles, depths, tool usage, and mathematical domains.
Method
Nemotron-Math combines 85K AoPS and 262K StackExchange-Math problems with six gpt-oss-120b generation configurations spanning reasoning modes and Python TIR.
Results
Nemotron-Math provides higher-quality supervision, improves competition and open-domain reasoning, and enables 2–3× faster 128K-context fine-tuning with 1–3% accuracy difference from joint training.
Takeaways & Limitations
Both Qwen3-8B and Qwen3-30B-A3B achieve 100% maj@16 accuracy on AIME24/25 with high reasoning and Python TIR.
Takeaways & Limitations
Sequential bucketed training can imbalance reasoning modes at long context lengths, potentially collapsing behavior toward uniformly long, high-depth reasoning.
Abstract
from arXiv · showhide
High-quality mathematical reasoning supervision requires diverse reasoning styles, long-form traces, and effective tool integration, capabilities that existing datasets provide only in limited form. Leveraging the multi-mode generation ability of gpt-oss-120b, we introduce Nemotron-Math, a large-scale mathematical reasoning dataset containing 7.5M solution traces across high, medium, and low reasoning modes, each available both with and without Python tool-integrated reasoning (TIR). The dataset integrates 85K curated AoPS problems with 262K community-sourced StackExchange-Math problems, combining structured competition tasks with diverse real-world mathematical queries. We conduct controlled evaluations to assess the dataset quality. Nemotron-Math consistently outperforms the original OpenMathReasoning on matched AoPS problems. Incorporating StackExchange-Math substantially improves robustness and generalization, especially on HLE-Math, while preserving accuracy on math competition benchmarks. To support efficient long-context training, we develop a sequential bucketed strategy that accelerates 128K context-length fine-tuning by 2--3$\times$ without significant accuracy loss. Overall, Nemotron-Math enables state-of-the-art performance, including 100\% maj@16 accuracy on AIME 2024 and 2025 with Python TIR.
1. Introduction
Existing mathematical reasoning datasets often provide limited variation in reasoning depth, style, tool use, and domain coverage. Nemotron-Math addresses these gaps with diverse long-form traces, broader problem sources, and efficient long-context training.
- Motivation: Existing datasets often emphasize competition-style problems and single-mode generation, limiting coverage of broader mathematical reasoning behaviors.These limitations concern reasoning-style diversity, reasoning depth, tool usage, and mathematical-domain breadth.
- Motivation: gpt-oss-120b enables dataset construction with controllable reasoning depth and detailed Python-integrated reasoning traces.Its high, medium, and low modes produce solutions of varying depth and length.
- Dataset construction: The dataset combines structured AoPS problems with diverse community-sourced StackExchange-Math questions to broaden domain coverage.Filtering reduces the retained collections to 85K AoPS problems and 262K StackExchange-Math problems.
- Dataset and motivation: Nemotron-Math contains 7.5M long-form traces from three reasoning modes, with and without Python TIR, across 347K curated problems.The corpus captures diverse reasoning depths, self-verification styles, and tool-usage behaviors, with traces up to 128K tokens.
- Long-context training: Sequential bucketed training progressively expands context from 16K to 128K tokens, improving throughput while introducing only minor accuracy trade-offs.The strategy groups samples by sequence length and optimizes parallelism configurations at each scale.
- Results: Nemotron-Math improves AIME25 pass@1 for Qwen3-30B-A3B by 13.1% over the baseline without Python TIR.With high reasoning and Python TIR, both evaluated Qwen3 models achieve 100% maj@16 accuracy on AIME24 and AIME25.
2. Nemotron-Math Overview
Nemotron-Math combines filtered AoPS and StackExchange-Math problems with multi-mode, tool-augmented solution generation. The resulting corpus contains 7.5M correctness-filtered traces with varied lengths and reasoning configurations.
- Problem sources: Nemotron-Math is built from two complementary sources: curated AoPS problems for mathematical rigor and StackExchange-Math questions for real-world diversity.The source design balances structured competition-style tasks with college-level and research-oriented questions.
- Solution generation: Each problem receives solutions in high, medium, and low reasoning modes, both with and without Python TIR, yielding six generation configurations.Eight solutions are generated per configuration, and high-mode solutions help verify or replace extracted reference answers.
- Problem filtering: The final problem pool contains 85K AoPS problems and 262K StackExchange-Math problems after removing proof-style, trivial, and otherwise unsuitable items.The source collections were reduced from 175K to 85K and from 651K to 262K, respectively.
- Solutions analysis: The corpus contains 7.5M generated solutions whose Python TIR variants are consistently larger and whose total size increases from low to high reasoning mode.These patterns follow correctness filtering, which removes solutions that fail to reach the expected answer.
- Solutions analysis: Most Nemotron-Math traces are short or medium length, with relatively few exceeding 64K tokens.Table 2 reports the distribution of traces across token-length buckets for both problem sources.
3. Experiments Setup
The experiments fine-tune two Qwen3 models with a consistent supervised training pipeline and evaluate them across competition and open-domain mathematical benchmarks. Evaluation varies reasoning mode, Python TIR, decoding, and answer-verification procedures.
- Training setup: Experiments fine-tune Qwen3-8B and Qwen3-30B-A3B using AdamW, a 2e-4 learning rate, global batch size 2048, and sequence packing.The pipeline uses no warmup and applies sequence packing for long-context reasoning data.
- Evaluation benchmarks: Models are evaluated on Comp-Math-24-25 and HLE-Math, covering competition-style and open-domain advanced mathematical problem solving.Comp-Math-24-25 includes HMMT-24-25, AIME24, and AIME25; HLE-Math uses its text-only Math subset.
- Evaluation protocol: Each final checkpoint is tested under six combinations of high, medium, or low reasoning mode and Python TIR or no Python TIR.Competition benchmarks use 16 solutions per problem, while HLE-Math uses 4 solutions per problem.
- Evaluation protocol: Competition answers are checked with math-verify, whereas HLE-Math uses Qwen2.5-32B-Instruct as an LLM judge.Baseline Qwen3 models are evaluated with default decoding settings for comparison.
4. Experimental Results
Controlled experiments show that Nemotron-Math improves over OpenMathReasoning, StackExchange-Math improves HLE-Math robustness without harming competition performance, and sequential bucketing preserves accuracy while reducing training cost.
- Dataset comparison: Nemotron-Math models consistently outperform OpenMathReasoning and the Mixed dataset across HMMT-24-25, AIME24, and AIME25.The comparison uses matched AoPS problems and evaluates pass@1 and maj@16.
- Effect of StackExchange-Math Integration: StackExchange-Math improves HLE-Math performance across all six reasoning configurations while maintaining comparable or slightly higher competition accuracy.The six configurations combine high, medium, and low reasoning modes with and without Python TIR.
- Sequential Bucketed Training Strategy: Sequential bucketed training progresses from 16K to 128K contexts with length-specific parallelism and sequence packing.The strategy addresses the inefficiency of applying fixed 128K settings to predominantly shorter traces.
- Sequential Bucketed Training Strategy: High-reasoning Python TIR reaches 100% maj@16 accuracy on both AIME24 and AIME25.The result is reported for Qwen3-30B-A3B after fine-tuning.
- Scaling with Model size and Architecture: Qwen3-8B and Qwen3-30B-A3B show similar learning dynamics and nearly identical final accuracy across benchmarks and tool settings.The notable exception is slightly higher Qwen3-8B accuracy on HLE-Math without Python TIR.
5. Related Work
Prior mathematical reasoning datasets advanced long-form supervision and tool use, but often offered limited depth or behavioral diversity. Nemotron-Math extends this line with multi-mode, Python-augmented supervision across varied mathematical problems.
- Long Chain-of-Thought Reasoning Data: Earlier datasets scaled mathematical chain-of-thought supervision, but their solution traces remained relatively shallow.The related work includes OpenMathInstruct-1/2, Skywork-MathQA, and NuminaMath.
- Long Chain-of-Thought Reasoning Data: DeepSeek-R1 and Qwen3 enabled longer, more coherent, structured trajectories with self-reflection and consistency checking.OpenMathReasoning leveraged these capabilities for olympiad-level problem supervision.
- Tool-Integrated Reasoning: Tool-integrated reasoning lets models invoke Python or symbolic solvers for numeric precision, symbolic manipulation, and multi-step verification.Program of Thoughts and subsequent systems developed this direction.
- Nemotron-Math: Nemotron-Math differs from predominantly single-mode datasets by combining high, medium, and low reasoning modes with and without Python augmentation.This produces a broader range of reasoning styles and tool-usage behaviors.
6. Conclusion
Nemotron-Math combines 7.5M multi-mode, tool-augmented solution traces with broad problem coverage and supports efficient long-context training. Fine-tuned models achieve 100% maj@16 accuracy on AIME24/25 under high-reasoning Python TIR.
- 7.5M long-form traces generated across multi-mode and tool-augmented settings cover 347K curated mathematical problems.The dataset combines structured AoPS problems with diverse StackExchange-Math questions.
- Sequential bucketed training delivers 2–3× faster 128K-context fine-tuning with only a 1–3% accuracy difference from full joint training.
- 100% maj@16 accuracy is achieved on AIME24/25 by Qwen3-8B and Qwen3-30B-A3B under high-reasoning Python TIR.The reported result applies to both models.
A. Training Configuration By Bucket Length
The training setup adapts parallelism to sequence-length buckets, improving short-context efficiency while documenting evaluation configurations and metrics.
- A. Training Configuration By Bucket Length: Tensor parallelism splits layer matrix multiplications, while context parallelism shards the sequence dimension for long-context training.Pipeline parallelism is used only lightly in the described configuration.
- A. Training Configuration By Bucket Length: A 16K bucket runs at around 18 seconds per step versus approximately 25 seconds under the fixed 128K parallelism configuration for Qwen3-30B-A3B.The speed difference reflects bucket-specific versus fixed parallelism settings.
- A. Training Configuration By Bucket Length: Nemotron-3-Nano evaluation reports pass@1 and maj@k, with k=16 for Comp-Math-24-25 and k=4 for HLE-Math.
- A. Training Configuration By Bucket Length: Learning-rate comparisons use pass@1 accuracy across high, medium, and low reasoning modes with and without Python TIR.Results are reported as mean ± standard deviation over 16 runs.
B. Further Evidence via NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 Evaluation
A pre-reinforcement-learning Nemotron-3-Nano checkpoint trained only on high-reasoning Nemotron-Math trajectories performs comparably to fine-tuned Qwen3-30B-A3B on competition mathematics.
- B. Further Evidence via NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 Evaluation: Comparable Comp-Math-24-25 accuracy is achieved by the high-reasoning-only SFT checkpoint and fine-tuned Qwen3-30B-A3B.The checkpoint is evaluated on Comp-Math-24-25 and HLE-Math with and without Python TIR.
C. Learning Rate Grid Search
The learning-rate grid search identifies 2e-4 as the generally strongest setting across reasoning modes, which is then fixed for all experiments.
- C. Learning Rate Grid Search: 2e-4 generally performs best across different reasoning modes and is fixed for all experiments.The search uses Qwen3-30B-A3B on a subset of the training data.
D. Answer Judgment Prompt
The answer-judgment prompt asks a model to explain briefly whether predicted and expected math answers are equivalent, then return only Yes or No. Its examples cover algebraic equivalence, numeric representations, unordered solutions, missing solutions, and incorrect simplification.
- Judgment procedure: The prompt requires brief reasoning followed by only a Yes-or-No judgment about whether two math answers are equivalent.It limits comparisons to trivial simplifications.
- Accepted equivalences: Equivalent answers receive Yes when factor order differs, fractions and decimals match, or solution order is irrelevant.Examples include reordered factors, 3/2 versus 1.5, and reversed solution lists.
- Rejected mismatches: The examples reject answers when simplification is incorrect, the number of solutions differs, or the predicted expression does not match the expected choice.The prompt illustrates these cases with factorial simplification, a trigonometric solution set, and a radical expression.
- Input format: The template presents the problem, predicted answer, and expected answer before requesting the equivalence judgment.This structure is used for each evaluation instance.