Source-linked AI summary
Llama-Nemotron: Efficient Reasoning Models
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Prasoon Varshney, Makesh Narsimhan, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi Mahabadi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Shaona Ghosh, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Chris Alexiuk, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung
TL;DR
Reasoning models need strong capabilities without excessive inference cost, while users need control over when detailed reasoning is used. Llama-Nemotron combines architecture search, staged distillation and post-training, and a dynamic reasoning toggle across three open model sizes. The paper reports that supervised fine-tuning transfers reasoning from strong teachers, whereas curriculum-driven reinforcement learning with verifiable rewards is needed to surpass teacher-level capabilities.
Problem
Reasoning increasingly depends on inference-time scaling, making inference efficiency and user control over reasoning behavior important design requirements.
Method
Llama-Nemotron uses Puzzle-based neural architecture search, recovery training, supervised fine-tuning, large-scale reinforcement learning, and final alignment across Nano, Super, and Ultra models.
Results
High-quality synthetic data from strong reasoning teachers effectively adds reasoning to smaller models, while curriculum-driven reinforcement learning with verifiable rewards is necessary to exceed teacher-model capabilities.
Takeaways & Limitations
The released models combine competitive reasoning with low memory requirements, efficient inference, an open license, and a dynamic reasoning toggle.
Takeaways & Limitations
LN-Super was identified for a future model refresh.
Abstract
from arXiv · showhide
We introduce the Llama-Nemotron series of models, an open family of heterogeneous reasoning models that deliver exceptional reasoning capabilities, inference efficiency, and an open license for enterprise use. The family comes in three sizes -- Nano (8B), Super (49B), and Ultra (253B) -- and performs competitively with state-of-the-art reasoning models such as DeepSeek-R1 while offering superior inference throughput and memory efficiency. In this report, we discuss the training procedure for these models, which entails using neural architecture search from Llama 3 models for accelerated inference, knowledge distillation, and continued pretraining, followed by a reasoning-focused post-training stage consisting of two main parts: supervised fine-tuning and large scale reinforcement learning. Llama-Nemotron models are the first open-source models to support a dynamic reasoning toggle, allowing users to switch between standard chat and reasoning modes during inference. To further support open research and facilitate model development, we provide the following resources: 1. We release the Llama-Nemotron reasoning models -- LN-Nano, LN-Super, and LN-Ultra -- under the commercially permissive NVIDIA Open Model License Agreement. 2. We release the complete post-training dataset: Llama-Nemotron-Post-Training-Dataset. 3. We also release our training codebases: NeMo, NeMo-Aligner, and Megatron-LM.
NVIDIA
The Llama-Nemotron release includes Nano, Super, and Ultra model variants plus a post-training dataset, with LN-Ultra presented as highly capable among open models.
- The released model family includes Nano-8B, Super-49B, Ultra-253B, and an Ultra continued-pretraining variant.
- The release also includes the Llama-Nemotron-Post-Training-Dataset.
- As of April 2025, Artificial Analysis identifies LN-Ultra as the most intelligent open model.
- LN-Ultra is described as delivering leading performance among open models across reasoning and non-reasoning benchmarks.
1. Introduction
Reasoning models improve difficult-task performance through long, inference-time reasoning, creating a need for efficient deployment and user control over reasoning behavior. Llama-Nemotron addresses these goals through heterogeneous model optimization, a dynamic reasoning toggle, staged training, and an open post-training dataset.
- Motivation: Long responses containing chains of thought, self-verification, reflection, and backtracking enable strong performance on difficult STEM and mathematics tasks.
- Motivation: Inference efficiency is a core design objective because reasoning increasingly relies on inference-time scaling and can constrain model intelligence and agentic pipelines.
- Contributions: The family comprises LN-Nano (8B), LN-Super (49B), and LN-Ultra (253B), with LN-Ultra reported to outperform DeepSeek-R1 while fitting on a single 8xH100 node.
- Contributions: Each model supports a lightweight system-prompt toggle between standard chat and reasoning modes during inference.
- Training: The five-stage training procedure combines NAS and FFN Fusion, recovery training, supervised fine-tuning, large-scale reinforcement learning, and final alignment.
- Resources: The released post-training dataset targets mathematical reasoning, coding, science, and instruction following using filtered synthetic responses from open-source models.
- Reported result: Artificial Analysis identifies LN-Ultra as the most intelligent open-sourced model as of April 2025.
2. Creating Inference-Optimized Models
Puzzle creates hardware-efficient Llama-derived variants by searching block-level architecture choices under deployment constraints, followed by distillation and continued pretraining. The resulting models target substantial throughput, latency, memory, and accuracy-efficiency gains.
- Puzzle framework: Puzzle uses block-wise local distillation to build alternative transformer blocks with explicit accuracy-efficiency tradeoffs for LN-Super and LN-Ultra.
- Puzzle framework: The framework includes attention removal and variable FFN dimensions, with compression levels ranging from 87% to 10% of the original hidden size.
- Puzzle framework: Empirical evaluation found attention removal and FFN compression most effective for improving throughput and memory savings.
- Puzzle framework: A mixed-integer programming solver selects one block per layer under hardware, latency, memory, or throughput constraints.
- Puzzle framework: FFN Fusion replaces consecutive FFN blocks with fewer, wider layers that can execute in parallel, reducing sequential depth and improving latency.
- Deployment targets: 5× throughput speedup over Llama 3.3-70B-Instruct is achieved by LN-Super at batch size 256 and TP1 on one H100 GPU.
- Deployment targets: 1.71× latency improvement is achieved by LN-Ultra after FFN Fusion, with support for up to 3M cached tokens at FP8 on an H100 node.
- Deployment targets: LN-Ultra consistently outperforms DeepSeek-R1 and Llama-3.1-405B in GPQA-Diamond accuracy and throughput across both evaluated settings.
3. Synthetic Data
The synthetic-data pipeline combines curated real and generated questions with teacher-produced reasoning traces, filtering, decontamination, and difficulty-oriented scaling across reasoning domains.
- Code Reasoning Data: The code dataset contains approximately 488K Python samples after reasoning-trace checks, code extraction, filtering, and syntax validation.It aggregates 28,904 unique competitive-programming questions before solution generation and refinement.
- Data Scaling Insights: Scaling code data from 25k to 736k samples produced continuous benchmark gains, with harder CodeContests problems providing the largest boosts.The scaling curve did not plateau, emphasizing large, diverse, challenging data.
- Data Sources: The dataset combines open-ended and multiple-choice questions from in-house and external sources, including StackOverflow-derived pairs and synthetic MCQs.
- Synthetic Question Generation: Synthetic questions span academic topics and difficulty levels, with Qwen2.5 generating and varying multiple-choice questions after format verification.Nemotron-4-340B-Instruct defines topics and subtopics; the pipeline follows OpenMathInstruct-2 for question variation.
- Benchmark Decontamination: Questions are decontaminated against major science benchmarks, including GPQA, MMLU, and MMLU-Pro, to support fair evaluation.
- Solution Generation: DeepSeek-R1 generates multiple reasoning traces, while majority voting infers likely answers when ground truth is unavailable.
4. Supervised Fine-Tuning
Supervised fine-tuning transfers teacher-model reasoning into Llama-Nemotron models while jointly supporting reasoning control and standard instruction following.
- General Methodology: Reasoning SFT uses large-scale, high-quality traces from teachers such as DeepSeek-R1 to elicit robust downstream reasoning abilities.
- General Methodology: Training mixes reasoning and non-reasoning examples conditioned on “detailed thinking on/off” system instructions.
- General Methodology: Higher learning rates are required to learn effectively from long reasoning traces, partly because token-loss averaging depends on sequence length.
- LN-Nano: LN-Nano uses three SFT stages: reasoning-only training, mixed reasoning-control data, and a final chat, instruction-following, and tool-calling blend.The first stage uses a learning rate of 1e−4 for four epochs and 32k-token effective sequences.
- LN-Super: LN-Super trains for one epoch on the full SFT dataset, while smaller runs suggested gains through 3–4 epochs with larger learning rates.Rejection fine-tuning was omitted because it yielded no gains in these experiments.
- LN-Ultra: LN-Ultra uses packed 24k-token sequences and warmup to mitigate instability from high learning rates, including gradient explosions.
5. RL for Reasoning
Reasoning RL is applied exclusively to LN-Ultra to extend capabilities beyond distillation, using GRPO, verifiable rewards, curriculum data, and optimized distributed infrastructure.
- Training Procedure: LN-Ultra receives reasoning RL because distillation lets it approach but not exceed DeepSeek-R1, whereas RL enables exploration and self-learning.The authors report that smaller models performed worse with RL than distillation and therefore were excluded.
- Training Procedure: GRPO trains LN-Ultra with 16 sampled responses per prompt, global batch size 576, and two gradient updates per rollout until reasoning-task convergence.
- Reward Design: The RL phase uses accuracy rewards judged against ground truth and format rewards enforcing thinking tags in reasoning mode and their absence otherwise.
- Curriculum Training: Training data is filtered by LN-Super pass rate, discarding prompts solved at rates of 0.75 or higher, and curriculum batching increases difficulty progressively.
- Infrastructure Optimization: FP8 generation reaches 32 tokens/s/GPU/prompt, delivering a 1.8x speedup over BF16 through direct acceleration and reduced memory usage enabling cudagraphs.The reported contributions are 1.4x from FP8 generation and an additional 0.4x from memory reduction.
6. RL for Preference Optimization
Preference-optimization RL improves instruction following, helpfulness, and chat quality for the Llama-Nemotron models while retaining broader reasoning capabilities.
- Instruction Following: Short RLOO training runs for LN-Super and LN-Ultra improve conventional instruction-following and reasoning benchmarks.The runs use synthetic prompts containing one to ten detailed instructions and fewer than 120 steps.
- General Helpfulness and Chat: LN-Super reaches an Arena Hard score of 88.3, surpassing the listed proprietary and larger open-model baselines.
- General Helpfulness and Chat: Two online RPO iterations increase LN-Super’s Arena Hard score from 69.1 to 88.1 using a reward model trained on HelpSteer2 prompts.
- LN-Ultra: LN-Ultra uses GRPO with eight sampled responses per prompt for a 30-step preference-optimization run.
- LN-Nano: LN-Nano uses two offline RPO rounds with on-policy data to improve reasoning control and instruction following.
7. Evaluations on Reasoning and Chat Benchmarks
The evaluation covers reasoning and non-reasoning benchmarks across Llama-Nemotron model sizes and reasoning modes. Results show strong performance, while revealing trade-offs between reasoning, instruction following, conversationality, and subsequent reinforcement-learning outcomes.
- Evaluation setup: Evaluations include AIME24, AIME25, GPQA-Diamond, LiveCodeBench, and MATH500 for reasoning, plus IFEval, BFCL V2 Live, and Arena-Hard for non-reasoning capabilities.LN-Nano uses AIME25-I, while LN-Super and LN-Ultra use all 30 AIME25 problems.
- Evaluation setup: Evaluations use 32k context, reasoning-specific sampling, up to 16 completions per prompt, and average pass@1 accuracy.Reasoning-heavy benchmarks can show high variance because of small datasets and generation randomness.
- LN-Nano: LN-Nano achieves strong reasoning performance despite its small size, with balanced math, coding, and STEM data supporting broad capability transfer.Early data-distribution choices particularly affected chemistry-related GPQA performance.
- LN-Super: LN-Super performs competitively across reasoning and non-reasoning tasks, matching Llama-3.3-70B in reasoning-off mode and outperforming DeepSeek-R1-Distilled-Llama-70B in reasoning-on mode.Reasoning-focused SFT lowers IFEval scores, prompting a dedicated IFEval RL run; instruction following and conversationality also trade off against each other.
- LN-Ultra: LN-Ultra matches or outperforms existing open-weight models across reasoning and non-reasoning benchmarks and achieves state-of-the-art GPQA performance among open models.It is optimized for a single 8×H100 node, unlike DeepSeek-R1, which requires 8×H200.
- LN-Ultra: SFT provides a reasoning foundation, while large-scale RL is critical for surpassing DeepSeek-R1, particularly on GPQA.A higher-scoring SFT checkpoint did not yield the best RL outcome; RL was initialized from an earlier checkpoint.
8. Evaluations on Judging Capability
The models are also evaluated on JudgeBench, an out-of-distribution task that tests whether systems distinguish high-quality from low-quality responses. The reported results show strong performance against proprietary and open-source models.
- JudgeBench: JudgeBench evaluates whether models can differentiate between high-quality and low-quality responses.It assesses an out-of-distribution judging capability beyond the models’ trained reasoning and chat tasks.
- Results: Llama-Nemotron models outperform top proprietary and open-source models on JudgeBench, with LN-Ultra the best open-source model.
- Scope: The JudgeBench evaluation extends assessment beyond reasoning and conversational performance to response-quality judgment.
9. Conclusions
The paper presents Llama-Nemotron as an openly released model family combining reasoning performance with efficient inference. Its conclusions emphasize teacher-based SFT, curriculum-driven RL, and a multi-stage post-training pipeline.
- Release: The Llama-Nemotron models are released under a permissive license with open-source weights, training data, and code.
- Overall capability: The model family performs competitively with state-of-the-art reasoning models while offering low memory requirements and efficient inference capabilities.
- Reasoning transfer: Supervised fine-tuning on high-quality synthetic teacher data effectively adds reasoning capabilities to smaller models.
- Reinforcement learning: Large-scale, curriculum-driven reinforcement learning from verifiable rewards is necessary to push reasoning beyond the teacher model’s capabilities.
- Post-training: Producing an all-around model requires several stages in the post-training pipeline.
Post-training
The supplied passages list the paper’s authors, including contributors associated with the Llama-Nemotron work. No post-training procedure or result is described in these passages.
- Authors: Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, and Ran El-Yaniv are listed among the authors.
- Authors: Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, and Gerald Shen are listed among the authors.
- Authors: The author list also includes Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, and Zhilin Wang.
Evaluations and Safety
The supplied passages identify contributors associated with the paper, but provide no substantive evaluation or safety findings.
- Soumye Singhal and colleagues are listed among the paper’s contributors.
- Joey Conway and colleagues are listed among the paper’s contributors.
- Tugrul Konuk and colleagues are listed among the paper’s contributors.