PaperlayerView exact revision on arXiv

Source-linked AI summary

NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model

NVIDIA, :, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, Alex Kondratenko, Alex Shaposhnikov, Alexander Bukharin, Ali Taghibakhshi, Amelia Barton, Ameya Sunil Mahabaleshwarkar, Amy Shen, Andrew Tao, Ann Guan, Anna Shors, Anubhav Mandarwal, Arham Mehta, Arun Venkatesan, Ashton Sharabiani, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Banghua Zhu, Barnaby Simkin, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Brandon Norick, Brian Yu, Bryan Catanzaro, Charles Wang, Charlie Truong, Chetan Mungekar, Chintan Patel, Chris Alexiuk, Christian Munley, Christopher Parisien, Dan Su, Daniel Afrimi, Daniel Korzekwa, Daniel Rohrer, Daria Gitman, David Mosallanezhad, Deepak Narayanan, Dima Rekesh, Dina Yared, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Eileen Long, Elliott Ning, Eric Chung, Erick Galinkin, Evelina Bakhturina, Gargi Prasad, Gerald Shen, Haifeng Qian, Haim Elisha, Harsh Sharma, Hayley Ross, Helen Ngo, Herman Sahota, Hexin Wang, Hoo Chang Shin, Hua Huang, Iain Cunningham, Igor Gitman, Ivan Moshkov, Jaehun Jung, Jan Kautz, Jane Polak Scowcroft, Jared Casper, Jian Zhang, Jiaqi Zeng, Jimmy Zhang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jonathan Cohen, Joseph Jennings, Julien Veron Vialard, Junkeun Yi, Jupinder Parmar, Kari Briski, Katherine Cheung, Katherine Luna, Keith Wyss, Keshav Santhanam, Kezhi Kong, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Kushan Ahmadian, Lawrence McAfee, Laya Sleiman, Leon Derczynski, Luis Vega, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Mark Cai, Markus Kliegl, Marta Stepniewska-Dziubinska, Matvei Novikov, Mehrzad Samadi, Meredith Price, Meriem Boubdir, Michael Boone, Michael Evans, Michal Bien, Michal Zawalski, Miguel Martinez, Mike Chrzanowski, Mohammad Shoeybi, Mostofa Patwary, Namit Dhameja, Nave Assaf, Negar Habibi, Nidhi Bhatia, Nikki Pope, Nima Tajbakhsh, Nirmal Kumar Juluru, Oleg Rybakov, Oleksii Hrinchuk, Oleksii Kuchaiev, Oluwatobi Olabiyi, Pablo Ribalta, Padmavathy Subramanian, Parth Chadha, Pavlo Molchanov, Peter Dykas, Peter Jin, Piotr Bialecki, Piotr Januszewski, Pradeep Thalasta, Prashant Gaikwad, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi Mahabadi, Rajen Patel, Ran El-Yaniv, Ranjit Rajan, Ria Cheruvu, Rima Shahbazyan, Ritika Borkar, Ritu Gala, Roger Waleffe, Ruoxi Zhang, Russell J. Hewett, Ryan Prenger, Sahil Jain, Samuel Kriman, Sanjeev Satheesh, Saori Kaji, Sarah Yurick, Saurav Muralidharan, Sean Narenthiran, Seonmyeong Bak, Sepehr Sameni, Seungju Han, Shanmugam Ramasamy, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shelby Thomas, Shizhe Diao, Shreya Gopal, Shrimai Prabhumoye, Shubham Toshniwal, Shuoyang Ding, Siddharth Singh, Siddhartha Jain, Somshubra Majumdar, Soumye Singhal, Stefania Alborghetti, Syeda Nahida Akter, Terry Kong, Tim Moon, Tomasz Hliwiak, Tomer Asida, Tony Wang, Tugrul Konuk, Twinkle Vashishth, Tyler Poon, Udi Karpas, Vahid Noroozi, Venkat Srinivasan, Vijay Korthikanti, Vikram Fugro, Vineeth Kalluru, Vitaly Kurin, Vitaly Lavrukhin, Wasi Uddin Ahmad, Wei Du, Wonmin Byeon, Ximing Lu, Xin Dong, Yashaswi Karnati, Yejin Choi, Yian Zhang, Ying Lin, Yonggan Fu, Yoshi Suhara, Zhen Dong, Zhiyu Li, Zhongbo Zhu, Zijia Chen

arXiv:2508.14444v4cs.CLcs.AIcs.LG

TL;DR

The paper addresses the challenge of improving reasoning-model throughput without sacrificing benchmark accuracy. It develops Nemotron-Nano-9B-v2 through large-scale pretraining, alignment, and Minitron-based compression, achieving comparable or better accuracy with substantially higher throughput and 128k-context inference on one A10G GPU.

  • Problem

    Reasoning workloads require long thinking traces, motivating a model that improves inference throughput while retaining strong benchmark accuracy.

  • Method

    The authors pretrain a 12B base model, align it with SFT, GRPO, DPO, and RLHF, then extend Minitron with pruning and distillation for reasoning-model compression.

  • Results

    Nemotron-Nano-9B-v2 achieves comparable or better accuracies at up to 6× higher throughput than Qwen3-8B and supports up to 128k-token inference on a single NVIDIA A10G GPU.

  • Takeaways & Limitations

    The released hybrid model targets efficient long-context reasoning inference while providing checkpoints, recipes, and most pre- and post-training datasets.

  • Takeaways & Limitations

    The compression target is constrained by a 19.66 GiB inference-memory budget for 128k sequence length and batch size at least 1 on an NVIDIA A10G.

Abstract

from arXiv · show

We introduce Nemotron-Nano-9B-v2, a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models. Nemotron-Nano-9B-v2 builds on the Nemotron-H architecture, in which the majority of the self-attention layers in the common Transformer architecture are replaced with Mamba-2 layers, to achieve improved inference speed when generating the long thinking traces needed for reasoning. We create Nemotron-Nano-9B-v2 by first pre-training a 12-billion-parameter model (Nemotron-Nano-12B-v2-Base) on 20 trillion tokens using an FP8 training recipe. After aligning Nemotron-Nano-12B-v2-Base, we employ the Minitron strategy to compress and distill the model with the goal of enabling inference on up to 128k tokens on a single NVIDIA A10G GPU (22GiB of memory, bfloat16 precision). Compared to existing similarly-sized models (e.g., Qwen3-8B), we show that Nemotron-Nano-9B-v2 achieves on-par or better accuracy on reasoning benchmarks while achieving up to 6x higher inference throughput in reasoning settings like 8k input and 16k output tokens. We are releasing Nemotron-Nano-9B-v2, Nemotron-Nano12B-v2-Base, and Nemotron-Nano-9B-v2-Base checkpoints along with the majority of our pre- and post-training datasets on Hugging Face.

1. Introduction

Nemotron Nano 2 is a hybrid Mamba-Transformer reasoning model designed to improve throughput while maintaining comparable or better accuracy than similarly sized models. The report combines new training data and recipes with alignment, compression, and distillation, and releases models, datasets, and recipes.

  • Contribution: Nemotron Nano 2 achieves on-par or better benchmark accuracies at 3×–6× higher throughput than Qwen3-8B in generation-heavy scenarios.The evaluated settings include 1k input / 8k output and 8k input / 16k output tokens.
  • Pretraining: The 12B base model was pretrained on 20 trillion tokens with FP8 precision and extended to support 128k-token contexts without degrading other benchmarks.The pretraining used a Warmup-Stable-Decay learning-rate schedule.
  • Alignment: Post-training combined multiple SFT stages with GRPO, DPO, and RLHF across reasoning, tool use, long-context, and instruction-following domains.Roughly 90 billion post-training tokens were used, with about 5% containing deliberately truncated reasoning traces for inference-time budget control.
  • Compression: Minitron-based pruning and distillation compressed both base and aligned models to enable 128k-token inference on a single NVIDIA A10G GPU.The target setting uses 22 GiB of memory and bfloat16 precision.

2. Pretraining

This section introduces the Nemotron-Nano-12B-v2-Base architecture and frames its evaluation against other state-of-the-art models on popular benchmarks.

  • Scope: The section covers the Nemotron-Nano-12B-v2-Base architecture, pretraining, and accuracy comparisons with other state-of-the-art models.
  • Architecture: Nemotron-Nano-12B-v2-Base uses a mixture of Mamba-2, self-attention, and FFN layers.Its 62-layer configuration contains 6 self-attention, 28 FFN, and 28 Mamba-2 layers, with hidden dimension 5120 and FFN hidden dimension 20480.

2.2. Pre-Training Data

Nemotron-Nano-12B-v2-Base uses curated web, multilingual, mathematical, code, and academic data alongside synthetic question-answer and reasoning data. The report describes data pipelines designed to preserve structure, improve quality, and broaden task coverage.

  • Data coverage: Pretraining draws on curated and synthetically generated data spanning general web crawl, multilingual, math, code, and academic categories.Separate curation pipelines are used for general web, math, and code data.
  • Web and multilingual data: The English web pipeline adds recent Common Crawl and CC-NEWS data, synthetic rephrasing, and global fuzzy deduplication.The Common Crawl updates include eight snapshots from CC-MAIN-2024-33 through CC-MAIN-2025-13.
  • Web and multilingual data: The multilingual pipeline extracts fifteen languages from three Common Crawl snapshots and applies heuristic filtering.The languages include Arabic, Chinese, Danish, Dutch, French, German, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, and Thai.
  • Math data: A new Common Crawl math-extraction pipeline renders raw HTML with lynx and uses model-based processing to preserve mathematical structure.It produces a 133B-token corpus and a higher-quality 52B-token subset.
  • Math data: The resulting math datasets improve math, code, and general-domain evaluations and surpass existing open math datasets.Reported evaluations include MATH-500, HumanEval+, MBPP+, MMLU, MMLU-STEM, and MMLU-Pro.
  • Code data: Code data comes from GitHub source code processed through license filtering, deduplication, and other quality-control stages, with synthetic QA in 11 programming languages.The QA generation prompts an LLM to create and solve questions from curated code snippets, followed by heuristic filtering.
  • Reasoning data: Synthetic reasoning data expands LSAT, LogiQA, and AQuA-RAT material through iterative generation, filtering, solution creation, and multiple-choice conversion.Generation uses Qwen3, DeepSeek-R1, and DeepSeek-V3 models with similar, harder, and varied-question prompts.
  • Academic data: Synthetic academic QA targets undergraduate- and graduate-level technical documents to improve comprehension and knowledge storage or extraction.The targeted subjects include math, chemistry, biology, physics, and medicine; documents are chunked into 512-token snippets and embedded for retrieval.

2.3. Data Mixture and Ordering

Nemotron-Nano-12B-v2-Base uses a thirteen-category curriculum that shifts from diverse data toward higher-quality datasets across three pre-training phases. Ablations also evaluate multilingual sources and fundamental-reasoning SFT-style data for downstream performance.

  • The pre-training mixture contains thirteen categories, including web crawl, math, Wikipedia, code, academic, multilingual, and synthetic SFT-style data.
  • Training begins with a diversity-promoting blend, then switches to primarily high-quality datasets at 60% and 90% of training.
  • The multilingual-data ablation compares Common Crawl, FineWeb-2, DiverseQA-wiki, and DiverseQA-crawl using Global-MMLU.
  • Curated Common Crawl multilingual data slightly outperformed FineWeb-2, while synthesized multilingual QA pairs performed much better than curated alternatives.
  • Fundamental Reasoning SFT-Style Data Ablation Study: Replacing 5% of Common Crawl data with FR-SFT data increased MMLU-Pro from 44.24 to 56.36 and raised average MATH by around 2 points.

2.4. FP8 Recipe

The entire pre-training run uses DeepSeek’s FP8 recipe, with E4M3 tensors and specified quantization layouts while retaining selected boundary layers in BF16.

  • The entire pre-training run uses DeepSeek’s FP8 training recipe.
  • Weights use 128x128 quantization blocks, activations use 1x128 tiles, and model weights remain natively in E4M3.
  • The first and last four linear layers remain in BF16, while master weights are kept in FP32.

2.5. Hyperparameters

Nemotron-Nano-12B-v2-Base is trained for 20 trillion tokens with 8192-token sequences, a global batch of 768, and a Warmup-Stable-Decay learning-rate schedule.

  • The model is trained on a token horizon of 20 trillion tokens.
  • Training uses sequence length 8192 and global batch size 768, corresponding to 6,029,312 tokens per batch.
  • The WSD schedule uses a stable learning rate of 4.5 · 10−4, decays during the final 3.6 trillion tokens, and reaches a minimum of 4.5 · 10−6.
  • Weight decay is 0.1, with Adam β1 set to 0.9 and β2 set to 0.95.

2.6. Long-Context Extension

A dedicated long-context phase extends the base model toward 128k-token inference using 512k-token training sequences and synthetic document-QA data. The blend preserves regular benchmark scores while extending context length.

  • The long-context phase uses 512k-token sequences, 8-way tensor parallelism, and 16-way context parallelism.
  • Phase LC contains 18.9 billion tokens and uses a global batch size of 12 to maintain roughly 6M tokens per batch.
  • Synthetic document-QA data is generated from documents longer than 32k tokens by sampling 1,024-token chunks and appending generated QA pairs.
  • The Phase LC blend downscales Phase 3 data to 80% and allocates 20% to long-context document-QA data.
  • The resulting blend extends context length without degrading regular benchmark scores.

2.7. Base Model Evaluations

The evaluation uses lm-evaluation-harness with task-specific protocols for mathematical and code reasoning. Accuracy results compare Nemotron-Nano-12B-v2-Base and its 9B pruned variant against Qwen3-8B Base and Gemma3-12B Base.

  • Evaluation setup: Evaluations build on lm-evaluation-harness with modifications for fair comparisons.The setup is described as the basis for the reported evaluations.
  • Evaluation setup: Mathematical reasoning uses greedy decoding on GSM8K, MATH, and MATH Level 5, plus pass@32 on AIME-2024.Math-Verify grades all generations.
  • Evaluation setup: Code evaluation covers the EvalPlus variants of HumanEval and MBPP with generation sanitization.The supplied passage introduces this code-task protocol but ends before its full description.
  • Results: Accuracy tables compare Nemotron-Nano-12B-v2-Base and its 9B pruned variant with Qwen3-8B Base and Gemma3-12B Base.The 9B pruned variant is discussed elsewhere in the paper.

3. Alignment

The alignment pipeline combines large-scale supervised fine-tuning across domains with targeted stages for tool use, long context, truncated reasoning, instruction following, and tool-calling optimization. It also includes multilingual, conversational, and safety data, while truncation training improves behavior under constrained thinking budgets.

  • Alignment overview: The alignment process converts the base checkpoint into an aligned 12B checkpoint through procedures summarized in Figure 4.Figure 4 presents the flow of the alignment procedures leading to the final Merged checkpoint.
  • Data and SFT: Large-scale SFT trains on approximately 80 billion prompt-response tokens spanning math, science, coding, tool calling, multilingual, conversational, and safety data.The domain distribution is reported in Table 7.
  • Data and SFT: Safety data combines harmful and benign prompts, with generated responses filtered by guard models after initial prompting.The sources include Nemotron Content Safety Dataset V2, HarmfulTasks, RedTeam2K, and gretel-v1.
  • SFT stages: Stage 1 SFT adds reasoning-free responses and concatenates samples into approximately 128k-token sequences to support direct answering and long-context ability.The reasoning-free examples expose the model to an answer mode without thinking traces.
  • SFT stages: Stage 2 removes concatenation to recover tool-calling patterns, while Stage 3 adds long-context data and abruptly truncated 1–2k-token reasoning traces.The truncation strategy preserves final answers and improves robustness across inference-time thinking budgets.
  • Targeted optimization: IFEval RL improves instruction adherence, whereas DPO uses on-policy WorkBench successes and failures to strengthen multi-step and multi-turn tool calling.The BFCL v3 benchmark evaluates the expanded tool-calling capabilities, while checkpoint selection remains important after IFEval RL.
  • Budget control: With truncation training, the model avoids compensatory final-answer expansion and consistently produces well-formed responses under short thinking budgets.Well-formedness requires only a single closing </think> tag; without truncation training, short budgets cause sharp declines.

4. Pruning and Distillation

The paper compresses the aligned 12B model through importance-guided pruning, candidate search, and distillation to meet long-context memory and throughput targets. The resulting 9B model enables 128k-token inference on an A10G GPU while improving throughput and preserving benchmark accuracy.

  • Compression is required because storing the 12B model’s weights alone needs 22.9 GiB, exceeding the A10G’s 22 GiB memory.
  • The method extends Minitron to compress reasoning models under memory and throughput constraints.The framework combines lightweight pruning with objectives tailored to the Nano 2 deployment setting.
  • 4.1. Importance Estimation: Importance estimation uses lightweight forward-pass calibration rather than gradients to rank layers, FFN neurons, embedding channels, and Mamba heads.The calibration procedures use 1024 samples and activation-based scores for several component types.
  • 4.1. Importance Estimation: Layers are pruned iteratively by temporarily removing each candidate and discarding the one with the lowest logits MSE.Lower MSE indicates a smaller effect on the original model’s predictions.
  • 4.1. Importance Estimation: Mamba heads are ranked within groups and pruned across affected projections, convolution, and SSM matrices to preserve structural computation.The paper focuses on head pruning because prior work found a better accuracy–throughput trade-off than pruning head channels.
  • 4.2. Architecture Search: The search combines depth and width pruning within a 19.66 GiB budget, evaluates top candidates with 19B-token distillation, and benchmarks throughput.Candidate axes include removing 6–10 layers, reducing embedding channels and FFN dimensions, and varying Mamba heads.
  • 4.4. Results: The final model retains 56 layers, uses 4480 embedding channels and a 15680-dimensional FFN, and delivers 3×–6× higher throughput than Qwen3-8B.It enables long-context inference on an NVIDIA A10G while surpassing Qwen3-8B in accuracy and remaining comparable to the 12B teacher on most benchmarks.

5. Conclusion

Nemotron-Nano-9B-v2 combines a hybrid Mamba-Transformer architecture with staged training and Minitron-based compression, achieving comparable or better accuracy at up to 6× higher throughput than Qwen3-8B. Compression enables 128k-token inference on a single NVIDIA A10G GPU, and the models and most training data are open-sourced.

  • Up to 6× higher throughput accompanies comparable or better accuracy than Qwen3-8B.
  • Nemotron-Nano-9B-v2 was produced through pre-training, multi-stage alignment, and Minitron compression using pruning and distillation.
  • 128k-token inference is supported in bfloat16 precision on one NVIDIA A10G GPU with 22 GiB of memory.
  • The release includes Nemotron-Nano-9B-v2, sibling and parent checkpoints, and most pre- and post-training data on Hugging Face.

A. Permissive Source Code Licenses

This section identifies source-code licenses accepted for removal decisions and associates each license with a ScanCode dataset key. The listed licenses span multiple software and documentation licensing families.

  • Source code with licenses outside the specified accepted list is removed.
  • The list covers additional permissive, public, software, font, hardware, and documentation license entries.
  • Further entries include wxWidgets, X11 Documentation, X11, and numerous X11-Style licenses.
  • The license registry includes named licenses paired with bracketed keys, such as X11-Style variants and ZLIB licenses.
  • The short names and keys come from the ScanCode license dataset.
Loading 2508.14444v4…