Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

1. Introduction

• Nemotron 3 Super combines MoE and hybrid Mamba-Attention design in a 120-billion-parameter model with 12 billion active parameters.The introduction motivates MoE for accuracy at lower active parameter counts and hybrid Mamba-Attention for inference throughput.Source: §1 · p. 1
• LatentMoE improves accuracy per parameter and per FLOP, while MTP accelerates inference through speculative decoding and improves overall model quality.Nemotron 3 Super is the first model in the series to use LatentMoE and is pre-trained in NVFP4.Source: §1 · p. 1
• The model was pre-trained on 25 trillion tokens in two phases: 20 trillion emphasizing diversity and coverage, followed by 5 trillion emphasizing data quality and benchmark accuracy.The first phase comprised 80% of pre-training and the second 20%.Source: §1 · p. 1
• Expanded agentic training used broader RL environments, more agentic data, multi-step tool-use post-training, and resilient asynchronous infrastructure, improving over Nemotron 3 Nano.The reported improvements span software engineering, terminal use, and general tool-use benchmarks.Source: §1 · pp. 1–2
• Up to 2.2× and 7.5× higher inference throughput than GPT-OSS-120B and Qwen3.5-122B, respectively, with comparable benchmark accuracies at 8k-input/64k-output lengths.The comparison used B200 GPUs and the best throughput from vLLM or TRT-LLM for each model.Source: §1 · p. 1, Figure 1 · p. 2

Checkpoints

• The project publicly shares Nemotron 3 Super’s training recipe through the Nemotron Developer Repository.The introduction presents this release as part of the project’s open materials.Source: §1 · p. 2
• Released checkpoints include post-trained FP8, NVFP4, and BF16 models, plus a BF16 base model and the Qwen3-Nemotron-235B-A22B-GenRM-2603 reward model.The GenRM is used for RLHF.Source: Checkpoints · p. 2, Data · p. 2

Data

• Released data include specialized pre-training material, synthetic capability datasets, and post-training RL environments and SFT datasets targeting broad agentic capabilities.The report covers pre-training, post-training, and quantization in Sections 2–4.Source: Data · pp. 2–3

2. Pretraining

• Nemotron 3 Super 120B-A12B Base achieves superior accuracy to Ling-flash-Base-2.0 and GLM-4.5-Air-Base across a comprehensive suite of benchmarks.The pretraining section covers a 25-trillion-token corpus, hyperparameters, architecture, NVFP4 training, and long-context extension.Source: §2 · p. 3

2.1.1. LatentMoE: Hardware-Aware Expert Design for Improved Accuracy per Byte

• LatentMoE projects tokens into a smaller latent space, reducing routed loads and communication by d/ℓ while expanding experts and active experts at similar inference cost.The design targets accuracy per parameter and accuracy per FLOP by reducing the hidden dimension bottleneck while preserving the effective nonlinear budget.Source: §2.1.1 · pp. 4–5, Figure 3 · p. 5

2.1.2. Multi-Token Prediction

• MTP improves representation learning and enables native speculative decoding, while shared parameters across prediction heads support longer autoregressive drafts without an external draft model.The shared-head design makes acceptance degradation with draft length substantially milder than with independently trained offset heads.Source: §2.1.2 · p. 6
• 3.45 average accepted tokens per verification step gives Nemotron 3 Super the highest overall SPEED-Bench acceptance length, ahead of DeepSeek-R1 and competitive with Qwen3-Next.With draft length fixed at 7, acceptance remains higher than DeepSeek-R1 at every draft position and closely tracks or exceeds Qwen3-Next across most indices.Source: §2.1.2 · p. 6, Table 2 · p. 7, Figure 4 · p. 7
• Increasing MTP draft depth from D = 1 to D = 3 shifts the throughput–latency Pareto frontier toward higher aggregate output tokens per second than MTP-disabled inference.The comparison uses an NVFP4 checkpoint with TRT-LLM on a B300 GPU and the SPEED-Bench Throughput-1k split.Source: §2.1.2 · p. 6, Figure 5 · p. 7

2.1.3. Hybrid Interleaved MoE Architecture and Global Anchors

• The 88-layer hybrid stack predominantly uses Mamba-2 blocks with periodic sparse MoE layers and strategically placed global attention anchors, reducing KV-cache overhead while preserving long-range interaction.The architecture supports up to 1M tokens and maintains a 12.7B active-parameter budget despite 120.6B total parameters.Source: §2.1.3 · pp. 7–8

2.2. NVFP4 Pretraining

• NVFP4 training remains stable through 25T tokens, but quantization underflows small gradients in routed expert layers, producing roughly 3x more zero-valued weight gradients than BF16 at the same token horizon.Switching to MXFP8 before annealing improved the loss trajectory but produced no downstream accuracy gains, so the final model stayed in NVFP4 throughout pretraining.Source: §2.2 · pp. 8–10

2.3.1. Data

• New specialized pretraining data include approximately 15M cleaned Python problem-solution pairs and a 0.2B-token dataset, improving HumanEval, MBPP, and CRUXEval-O by 1-2 points over the Nemotron 3 Nano base checkpoint when added to the last 100B-token redo.The datasets are released as Nemotron-Pretraining-Specialized-v1.1.Source: §2.3.1 · p. 11, §2.3.2 · p. 12, §2.3.3 · p. 12

2.3.4. Synthetic Economics

• Synthetic economics data covers cloze, calculation, sentence completion, and multiple-response questions across microeconomics, macroeconomics, and econometrics, with model-based verification.Qwen3-235B-A22B-Thinking-2507 generated step-by-step solutions and diversified questions from curated topic-term pairs.Source: §2.3.4 · p. 12

2.3.5. Synthetic Formal Logic

• Synthetic formal-logic data spans natural-language translation, conditional reasoning, and truth-table problems, with randomized personas, letters, and logical connectives.Qwen3-235B-A22B-Thinking-2507 generated and evaluated the problems and solutions.Source: §2.3.5 · pp. 12–13

2.3.6. Synthetic Multiple Choice

• Approximately 3.5M synthetic MMLU-style questions, totaling ~1.6B tokens, improve MMLU from 77.22 to 77.51, MATH Level 5 from 78.55 to 79.05, AIME-2024 from 53.3 to 56.7, and MBPP from 74.8 to 75.2.Questions are bootstrapped from MMLU auxiliary data, solved with sampled DeepSeek-V3 generations, and filtered by majority-vote agreement.Source: §2.3.6 · p. 13

2.3.7. Data Mixture and Ordering

• The pretraining corpus spans 16 categories, balances diversity against estimated quality, and shifts at 80% of tokens from broad Phase 1 data toward predominantly high-quality Phase 2 sources.The mixture includes web crawl, synthetic crawl, math, Wikipedia, code, academic, multilingual, finepdfs, and reasoning-focused SFT-style datasets.Source: §2.3.7 · pp. 13–14, Figure 10 · p. 14

2.4. Hyperparameters

• Pretraining used 25 trillion tokens with a Warmup-Stable-Decay schedule, 8,192-token sequences, 3,072-sequence batches, and a hybrid Mamba-MoE architecture with 512 experts and top-22 routing.It used BF16/NVFP4 precision, auxiliary-loss-free load balancing paired with standard load-balancing loss, and an MTP objective with loss scaling factor 0.3.Source: §2.4 · p. 14

2.5. Tracking Merge Evaluation

• Across 12 benchmarks, the best offline checkpoint merge consistently outperforms the corresponding trained checkpoint by 2–4 points on average, and the final 500B merge was selected for downstream alignment.During the stable learning-rate phase, merging yields a consistent 2–4-point improvement, while the gap narrows during the 5T-token decay phase.Source: §2.5 · p. 15, Figure 11 · p. 15

2.6. Long-Context Extension

• Long-context continuation used 34B tokens at 1,048,576 context length, followed by 17B tokens alternating 1M and 4K sequences to reduce the observed mathematical-performance impact.The blend used 20% document-QA data and 80% downscaled Phase 2 data, with 64-way context and expert parallelism.Source: §2.6 · p. 16

2.7. Base Model Evaluations

• The base-model evaluation reports Nemotron 3 Super 120B-A12B Base against Ling-flash-Base-2.0 and GLM-4.5-Air-Base in Table 4.Evaluation covers mathematical, code, general reasoning, multilingual, and long-context capabilities using specified decoding and sampling protocols.Source: §2.7 · p. 16, Table 4 · p. 17

3. Post-Training

• The post-training pipeline proceeds through SFT, three reinforcement-learning stages—RLVR, SWE-RL, and RLHF—and final MTP healing.The pipeline overview is provided in Figure 12.Source: §3 · p. 18, Figure 12 · p. 18

SFT objective and two-stage loss

• Two-stage SFT combines token-level averaging to induce reasoning with per-conversation normalization that restores long-input-short-output performance.Stage 2 normalizes each conversation by its output-token count before averaging, reducing the dominance of long outputs while retaining reasoning behavior.Source: §3.1 · p. 18, SFT objective and two-stage loss · pp. 18–19

3.1. Supervised Fine Tuning

• SFT scales agentic data and adds low-effort reasoning mode while retaining the Nemotron 3 Nano chat template.The training blend covers a wider range of agentic harnesses and interaction scenarios.Source: §3.1 · p. 18, §3 · p. 18

3.1.1. Data

• Agentic CLI data comprises roughly 15k direct-solution tasks, 3000 repository-based SWE questions, and 10k web-development tasks.The tasks include constraints such as markdown specifications, avoided test execution, and independent dependency setup in Node.js environments.Source: §3.1.1 · pp. 19–20
• The safety data adds an explicit response-policy classifier that selects supportive resources, refusals, or benign-partial answers for safety-sensitive prompts.A separate two-stage process generates policy-consistent reasoning traces and final responses, followed by content-moderation filtering.Source: §3.1.1 · p. 22
• Search-agent SFT records preserve multi-turn Thought–Action–Observation trajectories with supporting URLs and an average of 12 tool calls.The pipeline obfuscates knowledge-graph questions, solves them through web searches, and normalizes responses into validated JSON.Source: §3.1.1 · p. 23
• 279,116 specialized tool-use conversations span 838 domains, compared with 15,588 conversations across 5 domains in Nemotron 3 Nano.The conversations are generated through a six-stage synthetic pipeline covering domain, policy, scenario, trajectory, verification, and SFT selection.Source: §3.1.1 · p. 24

3.1.2. SFT Data Blend

• Stage 1 trains on over 7M samples, while Stage 2 retains 85% of that blend and adds 256K- and 512K-token long-context data.Compared with Nemotron 3 Nano, the blend substantially increases agentic-task volume, diversity, and allocation.Source: §3.1.2 · p. 25

3.1.3. Reasoning Control

• Nemotron 3 Super supports reasoning-off, regular, and low-effort reasoning modes, with the latter two optionally combined with inference-time budget control.These controls provide flexibility across accuracy-efficiency trade-offs for different application scenarios.Source: §3.1.3 · p. 25
• Low-effort reasoning is introduced during SFT using GPT-OSS-120B-generated samples covering math reasoning, STEM question answering, and instruction following.These samples comprise 2% of SFT data by sample count and are later optimized during RL.Source: §3.1.3 · p. 25
• SFT strips reasoning traces from random 3% of samples for reasoning-off mode, then adds 350 semi-on-policy steps that truncate 12% of rollouts to random reasoning budgets.The additional stage collects model roll-outs to support inference-time budget control.Source: §3.1.3 · p. 26

3.2. Reinforcement Learning

• The RL pipeline comprises multi-environment RLVR, a separate SWE-RL stage, RLHF, and an MTP-healing stage.SWE-RL is isolated because its rollouts are slower and typically require longer contexts, while RLHF targets instruction following, robustness, and interaction quality.Source: §3.2 · p. 26
• During MTP healing, only MTP heads are trained with negative log likelihood on RLVR prompts and generated responses, significantly improving MTP accuracy.The remaining model weights stay frozen, using an objective similar to SFT.Source: §3.2 · p. 26

3.2.1. Stage 1: Multi-environment RL from Verifiable Rewards

• Joint training across all 21 RLVR environments yields stable gains, whereas single-environment training causes severe regressions on other benchmarks.The environments span math, code, STEM, safety, chat, instruction following, long-context, puzzles, and agentic tasks.Source: §3.2.1 · pp. 26–27
• RLVR filters prompts already solved by the SFT model, sorts remaining samples by difficulty, and uses a curriculum across diverse environments.The strategy follows Nemotron 3 Nano and jointly informs updates with the complete environment distribution to help prevent task regressions.Source: §3.2.1 · pp. 26–27, §3.2 · p. 26
• Low-effort prompts use rewards based on correctness and generated-token count, comprising 2% of RL prompts initially and 1% later; this improves low-effort performance broadly.The mix covers subsets of Math, STEM QA, and withheld competitive-coding problems, with coding prompts excluded from SFT data.Source: §3.2.1 · p. 27

RLVR Data

• The RL training scales to 21 environments and 37 datasets, spanning math with and without Python execution, formal proof verification, competition-style code, STEM, safety, and agentic tool use.The safety data include an environment targeting reduced over-refusals, while math training uses both Python execution and no execution.Source: RLVR Data · p. 27

3.2.2. Stage 2: End-to-end RL for Software Engineering

• SWE-RL trains agents to solve GitHub issues through containerized OpenHands rollouts, binary ground-truth test rewards, and varied Claude Code- and Codex CLI-compatible harnesses.The shared harness varies tools and prompts during training, improving generalization and performance across target harnesses at inference.Source: §3.2.2 · p. 28

3.2.3. Stage 3: Reinforcement Learning from Human Feedback

• RLHF uses a principle-following GenRM initialized from Qwen3-235B-A22B-Thinking-2507 to guide identity and safety behavior during multi-environment RL and a final RLHF-only stage.GenRM training combines Helpsteer 3, commercially friendly lmarena-140k subsets, and newly collected human-preference data.Source: §3.2.3 · p. 28

3.2.4. Algorithm

• PivotRL reuses offline SFT expert trajectories at uncertain assistant turns, improving agentic RL efficiency without SFT’s out-of-distribution degradation across programming, search, terminal, and conversational tool-use domains.Domain-appropriate rewards credit actions similar to expert behavior rather than requiring exact matches.Source: §3.2.4 · pp. 28–29

3.2.5. Infrastructure

• The post-training stack uses NeMo Gym and NeMo RL for large-scale asynchronous RL, with decoupled generation and training, in-flight weight updates, and infrastructure supporting 1K-GPU runs.At scale, the system encountered hardware failures and frequent port-binding race conditions requiring resiliency and startup optimizations.Source: §3.2.5 · pp. 29–30

Evaluation setup

• The evaluation covers general knowledge, reasoning, agentic, instruction following, long-context, and multilingual capabilities using the Nemotron 3 Nano benchmark suite and evaluation stack.Results were collected via Nemo Evaluator SDK and, for most benchmarks, the Nemo Skills Harness.Source: §3.3 · pp. 30, 32

Reasoning Capabilities

• Across reasoning benchmarks, Nemotron 3 Super is competitive with GPT-OSS-120B while lagging slightly behind Qwen-3.5-122B.Reasoning is evaluated on AIME 25, HMMT Feb 25, GPQA, LiveCodeBench v5, SciCode, and HLE.Source: §3.3 · p. 32

Agentic capabilities

• Across agentic benchmarks, Nemotron 3 Super outperforms or is competitive with GPT-OSS-120B and is competitive with Qwen 3.5 122B on some harnesses.The evaluation covers TerminalBench, SWE-Bench, TauBench V2, BrowseComp, and BIRD.Source: §3.3 · p. 32

Chat and Instruction Following Capabilities

• Nemotron 3 Super is competitive with the baseline models across IFBench, Multi-Challenge, and Arena-Hard V2.These benchmarks assess chat and instruction-following capabilities.Source: §3.3 · p. 32

Long Context Capabilities

• Nemotron 3 Super is evaluated on Ruler with 100 samples per task and on AALCR for long-context capabilities.The reported passage specifies the evaluation scope but not comparative long-context results.Source: §3.3 · p. 32

Multilingual Capabilities

• Nemotron 3 Super matches or outperforms the baseline models on MMLU-ProX and WMT24++ en→xx.Multilingual capability is measured on both benchmarks.Source: §3.3 · p. 32

Evaluation setup

• Baseline comparisons use officially reported numbers when available, otherwise public aggregators consistent with the official protocol or scores computed with official evaluation settings.This procedure follows the Nemotron 3 Nano report when official numbers are unavailable.Source: §3.3 · p. 32

4.1. Nemotron 3 Super FP8 Checkpoint

• FP8 W8A8 and NVFP4 W4A4 checkpoints quantize weights and activations for Hopper and Blackwell deployment, respectively.FP8 calibration used 256 post-training SFT samples with context length 65536, while selected MoE GEMMs and Mamba Linear layers were quantized and caches assigned reduced precisions.Source: §4 · p. 33, §4.1 · p. 33

4.2. Nemotron 3 Super FP4 Checkpoint

• The final NVFP4 recipe combines MSE-minimizing weight scales, dynamic max-based activation scales, and selective promotion of sensitive layers through AutoQuantize.AutoQuantize estimates operator sensitivity, performance costs, and layer-wise allocations under deployment constraints using knapsack-style optimization.Source: §4.2 · p. 34
• 99.8% median accuracy relative to the BF16 baseline was achieved by the final NVFP4 PTQ recipe while retaining near-FP4 performance.The mixed-precision process completed in less than 2 hours on one 8-GPU B200 node using 512 SFT samples at sequence length 4096.Source: Table 8 · p. 35, §4.2 · p. 35

4.3. Mamba State Quantization

• Up to 40% increased verbosity occurred when directly casting the SSM cache to FP16 with W8A8, and up to 37% even with BF16 weights and activations.Both INT16 with per-block scaling and FP16 with stochastic rounding maintained accuracy and verbosity similar to the FP32 baseline; the paper selected FP16 with Philox<5> stochastic rounding.Source: §4.3 · pp. 36–37
• Mamba-cache quantization errors accumulate across recurrent decoding steps, motivating training-free remedies such as per-block scaling and stochastic rounding.FP32 per-block scaling over state-dimension blocks of 128 eliminated verbosity for INT16, while stochastic rounding removes the systematic bias introduced by round-to-nearest ties-to-even.Source: §4.3 · pp. 36–37

5. Conclusion

• Up to 2.2× higher throughput than GPT-OSS-120B is achieved while maintaining higher accuracy across a wide range of tasks.The model was quantized to FP8 and NVFP4 without sacrificing model accuracy.Source: §5 · p. 37
• Nemotron 3 Super combines 12B active and 120B total parameters in an MoE hybrid Mamba-Attention architecture with LatentMoE and MTP layers.LatentMoE improves accuracy, while MTP layers accelerate inference through speculative decoding.Source: §5 · p. 37
• The model was pretrained on 25 trillion text tokens with low-precision NVFP4 and then post-trained on diverse reinforcement-learning environments.The resulting system targets strong agentic capabilities.Source: §5 · p. 37
• Pre-trained, post-trained, and quantized Nemotron 3 Super checkpoints are released on HuggingFace.Source: §5 · p. 37

• Per-benchmark accuracy compares trained checkpoints with the best offline checkpoint merge across the full 25T-token training run.Source: Figure 17 · p. 49
• The 12 benchmarks cover general knowledge, code generation, mathematical reasoning, and commonsense understanding, with the shaded region marking the learning-rate decay phase.Source: Figure 17 · p. 49, A. Per-Benchmark Merge Evaluation · p. 49

B.1. PTQ Algorithm Ablation

• All linear layers except the final classification layer and attention linear layers are quantized to NVFP4.This defines the quantization scope used in the experiments.Source: §B.1 · p. 49
• The ablation compares evaluation accuracy across various PTQ algorithms.Table 10 reports the PTQ algorithm ablation, but the supplied passage does not provide its cell values.Source: Table 10 · p. 50

Sensitivity Measurement

• AutoQuantize measures each operator’s quantization sensitivity at its immediate or closest available output using a local second-order metric.For linear layers, the measurement point is the linear-layer output, and quantization error is compared against BF16.Source: §B.2 · p. 50
• Because full Hessian computation is expensive, AutoQuantize uses a diagonal Hessian approximation estimated with the diagonal Fisher information matrix.Source: §B.2 · p. 50

Format Selection

• AutoQuantize chooses each operator’s quantization format by solving a constrained optimization subject to a total deployment-cost budget.The optimization uses the selected operator formats and the total deployment cost budget as its constraint.Source: §B.2 · p. 50

Deployment-Restriction-Aware Search

• Fused Q, K, and V projections within each layer share one quantization format and are optimized as a single decision variable.Sensitivity and cost are aggregated over the fused QKV group.Source: §B.2.1 · pp. 50–51
• The fused-output quadratic sensitivity remains additive across Q, K, and V when shared formatting preserves additive branch contributions.This formulation relies on the stated second-order sensitivity approximation.Source: §B.2.1 · p. 51
• Within each MoE layer, sparse experts must share one format, so each expert’s up_proj and down_proj are assigned jointly as one operator-level decision.The restriction follows quantized MoE API requirements in vLLM and TensorRT-LLM.Source: §B.2.1 · p. 51
• MoE sensitivity is measured at the block output and deployment cost sums across sparse experts, while latent projections and shared experts remain independently assignable.The coupled MoE group includes all sparse-expert up_proj/down_proj operators within one layer.Source: §B.2.1 · p. 51