Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

1. Introduction

• The hybrid Mamba-Attention architecture reduces attention cost and KV-cache footprint to improve throughput, while Mixture-of-Experts improves accuracy per active parameter.The paper presents Nemotron 3 Ultra as the largest and most capable model in the Nemotron 3 family (Blake-man et al., 2025).Source: §1 · p. 1
• Pretraining used NVFP4, LatentMoE, and Multi Token Prediction on 20 trillion text tokens, split into 15 trillion for diversity and coverage and 5 trillion for higher-quality refinement.The authors report that LatentMoE improves accuracy per parameter over standard Granular MoEs and MTP enables faster inference with speculative decoding; the pretrained base also achieves significantly higher accuracy than DeepSeek v3.2, Mistral Large 3, Kimi-K2, and GLM-4.5.Source: §1 · p. 1
• 5.9×, 4.8×, and 1.6× higher inference throughput than GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B, and Qwen-3.5-397B-17B, respectively, with on-par accuracy at 8K input / 64K output.Figure 1 reports maximum throughput using NVFP4 on GB200; Nemotron 3 Ultra uses TRT-LLM and other models use vLLM, with the best available speculative-decoding setting selected.Source: §1 · p. 1, Figure 1 · p. 2
• Post-training combined curated SFT, unified RLVR across reasoning, agentic, code, safety, usability, and chat environments, and MOPD from more than ten specialized teacher models.MOPD provides dense token-level guidance on student-generated rollouts, and reasoning-effort control enables inference-time adjustment of the accuracy–compute trade-off.Source: §1 · pp. 1–2

Checkpoints

• The release includes Base, Post-Trained, and NVFP4-quantized checkpoints, alongside training recipes, data, and RL environments.The listed checkpoints also include BF16 post-trained, BF16 Base, and GenRM models; GenRM is used for RLHF.Source: §1 · p. 2, Checkpoints · p. 2, Data · p. 2

Data

• Adding synthetic legal datasets to Nemotron 3 Nano pretraining increased proxy LegalBench average accuracy from 64.6 to 74.7.The legal collection is intended to improve LLM legal capabilities.Source: Data · p. 3

2.1. Model Architecture

• The model uses a 550B-total, 55B-active hybrid Mamba-Attention MoE architecture with LatentMoE and two shared-parameter MTP heads for inference acceleration (NVIDIA, 2026).Each MTP head comprises one attention layer followed by one MoE layer.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2. NVFP4 Pretraining

• NVFP4 pretraining used E2M1 with two-dimensional weight block quantization, Random Hadamard Transforms on wgrad inputs, and stochastic gradient rounding; BF16 ablations showed an average loss gap below 0.4%.Across checkpoints at 5T, 10T, and 16T tokens, the initial gaps were 0.27%, 0.28%, and 0.25%; switching to BF16 did not resolve training divergence.Source: §2.2 · pp. 3–4, Figure 3 · p. 5

2.3. Pretraining Data

• Pretraining data included 173B refreshed GitHub code tokens and synthetic Q&A seeded from public benchmark training examples, with held-out test splits excluded.The synthetic datasets provide multiple-choice and open-ended formats, add answer-enriched context where appropriate, and undergo validation, deduplication, and task-specific filtering.Source: §2.3.1 · p. 4, §2.3.2 · pp. 5–6

2.3.2. Nemotron-Pretraining-Multiple-Choice and Nemotron-Pretraining-Generative

• Adding benchmark-oriented synthetic data raised MMLU-Pro from 64.8 to 66.6, average code from 73.2 to 75.1, commonsense understanding from 72.9 to 74.5, and GPQA from 30.8 to 41.9; average math remained stable at 87.6 to 87.9.These results came from a 100B-token continued-pretraining ablation on a Nemotron-family base checkpoint.Source: §2.3.2 · p. 6

2.3.3. Nemotron-Pretraining-Fact-Seeking

• Injecting fact-seeking data during the final 100B training tokens improved converted multiple-choice SimpleQA accuracy from 40.24 to 50.16.Because SimpleQA questions were converted to multiple-choice format, these scores are not directly comparable to the original SimpleQA scores.Source: §2.3.3 · p. 6

2.3.5. Nemotron-Pretraining-Legal

• Legal-specific datasets increased average LegalBench accuracy from 64.6 to 74.7 in a 100B-token ablation starting from a 14.9T-token Nemotron 3 Nano checkpoint.The evaluation covered over 100 LegalBench subtasks.Source: §2.3.5 · p. 8

2.3.6. Data Mixture and Ordering

• After about 15T tokens, training shifted from a diversity-biased phase 1 to a quality-biased phase 2; web crawl data made up about 49% and 38% of their tokens, respectively.The learning rate warmed up for 200B tokens to 2.5 × 10−4, then decayed over the final 5T tokens to 2.5 × 10−6 using a minus-sqrt schedule.Source: §2.3.6 · p. 8, §2.4 · pp. 8–9, Figure 4 · p. 9

2.5. Long-Context Extension

• The 33B-token long-context phase trained at 1,048,576-token context length for 92% of iterations and 4,096 tokens for the remaining 8%.The blend used 46% long-context data and 54% phase 2 data; 4K iterations used only math and code SFT-style data to maintain short-benchmark metrics, and sequences of different lengths were not mixed within an iteration.Source: §2.5 · p. 9

2.6. Base Model Evaluations

• Nemotron 3 Ultra Base evaluation results were reported against DeepSeek-V3.2-Exp-Base, Mistral-Large-3-675B-Base-2512, Kimi-K2-Base, and GLM-4.5-Base.The results are shown in Table 2; the supplied passage does not provide the scores.Source: §2.6 · p. 11

2.7. Model Stability

• Two pretraining divergences involved simultaneous increases in training cross-entropy loss and wgrad L2 norm.Both are shown in Figure 5.Source: Figure 5 · p. 11, §2.7 · p. 12

Divergence 1: Local Gradient Accumulation Precision for Output Layer

• At around 8T tokens, reducing output-layer gradient accumulation from FP32 to BF16 caused the first divergence; restoring FP32 gradient reduction re-stabilized training.With two MTP blocks scaled by 0.05 each, MTP-2 loss began spiking before training and validation loss.Source: Figure 5 · p. 11, Divergence 1: Local Gradient… · p. 12, Figure 6 · p. 12

Divergence 2: Undetermined

• After a rollback to the 15T-token checkpoint, starting learning-rate annealing immediately mitigated the second divergence, and the pretraining horizon was cut to 20T tokens.Ablations found mitigation with both 5T and 10T decay schedules; the cause remained unidentified.Source: Divergence 2: Undetermined · p. 12, Figure 7 · p. 13
• For Ultra, maximum-layer MaxVio rose from 4.8 at initialization to approximately 12 by 12T tokens, correlating with instability but not establishing causation.The median layer stayed around 1.2, while training-data MaxVio was consistently lower than validation-data MaxVio across Nano, Super, and Ultra.Source: Divergence 2: Undetermined · pp. 13–14
• Ultra residual norms spiked in early layers around 7.5T tokens and drastically around 11T, unlike the earlier-layer stabilization observed for Nano and Super.Residual norms differed by four orders of magnitude across Ultra’s depth, versus three for Super; the spikes indicated poor signal propagation.Source: Figure 8 · p. 13, Divergence 2: Undetermined · p. 14

3. Post-Training

• The redesigned post-training pipeline starts with general SFT, then prepares the student through RLVR and MOPD warmup before specialized-teacher training and iterative MOPD optimization.MOPD augments consecutive reinforcement-learning stages to support broad capability acquisition and targeted specialization.Source: §3 · p. 15, Figure 9 · p. 15

3.1. Supervised Fine Tuning

• SFT trains in two stages, extending packed sequences from 294,912 to 515,000 tokens and adding long-context data up to 512K tokens.Both stages use global batch size 64; the first runs for 204,800 samples and the second for 19,200. SFT retains two shared-weight MTP layers with auxiliary-loss scaling of 0.1.Source: §3.1 · p. 15

3.1.1. Data

• SFT data target long-context reasoning, medium-effort responses, and reasoning-budget control, with truncated traces masking the </think> tokens from training loss.The long-context set supports multi-document reasoning, sequential scanning, and synthetic-table queries; GPT-OSS-120B medium-effort samples cover math, STEM, and instruction following.Source: §3.1.1 · pp. 15–16
• Search training combines Wikidata-based multi-hop trajectories, 21.7K commercially cleared OpenResearcher trajectories, and BrowseComp examples requiring 50–100 searches.OpenResearcher data preserves browser-tool interactions for open-ended research and evidence-grounded search behavior.Source: §3.1.1 · pp. 16–17
• Agentic SFT includes 370K multi-turn terminal conversations and GitHub-issue resolution trajectories filtered for tool hygiene, task completion, and test verification.The terminal data spans software engineering, data processing, file operations, and scientific computing; issue-resolution rollouts use multiple models and harnesses.Source: §3.1.1 · p. 17
• Other SFT data spans math and proofs, science, multi-turn chat, competitive coding, and synthetic conversational tool-use trajectories.Competitive-coding data includes 1.2M Python, 1.0M C++14, and 1.3M Python tool-calling reasoning traces.Source: §3.1.1 · pp. 17–18
• Specialized SFT adds CUDA and RTL data, multilingual translations, and a multilingual safety blend of about 135K samples.The RTL set contains around 1.2M samples, while the safety blend includes approximately 45K English samples and 15K in each of six translated languages.Source: §3.1.1 · pp. 16, 18, 19

3.1.2. Data Packing

• Length-aware best-fit packing assigns each conversation to the sequence with the closest remaining capacity, without truncating or splitting conversations.The packer interleaves sources, deduplicates prompts within each pack, and shuffles completed packs to mix the data distribution.Source: §3.1.2 · pp. 19–20

3.2. Reinforcement Learning

• Mixed-environment RLVR spans diverse environments, while asynchronous GRPO training uses 8,192 global-batch samples and 16 rollouts per sample.Generation length begins at 48K tokens and later increases to 64K tokens.Source: §3.2 · p. 20

3.3. MOPD

• More than ten domain-specialized teachers address diluted per-domain learning signals as mixed-environment RLVR expands across environments.MOPD asynchronously pipelines student rollouts, teacher scoring, and optimization, then repeats teacher training and distillation for two iterations.Source: §3.3 · p. 20, Figure 10 · p. 21

3.3.1. Algorithm

• MOPD supplies dense token-level teacher signals by matching teacher distributions on student-generated states, unlike sparse, environment-dependent RLVR rewards.Asynchronous training decouples stale behavior-policy rollouts from the proximal trust-region policy and applies PPO-style clipping.Source: §3.3.1 · pp. 21–22

3.3.2. Specialized Teachers

• The specialized-teacher program includes an STEM reasoning teacher that matches or outperforms DeepSeek V4 Pro (High) on challenging benchmarks.The section also introduces an agentic safety teacher for resisting indirect prompt injections embedded in tool responses.Source: §3.3.2 · pp. 24–25, Table 3 · p. 25
• +2.4 improvements on LiveCodeBench v6 beyond the General Reasoning Teacher follow additional Competitive Coding RL; non-STEM-focused RL also improves all domains.The coding RL uses a filtered set of 3.5K prompts; the non-STEM RL uses a global batch size of 2,048.Source: §3.3.2 · p. 26

3.3.3. MOPD Warmup

• A brief light-SFT warmup on teacher-distribution data aligns student trajectories, improving agentic-domain performance after MOPD while yielding negligible HLE gains.The warmup’s limited scale induces minimal regression on unrelated domains, which MOPD subsequently recovers.Source: §3.3.3 · p. 27, Table 4 · p. 27

3.3.4. Results and Discussions

• MOPD results show gains over the RLVR student and recovery toward specialized teachers across domains.The RLVR checkpoint is the initial student checkpoint.Source: Table 5 · p. 27, §3.3.4 · p. 27
• MOPD improves the RLVR student across the evaluation suite and sometimes surpasses specialized teachers, especially on agentic and instruction-following benchmarks.Examples include Terminal Bench, GDPVal, SWE-Bench Verified, TauBench Telecom, BrowseComp, OmniScience, IFBench, and Multi-Challenge; MOPD2 also outperforms a teacher on Terminal Bench data-science tasks.Source: §3.3.4 · p. 28
• MOPD gains are smaller on self-contained reasoning benchmarks, especially HLE, when the teacher’s advantage comes from additional off-policy data rather than preferences over student-generated trajectories.Because MOPD scores student rollouts, supervision is less informative when the student rarely samples the reasoning paths needed for the missing capability.Source: §3.3.4 · p. 28
• Warmup substantially improves MOPD in agentic domains but has little effect on HLE, consistent with trajectory overlap mattering more in agentic tasks than missing reasoning capabilities.The passage describes this as consistent with the warmup ablation in Table 4.Source: §3.3.4 · p. 28

3.3.5. Limitations and Open Problems

• Preliminary variants that failed to improve under the current setup are reported as open research directions, not evidence that they are fundamentally ineffective.The authors encourage further exploration because these approaches could yield stronger MOPD methods with additional research.Source: §3.3.5 · p. 28
• Logit matching did not improve MOPD and sometimes underperformed sampled-token distillation on agentic benchmarks such as Terminal Bench.The authors hypothesize that matching full distributions can amplify poorly calibrated teacher logits on student prefixes with limited teacher support; when broader distributional supervision helps remains open.Source: §3.3.5 · pp. 28–29
• Future MOPD work could align teacher and student SFT stages or use specialized-teacher-generated SFT data before RL or distillation.These alternatives were not systematically evaluated because of time and resource constraints.Source: §3.3.5 · p. 29
• Mixing end-to-end agentic and reasoning environments creates substantial training inefficiency, so most agentic tasks use single-turn rollouts, similar to PivotRL (Yi et al., 2026).Whether end-to-end rollouts improve performance and how to address potential distribution mismatch (Wang et al., 2026a) remain open questions.Source: §3.3.5 · p. 29

• The MTP head drafts multiple future tokens from backbone hidden states for verification, using a shared recursive formulation that grows the draft horizon without extra parameters.Several draft tokens may be accepted in one verification step through rejection sampling or naive 1-1 token matching.Source: §3.4 · p. 29
• Teacher-forced MTP training differs from autoregressive inference, where later draft steps condition on increasingly noisy mixtures of backbone and MTP-generated hidden states, reducing deep-position acceptance.Source: §3.4 · p. 29
• Boosting continues from the MOPD checkpoint with the backbone fixed and only the MTP head updated on sampled earlier-step states, exposing it to inference-like noise.Training uses on-policy rollouts from general-purpose and agentic seed prompts, sampled at temp=1, for 12K steps at global batch size 64 with sequences capped at 8K tokens.Source: §3.4 · pp. 29–30
• The boosting objective uses temperature-scaled forward KL against backbone logits, disables gold-token cross-entropy, and uses temperature 2 across 7 MTP steps.Source: §3.4 · p. 30
• 3.15% on summarization tasks and 5.82% on coding tasks are the reported relative speculative-decoding speedup improvements over base MTP on SPEED-Bench.Boosted MTP increases acceptance length over base MTP, with gains across draft positions and the strongest improvement at deeper positions.Source: Table 6 · p. 30, §3.4 · p. 30

3.5. Reasoning Efficiency and Control

• Regular and medium-effort modes can combine with inference-time budget control, complementing agentic turn limits to provide accuracy-efficiency options for different application scenarios.Source: §3.5 · p. 31
• Medium-effort reasoning is introduced during SFT and optimized during RLVR, with approximately 2.5% of RLVR prompts covering math, STEM, and coding and length-based adjustments applied to rewards.The recipe was used for Nemotron 3 Ultra and Super; its effects generalized beyond those domains, and hyperparameters calibrated the resulting effort mode.Source: §3.5 · p. 31
• On Artificial Analysis Intelligence Index V4 tasks, Ultra’s medium-effort mode uses around 2.5X less tokens than regular mode at the cost of approximately 7% drop in accuracy.The comparison averages over 10 tasks, using Qwen 3.5 397B’s average token usage per task as the relative verbosity reference.Source: Figure 11 · p. 31, §3.5 · p. 31

3.6.1. Accelerating Rollout Generation with Multi-Token Prediction

• 1.46× speedup in mean per-step rollout-generation time came from MTP with k=5 versus the k=0 no-MTP baseline.The sweep compared k∈{0, 3, 5, 7}.Source: Figure 12 · p. 32, §3.6.1 · p. 32
• MTP’s benefit was concentrated in the slowest, long-tail generations, which emit more tokens and decode later at lower concurrency where speculative decoding is most effective.Figure 12 compares per-trajectory rollout times for k=5 and no MTP.Source: Figure 12 · p. 32, §3.6.1 · p. 32

3.6.2. Scaling RL Infrastructure

• Generation and sandbox/tool-calling failures accounted for ∼92% of observed RL software failures.The production cluster used NVIDIA GB200 nodes with Slurm orchestration and co-located CPUs for sandbox execution.Source: Table 7 · p. 32, §3.6.2 · p. 32

Ray GCS Scalability and Slurm Launch Overheads

• Startup cost fell from over 30 minutes to 10 minutes after replacing per-node srun calls with one multi-node invocation, reducing Slurm controller interactions from O(n) to O(1).Serial RPCs to slurmctld had become a scaling bottleneck as node count increased.Source: Ray GCS Scalability and Slurm Launch… · p. 33
• At 3K+ GPU scale, Ray GCS actor registrations caused 25–49-minute startups and thundering-herd failures; fixes removed 40% of registrations and shipped in Ray 2.55.The changes converted short-lived actors to tasks, pooled initialization actors per node, and tuned GCS.Source: Ray GCS Scalability and Slurm Launch… · p. 33

Topology-Aware NVLink Domain Placement

• Ray registers NVLink-domain UUIDs and sorts GPU bundles by domain and topology rank so Megatron EP groups stay within one NVLink domain.The placement process also uses Ray’s GPU pinning for rank and device assignment and corrects scrambled Slurm block ordering.Source: Topology-Aware NVLink Domain Placement · pp. 33–34
• 20% end-to-end throughput improvement on GB200 followed topology-aware placement that kept expert-parallel all-to-all traffic on NVLink.Training and generation actors were co-located in the same NVLink domain, avoiding fallback to InfiniBand.Source: Topology-Aware NVLink Domain Placement · p. 34

NUMA Binding for Policy and vLLM Workers

• 10%: binding policy and vLLM workers to the CPU socket local to their GPUs improved end-to-end throughput on GB200.Local placement keeps transfers, tokenization, preprocessing, and pinned-memory allocations on local DRAM and the socket-local C2C path.Source: NUMA Binding for Policy and vLLM Workers · p. 34

Checkpoint Save Blocking

• <1 second: asynchronous checkpointing and follow-on optimizations reduced exposed save time from ∼60 seconds per save.The optimizations included overlapping NCCL transfers with D2H copies, persistent workers, background finalization, cached save plans, and sharded optimizer-state saves.Source: Checkpoint Save Blocking · p. 34

JIT Cache and Initialization

• Three cache strategies reuse compiled artifacts: persist warm caches on shared storage, seed node-local /tmp before Ray starts, and bake FlashInfer cubins into the container.Subsequent JIT writes target local storage, and the baked cubins eliminate FlashInfer runtime compilation.Source: JIT Cache and Initialization · p. 35
• 99%: warm JIT caches reduced the initialization phase from 38.8 minutes to 0.4 minutes.This made initialization time negligible relative to step time.Source: JIT Cache and Initialization · p. 35

Multi-Node vLLM Operational Stability

• At scale, ABI mismatches, subprocess environment divergence, and incompatible JIT kernels during NCCL multi-node NVLink registration caused startup crashes or hangs.The issues affected kernel reuse, library resolution, and distributed initialization.Source: Multi-Node vLLM Operational Stability · p. 35
• Operational fixes unified shared-library versions, propagated environment variables to subprocesses, disabled affected NVLink registration paths, and added health checks, RPC timeouts, and cleanup.These measures were intended to improve stability across jobs.Source: Multi-Node vLLM Operational Stability · pp. 35–36

Container and Storage I/O

• Local container caching and single-writer cache archiving addressed startup and completion I/O storms from shared storage.Enroot reuses locally extracted images; JIT writes stay local during training, and one sidecar persists compressed cache tarballs for sequential reads at startup.Source: Container and Storage I/O · p. 36

3.6.3. Future Work

• Future work targets fail-fast fault isolation, independent recovery and scaling of generation or sandbox components, and fine-grained checkpointing of in-flight rollout and conversation state.The proposed checkpoints include KV cache and support replay from the last consistent snapshot.Source: §3.6.3 · p. 36

3.7.1. Evaluation Setup

• The evaluation suite spans agentic, reasoning, conversational, long-context, and multilingual tasks, using shared settings for resources, prompts, data, repeats, and metric extraction.Results were collected through Nemo Evaluator SDK using Nemo Gym, Nemo Skills, and Harbor, with model-card recommendations informing inference parameters.Source: §3.7.1 · pp. 36–37

3.7.2. Evaluation Results

• Nemotron 3 Ultra remains competitive with leading open models across agentic, reasoning, conversational, and other challenging evaluations.The comparison covers six open models, including some with substantially larger total parameter counts.Source: Table 10 · p. 38, §3.7.2 · pp. 38–39
• PinchBench score 90.0 is within 1.3 points of the best model, while ProfBench score 56.0 ties 1T-parameter Kimi-K2.6.Both benchmarks were held-out generalization gates, evaluated only once after the final model was produced and excluded from training-time monitoring and checkpoint selection.Source: §3.7.2 · p. 39
• IOI 2025 score 570.0 corresponds to top-3-human-level competitive programming, and IMOAnswerBench score 92.3 with tools demonstrates tool-integrated mathematical reasoning.The IOI score falls between the second- and third-ranked official human contestants; the IMOAnswerBench setting allows Python computation and verification.Source: §3.7.2 · p. 39
• Nemotron 3 Ultra supports 1M-token contexts with competitive million-token evaluation performance and achieves the highest AA-Omniscience non-hallucination score, 78.7.The reported reliability result suggests a lower tendency to produce unsupported answers when knowledge is uncertain.Source: §3.7.2 · p. 39

3.7.3. Test-time Scaling in Math Olympiad Problems

• The paper evaluates Nemotron 3 Ultra in a high-compute, search-based test-time scaling strategy on IMO-ProofBench Advanced, IMO 2025, Putnam 2025, and USAMO 2026.The strategy follows generate–verify–refine; Figure 13 tracks accuracy by refinement round, while Table 11 reports final accuracies.Source: §3.7.3 · p. 39, Figure 13 · p. 40, Table 11 · p. 40

4.3. Final Weight and GEMM Quantization Recipe

• The final 5.03 BPE recipe uses NVFP4 routed-expert GEMMs, FP8 shared-expert and Mamba linear GEMMs, and BF16 attention and MoE latent projections.Routed-expert weights use max-calibrated 4/6 scaling with dynamic max-based activation scaling; shared-expert and Mamba GEMMs use static max-calibrated per-tensor scales.Source: Table 12 · p. 41, §4.3 · p. 43

4.1. Bits per Element Selection

• AA-LCR improves by +2.4 points from 4.85 to 5.03 BPE, then plateaus at 64.2–65.0 through 7.19 BPE; 5.03 BPE is the smallest selected budget with no higher-BPE gains beyond noise.GPQA Diamond, SciCode, HLE, IFBench, and AA-Omniscience accuracy are flat within run-to-run noise across the sweep; the AA-Omniscience non-hallucination difference at 4.85 versus 5.03 BPE is attributed to variance.Source: §4.1 · pp. 41–42, Table 13 · p. 42

4.2. FP4 Algorithm Experiments

• At 5.03 BPE, Four-Over-Six reduces median routed-expert weight reconstruction relative MSE by 16.4% versus max calibration, and was selected for FP4 routed-expert scales.Four-Over-Six improves the balanced 5.03 BPE setting but degrades at 4.85 BPE; although MSE calibration further reduces weight MSE by 27.1%, downstream evaluations show no consistent accuracy increase.Source: Table 14 · p. 43, §4.2 · p. 43

4.4. Software Support in Model-Optimizer

• Megatron-LM quantized Ultra in 42 minutes versus about 2 hours for HuggingFace layerwise PTQ, supporting faster multi-node and long-sequence experiments.Its multi-node parallelism shards the model and experts across GPUs, while context parallelism supports calibration on sequences beyond 32K tokens.Source: Table 15 · p. 44, §4.4 · p. 44

4.5. SSM Cache Optimization

• Block-scaled INT8 with stochastic rounding largely preserves FP32 Mamba-cache accuracy and verbosity, whereas FP8 E4M3 degrades accuracy; the current release uses FP16 with stochastic rounding.Optimized 8-bit cache kernels with checkpointing support are under development; FP16 storage avoids block-scaling overhead while mitigating recurrent rounding bias from naive round-to-nearest storage.Source: §4.5 · p. 45

4.6. One NVFP4 checkpoint

• W4A16 is within 1 point of or above BF16 on four of five tasks; the single NVFP4 checkpoint runs native FP4 on Blackwell and W4A16 on Hopper.At relevant operating points, W4A16 is on or above W8A8’s throughput–latency Pareto, and enabling MTP moves its Pareto upward and into lower latency. The smaller W4 footprint also fits MTP on one 8-GPU H100 node.Source: §4.6 · pp. 45–46, Table 17 · p. 47

5.1. Performance across serving regimes

• Snapshotting the Mamba SSM state at every draft step enables rollback after rejected tokens, and coarser snapshots enable prefix caching across requests.Unlike Attention’s per-token KV cache, the Mamba state is overwritten each token, so earlier states are not directly available.Source: §5.1 · p. 48
• 1.6× over Qwen-3.5, Nemotron 3 Ultra leads on decode-heavy serving but trails on prefill-heavy workloads, reflecting Mamba-2’s sequence-constant decode cost and its higher active-parameter prefill cost.Figure 15 compares 8K/64K decode-heavy and 50K/2K prefill-heavy workloads; the analysis attributes prefill cost to active parameters and large-batch decode cost to total weight I/O.Source: §5.1 · p. 48, Figure 15 · p. 49

5.2. Inference at Ultra Scale

• Wide expert parallelism suits high-throughput serving, while wide tensor parallelism suits low-latency serving; combining TP, EP, and DP can outperform either alone in some low-latency settings.Wide EP benefits from rank-local computation but requires balancing requests across ranks and can replicate hot experts; wide TP reduces weight-read traffic until AllReduce becomes the bottleneck.Source: §5.2 · p. 49

5.1. Performance across serving regimes

• 2.89× speedup at draft length 6 is the peak for single-user NVFP4 decoding on one GB200 node at TP=4.At small batches, high draft lengths favor latency because verification is cheap; at large batches, compute-bound verification can make shorter drafts or disabling MTP preferable.Source: §5.1 · p. 48, Figure 16 · p. 50

5.2. Inference at Ultra Scale

• Up to roughly 10% throughput improvement comes from prefill-decode disaggregation on prefill-heavy workloads, with separate workers using tailored parallelism and scheduling.Hybrid cache transfer requires handling both KV cache and Mamba SSM state; semantic KV-event metadata and a NIXL host-resolution fix enable support in vLLM (Kwon et al., 2023).Source: §5.2 · pp. 49–50
• Roughly 5% throughput improvement comes from FlashInfer’s NVLinkOneSided all-to-all backend over vLLM’s default scheme on GB200 NVL72.The default all-to-all accounts for 15% to 20% of runtime on prefill-heavy workloads; DWDP (NVIDIA TensorRT LLM Team, 2026a) instead pulls expert weights to each rank to avoid all-to-all.Source: §5.2 · p. 50
• MoE-side chunking lets prefill chunks grow under wide expert parallelism without hitting routed-expert kernel resource limits.Small request-level chunks can hurt interactivity by leaving few pure-decode passes; large chunks create a union of tokens across data-parallel ranks, which MoE-side chunking splits within the kernel.Source: §5.2 · pp. 50–51
• Load-time weight padding resolves GEMM alignment violations for some TP, quantization, and hardware combinations, while runtime infrastructure ignores padded positions.The paper identifies kernel-friendly inner dimensions as a future way to avoid this padding.Source: §5.2 · p. 51

6. Conclusion

• Nemotron 3 Ultra has 550 billion total and 55 billion active parameters, using a MoE Hybrid Mamba-Attention architecture with LatentMoE and MTP.Source: §6 · p. 51
• 5x higher inference throughput than other state-of-the-art open LLMs is achieved with on-par accuracy.Source: §6 · p. 51
• The model was pre-trained on 20 trillion text tokens, then post-trained with SFT, RL, and MOPD; its pre-trained, post-trained, and quantized checkpoints and training data are open-sourced on HuggingFace.Source: §6 · p. 51

A.1. Benchmark Details

• TauBench V3 uses an added user-simulation prompt across domains, terminal knowledge-base search for banking, GPT-5.2 low-effort simulation, and an eight-trial average.DeepSeek-V4 models were evaluated under max reasoning.Source: §A.1 · p. 64
• ProfBench Search evaluates four professional domains with search and browsing enabled at 256K context, reporting scores averaged over 16 runs.Tasks use workflows judged by professional-annotated rubrics; context management is disabled.Source: §A.1 · p. 64
• BrowseComp uses Tavily search and browsing plus terminal access, storing retrieved pages per task so agents can inspect snippets and selectively search saved content.This setup preserves collected information across context resets while limiting context growth.Source: §A.1 · p. 64
• FAB v1.1 evaluation uses 200 questions: 50 public validation samples and 150 additional samples from a privately licensed validation set.The benchmark covers nine entry-level financial analyst task categories; its full private test set contains 337 questions.Source: §A.1 · p. 64
• In FAB v1.1, agents receive four SEC-focused tools without web search and an additional Google web-search tool via Tavily in the web-search condition.Most questions are expected to be answerable directly from EDGAR, with web search serving as a supplementary signal.Source: §A.1 · p. 64
• FAB v1.1 answers are judged against expert-written ground truth using GPT-5.2 as an LLM judge, with the mode of three evaluations reported.The evaluation closely follows the open-source harness.Source: §A.1 · p. 64

A.2. Harness Robustness

• Task distributions span five verticals; the supplied list names terminal use and software engineering, repo bug fixing, office and productivity, and general or multi-domain knowledge.The passage identifies five verticals but specifies only these four categories in the provided text.Source: §A.2 · pp. 64–65
• Each task distribution is trained under at least two harnesses, including custom internal harnesses.Source: §A.2 · p. 65