PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

NVIDIA, :, Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Arushi Goel, Mike Ranzinger, Greg Heinrich, Guo Chen, Lukas Voegtle, Philipp Fischer, Timo Roman, Karan Sapra, Collin McCarthy, Shaokun Zhang, Fuxiao Liu, Hanrong Ye, Yi Dong, Mingjie Liu, Yifan Peng, Piotr Zelasko, Zhehuai Chen, Nithin Rao Koluguri, Nune Tadevosyan, Lilit Grigoryan, Ehsan Hosseini Asl, Pritam Biswas, Leili Tavabi, Yuanhang Su, Zhiding Yu, Peter Jin, Alexandre Milesi, Netanel Haber, Yao Xu, Sarah Amiraslani, Nabin Mulepati, Eric Tramel, Jaehun Jung, Ximing Lu, Brandon Cui, Jin Xu, Zhiqi Li, Shihao Wang, Yuanguo Kuang, Shaokun Zhang, Huck Yang, Boyi Li, Hongxu Yin, Song Han, Bilal Kartal, Pavlo Molchanov, Adi Renduchintala, Charles Wang, David Mosallanezhad, Soumye Singhal, Luis Vega, Katherine Cheung, Sreyan Ghosh, Yian Zhang, Alexander Bukharin, Venkat Srinivasan, Johnny Greco, Andre Manoel, Maarten Van Segbroeck, Suseella Panguliri, Rohit Watve, Divyanshu Kakwani, Shubham Pachori, Jeffrey Glick, Radha Sri-Tharan, Aileen Zaman, Khanh Nguyen, Shi Chen, Jiaheng Fang, Qing Miao, Wenfei Zhou, Yu Wang, Zaid Pervaiz Bhat, Varun Praveen, Arihant Jain, Ramanathan Arunachalam, Tomasz Kornuta, Ashton Sharabiani, Amy Shen, Wei Huang, Yi-Fu Wu, Ali Roshan Ghias, Huiying Li, Brian Yu, Nima Tajbakhsh, Chen Cui, Wenwen Gao, Li Ding, Terry Kong, Manoj Kilaru, Anahita Bhiwandiwalla, Marek Wawrzos, Daniel Korzekwa, Pablo Ribalta, Grzegorz Chlebus, Besmira Nushi, Ewa Dobrowolska, Maciej Jakub Mikulski, Kunal Dhawan, Steve Huang, Jagadeesh Balam, Yongqiang Wang, Nikolay Karpov, Valentin Mendelev, George Zelenfroynd, Meline Mkrtchyan, Qing Miao, Omri Almog, Bhavesh Pawar, Rameshwar Shivbhakta, Sudeep Sabnis, Ashrton Sharabiani, Negar Habibi, Geethapriya Venkataramani, Pamela Peng, Prerit Rodney, Serge Panev, Richard Mazzarese, Nicky Liu, Michael Fukuyama, Andrii Skliar, Roger Waleffe, Duncan Riach, Yunheng Zou, Jian Hu, Hao Zhang, Binfeng Xu, Yuhao Yang, Zuhair Ahmed, Alexandre Milesi, Carlo del Mundo, Chad Voegele, Zhiyu Cheng, Nave Assaf, Andrii Skliar, Daniel Afrimi, Natan Bagrov, Ran Zilberstein, Ofri Masad, Eugene Khvedchenia, Natan Bagrov, Borys Tymchenko, Tomer Asida, Daniel Afrimi, Parth Mannan, Victor Cui, Michael Evans, Katherine Luna, Jie Lou, Pinky Xu, Guyue Huang, Negar Habibi, Michael Boone, Pradeep Thalasta, Adeola Adesoba, Dina Yared, Christopher Parisien, Leon Derczynski, Shaona Ghosh, Wes Feely, Micah Schaffer, Radha Sri-Tharan, Jeffrey Glick, Barnaby Simkin, George Zelenfroynd, Tomasz Grzegorzek, Rishabh Garg, Aastha Jhunjhunwala, Sergei Kolchenko, Farzan Memarian, Haran Kumar, Shiv Kumar, Isabel Hulseman, Anjali Shah, Kari Briski, Padmavathy Subramanian, Joey Conway, Udi Karpas, Jane Polak Scowcroft, Annie Surla, Shilpa Ammireddy, Ellie Evans, Jesse Oliver, Tom Balough, Chia-Chih Chen, Sandip Bhaskar, Alejandra Rico, Bardiya Sadeghi, Seph Mard, Katherine Cheung, Meredith Price, Laya Sleiman, Saori Kaji, Wesley Helmholz, Wendy Quan, Michael Lightstone, Jonathan Cohen, Jian Zhang, Oleksii Kuchaiev, Boris Ginsburg, Jan Kautz, Eileen Long, Mohammad Shoeybi, Mostofa Patwary, Oluwatobi Olabiyi, Andrew Tao, Bryan Catanzaro, Udi Karpas

arXiv:2604.24954v2cs.LGcs.AIcs.CV

TL;DR

Cross-modal reasoning across images, video, and audio is challenging because it requires sophisticated reasoning. Nemotron 3 Nano Omni adds native audio support and multimodal token reduction, achieving consistent gains over its predecessor across a broad evaluation suite.

  • Problem

    Cross-modal reasoning across images, video, and audio is challenging because multimodal understanding requires sophisticated reasoning.

    Source: §3.2.4 · p. 9

  • Method

    The paper presents an omni-modal model with native audio support, multimodal token-reduction techniques, and a staged training recipe.

    Source: §1 · p. 1, §5 · p. 16

  • Results

    Across a broad evaluation suite, Nemotron 3 Nano Omni delivers consistent gains over Nemotron Nano V2 VL and leading or competitive results across document, GUI, audio-video, and voice tasks.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The authors release model checkpoints alongside portions of training data and code to facilitate further research and development.

    Source: §5 · p. 16

Abstract

from arXiv · show

We introduce Nemotron 3 Nano Omni, the latest model in the Nemotron multimodal series and the first to natively support audio inputs alongside text, images, and video. Nemotron 3 Nano Omni delivers consistent accuracy improvements over its predecessor, Nemotron Nano V2 VL, across all modalities, enabled by advances in architecture, training data and recipes. In particular, Nemotron 3 delivers leading results in real-world document understanding, long audio-video comprehension, and agentic computer use. Built on the highly efficient Nemotron 3 Nano 30B-A3B backbone, Nemotron 3 Nano Omni further incorporates innovative multimodal token-reduction techniques to deliver substantially lower inference latency and higher throughput than other models of similar size. We are releasing model checkpoints in BF16, FP8, and FP4 formats, along with portions of the training data and codebase to facilitate further research and development.

NVIDIA · 1. Introduction

Nemotron 3 Nano Omni extends the Nemotron multimodal family with native audio support and improved reasoning across text, images, video, and audio. It combines a stronger MoE backbone, multimodal token reduction, and staged training to improve practical performance and inference efficiency.

1. Introduction

  • The model is particularly effective in real-world document understanding, long audio-video comprehension, and agentic computer use.It uses the Nemotron 3 Nano 30B-A3B backbone with C-RADIOv4-H1 vision and Parakeet-TDT-0.6B-v22 audio encoders, plus token-reduction techniques to lower latency and increase throughput.Source: §1 · p. 1
  • The model replaces the dense 12B backbone with a 30B-A3B MoE, adds native audio, uses dynamic image resolution, compresses video temporal tokens 2×, and extends context from 128K to 256K.The backbone change is intended to improve long multimodal sequence processing and inference throughput; dynamic resolution replaces tiling and better preserves native aspect ratios.Source: §1 · p. 1
  • A multi-stage training strategy progressively introduces modalities and scales context length to preserve text reasoning, mitigate catastrophic forgetting, and stabilize cross-modal alignment.The approach addresses modality alignment, training stability, and data balancing across heterogeneous sources.Source: §1 · p. 2
  • Nemotron 3 Nano Omni gains over Nemotron Nano V2 VL and ranks at or near the top on OCRBench-V2 [Liu et al., 2024], MMLongBench-DOC [Ma et al., 2024], VoiceBench [Chen et al., 2024a], WorldSense [Hong et al., 2026], and DailyOmni [Zhou et al., 2026].The introduction highlights leading results in document understanding, audio-visual reasoning, and audio benchmarks.Source: §1 · p. 2
  • On NVIDIA B200, Nemotron 3 Nano Omni delivers 3× higher single-stream output token throughput than Qwen3-Omni [Xu et al., 2025] and 9× higher throughput per GPU at a fixed interactivity target.Compared with Nemotron Nano V2 VL, it provides 3× higher throughput at the same interactivity target and 2× higher single-stream output token throughput; it is also the most cost-efficient open video understanding model on MediaPerf.Source: §1 · p. 2
  • The authors release BF16, FP8, and NVFP4 model checkpoints, along with portions of the training datasets, pipelines, and code.Released materials include about 6.9 million Nemotron-Image-Training-v3 samples, data-generation pipeline examples, Megatron-Bridge training code, and a NeMo RL guide.Source: §1 · p. 2

2. Model Architecture

Nemotron 3 Nano Omni combines modality-specific vision and audio encoders with the Nemotron 3 Nano language model via MLP projectors, then concatenates their tokens with text for the LLM.

2. Model Architecture

  • The model uses an encoder-projector-decoder that connects Nemotron 3 Nano 30B-A3B (NVIDIA et al., 2025a) with modality-specific vision and audio encoders through MLP projectors.Vision uses C-RADIOv4-H (Ranzinger et al., 2026; Heinrich et al., 2025), and audio is initialized from Parakeet-TDT-0.6B-v2 (Xu et al., 2023; Rekesh et al., 2023; Sekoyan et al., 2025).Source: §2 · p. 2
  • Dynamic-resolution image processing preserves native aspect ratio instead of using Nemotron Nano V2 VL's tiling strategy (NVIDIA et al., 2025c).Images use 16 × 16 patches, with 1,024–13,312 visual tokens per image; a 4× pixel-shuffle downsampling reduces tokens before projection. For square images, the token limits correspond to 512 × 512 and 1840 × 1840 pixels, respectively.Source: §2 · p. 2
  • Video frames use a Conv3D patch embedder that compresses every two frames into one, reducing video tokens by 2×.The architecture also optionally uses Efficient Video Sampling for higher throughput.Source: §2 · p. 2, Figure 1 · p. 3
  • Audio is encoded with a FastConformer-based Parakeet encoder after resampling to 16 kHz mono.Log-mel features use a 10 ms hop and three stride-2 convolutional subsampling layers, yielding ∼8× temporal downsampling, about 12.5 tokens per second (∼80 ms per token), and ∼375 tokens per 30-second clip. Training covers 0.5-second to 20-minute inputs, while context supports over 5 hours; shorter streams are not padded, and the final clip may be shorter than 30 seconds.Source: §2 · p. 3
  • For inputs with both visual and audio streams, modality tokens are interleaved in temporal order to enable joint temporal reasoning.Source: §2 · p. 3

3. Training Recipe & Datasets

The training recipe uses staged training: supervised fine-tuning progressively aligns modalities and expands instruction-following and context capacity, then reinforcement learning refines reasoning and safety.

3. Training Recipe & Datasets

  • Training proceeds in two stages: supervised fine-tuning (SFT) is followed by reinforcement learning (RL).Figure 2 illustrates the progression.Source: §3 · p. 3
  • SFT progressively aligns modalities, improves multimodal instruction-following, and extends context capacity.Source: §3 · p. 3
  • RL further refines reasoning and safety after SFT.Source: §3 · p. 3

3.1. SFT

SFT uses a seven-stage curriculum that progressively introduces modalities and lengthens context to promote stable cross-modal alignment, mitigate catastrophic forgetting, and improve multimodal understanding.

3.1.1. Stage 0: Vision projector warmup

  • Stage 0 trains only the vision MLP projector with other components frozen, using approximately 9.35M vision-text samples (∼15.5B tokens) across captioning, grounding, OCR, documents, GUI, and visual QA.The maximum context length is 16,384 tokens.Source: §3.1.1 · pp. 3–4

3.1.2. Stage 1: Vision SFT 16k

  • Stage 1 jointly fine-tunes the vision encoder and language model on approximately 86.3M samples (∼214.8B tokens), expanding data quality, reasoning traces, and domain and language coverage.Enhancements include re-annotating noisy labels, adding human- and model-generated reasoning traces, and combining public, curated, and synthetic data across GUI, grounding, charts, tables, documents, and video.Source: §3.1.2 · pp. 4–5

3.1.3. Stage 2: Audio projector warmup

  • Stage 2 warms up the audio projector while keeping the LLM, vision encoder, and Parakeet-TDT audio encoder frozen, using approximately 59.2M ASR samples (∼11.4B tokens).The Granary v1.1 data spans varied acoustic conditions, speaking styles, and languages.Source: §3.1.3 · p. 5

3.1.4. Stage 3: Audio projector & encoder

  • Stage 3 jointly trains the audio encoder and projector on expanded ASR, sound, music, and speech-understanding data, including captions, QA, and some reasoning traces.Synthetic audio captions and metadata are produced with open models and specialized music tools, then used to generate QA pairs.Source: §3.1.4 · p. 5

3.1.5. Stage 4: Omni SFT 16k

  • Stage 4 is the first all-modality stage with all parameters trainable, combining vision, text, safety, video, audio-video QA and captioning, ASR, and audio reasoning.Its largest data sources are vision (30.6B tokens), audio (9.7B), and short video (6.3B); videos under two minutes supply synthetic audio-visual QA and captions.Source: §3.1.5 · p. 6

3.1.6. Stage 5: Omni SFT 48k

  • At 49,152-token Stage 5, training emphasizes medium- and long-context video, omni-modal data, and reasoning traces while downsampling short video and omni data.The stage also incorporates safety data from Nemotron 3 Super and synthesizes omni-modal QA and reasoning traces from 20-second video clips.Source: §3.1.6 · pp. 6–7

3.1.7. Stage 6: Omni SFT 256k

  • 262,144-token Stage 6 uses approximately 34.0B tokens to strengthen long-context reasoning and document understanding, including documents spanning 10 to 100+ pages.The data covers text-only and vision domains; the audio encoder and projector are frozen to focus model capacity on long-context text and document understanding.Source: §3.1.7 · p. 7

3.1.8. Training Details

  • SFT training uses BF16 mixed precision, sequence packing, and tensor and expert parallelism; later stages add context parallelism to accommodate longer sequences.Videos can use up to 64 frames in Stages 1 and 4 and up to 256 in Stages 5 and 6, with patch-count augmentation to improve temporal information without increasing token count.Source: §3.1.8 · pp. 7–8

3.2. Reinforcement Learning

After SFT, the model undergoes a five-stage RL curriculum to improve instruction following, reasoning, and safety alignment across text, image, and video. Its staged training combines preference and text-only optimization with visual and unified multimodal RL.

3.2. Reinforcement Learning

  • The overall post-training curriculum proceeds through Preference Optimization, Text-RL-stage-1, Image-RL, Omni-RL, and Text-RL-stage-2.The rounds follow SFT and target instruction following, reasoning, and safety alignment across text, image, and video modalities.Source: §3.2 · p. 8

3.2.1. Preference Optimization

  • Preference Optimization uses Mixed Preference Optimization (MPO) (Wang et al., 2024a), combining Direct Preference Optimization (DPO) (Rafailov et al., 2023) with Binary Classifier Optimization (BCO) (Wang et al., 2024a).Vision-domain candidate responses are generated by rejection sampling and labeled positive or negative according to outcome correctness.Source: §3.2.1 · p. 8

3.2.2. Text-RL

  • Text-only RL updates LM parameters through multi-environment RLVR/RLHF and freezes input-token embeddings during text stages to mitigate representational drift.The approach reuses RL data and infrastructure from post-training Nemotron 3 Nano and Super (NVIDIA et al., 2025b, 2026).Source: §3.2.2 · pp. 8–9

3.2.3. Image RL

  • ImageRL trains outcome-based visual reasoning with verifiers and format rewards designed to distinguish answer correctness from common response-format errors.Tasks include chart, document, and text-rich image reasoning; STEM and math; games and puzzles; visual question answering; and visual grounding. Prompts with initial-policy pass rates of 0.8 or higher are discarded, and unanswerable or image-text-mismatched prompts teach abstention; the resulting corpus and verifier suite also feed OmniRL.Source: §3.2.3 · p. 9

3.2.4. Omni-RL

  • Omni-RL unifies reinforcement learning across image, video, and audio modalities to strengthen coherent cross-modal reasoning.Its corpus contains approximately 120K prompts across 113 sub-datasets, including image, video, audio, and text-only reasoning; ASR receives a verifier reward of 1 − WER after text normalization. Prompts are filtered by initial-policy pass rate (0.1–0.9, or 0.3–0.7 for AudioQA), with approximately 4K unanswerable or mismatched samples for abstention.Source: §3.2.4 · pp. 9–10

3.2.5. RL Training Details

  • RL training uses adapted Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025; Shao et al., 2024) with distributed multimodal training and deduplication.The setup uses NVIDIA B200 and H100 clusters, a global batch size of 4,096, and 16 rollouts per prompt; it also uses tensor, expert, and context parallelism and AdamW with β1 = 0.9 and β2 = 0.999.Source: §3.2.5 · p. 10

4. Experiments · 4.1. Visual Evaluations · 4.2. Audio Evaluations

The experiments assess vision, audio, and text reasoning, with additional analyses of video-sampling efficiency and quantization. Nemotron 3 Nano Omni improves over Nemotron Nano V2 VL across visual benchmarks and outperforms Qwen-family models on ASR and VoiceBench.

4. Experiments

  • The experiments evaluate reasoning over vision, audio, and text; separate analyses examine Efficient Video Sampling and quantization effects on efficiency and accuracy.The broader evaluation spans Sections 4.1–4.4, with EVS analysis in Section 4.6 and quantization analyses in Sections 4.7–4.8.Source: §4 · p. 10
  • Vision and audio evaluations use VLMEvalKit with a vLLM backend, while text evaluations use NeMo-Skills.The framework references are Duan et al. (2025) and Kwon et al. (2023).Source: §4 · p. 10

4.1. Visual Evaluations

  • Visual evaluations cover STEM reasoning, document understanding, OCR and charts, visual grounding and spatial reasoning, GUI understanding, and video understanding.The listed benchmarks include MMMU, MMLongBench-Doc, ScreenSpot, OSWorld, and Video-MME.Source: §4.1 · p. 10
  • Nemotron 3 Nano Omni improves over Nemotron Nano V2 VL across visual benchmarks and outperforms Qwen3-Omni in several categories.Table 7 compares Nemotron 3 Nano Omni with Nemotron Nano V2 VL and other omni-modal models.Source: §4.1 · p. 11, Table 7 · p. 11

4.2. Audio Evaluations

  • Audio evaluations span ASR, audio understanding, and voice interaction and reasoning, including long-form transcription and spoken-assistant safety.They use OpenASR’s English subset and TED-LIUM Longform for ASR, MMAU for audio QA, and VoiceBench for spoken interactions.Source: §4.2 · p. 11
  • On ASR and VoiceBench benchmarks, Nemotron 3 Nano Omni outperforms Qwen family models.The audio evaluation also covers audio understanding.Source: §4.2 · p. 11

4.3. Audio-Visual Evaluations · 4.4. Text-only evaluations

Sections 4.3 and 4.4 evaluate audio-visual and text-only capabilities using benchmark comparisons and specified evaluation protocols. The text-only evaluation aims to preserve the backbone’s benchmark performance while adding vision and audio understanding.

4.3. Audio-Visual Evaluations

  • DailyOmni (Zhou et al., 2026) tests cross-modal reasoning with 684 videos and 1,197 multiple-choice questions across six tasks, including temporal alignment and causal reasoning.The videos are segmented into 30- and 60-second clips; tasks also test event understanding and cross-modal consistency.Source: §4.3 · p. 11
  • Table 8 compares open-source models on ASR, long-form ASR, MMAU, and VoiceBench; ASR uses word error rate, where lower is better, while MMAU and VoiceBench scores are higher-is-better.ASR and MMAU use non-reasoning settings, while VoiceBench uses reasoning; the supplied passage gives no scores.Source: Table 8 · p. 12
  • Table 9 compares Nemotron 3 Nano Omni with other open-source models on Video+Audio (Omni) benchmarks, measured by accuracy, where higher is better.The supplied passage gives the comparison scope and metric direction, but no benchmark scores.Source: Table 9 · p. 12

4.4. Text-only evaluations

  • Text-only evaluations used a maximum output length of 131,072 tokens, temperature 1.0, and top-p 1.0.Source: §4.4 · p. 12
  • Text-only reporting averages Pass@1 over 8 AIME-2025 runs and 4 GPQA-Diamond (Rein et al., 2023) runs, while other listed benchmarks use one run.Single-run benchmarks are SciCode (Tian et al., 2024), LiveCodeBench v5 (07/24 - 05/25) (Jain et al., 2024), IFBench (Zhou et al., 2023), and TauBench V2 (telecom); MMLU-Pro assesses general academic and knowledge-intensive performance.Source: §4.4 · p. 12
  • The text-only evaluation aims to maintain the Nemotron 3 Nano 30B-A3B LLM backbone’s text-benchmark performance while adding vision and audio understanding.Table 10 compares Nemotron 3 Nano Omni, Nemotron 3 Nano LLM, and Qwen3-Omni across selected text-only benchmarks; the supplied passage gives no scores.Source: §4.4 · p. 13, Table 10 · p. 13

4.5. Reasoning budget control · 4.6. Conv3D and Efficient Video Sampling (EVS)

Reasoning-budget control yields selective accuracy gains in reasoning-on mode without observed degradation elsewhere. For video, Conv3D and EVS reduce input-token costs and improve time to first token (TTFT).

4.5. Reasoning budget control

  • The reasoning-budget comparison uses a 16,384-token maximum sequence length in both configurations; reasoning-on adds a 13K reasoning budget and 1,024-token grace period.The table compares base and reasoning-enabled settings.Source: §4.5 · p. 13, Table 11 · p. 13
  • Reasoning-budget adjustment yields accuracy gains on select benchmarks in reasoning-on mode, with no observed degradation on the others; gains may reflect early termination of malformed loops or truncation of overly verbose reasoning.The proposed explanations concern out-of-distribution tasks and problems requiring minimal or straightforward reasoning.Source: §4.5 · p. 13

4.6. Conv3D and Efficient Video Sampling (EVS)

  • Conv3D fuses every T=2 consecutive frames into a tubelet before the first ViT block, while runtime-only EVS retains the globally most-dissimilar tokens up to pruning rate q and pins the first tubelet as an anchor.Conv3D reduces tokens through the temporal dimension; EVS prunes in the spatial dimension after the ViT blocks and vision adapter.Source: §4.6 · p. 13
  • The Conv3D/EVS comparison fixes EVS at q=0.5 and reasoning off, reports benchmark accuracy at 128 and 256 sampled frames, and averages accuracy over three runs and TTFT over five concurrency-1 runs.TTFT runs use a synthetic 512-frame, 512 × 512 video at 30 fps.Source: §4.6 · pp. 13–14, Table 12 · p. 14
  • On BF16, TTFT falls from 7969 ms baseline to 5984 ms with Conv3D, 6452 ms with EVS, and 5313 ms stacked (−33% versus baseline), at about half a point of average accuracy cost.The same ordering holds on NVFP4.Source: Table 12 · p. 14, §4.6 · p. 14
  • For a 512-frame video, LLM input tokens fall from ∼141k with neither mechanism to ∼75k with Conv3D (−47%), and ∼42k with Conv3D plus EVS at q=0.5 (−70% versus baseline).Source: §4.6 · p. 14
  • At q=0.7, TTFT is reduced by ∼14% versus no-EVS while accuracy is essentially flat through q=0.7; accuracy slightly reduces at q=0.8 and drops beyond, with LongVideoBench most sensitive.The sweep uses BF16 with Conv3D enabled and reasoning off.Source: Table 13 · p. 14, §4.6 · p. 14

4.7. Quantization · 4.8. Inference efficiency

Nemotron 3 Nano Omni’s mixed-precision quantization reduces model-weight footprint while keeping median accuracy drops below 1% versus BF16 across 25 benchmarks. On NVIDIA B200, it also delivers higher single-stream and serving throughput than Qwen3-Omni and Nemotron Nano V2 VL.

4.7. Quantization

  • The NVFP4 recipe uses 4.98 effective bits per weight and 20.9 GB versus 61.5 GB for BF16; FP8 uses approximately 8.5 bpw and 32.8 GB.NVFP4 uses NVFP4 for routed MoE experts, FP8 for selected projections and shared experts, and BF16 for remaining language-model layers and vision/audio encoders and projectors; FP8 quantizes language-model linear layers except the MoE router and lm_head. Both use an FP8 KV cache.Source: §4.7 · p. 14
  • Median accuracy drops by less than 1% versus BF16 for both FP8 and NVFP4 across 25 text, image, video, and audio benchmarks.Table 14 presents accuracy across the evaluation suite for BF16, FP8, and NVFP4.Source: §4.7 · p. 15, Table 14 · p. 15

4.8. Inference efficiency

  • NVFP4 reaches up to 7.5× BF16 output-token throughput at iso-interactivity on single-image reasoning, at 18,200 versus 2,400 tok/s and 150 tok/s/user.The comparison is on NVIDIA B200.Source: §4.8 · p. 15
  • At concurrency 1, single-stream generation exceeds 500 output tokens/s, approximately 2.4–2.9× Qwen3-Omni’s 175–210 output tokens/s and 2× Nemotron Nano V2 VL’s 250 output tokens/s.The rate is sustained with longer sequences and larger multimodal inputs, including long videos and multi-document workloads.Source: §4.8 · p. 15
  • For multi-document workloads, time-to-first-token is approximately 1.3 s versus more than 2.5 s for Qwen3-Omni.Source: §4.8 · p. 15
  • At 50 output tokens/s per user, throughput is 9× Qwen3-Omni on long-video workloads, 7.5× on multi-document workloads, and 3× Nemotron Nano V2 VL.Measurements used a single NVIDIA B200; the multi-document workload contains 32 images, and the long-video workload contains 512 frames.Source: §4.8 · p. 15
  • At maximum concurrency on one NVIDIA B200, Nemotron 3 Nano Omni reaches 5,000 output tokens/s on a multi-document workload.At the 50 output tokens/s per-user target, its throughput is 7.5× Qwen3-Omni on multi-document workloads and 9× on long-video workloads.Source: §4.8 · p. 15

5. Conclusion

Nemotron 3 Nano Omni extends the Nemotron multimodal family with native audio support, stronger multimodal reasoning, and efficient processing of long, heterogeneous inputs. It combines a multimodal architecture and staged training with broad benchmark gains and open releases intended to support further research.

5. Conclusion

  • Built on the Nemotron 3 Nano 30B-A3B MoE hybrid backbone, the model combines C-RADIOv4-H and Parakeet-TDT encoders, dynamic image resolution, Conv3D video compression, and 256K context.Its multi-stage training progressively introduces modalities and extends context to support cross-modal alignment while preserving the base LLM's text reasoning ability.Source: §5 · p. 16
  • Nemotron 3 Nano Omni improves consistently over Nemotron Nano V2 VL, leads or competes across document, GUI, audio-video, and voice benchmarks, and retains the backbone's text reasoning performance.Evaluations include OCRBench-V2, MMLongBench-Doc, ChartQA, CharXiv, ScreenSpot, ScreenSpot-Pro, OSWorld, WorldSense, DailyOmni, and VoiceBench.Source: §5 · p. 16
  • Multimodal token reduction enables substantially lower inference latency and several-fold higher throughput than comparably sized models; BF16, FP8, and FP4 checkpoints, training data, and code are released.The release aims to enable further community research and development in efficient omni-modal modeling.Source: §5 · p. 16

Related papers

Loading 2604.24954v2…