PaperlayerView exact revision on arXiv

Source-linked AI summary

NVIDIA Nemotron Nano V2 VL

NVIDIA, :, Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Guo Chen, Karan Sapra, Zhiding Yu, Adi Renduchintala, Charles Wang, Peter Jin, Arushi Goel, Mike Ranzinger, Lukas Voegtle, Philipp Fischer, Timo Roman, Wei Ping, Boxin Wang, Zhuolin Yang, Nayeon Lee, Shaokun Zhang, Fuxiao Liu, Zhiqi Li, Di Zhang, Greg Heinrich, Hongxu Yin, Song Han, Pavlo Molchanov, Parth Mannan, Yao Xu, Jane Polak Scowcroft, Tom Balough, Subhashree Radhakrishnan, Paris Zhang, Sean Cha, Ratnesh Kumar, Zaid Pervaiz Bhat, Jian Zhang, Darragh Hanley, Pritam Biswas, Jesse Oliver, Kevin Vasques, Roger Waleffe, Duncan Riach, Oluwatobi Olabiyi, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Pritam Gundecha, Khanh Nguyen, Alexandre Milesi, Eugene Khvedchenia, Ran Zilberstein, Ofri Masad, Natan Bagrov, Nave Assaf, Tomer Asida, Daniel Afrimi, Amit Zuker, Netanel Haber, Zhiyu Cheng, Jingyu Xin, Di Wu, Nik Spirin, Maryam Moosaei, Roman Ageev, Vanshil Atul Shah, Yuting Wu, Daniel Korzekwa, Unnikrishnan Kizhakkemadam Sreekumar, Wanli Jiang, Padmavathy Subramanian, Alejandra Rico, Sandip Bhaskar, Saeid Motiian, Kedi Wu, Annie Surla, Chia-Chih Chen, Hayden Wolff, Matthew Feinberg, Melissa Corpuz, Marek Wawrzos, Eileen Long, Aastha Jhunjhunwala, Paul Hendricks, Farzan Memarian, Benika Hall, Xin-Yu Wang, David Mosallanezhad, Soumye Singhal, Luis Vega, Katherine Cheung, Krzysztof Pawelec, Michael Evans, Katherine Luna, Jie Lou, Erick Galinkin, Akshay Hazare, Kaustubh Purandare, Ann Guan, Anna Warno, Chen Cui, Yoshi Suhara, Shibani Likhite, Seph Mard, Meredith Price, Laya Sleiman, Saori Kaji, Udi Karpas, Kari Briski, Joey Conway, Michael Lightstone, Jan Kautz, Mohammad Shoeybi, Mostofa Patwary, Jonathen Cohen, Oleksii Kuchaiev, Andrew Tao, Bryan Catanzaro

arXiv:2511.03929v2cs.LGcs.AIcs.CV

TL;DR

Real-world document understanding, long-video comprehension, and reasoning motivate Nemotron Nano V2 VL, which builds on a hybrid Mamba-Transformer LLM with staged training and efficient video sampling. The model reports improvements over its predecessor across multimodal and text tasks, alongside faster throughput in long-document and video settings.

  • Problem

    Nemotron Nano V2 VL targets strong real-world document understanding, long-video comprehension, and reasoning, areas addressed through improvements over the previous model across vision and text domains.

    Source: §1 · p. 1

  • Method

    The model combines a Nemotron Nano V2 hybrid Mamba-Transformer LLM with multimodal staged SFT and Efficient Video Sampling.

    Source: §1 · p. 1

  • Results

    The model reports substantial improvements over Llama-3.1-Nemotron-Nano-VL-8B across multimodal and text tasks, including 35% higher throughput in long multi-page document understanding.

    Source: §1 · p. 1, §5 · p. 14

  • Takeaways & Limitations

    The authors release BF16, FP8, and FP4 weights along with substantial portions of training data, recipes, and code to support continued research and development.

    Source: §1 · p. 1

Abstract

from arXiv · show

We introduce Nemotron Nano V2 VL, the latest model of the Nemotron vision-language series designed for strong real-world document understanding, long video comprehension, and reasoning tasks. Nemotron Nano V2 VL delivers significant improvements over our previous model, Llama-3.1-Nemotron-Nano-VL-8B, across all vision and text domains through major enhancements in model architecture, datasets, and training recipes. Nemotron Nano V2 VL builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, and innovative token reduction techniques to achieve higher inference throughput in long document and video scenarios. We are releasing model checkpoints in BF16, FP8, and FP4 formats and sharing large parts of our datasets, recipes and training code.

NVIDIA

The NVIDIA entry presents model-format and dataset headings.

  • The entry lists model formats BF16, FP8, and NVFP4-QAD alongside a Dataset heading.Source: NVIDIA · p. 1

1. Introduction

Nemotron Nano V2 VL is an efficient 12B vision-language model with reported gains over its predecessor across multimodal and text tasks. It combines a hybrid Mamba-Transformer backbone, staged training, extended context, and efficient video sampling, while releasing model weights and training resources.

1. Introduction

  • The model reports substantial multimodal and text-understanding improvements over Llama-3.1-Nemotron-Nano-VL-8B, including strong performance across reasoning, document, video, visual-question-answering, and STEM tasks.It reports OCRBench v2 private-data leaderboard accuracy and attributes gains to model, dataset, and training improvements.Source: §1 · p. 1
  • Context length expands from 16K to 128K tokens to improve handling of long videos and complex reasoning tasks.Source: §1 · p. 1
  • The release includes BF16, FP8, and FP4 model weights, substantial portions of SFT data, recipes, and code.Released data and tooling include a dataset of over 8 million samples and the NVPDFTex compiler toolchain for annotated OCR ground truth.Source: §1 · pp. 1–2
  • The model builds on a 12B Nemotron Nano V2 reasoning LLM and RADIOv2.5 vision encoder, using multimodal fusion and staged SFT across vision and text domains.Additional SFT stages target video understanding, long-context performance, and recovery of text-only capabilities.Source: §1 · p. 1
  • 35% higher throughput is reported for long multi-page document understanding versus Llama-3.1-Nemotron-Nano-VL-8B, while EVS accelerates video throughput by 2x or more with minimal or no accuracy impact.Source: §1 · p. 1
  • Reasoning-on and reasoning-off modes are supported, with reasoning-on enabling extended reasoning for more complex problem-solving and balancing efficiency with task performance.Source: §1 · pp. 1–2

2. Model Architecture

The model combines a vision encoder, an MLP projector, and a language model, processing images and video through a shared visual-to-language path. Image tiling adapts to resolution, while video inputs use sampled frames.

2. Model Architecture

  • Image tile count varies with aspect ratio and video frames are sampled uniformly; resized visual inputs pass through RADIO and an MLP connector before interleaving with text embeddings for the LLM.Source: Figure 1 · p. 2
  • The architecture comprises a vision encoder, an MLP projector, and a language model initialized from RADIOv2 and Nemotron-Nano-12B-V2, respectively.Source: §2 · p. 2
  • Images are resized and divided into 512-pixel tiles, with pixel shuffle reducing each tile from 1024 to 256 visual tokens; training allows up to 12 tiles plus a global-context thumbnail.Source: §2 · p. 3

3. Training Recipe & Datasets

The training recipe progresses from connector alignment to multimodal and text SFT, then extends context and targets code and long-context recovery. Training details address variable sequence lengths, numerical precision, and parallelism.

3.1. Stage 0

  • Stage 0 freezes the vision encoder and language model while training the MLP connector on about 2.2 million multimodal samples, up to 36 billion tokens, across captioning, VQA, grounding, OCR, and document extraction.Source: §3.1 · p. 3

3.2. SFT Stage 1: 16K context length

  • In SFT Stage 1, all components are unfrozen for training at a 16,384-token maximum sequence length on approximately 32.5 million samples, including 6.5 million text-reasoning samples to preserve backbone text comprehension.The multimodal datasets total 26 million samples (approximately 72 billion tokens); the corpus also includes human-annotated and model-generated reasoning traces.Source: §3.2 · pp. 3–4

3.3. SFT Stage 2: 49K context extension

  • Stage 2 extends context to 49,152 tokens using about 11 million samples and 55 billion tokens; a 25% reuse of Stage 1 data balances training efficiency and accuracy across text, vision, multi-frame, and video benchmarks.It adds about 1.4 million video and multi-image samples, with non-QA data converted into QA formats using templates or synthesized questions and answers.Source: §3.3 · pp. 4–5

3.4. SFT stage 3: 49K text recovery

  • Stage 3 uses 1 million code-reasoning samples, or 15 billion tokens, at 49,152-token maximum sequence length to recover text-code performance after Stages 1 and 2.Source: §3.4 · p. 5

3.5. SFT stage 4: 300K context extension

  • Stage 4 trains on about 74,000 long-context samples averaging 160K tokens, with a 311,296-token maximum sequence length; this stage improves accuracy on long-context benchmarks such as RULER.Source: §3.5 · p. 5

3.6. Training details

  • Training uses FP8 across stages while retaining selected LLM and vision-encoder layers in BF16; the authors report no instabilities and benchmark scores close to full BF16.Source: §3.6 · p. 5
  • Video training inputs use 2 frames per second up to 128 frames, with longer videos uniformly sampled to 128 frames; online sequence packing reduces padding tokens for variable-length samples.Source: §3.6 · pp. 5–6
  • Training details include loss square-averaging, AdamW with β1=0.9 and β2=0.999, cosine annealing with linear warmup, and context parallelism for long sequences.Stages 2–3 use 2-way and Stage 4 uses 8-way context parallelism; training uses Megatron, Transformer Engine, Megatron Energon, and H100 GPUs.Source: §3.6 · p. 6, Figure 2 · p. 6

4. Experiments

The experiments assess multimodal and pure-text performance, training-stage effects, reasoning budgets, video sampling, image processing, and quantization across 45 benchmarks in seven categories.

4.1. Multimodal Evaluations

  • Nemotron Nano V2 VL improves consistently over Llama-3.1-Nemotron-Nano-VL-8B across benchmarks, with gains attributed to its backbone, datasets, and training recipe.Table 4 compares the previous-generation VLM with the new model after each training stage.Source: Table 4 · p. 9, §4.1 · p. 9

4.2. Pure Text Evaluations

  • LiveCodeBench drops from 70.0 to 50.87 after Stage 1; Stage 3 restores it, while RULER improves across Stages 2–4 and final text reasoning is largely preserved.Vision benchmarks remain stable between SFT Stages 2 and 3.Source: Table 6 · p. 10, §4.2 · p. 11

4.3. Reasoning Budget Control

  • Tuned reasoning budgets improve reasoning-on accuracy across multiple tasks, potentially by ending looping or overly verbose traces; unrestricted reasoning can score below reasoning-off.The authors suggest early termination of malformed loops and truncation of unnecessary reasoning as possible explanations.Source: §4.3 · pp. 11–12

4.4. Inference with Efficient Video Sampling

  • EVS prunes temporally static patches while preserving positional identity and semantics; higher EVS ratios lower TTFT and raise throughput with minor accuracy impact in BF16 and FP8.Figure 4 presents the BF16 and FP8 ablations, including accuracy, time-to-first-token, and throughput.Source: §4.4 · p. 12, Figure 4 · p. 12

4.5. Image Processing Ablations

  • Native-resolution processing matches or exceeds tiling on several benchmarks, but OCRBench and OCRBench-V2 (English) drop; size-and-aspect-ratio matching recovers OCRBench alone.The OCRBench-V2 (English) gap persists and is identified for future investigation.Source: §4.5 · p. 13, Table 7 · p. 13

4.6. Quantization

  • PTQ and QAD produce deployable checkpoints to bridge training’s dynamic FP8 scaling and inference stacks’ static quantization; QAD addresses the small accuracy drop from NVFP4 PTQ.Table 8 compares accuracy across precisions and strategies in vLLM.Source: §4.6 · pp. 13–14, Table 8 · p. 14

5. Conclusion

The conclusion presents Nemotron Nano V2 VL as an efficient 12B vision-language model whose multi-stage training improves multimodal and text understanding while aiming to preserve the backbone’s text capabilities.

5. Conclusion

  • Nemotron Nano V2 VL is an efficient 12B vision-language model built upon the Nemotron-Nano-V2 LLM.The authors report improvements in multimodal understanding, text understanding, and reasoning relative to their predecessor.Source: §5 · p. 14
  • The multi-stage training strategy enhances visual understanding while preserving the original backbone’s text comprehension abilities.The paper also investigates efficient video sampling at inference and alternative image-processing pipelines.Source: §5 · p. 14
  • The authors release model weights in BF16, FP8, and FP4 formats, plus substantial portions of the SFT dataset and associated tooling.Source: §5 · p. 14

6. Contributors

The contributor roster is organized into five listed groups spanning model training and data, systems, inference, evaluation, and leadership.

6. Contributors

Loading 2511.03929v2…