PaperlayerView exact revision on arXiv

Source-linked AI summary

NVLM: Open Frontier-Class Multimodal LLMs

Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

arXiv:2409.11402v2cs.CLcs.AIcs.CVcs.LGcs.MM

TL;DR

Open multimodal LLM comparisons are difficult because proprietary architectures and training methods are undisclosed, while open models vary in key design and data choices. NVLM-1.0 compares architectures and introduces a hybrid model, high-resolution tile tagging, and curated training data. The authors report competitive vision-language performance alongside preserved or improved text-only performance after multimodal training.

  • Problem

    Proprietary multimodal models’ architectures, training data, and methods are undisclosed, and differences among open models make direct architectural comparisons challenging.

    Source: §1 · p. 4, §5.2 · p. 14

  • Method

    NVLM-1.0 compares decoder-only and cross-attention designs and introduces a hybrid architecture, 1-D tile tagging, and curated multimodal and text-only training data.

    Source: §1 · pp. 5–6

  • Results

    NVLM-1.0 achieves state-of-the-art vision-language results rivaling leading models while preserving text-only performance and improving text-only math and coding benchmarks after multimodal training.

    Source: §1 · p. 6, §7 · p. 19

  • Takeaways & Limitations

    The paper concludes that its architecture, high-resolution input, and data-curation findings provide insights for developing multimodal models with strong vision-language and text-only capabilities.

    Source: §7 · p. 19

    Because benchmark training splits are included in SFT, those benchmark evaluations are not zero-shot; the proprietary models’ evaluation settings are unknown.

    Source: §5.2 · pp. 14–15

Abstract

from arXiv · show

We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g., Flamingo). Based on the strengths and weaknesses of both approaches, we propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for tile-based dynamic high-resolution images, which significantly boosts performance on multimodal reasoning and OCR-related tasks. Regarding training data, we meticulously curate and provide detailed information on our multimodal pretraining and supervised fine-tuning datasets. Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures. Notably, we develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks while maintaining and even improving text-only performance compared to their LLM backbones. To achieve this, we craft and integrate a high-quality text-only dataset into multimodal training, alongside a substantial amount of multimodal math and reasoning data, leading to enhanced math and coding capabilities across modalities. To advance research in the field, we release the model weights at https://huggingface.co/nvidia/NVLM-D-72B and will open-source the training code for the community soon.

NVIDIA

The supplied material identifies the NVLM-D1.0 72B model and highlights its broad multimodal capabilities and comparison with leading models.

NVIDIA

  • The listed contributors are marked as equal contributors, and one contributor is identified as leading the effort.Source: NVIDIA · p. 1
  • Figure 1 presents examples of chart and table understanding, OCR, localization, image description, meme and scene understanding, math reasoning, and coding.Source: Figure 1 · p. 3
  • Figure 2 reports that NVLM 1.0 performs on par with leading models across vision-language and text-only tasks, with a text-only math and coding gain for NVLM-D1.0 72B.The caption reports a 4.3-point average accuracy increase after multimodal training.Source: Figure 2 · p. 4

1 Introduction

The introduction identifies gaps in fair architectural comparison, high-resolution reasoning, and text-only retention in open multimodal models. NVLM-1.0 addresses these with three architectures, tile tagging, curated data, and a training approach that preserves or improves text-only performance.

1 Introduction

  • The introduction frames the gap as limited transparency about proprietary systems and difficulty comparing open models with differing architectures, backbones, encoders, and training data.Source: §1 · pp. 4–5
  • NVLM-1.0 introduces decoder-only, cross-attention, and hybrid models, reporting state-of-the-art results that rival leading proprietary and open-access multimodal LLMs.Source: §1 · p. 5
  • A matched comparison finds cross-attention NVLM-X more computationally efficient on high-resolution images, while decoder-only NVLM-D offers unified reasoning and higher OCR-task accuracy.The proposed NVLM-H combines multimodal reasoning strengths with improved high-resolution computational efficiency.Source: §1 · p. 5
  • A text-based 1-D tile tag before each tile’s image tokens achieves the best accuracy in ablations, targeting both OCR and multimodal reasoning tasks.Source: §1 · p. 5
  • The authors report that dataset quality and task diversity matter more than scale, including in pretraining, and that abundant, diverse pretraining data also benefits decoder-only models.They provide detailed information on pretraining and SFT data and describe a larger task-oriented SFT collection than previous studies.Source: §1 · p. 5
  • NVLM’s text-only dataset in multimodal SFT preserves text-only performance and is associated with noticeable gains on text-only math and coding benchmarks across models.The authors attribute improved reasoning to high-quality text-only data and substantial multimodal math data.Source: §1 · p. 6

2 Qualitative Study

The qualitative study illustrates NVLM-D1.0 72B handling varied image types and tasks, including cases that combine visual recognition with reasoning.

2 Qualitative Study

  • A qualitative analysis shows NVLM-1.0 handles diverse images and instructions, including memes, object-centric and real-world scene images, handwritten pseudocode, tables, and charts.Source: §2 · p. 6
  • The model interprets a meme by recognizing its text labels and reasoning about the contrast between the depicted lynx and domestic cat.Source: §2 · p. 6
  • Examples also show location-sensitive answers and math reasoning or coding based on visual information in tables and handwritten pseudocode.Source: §2 · p. 6

3 Preliminaries

The preliminaries describe multimodal LLM components and two common architectures, then explain dynamic high-resolution tiling and its trade-off between OCR performance and reasoning.

3 Preliminaries

  • Multimodal LLMs connect visual inputs with language tasks and typically combine a large language model with a vision encoder.Source: §3 · p. 6, §3.1 · p. 6

3.1 Essential Building Blocks

  • The paper uses an instruction-tuned Qwen2-72B backbone and frozen InternViT-6B vision encoder as defaults.The paper says freezing the vision encoder simplifies training while still delivering strong results.Source: §3.1 · pp. 6–7

3.2 Architectural Designs

  • Decoder-only models project image features into the text embedding space, while cross-attention models process image tokens through cross-attention layers; NVLM-X omits Flamingo’s Perceiver module.Cross-attention can be computationally efficient for high-resolution images by avoiding long sequences of image tokens in the LLM decoder, though it is more complex to implement.Source: §3.2 · pp. 7–8

3.3 High-Resolution Inputs

  • Dynamic high-resolution input divides images into aspect-ratio-based tiles processed by the vision encoder, but concatenating all tile tokens can degrade reasoning despite improving OCR-related performance.The paper illustrates 12 tiles producing 3072 tokens for an 896 × 672 image, versus 512 tokens for a 448 × 224 image.Source: §3.3 · p. 8

4 NVLM: Models and Training Methods

NVLM-1.0 includes decoder-only, cross-attention, and hybrid models sharing a dynamic high-resolution vision pathway but handling tile features differently. The section details each design and a unified two-stage training method.

4.1 Shared Vision Pathway

  • Dynamic high-resolution processing matches each image to 1–6 local 448×448 tiles, adds a global thumbnail, and reduces each tile’s 1,024 vision tokens to 256 by pixel shuffle.The authors note that simply concatenating high-resolution tile tokens can degrade reasoning-related performance, motivating architecture-specific handling.Source: §4.1 · p. 8

4.2 NVLM-D: Decoder-only Model

  • NVLM-D connects the vision encoder to the LLM with a two-layer MLP and signals each tile’s start and position using text-based tile tags before its 256 image tokens.The tile tags distinguish the design from globally flattening tokens and inserting row-ending newlines; improvements are reported particularly as model size and training data scale.Source: §4.2 · pp. 9–10, Figure 4 · p. 10
  • 1-D tile tags generally outperform other tags, with tile tagging improving OCR tasks and moderately improving MathVista and MMMU.The authors hypothesize that 1-D tags generalize better at test time despite not encoding 2-D tile layout.Source: §4.2 · p. 10

4.3 NVLM-X: X-attention Model

  • NVLM-X reads image tokens directly through gated cross-attention, avoiding a perceiver resampler that the authors found could disrupt spatial relationships important for document OCR.During multimodal SFT, the LLM is unfrozen and a high-quality text-only dataset is blended in to maintain text-only performance.Source: §4.3 · pp. 10–11
  • With NVLM-X, adding 1-D tile tags improves performance across benchmarks, including MMMU, MathVista, and OCR-related tasks.The reported reasoning comparisons are MMMU 54.1 vs. 53.0 and MathVista 59.6 vs. 57.6 for tagged versus untagged DHR.Source: §4.3 · p. 11, Table 2 · p. 11
  • NVLM-X adds gated cross-attention parameters but processes high-resolution images more efficiently than NVLM-D, whose longer decoder sequences increase memory use and reduce throughput.NVLM-D instead processes tokens from different modalities jointly in the decoder, though long high-resolution sequences may make reasoning challenging.Source: §4.3 · p. 11

4.4 NVLM-H: Hybrid Model

  • NVLM-H sends thumbnail tokens through decoder self-attention for joint multimodal reasoning while processing regular tiles through gated cross-attention for finer image details.The authors report improved high-resolution capability versus NVLM-X and higher training throughput than NVLM-D.Source: §4.4 · p. 12

4.5 Model Configurations and Training Method

  • Across NVLM models, pretraining freezes the LLM and vision encoder while training alignment modules; SFT keeps the vision encoder frozen and trains the LLM and alignment modules with multimodal and text-only data.The shared vision encoder is InternViT-6B-448px-V1-5, and the 72B models use Qwen2-72B-Instruct as their LLM backbone.Source: §4.5 · p. 12

5 Training Data

The training data section describes curated multimodal pretraining and supervised fine-tuning datasets, alongside a text-only SFT blend intended to support vision-language capabilities without sacrificing text performance.

5.1 Multimodal Pretraining Data

  • At pretraining, dataset quality matters more than scale; larger, noisier image-text data was less effective, while interleaved datasets had minimal downstream impact.The authors suggest inaccurate image-text alignment as a possible reason and hypothesize that interleaved data needs more filtering and recaptioning.Source: §5.1 · p. 13
  • Pretraining spans captioning, natural-image and chart/document VQA, visual math reasoning, and OCR; blending small SFT datasets into pretraining caused overfitting and weaker zero-shot reasoning.The reported zero-shot benchmarks include MMMU and MathVista.Source: §5.1 · p. 13
  • Diverse pretraining data consistently improves benchmarks, particularly math reasoning and OCR-related tasks.The reported comparison uses the same high-quality SFT data after pretraining.Source: §5.1 · pp. 13–14

5.2 Multimodal SFT Data

  • The multimodal SFT blend covers varied image understanding, chart and document analysis, OCR, and visual math, with multimodal math data also improving text-only math benchmarks.The cited text-only benchmarks are GSM8K and MATH.Source: §5.2 · p. 14
  • ChartQA, DocVQA, VQAv2, TextVQA, and AI2D training splits enter SFT, so their test results are not zero-shot; no augmentation was applied.The authors say augmentation can improve benchmark scores but contributes minimal value to out-of-domain generalization.Source: §5.2 · p. 14

5.3 Text-only SFT Data

  • A curated text-only SFT dataset is added during multimodal fine-tuning to preserve backbone text performance; it combines general, math, and code data with refined responses.The prompts are decontaminated against benchmark test sets.Source: §5.3 · p. 15

6 Results

NVLM-1.0 is evaluated across vision-language and text-only benchmarks, with comparisons to leading proprietary and open-access models. The results report strong multimodal performance and improved text-only accuracy, while a frozen LLM trades some vision-language performance for retained text performance.

6.1 Benchmarks

  • Evaluation covers nine vision-language and four text-only benchmarks, spanning multimodal reasoning, visual math, image understanding, OCR, and coding.AI2D uses a default test setting that replaces image text with answer letters; its no-mask setting also includes the corresponding answer values.Source: §6.1 · pp. 15–16

6.2 Baseline Models

  • The comparison includes 72B NVLM-D, NVLM-X, and NVLM-H models and leading proprietary and open-access baselines; 34B models are used for faster, unoptimized ablations.The paper notes that its 34B results are not carefully checkpoint-selected or hyperparameter-optimized.Source: §6.2 · p. 16

6.3 Main Results

  • 853 on OCRBench and 85.4 on VQAv2 for NVLM-D are reported as the highest scores among leading models, while NVLM-H’s 60.2 on MMMU (Val) leads open-access models.The paper reports that NVLM-1.0 72B models rival leading proprietary and open-access models overall.Source: §6.3 · p. 16, Table 7 · p. 17
  • NVLM-X reports frontier-class results among cross-attention models and faster training and inference than its decoder-only counterpart.The paper also reports text-only performance improvements for NVLM-1.0, contrasting with degradation in some open-access multimodal models.Source: §6.3 · p. 17

6.4 Text-only Performance

  • NVLM-D1.0 72B average accuracy across four text-only benchmarks rises from 79.8 to 84.1 relative to its Qwen2-72B-Instruct backbone.The authors attribute improved math performance to high-quality text-only SFT and multimodal math data.Source: §6.4 · pp. 17–18

6.5 Frozen versus Unfrozen LLM during Mutimodal SFT

  • Freezing NVLM-X’s LLM retains text performance but moderately reduces vision-language performance compared with the unfrozen model.The frozen 72B model is comparable to the smaller NVLM-X-34B, with relative strengths on reasoning and natural-image understanding but weaker OCR-related results.Source: §6.5 · p. 18, Table 9 · p. 18

7 Conclusion

The conclusion presents NVLM-1.0 as a multimodal model family that achieves strong vision-language results without compromising text-only performance. Qualitative examples illustrate instruction following, scene understanding, meme interpretation, and mathematical reasoning.

7 Conclusion

  • NVLM-1.0 is reported to achieve state-of-the-art vision-language results while preserving text-only performance, with the paper summarizing insights on architecture, high-resolution inputs, and training data.The conclusion says the model weights and training code are being made available to the community.Source: §7 · p. 19

A Qualitative Examples from the NVLM-1.0-D 72B Model

  • Examples show NVLM-D 72B controlling response length, describing images, recognizing potential dangers, interpreting memes, and giving step-by-step math reasoning.These are presented as qualitative examples rather than benchmark results.Source: Figure 5 · p. 28, Figure 6 · p. 28, Figure 7 · p. 29

B Training Hyperparameters

The appendix points to separate tables for pretraining and supervised fine-tuning hyperparameters.

B Training Hyperparameters

C Perceiver Resampler in Flamingo Impacts OCR Performance

The section tests whether Flamingo’s perceiver resampler affects document OCR fine-tuning. Removing it while training only cross-attention reaches zero training loss at the same iteration, unlike the original model.

C Perceiver Resampler in Flamingo Impacts OCR Performance

D Evaluation Details of AI2D

The appendix illustrates two AI2D evaluation settings and provides examples of task-specific pretraining formats and the ChatML template used for supervised fine-tuning.

D Evaluation Details of AI2D

E Data Formats and ChatML Tamplate

  • Figure 11 shows examples of training formats for various pretraining tasks, marking where visual features are inserted and which text is supervised by loss.The emerald <image> tag marks feature insertion; blue text marks ground truth associated with loss.Source: E Data Formats and ChatML Tamplate · p. 31, Figure 11 · p. 32
  • Figure 12 shows the ChatML template used in supervised fine-tuning, with visual-feature insertion and loss-associated ground-truth text indicated.The emerald <image> tag marks feature insertion; blue text marks ground truth associated with loss.Source: E Data Formats and ChatML Tamplate · p. 31, Figure 12 · p. 32

F Unoptimized Results Using Yi-34B as the Backbone LLM

The appendix reports reference evaluations for unoptimized Yi-34B-backbone models. Despite limited ablation-study optimization, the 34B NVLM models significantly outperform VILA-1.5 40B and Cambrian-1 34B.

F Unoptimized Results Using Yi-34B as the Backbone LLM

  • The unoptimized 34B NVLM models significantly outperform VILA-1.5 40B and Cambrian-1 34B [139].The models were used for faster ablations and iterations without detailed checkpoint selection or hyperparameter optimization.Source: F Unoptimized Results Using Yi-34B as… · p. 33
  • Table 12 evaluates 34B models on vision-language and text-only benchmarks, and defines Text-only Avg. 4 as the average accuracy change versus the backbone across four text benchmarks.The four benchmarks are MMLU, GSM8K, MATH, and HumanEval; full results are referenced in Table 8.Source: Table 12 · p. 33
  • Table 13 covers the 34B models’ results on MMLU, GSM8K, MATH, and HumanEval.Source: Table 13 · p. 33
Loading 2409.11402v2…