NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints

1 Introduction

• Under data constraints, the paper studies whether native MLLMs can feasibly achieve or surpass top-tier performance at acceptable cost.Prior evaluations often assume infinite training resources despite limited data and large-scale training challenges.Source: §1 · pp. 1–2
• Pre-trained LLM initialization benefits multimodal training convergence, while combining visual encoders and MoEs improves on a vanilla decoder-only LLM.These findings inform a meta-architecture designed to balance performance and training cost.Source: §1 · p. 2
• The optimal visual-encoder size increases approximately proportionally with LLM size in log scale, while encoder-scaling returns are bounded by LLM capacity.The authors contrast this with compositional models that use one encoder size across LLM scales.Source: §1 · p. 2
• With ~600M pre-training image-text pairs, NaViL achieves competitive performance against current top-tier compositional MLLMs; experiments also report top-tier performance with nearly 600M pre-training data.The authors describe the recipe as simple and cost-effective.Source: §1 · p. 2
• NaViL explores visual-encoder and LLM scaling and reports a positive correlation between their scales.The paper also reports large-scale pre-training and fine-tuning experiments.Source: §1 · p. 2

2 Related Work

• Most existing MLLMs follow a compositional paradigm, connecting a pretrained visual encoder and LLM through a projector before multimodal alignment fine-tuning.Prior work mainly improves visual encoders or designs connectors.Source: §2 · p. 3
• Compositional training struggles to explore joint vision-language scaling, and its training-pipeline and alignment limitations are increasingly recognized.Source: §2 · p. 3
• Native MLLMs jointly pre-train vision and language end to end, but training from scratch is costly and systematic study of architecture and scaling remains lacking.Recent work initializes from pretrained LLM checkpoints to facilitate convergence.Source: §2 · p. 3

3.1 Problem Setup

• The setup defines native MLLMs as jointly optimizing vision and language end-to-end, and studies a visual encoder, an LLM, and MoEs injected into the LLM.Its design questions concern component architecture and how to scale the visual and linguistic components.Source: §3.1 · p. 3, §3.2 · p. 3

3.2.1 Initialization of LLM

• LLM initialization improves convergence: training from scratch requires over 10x more data to reach comparable loss, and still trails on zero-shot captioning.The reported captioning gap favors initialization even when the noninitialized model receives significantly more data.Source: §3.2.1 · p. 4, Figure 2 · p. 4

3.2.2 Effectiveness of MoEs

• MoE architecture accelerates convergence, reaching the same validation loss as the vanilla LLM with only 1/10 of the data, without increasing training or inference cost.One expert is activated to maintain consistent inference costs.Source: §3.2.2 · p. 5

3.2.3 Optimizing the Visual Encoder Architecture

• Across a fixed 600M parameter budget, a range of encoder depth-width configurations performs near-optimally; shallower encoders converge faster early, while deeper ones perform slightly better with larger datasets.Extremely high or low depths perform worse than moderate configurations.Source: §3.2.3 · p. 5, Figure 4 · p. 5

3.3.1 Scaling up Visual Encoder and LLM Independently

• LLM scaling follows the conventional scaling law, while visual-encoder gains diminish beyond a size where LLM capacity constrains performance.With a fixed LLM, further encoder scaling yields only marginal loss reduction; the study examines both components independently and across training-data sizes.Source: §3.3.1 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6

3.3.2 Scaling up Visual Encoder and LLM Together

• The optimal visual-encoder size increases log-linearly with LLM size, indicating that the two components should be scaled jointly.The optimal size is the smallest encoder whose loss difference from an encoder twice its size is below 1% of the loss with the 75M encoder.Source: Figure 7 · p. 6, §3.3.2 · pp. 6–7

4.1 Architecture

• NaViL encodes images as visual tokens, concatenates them with text tokens, and feeds the multimodal sequence into the LLM.The architecture supports images of any resolution; image boundary and row tokens mark image extent and spatial positions.Source: §4.1 · p. 7, Figure 8 · p. 7
• Visual Multi-scale Packing downsamples an image into multiple scales, encodes them separately, and concatenates their visual embeddings for the LLM.An end-of-scale token marks the end of each scale image.Source: §4.1 · p. 7

4.2 Training

• In multimodal generative pre-training, NaViL first trains vision-specific parameters on 500 million image-text pairs while keeping textual parameters frozen.The data comprise 300 million web-dataset samples and 200 million images paired with captions synthesized by existing MLLMs.Source: §4.2 · p. 7
• A subsequent 185-million-example phase unfreezes self-attention text parameters for cross-modal alignment, followed by 68 million higher-quality examples with all parameters unfrozen.The final stage is supervised fine-tuning.Source: §4.2 · pp. 7–8

5.1 Experimental Setups

• The evaluation covers seven general multimodal benchmarks and seven visual question-answering benchmarks across domains including OCR, charts, and documents.The general set includes MMVet, MMMU, MMBench-EN, MME, MathVista, OCRBench, and CCBench.Source: §5.1 · p. 8

5.2 Main Results

• NaViL exceeds existing native MLLMs overall and is comparable to same-LLM InternVL-2.5-2B on most general multimodal benchmarks.InternVL-2.5-2B uses a 300M visual encoder distilled from a 6B encoder.Source: §5.2 · p. 9
• NaViL leads prior state-of-the-art native MLLMs on average across visual question-answering tasks and is roughly on par with compositional baselines.It significantly outperforms the previous state-of-the-art native MLLM on DocVQA, ChartQA, and InfoVQA, but remains behind the best compositional MLLMs.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Qualitative Experiments

• With a 1.2B rather than 150M visual encoder, shallow-layer image tokens attend more globally instead of remaining strongly local.The authors interpret this as evidence that a sufficiently large encoder can pre-extract high-level semantic information from the whole image.Source: §5.3 · p. 10, Figure 9 · p. 10
• A 1.2B visual encoder also has higher first-layer attention weights between visual and text tokens than the 150M counterpart.The authors identify earlier cross-modal interaction as an explanatory perspective for improved performance with larger encoders.Source: §5.3 · p. 10, Figure 9 · p. 10

• Visual-encoder scaling is limited by LLM capacity, and the optimal encoder size scales log-proportionally with LLM size.The paper contrasts this encoder-scaling behavior with conventional LLM scaling.Source: §6 · p. 10
• NaViL combines pre-trained LLM initialization, visual encoders, and MoE architecture, and achieves competitive performance on diverse multimodal benchmarks.Source: §6 · p. 10
• Limited computation confines the scaling study to models up to 9B parameters and to visual and linguistic modalities.The authors identify larger scales and broader modalities as directions for future investigation.Source: §6 · p. 10

A NaViL-9B: Scaling up to 9B parameters

• NaViL achieves comparable performance to two compositional counterparts while using substantially fewer training tokens.The comparison is reported in Table 3; the NaViL-9B run uses 450B tokens because of limited computational resources and time.Source: A NaViL-9B: Scaling up to 9B parameters · p. 15, Table 3 · p. 15, B More discussions on Compositional… · p. 15
• NaViL-9B outperforms similarly sized native MLLMs on almost all benchmarks and is competitive with InternVL-2.5-8B.The authors report this as evidence that their native MLLM scales to larger parameter sizes with consistent performance gains.Source: A NaViL-9B: Scaling up to 9B parameters · p. 15

B More discussions on Compositional MLLMs and Native MLLMs

• Compositional MLLMs separately pre-train vision and language components with different objectives, whereas native MLLMs jointly optimize both modalities end-to-end with next-token prediction.The paper describes the compositional training process as more complex and harder to scale.Source: Figure 10 · p. 15, B More discussions on Compositional… · pp. 15–16

C More Related Works

• Prior neural scaling-law research spans language, vision, and multimodal models, including findings on compute efficiency and the relative importance of tokens and active parameters.The review cites an unbalanced scaling law for native MoE multimodal models in which scaling training tokens is more critical than scaling active parameters as compute grows.Source: C More Related Works · p. 16

D Implementation Details

• Multimodal pre-training and supervised fine-tuning use high-quality data from InternVL-2.5, while pure-language data primarily comes from InternLM2.5.The multimodal data spans image captioning, general question answering, multi-turn dialogue, charts, OCR, documents, and knowledge.Source: D Implementation Details · p. 16

E The NLP capability

• NaViL maintains the NLP capabilities of its Qwen3-8B initialization and performs more strongly than other native MLLMs on three common NLP tasks.The authors attribute this to the modality-specific MoE architecture and report this despite using little high-quality text data.Source: E The NLP capability · p. 16

F More Qualitative Results

• Table 4 compares NaViL-9B with existing MLLMs on multimodal benchmarks and identifies activated-parameter counts.Its caption also notes that a comparison model uses a distilled visual encoder and that some results were reproduced.Source: Table 4 · p. 17, F More Qualitative Results · p. 17
• Average scores normalize each metric to 0–100, and MME scores sum perception and cognition.The caption also marks best and second-best native-MLLM scores and identifies the Chameleon-7B performance source as [43].Source: F More Qualitative Results · pp. 16–17
• Table 5 compares NaViL and existing native MLLMs on three common NLP tasks, with most models evaluated using OpenCompass.Source: Table 5 · p. 17
• Tables 6–8 list architecture hyperparameters and training-recipe hyperparameters for NaViL-2B and NaViL-9B.Source: Table 6 · p. 18, Table 7 · p. 18, Table 8 · p. 18

Image Captioning and Visual Question Answering

• The visual question-answering examples range from describing scenes and extracting nutrition-label and receipt details to answering targeted questions about people, roads, and locations.The receipt description identifies a lunch order for five; a separate response answers that the left exit leads to Reston, Herndon, and Dulles Airport.Source: Image Captioning and Visual Question… · pp. 19–20

Multi-lingual OCR

• The multilingual OCR example extracts a Chinese essay and provides an English translation.Source: Multi-lingual OCR · p. 21

Document Understanding

• Document-understanding responses provide an address, identify Camel cigarette styles for venue resale, and name KBA as having discretion over special-event flyer distribution.Source: Document Understanding · p. 21

LaTeX and Math

• For the parallelogram prompt, the response uses area 148 m2 and base 8 m to solve for a height of 18.5 m.Source: LaTeX and Math · p. 22
• Using the table values, the response substitutes 29 kg for 2016 and 40 kg for 2017 into the rate-of-change formula, yielding 11 kg/year between 2016 and 2017.Source: LaTeX and Math · p. 22