PaperlayerView exact revision on arXiv

Source-linked AI summary

InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang

arXiv:2504.10479v3cs.CV

TL;DR

MLLMs commonly retrofit text-only LLMs through complex, resource-intensive pipelines that create modality-alignment challenges. InternVL3 instead jointly pre-trains linguistic and multimodal capabilities, adding V2PE, post-training, test-time scaling, and infrastructure optimizations. It improves across diverse multimodal tasks, with InternVL3-78B achieving 72.2 on MMMU and remaining competitive with closed-source models, though gaps and benchmark-specific declines remain.

  • Problem

    Post-hoc MLLM training introduces modality-alignment challenges and requires specialized data, parameter freezing, and multi-stage fine-tuning.

  • Method

    InternVL3 jointly pre-trains on pure-text and diverse multimodal data, supplemented by V2PE, SFT, MPO, test-time scaling, and workload-balancing infrastructure.

  • Results

    InternVL3 surpasses predecessors across diverse multimodal tasks; InternVL3-78B scores 72.2 on MMMU, setting a new open-source MLLM standard.

  • Takeaways & Limitations

    Native multimodal pre-training supports strong multimodal and linguistic capabilities while reducing the complexities associated with post-hoc MLLM training.

  • Takeaways & Limitations

    InternVL3 still trails Qwen2.5-VL on certain benchmarks, and minor declines on MMHal show that improvements are not fully consistent.

Abstract

from arXiv · show

We introduce InternVL3, a significant advancement in the InternVL series featuring a native multimodal pre-training paradigm. Rather than adapting a text-only large language model (LLM) into a multimodal large language model (MLLM) that supports visual inputs, InternVL3 jointly acquires multimodal and linguistic capabilities from both diverse multimodal data and pure-text corpora during a single pre-training stage. This unified training paradigm effectively addresses the complexities and alignment challenges commonly encountered in conventional post-hoc training pipelines for MLLMs. To further improve performance and scalability, InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure. Extensive empirical evaluations demonstrate that InternVL3 delivers superior performance across a wide range of multi-modal tasks. In particular, InternVL3-78B achieves a score of 72.2 on the MMMU benchmark, setting a new state-of-the-art among open-source MLLMs. Its capabilities remain highly competitive with leading proprietary models, including ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Pro, while also maintaining strong pure-language proficiency. In pursuit of open-science principles, we will publicly release both the training data and model weights to foster further research and development in next-generation MLLMs.

1 Introduction

InternVL3 addresses the alignment and efficiency challenges of post-hoc multimodal training through native multimodal pre-training and complementary scaling and post-training techniques. It surpasses prior open-source MLLMs across diverse tasks, reaches 72.2 on MMMU, and remains competitive with leading closed-source models.

  • Motivation: Post-hoc MLLM pipelines introduce modality-alignment challenges and require specialized data, parameter freezing, and multi-stage fine-tuning.These resource-intensive strategies motivate more efficient multimodal training paradigms.
  • Approach: InternVL3 jointly learns linguistic and multimodal capabilities from pure-text corpora and diverse multimodal datasets during pre-training.This native paradigm replaces retrofitting a text-only LLM with visual processing after text pre-training.
  • Approach: V2PE, SFT, MPO, test-time scaling, and optimized infrastructure improve InternVL3’s multimodal-context capacity, efficiency, and performance.V2PE accommodates longer multimodal contexts, while SFT and MPO provide advanced post-training.
  • Results: 72.2 on MMMU establishes InternVL3-78B as a new open-source MLLM benchmark leader across a wide range of multimodal tasks.Evaluations cover reasoning, document, image/video, real-world, hallucination, grounding, and multilingual capabilities.
  • Results: InternVL3 significantly outperforms existing open-source MLLMs and remains highly competitive with state-of-the-art closed-source commercial models.The reported comparisons include Qwen2.5-VL, Step-1o, GLM-4v-Plus, GPT-4o, and Gemini-2.5-Pro.
  • Open Science: The authors plan to release InternVL3’s training data and model weights to support transparency, reproducibility, and further open-source research.The release is intended to foster continued development of next-generation MLLMs.

2 InternVL3

InternVL3 combines native multimodal pre-training with architectural, post-training, and infrastructure techniques to improve multimodal capability, scalability, and training efficiency.

  • Model Architecture: The model follows the ViT-MLP-LLM framework and uses pre-trained ViT and base LLM components, with a randomly initialized two-layer MLP.Pixel unshuffle reduces visual token counts to one-quarter, representing each 448×448 image tile with 256 visual tokens.
  • Model Architecture: V2PE assigns smaller increments to visual tokens while retaining increment 1 for textual tokens, supporting longer multimodal contexts.The visual increment remains constant within each image and is randomly selected during training; inference can select it based on sequence length.
  • Native Multimodal Pre-Training: InternVL3 jointly trains on text-only and multimodal corpora in one stage, updating all model parameters together.This replaces language-only pre-training followed by multimodal adaptation and aligns linguistic and visual features during optimization.
  • Native Multimodal Pre-Training: Multimodal autoregressive training predicts text tokens conditioned on visual and textual context rather than directly predicting visual tokens.Square averaging is used to reduce gradient bias associated with response length during loss weighting.
  • Post-Training and Infrastructure: InternVL3 combines supervised fine-tuning, mixed preference optimization, test-time scaling, and infrastructure optimizations to improve performance and resource utilization.Its training setup addresses computational imbalance arising from varying proportions of visual and textual tokens; the reported configuration uses approximately 200 billion tokens.

3 Experiments

InternVL3 is evaluated across broad multimodal, reasoning, and mathematical benchmarks, showing substantial gains over earlier models and competitive performance against proprietary systems. Test-time best-of-N scaling further improves reasoning results.

  • InternVL3 demonstrates substantial improvements across multimodal reasoning, OCR, chart and document understanding, hallucination evaluation, and long-video tasks.The reported gains are attributed primarily to enhanced training strategies, refined testing methodologies, and an expanded training corpus.
  • Multimodal reasoning and mathematics: 72.2 on MMMU is achieved by InternVL3, highlighting performance on complex multidisciplinary multimodal reasoning.
  • InternVL3 narrows the gap with ChatGPT-4o-latest and Claude-3.5 Sonnet, and surpasses them on some AI2D and ChartQA benchmarks.Gemini2.5 Pro remains ahead on selected tasks such as HallusionBench.
  • Test-time scaling: Best-of-8 raises Vision-Only MathVerse performance by approximately 6.0 points for InternVL3-38B and 3.2 points for InternVL3-78B.The result supports the effectiveness of test-time scaling for reasoning.

3.3 OCR, Chart, and Document Understanding

InternVL3 maintains strong performance across nine OCR, chart, and document-understanding benchmarks, with larger variants showing increasingly pronounced gains. Its high-scale models achieve particularly strong OCRBench and VCR results.

  • InternVL3 is evaluated across nine benchmarks covering AI2D, ChartQA, TextVQA, DocVQA, InfoVQA, OCRBench, SEED-2-Plus, CharXiv, and VCR.
  • InternVL3-2B reaches 78.7/87.4 on AI2D and 88.3 on DocVQA, outperforming similarly sized Qwen2-VL-2B on several visual-textual tasks.Its TextVQA performance is 77.0, comparable to Qwen2-VL-2B.
  • InternVL3-8B achieves 85.2/92.6 on AI2D, 92.7 on DocVQA, and 94.5/98.1 on VCR.The passage describes these mid-scale results as substantial gains.
  • InternVL3-78B records 906 on OCRBench and 96.0/98.6 on VCR, surpassing corresponding metrics of comparable models.

3.4 Multi-Image Understanding

InternVL3 improves multi-image understanding across parameter scales, with especially strong gains on inter-image reasoning benchmarks. The largest model remains competitive with leading closed-source systems, though a gap persists on MuirBench.

  • The evaluation covers BLINK, Mantis-Eval, MMIU, MuirBench, MMT-Bench, and MIRB, testing cross-image reasoning and context integration.
  • InternVL3-1B achieves 42.9 on BLINK and 52.9 on MMT-Bench, improving modestly and consistently over preceding models.
  • InternVL3-2B reaches 65.9 on Mantis-Eval, improving by over 11 points relative to InternVL2.5-2B, while attaining 59.5 on MMT-Bench.
  • InternVL3-78B reaches 66.3 on BLINK and 73.2 on MMT-Bench, positioning it as a competitive alternative to GPT-4o.
  • InternVL3 remains behind other MLLMs such as Qwen2.5-VL on certain benchmarks, including MuirBench.The passage identifies further training-data curation and model refinement as potential future work.

3.5 Real-World Comprehension

InternVL3 is evaluated on four real-world comprehension benchmarks and shows stronger performance at larger scales. InternVL3-78B is competitive with GPT-4o, surpassing it on RealWorldQA and MME-RealWorld while closely matching its R-Bench performance.

  • InternVL3-1B scores 58.2 on RealWorldQA, 46.0 on MME-RealWorld, 43.8 on WildVision, and 60.4 on R-Bench.
  • The evaluation covers RealWorldQA, MME-RealWorld, WildVision, and R-Bench, targeting realistic and complex tasks.
  • InternVL3-78B achieves 78.0 on RealWorldQA, 65.4 on MME-RealWorld, 73.6 on WildVision, and 77.4 on R-Bench.
  • InternVL3-78B surpasses GPT-4o on RealWorldQA and MME-RealWorld while closely matching its R-Bench performance.GPT-4o scores 75.4 on RealWorldQA and 80.6 on WildVision.

3.7 Multimodal Hallucination Evaluation

InternVL3 improves multimodal hallucination handling across scales and remains competitive with leading closed- and open-source models. However, minor declines on some benchmarks show that more consistent gains remain unresolved.

  • InternVL3 demonstrates overall competitive performance and consistent improvements over previous InternVL models across multimodal hallucination benchmarks.The evaluation uses HallusionBench, MMHal-Bench, CRPE, and POPE.
  • InternVL3-1B reaches 41.4 on HallusionBench, improving over the InternVL2.5-1B baseline score of 39.0.
  • InternVL3-38B scores 57.1 on HallusionBench, while InternVL3-78B reaches 59.1 and improves CRPE to 79.2.
  • Minor declines on benchmarks such as MMHal indicate that more consistent improvements through data and training optimization remain a future direction.

3.10 Video Understanding

InternVL3 shows improving video-understanding performance as model capacity increases across six established benchmarks. Its strengths vary by benchmark, with advantages over competing open-source models on MVBench and MLVU despite occasional weaker results on Video-MME.

  • InternVL3 is evaluated on Video-MME, MVBench, MMBench-Video, MLVU, LongVideoBench, and CG-Bench.Video-MME, MMBench-Video, MLVU, and LongVideoBench use 16, 32, 48, and 64-frame settings, while MVBench uses 16 frames and CG-Bench uses 32.
  • InternVL3-2B improves over earlier 2B variants, scoring 58.9/61.4 on Video-MME and also improving on MVBench and MLVU.
  • InternVL3-78B reaches 72.7/75.7 on Video-MME, while InternVL3-38B records 72.7/75.0 and InternVL3-14B records 70.4/73.0.
  • InternVL3-78B reaches 48 on CG-Bench, with performance steadily improving as model size increases.
  • InternVL3 tends to outperform Qwen2.5-VL models on MVBench and MLVU, although Qwen2.5-VL-72B scores 73.3/79.1 on Video-MME.

3.12 Spatial Reasoning

InternVL3 outperforms competing multimodal models on spatial reasoning and scales strongly across model sizes. Its results also indicate robust performance on specific spatial subcategories relevant to 3D scene understanding.

  • InternVL3-8B scores 42.1 on VSI-Bench, leading all open-source MLLMs in the benchmark.
  • InternVL3-38B and InternVL3-78B score 48.9 and 48.4 on VSI-Bench, exceeding GPT-4o, Gemini-1.5 Flash, and Gemini-1.5 Pro.
  • VSI-Bench measures spatial reasoning through tasks that construct three-dimensional environmental representations from visual inputs.The passage identifies autonomous driving as a relevant application.
  • InternVL3 scores 71.2 in object counting, 53.7 in absolute distance estimation, 55.9 in relative distance estimation, and 54.5 in appearance order prediction.

3.13 Evaluation on Language Capability

InternVL3 is evaluated on language capabilities across knowledge, understanding, reasoning, mathematics, and coding benchmarks against corresponding Qwen2.5 Chat models. The InternVL3 variants consistently outperform the Qwen2.5 Chat models across most benchmarks, with language improvements attributed to pure-language data and joint optimization.

  • Language benchmark comparison: The evaluation spans general knowledge, linguistic understanding, reasoning, mathematics, and coding tasks.Examples include MMLU, GSM8K-Test, MATH, HumanEval, and MBPP.
  • Language benchmark comparison: InternVL3 variants consistently outperform corresponding Qwen2.5 Chat models across most language-capability benchmarks.Both series use the same pre-trained Qwen2.5 base model for initialization.
  • Sources of language capability: Approximately 25% pure-language data, joint parameter optimization, and high-quality textual post-training contribute to stronger language proficiency.The same approach is described as strengthening multimodal comprehension as well.

3.14 Ablation Study

The ablation studies examine native multimodal pre-training, V2PE, and MPO as components of InternVL3. Results indicate that native pre-training provides strong multimodal capability, V2PE improves most evaluated metrics, and MPO improves reasoning performance beyond SFT counterparts.

  • Native multimodal pre-training: Native multimodal pre-training achieves performance comparable to a fully multi-stage-trained InternVL2-8B baseline on most multimodal benchmarks.The comparison keeps architecture, initialization parameters, and training data unchanged.
  • Native multimodal pre-training: Instruction tuning on higher-quality data produces further gains across the evaluated multimodal tasks.This follows native multimodal pre-training in the ablation setup.
  • Variable Visual Position Encoding: V2PE yields significant performance gains across most evaluation metrics on standard multimodal benchmarks.The study evaluates V2PE during native multimodal pre-training and varies the positional increment δ.
  • Variable Visual Position Encoding: Small δ values can achieve optimal performance even on tasks primarily involving short contexts.The result comes from ablations varying the positional increment δ.
  • Mixed Preference Optimization: InternVL3-78B and InternVL3-38B outperform their no-MPO counterparts by 4.1 and 4.5 points, respectively, across seven multimodal reasoning benchmarks.Because MPO training data is a subset of SFT data, the reported gains primarily reflect the training algorithm.

4 Conclusion

InternVL3 combines native multimodal pre-training with V2PE, post-training, and test-time scaling to establish a strong open-source multimodal model while preserving linguistic capabilities. InternVL3-78B scores 72.2 on MMMU, and the authors plan to release training data and model weights publicly.

  • Conclusion: InternVL3 jointly learns linguistic and multimodal capabilities during pre-training rather than adapting a text-only model through a post-hoc pipeline.The approach is presented as avoiding typical training complexity and optimization challenges.
  • Conclusion: V2PE, supervised fine-tuning, mixed preference optimization, and test-time scaling support extended contexts, performance, and scalability.These components are identified as part of the overall InternVL3 system.
  • Conclusion: 72.2 on MMMU is achieved by InternVL3-78B, exceeding previous open-source MLLMs and narrowing the gap with leading proprietary models.The conclusion names Gemini-2.5 Pro as an example of a proprietary counterpart.
  • Conclusion: The authors will publicly release InternVL3’s training data and model weights to support continued multimodal research and development.The release is framed as part of the work’s open-science commitment.
Loading 2504.10479v3…