PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang

arXiv:2412.05271v5cs.CV

TL;DR

Open-source MLLMs still trail closed-source systems in performance and efficiency, motivating better scaling strategies and data quality. InternVL 2.5 studies scaling across model components, datasets, and inference while retaining its core architecture, and achieves competitive results across diverse tasks, including surpassing 70% on MMMU. The paper also identifies remaining limitations in repetitive generation and integrated multimodal capability.

  • Problem

    Open-source multimodal models remain below desired performance and efficiency levels despite offering transparent alternatives to closed-source systems.

  • Method

    InternVL 2.5 retains InternVL 2.0’s architecture while systematically studying vision encoders, language models, dataset sizes, inference time, data quality, and test-time strategies.

  • Results

    InternVL 2.5 performs competitively across diverse multimodal and language benchmarks and is the first open-source MLLM to surpass 70% on the MMMU benchmark.

  • Takeaways & Limitations

    InternVL 2.5 provides an open-source multimodal model and scaling analysis for developing and applying multimodal AI systems.

  • Takeaways & Limitations

    Data filtering does not completely eliminate repetitive generation, and InternVL2.5-78B still does not surpass Qwen2-VL-72B on MMVet v2.

Abstract

from arXiv · show

We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see https://huggingface.co/spaces/OpenGVLab/InternVL

1 Introduction

InternVL 2.5 targets the open-source performance and efficiency gap with closed-source MLLMs by systematically studying scaling across models, data, and inference. It retains a widely used multimodal architecture while introducing broader data support and achieves competitive performance across diverse evaluations, including surpassing 70% on MMMU.

  • Motivation: Closed-source models such as GPT-4o and Claude-3.5-Sonnet set a performance target, while open-source alternatives remain limited in performance and efficiency.Open-source models offer transparent alternatives but still fall short of desired performance and efficiency levels.
  • Approach: InternVL 2.5 systematically investigates how vision encoders, language models, dataset sizes, and inference time affect multimodal performance.The study examines the relationship between model scaling and performance across these factors.
  • Approach: InternVL 2.5 builds on the InternVL 2.0 architecture and adds multi-image and video data alongside single-image and text-only data.The model retains the ViT-MLP-LLM paradigm while extending supported data types.
  • Contribution: InternVL 2.5 is released as an open-source tool for developing and applying multimodal AI systems.The release is intended to encourage further research in multimodal AI.
  • Results: InternVL 2.5 achieves competitive performance across diverse benchmarks spanning reasoning, document, video, multilingual, grounding, hallucination, and language tasks.The evaluations include multi-discipline reasoning, document understanding, multi-image/video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing.
  • Results: InternVL 2.5 becomes the first open-source MLLM to surpass 70% on the MMMU validation set.This result is presented as a benchmark for open-source multimodal models.

2 Model Architecture

InternVL 2.5 retains the ViT-MLP-LLM architecture while combining upgraded vision and language components with dynamic-resolution multimodal preprocessing. Its design supports single-image, multi-image, and video inputs through tile allocation and token reduction strategies.

  • Overall architecture: InternVL 2.5 retains the ViT-MLP-LLM architecture used in InternVL 1.5 and InternVL 2.0.The architecture combines a pretrained InternViT vision encoder with language models through an MLP projector.
  • Overall architecture: The model integrates InternViT-6B or InternViT-300M with language models of different sizes and types using a randomly initialized 2-layer MLP projector.The language backbones include InternLM 2.5 and Qwen 2.5.
  • Overall architecture: Pixel unshuffle reduces each 448×448 image tile from 1024 visual tokens to 256 tokens.This represents a one-quarter reduction intended to improve scalability for high-resolution processing.
  • Input preprocessing: Dynamic resolution divides images into 448×448 tiles according to their aspect ratio and input resolution.InternVL 2.0 and 2.5 extend this preprocessing beyond single-image datasets to multi-image and video data.
  • Vision encoder: InternVL uses two vision-encoder sizes: InternViT-6B and InternViT-300M.InternViT-300M is a distilled variant that was further refined through incremental pretraining on a more diverse data mixture using next-token prediction loss.
  • Model variants: InternVL 2.5 upgrades both the vision encoder and language model relative to earlier InternVL versions.The reported OpenCompass scores for earlier versions come from the leaderboard, whereas InternVL 2.5 scores come from local testing.
  • Input preprocessing: For single-image data, all nmax tiles serve one image; for multi-image data, tiles are distributed proportionally across images; for video, nmax = 1 and frames are resized to 448×448.These configurations adapt tile allocation to different data types.

3 Training Strategy

InternVL 2.5 combines dynamic high-resolution processing with a staged training pipeline and progressive scaling strategy. These choices support multimodal integration, reusable vision components, and reduced training cost.

  • Dynamic High-Resolution: Dynamic high-resolution processing matches image aspect ratios to tiled inputs while constraining tile counts between nmin and nmax.The method selects target ratios, resizes images, and splits them into 448×448 tiles.
  • Dynamic High-Resolution: Multi-image samples distribute nmax tiles across images, whereas video samples use nmax = 1 with fixed-resolution 448×448 frames.Video inputs can still produce 8,192 or 16,384 visual tokens when 32 or 64 frames are extracted.
  • Single Model Training Pipeline: The three-stage pipeline uses MLP warmup, optional ViT incremental learning, and full-model instruction tuning to progressively integrate vision and language.Stage 1 trains only the projector; Stage 1.5 trains the vision encoder and projector; Stage 2 trains the full model on multimodal instruction data.
  • Progressive Scaling: ∼120B tokens train InternVL2.5-78B, compared with 1.4T tokens processed by Qwen2-VL.The authors present this as evidence that progressive scaling can reduce training cost through reuse of pretrained components.
  • Single Model Training Pipeline: Stage 2 requires strict data quality controls because even a few thousand noisy samples can cause repetitive or erroneous outputs.The LLM becomes trainable in this stage and is especially sensitive to anomalous data.

4 Data Organization

InternVL 2.5 organizes training data through selective augmentation, resolution and sampling controls, data packing, and filtering. The filtering pipeline targets repetitive and anomalous samples that can impair inference and test-time scaling.

  • Dataset Configuration: Data augmentation is enabled for image datasets but disabled for videos so video frames retain consistent image quality.The configuration applies augmentation conditionally according to dataset characteristics.
  • Dataset Configuration: The maximum tile number nmax controls input resolution, with larger values for multi-image or high-resolution data and nmax = 1 for videos.Examples include nmax = 24 or 36 for complex visual inputs, nmax = 6 or 12 for lower-resolution images, and nmax = 1 for videos.
  • Dataset Configuration: The repeat factor r ∈(0, 4] adjusts dataset sampling frequency through down-sampling or up-sampling to balance training data.Values below 1 reduce a dataset’s weight, while values above 1 increase its effective number of epochs.
  • Multimodal Data Packing: Data packing concatenates samples to reduce padding and improve GPU utilization while respecting both LLM sequence length and ViT image-tile limits.Packed samples preserve independent attention contexts and positional indices for each constituent sample.
  • Data Quality: A few thousand anomalous or repetitive samples can trigger aberrant behavior, including repetitive loops in long-form outputs and CoT reasoning.The authors identify repetitive generation as particularly detrimental to test-time scaling.
  • Data Filtering: The filtering pipeline uses LLM quality scoring, repetition detection, and heuristic rules for text, while multimodal data receives repetition and heuristic filtering.Text samples below a domain-specific quality threshold, such as 7, are removed.
  • Data Filtering: Data filtering reduced anomalous behavior and improved CoT reasoning, but did not completely eliminate repetitive outputs.The authors attribute the remaining issue partly to noise introduced during LLM pre-training.

5 Evaluation on Multimodal Capability

InternVL 2.5 is evaluated across diverse multimodal reasoning benchmarks, with direct-answer and Chain-of-Thought results reported. It improves substantially over earlier InternVL models and other open-source systems, approaching advanced closed-source models on several tasks.

  • 5.1.1 Benchmarks: The evaluation spans multidisciplinary reasoning and mathematical benchmarks covering college-level disciplines, visual mathematics, diagrams, and Olympiad problems.MMMU tests expert-level reasoning across six disciplines, while MathVista, MATH-Vision, MathVerse, and OlympiadBench assess visual mathematical problem solving.
  • 5.1.2 Evaluation Results: InternVL 2.5 reports the higher score from direct-answer and CoT reasoning settings on the evaluated benchmarks.The reported MMMU results use the maximum accuracy across both approaches.
  • 5.1.2 Evaluation Results: 70+ on MMMU validation was achieved by InternVL2.5-78B, improving 7.4 points over InternVL2-Llama3-76B.The results also show gains over existing open-source models and progress toward GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro.
  • 5.1.2 Evaluation Results: CoT majority voting improved InternVL2-Llama3-76B on MMMU from 62.7 to 65.3, demonstrating the potential of test-time scaling.The paper observes a similar phenomenon in InternVL 2.5.
  • 5.1.2 Evaluation Results: 72.3% accuracy was achieved by InternVL2.5-78B on the MathVista test-mini set.The paper reports overall improvement over InternVL 2.0 on OlympiadBench and attributes part of the advancement to data filtering that mitigated CoT deadlocks.

5.2 OCR, Chart, and Document Understanding

InternVL 2.5 is assessed on OCR, chart, document, and visual caption restoration tasks. It generally improves over InternVL 2.0 and performs comparably to strong open-source systems, while the 2B model lags Qwen2-VL-2B on several OCR benchmarks.

  • 5.2.1 Benchmarks: The evaluation covers nine OCR-related benchmarks spanning diagrams, charts, text-rich images, documents, infographics, OCR, scientific charts, and caption restoration.Metrics include accuracy, ANLS, OCRBench’s maximum score, chart reasoning measures, and VCR EM/Jaccard.
  • 5.2.2 Evaluation Results: InternVL 2.5 significantly improves over InternVL 2.0 across model scales and is comparable to Qwen2-VL-72B on most OCR-related benchmarks.The comparison includes open-source and closed-source models and attributes the gains to training-strategy and data-quality improvements.
  • 5.2.2 Evaluation Results: At the 2B scale, InternVL2.5-2B underperforms Qwen2-VL-2B on TextVQA, DocVQA, and InfoVQA.The paper suggests that vision-encoder size may matter relatively more for OCR at a fixed total parameter budget.
  • 5.2.2 Evaluation Results: 93.2/97.6 EM/Jaccard scores were achieved by the 2B model on VCR, compared with 32.9/59.2 for the previous generation.The paper attributes this improvement to approximately 22K focused VCR training samples that strengthened task-specific instruction following.

5.3 Multi-Image Understanding

InternVL 2.5 improves multi-image understanding across model scales and leads on four real-world benchmarks. At larger scales it approaches or exceeds GPT-4o on some multi-image tasks, but gaps remain on others and in open-ended response personalization.

  • 5.3.1 Benchmarks: The multi-image evaluation spans BLINK, Mantis-Eval, MMIU, MuirBench, MMT-Bench, and MIRB, covering perception, reasoning, integration, and multi-image relations.Table 8 also reports RealWorldQA, MME-RealWorld, WildVision, and R-Bench for real-world understanding.
  • 5.3.2 Evaluation Results: InternVL 2.5 consistently improves over InternVL 2.0 across all model scales on multi-image benchmarks.At 2B, Mantis-Eval rises from 48.4 to 54.8 and MuirBench from 32.5 to 40.6; the paper links gains to additional curated multi-image datasets.
  • 5.3.2 Evaluation Results: 70.8 on MMT-Bench surpassed GPT-4o’s 65.4, while 55.8 on MMIU closely matched GPT-4o’s 55.7 for InternVL2.5-78B.The paper highlights model scaling and high-quality multi-image training data as important factors.
  • 5.3.2 Evaluation Results: A gap of around 5 points versus GPT-4o remains on BLINK and MuirBench.The paper suggests additional high-quality multi-image training data as a possible direction for improvement.
  • 5.4.2 Evaluation Results: InternVL 2.5 achieves leading performance across RealWorldQA, MME-RealWorld, WildVision, and R-Bench and significantly outperforms InternVL 2.0.These benchmarks target spatial understanding, high-resolution real-world tasks, human preferences, and robustness to image distortions.
  • 5.4.2 Evaluation Results: The model produces accurate and concise responses across most tasks but has potential for more personalized and detailed answers.The paper identifies open-ended tasks and complex interactions as future improvement areas.

5.5 Comprehensive Multimodal Evaluation

InternVL 2.5 is evaluated across comprehensive multimodal benchmarks covering perception, reasoning, integrated capabilities, and visual-language understanding. It improves over InternVL 2.0, especially at smaller scales, while remaining behind leading models on MMVet v2.

  • Benchmark Coverage: Comprehensive evaluation uses MME, MMBench, MMVet, and MMStar to assess diverse multimodal capabilities.These benchmarks cover perception, cognitive reasoning, integrated recognition and generation, and advanced visual-language understanding.
  • Results: InternVL 2.5 consistently outperforms InternVL 2.0 across model sizes, with especially strong gains for 1B–8B models.The reported comparison emphasizes improvements among smaller models.
  • Results: 81.1/79.3 is InternVL2.5-4B’s MMBench v1.0 score, compared with 78.6/73.9 for InternVL2-4B.The benchmark reports English and Chinese scores in the same order for both models.
  • Results: 84.6/82.6 is InternVL2.5-8B’s MMBench v1.0 score, compared with 81.7/81.2 for InternVL2-8B.The 8B comparison also reports English and Chinese results.
  • Remaining Gap: InternVL2.5-78B still does not surpass Qwen2-VL-72B on MMVet, while MMVet v2 remains led by GPT-4o and Claude-3.5-Sonnet.The authors identify this open-source versus closed-source capability gap as a direction for future development.

5.6 Multimodal Hallucination Evaluation

InternVL 2.5 is assessed for hallucinations across four benchmarks covering visual, language, object, and relational factuality. It improves over InternVL 2.0, including for smaller models, while long responses can still contain hallucinations.

  • Benchmark Coverage: Hallucination evaluation uses HallusionBench, MMHal-Bench, CRPE, and POPE to measure multimodal factual inaccuracies.The benchmarks cover image-context reasoning, image-grounded answer quality, object relations, and object hallucination.
  • Results: InternVL2.5-1B and InternVL2.5-2B improve on all hallucination benchmarks relative to their InternVL 2.0 counterparts.The reported gains are particularly notable among smaller models.
  • Results: 39.0 is InternVL2.5-1B’s HallusionBench score, up from 34.0 for the earlier version.The comparison indicates a five-point improvement on this benchmark.
  • Results: 42.6 is InternVL2.5-2B’s HallusionBench score, nearly five points above the previous 2B model.The passage presents this as another substantial small-model improvement.
  • Results: 57.4 is InternVL2.5-78B’s HallusionBench score, compared with 58.1 for Qwen2-VL-72B and 55.0 for GPT-4o.The largest model shows relatively low hallucination rates, but long responses can still contain hallucinations.

5.7 Visual Grounding

InternVL 2.5 evaluates referring expression comprehension on RefCOCO, RefCOCO+, and RefCOCOg, which test localization from varied textual descriptions. Performance improves at both 8B and 78B scales, reaching state-of-the-art average performance at 78B.

  • Task and Datasets: Visual grounding evaluates whether models identify target objects from descriptions on RefCOCO, RefCOCO+, and RefCOCOg.The datasets vary in object focus, attribute-based descriptions, and expression complexity.
  • Results: 87.6 is InternVL2.5-8B’s average grounding score, up from 82.9 for InternVL2.0-8B.Its score is comparable to Qwen2-VL-7B’s 87.9, though below specialized or larger grounding models.
  • Results: 92.3 is InternVL2.5-78B’s average grounding score, a 2.3-point improvement over InternVL2-Llama3-76B.The passage reports this as state-of-the-art performance and says it surpasses Qwen2-VL-72B.
  • Multilingual Evaluation: The multilingual evaluation covers MMMB, Multilingual MMBench, and MTVQA across six languages and text-centric visual question answering.The evaluated languages are English, Chinese, Portuguese, Arabic, Turkish, and Russian.
  • Multilingual Results: InternVL2.5-78B and Qwen2-VL-72B show similar multilingual performance, consistent with their shared underlying language model.The authors suggest that multilingual capability is largely inherited from the language model.

5.9 Video Understanding

InternVL 2.5 is evaluated on six video benchmarks spanning short-video comprehension, temporal reasoning, and long-video understanding. It consistently improves over InternVL 2.0 and scales better with more input frames, while some gaps to closed-source models remain.

  • Benchmark Coverage: Video evaluation covers Video-MME, MVBench, MMBench-Video, MLVU, LongVideoBench, and CG-Bench across short and long video tasks.The benchmarks assess temporal awareness, multimodal reasoning, long-video detail retrieval, and clue-based understanding.
  • Results: InternVL 2.5 consistently improves over InternVL 2.0 across all evaluated video benchmarks.The comparison includes both short-video and long-video settings.
  • Results: 50.3/52.3 is InternVL2.5-1B’s Video-MME score, up from 42.9/45.4 for InternVL2.0-1B.On MVBench, the same model improves from 57.5 to 64.3.
  • Frame Scaling: InternVL 2.5 benefits from increasing input frames, unlike InternVL 2.0 models that typically degrade beyond 16 or 32 frames.The authors attribute this to higher-quality video data and changing training sampling from 4–24 to 8–32 frames.
  • Large-Model Comparisons: 76.4 versus 73.6 is InternVL2.5-78B’s MVBench comparison with Qwen2-VL-72B, while its Video-MME subtitle score is 74.0 versus 77.8.The largest model leads open-source comparisons on several benchmarks but trails some closed-source systems on LongVideoBench.

6 Evaluation on Language Capability

The evaluation covers five language dimensions across diverse benchmarks, while pure language performance exposes a weakness in InternVL 2.0 that InternVL 2.5 addresses through higher-quality language data.

  • Evaluation Scope: Five evaluation dimensions cover comprehensive examination, language and knowledge, reasoning, mathematics, and coding.The benchmarks use varied datasets and few-shot settings to assess these capabilities.
  • Pure Language Performance: InternVL 2.0 scored 39.2 at 2B and 67.2 at 8B on average pure language benchmarks, trailing their foundational LLM counterparts by 2.1 and 2.3 points.InternVL2-2B was compared with InternLM2-1.8B-Chat, while InternVL2-8B was compared with InternLM2.5-7B-Chat.
  • Data Improvements: InternVL 2.5 uses a large collection of high-quality open-source pure language instruction data with rigorous filtering to improve data quality.The filtering pipeline removes low-quality samples before training.
  • Results: These data improvements mitigate the earlier language-performance decline, allowing InternVL 2.5 to match or surpass the original LLM on several tasks.

7 Evaluation on Vision Capability

Vision capability is evaluated through global-view classification and local-view segmentation. Across InternViT versions, simple linear probing declines while stronger adaptation methods increasingly benefit from richer, more complex representations.

  • Evaluation Design: Vision evaluation separates global-view semantic quality through image classification from local-view semantic quality through semantic segmentation.The classification evaluation uses ImageNet variants, while segmentation uses ADE20K and COCO-Stuff-164K.
  • Image Classification: Classification experiments compare linear probing with attention pooling probing across ImageNet-1K and five ImageNet variants.Both methods train on ImageNet-1K and evaluate robustness across IN-ReaL, IN-V2, IN-A, IN-R, and IN-Sketch.
  • Image Classification: Linear probing declines across InternViT versions, while attention pooling consistently exceeds the gray baseline and the average gap ∆ grows from 3.5 to 6.7.
  • Representation Analysis: The results suggest final-layer features become less linearly separable as representations evolve toward more complex, open-ended semantic information.Attention pooling selects relevant features from the enriched representation space and offsets reduced linear separability.
  • Semantic Segmentation: Segmentation linear-probing mIoU falls from 45.0 in InternViT-6B-224px to 37.5 in InternViT-6B-448px-V2.5.This decline is consistent with a shift toward more complex and open-ended information.
  • Semantic Segmentation: Head tuning surpasses the 51.9 baseline across later versions, with ∆1 increasing from 6.9 to 15.1 by InternViT-6B-448px-V2.5.The pattern indicates preserved feature quality despite reduced linear separability.
  • Overall Vision Findings: Increasing ∆1 and ∆2 reflect a shift from simple linearly separable features toward nonlinear representations as the vision encoder improves.The authors associate this evolution with stronger extraction of open-ended visual features and effective ViT incremental learning.

8 Conclusion

InternVL 2.5 extends InternVL 2.0 with improved training, testing, and data quality while systematically examining how scaling affects multimodal performance. Extensive evaluations report competitive results across diverse tasks, including surpassing 70% on MMMU.

  • Conclusion: InternVL 2.5 builds on InternVL 2.0’s architecture with improvements in training strategies, testing strategies, and data quality.
  • Conclusion: The study analyzes scaling relationships across vision encoders, language models, dataset sizes, and test-time configurations.
  • Conclusion: InternVL 2.5 achieves competitive performance across multi-discipline reasoning, document understanding, video understanding, and multilingual processing tasks.
  • Conclusion: InternVL 2.5 is the first open-source MLLM reported to surpass 70% on MMMU.The conclusion states that this narrows the gap between open-source and commercial models such as OpenAI o1.
Loading 2412.05271v5…