Source-linked AI summary
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang
TL;DR
Open-source multimodal models still trail proprietary systems in scale, resolution, and multilingual capability. InternVL 1.5 addresses these gaps with continuous vision-encoder learning, dynamic high-resolution tiling, and bilingual data, achieving state-of-the-art results on 8 of 18 benchmarks and surpassing leading proprietary models on four benchmarks.
Problem
Open-source MLLMs trail proprietary models in parameter scale, image resolution, and multilingual capability.
Method
InternVL 1.5 combines continuous learning for InternViT-6B, dynamic 448×448 tiling up to 40 tiles, and high-quality bilingual datasets.
Results
8 of 18 benchmarks achieve state-of-the-art results, while the model surpasses leading proprietary models on four specific benchmarks.
Takeaways & Limitations
The reported results indicate that InternVL 1.5 narrows the performance gap between open-source and proprietary multimodal models, especially in OCR and Chinese-related understanding.
Abstract
from arXiv · showhide
In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$\times$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.
1. Introduction
InternVL 1.5 targets the capability divide between open-source and proprietary multimodal models with a stronger vision encoder, dynamic high resolution, and bilingual data. Across 18 benchmarks, it reports competitive performance, including state-of-the-art results on 8 benchmarks and superiority over leading proprietary models on four specific benchmarks.
- Motivation: Open-source MLLMs lag proprietary models in parameter scale, image resolution, and multilingual capability.Proprietary models typically use at least 100 billion parameters and dynamic resolution, whereas open-source models commonly use smaller vision and language components with fixed resolutions.
- Approach: InternVL 1.5 combines continuous learning for InternViT-6B, 448×448 tiling with up to 40 tiles, and diverse English-Chinese data.The design includes a thumbnail for global context and a translation pipeline that can extend to additional languages.
- Capabilities: Up to 4K resolution balances computational efficiency with detail preservation across scene description and document understanding.Users can select low resolution for scene subjects or high resolution for document understanding.
- Capabilities: Continuous learning strengthens InternViT-6B visual representations, while its visual features remain compatible with LLMs exceeding 20 billion parameters.The model combines strong visual and linguistic processing for multimodal understanding.
- Evaluation: 8 of 18 benchmarks achieve state-of-the-art results, and InternVL 1.5 surpasses Grok-1.5V, GPT-4V, Claude-3 Opus, and Gemini Pro 1.5 on four benchmarks.The strongest reported comparisons occur particularly on OCR-related datasets including TextVQA, ChartQA, and DocVQA.
2. Related Work
Related work establishes MLLMs through proprietary and open-source systems, while highlighting persistent challenges in vision encoders and image resolution. InternVL 1.5 follows the common ViT-MLP-LLM pattern while addressing these challenges with a large vision encoder and flexible resolution.
- Proprietary MLLMs: Proprietary MLLMs such as GPT-4V and Gemini extend language models with visual inputs for multimodal processing.These systems represent a major evolution of commercial multimodal models.
- Open-source MLLMs: Open-source MLLMs commonly train on small fixed resolutions such as 336×336 or 448×448, producing sub-optimal performance.The related-work discussion identifies resolution as a central limitation of existing open-source systems.
- Vision Foundation Models: Vision foundation model research explores mixtures and dual-branch encoders to support complementary low- and high-resolution visual features.Examples combine CLIP, DINOv2, ConvNext, SigLIP-L, or SAM-B pathways.
- InternVL 1.5 Architecture: InternVL 1.5 uses the ViT-MLP-LLM configuration, integrating pre-trained InternViT-6B and InternLM2-20B through an MLP projector.The architecture follows a configuration used by several existing open-source MLLMs.
3. InternVL 1.5
InternVL 1.5 combines a ViT-MLP-LLM architecture with a portable vision encoder, dynamic tiled resolution, and broad bilingual training data. Its dataset and benchmark design span OCR, general multimodal, mathematical, and conversational tasks.
- Architecture: InternVL 1.5 integrates InternViT-6B and InternLM2-20B through a randomly initialized MLP projector.The architecture follows the ViT-MLP-LLM configuration.
- Dynamic High-Resolution: Dynamic resolution divides images into 448×448 tiles, using 1–12 tiles during training and zero-shot scaling to 40 tiles at testing.Pixel shuffle reduces visual tokens to one-quarter, representing a 448×448 image with 256 visual tokens.
- Strong Vision Encoder: Continuous pre-training addresses degradation on high-resolution and non-web images, including documents, while selecting features from the fourth-to-last layer.The update reduces InternViT-6B from 48 to 45 layers by discarding the last three layers.
- Strong Vision Encoder: InternViT-6B remains compatible and portable when the language model changes from Nous-Hermes-2-Yi-34B to InternLM2-20B.The authors report that its learned visual features are not tightly bound to a specific LLM.
- Dynamic High-Resolution: Dynamic aspect-ratio matching selects among 35 predefined tile layouts while preserving natural image proportions.Training permits at most 12 tiles because of computational constraints.
- Datasets: The training corpus covers captioning, detection, grounding, OCR, documents, charts, mathematics, scientific images, conversation, and text-only language data.OCR sources constitute 32.0% of the pre-training data, while captioning sources constitute 53.9%.
- Evaluation: Evaluation covers OCR-related, general multimodal, mathematical, and multi-turn conversation benchmarks.Table 2 compares models on 16 multimodal benchmarks, including DocVQA, ChartQA, MMBench-CN, MMMU, and MathVista.
- Bilingual Dataset: A translation pipeline converts English datasets into Chinese with open-source LLMs or GPT-3.5 and can extend to more languages through prompts.The authors report enhanced Chinese capabilities from translated datasets.
4. Experiments
InternVL 1.5 is evaluated across multimodal benchmarks, ablations, resolution settings, and qualitative comparisons with proprietary models. It performs strongly across OCR, general understanding, Chinese scenes, and several visual reasoning settings, while retaining some limitations in multi-turn conversation and task-dependent resolution scaling.
- Experimental Setup: InternVL 1.5 handles up to 40 tiles at test time, equivalent to 4K resolution, despite training with at most 12 tiles.Images are divided into 448×448-pixel tiles according to aspect ratio and resolution.
- Quantitative Results: InternVL 1.5 achieves leading OCR performance, including state-of-the-art results on ChartQA and OCRBench against competing proprietary models.The evaluation covers document, chart, infographic, scene-text, and aggregate OCR capabilities.
- Quantitative Results: InternVL 1.5 significantly surpasses open-source and proprietary methods on MMBench-CN and CCBench, while slightly declining from InternVL 1.2 on MMMU.The authors attribute the MMMU decline to the smaller language model, a pattern also observed on MMT-Bench.
- Ablation Study: A larger 6B vision foundation model helped InternVL 1.2 outperform LLaVA-NeXT on 9 of 11 evaluated datasets after selected datasets were excluded.The comparison is not fully equivalent because the models differ in resolution training and other settings.
- Ablation Study: Higher resolution benefits OCR tasks, whereas several general benchmarks show slight performance declines at higher resolutions.InternVL 1.5 therefore adjusts resolution by task, preserving resources when additional detail is unnecessary.
5. Conclusion
InternVL 1.5 combines a strong continuously learned vision encoder, dynamic high-resolution processing, and a bilingual dataset to narrow the gap between open-source and proprietary multimodal models. The reported evaluations show robust broad performance, especially for OCR and Chinese-related understanding, while leaving further challenges for the field.
- Conclusion: InternVL 1.5 narrows the performance gap between open-source and proprietary multimodal models through a strong vision encoder, dynamic high resolution, and bilingual data.The model is presented as an open-source MLLM for multimodal understanding.
- Conclusion: InternVL 1.5 shows robust performance across varied benchmarks, with particular strengths in OCR-related tasks and Chinese-related scene understanding.The conclusion summarizes competitive performance with leading proprietary models.
- Conclusion: The authors identify continuing challenges and call for further enhancement and collaboration to expand open-source multimodal models.This is stated as the work’s forward-looking conclusion.