Source-linked AI summary
Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Jifeng Dai, Wenhai Wang
TL;DR
MLLMs remain costly to train and deploy, and long-tail specialized tasks pose a challenge. Mini-InternVL distills visual knowledge into a compact encoder and applies a unified transfer approach; Mini-InternVL-4B reaches 90% of InternVL2-76B’s performance with 5% of its parameters, while adapted models match proprietary systems in target domains.
Problem
MLLMs face high computational costs and poor performance on long-tail domain-specific tasks, while downstream adaptation lacks a consensus framework.
Source: §1 · p. 1Method
Mini-InternVL distills InternViT-6B knowledge into InternViT-300M and standardizes architecture, data format, and training schedule for downstream transfer.
Source: §1 · p. 2Results
Mini-InternVL-4B achieves 90% of InternVL2-76B’s performance with 5% of its parameters, and adapted models match proprietary commercial models in target domains.
Source: §1 · p. 2Takeaways & Limitations
The unified transfer approach applies across autonomous driving, medical images, and remote sensing at low marginal cost.
Source: §1 · p. 2After SFT, multiple-choice performance does not improve, which the authors attribute to a lack of multiple-choice question data in training.
Source: §4.2.3 · p. 11
Abstract
from arXiv · showhide
Multimodal large language models (MLLMs) have demonstrated impressive performance in vision-language tasks across a broad spectrum of domains. However, the large model scale and associated high computational costs pose significant challenges for training and deploying MLLMs on consumer-grade GPUs or edge devices, thereby hindering their widespread application. In this work, we introduce Mini-InternVL, a series of MLLMs with parameters ranging from 1B to 4B, which achieves 90% of the performance with only 5% of the parameters. This significant improvement in efficiency and effectiveness makes our models more accessible and applicable in various real-world scenarios. To further promote the adoption of our models, we develop a unified adaptation framework for Mini-InternVL, which enables our models to transfer and outperform specialized models in downstream tasks, including autonomous driving, medical images, and remote sensing. We believe that our study can provide valuable insights and resources to advance the development of efficient and effective MLLMs. Code is available at https://github.com/OpenGVLab/InternVL.
1 Introduction
MLLMs face high computational costs and limitations on long-tail domain tasks, while specialized adaptation lacks a consensus framework. Mini-InternVL combines a compact, distilled vision encoder with a standardized transfer approach, reporting strong general and domain-specific performance.
Motivation
- MLLMs’ computational burden and poor long-tail domain-specific performance hinder practical application, while domain adaptation lacks a consensus framework.Existing approaches vary in model design, data format, and training schedule across domains.Source: §1 · pp. 1–2
- The paper identifies a need for a strong vision encoder and a general transfer-learning paradigm that supports efficient application across domains at low marginal cost.Source: §1 · p. 2
Results
- Mini-InternVL-4B reaches 90% of InternVL2-76B’s performance with 5% of its parameters, while adapted models rival commercial systems in target domains.Evaluation covers general and domain-specific benchmarks and examines the effect of domain-adaptation sample size.Source: §1 · p. 2
Model design
- Mini-InternVL uses a CLIP-initialized InternViT-300M distilled from InternViT-6B, paired with pre-trained LLMs in 1B, 2B, and 4B model variants.The lightweight vision encoder is described as robust across visual domains.Source: §1 · p. 2
Transfer learning
- The transfer paradigm standardizes model architecture, data format, and training schedule for autonomous driving, medical images, and remote sensing.The authors report that adapted models match proprietary commercial models within target domains.Source: §1 · p. 2
2 Related Works
Prior MLLMs include connector-based, encoder-free, and lightweight designs, but inference cost and vision encoders’ limited domain coverage remain concerns. The paper positions visual-knowledge distillation and standardized downstream transfer as responses to gaps in vision representation and domain adaptation.
Model design
- Figure 1 contrasts InternViT-6B teacher distillation with the student architecture: InternViT-300M connects to lightweight LLMs through an MLP, with pixel unshuffle reducing visual tokens to one-quarter.Source: Figure 1 · p. 3
Multimodal Large Language Models
- Lightweight MLLMs reduce reliance on expensive computing, but CLIP-based encoders trained on natural images can limit generalization beyond the general domain.Other approaches include connectors, extra fusion layers, and encoder-free architectures; some fusion designs add training cost or inference burden.Source: §2 · pp. 2–3
Vision Foundation Models for MLLMs
- Multi-domain vision representation has been pursued with feature mixtures and dual encoders, but these pathways can be complex and still lack comprehensive visual knowledge.The paper contrasts these methods with InternViT’s multi-domain representations and proposes transferring visual knowledge to lightweight models.Source: §2 · p. 3
Domain-Specialized Adaptation of MLLMs
- Domain-specific MLLMs often require architecture changes, extensive domain data, or customized training, and no universally accepted downstream adaptation framework exists.The paper proposes a transfer paradigm intended to reduce disparities among MLLMs across fields.Source: §2 · p. 3
3 Method
Mini-InternVL combines a lightweight vision encoder with language models, then applies a standardized transfer framework to downstream domains. Its methods include distilling InternViT-300M, formatting tasks as VQA, and fine-tuning for domain-specific applications.
3.1 Mini-InternVL
- Mini-InternVL has 1B, 2B, and 4B variants combining InternViT-300M, an MLP projector, and Qwen2-0.5B, InternLM2-1.8B, or Phi-3-mini.The framework also uses dynamic-resolution inputs and pixel unshuffle; a 448×448 image becomes 256 visual tokens, and the model supports up to 40 tiles.Source: §3.1 · p. 4
- Training has two stages: MLP-only language-image alignment on diverse task data, followed by full-parameter visual instruction tuning.The instruction-tuning data covers captioning, chart interpretation, OCR, and cross-disciplinary reasoning.Source: §3.1 · p. 4
3.2 InternViT-300M
- InternViT-300M transfers InternViT-6B knowledge into a lightweight vision encoder initialized from CLIP-ViT-L-336px, using negative cosine similarity between their last K transformer-layer hidden states.Distillation data includes natural images, OCR images, charts, and multidisciplinary images, resized to 448 ×448.Source: §3.2 · p. 5
3.3 Domain Adaptation
- The domain-adaptation framework standardizes model architecture, data format, and training schedule across downstream applications, addressing the lack of a universally accepted adaptation framework.It formats conventional visual tasks as VQA; existing VQA datasets can be used directly as instruction-following data.Source: §3.3 · p. 5, Figure 2 · p. 5
- Specialized tasks are cast as conversational inputs and outputs, including multiple-choice image classification, reference-and-box visual grounding, and region-conditioned perception.Region perception can use boxes, masks, or contours on the image, or normalized box coordinates in the question.Source: §3.3 · pp. 5–6
- For multi-view driving, six camera views are tiled in a fixed sequence with a thumbnail and camera-position labels; video uses interleaved image-token frames.For 448×448 images, the video format accommodates sequences of up to 40 frames. Domain adaptation uses full-parameter fine-tuning, and adding general multimodal data can improve generalization to other tasks.Source: §3.3 · pp. 6–7
4 Experiments
Experiments compare Mini-InternVL on general multimodal benchmarks, transfer it to three specialized domains, and test design and adaptation choices. Results cover benchmark performance, domain transfer, and ablations of data and training methods.
4.2.1 Multi-View Image-Based Autonomous Driving
- Mini-InternVL-2B scores 0.5958 on the CVPR 2024 Autonomous Driving Challenge, comparable to the leaderboard’s best with one-tenth its parameters.On MME-RealWorld autonomous driving, DriveLM-only domain training yields an improvement of over 10 points and surpasses LLaVA-OneVision-7B, GPT-4o, and Claude 3.5 Sonnet.Source: Table 4 · p. 8, §4.2.1 · p. 9, Table 5 · p. 9
4.1 Results on General Multimodal Benchmark.
- Mini-InternVL achieves approximately 90% of InternVL2-Llama3-76B’s performance using 5% of its parameters, with strong results across most benchmarks.The 4B model is particularly strong on MMBench, ChartQA, DocVQA, and MathVista, matching Gemini-Pro-1.5 on those benchmarks.Source: §4.1 · p. 8
4.2.2 Autonomous Driving with Temporal Information
- On BDD-X, Mini-InternVL performs comparably to DriveGPT4 across four tasks and surpasses ADAPT, despite lacking DriveGPT’s large-scale proprietary-domain pretraining.The paper reports action-description and justification metrics alongside open-loop control-signal prediction metrics.Source: Table 6 · p. 10, Table 7 · p. 10, §4.2.2 · p. 10
4.2.3 Medical Image Question Answering
- After medical supervised fine-tuning, performance improves on most metrics, including 2D classification, detection, and multi-class accuracy, but not multiple-choice questions.The authors attribute the multiple-choice result to a lack of such data in the training set.Source: Table 9 · p. 11, §4.2.3 · p. 11
4.2.4 Remote Sensing
- Remote-sensing evaluation covers RSVQA VQA subsets and DIOR-RSVG visual grounding; one epoch on DIOR-RSVG was insufficient, so its samples were repeated.The evaluated RSVQA subsets are Presence, Comparison, and Rural/Urban.Source: §4.2.4 · pp. 11–12, Table 11 · p. 12
4.3 Ablation Study
- Replacing InternViT with CLIP-ViT-L-336px under the same recipe performs worse on general benchmarks, especially document tasks, and after driving transfer.The authors interpret the comparison as evidence that distillation enables InternViT to acquire visual world knowledge.Source: §4.3 · p. 13
- For driving, performance peaks at a general-data ratio of r = 0.25; using one-quarter of the full dataset cuts compute with minor performance loss, while fixed data ratios largely preserve general scores.The ratio experiment finds that adding an appropriate amount of general data improves domain performance and reduces degradation on general benchmarks.Source: §4.3 · pp. 13–14
- Across adaptation methods, full-parameter fine-tuning scores highest on domain tasks, LoRA underperforms on visual grounding, and freezing the vision encoder balances performance and efficiency.Performance converges after 1500 steps; all three methods maintain stable MMBench scores across training steps.Source: Figure 5 · p. 13, §4.3 · p. 14
5 Conclusion
Mini-InternVL pairs a compact, distilled vision encoder with lightweight MLLMs and a unified transfer format. It achieves roughly 90% of larger models’ performance with fewer parameters and comparable performance to domain-specific approaches after adaptation.
5 Conclusion
- Mini-InternVL uses InternViT-300M, distilled from a more capable vision model, to bring multi-domain visual knowledge into a compact encoder.The paper presents this as addressing limitations of encoders such as CLIP-ViT.Source: §5 · p. 14
- The models achieve approximately 90% of larger models’ performance with significantly fewer parameters, particularly excelling at OCR and domain-specific image understanding.Source: §5 · p. 14
- A unified transfer format supports adaptation across specialized domains, where Mini-InternVL achieves performance comparable to domain-specific approaches.Source: §5 · p. 14