Source-linked AI summary
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, Chunhua Shen
TL;DR
MobileVLM V2 addresses the challenge of deploying capable vision-language models in resource-constrained environments. It combines high-quality data, improved training strategies, and a lightweight projector, achieving strong accuracy–latency tradeoffs and outperforming many larger models.
Problem
Deploying capable vision-language models in real scenarios such as mobile devices remains challenging.
Method
MobileVLM V2 combines high-quality multimodal data, full projector and language-model training, and a lightweight projector for efficient vision-language alignment.
Results
MobileVLM V2 achieves a new state-of-the-art accuracy–latency Pareto front and outperforms many larger models with substantial inference advantages.
Takeaways & Limitations
The results support using MobileVLM V2 as an efficient baseline for advanced vision-language models in resource-limited scenarios.
Takeaways & Limitations
Combining MobileVLM V2 with mixture-of-experts designs without sacrificing its memory and latency advantages remains unresolved.
Abstract
from arXiv · showhide
We introduce MobileVLM V2, a family of significantly improved vision language models upon MobileVLM, which proves that a delicate orchestration of novel architectural design, an improved training scheme tailored for mobile VLMs, and rich high-quality dataset curation can substantially benefit VLMs' performance. Specifically, MobileVLM V2 1.7B achieves better or on-par performance on standard VLM benchmarks compared with much larger VLMs at the 3B scale. Notably, our 3B model outperforms a large variety of VLMs at the 7B+ scale. Our models will be released at https://github.com/Meituan-AutoML/MobileVLM .
1. Introduction
MobileVLM V2 targets capable VLM deployment in resource-constrained scenarios by combining high-quality data, improved training, and a lightweight projector. The models are presented as faster and stronger baselines for mobile-scale vision-language applications.
- Mobile VLM deployment remains challenging for mobile devices, self-driving cars, and embodied AI systems.
- MobileVLM V2 establishes faster and stronger baselines built upon MobileVLM.
- Figure 1 compares average benchmark performance and inference speed for state-of-the-art VLMs on an NVIDIA Jetson Orin using llama.cpp.
- The approach combines 1.2 million ShareGPT4V image-text pairs, diverse academic tasks, comprehensive parameter training, and a lightweight projector.
- The projector enhances image-token positional representations, enabling substantial token compression without much performance degradation.
1. We explore and evaluate the performance of increasing
The paper explores scaling MobileVLM V2 while improving performance and inference efficiency. Its contributions emphasize high-quality data, training strategies, and lightweight visual-token reduction.
- MobileVLM V2 uses training data for small VLMs to significantly bridge the gap between small and large vision language models.
- The training scheme is designed to exploit high-quality multimodal data for mobile scenarios.
- The lightweight projector significantly reduces visual tokens with slight performance drops.
- MobileVLM V2 achieves a new state-of-the-art tradeoff between performance and inference speed across several vision-language benchmarks.
- Scaling the model to 7B parameters outperforms previous state-of-the-art models with clear margins.
2. Related Work
Related work traces VLM progress through large language models, multimodal understanding, and efforts to deploy multimodal systems on edge devices. It situates MobileVLM V2 within this broader development.
- Large language models such as GPT-3, PaLM, OPT, BLOOM, and LLaMA advanced natural language processing research.
- Instruction-tuned models including Alpaca and Vicuna enhanced language interaction abilities through human-guided annotations.
- Multimodal language models extend language models to comprehend image semantics and answer queries about visual content.
- Several recent works address the demand for deploying multimodal large language models on edge devices.
3. Method
MobileVLM V2 combines a pre-trained vision encoder, MobileLLaMA language model, and lightweight downsample projector with a two-stage training process. Its method emphasizes efficient visual-token reduction, unified projector–language-model optimization, and diverse multimodal data for mobile deployment.
- Overall Architecture: The architecture uses a pre-trained vision encoder, MobileLLaMA, and the mobile-friendly LDPv2 projector to align image features with language processing.The vision encoder extracts image features, the projector aligns them with the language model, and MobileLLaMA generates final answers.
- Language Model: MobileLLaMA processes concatenated visual and text tokens and generates the final response autoregressively.The adopted language models are MobileLLaMA-1.4B-Chat and MobileLLaMA-2.7B-Chat.
- Lightweight Downsample Projector: LDPv2 applies feature transformation, 2 × 2 average pooling, and positional enhancement to reduce visual tokens with fewer parameters.It uses pointwise convolutions, average pooling, and PEG with a skip connection; its positional component reduces parameters by 99.8% compared with LDP.
- Training Strategy: Training has pre-training and multi-task stages, with the visual encoder frozen while the projector and language model are optimized in both stages.This differs from prior training paradigms that freeze more components during pre-training.
- Training Data: The training data combines 1.2 million ShareGPT4V-PT image-text pairs with task-specific datasets covering dialogue, OCR, scene understanding, and location understanding.Visual Dialog, TextVQA, COCO Caption, SBU, and VSR support these diverse capabilities; the SBU data was cleaned and refined.
4. Experiment
MobileVLM V2 is evaluated against state-of-the-art VLMs for accuracy, speed, scaling, and deployment latency. Across these comparisons, it offers a strong performance–efficiency tradeoff, including higher benchmark scores at substantially faster generation speeds.
- Comparison with SOTA methods: MobileVLM V2 3B is 75% faster than MoE-LLaVA-2.7B×4 while scoring 1.4 points higher on the average benchmark score.The comparison uses average accuracy across six VLM benchmarks.
- Latency Comparison: 37.37 tokens/s for MobileVLM V2 1B and 28.97 tokens/s for 3B are both 1.65× faster than MoE-LLaVA counterparts, with higher average performance.Latency is measured with PyTorch on an NVIDIA A100 GPU using batch size 1 and 256 generated tokens.
- Comparison with MobileVLM: MobileVLM V2 improves MobileVLM’s average accuracy by 5.3 points while sharing the same vision and language encoders.The paper attributes the improvement to enhanced data, training strategies, and the new projector design.
- Model Scaling Analysis: MobileVLM V2 7B is nearly 20% faster than ShareGPT4V 7B and scores 1.3 points higher on average across the evaluated benchmarks.The scaling comparison evaluates accuracy and inference speed against large-scale VLMs including LLaVA-1.5 7B and ShareGPT4V 7B.
- Model Scaling Analysis: MobileVLM V2 7B without average pooling reaches an average score of 73.5, outperforming LLaVA-1.5 by 4.7 points.Removing token reduction mainly improves TextVQA performance, where small objects can make token reduction harmful.
5. Ablation Study
The ablation study separates the effects of data enhancement, training strategy, and projector design in MobileVLM V2. Results show that richer data and joint projector–language-model finetuning improve performance, while token reduction provides efficiency with task-dependent tradeoffs.
- Data Strategy: Replacing the pretraining data with ShareGPT4V and expanding instruction tuning to 2.4M examples improves GQA, SQA, and TextVQA performance.The model architecture and training strategy remain unchanged in this data-enhancement ablation.
- Training Strategy: Simultaneously finetuning the projector and language model enables comprehensive performance improvement with enhanced data and small-scale VLMs.The language model is unlocked throughout the training stage rather than keeping it fixed.
- Projector Design: The original LDPv1 reduces visual tokens by 75%, from 576 to 144, while maintaining almost equivalent performance.With the proposed data and training configurations, the corresponding architectures achieve average improvements of 5.4 and 4.5 points.
6. Conclusion
MobileVLM V2 extends MobileVLM with data scaling, improved training strategies, and efficient modality alignment for small VLMs. The resulting models target a strong accuracy–latency tradeoff and resource-limited deployment.
- MobileVLM V2 combines data scaling, improved training strategies, and efficient modality alignment to improve small VLM performance.
- The method achieves a new state-of-the-art Pareto front in accuracy and latency with comparable training cost.
- The models offer substantial inference advantages over many larger models for resource-limited scenarios.
A. Dialogue formats of various datasets.
MobileVLM V2 uses separate pre-training and multi-task training data. The paper provides examples of the resulting multi-task data formats in two tables.
- Pre-training uses 1.2 million image-text pairs from ShareGPT4V.These pairs primarily include COCO, SAM, and LLaVA-1.5 pre-training data.
- Multi-task training uses 2.4 million high-quality instruction data.
- Tables 8 and 9 provide detailed formats and examples of the multi-task training data.
B. Examples of MobileVLM V2.
The examples show MobileVLM V2 handling multimodal conversations across visual understanding, reasoning, OCR, and world-knowledge tasks. Responses include descriptions, classifications, spatial answers, and multi-turn attributes.
- Multimodal capabilities: The examples cover fine-grained attribute understanding, image understanding, and relation reasoning.
- OCR and world knowledge: The examples include OCR, world knowledge, and multi-turn questions about object counts, colors, parking, and helmets.
- Visual reasoning: The model answers visual question-answering tasks involving equal parts, spatial relations, and object descriptions.
- Data examples: Figure 6 presents MobileVLM V2 1.7B examples in various scenes, while Tables 8 and 9 show multi-task training-data examples.