Source-linked AI summary
MiniVLM: A Smaller and Faster Vision-Language Model
Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang, Xiujun Li, Lijuan Wang, Lei Zhang, Jianfeng Gao, Zicheng Liu
TL;DR
Large pre-trained vision-language models perform well, but their size and computation make them impractical for resource-limited applications, and lightweight alternatives are less explored. MiniVLM combines an efficient visual extractor and compact transformer with teacher-generated captions and tags in pre-training. It retains 94 −97% of accuracy on multiple VL tasks with 27% parameters and 1% FLOPS compared to a state-of-the-art VL model.
Problem
Large vision-language models can be impractical under constrained training or inference budgets, while building lightweight models is less explored.
Source: §1 · pp. 1–2Method
MiniVLM combines the TEE visual feature extractor and MiniLM fusion transformer, with pre-training augmented by pseudo-captioned images and high-quality tags from large models.
Source: §1 · p. 2Results
MiniVLM retains 94 −97% accuracy on multiple VL tasks with 27% parameters and 1% FLOPS compared to a state-of-the-art VL model.
Source: §5 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on downstream VL tasks. While existing research has been focused on achieving high accuracy with large pre-trained models, building a lightweight model is of great value in practice but is less explored. In this paper, we propose a smaller and faster VL model, MiniVLM, which can be finetuned with good performance on various downstream tasks like its larger counterpart. MiniVLM consists of two modules, a vision feature extractor and a transformer-based vision-language fusion module. We design a Two-stage Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet network, to significantly reduce the time cost of visual feature extraction by $95\%$, compared to a baseline model. We adopt the MiniLM structure to reduce the computation cost of the transformer module after comparing different compact BERT models. In addition, we improve the MiniVLM pre-training by adding $7M$ Open Images data, which are pseudo-labeled by a state-of-the-art captioning model. We also pre-train with high-quality image tags obtained from a strong tagging model to enhance cross-modality alignment. The large models are used offline without adding any overhead in fine-tuning and inference. With the above design choices, our MiniVLM reduces the model size by $73\%$ and the inference time cost by $94\%$ while being able to retain $94-97\%$ of the accuracy on multiple VL tasks. We hope that MiniVLM helps ease the use of the state-of-the-art VL research for on-the-edge applications.
1. Introduction
MiniVLM targets the practical gap between large, costly vision-language models and the need for lightweight models on resource-limited devices. It combines an efficient visual extractor and compact transformer with pre-training improvements, retaining strong downstream performance at substantially reduced size and computation.
1. Introduction
- Large vision-language models can be impractical under constrained training or inference budgets, while lightweight VL models remain less explored.Existing systems use large vision and fusion networks; shrinking them without substantial performance loss is challenging.Source: §1 · pp. 1–2
- MiniVLM retains 94 −97% accuracy on multiple VL tasks with 27% parameters and 1% FLOPS compared to state-of-the-art VL models.The model combines TEE-0 and MiniLM; the large captioning and tagging models used in pre-training add no fine-tuning or inference overhead.Source: §1 · p. 2
- MiniVLM pre-training adds 7M pseudo-captioned Open Images and high-quality tags from a large tagging model to improve cross-modal alignment.The large tagging model is used offline, with no runtime impact during fine-tuning or inference.Source: §1 · p. 2
- The paper presents MiniVLM as a smaller, faster model that can be fine-tuned for multiple downstream tasks.Its design responds to the high latency of common visual extractors and the limited investigation of compact transformers in VL pre-training.Source: §1 · p. 2
- The paper introduces TEE to extract image-region features faster and uses MiniLM as the fusion transformer after comparing compact BERT structures.TEE is designed to reduce visual-feature computation while retaining feature quality for downstream tasks.Source: §1 · p. 2
2. Related work
Prior vision-language pre-training commonly combines visual features with text in transformer models, while relying on expensive region extractors and large fusion models. Related work also explores grid features, compact BERT structures, and pseudo-labeling as alternatives or efficiency strategies.
2. Related work
- Most vision-language pre-training feeds visual features and text tokens into BERT-based models, with masked-language, image-text matching, and masked-region objectives commonly used.Fusion approaches include early, late, and full fusion; examples include InterBERT [28], ViLBERT [35], and OSCAR [26].Source: §2 · p. 2
- Faster R-CNN region features are widely used in VL tasks, while grid-feature approaches remove region operations but leave open how to select features efficiently.Prior grid approaches include converting Faster R-CNN to a fully convolutional network [17] and training convolutional features jointly with fusion [15].Source: §2 · p. 3
- Region features offer top-K selection by confidence, typically using 50 regions rather than the 300–600 grid features reported for [17], limiting fusion cost.The paper therefore retains region features for its compact model.Source: §2 · p. 3
- One-stage detectors avoid the RPN, but effective region-feature extraction from them for VL remains open; the paper instead uses a lightweight two-stage architecture.Two-stage detectors generate proposals with an RPN and extract features with RoIPool or RoIAlign.Source: §2 · p. 3
- Compact BERT strategies reduce dimensions, use bottlenecks, or share parameters; MiniVLM selects MiniLM after comparing approaches on VL tasks.Examples include TinyBERT [18], MobileBERT [49], and ALBERT [22].Source: §2 · p. 3
- Data distillation uses pseudo-labels from a strong pre-trained model to exploit raw images, while distillation of both MiniVLM modules is left for future work.Prior demonstrations cited in image classification include [62] [22], and in object detection.Source: §2 · p. 3
3. MiniVLM
MiniVLM pairs a detector-based visual feature extractor with a transformer fusion module, adapting the prediction head for downstream tasks. Its design simplifies visual extraction and uses large-scale pre-training data and teacher-generated labels to support a compact model.
3. MiniVLM
- MiniVLM combines a detector-based feature extractor and transformer fusion module, with task-specific prediction heads requiring minimal changes.The model is fine-tuned for varied downstream VL tasks.Source: §3 · p. 4
3.1. Model architecture
- TEE prioritizes rich visual representations and bounding-box recall over precise box locations, which VL tasks do not require to the same degree as detection.This design choice supports a more efficient extractor without significant downstream accuracy degradation.Source: §3.1 · p. 4
- TEE uses EfficientNet and BiFPN with depthwise and pointwise convolutions, taking stride-4, -8, -16, and -32 features and adding a stride-64 output.Starting from stride 4, rather than 8 as in EfficientDet, incorporates higher-resolution feature maps.Source: §3.1 · p. 4
- TEE scales input size, network depth, and width across variants, while MiniVLM uses TEE-0; inference supplies region features, boxes, and region-class tags to the transformer.The extractor itself supplies the tags, so no separate tagging model is used at inference.Source: §3.1 · p. 4
- The fusion module uses MiniLM’s 12 layers, 384 hidden size, and 1536 intermediate size, combining region features and box encodings with object tags and task text.The text input can be a VQA question or a [CLS] start token for captioning.Source: §3.1 · p. 4
3.2. Pre-training
- Pre-training first trains TEE on large classification and detection datasets before Visual Genome fine-tuning, and uses large-scale image-text data for transformer representations.A captioning model pseudo-labels 7M Open Images, while high-quality teacher tags and available ground-truth tags support cross-modal learning; fine-tuning tags come from the vision model itself.Source: §3.2 · pp. 4–5
4. Experiment
The experiments evaluate MiniVLM’s compression choices, pretraining improvements, and downstream performance. They also analyze how visual backbones, object tags, and compact transformer structures affect accuracy or efficiency.
4.1. Implementation details
- TEE is pretrained on ImageNet, then Objects365, and finally Visual Genome; OI-Caps-7M adds pseudo-captions generated from Open Images V6.The V+L pretraining combines MS COCO, Conceptual Captions, SBU captions, Flicker30k, GQA, VQA, and VG-QA.Source: §4.1 · p. 5
4.2. Smaller and faster
- MiniVLM uses 27% of the parameters and 1% of the FLOPS of the model in [26], while TEE uses 4.4/767.0 = 1% of R101-F FLOPS.Processing one image at a time, TEE takes 393.9 ± 43.8 ms, or 3.2% of R101-F’s 12.3 ± 3.2 seconds.Source: §4.2 · pp. 5–6
4.3. Retaining high accuracy
- Replacing only MiniLM or TEE-0 reduced CIDEr by 4.9 or 5.0 points; replacing both reduced it by 8.7 points.The two-module compact model is the baseline before the reported pretraining improvements.Source: §4.3 · p. 6
- Objects365 pretraining added 1.7 CIDEr, high-quality tags added 1.0, and OI-Caps-7M added 2.1, for a total gain of 4.8 CIDEr.The passage reports a similar trend for VQA.Source: §4.3 · p. 6
4.4. Results on downstream VL tasks
- MiniVLM retains 94–97% of OSCARB accuracy across downstream tasks; its COCO captioning score is 119.8 CIDEr versus OSCARB’s 123.7.Against grid-feature [17], MiniVLM scores 119.8 versus 113.8 CIDEr, with 4.4 versus 161.2 FLOPS for feature extraction.Source: §4.4 · p. 7, Table 6 · p. 8, Table 7 · p. 8
4.5. Analysis
- Object tags improved captioning by 2+ points and VQA by 1+; stronger TEE backbones also scored higher on both tasks.Higher-quality tags helped further, while random transformer initialization gave results comparable to text-pretrained weights.Source: §4.5 · p. 7, Figure 3 · p. 7
- MiniLM had a better speed-accuracy trade-off than the tested compact BERT structures, with thinner BERT variants favored over shallower ones.Figure 4 presents the comparison for captioning and VQA (test-dev).Source: §4.5 · p. 8, Figure 4 · p. 8
5. Conclusion
MiniVLM is a compact, faster vision-language model designed for resource-constrained devices. It retains 94–97% of state-of-the-art accuracy with 27% of the parameters and 1% of the FLOPS.
5. Conclusion
- 94–97% of accuracy with 27% parameters and 1% FLOPS compared to the state-of-the-art VL model.The model is fine-tuned on various downstream VL tasks.Source: §5 · p. 9
- TEE reduces computation by simplifying the region head and replacing regular convolutions with pointwise and depthwise convolutions.MiniVLM also leverages large models and large-scale datasets to improve small-model pre-training.Source: §5 · p. 9