Source-linked AI summary
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, Yuan Cao
TL;DR
Existing VLP methods require expensive annotations and complex, multi-objective pretraining, limiting scalability. SimVLM addresses this with raw images, weakly labeled image-text data, and one end-to-end PrefixLM objective, achieving strong results across discriminative and generative benchmarks while supporting zero-shot transfer.
Problem
Existing VLP approaches require expensive image captions or regional annotations and complicated dataset-specific objectives, limiting scalable pretraining.
Method
SimVLM trains from scratch on weakly labeled image-text data using raw image patches and a single end-to-end PrefixLM objective.
Results
SimVLM consistently outperforms existing VLP models on six vision-language benchmarks without additional data or task-specific customization.
Takeaways & Limitations
SimVLM demonstrates that generative VLP with weak supervision can achieve competitive multimodal performance while enabling zero-shot captioning, open-ended VQA, and cross-modality transfer.
Abstract
from arXiv · showhide
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.
1 INTRODUCTION
Existing vision-language pretraining methods use complex, annotation-heavy procedures, motivating SimVLM’s simpler weakly supervised alternative. SimVLM uses one PrefixLM objective and achieves strong benchmark performance and zero-shot generalization.
- Motivation: Existing VLP methods commonly use object detection, aligned image-text data, and auxiliary losses, complicating pretraining.These choices create a bottleneck for further quality improvement and usually lack zero-shot capability.
- Goals: SimVLM targets competitive benchmark performance, a simpler protocol, and text-guided zero-shot generalization.Its design is motivated by these three goals.
- Approach: SimVLM is trained end-to-end with a single PrefixLM objective on weakly aligned image-text pairs.The framework uses language modeling objectives rather than multiple task-specific objectives.
- Approach: The model uses ViT/CoAtNet-style processing of raw images and large-scale weakly labeled data, avoiding object detection requirements.These choices are intended to support scalable pretraining and zero-shot generalization.
- Results: SimVLM outperforms existing VLP models on six benchmarks without additional data or task-specific customization.It also supports zero-shot image captioning, open-ended VQA, and cross-modality transfer.
2 RELATED WORK
Prior VLP systems often depend on object detection and compound cross-modal losses, making optimization and data requirements complex. SimVLM instead uses raw images, weak labels, and a single language-modeling loss.
- Prior VLP: Many VLP methods require object detection for image-region feature regression or tagging during pretraining.This dependence introduces an auxiliary detection model such as Faster R-CNN.
- Prior VLP: Cross-modal objectives such as image-text matching, region prediction, contrastive loss, and alignment are often combined into compound losses.Balancing multiple losses and datasets complicates optimization.
- SimVLM: SimVLM takes raw image inputs and uses only a language-modeling loss instead of auxiliary region-detection models.The model is pretrained from scratch with large-scale weakly labeled data.
3 SIMVLM
SimVLM replaces masked or standard autoregressive language modeling with PrefixLM, combining bidirectional prefix context with autoregressive generation. Its architecture processes raw images as patches and supports scalable weakly supervised pretraining.
- Background: Masked language modeling reconstructs randomly masked tokens from corrupted textual context, while standard language modeling predicts tokens autoregressively.MLM is widely used in prior VLP models, whereas LM supports generation and text-induced zero-shot generalization.
- PrefixLM: PrefixLM applies bidirectional attention to a prefix and autoregressive factorization to the remaining tokens.This combines contextual representation learning with generation.
- PrefixLM: For image-text pairs, image features are prepended to text and the prefix length is sampled to include the image tokens.The loss is calculated on the text portion only.
- Architecture: The model uses Transformer layers with raw images mapped into flattened patch sequences and standard subword text tokens.Separate positional embeddings are used for image and text inputs, with 2D relative attention for image patches.
- Data: SimVLM is pretrained from scratch on large-scale noisy image-alt-text pairs without an object detection module.Its modality-agnostic formulation can also incorporate text-only data.
- Training: Compared with two-stage methods using multiple auxiliary objectives, SimVLM requires one-pass end-to-end pretraining with a single language-modeling loss.This is the framework’s central simplification.
4 EXPERIMENTS
SimVLM is evaluated across discriminative, generative, zero-shot, cross-modality, open-ended VQA, and single-modality benchmarks. It consistently outperforms prior methods while supporting transfer and generalization beyond standard supervised evaluation.
- 4 EXPERIMENTS: SimVLM achieves new state-of-the-art results across six vision-language benchmarks spanning VQA, captioning, visual reasoning, visual entailment, and multimodal translation.The evaluation includes three discriminative tasks and three generative tasks.
- 4.2 COMPARISON WITH EXISTING APPROACHES: SimVLMhuge improves nearly 4 absolute points over VinVL and pushes single-model VQA performance above 80%.SimVLMbase also outperforms prior methods while using less capacity.
- 4.3.1 ZERO-SHOT/FEW-SHOT IMAGE CAPTIONING: SimVLM demonstrates zero-shot and few-shot image-captioning generalization competitive with fully supervised CoCo baselines and stronger pretrained-model performance on NoCaps.The zero-shot setting directly decodes from the pretrained model, while the few-shot setting uses 1% of training data.
- 4.3.2 ZERO-SHOT CROSS-MODALITY TRANSFER: SimVLM transfers knowledge across modalities and languages, achieving comparable performance to supervised baselines on SNLI-VE and Multi30k.The experiments transfer from text-only data to image-text tasks, including English-to-German image translation.
- 4.3.3 OPEN-ENDED VQA: Generative SimVLM improves out-of-domain VQA performance by over 17 points and can generate answers outside the 3,129-answer candidate set.After additional pretraining on cleaner knowledge-rich data, open-ended VQA ability emerges, although the initial pretrained model struggles with meaningful answers to real questions.
- 4.4 ANALYSIS: SimVLM learns useful single-modality representations, performing competitively with BERT on GLUE and producing high-quality image representations for ImageNet evaluation.The model was not pretrained with a discriminative contrastive objective for ImageNet.
5 CONCLUSION
SimVLM presents a simple vision-language pretraining framework that processes whole images as patches and trains end-to-end with a single prefix language modeling objective.
- SimVLM processes whole images as patches instead of using object proposal systems.
- The model is trained end-to-end with a single prefix language modeling objective.
- The framework is presented as an alternative to existing vision-language pretraining paradigms.
A GENERATED EXAMPLES
SimVLM generates examples from image-only or image-text prefixes, while its text-only GLUE results are reported in a benchmark table.
- SimVLM uses image-only or image-text prefix inputs in the encoder and generates suffix text with the decoder.
- Figure 2 presents examples generated by SimVLM across various types.
- Table 7 reports text-only task performance on the GLUE development set and compares BERT with other VLP methods.
- The table bolds the overall best result and underlines the best VLP model.
B.1 PRETRAINING
SimVLM pretraining uses large-scale web data for image-text and text-only inputs and optimizes a single PrefixLM objective from scratch.
- SimVLM pretraining uses Transformer variants aligned with corresponding ViT variants and convolutional stages based on ResNet architectures.Base and Large use the first three blocks of ResNet-101 and ResNet-152, while Huge uses a larger ResNet-152 variant.
- The models use a fixed 16×16 patch size throughout pretraining.
- SimVLM trains on about 1.8B noisy image-text pairs from ALIGN without extra preprocessing or filtering beyond random resized cropping.
- Text-only pretraining uses the C4 dataset and follows its published preprocessing steps.
- All models train from scratch for about 1M steps using one PrefixLM objective, mixing image-text and text-only datasets within each batch.
B.2 FINETUNING
After pretraining, SimVLM is fine-tuned across discriminative and generative vision-language tasks using task-specific input formulations and standard downstream evaluation setups.
- Fine-tuning uses AdamW with the pretraining beta values, tunes the learning rate over three options, and adds dropout and stochastic depth regularization.Dropout and stochastic depth are set to 0.1, with stochastic depth applied to the convolutional stage and encoder.
- Visual question answering: VQA is formulated as classification over the 3,129 most frequent training-set answers from image-question inputs.The decoder’s final question-token activation feeds a task-specific linear classifier.
- Visual entailment: SNLI-VE predicts entailment, neutral, or contradiction from an image-based premise and a hypothesis sentence.
- Visual reasoning: NLVR2 evaluates whether a textual description is true for a pair of images by concatenating embeddings from two image-description pairs.
- Image captioning: Image captioning encodes an image and generates captions for COCO and NoCaps, using COCO training data without CIDEr optimization.
- Multimodal translation: Multimodal translation combines source sentences with image inputs in PrefixLM and generates target-language translations on Multi30k.
C MODEL PERFORMANCE ON LANGUAGE-ONLY TASK
SimVLM is evaluated on natural language understanding tasks using the GLUE benchmark.
- SimVLM is compared with prior vision-language pretraining methods on the GLUE benchmark.
D ERRATUM
The paper reports an error in its initial zero-shot COCO evaluation, updates the numbers, and adds few-shot results for broader capacity assessment.
- An error was found in the zero-shot COCO evaluations reported in the paper’s first version.
- The corrected numbers do not affect the other results, and the authors state that the main claims and conclusions still hold.
- Few-shot results were added for MSCOCO and NoCaps alongside the corrected zero-shot results.