Source-linked AI summary
Unified Vision-Language Pre-Training for Image Captioning and VQA
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, Jianfeng Gao
TL;DR
Vision-language pre-training lacks a single model that readily supports both generation and understanding. VLP shares one encoder-decoder Transformer, trains it with bidirectional and seq2seq masked prediction using attention masks, and achieves state-of-the-art results across captioning and VQA benchmarks.
Problem
A single unified vision-language model remains difficult because existing systems often support understanding only or use separately trained encoders and decoders.
Method
VLP shares a multi-layer Transformer for encoding and decoding and pre-trains on image-caption pairs with bidirectional and seq2seq masked language prediction controlled by self-attention masks.
Results
VLP achieves state-of-the-art results on image captioning and VQA across COCO Captions, Flickr30k Captions, and VQA 2.0.
Takeaways & Limitations
One parameter-shared architecture supports two distinct vision-language prediction objectives and downstream generation and understanding tasks.
Takeaways & Limitations
The authors identify interference between different objectives as a motivation for future multi-task fine-tuning work.
Abstract
from arXiv · showhide
This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.
Introduction
Existing vision-language pre-training models improve individual tasks but remain difficult to unify across generation and understanding. VLP addresses this with shared encoder-decoder pre-training and reports strong results across captioning and VQA benchmarks.
- Vision-language pre-training extends language-model pre-training to image-text representations for tasks such as VQA and video captioning.
- Existing models commonly pre-train contextualized representations and then fine-tune separate downstream task models.
- A single model applicable to both vision-language generation and understanding remains challenging because many systems target understanding only or separate encoders and decoders.
- VLP shares a multi-layer Transformer for encoding and decoding and supports both tasks through bidirectional and seq2seq masked language prediction.
- VLP achieves state-of-the-art results on image captioning and VQA across COCO Captions, Flickr30k Captions, and VQA 2.0 while speeding fine-tuning.
Related Work
Prior work largely builds vision-language pre-training on BERT-style encoders, while related language models demonstrate how attention masks can support multiple prediction directions.
- BERT-based vision-language models pre-train contextualized image-text representations by predicting masked words or regions, then fine-tune downstream tasks.
- UniLM uses one shared Transformer with distinct self-attention masks for unidirectional, bidirectional, and sequence-to-sequence language modeling.
- VideoBERT addresses generation and understanding tasks but separates its visual encoder and language decoder.
Vision-Language Pre-training
VLP represents images with detector-based region features and combines them with masked sentence tokens in one Transformer. Bidirectional and seq2seq objectives differ through self-attention masks that control available context.
- Vision-Language Transformer Network: The model represents each image with a fixed number of detector-generated object regions, including visual, class-probability, and geometric features.
- Vision-Language Transformer Network: A single Transformer network unifies the encoder and decoder for class-aware region embeddings, word embeddings, and special tokens.
- Pre-training Objectives: Masked language modeling reconstructs selected sentence tokens after replacing them with [MASK], random, or original tokens using 80%, 10%, and 10% probabilities.
- Pre-training Objectives: Bidirectional prediction permits unrestricted visual-language message passing, whereas seq2seq prediction blocks attention to future words.
- Pre-training Objectives: The two objectives alternate across batches, with their proportions controlled by λ and 1 −λ.
- Pre-training Objectives: Incorporating region class probabilities into region features outperforms using masked region classification as the visual pre-training task.
Fine-Tuning for Downstream Tasks
VLP is fine-tuned differently for image captioning and visual question answering: generation uses iterative masked-word prediction, while VQA uses multi-label classification.
- Image Captioning: Captioning fine-tuning generates words iteratively by replacing each [MASK] with a sampled word and appending a new [MASK].The model uses the seq2seq objective during fine-tuning and inference.
- Visual Question Answering: VQA is formulated as multi-label classification over the 3,129 most frequent answers.A classifier uses the element-wise product of the final [CLS] and [SEP] hidden states.
- Evaluation: Table 3 reports COCO Captions results for single models optimized with CIDEr.The table highlights the top result for each metric.
- Evaluation: The pre-training does not require extensive human annotations, making it more general than approaches with that requirement.This comparison concerns the data requirements of the pre-training procedure.
Experiments and Results
Experiments evaluate Unified VLP across captioning and VQA benchmarks, comparing pre-training levels, initialization choices, visual-region objectives, and qualitative behavior. The model improves downstream performance and learning speed, with especially strong gains on Flickr30k and shared-task successes and failures.
- Experimental setup: Unified VLP is evaluated on COCO Captions, Flickr30k, and VQA 2.0 using downstream splits and implementation settings described for pre-training and fine-tuning.Pre-training uses Conceptual Captions, while captioning and VQA use their respective benchmark datasets.
- Benchmark results: The full model outperforms state-of-the-art methods on three of four COCO metrics, VQA overall accuracy, and all four Flickr30k metrics.All image-captioning methods are single models trained with cross-entropy optimization for comparison.
- Pre-training analysis: Joint bidirectional and seq2seq pre-training combines representations for generation and understanding, yielding slightly compromised but decent accuracy across downstream tasks.Seq2seq-only pre-training favors captioning, while bidirectional pre-training favors VQA; the joint model generally performs well across both.
- Pre-training analysis: Vision-language pre-training accelerates downstream learning and improves overall accuracy relative to weaker pre-training levels, including scratch, BERT, and UniLM initialization.The paper describes four pre-training degrees, from no pre-training through full Vision-Language Pretraining.
- Model analysis: Using region class probabilities as model input performs better than using class prediction as a masked-region pretext, which negatively affects captioning performance.The authors hypothesize that noisy detector class labels compromise representations when used as a prediction pretext.
- Qualitative analysis: Qualitative examples show better image detail capture and correct answers in successful cases, while shared representations coincide with the same activity mistake across captioning and VQA.The shared error concerns confusing kayaking or boating with surfing.
Conclusion
The paper presents VLP as a unified model for image captioning and VQA, using shared representations and avoiding separate pre-trained models for different downstream task types. Its unified design achieves strong downstream accuracy, while slightly compromising relative to separate objective-specific models.
- VLP supports both vision-language generation and understanding tasks within one unified pre-training framework.
- The model uses parameter sharing across bidirectional and seq2seq objectives, avoiding separate pre-trained models for different downstream task types.
- Large-scale unsupervised pre-training significantly speeds downstream learning and improves model accuracy on image captioning and VQA.
- VLP yields slightly compromised but decent state-of-the-art accuracy across downstream tasks compared with separate pre-trained models.
Appendix
The appendix reports validation-set results and implementation details for the downstream experiments. It specifies the VQA training-set protocol and the region-feature extraction setup.
- Validation results cover COCO Captions, VQA 2.0, and Flickr30k, with separate test-set training protocols noted for VQA.VQA validation results use only the training set, whereas reported test results use both training and validation sets.
- Image regions are extracted with a Faster R-CNN variant using a ResNeXt-101 FPN backbone pre-trained on Visual Genome.
- The implementation uses exactly 100 regions per image for region-based visual features.