Source-linked AI summary
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee
TL;DR
Vision-and-language tasks lack a unified foundation for learning visual grounding, instead typically learning it separately during task training. ViLBERT pretrains a two-stream, co-attentional model and achieves state-of-the-art performance across four established tasks, with improvements of 2 to 10 percentage points over task-specific baselines.
Problem
Vision-and-language approaches lack a unified foundation for aligning language and visual stimuli, often learning grounding separately during task training.
Method
ViLBERT pretrains separate vision and language streams that exchange information through co-attentional transformer layers on paired image-text data.
Results
2 to 10 percentage points improvements across four tasks yield state-of-the-art performance against task-specific baselines.
Takeaways & Limitations
Visual grounding can serve as a transferable capability across multiple established vision-and-language tasks from a common pretrained architecture.
Takeaways & Limitations
The evaluation omits visually grounded dialog, embodied tasks, and text-generation tasks, leaving extensions to long multimodal sequences and output decoding unresolved.
Abstract
from arXiv · showhide
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.
1 Introduction
ViLBERT addresses visual grounding by extending BERT into a task-agnostic joint model for paired vision-and-language data. It uses separate visual and linguistic streams with co-attentional transformer layers, pretrains on Conceptual Captions through two proxy tasks, and transfers to four vision-and-language tasks.
- Motivation: Vision-and-language systems must align natural language with visual stimuli, a capability referred to as visual grounding.
- Approach: ViLBERT extends BERT to jointly reason about text and images while learning task-agnostic visual grounding from paired visiolinguistic data.
- Architecture: Separate visual and linguistic processing streams communicate through co-attentional transformer layers, accommodating differing modality-specific processing needs.
- Pretraining: ViLBERT is pretrained on Conceptual Captions with two proxy tasks: predicting masked-word and masked-region semantics, and determining whether an image and text segment correspond.
- Transfer: The pretrained model is applied to visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval.
2 Approach
ViLBERT extends BERT into a two-stream model that separately processes image regions and text while exchanging information through co-attentional transformer layers. It is pretrained with masked multi-modal modelling and multi-modal alignment prediction to learn joint visual-linguistic representations.
- Model architecture: ViLBERT uses two parallel BERT-style streams for image regions and text, combining transformer blocks with co-attentional layers for cross-modal information exchange.The visual stream receives region features, while the text stream receives token sequences and produces modality-specific final representations.
- Model architecture: Co-attention passes each modality’s keys and values into the other modality’s multi-headed attention, producing representations conditioned on the other modality.This performs image-conditioned language attention in the visual stream and language-conditioned image attention in the linguistic stream.
- Input representations: Image inputs are represented as object-detection region features augmented with spatial encodings based on normalized box coordinates and image-area coverage.Because image regions lack a natural ordering, spatial location supplies the positional information.
- Pretraining objectives: ViLBERT pretrains on paired image-text data using masked multi-modal modelling and multi-modal alignment prediction.The first reconstructs masked words or image regions from the remaining inputs, while the second predicts whether an image and text are aligned.
3 Experimental Settings
ViLBERT is pretrained on millions of automatically collected image-caption pairs using specified vision-language training tasks, then fine-tuned end-to-end for established vision-and-language tasks with minimal task-specific modifications. The experiments also include a zero-shot retrieval setting that applies pretrained multimodal alignment without fine-tuning.
- Pretraining Data: 3.1 million image-caption pairs from Conceptual Captions are used to train ViLBERT after broken links reduce the original 3.3 million-pair collection.The dataset is automatically scraped from alt-text-enabled web images and contains noisy, sometimes non-human-like captions.
- Implementation Details: The linguistic stream is initialized with BERTBASE, while visual features come from Faster R-CNN with a ResNet-101 backbone pretrained on Visual Genome.BERTBASE has 12 transformer layers, hidden size 762, and 12 attention heads; visual streams use hidden size 1024 and 8 attention heads.
- Implementation Details: Training uses 8 TitanX GPUs, batch size 512, 10 epochs, Adam with an initial learning rate of 1e-4, linear decay with warmup, and equally weighted task losses.The model retains 10 to 36 high-scoring detected regions per image.
- Transfer Tasks: ViLBERT transfers to four established vision-and-language tasks and one diagnostic task by modifying the pretrained model and fine-tuning the entire architecture end-to-end.The task-specific modification is typically only a learned classification layer, contrasting with specialized task-specific model development.
- Transfer Tasks: The transfer suite covers VQA, VCR, referring-expression grounding, and caption-based image retrieval, using task-specific datasets and lightweight prediction heads.VQA uses VQA 2.0 with 1.1 million questions and 3,129 possible answers; VCR uses 290k multiple-choice problems from 110k movie scenes; retrieval uses Flickr30k with 31,000 images and five captions each.
- Zero-Shot Retrieval: In zero-shot caption-based image retrieval, pretrained multimodal alignment is applied directly to Flickr30k without task-specific fine-tuning to test transferable text grounding.This setting is intended to assess generalization across visual and linguistic variation.
4 Results and Analysis
ViLBERT outperforms task-specific state-of-the-art models across four transfer tasks, with gains attributable to its two-stream architecture and Conceptual Captions pretraining. Results also show benefits from increased model depth, larger pretraining datasets, and semantically meaningful zero-shot alignment.
- Results: ViLBERT outperforms task-specific state-of-the-art models across all four transfer tasks using a single base architecture and one classifier per task.It achieves state-of-the-art on VCR, RefCOCO+, and image retrieval by 7-10 percentage points.
- Results: ViLBERT improves performance over the single-stream baseline across tasks for both pretrained and non-pretrained models, especially on VQA and RefCOCO+.The comparison isolates the impact of the two-stream architecture while retaining BERT initialization and Faster R-CNN visual features in the baseline.
- Results: Pretraining under ViLBERT’s proxy tasks improves performance by between 2% and 13% across tasks and also benefits the single-stream architecture.This supports improved visiolinguistic representations and generalization across model architectures.
- Effect of Visual Stream Depth: VQA and image retrieval performance increases monotonically with visual-stream depth until six layers, while zero-shot image retrieval continues gaining as depth increases.Different tasks perform better at different depths, implying different context-aggregation needs.
- Benefits of Large Training Sets: Pretraining accuracy grows monotonically as the Conceptual Captions dataset expands from 25% to 50% and the full dataset, suggesting benefits from more data.The experiment uses random 25% and 50% subsets with the same pretraining and fine-tuning setup.
- What does ViLBERT learn during pretraining?: 31.86 vs 58.20 R1 shows zero-shot retrieval trails fine-tuning, yet 31.86 vs 48.60 R1 indicates meaningful alignment without Flickr30k images or captions.Qualitative examples further examine what ViLBERT learns during Conceptual Captions pretraining.
5 Related Work
Related work situates ViLBERT within self-supervised representation learning for vision and language, while identifying vision-and-language task families not addressed and concurrent video-language pretraining.
- Self-Supervised Learning: Self-supervised learning trains deep models on automatically derived supervision from regularities in existing data, with progress reported separately in vision and language.The passage notes that self-supervised image representations still lag behind those trained with image classification.
- Vision-and-Language: The approach does not cover visually grounded dialog, embodied question answering and instruction following, or image and video captioning.These task families are proposed as potential beneficiaries of similar self-supervised methods.
- Self-Supervised Learning for Vision-And-Language: Concurrent work learns joint video-language representations from cooking videos paired with text-to-speech transcripts using self-supervised tasks paralleling this approach.That work uses a unified BERT architecture for visual and linguistic inputs, similar to the Single-Stream baseline considered here.
6 Conclusion
ViLBERT jointly models image content and text through a two-stream architecture with co-attentional transformer blocks, pretrained on a large automatically collected dataset to learn visual grounding. Transferred with only a classifier addition, it outperforms ablations and exceeds state-of-the-art on multiple established vision-and-language tasks.
- 6 Conclusion: ViLBERT jointly models image content and text using a novel two-stream architecture with co-attentional transformer blocks.The architecture is designed to learn joint representations across visual and textual inputs.
- 6 Conclusion: Pretraining on a large, automatically collected dataset enables ViLBERT to learn visual grounding before task transfer.The model is pretrained as a joint image-and-text representation rather than learning grounding only during downstream task training.
- 6 Conclusion: Transferred to multiple established vision-and-language tasks, ViLBERT outperforms sensible ablations and exceeds state-of-the-art.Task adaptation requires only minor architectural additions, specifically a classifier.