Source-linked AI summary
VisualBERT: A Simple and Performant Baseline for Vision and Language
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang
TL;DR
Vision-language tasks require models to connect detailed image semantics with natural-language references, while many existing systems are task-specific. VisualBERT jointly processes text and detected image regions with Transformer self-attention and uses visually grounded caption pre-training. Across four tasks, it outperforms or rivals state-of-the-art models, while analysis indicates implicit entity and dependency grounding; its attention-based grounding can nevertheless assign little total attention to image regions.
Problem
Vision-language tasks require systems to understand detailed image semantics and how objects, attributes, relations, and actions are expressed in language.
Method
VisualBERT jointly processes text and object-proposal features with Transformer self-attention and pre-trains on paired captions using visually grounded objectives.
Results
Across VQA, VCR, NLVR2, and Flickr30K, VisualBERT outperforms or rivals state-of-the-art models and learns implicit grounding of entities and certain dependency relationships.
Takeaways & Limitations
A simple jointly trained vision-language representation can support multiple tasks while internally aligning language with image regions without explicit alignment supervision.
Takeaways & Limitations
Some heads that accurately ground entities allocate less than 20% of their attention from entities to image regions.
Abstract
from arXiv · showhide
We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.
1 INTRODUCTION
VisualBERT is a simple, flexible vision-language model that jointly processes text and image-region inputs, using visually grounded pre-training to support multiple tasks. Experiments show strong performance and internal grounding of concepts and some syntactic relations.
- Vision-and-language tasks test understanding of objects, attributes, spatial relationships, actions, intentions, and their grounding in language.
- VisualBERT jointly feeds text and unordered object-proposal features through Transformer layers to capture associations between words and image regions.It combines BERT with pre-trained object-proposal systems such as Faster-RCNN.
- Two visually grounded objectives pre-train VisualBERT on caption data: masked-word prediction using visual context and sentence-image matching.
- Attention analysis shows implicit grounding of visual concepts and certain syntactic dependencies, with alignments refined across Transformer layers.For example, “walking” aligns to the man region, while an incorrect “man”–“shirt” alignment is corrected in higher layers.
- Four tasks—VQA, VCR, NLVR2, and Flickr30K—show that COCO-pre-trained VisualBERT outperforms or rivals state-of-the-art models.
2 RELATED WORK
Prior vision-language systems commonly combine separate text and image modules for task-specific applications. VisualBERT instead uses a general Transformer-based joint representation that implicitly models relations between language and image regions.
- Vision-language research includes VQA, textual grounding, and visual reasoning, with many systems built for specific tasks.
- Typical systems contain a text encoder, image feature extractor, multimodal fusion module, and answer classifier.
- VisualBERT is general, easily adapted to new tasks, and incorporable into task-specific models.
- Prior work models image semantics through annotations, attention-based object relations, or explicit graphs; VisualBERT captures implicit relations with self-attention and caption pre-training.
- Compared with a concurrent study, VisualBERT’s results are consistent on intersecting tasks, but methods are not wholly comparable because visual representations and pre-training resources differ.
3 A JOINT REPRESENTATION MODEL FOR VISION AND LANGUAGE
VisualBERT extends BERT into a joint vision-language representation model by treating detected image regions as visual embeddings processed alongside text. It uses Transformer self-attention and staged pre-training before task-specific fine-tuning.
- 3.1 BACKGROUND: BERT maps subwords to token, segment, and position embeddings, then contextualizes them with a multilayer Transformer.
- 3.2 VISUALBERT: VisualBERT reuses Transformer self-attention to implicitly align text elements with image regions represented by detector-derived visual embeddings.
- 3.2 VISUALBERT: Each visual embedding combines a region feature, an image-segment embedding, and optionally position information from aligned words.
- 3.3 TRAINING PROCEDURE: COCO provides paired images and captions for VisualBERT’s multimodal training procedure.
- 3.3 TRAINING PROCEDURE: Task-agnostic pre-training masks text while retaining image regions and trains sentence-image prediction using captions associated with the same image.
- 3.3 TRAINING PROCEDURE: Before fine-tuning, task-specific masked language modeling with images adapts VisualBERT to the target domain.
- 3.3 TRAINING PROCEDURE: Fine-tuning introduces task-specific inputs, outputs, and objectives while training the Transformer to maximize task performance.
4 EXPERIMENT
VisualBERT is evaluated across four vision-and-language applications using task-specific visual representations and model variants. Across these benchmarks, it is reported as simpler than comparable methods while outperforming prior systems or state-of-the-art baselines.
- Experimental setup: VisualBERT is evaluated on VQA, VCR, NLVR2, and Flickr30K region-to-phrase grounding.The experiments use the Karpathy train split and follow task-specific detector and region-feature settings.
- Experimental setup: The full VisualBERT model uses BERT initialization, COCO pre-training, task-data pre-training, and task fine-tuning.The ablations remove early fusion or COCO pre-training to test their importance.
- VQA: In VQA, VisualBERT is significantly simpler and outperforms existing work in comparable settings.The comparison includes baselines using the same visual features and number of region proposals.
- VCR: On VCR, the model without COCO pre-training outperforms R2C by a large margin, while the full model further improves results.COCO pre-training still helps despite the domain difference between COCO images and movie scenes.
- NLVR2: On NLVR2, both ablation models significantly outperform MaxEnt, while the full model widens the gap.VisualBERT assigns different segment embeddings to features from the two images.
- Flickr30K Entities: On Flickr30K, VisualBERT outperforms BAN, with no significant difference between the full model and the no-early-fusion ablation.The authors suggest that a shallower architecture may be sufficient for this task.
5 ANALYSIS
The analysis identifies task-agnostic pre-training and early vision-language fusion as VisualBERT’s most important design choices, then examines implicit entity and syntactic grounding through attention.
- Analysis: 64.8, 65.5, 66.7, 67.1, and 67.4 accuracy are obtained with 9, 18, 36, 72, and 144 object proposals, respectively, on development data.Increasing the number of proposals improves accuracy in this preliminary experiment.
- 5.1 Ablation Study: Removing BERT initialization reduces performance less than expected, suggesting COCO pre-training learns many useful aspects of grounded language.The analysis still finds language-only BERT weights important.
- 5.1 Ablation Study: Removing the sentence-image prediction objective has a positive but less significant effect than removing other components.This objective contributes, but less strongly than task-agnostic pre-training and early fusion.
- 5.1 Ablation Study: Task-agnostic pre-training and early fusion are identified as VisualBERT’s most important design choices.Both paired vision-language pre-training and multiple early interaction layers contribute substantially to performance.
- 5.2 Dissecting Attention Weights: Many attention heads accurately ground entities without direct supervision, with grounding accuracy appearing to improve in higher Transformer layers.The analysis compares heads against a baseline that selects the region with the highest detection confidence.
- 5.2 Dissecting Attention Weights: VisualBERT contains heads that detect dependency relationships and implicitly associate syntactic arguments, such as verbs with their corresponding visual regions.The evaluation examines dependency relations including “pobj”, “amod”, “nsubj”, and “dobj”.
- 5.3 Qualitative Analysis: Some heads achieve entity-grounding accuracy while assigning less than 20% of their attention from entities to image regions.Accuracy therefore reflects the top attended region, not necessarily strong overall visual attention.
- 5.3 Qualitative Analysis: Across successive Transformer layers, VisualBERT can refine ambiguous alignments and disentangle visually corresponding entities.Qualitative examples show initially overlapping attention becoming more specific by the end of computation.
6 CONCLUSION AND FUTURE WORK
VisualBERT is presented as a pre-trained model for joint vision-language representation that achieves strong performance across four evaluation tasks. The analysis suggests its attention mechanism captures information interpretably, while future work could extend the model to image-only tasks and larger caption datasets.
- Conclusion: VisualBERT achieves strong performance on four evaluation tasks despite its simple design.The conclusion characterizes the model as a pre-trained joint vision-language representation.
- Conclusion: Attention analysis suggests that VisualBERT captures information in an interpretable way.The conclusion connects the model’s performance with its attention-based analysis.
- Future Work: Future work includes extending VisualBERT to image-only tasks and pre-training it on larger caption datasets.The paper names scene graph parsing, situation recognition, Visual Genome, and Conceptual Captions as directions.
IMPLEMENTATION AND TRAINING DETAILS
The paper introduces implementation and training details for each evaluation task.
- Implementation and Training Details: Implementation and training details are provided separately for each evaluation task.
A VQA
For VQA, the paper adapts the task to a classification setup using a limited answer pool and a masked-token representation for prediction.
- VQA: VQA is treated as classification over a limited answer pool rather than open-ended generation.The processing follows Pythia’s procedure.
- VQA: A [MASK] token appended after the question provides the representation used for answer classification.The [MASK] representation is fed into an output layer.
- VQA: Multiple correct answers are assigned equal probability during training.
B VCR
VisualBERT’s VCR setup evaluates two multiple-choice subtasks using sequences that combine questions, choices, and images. The models use COCO task-agnostic pre-training and VCR task-specific pre-training with shared hyperparameters across variants.
- VCR is decomposed into question answering (Q →A) and answer justification (QA →R).
- Each VCR example becomes four input sequences formed by concatenating the question, a choice, and an image.
- VisualBERT variants receive COCO task-agnostic pre-training for 10 epochs with batch size 128 and maximum learning rate 1e-4.
- The variants also use task-specific pre-training with the same learning rate, batch size, optimizer, and warm-up ratio hyperparameters.
C NLVR2
For NLVR2, VisualBERT processes captions together with features from two images. Its training includes COCO pre-training and an auxiliary objective for deciding whether a caption is true.
- Each NLVR2 training example consists of a caption and image features from two images.
- VisualBERT variants are pre-trained on COCO for 10 epochs with batch size 64 and maximum learning rate 5e-5.
- Task-specific pre-training adds an auxiliary task that decides whether the caption in a training example is true.
D FLICKR30K
For Flickr30K region-to-phrase grounding, VisualBERT uses cross-entropy training when multiple boxes may align with the same phrase. Its pre-training uses only the masked-language-modeling-with-image objective before fine-tuning.
- Multiple boxes aligned to one phrase are trained with the same cross-entropy loss used for VQA, unlike BAN’s binary cross-entropy loss.
- VisualBERT is pre-trained on COCO with batch size 32 and learning rate 5e-5.
- During task-specific pre-training, only masked language modeling with the image is used for 10 epochs.
- The models are fine-tuned for a maximum of 5 epochs with batch size 32 and learning rate 2e-5.