VinVL: Revisiting Visual Representations in Vision-Language Models

Pengchuan Zhang; Xiujun Li; Xiaowei Hu; Jianwei Yang; Lei Zhang; Lijuan Wang; Yejin Choi; Jianfeng Gao

Source-linked AI summary

VinVL: Revisiting Visual Representations in Vision-Language Models

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao

arXiv:2101.00529v2Jan 2, 2021cs.CVcs.AIcs.CLcs.LG

TL;DR

The paper addresses the limited attention to object-centric visual representations in vision-language pre-training. It develops a stronger object detection model, integrates its features into VL pre-training and downstream tasks, and reports improved performance across seven VL tasks.

Problem
Existing VLP research mainly improves cross-modal fusion while leaving object-centric visual feature improvement relatively untouched, despite visual features being central to VL models.
Method
The paper develops a larger, VL-oriented object detection model trained on multiple datasets, then feeds its visual features into OSCAR+ for pre-training and fine-tuning across downstream VL tasks.
Results
The new object-centric visual representations substantially improve performance across all seven evaluated VL tasks and uplift state-of-the-art results across multiple public benchmarks.
Takeaways & Limitations
Visual representations matter significantly in VLP, with improved vision pre-training accounting for 95% of the reported VQA gain from 72.46 to 74.90.
Takeaways & Limitations
The reported R101-C4 reference mAP differs from the original report because of differences in the object-detection evaluation pipeline.

Abstract

from arXiv · show

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

1 Introduction

The paper argues that vision-language research has underemphasized object-centric visual representations and introduces a stronger detection model for VL tasks. Combined with OSCAR+ pre-training, the resulting features improve performance across seven VL tasks and benchmarks.

VLP commonly improves cross-modal fusion while treating the object detection model as fixed, motivating a direct study of visual representations.
The new X152-C4 detector is larger, designed for VL, and trained on multiple public detection datasets, producing 1848 object and 524 attribute categories.
The model produces richer concepts, attributes, and semantically meaningful regions than typical OpenImages-based detections, including concepts absent from the comparison model.
The authors feed fixed representations from the new detector into Transformer-based OSCAR+ pre-training on 8.85 million text-image pairs.
The resulting features significantly improve understanding and generation tasks, including VQA, GQA, retrieval, captioning, and NoCaps, establishing new state-of-the-art results across seven tasks.
The study also ablates object-category diversity, attribute training, data scale, model size, and architecture to analyze sources of improvement.

2 Improving Vision (V) in Vision Language (VL)

The paper improves the vision module in VL systems by developing a larger object detector trained on diverse datasets and adapted to produce object and attribute representations. These representations are then supplied as semantic tags and region features to downstream VL models.

VL systems combine an image understanding module, Vision, with a cross-modal module, VL, whose outputs vary by task such as answers, matching scores, or captions.
Recent VLP work largely treats Vision as a black box, despite progress in object detection datasets, architectures, and training methods.
The new detector enriches object and attribute categories, increases model size, and uses a much larger object-detection dataset to improve VL representations.
The training recipe pre-trains on four public datasets, then fine-tunes on Visual Genome with an attribute branch so the detector recognizes both objects and attributes.
The merged vocabulary contains 1848 object classes, comprising 1594 retained Visual Genome classes and 254 additional unmapped classes.
The authors choose C4 over FPN for VL region features based on experiments showing FPN does not provide more effective VL features, while X152-C4 can also detect objects better than X152-FPN.
For VL feature extraction, the detector outputs semantic object names q and region features v, with each region feature combining a 2048-dimensional representation and six-dimensional position encoding.

3 OSCAR+ Pre-training

OSCAR+ pre-trains joint image-text representations with image tags as anchors, combining masked-token and novel 3-way contrastive objectives across heterogeneous vision-language data.

OSCAR+ learns joint image-text representations using image tags as anchors for image-text alignment.
The pre-training corpus combines image captioning, visual question answering, and image tagging datasets with captions, questions, answers, and tags.
The OSCAR+ pre-training loss contains masked token loss and a novel 3-way contrastive loss.
The 3-way contrastive loss classifies matched triplets, caption- or question-polluted triplets, and tag- or answer-polluted triplets.
OSCAR+B and OSCAR+L use BERT base and large initializations, respectively, with projected region features matched to BERT's embedding size.

4 Adapting to VL Tasks

The pre-trained OSCAR+ model is adapted to seven VL understanding and generation tasks using task-specific input constructions, objectives, and prediction heads.

The study adapts OSCAR+ to five understanding tasks and two generation tasks with task-specific fine-tuning strategies.
Understanding tasks: VQA and GQA concatenate questions, object tags, and region features before classifying the [CLS] representation over candidate answers.
Generation tasks: Captioning and NoCaps use masked-token seq2seq fine-tuning with caption tokens attending causally to prior tokens and fully to image regions and tags.
Retrieval tasks: Image-text retrieval is formulated as binary classification of matched versus randomly mismatched image-text pairs using the [CLS] representation.
Understanding tasks: NLVR2 encodes each image-text pair separately, concatenates the two [CLS] outputs, and predicts whether the description is true of the image pair.

5 Experiments & Analysis

Experiments show that VinVL’s visual representations improve VL performance across seven tasks and that gains arise substantially from visual pre-training, richer data, and larger models.

5.1 Main Results: VINVL outperforms previous state-of-the-art models on all seven VL tasks, often by significantly large margins.The only reported exception is B@4 on image captioning.
5.2 Ablation Analysis: OSCAR+B with VinVL features improves VQA accuracy from 72.38 to 74.90 over the R101-C4 baseline.The reported gain is decomposed as 72.38 → 72.46 from OSCAR+ pre-training and 72.46 → 74.90 from improved visual features.
5.2 Ablation Analysis: Vision and vision-language pre-training provide additive VQA gains of 2.82 and 3.94 points, respectively, relative to the no-VLP R101-C4 baseline.The combined improvement is approximately 2.82 + 3.94 points.
5.2 Ablation Analysis: Models trained on four merged object-detection datasets outperform VG-only models, with larger improvements as model size increases.The study identifies model size and training-data size as compound contributors to the improvement from R101-C4 to VinVL.
5.2 Ablation Analysis: FPN underperforms C4 on some VL settings, while the gap diminishes when both architectures are trained on the merged dataset.The paper also reports that OD pre-training benefits object-detection tasks, though VG mAP is reduced by sparse, imbalanced, and missing annotations.
5.2 Ablation Analysis: Richer object vocabularies generally improve VQA, following VG-obj < ImageNet < VG w/o attr.The ablation compares vocabularies ranging from 317 shared object classes to 1,594 object classes, with and without 524 attributes.

6 Conclusion

The paper presents a new object-detection pre-training recipe for VL tasks and validates it through VL pre-training and fine-tuning across seven tasks. The resulting visual features substantially improve state-of-the-art performance, with gains attributed mainly to broader categories, attributes, data, model size, and architecture.

6 Conclusion: The new object-detection model is larger, VL-oriented, and trained on more data than the widely used baseline, producing richer visual features.The paper connects these design choices to broader coverage of visual objects and concepts.
6 Conclusion: The visual features are evaluated by feeding them into a VL fusion model, pre-training on paired text-image data, and fine-tuning across seven VL tasks.The evaluation spans multiple public benchmarks and includes both understanding and generation tasks.
6 Conclusion: The new object-detection model substantially improves state-of-the-art results across all seven VL tasks and multiple public benchmarks.The conclusion reports a cross-task improvement rather than a single-task gain.
6 Conclusion: Ablations attribute the improvement mainly to object-category diversity, visual-attribute training, training-data scale, model size, and model architecture.These factors are identified as the principal design choices behind the observed gains.

A Qualitative study of three pre-trained vision models

The qualitative comparison applies three pre-trained vision models to the same image and contrasts their detected objects and attributes. The proposed X152-C4 model produces broader, more coherent detections than the compared baselines, while the R101-C4 baseline includes obvious errors.

A Qualitative study of three pre-trained vision models: The qualitative study applies three pre-trained object-detection models to one image and lists their detection results for comparison.The image is the COCO test2015 image 000000028839.
A Qualitative study of three pre-trained vision models: X152-FPN trained on Open Images detects repeated surfboard, person, body-part, and clothing concepts in the test image.The listed detections include surfboards, a man, human legs, swimwear, shorts, a boy, and a human arm.
A Qualitative study of three pre-trained vision models: The R101-C4 model trained on Visual Genome produces several obviously wrong detections alongside objects, attributes, and scene descriptions.The reported errors are explicitly marked in red in the corresponding figure.
A Qualitative study of three pre-trained vision models: The proposed X152-C4 model, pre-trained on four datasets and fine-tuned on Visual Genome, produces repetitive detections but no obvious wrong detections.Its outputs cover objects, attributes, people, body parts, surfboards, water, sky, mountains, and clothing.

B.1 Pre-training Corpus

OSCAR+ pre-training combines image-text, visual-question-answering, and image-tagging data, using object tags and image features alongside text. Its masked-token and contrastive objectives train cross-modal representations, while tagging data enables corpus expansion through self-training.

B.1 Pre-training Corpus: The pre-training corpus combines captioning, visual-question-answering, and image-tagging datasets with text, tags, and image features.Image-tagging examples use machine-generated captions and human-annotated tags.
B.1 Pre-training Corpus: OSCAR+ scales pre-training through three corpus sizes—Small, Medium, and Large—and incorporates image-tagging data for self-training.OpenImages and YFCC are identified as large-scale tagging sources for corpus expansion.
B.2 OSCAR+ pre-training objectives: Masked Token Loss randomly masks 15% of caption and object-tag tokens and predicts them from surrounding tokens and image features.The objective applies to the combined token sequence of text and tags.
B.2 OSCAR+ pre-training objectives: The pre-training objectives include a 3-way contrastive loss designed to mimic text-image retrieval and visual question answering simultaneously.The passage identifies this as a distinct OSCAR+ objective.

B.3 Ablation of the two new techniques

The ablations examine self-training with tagging data, the 3-way contrastive loss, corpus scale, and visual-feature choice. They show that tagging data improves performance, the contrastive loss especially benefits retrieval, and VinVL visual features matter substantially for downstream VL performance.

Effect of self-training: Adding 1.7M OpenImages tagging examples to OSCAR+ pre-training significantly improves VQA performance.The comparison is between the Small and Medium VinVL pre-training settings.
Effect of self-training: The new image features substantially affect VL pre-training and downstream performance compared with features from the classical object-detection model.Figure 4 includes OSCAR and OSCAR+ baselines using the earlier visual features.
Effect of the new 3-way contrastive loss: The 3-way contrastive loss leaves VQA performance unchanged relative to OSCAR pre-training but significantly improves text-image retrieval.The reported comparison is summarized from Table 3.
Overall improvement from OSCAR to OSCAR+: OSCAR+ gains with the classical visual features are minor because the added tagging data are small relative to the original corpus and the contrastive loss helps retrieval more than VQA.The paper expects larger gains when OSCAR+ scales to much larger tagging datasets.

C Downstream Tasks Fine-tuning

The paper fine-tunes OSCAR+ across understanding, generation, retrieval, and reasoning tasks using task-specific objectives and evaluation procedures. It also examines vocabulary, feature extraction, and visual-model choices for VQA.

Downstream task procedures: OSCAR+ is fine-tuned for VQA, GQA, NLVR2, COCO retrieval, COCO captioning, and NoCaps using task-specific input and prediction procedures.The downstream tasks span both VL understanding and generation, with retrieval formulated as aligned-versus-unaligned image-text classification.
VQA: VQA concatenates questions, object tags, and region features before using the [CLS] representation for soft-target answer classification.The model is optimized with cross-entropy for multi-label answer prediction.
Generation: Image captioning fine-tunes OSCAR+ with a seq2seq objective that predicts masked caption tokens from image regions, captions, and object tags.Inference autoregressively replaces [MASK] with selected tokens until [SEP].
Generation: NoCaps training uses COCO only without pre-training initialization or additional image-text pairs, while applying an Open Images-trained detector to generate tags.The task evaluates describing novel objects from Open Images under its restriction guidelines.
VQA analyses: Figure 5 compares vocabulary effects by training setup and feature-extraction method, averaging rows and columns to summarize model and extraction quality.Figure 6 instead compares object and attribute vocabularies and extraction methods across training checkpoints using VQA accuracy.

D.1 Disentangling the effects of region proposals and model weights on R50-C4

The ablation separates region proposals from detector weights and finds that model weights contribute more to VQA gains than the source of region boxes. Richer visual semantics are more useful than COCO ground-truth regions for VL.

Feature extraction: Grid features perform worse than Visual Genome-proposed region features and require three times the training speed of region-feature models.Closing the gap with grid features would also increase cross-modal hardware memory and computation.
Feature extraction: Mean evaluation curves across training trajectories provide more robust evidence for the vocabulary and feature-extraction rankings than a single final number.The curves further confirm the conclusions from the earlier analysis.
Region proposals versus model weights: Pre-trained model weights contribute more to VQA improvement than region proposals when region boxes and feature extractors are alternated.Using R101-C4 boxes with X152-C4 features improves more than using X152-C4 boxes with R101-C4 features.
Region proposals versus model weights: COCO ground-truth object regions perform much worse than proposals from Visual Genome-trained models on VQA.The comparison includes 80-class object regions and 171-class object-and-stuff regions.
Region proposals versus model weights: VL object detection requires richer visual semantics than typical object-detection tasks because visual representations must align with language semantics.This conclusion is supported by the gap between COCO ground-truth regions and Visual Genome-trained proposals.

E.1 Two reasons why FPN performs worse than C4 on VL tasks.

FPN does not provide better VL region features than C4 because pre-training and box-head architecture matter more than multi-scale pooling. ImageNet initialization and the C4 convolutional head are beneficial.

Pre-training and architecture: C4 and FPN achieve the same VQA performance after vision pre-training on four datasets, 68.3 versus 68.2.With VG training and a randomly initialized FPN box head, C4 performance also matches FPN, implicating pre-training differences.
Pre-training and architecture: C4’s convolutional head has a better inductive bias for visual encoding than FPN’s MLP head.Randomly initialized R50-C4 features outperform randomly initialized R50-FPN features, while random C4 features nearly match ImageNet-pre-trained features.
Pooling methods: FPN pooling is not the root cause of its worse VQA performance than C4.The experiments compare adaptive, max, average, and concatenated pooling across FPN vision models and pre-training conditions.
Pooling methods: FPN extracts region features from multiple scales according to region area, whereas C4 extracts them from a single scale.This multi-scale design creates potential feature inhomogeneity, which the pooling experiments test directly.

E.3 Large-scale object-detection pre-training of C4 and FPN models

Large-scale pre-training comparisons favor the R152-C4 model for downstream VL tasks, despite some R50-FPN advantages on individual detection validation sets. Inference profiling shows that the vision backbone dominates runtime.

Detection pre-training comparison: R152-FPN is consistently worse than R152-C4 on mAP 50 across COCO with stuff, Objects365, OpenImages, and Visual Genome.For R50 models, FPN is slightly better on COCO and Objects365 but slightly worse on Visual Genome.
Experimental caveat: One experimental run failed, so its standard deviation is unavailable.This limits variability reporting for that experiment.
Detection pre-training comparison: The paper selects R152-C4 for downstream vision-language tasks after comparing C4 and FPN models trained on merged object-detection datasets.The selection follows the four-dataset validation comparison.
X152 pre-training ablation: The X152 ablation compares grid and region features across ImageNet-5k, merged four-dataset, Visual Genome-with-attributes, and sequentially trained vision models.All region features use boxes proposed by the best X152-C4 model.
Inference efficiency: Grid-feature models are faster than region-feature models, while vision models dominate end-to-end inference time, especially at larger scales.On CPU, backbone and C4-head computation nearly exhaust inference time, making NMS savings negligible.