Source-linked AI summary
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
TL;DR
Existing vision-language pre-training must learn image-text alignments from concatenated features despite weak supervision and ambiguous visual regions. Oscar introduces detected object tags as anchor points, pre-trains on 6.5 million image-text pairs, and reports new state-of-the-art results across multiple vision-language benchmarks. The paper also notes a fine-tuning choice that is not necessarily optimal for NLVR2.
Problem
Existing VLP methods learn alignments from concatenated image-region and text features without explicit alignment information, while visual regions can be over-sampled, noisy, and ambiguous.
Method
Oscar represents each training example as words, detected object tags, and image-region features, using object tags as anchor points during pre-training.
Results
Oscar was pre-trained on 6.5 million image-text pairs and achieved new state-of-the-art results on multiple established vision-language understanding and generation tasks.
Takeaways & Limitations
Object tags improve cross-modal representation learning as alignment anchors, with experiments analyzing their effectiveness for downstream vision-language tasks.
Takeaways & Limitations
The paper states that its fine-tuning choice for NLVR2 is not necessarily the best choice.
Abstract
from arXiv · showhide
Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.
1 Introduction
Existing vision-language pre-training learns image-text alignments from concatenated region and text features without explicit alignment information. Oscar introduces detected object tags as anchor points, pre-trains on 6.5 million pairs, and reports new state-of-the-art results across multiple vision-language tasks.
- Existing methods concatenate image-region and text features, leaving semantic alignment weakly supervised.Visual regions are also often over-sampled, noisy, and ambiguous.
- Oscar uses detected object tags as anchor points to ease image-text semantic alignment learning.The method defines training inputs as triples containing words, object tags, and image-region features.
- 49.7% of MS COCO image-text pairs share at least 1 object, while 22.2% share at least 2 and 12.9% share at least 3.
- Oscar is pre-trained on 6.5 million image-text pairs and fine-tuned on seven understanding and generation tasks.
- The paper reports new state-of-the-art results on multiple vision-language benchmarks and analyzes the effectiveness of object tags as alignment anchors.
2 Background
Vision-language pre-training represents image regions and words with separate embeddings and uses self-attention to learn cross-modal contextual representations. The background identifies ambiguity from overlapping regions and weak grounding between visual content and paired text as central challenges.
- Oscar’s dictionary view uses object tags as anchors to connect image regions with language-model word embeddings in semantic space.
- Existing VLP models use visual region features and word embeddings as inputs to self-attention Transformers that learn image-text alignments.
- Over-sampled image regions can overlap, making visual embeddings ambiguous; dog and couch are an illustrative example.
- Image-text alignment is weakly supervised because explicit correspondences between image regions or objects and text words or phrases are unavailable.
3 Oscar Pre-training
Oscar represents each image-text pair as a Word-Tag-Image triple and uses object tags as anchor points for cross-modal alignment. It trains with complementary dictionary-view masked token and modality-view contrastive objectives.
- Input Representation: Oscar adds object-tag embeddings q to word embeddings w and region features v, forming a Word-Tag-Image triple that anchors image-text alignment.The tags are detected from images and are often also represented in paired text, making tag-text alignment easier to identify.
- Input Representation: Object tags connect the linguistic semantic space shared by tags and words with the visual semantic space of image-region features.This two-view formulation distinguishes modality representations from the semantic spaces defined by different dictionaries.
- Dictionary View: Masked Token Loss randomly masks 15% of discrete word and tag tokens and predicts them from surrounding tokens together with all image features.The objective uses surrounding tokens and visual features to ground word and tag representations in image context.
- Modality View: Contrastive Loss groups object tags and region features as the image modality, then distinguishes matched text-image pairs from polluted pairs created by replacing tags.The classifier uses the fused [CLS] representation to predict whether an image representation is genuine or polluted.
- Pre-Training Objective: Oscar jointly optimizes masked token and contrastive losses, retaining a simpler objective while reporting superior experimental performance.The paper motivates the joint loss as a clear way to study dictionary and modality views separately.
- Implementation: The pre-training corpus contains 6.5 million text-tag-image triples built from existing vision-and-language datasets and 4.1 million unique images.The model variants OscarB and OscarL are initialized from BERT base and large, respectively.
4 Adapting to V+L Tasks
Oscar is adapted to seven downstream vision-and-language tasks through task-specific fine-tuning strategies. These include understanding tasks such as retrieval, VQA, GQA, and NLVR2, alongside image-captioning generation tasks including NoCaps.
- Task Adaptation: Oscar is fine-tuned on seven downstream tasks, comprising five vision-and-language understanding tasks and two generation tasks.The adaptation requires task-specific strategies because the downstream tasks pose different challenges.
- Understanding Tasks: Image-text retrieval trains Oscar as a binary classifier to distinguish aligned image-caption pairs from pairs formed with a different image or caption.The final [CLS] representation is used for predicting pair alignment.
- Generation Tasks: Image captioning uses a sequence-to-sequence objective over image regions, captions, and object tags, masking 15% of caption tokens for prediction.At inference, Oscar generates tokens autoregressively until producing [STOP], using beam search.
- Generation Tasks: NoCaps evaluates novel-object captioning using Open Images images; Oscar is trained on COCO with predicted Visual Genome and Open Images labels and without pre-training initialization.The task tests descriptions of objects absent from the training corpus.
- Understanding Tasks: VQA concatenates a question with object tags and region features, then uses the [CLS] output for multi-label answer classification with soft target scores.The soft targets reflect answer relevance to human responses.
- Understanding Tasks: GQA evaluates question-answering reasoning over 1,852 candidate answers using OscarB variants fine-tuned on different dataset splits.One variant is further fine-tuned from the unbalanced all-split to the balanced split.
- Understanding Tasks: NLVR2 processes each image-sentence pair separately and concatenates the two [CLS] outputs before binary classification of the statement.The task determines whether a natural-language statement is true about an image pair.
5 Experimental Results & Analysis
Oscar achieves strong performance across vision-language benchmarks while using fewer pre-training pairs than some large-model baselines. Qualitative visualizations and ablations support object tags as useful anchors for cross-modal alignment and downstream learning.
- 5.1 Performance Comparison with SoTA: 6.5 million pre-training pairs support Oscar’s comparison with models using 9.6 million pairs for UNITER and 9.18 million for LXMERT.The paper frames this as evidence of parameter and data efficiency relative to those baselines.
- 5.1 Performance Comparison with SoTA: Oscar outperforms existing VLP methods on all seven reported tasks and achieves new state-of-the-art results on six.The comparison covers both vision-language understanding and generation tasks.
- 5.1 Performance Comparison with SoTA: Oscar improves captioning over the comparable VLP method by more than 2 BLEU@4 points and 10 CIDEr points.The reported captioning result uses additional self-critical sequence training for sequence-level learning.
- 5.2 Qualitative Studies: Object tags bring same-class visual and textual representations closer while keeping related object classes distinguishable.The t-SNE analysis contrasts Oscar with a baseline without object tags and reports fewer mixtures among related classes.
- 5.2 Qualitative Studies: Oscar generates more detailed image captions than the baseline, with detected object tags guiding the text-generation process.The qualitative comparison attributes the difference to accurate and diverse tags detected by Faster R-CNN.
- 5.3 Ablation Analysis: Ground-truth, predicted, and no-tag settings isolate the contribution of object tags across VQA, image retrieval, and image captioning.The ablation uses the ground-truth tags as a performance upper bound and the same BERT base model across settings.
- 5.3 Ablation Analysis: Tags provide minor gains as image features but are more promising as anchor points, while region features remain more informative than tags for image representation.This conclusion comes from varying attention interactions on the COCO 1K retrieval test set.
- 5.3 Ablation Analysis: Using object tags improves the no-tag baseline with either Visual Genome or Open Images pre-training, while Visual Genome tags perform slightly better.The paper hypothesizes that Visual Genome provides a more diverse object set despite Open Images having higher detector precision.
6 Related Work
Prior vision-language pre-training methods learn cross-modal representations by combining visual regions and language, while Oscar uses object tags to align object regions in linguistic semantic space.
- Vision-language pre-training learns generic representations for tasks including visual question answering, image-text retrieval, and image captioning.
- Existing methods concatenate visual region features with language embeddings and rely heavily on Transformer self-attention to contextualize both modalities.
- Object tags provide semantic labels for aligning object-region features in the pre-trained linguistic semantic space.
- Earlier multimodal embedding methods project words and image regions into a shared space for tasks such as annotation, segmentation, captioning, and retrieval.
7 Conclusion
Oscar uses object tags as anchor points to align image and language modalities in a shared semantic space. Pre-training on 6.5 million text-image pairs yields new state-of-the-art results across six established vision-language tasks.
- Oscar uses object tags as anchor points to align image and language modalities in a shared semantic space.
- 6.5 million text-image pairs are used to pre-train Oscar models before downstream evaluation.
- Oscar achieves new state-of-the-art results on six established vision-language understanding and generation tasks.
A Fine-tuning Settings
The fine-tuning settings specify task-specific data splits, optimization schedules, and decoding procedures for retrieval, captioning, NoCaps, VQA, GQA, and NLVR2.
- Image-Text Retrieval: Image-text retrieval uses the COCO Karpathy split with 113,287 training images, 5,000 validation images, and 5,000 test images.
- Image Captioning: Image captioning fine-tunes Oscar directly on COCO without additional Conceptual Captions pre-training to test generation-task generalization.
- NoCaps: NoCaps uses object tags generated by a detector trained on Open Images and applies constrained beam search during inference.
- VQA: VQA fine-tuning runs for 25 epochs, using learning rates of 5e−5 for OscarB and 3e−5 for OscarL.
- GQA: GQA follows a procedure similar to VQA, with OscarB trained for 5 epochs using a learning rate of 5e−5 and batch size 128.
- NLVR2: NLVR2 fine-tunes Oscar models for 20 epochs with model-specific learning-rate sets and batch sizes of 72 for OscarB and 48 for OscarL.
B Pre-training Corpus
The pre-training corpus is documented through a table presenting its image and text statistics.
- Table 5 reports statistics for the images and text in the pre-training corpus.
C More Results
The section presents enlarged t-SNE visualizations for Oscar and a baseline without tags.
- Enlarged t-SNE visualizations compare Oscar with a baseline that uses no tags.The Oscar and baseline visualizations are shown in Fig. 7 and Fig. 8, respectively.