PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai

arXiv:1908.08530v4cs.CVcs.CLcs.LG

TL;DR

Visual-linguistic tasks lacked generic representations that could align image and language information across downstream tasks. VL-BERT extends Transformer attention to words and image regions, pre-trains with masked visual-linguistic and text-only objectives, and achieves state-of-the-art results across several tasks, including first place among single models on VCR.

  • Problem

    Visual-linguistic tasks lacked generic pre-trained representations that aggregate and align visual and linguistic information across tasks.

  • Method

    VL-BERT uses a multimodal Transformer over words and image RoIs, pre-trained by masking words or RoIs on visual-linguistic data and applying masked language modeling to text-only data.

  • Results

    VL-BERT achieves state-of-the-art performance on visual commonsense reasoning, visual question answering, and referring expression comprehension, including first place among single models on VCR.

  • Takeaways & Limitations

    The pre-training procedure better aligns visual-linguistic clues and benefits downstream visual-linguistic tasks.

Abstract

from arXiv · show

We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at \url{https://github.com/jackroos/VL-BERT}.

1 INTRODUCTION

VL-BERT addresses the lack of generic pre-trained representations for visual-linguistic tasks by extending Transformer attention to jointly process words and image regions. It is pre-trained with visual-linguistic and text-only objectives, and achieves strong downstream results.

  • Motivation: Visual-linguistic tasks lacked generic pre-trained feature representations, leaving models to combine separately pre-trained vision and language networks in task-specific ways.Such task-specific training may overfit when target-task data is scarce.
  • VL-BERT: VL-BERT uses a multimodal Transformer whose elements are sentence words, image regions-of-interest, or special format-disambiguating elements.Each element aggregates information from others using content, position, and category compatibility.
  • VL-BERT: Stacked multimodal Transformer layers produce representations that aggregate and align visual-linguistic clues, with task-specific branches added for downstream tasks.
  • Pre-training: VL-BERT is pre-trained on visual-linguistic data by masking words or RoIs, alongside text-only masked language modeling for long and complex sentences.
  • Results: VL-BERT achieves state-of-the-art performance across visual commonsense reasoning, visual question answering, and referring expression comprehension.It achieves first place among single models on the visual commonsense reasoning leaderboard.

2 RELATED WORK

Prior work developed generic representations separately in computer vision and NLP, while visual-linguistic models initially relied on task-specific combinations. Video pre-training and concurrent multimodal methods motivated image-based generic visual-linguistic pre-training, which VL-BERT differentiates through joint corpora and visual tuning.

  • Computer Vision: ImageNet-pre-trained CNNs became generic feature representations for numerous computer vision tasks.
  • Natural Language Processing: Transformer-based pre-training methods, including BERT, established generic representations for diverse NLP tasks.
  • Visual-Linguistic Tasks: Earlier visual-linguistic models combined off-the-shelf vision and NLP features ad hoc and trained only on each task dataset.
  • Visual-Linguistic Pre-training: VideoBERT and CBT pre-trained on video clips, but their video-oriented setting left image-based visual-linguistic tasks to be studied.VideoBERT clustered clips into visual words, losing visual content and hindering visual-network updates.
  • Concurrent Work: Concurrent methods explored two-stream cross-modal Transformers and unified single-stream architectures for generic visual-linguistic representations.Their emergence indicated the importance of pre-trainable generic representations for visual-linguistic tasks.
  • VL-BERT Differences: VL-BERT jointly pre-trains on visual-linguistic and text-only datasets, omits sentence-image relationship prediction, and updates Fast R-CNN visual features.The authors report that joint pre-training improves generalization on long and complex sentences.

3 VL-BERT

VL-BERT extends BERT's bidirectional Transformer to jointly model linguistic words and visual regions through multimodal attention. Its architecture combines visual appearance and geometry features with linguistic, segment, and positional embeddings, and supports cross-modal pre-training.

  • Architecture: VL-BERT processes words, image RoIs, and special elements with a multi-layer bidirectional Transformer encoder.Attention models dependencies among all input elements rather than sentence words alone.
  • Architecture: Each input element combines token, visual feature, segment, and sequence position embeddings, with visual feature embeddings newly introduced for visual clues.The input begins with [CLS], separates modalities and sentences with [SEP], and ends with [END].
  • Visual Feature Embedding: Visual feature embeddings concatenate Fast R-CNN appearance features with geometry embeddings before a fully connected projection.RoI geometry uses normalized corner coordinates and sine-cosine functions of different wavelengths.
  • Input Formatting: Segment embeddings distinguish words from two sentences and RoIs, enabling formats such as <Question, Answer, Image> and <Caption, Image>.The representation remains generic across task-specific input formats because sequence order is separately encoded by positional embeddings.
  • Pre-training Tasks: Masked RoI classification predicts a masked region's category from linguistic and other contextual clues while preventing visual leakage from that region.The masked RoI pixels are zeroed before Fast R-CNN processing, and detector-predicted categories provide training labels.
  • Pre-training Tasks: Visual-linguistic pre-training improves detailed visual-language alignment, while text-only pre-training supports understanding of long and complex sentences.These capabilities target downstream tasks requiring region-language alignment or complex textual understanding.

4 EXPERIMENT

Experiments evaluate VL-BERT across VCR, VQA, and referring expression comprehension, using task-specific input and prediction adaptations. Results show improvements over non-pre-trained and task-specific or concurrent methods, while ablations identify which pre-training choices help or hurt.

  • Pre-training: VL-BERT is jointly pre-trained on Conceptual Captions, BooksCorpus, and English Wikipedia before downstream fine-tuning.Visual content uses RoIs extracted by Faster R-CNN; the model can then be fine-tuned with simple task-specific changes.
  • Visual Commonsense Reasoning: In VCR, VL-BERT uses Question, Answer, and Image inputs, with [CLS] classification and joint answer-correctness and RoI-classification losses.The protocol follows R2C, but VL-BERT omits ad-hoc Grounding, Contextualization, and Reasoning modules.
  • Visual Commonsense Reasoning: 1.0% improvement in the final Q →AR task comes from pre-training, while VL-BERT also outperforms R2C by large margins and reaches state-of-the-art VCR performance.The comparison uses the same input, output, and experimental protocol as R2C, while training the generic representation end-to-end.
  • Visual Question Answering: In VQA, a [MASK] answer element and detector-generated RoIs support multi-class answer prediction from Question, Answer, and Image inputs.VL-BERT improves performance by 1.6% with pre-training and surpasses BUTD by over 5% in accuracy.
  • Referring Expression Comprehension: For RefCOCO+, VL-BERT classifies candidate RoIs for an appearance-based referring expression and selects the highest-scoring RoI at inference.The model significantly improves performance over MAttNet without task-specific architecture designs and performs comparably to ViLBERT.
  • Ablation Study: Across all three downstream tasks, pre-training improves performance; masked language modeling with visual clues helps broadly, while sentence-image relationship prediction hurts all three.Masked RoI classification with linguistic clues further improves RefCOCO+ but does not improve VCR or VQA; adding text-only data improves performance relative to the corresponding setting.

5 CONCLUSION

VL-BERT is a pre-trainable generic visual-linguistic representation built on Transformer architecture and trained with visual-linguistic and text-only data. The authors report improved visual-linguistic alignment and benefits across downstream tasks.

  • VL-BERT uses a Transformer backbone instead of ad-hoc task-specific modules for visual-linguistic representation learning.
  • Pre-training combines the massive-scale Conceptual Captions dataset with a text-only corpus.
  • The authors report that pre-training better aligns visual-linguistic clues and benefits downstream tasks.
  • The paper identifies image caption generation as a future downstream task for improved pre-training methods.

A.1 COMPARISON AMONG VL-BERT AND OTHER WORKS

The appendix compares VL-BERT with concurrent efforts to pre-train generic visual-linguistic representations. The cited comparison specifically notes the datasets used to pre-train LXMERT.

  • Table 5 compares VL-BERT with other concurrent works seeking pre-trainable generic visual-linguistic representations.
  • LXMERT is pre-trained on COCO Caption, VG Caption, VG QA, VQA, and GQA datasets.

A.2 DETAILED EXPERIMENT SETTINGS

The experiments use large-scale GPU training for pre-training and task-specific fine-tuning. Figure 3 visualizes normalized cross-modal attention from text tokens to image regions.

  • Pre-training runs for 250k iterations on 16 Tesla V100 GPUs with batches split between Conceptual Captions image-caption pairs and text sequences.Each mini-batch contains 128 visual-linguistic samples and 128 text-only samples.
  • VCR: VCR fine-tuning uses 20 epochs, batches of 256 question-answer-image triplets, and learning-rate drops at epochs 14 and 18.
  • VQA: VQA fine-tuning uses 20 epochs, batches of 256 question-answer-image triplets, warmup over 2,000 steps, and linear learning-rate decay.
  • RefCOCO+: RefCOCO+ fine-tuning uses 20 epochs, batches of 256 query-image pairs, warmup over 500 steps, and linear learning-rate decay.
  • Attention visualization: Figure 3 encodes text-to-RoI attention magnitude through line intensity, rescaled from 0 to 1 across heads within each layer.

A.3 VISUALIZATION OF ATTENTION MAPS IN VL-BERT

The authors visualize pre-trained VL-BERT attention on COCO images before downstream fine-tuning. Different heads exhibit distinct patterns, including focused or uniform text-to-region attention.

  • Attention maps are visualized for pre-trained VL-BERT without fine-tuning on downstream tasks.
  • On the COCO val2017 set, different attention heads show different visual-linguistic attention patterns.
  • Some heads make text tokens attend more strongly to associated image RoIs.
  • Other heads distribute text-token attention uniformly across image RoIs.
  • The visualizations demonstrate VL-BERT’s ability to aggregate and align visual-linguistic contents.
Loading 1908.08530v4…