PaperlayerView exact revision on arXiv

Source-linked AI summary

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee

arXiv:1908.02265v1cs.CVcs.CL

TL;DR

Vision-and-language tasks lack a unified foundation for learning visual grounding, instead typically learning it separately during task training. ViLBERT pretrains a two-stream, co-attentional model and achieves state-of-the-art performance across four established tasks, with improvements of 2 to 10 percentage points over task-specific baselines.

  • Problem

    Vision-and-language approaches lack a unified foundation for aligning language and visual stimuli, often learning grounding separately during task training.

    Source: §1 · p. 1

  • Method

    ViLBERT pretrains separate vision and language streams that exchange information through co-attentional transformer layers on paired image-text data.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    2 to 10 percentage points improvements across four tasks yield state-of-the-art performance against task-specific baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Visual grounding can serve as a transferable capability across multiple established vision-and-language tasks from a common pretrained architecture.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation omits visually grounded dialog, embodied tasks, and text-generation tasks, leaving extensions to long multimodal sequences and output decoding unresolved.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.

1 Introduction

ViLBERT addresses visual grounding by extending BERT into a task-agnostic joint model for paired vision-and-language data. It uses separate visual and linguistic streams with co-attentional transformer layers, pretrains on Conceptual Captions through two proxy tasks, and transfers to four vision-and-language tasks.

Motivation

  • Vision-and-language systems must align natural language with visual stimuli, a capability referred to as visual grounding.Source: §1 · p. 1

Approach

  • ViLBERT extends BERT to jointly reason about text and images while learning task-agnostic visual grounding from paired visiolinguistic data.Source: §1 · p. 2

Architecture

  • Separate visual and linguistic processing streams communicate through co-attentional transformer layers, accommodating differing modality-specific processing needs.Source: §1 · p. 2, Figure 1 · p. 2

Pretraining

  • ViLBERT is pretrained on Conceptual Captions with two proxy tasks: predicting masked-word and masked-region semantics, and determining whether an image and text segment correspond.Source: §1 · pp. 1–2

Transfer

  • The pretrained model is applied to visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval.Source: §1 · p. 2

2 Approach

ViLBERT extends BERT into a two-stream model that separately processes image regions and text while exchanging information through co-attentional transformer layers. It is pretrained with masked multi-modal modelling and multi-modal alignment prediction to learn joint visual-linguistic representations.

Model architecture

  • ViLBERT uses two parallel BERT-style streams for image regions and text, combining transformer blocks with co-attentional layers for cross-modal information exchange.The visual stream receives region features, while the text stream receives token sequences and produces modality-specific final representations.Source: §2.2 · p. 3
  • Co-attention passes each modality’s keys and values into the other modality’s multi-headed attention, producing representations conditioned on the other modality.This performs image-conditioned language attention in the visual stream and language-conditioned image attention in the linguistic stream.Source: Figure 2 · p. 3, §2.2 · p. 4

Input representations

  • Image inputs are represented as object-detection region features augmented with spatial encodings based on normalized box coordinates and image-area coverage.Because image regions lack a natural ordering, spatial location supplies the positional information.Source: §2.2 · p. 4

Pretraining objectives

  • ViLBERT pretrains on paired image-text data using masked multi-modal modelling and multi-modal alignment prediction.The first reconstructs masked words or image regions from the remaining inputs, while the second predicts whether an image and text are aligned.Source: §2.2 · p. 4, Figure 3 · p. 4

3 Experimental Settings

ViLBERT is pretrained on millions of automatically collected image-caption pairs using specified vision-language training tasks, then fine-tuned end-to-end for established vision-and-language tasks with minimal task-specific modifications. The experiments also include a zero-shot retrieval setting that applies pretrained multimodal alignment without fine-tuning.

Pretraining Data

  • 3.1 million image-caption pairs from Conceptual Captions are used to train ViLBERT after broken links reduce the original 3.3 million-pair collection.The dataset is automatically scraped from alt-text-enabled web images and contains noisy, sometimes non-human-like captions.Source: §3.1 · p. 5

Implementation Details

  • The linguistic stream is initialized with BERTBASE, while visual features come from Faster R-CNN with a ResNet-101 backbone pretrained on Visual Genome.BERTBASE has 12 transformer layers, hidden size 762, and 12 attention heads; visual streams use hidden size 1024 and 8 attention heads.Source: §3.1 · p. 5
  • Training uses 8 TitanX GPUs, batch size 512, 10 epochs, Adam with an initial learning rate of 1e-4, linear decay with warmup, and equally weighted task losses.The model retains 10 to 36 high-scoring detected regions per image.Source: §3.1 · p. 5

Transfer Tasks

  • ViLBERT transfers to four established vision-and-language tasks and one diagnostic task by modifying the pretrained model and fine-tuning the entire architecture end-to-end.The task-specific modification is typically only a learned classification layer, contrasting with specialized task-specific model development.Source: §3.2 · p. 5
  • The transfer suite covers VQA, VCR, referring-expression grounding, and caption-based image retrieval, using task-specific datasets and lightweight prediction heads.VQA uses VQA 2.0 with 1.1 million questions and 3,129 possible answers; VCR uses 290k multiple-choice problems from 110k movie scenes; retrieval uses Flickr30k with 31,000 images and five captions each.Source: §3.2 · pp. 5–6

Zero-Shot Retrieval

  • In zero-shot caption-based image retrieval, pretrained multimodal alignment is applied directly to Flickr30k without task-specific fine-tuning to test transferable text grounding.This setting is intended to assess generalization across visual and linguistic variation.Source: §3.2 · p. 6

4 Results and Analysis

ViLBERT outperforms task-specific state-of-the-art models across four transfer tasks, with gains attributable to its two-stream architecture and Conceptual Captions pretraining. Results also show benefits from increased model depth, larger pretraining datasets, and semantically meaningful zero-shot alignment.

Results

  • ViLBERT outperforms task-specific state-of-the-art models across all four transfer tasks using a single base architecture and one classifier per task.It achieves state-of-the-art on VCR, RefCOCO+, and image retrieval by 7-10 percentage points.Source: Table 1 · p. 7, §4 · p. 7
  • ViLBERT improves performance over the single-stream baseline across tasks for both pretrained and non-pretrained models, especially on VQA and RefCOCO+.The comparison isolates the impact of the two-stream architecture while retaining BERT initialization and Faster R-CNN visual features in the baseline.Source: §4 · pp. 6–7
  • Pretraining under ViLBERT’s proxy tasks improves performance by between 2% and 13% across tasks and also benefits the single-stream architecture.This supports improved visiolinguistic representations and generalization across model architectures.Source: §4 · p. 7

Effect of Visual Stream Depth

  • VQA and image retrieval performance increases monotonically with visual-stream depth until six layers, while zero-shot image retrieval continues gaining as depth increases.Different tasks perform better at different depths, implying different context-aggregation needs.Source: §4 · p. 7, Table 2 · p. 7

Benefits of Large Training Sets

  • Pretraining accuracy grows monotonically as the Conceptual Captions dataset expands from 25% to 50% and the full dataset, suggesting benefits from more data.The experiment uses random 25% and 50% subsets with the same pretraining and fine-tuning setup.Source: §4 · p. 7, Table 3 · p. 8

What does ViLBERT learn during pretraining?

  • 31.86 vs 58.20 R1 shows zero-shot retrieval trails fine-tuning, yet 31.86 vs 48.60 R1 indicates meaningful alignment without Flickr30k images or captions.Qualitative examples further examine what ViLBERT learns during Conceptual Captions pretraining.Source: §4 · p. 8, Figure 5 · p. 8

5 Related Work

Related work situates ViLBERT within self-supervised representation learning for vision and language, while identifying vision-and-language task families not addressed and concurrent video-language pretraining.

Self-Supervised Learning

  • Self-supervised learning trains deep models on automatically derived supervision from regularities in existing data, with progress reported separately in vision and language.The passage notes that self-supervised image representations still lag behind those trained with image classification.Source: §5 · p. 8

Vision-and-Language

  • The approach does not cover visually grounded dialog, embodied question answering and instruction following, or image and video captioning.These task families are proposed as potential beneficiaries of similar self-supervised methods.Source: §5 · p. 8

Self-Supervised Learning for Vision-And-Language

  • Concurrent work learns joint video-language representations from cooking videos paired with text-to-speech transcripts using self-supervised tasks paralleling this approach.That work uses a unified BERT architecture for visual and linguistic inputs, similar to the Single-Stream baseline considered here.Source: §5 · p. 9

6 Conclusion

ViLBERT jointly models image content and text through a two-stream architecture with co-attentional transformer blocks, pretrained on a large automatically collected dataset to learn visual grounding. Transferred with only a classifier addition, it outperforms ablations and exceeds state-of-the-art on multiple established vision-and-language tasks.

6 Conclusion

  • ViLBERT jointly models image content and text using a novel two-stream architecture with co-attentional transformer blocks.The architecture is designed to learn joint representations across visual and textual inputs.Source: §6 · p. 9
  • Pretraining on a large, automatically collected dataset enables ViLBERT to learn visual grounding before task transfer.The model is pretrained as a joint image-and-text representation rather than learning grounding only during downstream task training.Source: §6 · p. 9
  • Transferred to multiple established vision-and-language tasks, ViLBERT outperforms sensible ablations and exceeds state-of-the-art.Task adaptation requires only minor architectural additions, specifically a classifier.Source: §6 · p. 9

Related papers

Loading 1908.02265v1…