PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, Desmond Elliott

arXiv:2011.15124v2cs.CLcs.CV

TL;DR

Reported performance differences between single- and dual-stream V&L BERTs are unclear. The paper unifies these architectures in a mathematical framework and compares five models in controlled experiments; under matched data and hyperparameters, they perform similarly, with embedding layers also playing a key role.

  • Problem

    Reported downstream-performance differences between single- and dual-stream models are unclear, with prior work disagreeing about whether either family is superior.

    Source: §1 · p. 1

  • Method

    The paper unifies single- and dual-stream architectures in a mathematical framework and conducts controlled comparisons of five V&L BERTs across four downstream tasks.

    Source: §1 · pp. 1–2, §5.2 · p. 8

  • Results

    Under matched data and hyperparameters, considered models perform similarly; single- and dual-stream families are on par, embedding layers matter, and results vary with random initialization.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    The findings support comparing stream families under matched conditions while treating embedding layers as consequential to model performance.

    Source: §7 · p. 12

    Experiments cover frozen visual encoders, BERTBASE-initialized base models pretrained on Conceptual Captions, 10 epochs, and a small hyperparameter grid; a controlled study of objectives is left for future work.

    Source: §5.8 · p. 11

Abstract

from arXiv · show

Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorised into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five V&L BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

1 Introduction

The paper examines why reported performance differences between single- and dual-stream V&L BERTs are difficult to interpret. It unifies the architectures mathematically and uses controlled studies to compare them and investigate potential confounds.

1 Introduction

  • Reported differences between single- and dual-stream models remain unclear, with prior papers disagreeing about which family is superior or whether a conclusion is possible.The figure contrasts published results with runs using the same Conceptual Captions data and NLVR2 fine-tuning conditions; marker area represents pretraining-data amount.Source: §1 · p. 1, Figure 1 · p. 1
  • The paper develops a unified mathematical framework in which proposed single- and dual-stream architectures are particular instances, then compares encoders in controlled experiments.The studies examine pretraining data and objectives, hyperparameters, initialization and fine-tuning variance, stream family, and embedding choice.Source: §1 · pp. 1–2
  • The contributions include VOLTA, a PyTorch implementation of the framework intended to speed research in multimodal pretraining, alongside the framework and controlled studies.The paper presents these as contributions to understanding and comparing V&L BERTs.Source: §1 · p. 2
  • Under matched training conditions, single- and dual-stream families perform equally well, although individual models differ and embedding layers play a key role.The paper also reports sensitivity to weight initialisation and cautions against state-of-the-art claims based on single runs.Source: §1 · p. 2

2 Vision-and-Language BERTs

V&L BERTs seek cross-modal representations from paired text and visual features. This section reviews their input embeddings, encoder designs, pretraining objectives, and other choices that can distinguish implementations and affect performance.

2 Vision-and-Language BERTs

  • V&L BERTs aim to produce cross-modal representations from token sequences and visual features that support downstream tasks grounded in both modalities.The section reviews input embeddings, encoder differences, and implementation choices that may affect performance.Source: §2 · p. 2

2.1 Input Embeddings

  • Language inputs use token, position, and segment embeddings; vision inputs typically use detector-extracted regions with features and spatial encodings, with model-specific variations.VL-BERT adds the whole-image feature to each token and uses a more complex geometry embedding; VISUALBERT omits spatial embeddings, while PIXEL-BERT extracts visual embeddings directly from raw images.Source: §2.1 · p. 2

2.2 Encoders

  • Single-stream encoders concatenate visual and linguistic features for joint processing, whereas dual-stream encoders process them separately before cross-modal layers model their interactions.The dual-stream design constrains interactions at each layer, while separate parameter sets increase expressive power; single-stream designs allow early, unconstrained fusion.Source: §2.2 · pp. 2–3

2.3 Pretraining Objectives

  • V&L BERTs jointly optimize weighted self-supervised objectives for language, vision, and cross-modal predictions.Common examples include masked language modelling, masked-region classification or feature regression, object-attribute prediction, and image–text matching.Source: §2.3 · p. 3

2.4 Further Distinctions

  • Implementations vary in attention-layer design, model and optimization settings, and pretraining data, whose size and type can affect downstream performance.LXMERT shares inter-modal attention parameters and uses an intra-modal layer consisting only of multi-head attention; datasets range from 3M–10M image–text pairs and may be in-domain or out-of-domain.Source: §2.4 · p. 3, Figure 2 · p. 3

3 A Unified Framework

The section places single- and dual-stream V&L BERTs within a common Transformer framework, then introduces a gated bimodal layer that can represent both and other interaction patterns.

3.1 Transformer Layers

  • A Transformer layer combines multi-head attention and a feed-forward block, with attention mapping queries to weighted sums of values using query–key scores.Multi-head attention projects inputs into multiple heads, concatenates their outputs, and applies a learned linear transformation.Source: §3.1 · p. 4

3.2 Single-stream Multimodal Transformers

  • Single-stream V&L BERTs concatenate embedded visual and textual inputs and attend across both modalities; approaches differ in embeddings, pretraining tasks, and training data.Source: §3.2 · p. 4

3.3 Dual-stream Multimodal Transformers

  • Dual-stream encoders use inter-modal attention for cross-modal interaction and intra-modal attention to process each modality independently.In inter-modal attention, keys and values come from the complementary modality; the rest of the layer follows the architecture-specific design.Source: §3.3 · p. 4

3.4 Dual-stream Attentions as Restricted Single-stream Attention

  • Single-stream attention computes intra- and inter-modal interactions together; dual-stream intra- and inter-modal attention are restricted versions that impose an inductive bias on layer interactions.The dual-stream score matrices are submatrices of the single-stream score matrix.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 Gated Bimodal Transformer Layers

  • Gates select permitted intra- and inter-modal interactions, while parameter-sharing controls let the gated bimodal layer recover inter-modal and single-stream blocks as special cases.The framework also encompasses combinations beyond those considered in existing encoders.Source: §3.5 · pp. 5–6
  • A gated bimodal layer regulates the four possible cross- and within-modality interactions and can tie parameters between modalities.It provides one framework for defining and evaluating existing V&L BERT architectures.Source: §3.5 · pp. 5–6

4 Experimental Setup

The controlled studies use VOLTA, a PyTorch implementation of the unified framework, and pretrain the models on Conceptual Captions. They evaluate common vision-and-language tasks using shared implementation choices.

4 Experimental Setup

  • VOLTA implements the unified framework in PyTorch to support research and control for implementation differences.The authors stress that a unified implementation helps measure architectural differences without possible implementation-detail confounds.Source: §4 · p. 6
  • After pruning broken links and images overlapping common test sets, pretraining uses 2.77M Conceptual Captions image–caption pairs.The dataset consists of images with weakly associated captions automatically collected from web pages.Source: §4 · p. 7
  • Evaluation spans VQA, image–text retrieval, referring expression, and multimodal verification tasks.The implementation extracts 36 regions of interest per image using Faster R-CNN with a ResNet-101 backbone.Source: §4 · pp. 6–7

5 Results

When pretrained on the same data, the models' downstream performance is similar and their reported gaps narrow. VOLTA results closely follow official-code implementations, supporting the correctness of the framework implementations.

5.1 Unified Data and Reimplementation

  • Official-code runs show lower VQAv2, NLVR2, and image-retrieval performance than reported papers, consistent with using less pretraining data.The authors relate the reduced data to diminishing returns as pretraining data size increases.Source: §5.1 · pp. 7–8
  • With shared pretraining data and tasks, the claimed performance gaps between V&L BERTs narrow, and their performance is similar.Figure 5 compares results from the original papers with official-code and VOLTA runs.Source: §5.1 · pp. 7–8
  • VOLTA performance closely follows official implementations, while VILBERT gains 2–3 points on NLVR2 and COCO image retrieval.The authors also report that VILBERT performs slightly worse than other models on VQAv2 in VOLTA, while LXMERT closes the gap with VILBERT on GQA.Source: §5.1 · p. 8

5.2 Controlled Setup

The controlled setup evaluates five models using shared hyperparameters and training choices. Most models perform similarly to their official setups, though LXMERT performs worse on two tasks.

5.2 Controlled Setup

  • The study evaluates VILBERT, LXMERT, VL-BERT, VISUALBERT, and UNITER on VQAv2, RefCOCO+, NLVR2, and Flickr30K.Models use a fixed set of hyperparameters and shared protocols for fine-tuning.Source: §5.2 · p. 8
  • Under shared hyperparameters, most models perform similarly to their official setups, while LXMERT performs worse on NLVR2 and VQAv2.Table 2 reports downstream results for models pretrained with VOLTA on 2.7M Conceptual Captions and fine-tuned on downstream tasks.Source: Table 2 · p. 8, §5.2 · p. 9
  • VQAv2 performance improves for VILBERT, showing that dual-stream models do not require different representation dimensions across streams.The controlled setup fixes VILBERT's visual representation dimension to match its linguistic stream.Source: §5.2 · pp. 8–9
  • VL-BERT performs similarly to its official setup, so adding ITM to the controlled pretraining objectives does not hurt downstream performance.The shared objectives are MLM, masked object classification with KL-divergence, and ITM.Source: §5.2 · pp. 8–9

5.3 Fine-tuning Variance

Repeated fine-tuning and pretraining runs reveal meaningful variation in downstream performance. Variation depends on the run and model, so single-run comparisons can be misleading.

5.3 Fine-tuning Variance

  • Across RefCOCO+ and NLVR2, fine-tuning has an average standard deviation of 0.3 points, while a model's minimum and maximum often differ by 1 or more points.Each model was fine-tuned 10 times per task, varying the seed, data order, and classification-layer initialization.Source: §5.3 · p. 9

5.4 Pretraining Variance

  • After 10 pretraining runs, LXMERT shows the largest variance among tested models, and its first run has Flickr30K text-retrieval recall 10 points below its mean.Each model was pretrained 10 times and fine-tuned once on four tasks; LXMERT also shows slower convergence.Source: §5.4 · p. 10
  • Pretraining-seed variation exceeds 1 point in several tasks for every model, although VILBERT varies less on VQA and retrieval and UNITER on referring expression.The authors attribute the varied runs to changes in data order and randomly initialized layers not inherited from BERT.Source: §5.4 · p. 10

5.5 Evaluating Local Decision Boundaries

On NLVR2 contrast sets, models that scored better on the standard test perform comparably to LXMERT. Controlled comparisons find no general performance difference between single- and dual-stream families, although individual models differ.

5.6 Single- or Dual-stream Architectures

  • Only Flickr30K text retrieval shows a significant single- versus dual-stream difference after Bonferroni correction (p < 0.005).The comparison groups VILBERT as the dual-stream model; LXMERT is excluded because of its sub-optimal performance in this setup.Source: Table 3 · p. 10, §5.6 · p. 10

5.5 Evaluating Local Decision Boundaries

  • On the NLVR2 contrast set, models that scored much better on the standard test achieve performance comparable to LXMERT.LXMERT loses around 15 points on perturbed samples; the authors interpret the pattern as evidence that the models exploited systematic gaps in the standard test set.Source: §5.5 · p. 10

5.6 Single- or Dual-stream Architectures

  • Pairwise tests find significant differences among models in every task, including VILBERT on Flickr30K text retrieval and UNITER on RefCOCO+.The post-hoc comparisons use a significance level of p < 0.005 after Bonferroni correction.Source: §5.6 · p. 11

5.7 The Importance of the Embeddings

The controlled experiments find that embedding choices substantially affect V&L BERT performance, while the study’s conclusions are bounded by its training and model settings.

5.7 The Importance of the Embeddings

  • Embedding layers play a crucial role in final performance, and are the only difference among VL-BERT, VISUALBERT, and UNITER in this setup.The authors report that embedding choice can have a drastic downstream impact, despite receiving little attention in the literature.Source: §5.7 · p. 11
  • VISUALBERT does not encode region locations in its embeddings and has considerably lower performance on RefCOCO+.The authors say this result shows that region-location information is extremely useful for this task.Source: §5.7 · p. 11
  • Swapping VILBERT and UNITER embeddings confirms their importance for referring-expression and retrieval tasks, but no single embedding layer performs best across both architecture families.The authors conclude that different embedding strategies are needed to maximize performance in the two families.Source: §5.7 · p. 11

5.8 Limitations

  • Experiments cover a specific pretrained, frozen visual encoder, BERTBASE variants, and CC; other visual encoders, pretraining data, and larger models remain future work.Some cited studies jointly learn the visual encoder with language, but that setting is outside this study’s experiments.Source: §5.8 · p. 11
  • The study uses 10 pretraining epochs and a small literature-based hyperparameter grid, and leaves a thorough controlled comparison of pretraining objectives for future work.The authors limit training to 10 epochs to reduce resource consumption.Source: §5.8 · p. 11

6 Reproducibility and the Environment

VOLTA is presented as a way to make V&L encoder comparisons more reproducible and efficient, against a backdrop of substantial compute and environmental costs.

6 Reproducibility and the Environment

  • VOLTA reduces implementation confounds and lets researchers preprocess visual and textual data once instead of creating model-specific formats.The authors present both as advantages for reproducible research and fair comparisons.Source: §6 · p. 11
  • Pretraining one model 10 times for four downstream tasks is estimated to cost about $6,000 and 200 GPU-compute days; the study’s training and evaluation used 1,500 GPU days.The authors also note severe environmental and carbon-footprint costs from V&L pretraining.Source: §6 · p. 12
  • The authors hope VOLTA will support easy, fair comparisons across architectures and help keep progress from being obscured by confounds.They also report distributing many pretrained models to amortize environmental costs.Source: §6 · p. 12

7 Conclusion

The paper unifies several V&L BERT architectures and compares them under controlled conditions. It finds sensitivity to initialization, parity between single- and dual-stream families, and an important role for embeddings.

7 Conclusion

  • Under a unified framework, the studied models vary with random initialization but achieve similar performance when trained with the same data and hyperparameters.The paper also finds that single- and dual-stream families are on par, while embedding layers play a crucial role in final performance.Source: §7 · p. 12
  • The conclusion recommends using similar pretraining datasets and reporting fine-tuning variance alongside the best-performing model.It also calls for more controlled evaluations of newly proposed V&L architectures.Source: §7 · p. 12
  • The authors distribute many pretrained V&L BERTs through VOLTA to amortize environmental costs.The paper reports substantial environmental and carbon-footprint costs from V&L pretraining.Source: §7 · p. 12
Loading 2011.15124v2…