PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Training End-to-End Vision-and-Language Transformers

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, Michael Zeng

arXiv:2111.02387v3cs.CVcs.CLcs.LG

TL;DR

Fully transformer-based vision-and-language models can be more efficient than region-feature-based approaches but often lose downstream performance. METER systematically evaluates end-to-end design choices and achieves strong VQA results, reaching 77.64% with 4M pre-training images and 80.54% when scaled up.

  • Problem

    Fully transformer-based VLP models often underperform region-feature-based models, motivating investigation into performant end-to-end designs.

  • Method

    METER systematically investigates vision and text encoders, multimodal fusion, architectural design, and pre-training objectives for end-to-end transformer-based VLP.

  • Results

    77.64% VQAv2 test-std accuracy using 4M images surpasses VinVL by 1.04% and ALBEF by 1.6%, while scaling reaches 80.54%.

  • Takeaways & Limitations

    METER demonstrates competitive performance with state-of-the-art models using 4M pre-training images and achieves new state of the art on VQA when scaled up.

  • Takeaways & Limitations

    The study mainly focuses on discriminative tasks, while generative tasks such as image captioning are under-investigated.

Abstract

from arXiv · show

Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly. In this paper, we present METER, a Multimodal End-to-end TransformER framework, through which we investigate how to design and pre-train a fully transformer-based VL model in an end-to-end manner. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion module (e.g., merged attention vs. co-attention), architectural design (e.g., encoder-only vs. encoder-decoder), and pre-training objectives (e.g., masked image modeling). We conduct comprehensive experiments and provide insights on how to train a performant VL transformer. METER achieves an accuracy of 77.64% on the VQAv2 test-std set using only 4M images for pre-training, surpassing the state-of-the-art region-feature-based model by 1.04%, and outperforming the previous best fully transformer-based model by 1.6%. Notably, when further scaled up, our best VQA model achieves an accuracy of 80.54%. Code and pre-trained models are released at https://github.com/zdou0830/METER.

1. Introduction

METER investigates how to train fully transformer-based vision-and-language models end to end, addressing the performance gap with region-feature-based VLP. Its systematic design study identifies effective choices and reports strong VQA results with limited pre-training data and at larger scale.

  • Motivation: Fully transformer-based VLP models often underperform region-feature-based models, despite potentially more efficient end-to-end processing.Prior systems commonly use frozen object detectors for region features, which can limit model capacity and require time-consuming feature extraction.
  • Approach: METER systematically studies vision encoders, text encoders, multimodal fusion, encoder-only versus encoder-decoder architectures, and pre-training objectives.The framework evaluates models pre-trained on COCO, Conceptual Captions, SBU Captions, and Visual Genome across VQA, visual reasoning, image-text retrieval, and visual entailment.
  • Findings: Cross-attention improves multimodal fusion over self-attention alone, while encoder-only models outperform encoder-decoder models for VQA and zero-shot image-text retrieval under fair comparison.The encoder-decoder architecture remains more flexible for tasks such as image captioning.
  • Results: 77.64% VQAv2 test-std accuracy using 4M images surpasses VinVL by 1.04% and ALBEF by 1.6%.When further scaled up, the best METER model reaches 80.54% on VQAv2 test-std.
  • Results: 80.54% VQAv2 test-std accuracy is achieved by the best METER model when further scaled up.

2. Glossary of VLP Models

The glossary organizes representative VLP models by how they encode images, distinguishing object-detector region features, CNN grid features, and ViT patch features. It situates METER-style end-to-end transformer models within the progression away from detector-based visual representations.

  • Image representations: VLP models are categorized by their image encoding strategy: region features, CNN grid features, or ViT patch features.The taxonomy is summarized in the paper’s representative-model glossary.
  • OD-based Region Features: Region-feature-based models use pre-trained object detectors, with co-attention or merged attention for multimodal fusion.ViLBERT and LXMERT use co-attention, whereas VisualBERT, VL-BERT, and UNITER use merged attention.
  • CNN-based Grid Features: CNN-based end-to-end models feed grid features and text directly into transformers or discretize grid features before cross-modal processing.PixelBERT and CLIP-ViL directly use CNN grid features, while SOHO first discretizes them using a learned vision dictionary.
  • ViT-based Patch Features: Vision transformers motivate ViT-based VLP because CNNs can have slightly worse accuracy/FLOPs trade-offs than ViT counterparts.
  • ViT-based Patch Features: ViT-based models directly use image patch features, but earlier approaches lagged behind state-of-the-art downstream performance such as visual question answering.ViLT, visual parsing, and ALBEF are cited as ViT-based examples.

3. The METER Framework

METER is an end-to-end vision-and-language framework that combines separate pretrained vision and text encoders with multimodal fusion, optional decoding, and configurable pre-training objectives. The framework investigates fusion architectures and masked image modeling through several patch-classification designs.

  • Framework overview: METER first extracts visual patch and text features with separate vision and text encoders, then produces cross-modal representations through a fusion module.A decoder may optionally process the fused representations before final outputs.
  • Encoders: The vision encoder uses ViTs that segment images into patches, while the text encoder processes subword sequences before multimodal fusion.The framework studies pretrained language encoders including BERT, RoBERTa, ELECTRA, ALBERT, and DeBERTa, as well as embedding-only encoding.
  • Multimodal fusion: METER compares merged attention, which concatenates modalities for one transformer block, with co-attention, which uses separate blocks and cross-attention.Cross-attention enables interaction between the independently processed text and visual features.
  • Architecture: The framework evaluates encoder-only and encoder-decoder architectures, with the latter using a decoder that attends to encoder representations and previously generated tokens.For VQA classification, the encoder-decoder variant feeds text to the encoder and a classification token to the decoder.
  • Pre-training objectives: METER uses masked language modeling and image-text matching as default objectives, while investigating masked image modeling through masked patch classification.Image-text matching classifies matched and mismatched image-caption pairs; masked patch classification uses either in-batch negatives or discrete codes.
  • Masked image modeling: With discrete-code masked patch classification, a DALL-E VQ-VAE tokenizes images, 15% of patches are masked, and the model predicts the corresponding discrete tokens.The alternative in-batch-negative method selects each original masked patch from candidate patches drawn from the training batch.

4. Experiments

METER’s experiments examine encoder choices, multimodal fusion, architecture, pre-training objectives, and practical training techniques across VL tasks. The analyses identify design choices associated with stronger downstream performance and report competitive VQAv2 results.

  • Experimental Setup: The experiments compare vision and language encoders, fusion modules, encoder-only versus encoder-decoder designs, and pre-training objectives across several VL tasks.The evaluation covers VQAv2, SNLI-VE, and Flickr30k retrieval in exploratory studies, followed by broader downstream comparisons.
  • Encoder Analysis: Pre-trained text encoders remain important, while RoBERTa is robust and CLIP-ViT-224/16 or Swin Transformer perform strongly on the vision side.Without VLP, removing the pre-trained text encoder degrades downstream performance; Swin Transformer reaches 72.38 VQA test-dev without VLP.
  • Encoder Analysis: 77.19/77.20 VQA test-dev/test-std scores are achieved by CLIP-ViT-224/16 after VLP, outperforming previous state-of-the-art region-based VinVL models.The result supports the effectiveness of the selected vision encoder in the VLP setting.
  • Training Techniques: Using separate learning rates for randomly initialized and pre-trained parameters improves performance over applying one learning rate to all model components.The authors attribute the degradation from a shared learning rate possibly to aggressive finetuning of parameters that already encode useful vision and language knowledge.
  • Training Techniques: Increasing image resolution from 224 to 576 improves VQA scores by about 3 points for CLIP-ViT-224/32 and 1 point for CLIP-ViT-224/16.The improvement is especially large when the image-resolution-to-patch-size ratio is low.
  • Architectures and Objectives: Co-attention outperforms merged attention, encoder-only models outperform encoder-decoder models on two discriminative tasks, and masked image modeling degrades VQAv2 and Flickr30k retrieval performance.Masked language modeling and image-text matching improve downstream performance, whereas both tested masked image modeling objectives reduce it.
  • Comparison with Prior Arts: 77.64% VQAv2 test-std accuracy is achieved with 4M pre-training images, exceeding VinVL by 1.04% and ALBEF by 1.6%.When further scaled up, the best METER model reaches 80.54% VQAv2 test-std accuracy and maintains fast ViT-based inference speed.

5. Conclusion

METER systematically investigates end-to-end fully transformer-based vision-and-language pre-training. With only 4M images it achieves competitive performance, and scaling further produces a new VQA state of the art.

  • METER systematically investigates how to train a fully-transformer VLP model in an end-to-end manner.
  • Competitive performance is achieved with only 4M images for pre-training.
  • 80.54% accuracy on VQAv2 test-std is achieved when METER is further scaled up.

A. Implementation Details

METER is pre-trained on four image-caption datasets totaling about 4M images and 9M pairs, then evaluated across four VL task types using standard dataset splits with a VQAv2-specific adjustment.

  • Four datasets—COCO, Visual Genome, Conceptual Captions, and SBU Captions—provide about 4M images and 9M image-caption pairs for pre-training.
  • Downstream evaluation covers VQAv2, NLVR2, COCO and Flickr30k retrieval, and SNLI-VE visual entailment.
  • Best models use AdamW, learning rates of 1e-5 for bottom encoders and 5e-5 for the cross-modal module, and 100k steps on 8 NVIDIA A100 GPUs.
  • Fine-tuning selects learning rates from {1e-6, 2e-6, 5e-6, 1e-5} and image resolutions from {288, 384, 576}, with RandAugment applied.

B. Inference Time

METER is evaluated for inference speed and image captioning. It is faster than region-feature-based models, while its encoder-only architecture delivers reasonable captioning performance but is less suitable than encoder-decoder designs for generation.

  • Inference Time: METER is significantly faster than region-feature-based models and comparable to other ViT-based models, although slower than ViLT.
  • Inference Time: Inference time is measured as the average time to process one VQA instance on one NVIDIA V100 GPU.
  • Image Captioning: METER is also fine-tuned on COCO image captioning using maximum likelihood estimation.
  • Image Captioning: Reasonable image-captioning performance is achieved with an encoder-only architecture, while the authors expect encoder-decoder models to be more suitable for generative tasks.

D. Multi-scale Feature Fusion

Multi-scale feature fusion aggregates representations from different layers of the pre-trained text and visual encoders before cross-modal processing. The reported fusion strategy helps without VLP but can hurt after pre-training.

  • Motivation: Representations from different encoder layers are aggregated because layers can encode different types of information.
  • Results: Fusion improves performance without vision-and-language pre-training but can degrade performance after pre-training.
  • Method: For each text token or image patch, a gate is computed for every encoder layer and the representations are combined by a weighted sum.
  • Method: The fused representation is passed to the top cross-modal layers, and fusion is applied in both text and visual encoders.

E. Correlation between Vision-and-Language Tasks and Vision or Language Tasks

METER finds little to no correlation between performance on vision-and-language tasks and performance on pure language or vision tasks. This conclusion is based on encoder variations evaluated across VQAv2, SQuAD, and ImageNet.

  • The analysis varies text and vision encoders and compares VQAv2 test-dev performance with SQuAD or ImageNet performance.
  • Figure 6 presents the correlation between model performance on vision-and-language tasks and pure vision or language tasks.
  • Pearson correlations were -0.09 for VL versus language and 0.41 for VL versus vision, indicating little to no correlation in either setting.The corresponding p-values were 0.88 and 0.36.

F. Unimodal Tasks

After vision-and-language pre-training, text-only performance becomes slightly worse on average, while image-only performance improves for Swin Transformer but not for CLIP-ViT. These experiments discard the multimodal encoder, leaving multimodal-to-unimodal transfer as an open problem.

  • VLP slightly worsens average GLUE performance for the text encoder.The text-only evaluation fine-tunes the bottom text encoder on GLUE tasks.
  • VLP improves image-only performance for Swin Transformer but not for CLIP-ViT, possibly because of domain issues.Image-only evaluation uses linear classifiers on learned representations for CIFAR-10 and CIFAR-100.
  • The experiments discard the rest of the network, and using the multimodal encoder to improve unimodal performance remains an open problem.

G. Analysis on Pre-training Datasets

Pre-training dataset size and domain similarity both contribute to downstream performance. Among the evaluated datasets, CC, VG, and COCO produce the highest scores because CC and VG are largest while COCO most closely matches downstream domains.

  • Both data size and domain similarity contribute to downstream task performance.
  • CC and VG are the largest datasets, while COCO most closely matches downstream task domains.
  • Models pre-trained on CC, VG, and COCO obtain the highest scores on VQAv2 and Flickr30k zero-shot retrieval.Each dataset is evaluated after 50k pre-training steps with the same CLIP-ViT-224/32 image encoder, BERT-base-uncased text encoder, and coattention fusion.
  • Table 17 reports results for models pre-trained with different datasets.

I. Limitations

The paper’s limitations concern its emphasis on discriminative tasks, limited pre-training scale, and English-only experiments. Generative capabilities, larger-scale behavior, and cross-lingual generalization remain insufficiently investigated.

  • Generative Tasks: Generative tasks such as image captioning are under-investigated because the paper mainly focuses on discriminative tasks.Experiments on COCO image captioning are relegated to the appendix.
  • Scalability: The models are pre-trained with 4M or 14M images, so performance on larger datasets remains unclear.
  • English Data: The experiments use only English data, leaving generalization to other languages for future investigation.
  • Figures 7 and 8 visualize attention maps for METER-SwinBASE and METER-CLIP-ViTBASE on caption tokens.
Loading 2111.02387v3…