PaperlayerView exact revision on arXiv

Source-linked AI summary

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Hao Tan, Mohit Bansal

arXiv:1908.07490v3cs.CLcs.CVcs.LG

TL;DR

Vision-and-language reasoning requires understanding visual content and language semantics, including how they align. LXMERT combines three Transformer encoders and pre-trains them on image-and-sentence tasks; it achieves state-of-the-art results on VQA and GQA and improves NLVR2 accuracy by 22% absolute.

  • Problem

    Vision-and-language reasoning requires understanding visual content and language semantics, alongside alignment and relationships between the modalities.

    Source: §1 · p. 1

  • Method

    LXMERT combines object-relationship, language, and cross-modality Transformer encoders, pre-trained on five image-and-sentence tasks to learn intra- and cross-modal relationships.

    Source: §1 · pp. 1–2

  • Results

    LXMERT achieves state-of-the-art results on VQA and GQA and improves NLVR2 accuracy by 22% absolute (54% to 76%).

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Ablation analyses support the effectiveness of several model components and training methods.

    Source: §7 · p. 9

Abstract

from arXiv · show

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert

1 Introduction

LXMERT addresses vision-and-language reasoning’s need to understand visual content, language semantics, and cross-modal relationships, an area the introduction describes as under-developed for large-scale pretraining and fine-tuning. It combines three Transformer encoders and diverse pretraining tasks to learn these connections.

1 Introduction

  • The framework uses three Transformer encoders: an object relationship encoder, a language encoder, and a cross-modality encoder focused on vision-language interactions.Its focus includes representations of a single image and its descriptive sentence.Source: §1 · pp. 1–2
  • Five pretraining tasks—masked cross-modality language modeling, masked-object RoI-feature regression and detected-label classification, cross-modality matching, and image question answering—train inference within and across modalities.The tasks support learning both intra-modality and cross-modality relationships.Source: §1 · p. 2
  • LXMERT achieves state-of-the-art overall accuracy on VQA and GQA; on NLVR2, accuracy improves from 54% to 76% (+22 points) and consistency from 12% to 42% (+30 points).For NLVR2, the model was fine-tuned and evaluated on real-world images that were not used for pretraining.Source: §1 · p. 2
  • Ablations assess model components and pretraining tasks against removals and alternatives, including BERT variants; the introduction also presents attention visualizations for all three encoders.The authors report that these comparisons demonstrate the effectiveness of the components and diverse pretraining tasks, and the need for a cross-modality pretraining framework.Source: §1 · p. 2

2 Model Architecture

LXMERT converts an image and its related sentence into object- and word-level features, then uses self-attention and cross-attention to produce language, vision, and cross-modality representations. Its architecture combines separate single-modality encoders with a cross-modality encoder that exchanges information between vision and language.

2.1 Input Embeddings

  • WordPiece tokens receive word and position embeddings, while each detected image object combines its 2048-dimensional RoI feature and bounding-box coordinates into a position-aware embedding.Object positions provide spatial information for reasoning and are needed for masked object prediction; the inputs’ object order is otherwise unspecified.Source: §2.1 · pp. 2–3

2.2 Encoders

  • Language and object-relationship encoders apply self-attention and feed-forward sub-layers within each modality; cross-modality layers add bidirectional cross-attention to exchange information and align entities.Each cross-modality layer also has two self-attention and two feed-forward sub-layers; residual connections and layer normalization follow each sub-layer.Source: §2.2 · pp. 3–4

2.3 Output Representations

  • LXMERT’s cross-modality encoder produces language and vision feature sequences, while the [CLS] token’s language feature serves as the cross-modality representation.The [CLS] token is prepended to the sentence words.Source: §2.3 · p. 4

3 Pre-Training Strategies

LXMERT is pre-trained on a large aggregated vision-and-language dataset using tasks designed to learn relationships within and across modalities. The strategy combines masked language and object prediction with cross-modal matching and image-question answering.

3.1.1 Language Task: Masked

  • Masked cross-modality LM predicts randomly masked words (15%) using language and visual information, allowing visual context to resolve ambiguity.The passage contrasts this with BERT’s language-only masked-word prediction.Source: §3.1.1 · p. 4

3.1.2 Vision Task: Masked Object Prediction

  • Masked object prediction zeros 15% of RoI features and trains feature regression and detected-label classification, using visible objects and language as context.Detected labels come from Faster R-CNN because object annotations differ across datasets; the authors note that these labels are noisy.Source: §3.1.2 · pp. 4–5

3.1.3 Cross-Modality Tasks

  • Cross-modality pre-training classifies image-sentence matches after 50% sentence replacement and predicts answers for matched image questions, about one-third of training sentences.The authors report that image-QA pre-training leads to a better cross-modality representation.Source: §3.1.3 · p. 5

3.2 Pre-Training Data

  • The pre-training data comprise 9.18M aligned image-and-sentence pairs from 180K images, with about 100M words and 6.5M image objects.The data aggregate five vision-and-language datasets; only train and development splits are used, avoiding test data during pre-training.Source: §3.2 · p. 5

3.3 Pre-Training Procedure

  • Pre-training uses a frozen Faster R-CNN feature extractor, 36 objects per image, and encoder depths of 9 language, 5 cross-modality, and 5 object-relationship layers.All encoder and embedding parameters are initialized from scratch, losses receive equal weights, and training runs for 20 epochs (about 670K steps) with batch size 256; image-QA training is limited to the last 10 epochs.Source: §3.3 · p. 6

4 Experimental Setup and Results

LXMERT is evaluated on VQA v2.0, GQA, and NLVR2, where it improves on prior best results across all three benchmarks.

4.1 Evaluated Datasets

  • The evaluation uses VQA v2.0, GQA, and NLVR2.Source: §4.1 · p. 6

4.2 Implementation Details

  • For NLVR2, LXMERT encodes each image-statement pair separately, then classifies using the concatenation of the two cross-modality outputs.For VQA and GQA, the model is fine-tuned from the pre-trained snapshot without data augmentation; GQA uses raw questions and images without functional-program or scene-graph supervision.Source: §4.2 · p. 6

4.3 Empirical Comparison Results

  • LXMERT gains 2.1% overall accuracy over BAN+Counter on VQA, 3.2% accuracy over BAN on GQA, and reaches 76.2% Accu on NLVR2 Test-U, a 22% improvement.The NLVR2 Test-U set is unreleased.Source: §4.3 · p. 7
  • VQA Binary/Other question sub-categories improve by 2.4%, while Number performance is equal to or better than BAN+Counter despite no explicit counting module.Source: §4.3 · p. 7
  • GQA open-domain question accuracy improves by 4.6% over the state-of-the-art method.Source: §4.3 · p. 7
  • NLVR2 consistency, the proportion of unique sentences whose related image pairs are all correctly predicted, reaches 42.1%, a 3.5-times improvement.Source: §4.3 · p. 7

5 Analysis

The analysis compares model alternatives and pre-training ablations, finding stronger results with LXMERT pre-training and combined visual objectives; it also examines attention across the three encoders.

5.1 BERT versus LXMERT

  • LXMERT achieves 74.9% accuracy on NLVR2, while results without LXMERT pre-training are around 22% absolute lower.On the public test set, LXMERT achieves 74.5% Accu and 39.7% Cons.Source: §5.1 · p. 7
  • Position-aware embeddings and cross-modality layers improve on BUTD, but language-only pre-training plateaus after three layers and remains 3.4% below full LXMERT.Replacing BUTD’s GRU with BERT yields results comparable to its LSTM encoder.Source: §5.1 · pp. 7–8
  • BERT initialization improves fine-tuning over random initialization but underperforms full LXMERT; its early pre-training loss advantage disappears after three epochs.The authors suggest BERT’s language-only pre-training may not teach vision-language connections.Source: §5.1 · p. 8

5.2 Effect of the Image QA Pre-training Task

  • Image-QA pre-training improves results on all three datasets, including NLVR2 by 2.1%, while data augmentation decreases results both with and without QA pre-training.The comparison uses 20 epochs without QA against the original schedule of 10 epochs without QA followed by 10 with QA; the NLVR2 images and statements are not used in pre-training.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Effect of Vision Pre-training tasks

  • RoI-feature regression and detected-label classification each yield reasonable results, and jointly pre-training with both achieves the highest results.Without vision pre-training tasks, results are similar to BERT+3 CrossAtt.Source: §5.3 · pp. 8–9, Table 5 · p. 8

5.4 Visualizing LXMERT Behavior

  • The appendix visualizes attention graphs in the language, object-relationship, and cross-modality encoders.Source: §5.4 · p. 9

6 Related Work

LXMERT builds on prior attention, Transformer, and object-region approaches, while introducing a multi-component architecture and additional pre-training tasks for cross-modal reasoning. Compared with ViLBERT and VisualBERT on overlapping tasks, it reports accuracy margins on VQA 2.0 and NLVR2.

6 Related Work

  • LXMERT adapts Transformer encoders for single modalities and cross-modal interactions, extending BUTD’s object RoI features with positional embeddings and object-relationship encoding.Its architecture is related to bi-directional attention, Transformer, and BUTD.Source: §6 · p. 9
  • Prior cross-modality pre-training included XLM’s joint cross-lingual representations and VideoBert’s masked language modeling over words and quantized visual tokens, but used a single Transformer encoder and BERT-style token-based tasks.LXMERT develops a different architecture and pre-training tasks for cross-modality needs.Source: §6 · p. 9

Recent works since our EMNLP submission

  • LXMERT reports a 1.5% accuracy margin on VQA 2.0 and a 9% accuracy margin on NLVR2 (15% in consistency) over ViLBERT and VisualBERT on overlapping tasks.It uses an object-relationship encoder and cross-modality layers, plus RoI-feature regression and image question answering; it also ranked in the top three on both VQA and GQA challenges among more than 90 teams.Source: Recent works since our EMNLP submission · p. 9

7 Conclusion · Appendix

LXMERT learns vision–language connections with a cross-modality encoder and large-scale pre-training on image–sentence pairs. The paper reports strong results on VQA, GQA, and NLVR2, and analyzes the effectiveness of model components and training methods.

7 Conclusion

  • LXMERT improves performance on NLVR2 by 22% and achieves state-of-the-art results on VQA and GQA.The conclusion presents NLVR2 generalizability alongside results on the two image question-answering datasets.Source: §7 · p. 9
  • The framework combines Transfermer encoders with a novel cross-modality encoder to learn connections between vision and language.Source: §7 · p. 9
  • LXMERT is pre-trained on a large-scale dataset of image-and-sentence pairs using diverse pre-training tasks.Source: §7 · p. 9
  • The paper uses detailed analysis and ablation studies to demonstrate the effectiveness of several model components and training methods.Source: §7 · p. 9

A Evaluated Datasets Description · B Details of NLVR2 Fine-tuning

The section characterizes VQA, GQA, and NLVR2 and describes how LXMERT is fine-tuned to classify whether an NLVR2 statement matches a pair of images.

A Evaluated Datasets Description

B Details of NLVR2 Fine-tuning

C Training, Validation, and Testing Splits

LXMERT’s data splits were designed to keep testing images out of pre-training and fine-tuning. Pre-training, validation, and test protocols differed across VQA, GQA, and NLVR2.

Training, Validation, and Testing Splits

  • All dataset testing images were excluded from pre-training and fine-tuning to prevent test-image exposure.The authors provide split details and reproducible code.Source: C Training, Validation, and Testing… · p. 12
  • Pre-training used COCO training images, COCO validation images except a 5k-image minival set, and Visual Genome images, excluding COCO test data.The 5k minival was sampled from COCO validation; COCO test captions and questions were excluded even though available.Source: C Training, Validation, and Testing… · p. 12
  • VQA used minival-related data for validation and the remaining train+val data for fine-tuning; GQA used testdev for validation and joint train+validation for fine-tuning.Testing used VQA v2.0 test-dev and teststandard, and GQA test-standard.Source: C Training, Validation, and Testing… · p. 12
  • NLVR2 retained its original splits: fine-tuning on train, model selection on val, and testing on Test-P and Test-U.Its images are not from MS COCO or Visual Genome.Source: C Training, Validation, and Testing… · p. 12

D Training Details of ‘BERT versus LXMERT’ · E Visualizing LXMERT Behavior

BERT-only experiments use longer training and a higher learning rate than the stated 5e−5 alternative, while LXMERT’s attention visualizations cover its language, object-relationship, and cross-modality encoders.

D Training Details of ‘BERT versus LXMERT’

  • BERT-only experiments are trained for 20 epochs, with batch size 64/128, because BERT was not pre-trained on these cross-modality datasets.Source: D Training Details of ‘BERT versus… · p. 12
  • The BERT-only learning rate is 1e−4 instead of 5e −5.Source: D Training Details of ‘BERT versus… · p. 12
  • LXMERT’s language-encoder attention graphs show behavior similar to the original BERT encoder’s graphs.Source: Figure 3 · p. 13
  • In the language-encoder and BERT visualizations, attention points to next words in one pair of graphs and previous words in the other.Source: Figure 3 · p. 13
  • A first-layer visualization shows an attention graph and its recovered scene graph for LXMERT’s object-relationship encoder.Source: Figure 4 · p. 13

E Visualizing LXMERT Behavior

E.1 Language Encoder · E.2 Object-Relationship Encoder · E.3 Cross-Modality Encoder

The encoder visualizations show BERT-like language attention, object connections that resemble a scene graph, and cross-modal attention patterns linking words and objects. The authors also suggest that articles may serve as unified attention targets and identify pairwise noun and verb relations as a direction for future pretraining.

E.1 Language Encoder

  • The language comparison uses the same sentence for LXMERT and BERT, with LXMERT attention taken from the pretrained model before task-specific fine-tuning.Source: §E.1 · p. 13
  • On that sentence, LXMERT’s second layer and BERT’s third layer attend to next words, while both models’ fourth layers attend to previous words.The authors describe the encoders’ attention behavior as similar.Source: §E.1 · p. 13

E.2 Object-Relationship Encoder

  • The object-relationship encoder’s first-layer attention highlights selected objects whose connections form a scene graph, suggesting it learns object relationships.The authors manually drew the connections according to the attention graph; other objects were mostly unattended.Source: §E.2 · p. 13

E.3 Cross-Modality Encoder

  • Cross-modality attention focuses on nouns and pronouns, which the authors describe as informative words in vision-and-language tasks.Source: §E.3 · pp. 13–14, Figure 5 · p. 13
  • For non-plural nouns, cross-modality attention instead focuses on articles.Source: Figure 5 · p. 13, §E.3 · p. 14
  • The authors speculate that articles may act like [CLS] and [SEP] as unified attention targets, and identify pretraining for image-text noun-noun and noun-verb relations as a direction to explore.Source: §E.3 · p. 14

Related papers

Loading 1908.07490v3…