Source-linked AI summary
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Hao Tan, Mohit Bansal
TL;DR
Vision-and-language reasoning requires representations that connect visual content, language, and their relationships, but large-scale pretraining for this modality pair remains under-developed. LXMERT combines three Transformer encoders with five pretraining tasks, achieving state-of-the-art results on VQA and GQA and a 22% absolute NLVR2 improvement from 54% to 76%.
Problem
Large-scale pretraining and fine-tuning for vision-and-language pairs remains under-developed despite progress in separate vision and language representations.
Method
LXMERT combines object relationship, language, and cross-modality encoders and pretrains them on image-and-sentence pairs with five diverse tasks.
Results
LXMERT achieves state-of-the-art results on VQA and GQA and improves NLVR2 accuracy by 22% absolute, from 54% to 76%.
Takeaways & Limitations
The pretrained cross-modality model generalizes from visual question answering to challenging visual reasoning, while ablations support the effectiveness of its components and training methods.
Abstract
from arXiv · showhide
Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert
1 Introduction
LXMERT addresses the under-developed problem of large-scale vision-and-language pretraining by learning visual, linguistic, and cross-modal representations together. Its pretraining framework achieves strong results across visual question answering and visual reasoning tasks.
- LXMERT introduces a pre-trained vision-and-language cross-modality framework built from object relationship, language, and cross-modality encoders.
- Five pretraining tasks learn masked language, masked objects, cross-modality matching, and image question answering to build intra-modality and cross-modality relationships.
- LXMERT outperforms previous methods across VQA and GQA question categories and achieves state-of-the-art overall accuracy.
- 22% absolute accuracy improvement on NLVR2 raises performance from 54% to 76%, demonstrating generalizability to challenging visual reasoning.
2 Model Architecture
LXMERT represents images as object sequences and sentences as word sequences, then processes them with separate modality encoders and a cross-modality encoder. Self-attention models within-modality structure, while bidirectional cross-attention aligns vision and language.
- Input Embeddings: The model converts an image and sentence into object-level image embeddings and word-level sentence embeddings before encoding them.
- Input Embeddings: Words receive index-aware embeddings, while detected objects combine RoI features with position features to create position-aware visual embeddings.
- Encoders: Separate language and object-relationship Transformer encoders model each modality independently before cross-modal interaction.
- Encoders: Each cross-modality layer uses two self-attention sub-layers, bidirectional cross-attention, and two feed-forward sub-layers.
- Encoders: Cross-attention exchanges information and aligns entities between modalities, producing language, vision, and cross-modality outputs.
3 Pre-Training Strategies
LXMERT pretrains on aggregated image-and-sentence data with masked-modality and explicit cross-modality tasks. The procedure combines five tasks, fixed visual inputs, and staged optimization before task-specific fine-tuning.
- Pretraining Overview: LXMERT pretrains on a large aggregated dataset with diverse modality-specific and cross-modality tasks to learn connections between vision and language.
- Language Task: Masked cross-modality language modeling masks words with probability 0.15 and predicts them using visible language and visual information.
- Vision Task: Masked object prediction masks RoI features with probability 0.15 and uses feature regression and detected-label classification to infer object properties.
- Cross-Modality Tasks: Cross-modality matching replaces each sentence with probability 0.5 and trains a classifier to identify whether the image and sentence match.
- Cross-Modality Tasks: Image question answering uses matched image-question pairs, and the authors report that including it improves cross-modality representations.
- Pretraining Procedure: The model is pretrained for 20 epochs, with image question answering used only during the last 10 epochs, and then fine-tuned for 4 epochs.
4 Experimental Setup and Results
LXMERT is evaluated on VQA, GQA, and NLVR2 using fine-tuning from pre-trained parameters, with comparisons against prior results and ablations. It improves reported state-of-the-art performance across these visual-question-answering and visual-reasoning benchmarks.
- Evaluation Setup: LXMERT is evaluated on VQA v2.0, GQA, and NLVR2, using fine-tuning from a pre-trained snapshot without data augmentation on VQA and GQA.For NLVR2, the model encodes both image-statement pairs and classifies their concatenated representations.
- VQA: 2.1% higher overall accuracy than BAN+Counter is reported on VQA, with 2.4% gains on Binary and Other questions.LXMERT's Number performance is equal to or better than the comparison despite lacking an explicit counting module.
- GQA: 3.2% higher accuracy than BAN is reported on GQA, including a 4.6% improvement on Open questions.The authors associate the larger gain with GQA's greater visual-reasoning requirements.
- NLVR2: 76.2% accuracy on NLVR2 Test-U improves the prior result by 22%, while consistency reaches 42.1%, a 3.5-times improvement.The evaluation uses the Accu and Cons metrics; each NLVR2 statement is associated with multiple image pairs.
5 Analysis
The analysis evaluates LXMERT against alternative architectures and pre-training strategies, finding that cross-modality and image-QA pre-training contribute substantially to performance.
- BERT versus LXMERT: 22% absolute lower accuracy occurs without LXMERT pre-training, compared with the full model’s 74.9% on NLVR2.The comparison evaluates approaches using BERT without LXMERT pre-training.
- BERT versus LXMERT: 74.5% Accu and 39.7% Cons are achieved by LXMERT on the public NLVR2 test set.
- BERT versus LXMERT: A 3.4% gap remains between models without cross-modality pretraining and the full LXMERT framework after adding three cross-attention layers.Without cross-modality pretraining, results become stationary after three layers.
- BERT versus LXMERT: Pre-training LXMERT from BERT parameters improves early pre-training loss but is eventually caught up by random initialization and remains weaker than the full model.The paper suggests BERT’s single-modality pre-training may not adequately capture vision-language connections.
- Effect of the Image QA Pre-training Task: 2.1% improvement on NLVR2 is obtained with image-QA pre-training, which improves results on all three evaluated datasets.The comparison uses 20 epochs without image-QA loss versus the original procedure including image-QA loss.
- Effect of the Image QA Pre-training Task: QA pre-training outperforms data augmentation, while adding data augmentation decreases results both with and without QA pre-training.
- Vision Pre-training Tasks: Joint RoI-feature regression and detected-label classification achieve the highest results among the compared vision pre-training configurations.Each visual task also produces reasonable results independently.
6 Related Work
LXMERT builds on bi-directional attention, Transformers, BUTD, and cross-modality pre-training, while distinguishing itself through a more detailed architecture and additional pre-training tasks.
- Model Architecture: LXMERT uses Transformer encoders for single modalities and designs a Transformer-based cross-modality encoder.
- Pre-training: Prior cross-modality pre-training includes XLM’s cross-lingual representations and VideoBERT’s masked language modeling over language and visual tokens.
- Context: The paper’s results were submitted to EMNLP in 2019 and compared with related preprints released afterward.
- Pre-training: LXMERT differs from ViLBERT and VisualBERT through its object-relationship encoder, cross-modality layers, RoI-feature regression, and image question answering tasks.
7 Conclusion
LXMERT is a Transformer-based cross-modality framework pretrained on image-sentence pairs, achieving strong visual-language reasoning results and supporting detailed component analyses.
- LXMERT learns vision-language connections with Transformer encoders and a novel cross-modality encoder.
- 22% improvement on NLVR2 is reported alongside state-of-the-art results on VQA and GQA.
- Ablation studies support the effectiveness of the model components and pre-training methods.
A Evaluated Datasets Description
The evaluation covers VQA, GQA, and NLVR2, spanning single-image question answering and visual reasoning over image pairs and statements.
- VQA: VQA answers natural-language questions about images using VQA v2.0, which contains 1.1M questions and reduces answer bias relative to VQA v1.0.
- GQA: GQA requires spatial understanding and multistep inference for single-image questions generated from ground-truth scene graphs.The dataset contains 22M questions.
- NLVR2: NLVR2 tests whether a natural-language statement correctly describes a pair of related images, using images and sentences not covered in pre-training.The dataset has 86K training, 7K development, and 7K test examples.
- NLVR2: The NLVR2 classifier predicts a label from the images and statement and is trained with log-likelihood or equivalent binary cross-entropy loss.
- NLVR2: For NLVR2, LXMERT concatenates the two image-sentence cross-modality representations before applying a GeLU classifier.
C Training, Validation, and Testing Splits
The paper uses carefully separated data splits for pre-training, validation, fine-tuning, and testing to prevent testing images from entering earlier stages.
- Testing images are excluded from all pre-training and fine-tuning steps, with dataset splits and reproducible code publicly released.The authors explicitly describe split precautions and provide implementation resources through the LXMERT repository.
- A 5k-image MS COCO validation subset serves as the mini-validation set, while remaining eligible COCO and Visual Genome images support pre-training.MS COCO test captions and questions are excluded from pre-training.
- VQA fine-tuning validates on data associated with the mini-validation images and tests on the test-dev and test-standard splits.The remaining train-plus-validation data is used for fine-tuning.
D Training Details of ‘BERT versus LXMERT’
This section describes the comparative BERT training setup and visualizes attention behavior across LXMERT’s language, object-relationship, and cross-modality encoders.
- Training details: BERT-only experiments run for 20 epochs with batch size 64/128 and use learning rate 1e−4 rather than 5e−5.
- Language encoder comparison: LXMERT and BERT attention graphs are compared across panels showing attention to next words versus previous words.Panels a and b show next-word attention, while panels c and d show previous-word attention.
- Object-relationship encoder: The object-relationship visualization includes an attention graph and a recovered scene graph from LXMERT’s first encoder layer.
- Attention visualizations: Attention visualizations examine LXMERT’s language, object-relationship, and cross-modality encoders.
E.1 Language Encoder
LXMERT’s language encoder exhibits attention patterns similar to BERT, while its object and cross-modality encoders focus on structured relationships among visual objects and linguistic elements.
- Cross-modality encoder: Figure 5 visualizes cross-modality attention connecting objects with words, including noun, pronoun, and article-focused patterns.
- Language encoder: Both LXMERT and BERT show layers attending to next words and other layers attending to previous words.The comparison uses the same sentence for LXMERT and BERT attention visualizations.
- Object-relationship encoder: LXMERT’s object-relationship encoder attends selectively to high-scoring objects, whose connections form a scene graph.The manually recovered connections are described as faithfully representing the figure’s scene graph.
- Cross-modality encoder: The cross-modality encoder focuses attention on nouns and pronouns because they are informative words in vision-and-language tasks.
- Cross-modality encoder: For non-plural nouns, attention shifts toward articles, which may serve as unified target entries for attention layers.The paper presents this as a possible role rather than a specifically designed behavior.