X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

1 Introduction

• LXMERT fails to produce high-fidelity, semantically meaningful images conditioned on captions despite strong performance on related tasks.The paper probes LXMERT as a representative vision-and-language model.Source: §1 · p. 1
• Masked visual-feature regression is challenging to optimize, adds inference noise, and compounds that noise during iterative sampling.A downstream generator cannot recover from the noisy predictions, even after fine-tuning on them.Source: §1 · pp. 1–2
• X-LXMERT generates semantically consistent imagery that rivals specialized state-of-the-art models, with little question-answering degradation versus LXMERT.The same refinements also produce X-UNITER images similar in quality to X-LXMERT.Source: §1 · p. 2
• X-LXMERT uses grid inputs, discrete visual representations, broad uniform masking ratios, and objective-aligned pre-training datasets to support image generation.The resulting model also performs discriminative tasks.Source: §1 · p. 2
• The refinements are not tied to LXMERT’s architecture and are designed to apply to a wider family of multimodal BERT models.X-UNITER provides an example: its generated images are similar in quality to X-LXMERT’s.Source: §1 · p. 2

Visual-Language transformer models

• Vision-and-language models improve performance on discriminative tasks and image captioning, using either joint single-stream or two-stream architectures.Many consume object-detection features; the paper probes image generation and proposes extensions for reliable image generation.Source: §2 · p. 2

Sequence generation with undirectional transformer

• For bidirectional transformers without a prescribed generation order, prior work uses Gibbs sampling, causal attention, or uniform masking for arbitrary-order generation.The paper adapts these strategies to grid-based image generation.Source: §2 · p. 2

Text-to-image synthesis

• Text-to-image methods include GAN-based and multi-stage generation, while X-LXMERT also answers questions and generates captions through a modular design.Its compact image generator can be replaced with other image-generation architectures.Source: §2 · p. 2

Grid visual representation

• Grid representations can be obtained from CNNs without expensive object-detection annotations and have performed nearly as well as bounding-box representations for VQA.The paper leverages grid visual representations to enable image generation with LXMERT.Source: §2 · p. 3

3 Background: Revisiting LXMERT

• LXMERT takes image and text inputs and represents them together with special CLS and EOS tokens.Its input sequence includes visual and word tokens.Source: §3 · p. 3
• LXMERT combines single-modality encoders with a cross-modality encoder that uses bidirectional cross-attention to exchange information and align entities.The CLS embedding serves as the cross-modality output.Source: §3 · p. 3
• LXMERT pre-trains with masked language modeling, masked visual-feature regression, masked object classification, image-text matching, and question answering.It is then fine-tuned for downstream tasks.Source: §3 · p. 3

4.1 Grid Image Features

• The probe replaces object-detection regions with a uniform N × N grid, using RoI Pooling to extract grid features.The grid supports Gibbs sampling of predicted visual features at T = N × N image locations.Source: §4.1 · p. 3

4.2 Image Generation

• The compact generator projects grid features through convolutional layers into residual blocks with SPADE layers to produce images.The described setup uses 8 × 8 grid features and outputs 256 × 256 images; the figure depicts a quantized 8 × 8 latent map.Source: §4.2 · p. 3, Figure 1 · p. 4
• The generator reconstructs images from ground-truth pre-trained grid features, but reconstruction errors remain versus modern autoencoders.The paper attributes errors mainly to the frozen encoder backbone and low-dimensional grid features.Source: §4.2 · pp. 3–4

4.3 Can LXMERT Paint?

• LXMERT’s predicted visual features are too noisy for meaningful image generation, and iterative Gibbs sampling propagates the error across the feature set.The high validation regression loss contributes to noisy predictions that are unsuitable for the downstream generator.Source: §4.3 · p. 4

5 X-LXMERT

• X-LXMERT adds discretized visual representations, uniform masking across varied ratios, and objective-aligned pre-training data, then uses Gibbs sampling to generate visual features from text.The extension is presented as a strategy that can be applied beyond LXMERT.Source: §5 · p. 4

5.1 From LXMERT to X-LXMERT

• Visual tokens are mapped to K-means cluster IDs and predicted with Cluster-Centroid Classification instead of feature regression.The paper reports that discretization improves feature prediction, limits noise propagation across sampling iterations, and generates richer imagery.Source: §5.1 · p. 4
• Uniform masking samples a vision-side masking ratio from [0,1] before randomly selecting positions, exposing the model to varied masking ratios.The paper reports that this benefits image generation; low-probability Bernoulli masking does not prepare the model for predicting all visual tokens, while very high probabilities harm pre-training.Source: §5.1 · pp. 4–5
• For visual-cluster prediction, X-LXMERT drops QA data and Visual Genome captions because questions can be ambiguous and many captions describe only small image regions.Those data are considered unsuited to the Cluster-Centroid Classification objective.Source: §5.1 · p. 5

5.2 Sampling Strategies for X-LXMERT

• Visual features are sampled iteratively rather than predicted all at once, with locations updated either one-by-one or in parallel.The grid starts with MASK tokens; the paper describes autoregressive and non-autoregressive strategies, including random-order autoregressive updates.Source: §5.2 · p. 5
• Mask-Predict-4 is selected for X-LXMERT after it consistently performs well across generation metrics; its updates begin at all grid locations and then decrease linearly.Within each iteration, the least-confident positions are updated.Source: §5.2 · p. 5

5.3 Training Details

• The image generator is trained with four losses and then kept fixed, while pre-training uniformly samples whether to mask image, text, or neither.Generator and discriminator losses are hinge adversarial, AC-GAN, feature matching, and perceptual; X-LXMERT pre-training runs for 20 epochs.Source: §5.3 · p. 6

6.1 Evaluating Image Generation

• Image-generation experiments use MS COCO captions and compare X-LXMERT with LXMERT and specialized text-to-image models.Image generation is difficult to evaluate because captions can admit varied outputs and image quality is subjective.Source: §6.1 · p. 6
• Automated image-quality metrics are Inception score for diversity and Fréchet Inception Distance for authenticity, while R-precision variants assess text-image conditioning.R-prec-easy uses randomly sampled negatives; R-prec-hard uses negatives that swap a caption word for another from the same category.Source: §6.1 · p. 6
• Human pairwise evaluations compare X-LXMERT with DM-GAN on caption-match preference and perceived realism, with ties allowed.Each evaluation used 5,000 image pairs.Source: §6.1 · p. 6
• HUMMUS tests whether image content supports recovery of a masked caption word, using majority agreement among annotators as its score.It was run on 2,800 image-caption pairs, with five annotators per pair; the masked words come from COCO nouns, verbs, and colors.Source: §6.1 · pp. 6–7

6.2 Evaluating Visual Question Answering

• Visual question answering uses VQA2.0 and GQA, where models answer questions about an image.Source: §6.2 · p. 7

6.3 Evaluating Visual Reasoning

• Visual reasoning is evaluated on NLVR2 dev and test-P, requiring a true-or-false judgment about a caption and two images.The paper describes NLVR2 as particularly challenging for contemporary vision-and-language models.Source: §6.3 · p. 7

7.1 Quantitative Results

• X-LXMERT significantly outperforms LXMERT across generation metrics, is comparable to AttnGAN and ControlGAN, and is competitive with DM-GAN on semantic metrics despite lower IS and FID.HUMMUS scores are on par with DM-GAN and significantly better than LXMERT; the authors note that HUMMUS misses synonyms and other valid descriptors.Source: §7.1 · pp. 7–8
• Human annotators prefer X-LXMERT to DM-GAN for both semantic match and image fidelity.The comparison uses 5,000 image pairs.Source: §7.1 · p. 8
• X-LXMERT shows a 1.5–2.5% drop on VQA2.0 and GQA and a roughly 2% drop on NLVR2, while retaining competitive performance.The implementation uses an 8×8 grid; the authors suggest that increasing grid size may shrink the VQA2.0, GQA, and NLVR2 gaps.Source: §7.1 · p. 8

7.2 From X-LXMERT to X-UNITER

• Adding the refinements to UNITER changes its image generation from very poor with grid inputs to scores comparable to X-LXMERT.This result supports the authors’ claim that the extensions are not specific to LXMERT’s architecture.Source: §7.2 · p. 8, Table 2 · p. 8

7.3 Qualitative Results

• Generated images preserve high-level semantics, though they lack fine detail; for complex scenes, X-LXMERT preserves some semantics better than DM-GAN.The paper gives “two people,” “clock tower,” and “home office” as examples.Source: §7.3 · p. 8, Figure 3 · p. 8
• Across 140 random autoregressive sampling steps, images improve from coarse salient objects to details and background.The intermediate-image figure illustrates the progression.Source: §7.3 · pp. 8–9, Figure 4 · p. 9
• With Gibbs sampling, prefix words such as “A,” “The,” “What,” and “Where” control generated text, including long meaningful captions and questions.The examples use common starting words as prefixes.Source: §7.3 · p. 9, Figure 5 · p. 9

7.4 Ablation Studies

• Discrete visual representations and uniform masking are critical for high-quality images, while updating pre-training data for the CCC objective is less critical but still useful.Image quality is fairly robust across sampling strategies, except TL→BR tends to produce worse results; differences between strategies are small.Source: §7.4 · p. 9, Table 4 · p. 9

• X-LXMERT unifies image generation, captioning, question answering, and visual reasoning.Source: §8 · p. 9
• LXMERT cannot generate meaningful images conditioned on text.Source: §8 · p. 9
• The authors report that their extensions can be applied to other vision-and-language transformer models.Source: §8 · p. 9

A Qualitative samples

• X-LXMERT samples include comparisons with DM-GAN, ground-truth-cluster reconstructions, and LXMERT, plus examples placing the same subject in varied contexts.Source: A Qualitative samples · p. 13

B Source code

• The project page includes an iterative-generation animation and demo, and the source code is available on GitHub.Source: B Source code · p. 13

C LXMERT / X-LXMERT details

• The reimplemented LXMERT models have 226.5M trainable parameters, slightly below the original implementation’s 228M because the MVFR and MOC heads share weights.Source: C LXMERT / X-LXMERT details · p. 13

C.1 LXMERT Architecture

• LXMERT combines text and object embedders with object-relation, language, and cross-modal encoders, followed by task-specific heads.Source: §C.1 · p. 13

C.2 X-LXMERT Architecture

• X-LXMERT uses 8×8 grid features quantized against 10,000 centroids and replaces LXMERT’s MOC and MVFR tasks with a cluster-classification task.The CCC head’s output dimension is changed from 1,600 to 10,000.Source: §C.2 · pp. 13–14

C.3 Datasets

• Pretraining reuses LXMERT’s datasets, but X-LXMERT uses only COCO captions for its CCC task.Source: §C.3 · p. 14

C.4 Visual vocabulary clustering

• The visual vocabulary is clustered from 2.6 million sampled grid features drawn from COCO train2014 images.The source grid features total 8 × 8 × 82,783, or 5.3 million.Source: §C.4 · p. 15

C.5 Training

• Pretraining samples a modality to mask and uses uniform image masking ratios, which the authors report benefits image generation.Image masking ratios are sampled uniformly from [0,1].Source: §C.5 · p. 15

C.6 Finetuning

• For VQA, GQA, and NLVR2 finetuning, task heads train alongside the pretrained models; finetuning takes 3, 5, and 1 hours, respectively.Source: §C.6 · p. 15

D.1 Generator Architecture

• The generator maps quantized 8×8 grid features to 256×256 RGB images through five residual upsampling blocks, with 1.7M trainable parameters.Its blocks use spatially adaptive instance normalization and spatial Gaussian noise.Source: §D.1 · pp. 15–16

D.2 Discriminator Architecture

• The discriminator downsamples images through five residual blocks and predicts both image-domain authenticity and cluster ids for 8×8 spatial layouts.It has 0.5M trainable parameters.Source: §D.2 · p. 16

D.3 Dataset

• The generator and discriminator are trained on the 82,783-image COCO train2014 split.Source: §D.3 · p. 16

D.4 Training

• The generator and discriminator use hinge adversarial, AC-GAN, discriminator feature-match, and perceptual losses, weighted 1, 1, 10, and 10.Adversarial loss guides realism; the other losses guide similarity to target images using spatial-layout inputs.Source: §D.4 · p. 16, Total loss · p. 16
• Training uses Adam with learning rates 0.0004 for the generator and 0.0001 for the discriminator for 60 epochs.The batch size is 96; training takes 15 hours on eight Titan V GPUs.Source: §D.4 · p. 16

Adversarial loss

• The adversarial loss uses the discriminator’s adversarial head to score generated and target images.The text identifies the generated image as X-hat and the target image as X.Source: Adversarial loss · p. 16, §D.5 · p. 16

AC-GAN loss

• The AC-GAN loss uses the discriminator’s classification head.Source: AC-GAN loss · p. 16

Discriminator feature match loss

• The discriminator feature-match loss uses features from its residual blocks.Source: Discriminator feature match loss · p. 16

Perceptual loss

• The perceptual loss uses features from ResNet-50 residual blocks conv2_x through conv5_x.Source: Perceptual loss · p. 16

E.1 Image metrics

• The evaluation generates images from 30,000 randomly sampled MS COCO val2014 captions for automatic image metrics.A subset of the captions is used for automatic evaluation.Source: §E.1 · p. 18
• Inception Score and FID are calculated using all 30,000 generated images.The authors follow Zhu et al. (2019) for both metrics.Source: §E.1 · p. 18
• R-precision-easy uses 99 negative captions per caption from different val2014 images, while R-precision-hard tests noun, verb, color, and number words with nine same-category word swaps.For each hard category, 1,000 captions containing a category word are sampled; category words are identified with spaCy, and results are reported per category.Source: §E.1 · p. 18, Table 6 · p. 19

E.2 Human evaluation

• HUMMUS evaluates masked noun, verb, and color words on 100 randomly sampled images per category, with 280 crowdworkers completing the task.Workers annotated a median of 13 images each, and per-category scores are reported in Table 7.Source: §E.2 · p. 18, Table 7 · p. 19
• Pairwise preference evaluation separately asks which image best matches a caption and which looks more realistic, using 357 crowdworkers.Workers completed a median of 17 annotations each; semantic preference includes the caption, while fidelity preference does not.Source: §E.2 · p. 18