Masked Vision and Language Modeling for Multi-modal Representation Learning

1 INTRODUCTION

• Existing approaches often learn p(T|I) without p(I|T), potentially biasing cross-modal retrieval toward one direction.Other methods that mask both modalities rely on frozen object detectors or pre-trained image tokenizers.Source: §1 · pp. 1–2
• Joint masked V+L modeling reconstructs each modality using its masked input and the other modality’s unmasked signal.The method explicitly uses cross-attention so reconstruction learns interactions between vision and language.Source: §1 · pp. 1–2
• The proposed approach avoids frozen bottlenecks by training end-to-end on masked image and text signals, implicitly aligning image patches with language tokens.For example, visual dog-face information can reconstruct the masked token “dog,” while text can reconstruct masked image patches.Source: §1 · p. 2
• With common V+L alignment tasks, the method achieves state-of-the-art performance across a broad range of V+L tasks.The reported contributions include image-text retrieval, VQA, NLVR, and related evaluations.Source: §1 · p. 2
• Only ∼40% of the data used by state-of-the-art models is sufficient to match their performance, while the method performs significantly better than competitors with limited training data.The paper evaluates subsets containing approximately 39%, 25%, and 16% of the 4M pre-training data.Source: §1 · p. 2

2 RELATED WORK

• Cross-attention aligns modalities effectively but is computationally expensive because all possible image-text pairs may require processing.Hybrid methods reduce this burden by applying cross-attention only to pairs with high unimodal similarity.Source: §2 · pp. 2–3
• Contrastive learning avoids expensive cross-attention but generally requires millions of image-text pairs for strong zero-shot performance.Hybrid approaches combine contrastive loss with selective cross-attention to complement these limitations.Source: §2 · p. 3
• Prior masked modeling work typically treats MLM and MIM independently, predicts only one modality, or relies on object detectors and pre-trained image tokenizers.The proposed framework directly reconstructs RGB pixels and text tokens, eliminating those model and data dependencies.Source: §2 · pp. 2–3

3.1 MASKED VISION AND LANGUAGE MODELING

• Cross-attention is used in both reconstruction and multimodal alignment, enabling representations to incorporate information from the other modality.The image and text cross-modality encoders generate attentions from the opposite modality's features.Source: §3.1 · p. 4
• Random masking operates on [MASK] text tokens and raw image patches, with 32 × 32 masking patches preventing simple copying from neighboring pixels.The image encoder divides 224 × 224 inputs into 16 × 16 patches, while the larger masking patch encourages reconstruction beyond local neighborhoods.Source: §3.1 · p. 4
• MaskVLM jointly reconstructs masked images from text and masked text from images, using cross-attention so each modality contributes to the other's reconstruction.Image reconstruction uses masked image features and unmasked text, while text reconstruction uses masked text and the original image.Source: §3.1 · p. 4

3.2 MULTI-MODAL ALIGNMENT

• The model combines masked vision-language modeling with image-text contrastive learning and image-text matching, defining the total loss as L = LMV LM + LIT C + LIT M.MaskVLM uses normalized image and text features for contrastive alignment and predicts whether image-text pairs are aligned for matching.Source: §3.2 · pp. 4–5

3.3 PROBABILISTIC INTERPRETATION

• Unlike approaches modeling only p(T|I) or masked frozen visual features, MaskVLM estimates both p(I|T) and p(T|I) end-to-end to directly minimize variation of information.The probabilistic interpretation connects bidirectional conditional modeling to estimation of the multimodal joint distribution p(I, T).Source: §3.3 · p. 5

4.1 PRE-TRAINING DATASETS AND DOWNSTREAM TASKS

• The experiments use 4.1M unique images and 5.2M image-text pairs from CC, SBU, VG, and COCO, evaluating retrieval, VQA, NLVR2, and visual entailment.Retrieval is evaluated on COCO and Flickr30k, including zero-shot Flickr30k evaluation with the pre-trained model directly.Source: §4.1 · pp. 5–6

4.2 IMPLEMENTATION DETAILS

• The model is initialized with ImageNet-pre-trained ViT and RoBERTa encoders, using 60% image masking and 30% text masking during pre-training.The experiments use 50 epochs for the 4M dataset, 30 epochs otherwise, and 16 NVIDIA Tesla V100 GPUs.Source: §4.2 · p. 6

4.3 EVALUATION ON IMAGE-TEXT RETRIEVAL, VQA, NLVR, AND VE

• MaskVLM achieves state-of-the-art retrieval performance and consistently leads VQA, NLVR2, and visual entailment except on the NLVR2 validation split.It achieves the best retrieval metrics on COCO and Flickr30k except Flickr30k image-retrieval R@10 and text-retrieval R@5, and leads the listed downstream tasks relative to comparators with similar data scales.Source: Table 1 · p. 6, §4.3 · pp. 7–8, Table 3 · p. 7

4.4 EVALUATION WITH LIMITED PRE-TRAINING DATA

• With reduced pre-training data, MaskVLM’s advantage over ALBEF grows as the CC subset decreases, reaching 8.71 R@1 points for COCO image retrieval and 9.04 for text retrieval.MaskVLM trained with CC25% + COCO also beats Codebook trained with CC50% + COCO by 1.90 and 2.76 R@1 points for image and text retrieval.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 ABLATION STUDY

• Adding both MLM and MIM to ITC and ITM improves finetuned R@1 by 0.92 for image retrieval and 2.10 for text retrieval over ITC + ITM + MLM, while achieving the best zero-shot retrieval.Source: §4.5 · p. 9

4.6 QUALITATIVE RESULTS

• Qualitative reconstructions show that original images provide visual evidence missing from masked images, enabling more specific text predictions such as “dog,” “white fence,” and “cave.”The examples support use of information from both image and text modalities during masked signal reconstruction.Source: §4.6 · p. 9, Figure 5 · p. 9

• Masked vision and language modeling is proposed as a pre-training task for learning V+L representations.Source: §5 · p. 9
• The method is given a probabilistic interpretation to clarify its contribution.Source: §5 · p. 9
• The approach consistently achieves state-of-the-art performance across a broad range of V+L tasks in large-scale and limited-data regimes.Source: §5 · p. 9

• Downstream evaluation uses task-specific fine-tuning procedures, including 15 epochs for retrieval and VQA and 5 epochs for NLVR and VE.Batch sizes and learning rates are also adjusted by task.Source: §A.1 · p. 15
• Image-text retrieval is evaluated on COCO and Flickr30k using established splits and average recall to select the best validation model.The pretrained model is fine-tuned for 15 epochs with batch size 256 and learning rate 1 × 10^-5.Source: §A.1 · p. 15
• VQA uses VQA v2.0 with Visual Genome training samples and reports both test-dev and test-std performance with answer-frequency-weighted loss.Source: §A.1 · p. 15
• NLVR uses established data splits and direct fine-tuning without the additional text-assignment pre-training step used by several compared methods.Source: §A.1 · p. 15
• The VQA and NLVR architectures are illustrated in Figure 6.Source: Figure 6 · p. 15

• MaskVLM uses ImageNet-pretrained ViT and RoBERTa encoders, with detailed architectures visualized in Figure 7.Source: §A.2 · p. 15, Figure 7 · p. 16
• Cross-modality encoders use outputs from the other modality as keys and values for cross-attention.Figure 7 marks these cross-modal key and value connections with dotted lines.Source: §A.2 · p. 15, Figure 7 · p. 16
• For MIM, masked image features query unmasked text features, while MLM uses masked text features to query unmasked image features.Source: §A.2 · p. 16
• The reproducibility setup avoids per-loss weighting and layer decay during fine-tuning.Source: §A.2 · p. 16

• MaskVLM with one modality masked at a time performs slightly better than the version masking both modalities simultaneously.Source: §A.3 · p. 16
• Both reconstruction strategies achieve higher R@1 for image and text retrieval on COCO and Flickr30k than ALBEF.Source: §A.3 · p. 16
• The masking-strategy comparison evaluates fine-tuned image-text retrieval under the experimental setup described for the main retrieval evaluation.Source: §A.3 · p. 16

• The average recall varies by at most 0.26 for image retrieval and 0.10 for text retrieval across the tested image masking ratios.These comparisons use finetuned Flickr30k retrieval performance after pre-training on CC 50% + COCO.Source: §A.4 · p. 16, Table 7 · p. 17
• Increasing the text masking ratio from 0.15 to 0.3 leads to higher recall for both image and text retrieval.Source: §A.4 · p. 17, Table 7 · p. 17

• The evaluation covers UC Merced Land Use, MIT-67, CUB-200, Oxford Flowers, Caltech-256, and ImageNet-1K.Both models are pre-trained with the 4M dataset; ImageNet evaluation is not strictly zero-shot because both use ImageNet-pretrained weights.Source: §A.5 · p. 17
• MaskVLM consistently outperforms ALBEF across all datasets in image recognition.Table 8 reports Top-1 accuracy using ITC and ITM scores, with prompt-engineering results also included.Source: §A.5 · p. 17, Table 8 · p. 17
• Prompt engineering improves MaskVLM’s average accuracy but lowers ALBEF’s average accuracy across the datasets.Source: §A.5 · p. 17

A.6 ADDITIONAL EXAMPLES FOR THE QUALITATIVE ANALYSIS

• MaskVLM reconstructs masked text using both masked and original images, leveraging visual and linguistic information jointly.Figure 8 distinguishes reconstructions from masked images (“Recon (mask)”) and original images (“Recon (org)”).Source: §A.6 · p. 17, Figure 8 · p. 18

A.7 STATISTICS OF THE PRE-TRAINING DATASET

• The 4M pre-training dataset statistics are reported, but invalid web-data URLs can cause slightly different image-text pair counts across downloads.Source: §A.7 · p. 18, Table 9 · p. 18