Source-linked AI summary
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei
TL;DR
Vision-language foundation models need a general-purpose approach that handles multiple modalities and downstream task formats while scaling across data and model size. BEIT-3 addresses this gap with Multiway Transformers and unified masked “language” modeling over images, texts, and image-text pairs. It achieves state-of-the-art transfer performance across a broad range of vision and vision-language benchmarks, including a new state-of-the-art result on NLVR2.
Problem
Existing vision-language foundation models use task-specific architectures and multiple pretraining objectives, motivating a general-purpose model that handles multiple modalities and downstream formats.
Method
BEIT-3 uses Multiway Transformers with modality-specific experts and shared self-attention, trained by masked data modeling on images, texts, and image-text pairs.
Results
BEIT-3 achieves state-of-the-art transfer performance across a broad range of vision and vision-language tasks, with NLVR2 performance exceeding 90%.
Takeaways & Limitations
Unified masked modeling and Multiway Transformers provide a simple, effective direction for scaling general-purpose multimodal foundation models.
Abstract
from arXiv · showhide
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).
1 Introduction: The Big Convergence
BEIT-3 advances multimodal pretraining through a unified architecture, a single masked modeling objective across modalities, and scaling to large models and datasets. The resulting model transfers strongly across vision and vision-language tasks using publicly accessible resources.
- Architecture: BEIT-3 adopts Multiway Transformers as one unified architecture supporting modality-specific encoding and cross-modality fusion across downstream task formats.The modular design shares parameters across modalities while accommodating vision, language, and multimodal processing.
- Pretraining task: A single mask-then-predict objective models images as Imglish, texts as English, and image-text pairs as parallel sentences.Masked modeling recovers corrupted text tokens or image patches and learns cross-modal alignments from paired data.
- Scaling: Scaling model and data sizes supports transfer to diverse downstream tasks, while treating images as a foreign language enables reuse of large-scale language-model pretraining pipelines.The experiments use publicly accessible resources for academic reproducibility.
- Results: BEIT-3 achieves state-of-the-art transfer performance across a broad range of vision and vision-language tasks, including detection, segmentation, classification, reasoning, question answering, captioning, and retrieval.The evaluation covers COCO, ADE20K, ImageNet, NLVR2, VQAv2, Flickr30K, and other benchmarks, using public resources for pretraining and finetuning.
2 BEIT-3: A General-Purpose Multimodal Foundation Model
BEIT-3 combines a shared Multiway Transformer with modality experts and unified masked data modeling over monomodal and multimodal inputs. This general-purpose design can be reused across vision, retrieval, multimodal understanding, and generation tasks.
- Backbone Network: Multiway Transformers: Each Multiway Transformer block combines shared self-attention with modality-specific feed-forward experts, including vision-language experts in the top three layers.Shared attention learns multimodal alignment and enables deep fusion, while experts capture modality-specific information.
- Backbone Network: Multiway Transformers: The unified architecture supports image backbones, dual encoders for efficient image-text retrieval, and fusion models for multimodal understanding and generation.The model can be repurposed across classification, detection, segmentation, retrieval, and multimodal tasks.
- Pretraining Task: Masked Data Modeling: Masked data modeling randomly masks text tokens or image patches and trains BEIT-3 to recover them across monomodal and image-text data.The single mask-then-predict task learns representations and modality alignment while remaining scaling-up friendly.
- Scaling Up: BEIT-3 Pretraining: BEIT-3 uses a 40-layer Multiway Transformer with 1.9B parameters, including vision, language, vision-language, and shared self-attention components.When used as a vision encoder, only vision-related parameters—about 1B comparable to ViT-giant—are activated.
- Scaling Up: BEIT-3 Pretraining: The pretraining corpus combines public monomodal and multimodal resources, including about 15M images, 21M image-text pairs, 14M ImageNet-21K images, and 160GB of text.All listed data are academically accessible.
3 Experiments on Vision and Vision-Language Tasks
BEIT-3 is evaluated across a broad set of vision-language and vision benchmarks, achieving state-of-the-art transfer performance on many tasks. Its task-specific configurations include fusion encoders, dual encoders, conditional generation, and vision backbones.
- Vision-Language Downstream Tasks: 84.03 vqa-score is achieved on VQAv2 test-dev and test-standard splits with a single model, exceeding previous models by more than 1.7 points.BEIT-3 is finetuned as a fusion encoder that jointly models image-question pairs and predicts among frequent answer candidates.
- Vision-Language Downstream Tasks: Above 90% accuracy on NLVR2 establishes a new state of the art, outperforming CoCa by about 5.6 points.The model jointly encodes two image-text pairs and classifies the concatenated pooled outputs.
- Vision-Language Downstream Tasks: BEIT-3 creates new state-of-the-art results for COCO image captioning, ADE20K semantic segmentation, and ImageNet-1K classification.Captioning uses masked finetuning, segmentation uses a dense prediction adapter, and classification is formulated as image-to-text retrieval.
- Vision-Language Downstream Tasks: 3.0/4.0 absolute improvement on COCO top-1 image-to-text/text-to-image retrieval and 0.8/2.4 on Flickr30K are obtained by dual-encoder BEIT-3.The model outperforms prior and fusion-encoder-based systems, while dual encoders are more efficient for retrieval inference.
4 Conclusion
The paper presents BEIT-3 as a general-purpose multimodal foundation model that unifies masked modeling across images, texts, and image-text pairs. It reports state-of-the-art performance across broad vision and vision-language benchmarks and identifies Multiway Transformers as an effective general-purpose architecture.
- 4 Conclusion: BEIT-3 models images as a foreign language and applies masked “language” modeling uniformly to images, texts, and image-text pairs.The paper calls the image modality “Imglish” and uses this unified formulation across modalities.
- 4 Conclusion: BEIT-3 achieves state-of-the-art performance across a wide range of vision and vision-language benchmarks.The conclusion characterizes the model as simple and effective and presents it as a direction for scaling multimodal foundation models.
- 4 Conclusion: Multiway Transformers effectively model different vision and vision-language tasks as a general-purpose modeling option.The architecture supports modality-specific encoding and multimodal task transfer within the shared model.
A Effects of Intermediate Finetuning for Retrieval
BEIT-3 is evaluated on COCO and Flickr30K retrieval with direct downstream finetuning, without intermediate finetuning on pretraining image-text pairs. The corresponding results are reported in Table 10.
- A Effects of Intermediate Finetuning for Retrieval: BEIT-3 is directly finetuned on COCO and Flickr30K without intermediate finetuning on the pretraining data.Table 10 reports these image-text retrieval finetuning results.
- A Effects of Intermediate Finetuning for Retrieval: BEIT-3 outperforms previous state-of-the-art models without using an image-text contrastive objective during pretraining.The result is presented as evidence that masked data modeling can learn cross-modal representations.
B Hyperparameters Used for Pretraining
Table 11 presents the hyperparameters used for BEIT-3 pretraining.
- B Hyperparameters Used for Pretraining: Table 11 contains the hyperparameters for pretraining BEIT-3.The supplied passage identifies the table but does not enumerate its settings.
C Hyperparameters Used for Finetuning
Tables 12–17 provide fine-tuning hyperparameters for BEiT-3 across vision-language and vision tasks.
- Table 12 specifies hyperparameters for fine-tuning BEiT-3 on NLVR2 and VQAv2.
- Table 13 specifies hyperparameters for fine-tuning BEiT-3 on COCO image captioning.
- Table 14 specifies hyperparameters for fine-tuning BEiT-3 on image-text retrieval.
- Table 15 specifies hyperparameters for fine-tuning BEiT-3 on semantic segmentation.
- Tables 16 and 17 specify hyperparameters for fine-tuning BEiT-3 on object detection and image classification, respectively.