PaperlayerView exact revision on arXiv

Source-linked AI summary

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei

arXiv:2208.10442v2cs.CVcs.CL

TL;DR

Vision-language foundation models need a general-purpose approach that handles multiple modalities and downstream task formats while scaling across data and model size. BEIT-3 addresses this gap with Multiway Transformers and unified masked “language” modeling over images, texts, and image-text pairs. It achieves state-of-the-art transfer performance across a broad range of vision and vision-language benchmarks, including a new state-of-the-art result on NLVR2.

  • Problem

    Existing vision-language foundation models use task-specific architectures and multiple pretraining objectives, motivating a general-purpose model that handles multiple modalities and downstream formats.

    Source: §1 · p. 2

  • Method

    BEIT-3 uses Multiway Transformers with modality-specific experts and shared self-attention, trained by masked data modeling on images, texts, and image-text pairs.

    Source: §2.1 · p. 3, §1 · p. 3, §4 · p. 9

  • Results

    BEIT-3 achieves state-of-the-art transfer performance across a broad range of vision and vision-language tasks, with NLVR2 performance exceeding 90%.

    Source: §1 · p. 3, §3.1 · p. 6, §4 · p. 9

  • Takeaways & Limitations

    Unified masked modeling and Multiway Transformers provide a simple, effective direction for scaling general-purpose multimodal foundation models.

    Source: §1 · p. 2, §4 · p. 9

Abstract

from arXiv · show

A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).

1 Introduction: The Big Convergence

BEIT-3 advances multimodal pretraining through a unified architecture, a single masked modeling objective across modalities, and scaling to large models and datasets. The resulting model transfers strongly across vision and vision-language tasks using publicly accessible resources.

Architecture

  • BEIT-3 adopts Multiway Transformers as one unified architecture supporting modality-specific encoding and cross-modality fusion across downstream task formats.The modular design shares parameters across modalities while accommodating vision, language, and multimodal processing.Source: §1 · p. 2

Pretraining task

  • A single mask-then-predict objective models images as Imglish, texts as English, and image-text pairs as parallel sentences.Masked modeling recovers corrupted text tokens or image patches and learns cross-modal alignments from paired data.Source: §1 · pp. 2–3

Scaling

  • Scaling model and data sizes supports transfer to diverse downstream tasks, while treating images as a foreign language enables reuse of large-scale language-model pretraining pipelines.The experiments use publicly accessible resources for academic reproducibility.Source: §1 · pp. 2–3

Results

  • BEIT-3 achieves state-of-the-art transfer performance across a broad range of vision and vision-language tasks, including detection, segmentation, classification, reasoning, question answering, captioning, and retrieval.The evaluation covers COCO, ADE20K, ImageNet, NLVR2, VQAv2, Flickr30K, and other benchmarks, using public resources for pretraining and finetuning.Source: §1 · p. 3

2 BEIT-3: A General-Purpose Multimodal Foundation Model

BEIT-3 combines a shared Multiway Transformer with modality experts and unified masked data modeling over monomodal and multimodal inputs. This general-purpose design can be reused across vision, retrieval, multimodal understanding, and generation tasks.

Backbone Network: Multiway Transformers

  • Each Multiway Transformer block combines shared self-attention with modality-specific feed-forward experts, including vision-language experts in the top three layers.Shared attention learns multimodal alignment and enables deep fusion, while experts capture modality-specific information.Source: §2.1 · p. 3
  • The unified architecture supports image backbones, dual encoders for efficient image-text retrieval, and fusion models for multimodal understanding and generation.The model can be repurposed across classification, detection, segmentation, retrieval, and multimodal tasks.Source: §2 · p. 3, §2.1 · p. 4, Figure 3 · p. 4

Pretraining Task: Masked Data Modeling

  • Masked data modeling randomly masks text tokens or image patches and trains BEIT-3 to recover them across monomodal and image-text data.The single mask-then-predict task learns representations and modality alignment while remaining scaling-up friendly.Source: §2.2 · p. 4

Scaling Up: BEIT-3 Pretraining

  • BEIT-3 uses a 40-layer Multiway Transformer with 1.9B parameters, including vision, language, vision-language, and shared self-attention components.When used as a vision encoder, only vision-related parameters—about 1B comparable to ViT-giant—are activated.Source: §2.3 · p. 5
  • The pretraining corpus combines public monomodal and multimodal resources, including about 15M images, 21M image-text pairs, 14M ImageNet-21K images, and 160GB of text.All listed data are academically accessible.Source: §2.3 · p. 5, Table 3 · p. 5

3 Experiments on Vision and Vision-Language Tasks

BEIT-3 is evaluated across a broad set of vision-language and vision benchmarks, achieving state-of-the-art transfer performance on many tasks. Its task-specific configurations include fusion encoders, dual encoders, conditional generation, and vision backbones.

Vision-Language Downstream Tasks

  • 84.03 vqa-score is achieved on VQAv2 test-dev and test-standard splits with a single model, exceeding previous models by more than 1.7 points.BEIT-3 is finetuned as a fusion encoder that jointly models image-question pairs and predicts among frequent answer candidates.Source: §3.1 · p. 6, Table 4 · p. 6
  • Above 90% accuracy on NLVR2 establishes a new state of the art, outperforming CoCa by about 5.6 points.The model jointly encodes two image-text pairs and classifies the concatenated pooled outputs.Source: §3.1 · p. 6
  • BEIT-3 creates new state-of-the-art results for COCO image captioning, ADE20K semantic segmentation, and ImageNet-1K classification.Captioning uses masked finetuning, segmentation uses a dense prediction adapter, and classification is formulated as image-to-text retrieval.Source: §3.1 · pp. 6–7, §3.2 · pp. 8–9
  • 3.0/4.0 absolute improvement on COCO top-1 image-to-text/text-to-image retrieval and 0.8/2.4 on Flickr30K are obtained by dual-encoder BEIT-3.The model outperforms prior and fusion-encoder-based systems, while dual encoders are more efficient for retrieval inference.Source: Table 5 · p. 7, §3.1 · p. 8

4 Conclusion

The paper presents BEIT-3 as a general-purpose multimodal foundation model that unifies masked modeling across images, texts, and image-text pairs. It reports state-of-the-art performance across broad vision and vision-language benchmarks and identifies Multiway Transformers as an effective general-purpose architecture.

4 Conclusion

  • BEIT-3 models images as a foreign language and applies masked “language” modeling uniformly to images, texts, and image-text pairs.The paper calls the image modality “Imglish” and uses this unified formulation across modalities.Source: §4 · p. 9
  • BEIT-3 achieves state-of-the-art performance across a wide range of vision and vision-language benchmarks.The conclusion characterizes the model as simple and effective and presents it as a direction for scaling multimodal foundation models.Source: §4 · p. 9
  • Multiway Transformers effectively model different vision and vision-language tasks as a general-purpose modeling option.The architecture supports modality-specific encoding and multimodal task transfer within the shared model.Source: §4 · p. 9

A Effects of Intermediate Finetuning for Retrieval

BEIT-3 is evaluated on COCO and Flickr30K retrieval with direct downstream finetuning, without intermediate finetuning on pretraining image-text pairs. The corresponding results are reported in Table 10.

A Effects of Intermediate Finetuning for Retrieval

B Hyperparameters Used for Pretraining

Table 11 presents the hyperparameters used for BEIT-3 pretraining.

B Hyperparameters Used for Pretraining

  • Table 11 contains the hyperparameters for pretraining BEIT-3.The supplied passage identifies the table but does not enumerate its settings.Source: Table 11 · p. 16

C Hyperparameters Used for Finetuning

Tables 12–17 provide fine-tuning hyperparameters for BEiT-3 across vision-language and vision tasks.

  • Table 12 specifies hyperparameters for fine-tuning BEiT-3 on NLVR2 and VQAv2.Source: Table 12 · p. 17
  • Table 13 specifies hyperparameters for fine-tuning BEiT-3 on COCO image captioning.Source: Table 13 · p. 17
  • Table 14 specifies hyperparameters for fine-tuning BEiT-3 on image-text retrieval.Source: Table 14 · p. 17
  • Table 15 specifies hyperparameters for fine-tuning BEiT-3 on semantic segmentation.Source: Table 15 · p. 18
  • Tables 16 and 17 specify hyperparameters for fine-tuning BEiT-3 on object detection and image classification, respectively.Source: Table 16 · p. 18, Table 17 · p. 18

Related papers

Loading 2208.10442v2…