BEiT: BERT Pre-Training of Image Transformers

• Vision Transformers are data-hungry, and BERT-style image pre-training lacks a predefined patch vocabulary while pixel reconstruction can waste capacity on short-range details.The paper motivates self-supervised pre-training as a way to leverage large-scale image data; earlier vision Transformer approaches include contrastive learning and self-distillation.Source: §1 · p. 1
• BEIT masks image patches and trains a Transformer to predict their original discrete visual tokens, obtained by tokenizing the image with a discrete VAE.This replaces raw-pixel recovery with prediction over visual tokens.Source: §1 · pp. 1–2, Figure 1 · p. 2
• BEIT outperforms from-scratch training and previous strong self-supervised models on image classification and semantic segmentation; supervised intermediate fine-tuning further improves performance.The paper also reports faster, more stable fine-tuning and reduced end-task training costs.Source: §1 · p. 2
• The work proposes masked image modeling for self-supervised vision Transformer pre-training and gives a theoretical explanation from the variational autoencoder perspective.Ablations indicate that the proposed techniques are critical to BERT-style pre-training effectiveness for images.Source: §1 · p. 2
• Self-supervised BEIT learns to distinguish semantic regions and object boundaries without human annotation.The paper also reports that pre-training reveals reasonable semantic regions from supervision signals in images.Source: §1 · p. 2

2.1 Image Representations

• The method uses two image representations: raw-pixel patches as inputs and discrete visual tokens as pretraining targets.Patches are flattened and linearly projected, in a manner likened to BERT word embeddings.Source: §2.1 · p. 3, §2.1.1 · p. 3

2.1.1 Image Patch

• The image is split into patches that preserve raw pixels; experiments use 224 × 224 images divided into a 14 × 14 grid of 16 × 16 patches.The patches are flattened and linearly projected into vectors for Transformer input.Source: §2.1.1 · p. 3

2.1.2 Visual Token

• Following [RPG+21], BEiT obtains discrete visual tokens with a dVAE tokenizer and decoder trained to reconstruct images from codebook tokens.Because the tokens are discrete, the tokenizer uses Gumbel-softmax relaxation for training and a uniform prior.Source: §2.1.2 · p. 3

2.2 Backbone Network: Image Transformer

• BEiT uses a standard Transformer backbone, with linearly projected patch embeddings, a prepended [S] token, and learnable 1D positional embeddings.The encoder’s final-layer vectors represent the image patches.Source: §2.2 · p. 3

2.3 Pre-Training BEIT: Masked Image Modeling

• BEiT masks about 40% of image patches in blocks, replaces them with a learned embedding, and predicts their corresponding discrete visual tokens from Transformer representations.Unlike pixel-level auto-encoding, the objective predicts discrete tokens that summarize details as higher-level abstractions; the paper reports that ablations show it outperforms pixel-level auto-encoding.Source: §2.3 · pp. 3–4

2.4 From the Perspective of Variational Autoencoder

• BEiT pretraining has two stages: train a dVAE tokenizer and decoder for image reconstruction, then learn the visual-token prior while keeping those modules fixed.This two-stage procedure is described as similar to [vdOVK17] [RvdOV19].Source: §2.4 · p. 4

2.5 Pre-Training Setup

• Pretraining uses ImageNet-1K’s roughly 1.2M training images without labels and runs for about 500k steps, or 800 epochs, with a 2k batch size.The setup uses a ViT-Base-like 12-layer, 768-hidden-size Transformer and 8192-token vocabulary.Source: §2.5 · p. 5

2.6 Fine-Tuning BEIT on Downstream Vision Tasks

• For downstream tasks, BEiT is fine-tuned end-to-end with task layers; classification uses average pooling and a linear softmax classifier, while segmentation uses a deconvolution decoder.The model can also be fine-tuned on an intermediate data-rich dataset before the target task.Source: §2.6 · p. 5

3.1 Image Classification

• BEIT outperforms previous self-supervised Transformer methods on ImageNet fine-tuning and improves over random initialization on ImageNet.The comparison includes methods trained with different pre-training data and scales; the passage identifies iGPT-1.36B and ViT-JFT300M as the most comparable auto-encoding approaches.Source: §3.1 · p. 6, Table 1 · p. 7
• Intermediate fine-tuning on labeled ImageNet data provides an additional gain after BEIT self-supervised pre-training, complementing supervised pre-training.Source: §3.1 · p. 6
• Higher resolution improves BEIT results by 1+ points; BEIT384 outperforms ImageNet-22K-supervised ViT384 at the same resolution.Scaling from base to large improves BEIT by 2.0 and BEIT384 by 1.7; the corresponding supervised ViT384-L gain over ViT384 is 1.2.Source: §3.1 · p. 6, Table 1 · p. 7
• Fine-tuning BEIT achieves better performance and converges faster than training DeiT from scratch, reaching reasonable results within very few epochs.Source: §3.1 · p. 7, Table 2 · p. 7

3.2 Semantic Segmentation

• On ADE20K semantic segmentation with SETR-PUP [ZLZ+20] as the task layer and single-scale inference, BEIT outperforms supervised ImageNet pre-training without manual pre-training annotations; intermediate ImageNet fine-tuning further improves BEIT.Source: §3.2 · pp. 7–8, Table 3 · p. 7

3.3 Ablation Studies

• Ablations show blockwise masking helps both tasks, visual-token prediction is key, recovering all tokens hurts, and longer pre-training improves downstream performance.Pixel-level auto-encoding performs worse than the visual-token task and even worse than training a vision Transformer from scratch; blockwise masking helps pixel-level auto-encoding by relieving short-distance dependency.Source: §3.3 · p. 8, Table 4 · p. 8

3.4 Analysis of Self-Attention Map

• BEIT self-attention distinguishes semantic regions and separates objects without manual annotations or task-specific supervision.The attention visualizations use last-layer query-key scores on MS COCO images excluded from pre-training; the passage suggests this learned knowledge may improve generalization, especially on small-scale datasets.Source: §3.4 · p. 8

Self-supervised visual representation learning

• Prior visual self-supervision used patch-order [NF16], colorization [ZIE16], rotation, masked real/fake prediction [TLL19], contrastive views, negative-free BYOL [GSA+20]/SimSiam [CH20], and clustering.Contrastive methods commonly rely on large memory banks or batches to obtain informative negatives.Source: §4 · pp. 8–9

Self-supervised vision Transformers

• Unlike iGPT’s clustered color tokens and ViT’s masked-patch mean-color prediction, BEiT inputs image patches and predicts discrete-VAE visual tokens.The paper says these visual tokens achieve better performance and are better theoretically motivated than heuristically designed pre-training tasks.Source: §4 · p. 9
• BEiT reports several times higher pre-training throughput and improved memory consumption compared with mainstream masked-auto-encoding, contrastive-learning, and self-distillation approaches.The passage points to Appendix E for throughput results but gives no numerical memory-consumption value.Source: §4 · p. 9

5 Conclusion

• BEiT introduces self-supervised pre-training for vision Transformers, supporting strong fine-tuning on image classification and semantic segmentation and making BERT-like masked-input pre-training work well.The framework uses auto-encoding with masked input.Source: §5 · p. 9
• BEiT automatically acquires knowledge about semantic regions without using human-annotated data.Source: §5 · p. 9
• Future work will scale BEiT pre-training in data and model size and pursue unified multimodal pre-training using similar objectives and a shared text-image architecture.Source: §5 · pp. 9–10

A Architecture Variants of Vision Transformer

• The experiments use standard ViT for fair comparisons, with base-size ablations pretrained for 300 epochs on ImageNet-1K under the Section 3.3 setting.Source: A Architecture Variants of Vision… · p. 14
• LayerScale [TCS+21] and relative position bias [RSR+20] improve ImageNet classification and ADE20K semantic segmentation; the improved architecture is called BEIT+.BEIT+ is used for the experiments in Appendix B.Source: A Architecture Variants of Vision… · p. 14, Table 5 · p. 14
• Vanilla Transformers are empirically the most stable when scaling to billions of parameters, so extra-large models omit LayerScale.Source: A Architecture Variants of Vision… · p. 14

B Comparison with Large-Scale Supervised Pre-Training

• BEIT+ uses LayerScale and relative position bias, with 150 ImageNet-22K pre-training epochs followed by 90 epochs of intermediate fine-tuning.The setup also uses an in-house dataset of about 70M labeled images as a drop-in replacement for ImageNet-22K.Source: B Comparison with Large-Scale… · p. 14
• 89.5% top-1 accuracy on ImageNet was achieved by BEIT-L after intermediate fine-tuning on an in-house 70M-image dataset.With ImageNet-22K (14M), BEIT-L achieves comparable performance to ViT-L trained on Google JFT-3B.Source: Table 6 · p. 14, B Comparison with Large-Scale… · p. 15
• BEIT-L achieves state-of-the-art performance on ADE20K semantic segmentation using UperNet at 640 × 640 resolution.The evaluation follows Swin's task-layer and resolution setup.Source: B Comparison with Large-Scale… · p. 15, Table 7 · p. 15

C Ablation Studies of Image Tokenizer

• The tokenizer was re-trained on ImageNet-1K without labels using the same 8K codebook size as DALL-E [RPG+21].The reimplementation was based on the DALLE-pytorch codebase.Source: C Ablation Studies of Image Tokenizer · p. 15, Table 8 · p. 15
• The reimplemented tokenizer was plugged into pre-training using the same experimental setup as the Section 3.3 ablation studies.Source: C Ablation Studies of Image Tokenizer · p. 15
• The reimplemented tokenizer achieves comparable reconstruction loss and ImageNet fine-tuning performance to the off-the-shelf DALL-E tokenizer.Table 8 reports ImageNet-1K top-1 accuracy and mean absolute error of normalized RGB values for image reconstruction.Source: Table 8 · p. 15, C Ablation Studies of Image Tokenizer · p. 15

D Linear Probes on ImageNet

• Discriminative methods such as contrastive learning [CXH21] and self-distillation [CTM+21] pretrain global image-level features, whereas generative methods such as iGPT and BEiT do not, making linear probing harder.Source: D Linear Probes on ImageNet · p. 15
• The probe averages patch hidden states and attaches a linear layer at an intermediate Transformer layer, with the best layer at 9 for BEiT-B and 14 for BEiT-L.The probe layer is trained for 50 epochs with AdamW, a 4e-3 learning rate with cosine decay, batch size 1024, weight decay 1e-4, and DINO-style random-resized crops and horizontal flips; evaluation uses central crops.Source: D Linear Probes on ImageNet · pp. 15–16
• Discriminative methods outperform generative pretraining on ImageNet linear probes; scaling BEiT from base to large significantly improves accuracy, and BEiT beats iGPT by a large margin with fewer parameters.Global feature aggregation benefits DINO and MoCo v3 under linear probing, but full fine-tuning eliminates that gap; the base-to-large gap is smaller for MoCo v3.Source: Table 9 · p. 16, D Linear Probes on ImageNet · p. 16

E Multi-Task Pre-Training with DINO

• Multi-task BEIT+DINO improves ADE20K semantic segmentation and achieves comparable ImageNet classification results.The comparison uses SETR-PUP [ZLZ+20] for ADE20K and reports single-scale mIoU; larger pre-training throughput indicates faster training.Source: E Multi-Task Pre-Training with DINO · p. 16, Table 10 · p. 16
• BEIT is more pre-training-efficient than DINO, which uses two Transformer copies for self-distillation and multi-crop augmentation [CMM+20].Throughput comparisons use the same batch size; BEIT's lower memory use allows larger batches and can yield greater practical speedup.Source: E Multi-Task Pre-Training with DINO · p. 16

F Image Classification on CIFAR-100

• CIFAR-100 fine-tuning evaluates 100 classes across 60k images using the same experimental setup as Section 3.1.Source: F Image Classification on CIFAR-100 · p. 17
• BEIT also outperforms MoCo v3 on CIFAR-100.Source: F Image Classification on CIFAR-100 · p. 17
• Intermediate fine-tuning on ImageNet-1K further improves BEIT's CIFAR-100 results.Source: F Image Classification on CIFAR-100 · p. 17
• 90.1% top-1 accuracy: BEIT exceeds ViT trained from scratch at 48.5% on CIFAR-100.The ViT result is from [CXH21]; the passage attributes BEIT's result to pre-training.Source: Table 11 · p. 17, F Image Classification on CIFAR-100 · p. 17

G Hyperparameters for Pre-Training

• Table 12 covers BEIT pre-training on ImageNet-1K.Source: Table 12 · p. 17

H Hyperparameters for Image Classification Fine-Tuning

• Table 13 covers BEIT fine-tuning for image classification on ImageNet-1K and CIFAR-100.Source: Table 13 · p. 18
• The classification fine-tuning table includes ImageNet-1K.Source: Table 13 · p. 18
• The classification fine-tuning table also includes CIFAR-100.Source: Table 13 · p. 18

I Hyperparameters for ADE20K Semantic Segmentation Fine-Tuning

• Table 14 covers BEIT fine-tuning on ADE20K for semantic segmentation.Source: Table 14 · p. 18