PaperlayerView exact revision on arXiv

Source-linked AI summary

iBOT: Image BERT Pre-Training with Online Tokenizer

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, Tao Kong

arXiv:2111.07832v3cs.CV

TL;DR

MIM for Vision Transformers lacks the semantically meaningful visual tokenizer that supports MLM, and visual semantics typically motivate a separate tokenizer-training stage. iBOT uses self-distillation with an online teacher tokenizer jointly optimized with MIM, achieving state-of-the-art results across classification and other downstream tasks, including 82.3% linear probing and 87.8% fine-tuning accuracy under the reported ImageNet-22K pre-training setup.

  • Problem

    MIM needs a semantically meaningful visual tokenizer, but visual semantics are difficult to extract and commonly suggest a separate tokenizer-pretraining stage.

  • Method

    iBOT performs MIM by self-distilling masked patch tokens from a teacher network used as an online tokenizer while jointly self-distilling the class token.

  • Results

    iBOT achieves state-of-the-art results across classification, object detection, instance segmentation, and semantic segmentation, including 82.3% linear probing and 87.8% fine-tuning accuracy in the reported ImageNet-22K setup.

  • Takeaways & Limitations

    The learned part-level semantic patterns support image recognition and robustness against common image corruptions while transferring to dense downstream tasks.

  • Takeaways & Limitations

    The MIM formulation uses an independence assumption under which masked tokens are reconstructed separately.

Abstract

from arXiv · show

The success of language Transformers is primarily attributed to the pretext task of masked language modeling (MLM), where texts are first tokenized into semantically meaningful pieces. In this work, we study masked image modeling (MIM) and indicate the advantages and challenges of using a semantically meaningful visual tokenizer. We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer. Specifically, we perform self-distillation on masked patch tokens and take the teacher network as the online tokenizer, along with self-distillation on the class token to acquire visual semantics. The online tokenizer is jointly learnable with the MIM objective and dispenses with a multi-stage training pipeline where the tokenizer needs to be pre-trained beforehand. We show the prominence of iBOT by achieving an 82.3% linear probing accuracy and an 87.8% fine-tuning accuracy evaluated on ImageNet-1K. Beyond the state-of-the-art image classification results, we underline emerging local semantic patterns, which helps the models to obtain strong robustness against common corruptions and achieve leading results on dense downstream tasks, eg., object detection, instance segmentation, and semantic segmentation.

1 INTRODUCTION

The paper adapts masked language modeling’s token-prediction paradigm to vision, focusing on the challenge of obtaining semantically meaningful visual tokens. iBOT addresses this with an online tokenizer jointly optimized through self-distillation and reports strong classification, robustness, and downstream-task results.

  • 1 INTRODUCTION: MIM adapts MLM’s masked-token reconstruction paradigm to Vision Transformers while targeting images’ internal structures rather than only global views.The motivation is to make MIM a standard pre-training component for Vision Transformers.
  • 1 INTRODUCTION: Visual tokenization is difficult because image semantics emerge progressively from continuous visual data, unlike linguistic semantics extracted from word-frequency statistics.This motivates exploring a jointly optimized tokenizer instead of relying on a separately pretrained tokenizer.
  • 1 INTRODUCTION: iBOT performs MIM through self-distillation, using a teacher network as an online tokenizer while the target network predicts masked patch-token outputs.The target receives a masked image, whereas the online tokenizer receives the original image.
  • 1 INTRODUCTION: The online tokenizer is jointly optimized with MIM through momentum updates, avoiding an extra tokenizer-pretraining stage.This provides a single-stage training pipeline.
  • 1 INTRODUCTION: 82.3% linear probing accuracy and 87.8% fine-tuning accuracy are reported on ImageNet-1K with ImageNet-22K pre-training, alongside emerging part-level semantics and strong downstream performance.The paper also reports benefits for robustness against common corruptions and dense tasks including detection and segmentation.

2 PRELIMINARIES

MIM masks image patches and trains a model to recover them from the corrupted image, while the paper formulates this prediction as knowledge distillation. Self-distillation instead uses past teacher-network iterations and cross-view predictions to provide the supervisory signal.

  • 2.1 MASKED IMAGE MODELING AS KNOWLEDGE DISTILLATION: MIM samples a random mask over an image-token sequence according to prediction ratio r and replaces selected patch tokens with a mask token.The corrupted image is then used to recover the masked tokens.
  • 2.1 MASKED IMAGE MODELING AS KNOWLEDGE DISTILLATION: MIM recovers the masked tokens from the corrupted image by maximizing their conditional log-likelihood, with separate reconstruction treated under an independence assumption.The objective sums contributions from masked positions.
  • 2.1 MASKED IMAGE MODELING AS KNOWLEDGE DISTILLATION: In BEiT, a categorical prediction distribution is trained against tokens produced by a pretrained discrete VAE tokenizer, making the objective resemble knowledge distillation.The discrete VAE clusters image patches into K categories and assigns one-hot category encodings.
  • 2.2 SELF-DISTILLATION: Self-distillation replaces the fixed tokenizer distribution with predictions from earlier iterations of the model and applies a teacher-student objective to two augmented image views.The teacher and student share the same backbone and projection-head architecture, while teacher parameters are obtained by exponential moving average.
  • 2.2 SELF-DISTILLATION: The teacher-to-student knowledge transfer is trained by minimizing cross-entropy between their predictive categorical distributions.The objective is symmetrized with another cross-entropy term between the two views.

3 IBOT

iBOT performs masked image modeling through self-distillation, using a jointly learnable teacher network as an online visual tokenizer. Cross-view [CLS] distillation acquires semantics, while patch-token distillation reconstructs masked tokens.

  • Framework: iBOT casts masked image modeling as self-distillation, with the teacher network serving as an online visual tokenizer.The tokenizer is jointly learnable with the MIM objective and requires no extra pre-training stage.
  • Masked Image Modeling: The MIM objective distills teacher outputs from non-masked patch tokens to supervise reconstruction of masked student patch tokens.Blockwise masking is applied to augmented views, and the loss is symmetrized across views.
  • Semantic Tokenization: Cross-view [CLS] self-distillation bootstraps visual semantics for the online tokenizer.iBOT shares projection-head parameters between [CLS] and patch tokens to transfer semantic abstraction to patch-token learning.
  • Semantic Tokenization: iBOT uses softmax token distributions rather than one-hot token ids because image patches can have ambiguous semantic meanings.This design treats tokenization as a distributional supervisory signal for image patches.
  • Architecture: Sharing the entire projection head between [CLS] and patch tokens empirically produces the best performance among the studied sharing strategies.The implementation uses a 3-layer MLP projection head with output dimension 8192.

4 EXPERIMENT

iBOT is evaluated across classification, transfer, dense prediction, visual semantics, robustness, and tokenizer ablations. The experiments report strong ImageNet results, favorable transfer and dense-task performance, emerging part-level patterns, and robustness against image changes.

  • Experiment setup: The experiments evaluate iBOT using k-NN, linear probing, fine-tuning, semi-supervised learning, and unsupervised learning protocols.Downstream evaluation also covers object detection, instance segmentation, semantic segmentation, and transfer learning.
  • Classification on ImageNet-1K: iBOT reaches state-of-the-art ImageNet-1K k-NN and linear-probing performance across ViT and Swin backbones.With ImageNet-22K pre-training, ViT-L/16 reaches 82.3% linear probing accuracy.
  • Classification on ImageNet-1K: 87.8% top-1 accuracy is achieved after fine-tuning an ImageNet-22K-pretrained iBOT model on 512-size images.ImageNet-22K pre-training also gives 84.4% and 86.6% top-1 accuracy for ViT-B/16 and ViT-L/16 at the reported setting.
  • Classification on ImageNet-1K: iBOT improves over DINO by 1.6% with 1% labels and 0.8% with 10% labels in semi-supervised ImageNet-1K evaluation.It also achieves 32.8% NMI in unsupervised learning, 1.8% above the previous state of the art.
  • Properties of iBOT representations: Patch-token visualizations show emerging high-level semantics and low-level textures, including shared part semantics and shared part textures.The analysis uses the highest-confidence patch tokens from an 800-epoch ViT-S/16 model.
  • Properties of iBOT representations: iBOT separates different objects or parts in self-attention maps and improves robustness against background changes, occlusion, and out-of-distribution examples.Part-wise linear classification compares patch-token and [CLS]-token representations to study recognition effects.
  • Ablation study on tokenizer: The tokenizer ablation evaluates self-distillation on the [CLS] token and alternatives including standalone DINO and a pre-trained DALL-E encoder.These experiments use 300-epoch ViT-S/16 pre-training with prediction ratio r = 0.3 and no multi-crop augmentation.

5 RELATED WORK

Prior visual representation methods enforce invariance across distorted views, while masked image prediction recovers pixels, uses contrastive objectives, or predicts tokenizer-generated labels. MIM with discrete visual tokenizers reframes prediction as classification and reduces emphasis on high-frequency details.

  • Visual Representation Learning: Self-supervised vision methods learn augmentation invariance through contrastive learning, asymmetric networks, or explicit distribution matching.These approaches enforce similarity across distorted views while avoiding model collapse.
  • Masked Prediction in Images: Masked image prediction has used raw-pixel recovery, mask contrastive learning, and discrete-VAE tokenizers.In multimodal learning, masked region classification instead uses category distributions from pre-trained detection models.

6 CONCLUSION

The paper presents iBOT as BERT-like pre-training for Vision Transformers, using self-distillation with an online tokenizer. It reports strong downstream performance and emerging part-level semantics linked to recognition and robustness.

  • Conclusion: iBOT performs masked image modeling through self-distillation with an online tokenizer.The framework targets classification, object detection, instance segmentation, and semantic segmentation.
  • Conclusion: The paper identifies emerging part-level semantics in MIM-pretrained models that support recognition accuracy and robustness against common image corruptions.The stated future direction is scaling iBOT to larger datasets or models and studying unlabelled data in the wild.

B MULTI-CROP

Multi-crop augmentation can destabilize iBOT because masked and non-masked views have mismatched distributions, but random MIM with non-masked global crops stabilizes training and improves performance. The default crop-scale split is s = 0.32.

  • Stability of MIM Pre-trained with Multi-Crop: Direct multi-crop use can cause instability and degrade accuracy because masked and non-masked images have distribution mismatch.The instability is monitored through NMI training curves.
  • Stability of MIM Pre-trained with Multi-Crop: Local crops contain few meaningful contents to predict, and increasing their minimum scale from 0.05 to 0.2 mitigates the performance drop.This comparison concerns the pipeline that performs MIM for all crops.
  • Stabilizing the Training with Non-Masked Global Crops: Random MIM with non-masked global crops provides a stable multi-crop pipeline and yields a substantial performance gain.iBOT uses the pipeline that randomly selects one of the global crops for MIM.
  • Range of Scales in Multi-Crop: Local crops are sampled from (0.05, s), while global crops are sampled from (s, 1), where s divides the two scale ranges.The empirically optimal value is s = 0.32 for both small-size and base-size models.
  • State-of-the-Art Comparison w/o and w/ Multi-Crop: iBOT achieves state-of-the-art performance without multi-crop and higher performance with multi-crop in the reported comparisons.The multi-crop comparison reports 79.4% linear probing accuracy.
  • Effective Training Epochs: Effective training epochs multiply pre-training epochs by a scaling factor accounting for additional images and resolutions introduced by multi-crop augmentation.The reported scaling factor is approximately 4 for DINO and iBOT, approximately 3 for SwAV or DINO with RN50, and 2 for contrastive methods without multi-crop.

C ADDITIONAL IMPLEMENTATIONS

The appendix reports evaluation protocols and implementation choices for classification, semi-supervised learning, detection, instance segmentation, and semantic segmentation.

  • Evaluation Protocols: Semi-supervised evaluation fine-tunes pretrained models end-to-end with a linear classification head.
  • Evaluation Protocols: 62.5% versus 61.9% shows frozen-backbone Vision Transformers outperform end-to-end fine-tuning with 1% of training data.
  • Evaluation Protocols: 65.9% versus 62.5% shows DINO’s logistic regressor outperforms its frozen-feature multiclass linear classifier at 1% data.
  • Evaluation Protocols: 75.1% versus 73.4% shows end-to-end fine-tuning from the first projection-head layer performs best with 10% data.
  • Dense Prediction: Detection and instance segmentation use Cascade Mask R-CNN, while semantic segmentation uses either a linear head or UPerNet.
  • Dense Prediction: Appending [CLS] LayerNorm to patch tokens consistently improves linear semantic segmentation, but provides no substantial UPerNet gain.

D ADDITIONAL RESULTS

Additional experiments evaluate dense prediction, retrieval, transfer, and robustness, including occlusion, shuffle, and background changes.

  • Dense Downstream Tasks: The appendix provides detailed dense-task results for object detection, instance segmentation, and semantic segmentation with small- and base-size models.
  • Dense Downstream Tasks: Evaluation reports APb 75 for detection, APm 50 and APm 75 for instance segmentation, and mAcc for semantic segmentation.
  • Transfer and Retrieval: ImageNet-1K pre-training performs better than ImageNet-22K for frozen-feature k-NN and linear probing, opposite to fine-tuning results.
  • Robustness: iBOT gains performance on nearly all ImageNet-9 background variants except Only-BG-T, where neither foreground nor its mask is visible.
  • Robustness: iBOT shows stronger robustness against occlusion than its supervised baseline across random, salient, and non-salient patch dropping settings.
  • Robustness: iBOT retains accuracy better under patch shuffling than supervised and ResNet-50 baselines, indicating less reliance on positional embeddings.

E ADDITIONAL ABLATIONS

Ablations examine training settings, projection-head design, self-distillation alternatives, token labels, and tokenizer choices, alongside efficiency results.

  • Ablations: Continuous soft labels are crucial for MIM; using only sharpening reaches 69.4 k-NN and 73.9 linear probing accuracy.
  • Ablations: Directly adding the class-token and MIM losses without scaling gives the best linear probing accuracy among tested ratios.
  • Ablations: Larger output dimensions provide no substantial gain, so iBOT uses K = 8192 by default.
  • Training and Efficiency: Performance is insensitive to prediction ratios from 0.05 to 0.4, while adding variance to the ratio consistently improves performance.
  • Training and Efficiency: 800 epochs improve linear probing, while iBOT reaches SwAV’s 800-epoch accuracy in fewer than 100 pre-training epochs.
  • Training and Efficiency: 300-epoch iBOT requires 73.3h and achieves 0.4% higher linear probing accuracy than 800-epoch DINO requiring 180.0h.
  • Tokenizer Comparison: The tokenizer comparison evaluates DALL-E, mean-color, patch-clustering, and standalone DINO tokenizers on ImageNet-1K validation accuracy.

G.1 PATTERN LAYOUT

The pattern-layout visualizations show that iBOT organizes patch tokens around both semantic structures and low-level visual details, while attention highlights representative object parts.

  • Pattern Layout: The visualizations display additional high-level semantic and low-level detail patterns using top-confidence patches.
  • Pattern Layout: iBOT can be represented by several representative patches, a behavior linked in the paper to robustness and recognition performance.
  • Pattern Layout: BEiT assigns discrete DALL-E token ids, whereas DINO assigns each patch the highest-probability index from a 65536-dimensional distribution.
  • Pattern Layout: Class-token semantic patterns also emerge in DINO, so they are not unique to iBOT’s masked-image modeling.

G.2 SELF-ATTENTION VISUALIZATIONS

The paper provides additional visualizations of self-attention maps from multiple heads in the last layer, as shown in Figure 18.

  • Additional self-attention maps are visualized across multiple heads of the last layer.These visualizations extend the analysis presented in Figure 18.
  • The visualizations use the same setting as Section 4.3.2.
  • Figure 18 is the referenced figure for the additional self-attention-map analysis.

G.3 SPARSE CORRESPONDENCE.

The paper evaluates iBOT through sparse correspondence and visualizes the resulting semantic and attention patterns. The examples show mostly correct matching across augmented views and across same-class images despite substantial visual differences.

  • Sparse correspondence: The sparse correspondence task matches overlapping patches from augmented views or patches from two images labeled as one class.With ViT-S/16, at most 14 × 14 matched pairs can be extracted, making the correspondence sparse.
  • Sparse correspondence: iBOT empirically matches most correspondences correctly between two augmented views of the same image.The evaluation uses a ViT-S/16 model pretrained for 800 epochs, with scores averaged across multiple heads of the last layer.
  • Sparse correspondence: iBOT matches corresponding parts across same-class images despite large differences in texture, color, size, and position.Examples include matching tiles and windows between two cars and semantic parts across animal images.
  • Visual patterns: Patch-token patterns capture high-level semantic parts such as human hair, shoulders and arms, elbows, animal ears, noses, and wings.
  • Visual patterns: Other patch-token patterns group low-level details, including shared textures and object shapes.Examples include dotted and striped textures across different animal parts.
  • Visual patterns: Compared with BEiT and DINO layouts, iBOT displays stronger semantic patterns rather than primarily clustering color, texture, or local details.The comparison is visualized using layouts extracted from the DALL-E encoder, BEiT, and DINO.
Loading 2111.07832v3…