PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing the Dark Secrets of Masked Image Modeling

Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, Yue Cao

arXiv:2205.13543v2cs.CVcs.AIcs.LG

TL;DR

MIM's effectiveness across vision tasks leaves its mechanisms and transfer boundaries unclear. The paper compares MIM with supervised pre-training through visualizations and large-scale experiments, finding distinctive locality and attention-head diversity alongside stronger transfer on geometric, motion, and fine-grained tasks.

  • Problem

    The paper investigates MIM's key mechanisms and how its transferability compares with supervised pre-training across semantic, geometric, and motion tasks.

  • Method

    The paper compares MIM and supervised models through representational visualizations and fine-tuning experiments across semantic understanding, geometric and motion, and combined tasks.

  • Results

    MIM performs significantly better on weak-semantic geometric and motion tasks and fine-grained classification, while remaining highly competitive where supervised transfer is strong.

  • Takeaways & Limitations

    MIM's locality and sustained attention-head diversity help explain its effectiveness for Vision Transformers and its state-of-the-art results on geometric and motion benchmarks.

Abstract

from arXiv · show

Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction.

1 Introduction

The paper asks how MIM differs from supervised pre-training and where each transfers best. Through visualizations and broad fine-tuning experiments, it links MIM's representational properties to stronger performance on geometric, motion, and fine-grained tasks.

  • Representational analysis: MIM brings locality inductive bias across trained layers, whereas supervised models shift from local attention in lower layers to global attention in higher layers.The locality strength in MIM correlates with masking ratio and masked patch size.
  • Representational analysis: MIM preserves diverse attention heads across layers, while supervised models lose diversity in their last three layers, harming downstream fine-tuning.Removing the last several supervised-model layers improves fine-tuning, unlike for MIM models.
  • Representational analysis: MIM models show highly similar representations across layers, with CKA values around [0.9, 1.0], whereas supervised models vary more, around [0.5,1.0].Shuffling pre-trained layers hurts supervised models more than MIM models.
  • Transfer experiments: MIM substantially outperforms supervised models on fine-grained or differently categorized datasets, while supervised models remain better on categories sufficiently covered by ImageNet.On tasks where supervised transfer is strong, MIM remains highly competitive.
  • Transfer experiments: MIM models outperform supervised counterparts by large margins on geometric and motion tasks requiring weaker semantics and high-resolution localization.These include pose estimation, depth estimation, and video object tracking.
  • Transfer experiments: 78.9 AP on COCO test-dev, 78.0 AP on CrowdPose, 0.287 RMSE on NYUv2, 1.966 RMSE on KITTI, and 70.7 SUC on LaSOT are reported for MIM-pre-trained Swin-L.The results are described as state of the art without bells and whistles.

2 Background

The paper presents MIM as masked pixel prediction and studies it mainly with Transformer backbones. SimMIM provides a simple, broadly applicable framework for these analyses and experiments.

  • Masked Image Modeling: MIM masks part of an input image and predicts the masked signals from the visible ones.The paper uses SimMIM as its example framework because it is simple, effective, and generally applicable.
  • Masked Image Modeling: SimMIM uses random masking, jointly feeds masked and visible tokens to the encoder, applies a linear prediction head, and directly regresses RGB pixels with an L1 loss.Its masked patch size is moderately large, such as 32.
  • Backbone Architectures: The framework can be applied to Vision Transformers, Swin Transformers, and ConvNets.The paper uses ViT mainly for visualizations and Swin mainly for downstream experiments.

3 Visualizations

The visualizations compare MIM with supervised pre-training through attention locality, attention-head diversity, and cross-layer representation similarity. MIM preserves local, diverse, and highly similar representations across layers, whereas supervised models become more globally focused and less diverse in deeper layers.

  • 3.1.1 Local Attention or Global Attention?: MIM brings locality inductive bias across layers, while supervised models shift from local attention in lower layers to global attention in higher layers.In SimMIM, attention distance remains similar to supervised lower layers and becomes slightly smaller with depth.
  • 3.1.1 Local Attention or Global Attention?: Moderate AvgDist values of [10, 20] coincide with smaller averaged attention distances and higher fine-tuning accuracy.AvgDist measures the average distance from masked pixels to their nearest visible pixels; smaller or larger values perform worse.
  • 3.1.2 Focused Attention or Broad Attention?: MIM maintains diverse attention entropy across heads in every layer, with some heads focused and others attending broadly.Supervised models instead become broadly attentive in higher layers after focused lower-layer heads.
  • 3.1.3 Diversity on Attention Heads: Attention-head diversity remains high across MIM layers but diminishes with depth in supervised and contrastive models, nearly disappearing in their last three layers.The comparison uses KL divergence between attention distributions of different heads.
  • 3.1.3 Diversity on Attention Heads: Dropping two to eight final layers improves supervised COCO pose performance and leaves NYUv2 depth performance unchanged.The result links reduced model depth with the harmful effect of low diversity in supervised models’ final layers.
  • 3.2 Investigating the Representation Structures via CKA similarity: MIM feature representations are highly similar across layers, with CKA values around [0.9, 1.0], whereas supervised representations vary from [0.5,1.0].MoCo v3 behaves similarly to SimMIM for cross-layer CKA similarity.
  • 3.2 Investigating the Representation Structures via CKA similarity: Randomly shuffling layers reduces MIM pose performance from 75.5 to 75.2 (-0.3) and depth performance from 0.382 to 0.434 (-0.052).Supervised pre-trained models suffer larger drops under the same layer-shuffling test.

4 Experimental Analysis on Three Types of Downstream Tasks

The paper compares MIM and supervised pre-training across semantic, geometric, motion, and combined downstream tasks. MIM is strongest on fine-grained or weak-semantic tasks, while supervised pre-training remains competitive when categories overlap with ImageNet.

  • Semantic understanding tasks: MIM remarkably outperforms supervised counterparts on fine-grained classification and datasets with output categories differing from ImageNet.Supervised models perform better when target categories are sufficiently covered by ImageNet.
  • Geometric and motion tasks: MIM pre-training surpasses supervised counterparts by 2.2 AP on COCO test-dev and 4.2 AP on CrowdPose for pose estimation.With stronger detection results, MIM pre-trained SwinV2-L reaches 78.9 AP on COCO test-dev and 78.0 AP on CrowdPose.
  • Geometric and motion tasks: 0.304 RMSE on NYUv2 and 2.050 RMSE on KITTI are achieved by MIM-pre-trained SwinV2-B using a simple deconvolution head.These results outperform the previous state-of-the-art method and improve SwinV2-B over supervised ImageNet-22K pre-training by 0.03 RMSE.
  • Geometric and motion tasks: 70.7 SUC on LaSOT is achieved by SwinTrack with an MIM-pre-trained SwinV2-L backbone.The model uses 224 × 224 input images and a 112 × 112 template.
  • Combined tasks: 52.9/46.7 versus 51.9/45.7 box/mask AP shows MIM outperforming supervised pre-training on COCO object detection.MIM helps localization converge faster and better, whereas supervised pre-training benefits object classification more.

5 Related Work

Prior work established supervised ImageNet pre-training and examined transferability and internal representations, while self-supervised methods increasingly matched supervised performance on representative tasks.

  • Visual Pre-training: Supervised ImageNet classification has long provided strong fine-tuning performance across numerous semantic understanding tasks.Examples include classification, detection, segmentation, and video action recognition.
  • Understanding Pre-training: Earlier studies analyzed layer transferability, ImageNet model transfer, and Vision Transformer behavior using tools including CKA, loss landscapes, and Fourier analysis.Related investigations also examined pre-training in NLP after BERT.

6 Conclusion

The paper analyzes where MIM works well and identifies representational and transfer differences from supervised pre-training. Its findings emphasize locality, attention-head diversity, and strong performance on geometric and motion tasks.

  • Conclusion: MIM brings locality and attention-head diversity to trained models, helping Vision Transformers with large receptive fields optimize.The paper presents these findings as key visualization-based explanations for MIM behavior.
  • Conclusion: MIM performs very well on geometric and motion tasks with weak semantics and reaches state-of-the-art performance without bells and whistles.The conclusion frames this as the central experimental finding.
  • Conclusion: Most visualization observations transfer across ViT-B, SwinV2-B, and RepLKNet architectures.The cross-architecture checks use SimMIM visualizations.

A.1 Visualizations on Attention Maps

Additional visualizations on SwinV2-B show that MIM preserves local and global attention patterns, greater head diversity, and similar representation structures across layers than supervised training.

  • Local Attention or Global Attention?: SimMIM aggregates both local and global pixels across all layers, whereas supervised attention shifts from local focus in lower layers to global focus in higher layers.The SimMIM average attention distance resembles the lower layers of the supervised model.
  • Focused Attention or Broad Attention?: Swin-B visualizations reproduce the broader contrast between focused and broad attention observed with ViT-B.The passage reports a similar observation for the Swin-B backbone.
  • Diversity on Attention Heads: Different SimMIM attention heads aggregate different tokens across all layers, while supervised-head diversity decreases with depth.After supervised fine-tuning, the SimMIM model becomes more similar to a supervised model but retains an advantage from MIM.
  • Representation structures via CKA: CKA compares feature representations through Gram matrices and remains invariant to orthogonal transformation and isotropic scaling.This makes CKA suitable for comparing hidden representations with different dimensions.
  • Representation structures via CKA: SimMIM layers have almost identical representation structures, whereas supervised layers learn different structures across depth.The SwinV2-B CKA heatmaps reproduce the corresponding ViT-B observation.

B.1 Experimental Results

MIM pre-training is evaluated with RepLKNet-31B on ImageNet-1K classification and pose estimation, using the reported comparison setup and training configurations. It reaches parity with re-parameterized supervised pre-training on ImageNet-1K while retaining a substantial pose-estimation advantage.

  • RepLKNet-31B MIM pre-training uses 192 × 192 images, 6 × 6 patches, and 60% random masking before reconstruction with an ℓ1 loss.
  • MIM pre-training uses ImageNet-1k data with random resized cropping and flipping, AdamW, a 4e-4 base learning rate, and 300 total epochs.
  • Fine-tuning on ImageNet-1k uses AdamW, layer decay of 0.8, cosine scheduling, 100 total epochs, and batch size 2048.
  • MIM pre-training helps RepLKNet-31B address its optimization issue and achieve on-par ImageNet-1K performance with supervised training using the re-parametrization trick.The comparison concerns large-kernel convolutional models.
  • MIM models surpass supervised counterparts with the re-parametrization trick by large margins on pose estimation.The authors report this as evidence that MIM benefits geometric and motion tasks across different backbone architectures.

B.2 Visualizations

Visualizations compare local-versus-global behavior and cross-kernel diversity across supervised and SimMIM RepLKNet-31B models. SimMIM produces diverse kernel behaviors across layers, especially deeper ones, whereas supervised models show stronger layer-dependent patterns.

  • Convolutional-kernel visualizations normalize each depth-wise kernel channel and treat channels as attention heads, enabling attention-map analysis of RepLKNet.
  • Local Kernels or Global Kernels?: SimMIM kernels aggregate both local and global pixels across layers, while supervised RepLKNet-31B models shift from local focus in lower layers toward more global focus in higher layers.The comparison uses aggregated distance across channels and layers.
  • Focused Kernels or Broad Kernels?: SimMIM produces diverse entropy values across kernels in all layers, whereas supervised models show focused lower-layer attention and broader higher-layer attention.Entropy is compared across channels and layer numbers.
  • Diversity across Different Kernels: SimMIM has substantially greater diversity across convolution kernels than supervised counterparts, especially in deeper layers.The comparison is based on averaged KL divergence across layers.

C Detailed Results on Semantic Understanding Tasks

The semantic-understanding evaluation compares MIM and supervised SwinV2-B models on broad classification benchmarks and fine-grained classification settings. The supplied passages identify the benchmark structure and metrics but do not report the numerical outcomes.

  • The detailed comparisons include the fine-grained iNaturalist-18 classification dataset alongside the Kornblith and Concept Generalization benchmarks.
  • Table 4 compares MIM and supervised SwinV2-B models on the Kornblith 12-dataset classification benchmark.The reported metrics are top-1 accuracy and mean per-class accuracy; Pascal Voc 2007 is excluded because its metric is incompatible.

D Comparisons on Combined Task of Semantic Segmentation

On ADE-20K semantic segmentation, the supervised pre-trained model slightly outperforms the MIM counterpart, making the better pre-training choice difficult to predict for combined semantic-geometric tasks.

  • Supervised pre-training slightly outperforms MIM on ADE-20K semantic segmentation.The evaluation uses UperNet and Mask2former frameworks.
  • For combined tasks, the authors state that it may be difficult to predict which pre-trained model will perform better.

E Detailed Settings

The paper evaluates pretrained models across pose, depth, tracking, detection, segmentation, concept generalization, and fine-grained classification using standardized datasets, task-specific heads, and reported metrics. Detailed settings specify datasets, architectures, augmentations, optimization schedules, and evaluation procedures.

  • Concept Generalization: CoG uses five 1k-category ImageNet-22K subsets with increasing semantic gaps, transferring hyperparameters selected on CoG L1 to L2–L5.Top-1 accuracy is reported across the five levels.
  • Fine-grained classification: Fine-grained transfer includes the 12-dataset K12 benchmark and iNaturalist-18, whose training protocol searches hyperparameters before final evaluation.iNaturalist-18 contains 437,513 training images, 24,426 validation images, and more than 8,000 categories.
  • Pose estimation: Pose estimation uses COCO and CrowdPose with a heatmap-based top-down pipeline that upsamples backbone features through deconvolutions.Models are trained with standard dataset splits and evaluated using OKS-based AP.
  • Depth estimation: Depth estimation evaluates NYUv2 and KITTI using RMSE, with a deconvolutional head and maximum ranges of 10m and 80m, respectively.The head contains three deconvolutions with BN and ReLU plus a normal convolution.
  • Video object tracking: Tracking trains SwinTrack models on GOT10k, TrackingNet, LaSOT, and COCO, reporting SUC for TrackingNet and LaSOT and average overlap for GOT10k.Training uses 131072 sampled pairs per epoch for 300 epochs with separate head and backbone learning rates.
  • Detection and segmentation: Object detection and semantic segmentation use Mask-RCNN and UPerNet frameworks, respectively, with AdamW optimization and task-specific training schedules.Detection uses a 3×, 36-epoch schedule, while segmentation trains for 80K iterations at 640 × 640 resolution.
Loading 2205.13543v2…