Source-linked AI summary
Contrastive Masked Autoencoders are Stronger Vision Learners
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, Jiashi Feng
TL;DR
MIM learns holistic and local image representations but has limited instance discriminability, motivating a stronger vision learner. CMAE unifies MIM and contrastive learning with an online encoder-decoder, momentum encoder, feature decoder, and pixel shifting. It improves representation quality across classification, segmentation, and detection, including 85.3% ImageNet top-1 accuracy and 52.5% ADE20K mIoU.
Problem
MIM emphasizes local image relations and can learn less discriminative representations, while simply combining it with contrastive learning has produced only marginal gains.
Method
CMAE unifies masked reconstruction and contrastive learning using an online asymmetric encoder-decoder, momentum encoder, feature decoder, and pixel-shifting views.
Results
CMAE achieves state-of-the-art performance across image classification, semantic segmentation, and object detection, with reported gains over prior methods.
Takeaways & Limitations
CMAE representations preserve local context-sensitive features while modeling instance discriminativeness among different images.
Takeaways & Limitations
Randomly cropped paired views can adversely affect performance when their regions are far apart or scarcely semantically relevant.
Abstract
from arXiv · showhide
Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{https://github.com/ZhichengHuang/CMAE}.
1 INTRODUCTION
CMAE unifies masked image modeling and contrastive learning to retain local perceptibility while improving instance discriminability. Its feature decoder and weak spatial-shifting augmentation make the combination effective, yielding stronger transfer performance.
- MIM learns rich holistic representations through masked-patch reconstruction but emphasizes local image relations rather than discriminating among images.
- Prior attempts to combine contrastive learning with MIM achieved only marginal gains, making it non-trivial to exploit both frameworks together.
- CMAE uses an online asymmetric encoder-decoder for masked-image reconstruction and a momentum encoder that supplies contrastive-learning supervision.
- A feature decoder complements masked features for contrastive alignment, while weak spatial shifting generates plausible positive views compatible with MIM.
- 0.7% ImageNet-1K classification, 1.8% ADE20K mIoU, and 0.4% APb and 0.5% APm on COCO are CMAE's reported absolute gains over prior methods.
2 RELATED WORK
Contrastive learning provides instance-discriminative representations but often lacks spatial sensitivity, whereas MIM captures token relationships but can learn less discriminative features. CMAE addresses this complementarity with shifted views, masked reconstruction, feature prediction, and momentum encoding.
- Self-supervised vision methods broadly use contrastive learning or masked image modeling as their pretext task.
- Contrastive learning distinguishes images through instance-level representation comparisons but commonly focuses on global representations and lacks spatial-sensitive features.
- MIM reconstructs masked image content and models relationships among tokens, but its learned representations can be less discriminative across different images.
- Related methods use offline, online, or multimodal targets and siamese or prototype-based objectives to guide masked-image representation learning.
- CMAE combines masked reconstruction with spatially shifted views, an auxiliary feature decoder, and momentum-encoder embeddings for contrastive learning.
3 METHOD
CMAE unifies masked image modeling and contrastive learning through online and momentum branches, using pixel shifting and feature reconstruction to align their inputs and representations.
- Framework: CMAE uses an online asymmetric encoder-decoder for masked-image reconstruction and a momentum encoder for contrastive supervision.The online branch learns from visible patches, while the momentum branch processes the full image to preserve semantic integrity.
- Momentum encoder: The momentum encoder is updated by exponential moving average with µ fixed at 0.996, stabilizing feature changes during training.After pre-training, the online encoder extracts representations for downstream tasks.
- Decoder design: The online decoder contains separate pixel and feature decoders for reconstructing masked pixels and masked-token features.The feature decoder mean-pools recovered features into an image representation used for contrastive learning.
- View augmentations: Randomly separated crops can make positive pairs semantically misaligned, causing contrastive noise and hampering discriminative representation learning.Large-scale masking can amplify the input disparity between the online and momentum branches.
- View augmentations: Pixel shifting creates two nearby views from a shared master image, while masking remains exclusive to the online branch and color augmentation to the momentum branch.This produces distinct views while reducing the disparity between masked online inputs and full momentum inputs.
- Training objective: CMAE uses InfoNCE contrastive loss with a projection-prediction head to pull together same-image views and separate different-image samples.The feature-decoder representation is transformed before cosine similarity is computed against momentum-encoder features.
4 EXPERIMENTS
Experiments show that CMAE improves MIM representations across ImageNet classification, semantic segmentation, object detection, and transfer settings. Ablations attribute gains to pixel shifting, feature decoding, and the unified contrastive-MIM design.
- ImageNet classification: 84.7% top-1 accuracy on ImageNet is 1.1% higher than MAE, while CMAE also surpasses MoCo-v3 and DINO by 1.5% and 1.9%.
- ImageNet classification: 85.0% and 85.3% accuracy are achieved with hybrid convolutional ViT after 800 and 1600 pre-training epochs, exceeding ConvMAE by 0.4% and 0.7%.
- Transfer learning: 52.5 mIoU on ADE20K surpasses MAE by 2.9%, iBOT by 1.0%, CAE by 0.8%, and ConvMAE by 1.8%.
- Transfer learning: CMAE improves COCO results over MAE from 51.7 to 52.4 APb and 45.9 to 46.5 APm, and over ConvMAE from 52.5 to 52.9 APb and 46.5 to 47.0 APm.
- Method analysis: Pixel shifting raises accuracy from 83.1% to 83.6%, feature decoding raises it to 83.8%, and pixel shifting outperforms random cropping at 83.4% versus 83.0%.
- Transferability and scalability: CMAE remains better than MAE under partial fine-tuning, gains 5.9% in linear probing, scales across encoder sizes, and converges faster during fine-tuning.With one block fine-tuned, CMAE gains 2.5% over MAE and 3.8% over MoCo-v3; it surpasses MAE after 55 fine-tuning epochs.
5 CONCLUSION
The paper introduces CMAE, a self-supervised framework that unifies masked image modeling and contrastive learning through input-generation and architectural designs. Experiments show improved pre-trained representations and state-of-the-art results on classification, segmentation, and detection.
- CMAE improves MIM representation quality by combining contrastive learning with masked image modeling in a unified self-supervised framework.
- Its two main designs harmonize the objectives through input generation and architecture.
- CMAE achieves state-of-the-art performance on image classification, semantic segmentation, and object detection.