Source-linked AI summary
SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders
Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, Changwen Zheng
TL;DR
MAE still lacks the semantic decomposition available in language, motivating a visual analogue of words based on semantic parts. SemMAE learns semantic parts and uses them to guide masking from intra-part patterns toward whole-part relations, achieving improvements across vision tasks, including 84.5% ImageNet fine-tuning accuracy.
Problem
MAE lacks the semantic decomposition available in language because obtaining semantic parts in images is nontrivial.
Method
SemMAE learns semantic parts with a self-supervised ViT-based model and progressively masks patches within parts or whole parts during MAE training.
Results
SemMAE improves performance across classification and transfer tasks, achieving 84.5% top-1 ImageNet fine-tuning accuracy and surpassing MAE by 0.7%.
Takeaways & Limitations
Semantic-guided masking can integrate semantic information into MAE pre-training and facilitate learning image representations.
Takeaways & Limitations
The learned semantic parts are coarse because part-segmentation labels are unavailable, and small patches increase fine-tuning computation.
Abstract
from arXiv · showhide
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e., from intra-part patterns to inter-part relations. In particular, we achieve this in two steps. 1) Semantic part learning: we design a self-supervised part learning method to obtain semantic parts by leveraging and refining the multi-head attention of a ViT-based encoder. 2) Semantic-guided MAE (SemMAE) training: we design a masking strategy that varies from masking a portion of patches in each part to masking a portion of (whole) parts in an image. Extensive experiments on various vision tasks show that SemMAE can learn better image representation by integrating semantic information. In particular, SemMAE achieves 84.5% fine-tuning accuracy on ImageNet-1k, which outperforms the vanilla MAE by 1.4%. In the semantic segmentation and fine-grained recognition tasks, SemMAE also brings significant improvements and yields the state-of-the-art performance.
1 Introduction
Masked image modeling lacks the semantic decomposition available in language, motivating semantic parts as a visual analogue of words. SemMAE learns semantic parts and uses them to guide MAE masking.
- Masked image modeling remains different from masked language modeling because images lack an obvious semantic decomposition analogous to words.
- Semantic parts are proposed as a visual analogue of words because objects consist of different parts and part-based representations fit object structure.
- The paper learns semantic parts self-supervisedly with a ViT-based encoder, attention maps, and a StyleGAN-based decoder.
- The optimized attention maps provide spatial information, while the class token supplies texture information through AdaIN in the decoder.
- SemMAE combines semantic part learning with semantic-guided masking to test whether semantic parts facilitate MAE learning.
2 Related work
Related work spans semantic part learning and masked image modeling. Existing part-learning methods often target single-class datasets, while masking strategies vary in their spatial organization and prediction targets.
- Semantic part learning: Part-learning methods mine information using spatial priors, convolutional-channel semantics, contrastive proxy tasks, or few-shot classifiers over pretrained features.
- Semantic part learning: Existing few-shot part-segmentation approaches mainly address fine-grained single-class datasets such as birds, cars, or human faces.
- Semantic part learning: Unsupervised part learning on multi-class datasets such as ImageNet is described as more challenging than prior single-class settings.
- Masked image modeling: Masked image modeling research studies both regression targets and masking strategies, including discrete tokens, HOG features, and raw-pixel regression.
- Masked image modeling: Existing masking strategies include block-wise masking of neighboring tokens and random masking with large masked patches or high masking proportions.
3 Semantic-guided masked autoencoders
SemMAE learns semantic parts from ViT features and uses them to replace fixed random masking with a curriculum from intra-part reconstruction to inter-part reasoning.
- Semantic Part Learning: SemMAE extracts class and patch tokens with an iBOT-pretrained ViT, embeds the class token into part tokens, and correlates part tokens with patch tokens to produce attention maps.The attention maps represent the likelihood of each semantic part across image positions.
- Semantic Part Learning: A StyleGAN-based decoder reconstructs the original image from attention maps while injecting class-token texture through AdaIN.The reconstruction objective encourages attention maps to encode spatial information; a diversity constraint encourages different maps to differ.
- Semantic-Guided Masking: Argmax over attention maps assigns each patch to a semantic part, providing the segmentation used to generate semantic-guided masks.Patches assigned to the same part form the visual analogue of a word for masking purposes.
- Semantic-Guided Masking: Early training masks a portion of patches within every part so the model learns intra-part patterns from visible patches in the same part.The masking schedule then progresses toward masking complete parts and predicting patches in the remaining parts to learn inter-part relations.
- Semantic-Guided Masking: An interpolation parameter α increases during training, shifting masking from partial patches in each part toward randomly selected whole parts while maintaining the target mask ratio.Algorithm 1 computes both masking settings and interpolates between their per-part masked-patch counts.
4 Experiments
Experiments show that semantic information improves MAE representation learning, with benefits from precise parts and adaptive masking extending across ImageNet classification and fine-grained recognition.
- Semantic-guided MAE: 1.3% and 1.9% linear-probing accuracy gains show that integrating semantic information improves MAE with 16×16 and 8×8 patches, respectively.The experiments compare settings with different patch sizes.
- Semantic-guided MAE: 8×8 patches improve SemMAE because larger patches produce imprecise part segments, while removing 3/4 background patches avoids increased pre-training and probing cost.The smaller patch size is adopted in subsequent SemMAE experiments.
- Semantic-guided MAE: Masking 75% of parts causes a 13.9% performance drop versus random masking, whereas gradually shifting from masking 75% of patches per part to 75% of parts performs best with γ = 2.The two stages encourage intra-part patterns and inter-part relations, respectively.
- Semantic part learning: The proposed semantic part learner produces more complete, less noisy part maps than iBOT and generates parts that benefit MAE representation learning.The evaluation uses qualitative comparisons and downstream ImageNet classification because part segmentation ground truth is unavailable.
- Compared with other methods on ImageNet: 84.5% top-1 ImageNet fine-tuning accuracy lets SemMAE outperform SimMIM by 0.9% and MAE by 0.7%, while linear probing exceeds competitive MIM methods by 0.8%.SemMAE also surpasses previous contrastive-learning methods for fine-tuning.
- Downstream tasks: SemMAE surpasses MAE by 0.3%, 0.6%, and 0.2% on iNaturalists, CUB-Bird, and Stanford-Cars, respectively, demonstrating transfer ability for fine-grained classification.The experiments evaluate transfer learning on fine-grained datasets.
5 Conclusion
SemMAE uses semantic parts to guide masked autoencoder pre-training, integrating semantic information beyond random masking. The approach improves image representation, while coarse parts and fine-tuning cost remain limitations.
- SemMAE studies semantic parts as a visual analogue of words and integrates them into masked autoencoder pre-training through semantic-guided masking.The strategy is presented as an alternative to mainstream random masking.
- The learned semantic parts facilitate MAE learning and extensive experiments report superior results for SemMAE.
- Semantic parts remain coarse because part segmentation labels are unavailable, with approximately 6 parts per image.The paper states that this is not yet an ideal visual analogue of words.
- Using a small patch size increases computational cost during fine-tuning.
Checklist
The checklist records the paper’s compliance statements, including documentation of limitations, experimental details, resources, and asset licensing. It also records that error bars were not reported.
- The paper states that licenses and creator citations for existing assets are addressed, while another checklist entry records proprietary code and data.
- The paper states that its main claims accurately reflect its contributions and scope and that its limitations are described in Section 5.
- The checklist states that code, data, reproduction instructions, training details, and compute-resource information are provided, while error bars are not reported.