Source-linked AI summary
Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
Alexandre Eymaël, Renaud Vandeghen, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, Marc Van Droogenbroeck
TL;DR
Video-based SiamMAE learns object-centric representations but requires video data and extensive training. CropMAE instead reconstructs heavily masked crops from paired views of the same image, achieving competitive propagation performance, faster learning, and 98.5% masking without explicit motion.
Problem
SiamMAE is restricted to video frames and requires extensive training on large video datasets, while the role of explicit motion in learning object-centric representations remains unresolved.
Method
CropMAE trains a shared Siamese encoder on differently cropped views of the same image, reconstructing one view after masking over 98.5% of its patches.
Results
CropMAE achieves on-par performance with state-of-the-art methods on three video propagation tasks, learns useful representations without explicit motion, and reaches J &Fm 58.0 after 150 ImageNet-Subset epochs or 250 K400 epochs.
Takeaways & Limitations
Object-centric propagation features can be learned from still images through implicit crop transformations rather than requiring explicit video motion.
Takeaways & Limitations
The scalability of CropMAE across model size, patch size, data quantity, and differences between images and video frames requires further investigation.
Abstract
from arXiv · showhide
Self-supervised pre-training of image encoders is omnipresent in the literature, particularly following the introduction of Masked autoencoders (MAE). Current efforts attempt to learn object-centric representations from motion in videos. In particular, SiamMAE recently introduced a Siamese network, training a shared-weight encoder from two frames of a video with a high asymmetric masking ratio (95%). In this work, we propose CropMAE, an alternative approach to the Siamese pre-training introduced by SiamMAE. Our method specifically differs by exclusively considering pairs of cropped images sourced from the same image but cropped differently, deviating from the conventional pairs of frames extracted from a video. CropMAE therefore alleviates the need for video datasets, while maintaining competitive performances and drastically reducing pre-training and learning time. Furthermore, we demonstrate that CropMAE learns similar object-centric representations without explicit motion, showing that current self-supervised learning methods do not learn such representations from explicit object motion, but rather thanks to the implicit image transformations that occur between the two views. Finally, CropMAE achieves the highest masking ratio to date (98.5%), enabling the reconstruction of images using only two visible patches. Our code is available at https://github.com/alexandre-eymael/CropMAE.
1 Introduction
CropMAE addresses the cost and scope limits of video-based SiamMAE by using differently cropped views of the same image. It retains competitive downstream performance while accelerating training and reaching a 98.5% masking ratio with only two visible patches.
- Background: Masked image modeling can require extensive data and training because pixel reconstruction demands learning the data distribution, especially with ViTs lacking visual inductive biases.Contrastive learning has a different weakness: it depends on carefully chosen transformations.
- Motivation and contribution: SiamMAE is limited to video frames and requires 2,000 training epochs on large datasets such as K400 to reach state-of-the-art performance.Image datasets are typically larger and cheaper to decode than video datasets, motivating an image-based alternative.
- Motivation and contribution: CropMAE replaces video-frame pairs with differently cropped views of one image, enabling image- or video-based pre-training while retaining competitive downstream performance.The method simulates viewpoint changes, object transformations, motion, and occlusions through random views, avoiding a dependence on video datasets.
- Main contributions: 98.5% masking uses only two visible patches for a ViT/16, exceeding masking ratios explored in existing studies while still producing a meaningful self-supervised task.The authors identify this as an optimal and challenging masking setting for CropMAE.
2 Related Work
The related work situates CropMAE among visual representation learning, contrastive learning, masked image modeling, and Siamese masked autoencoders. These approaches differ in their objectives, augmentations, masking procedures, and reliance on video frames.
- Visual representation learning: Visual self-supervised learning learns generalizable image or video representations through pretext tasks for downstream tasks including classification, detection, and video segmentation.CropMAE is positioned as a visual self-supervised representation method for propagation tasks.
- Contrastive self-supervised learning: Contrastive learning makes augmented views of the same image similar and supports robust correspondence representations, but CropMAE relies less on augmentations and avoids representation-collapse issues.The approach has been applied to both images and videos.
- Masked image modeling: Masked image modeling divides images into patches, masks many of them, and reconstructs the missing content with a denoising autoencoder whose decoder is discarded after training.The encoder is retained as the feature extractor after pre-training.
- Siamese masked autoencoders: SiamMAE extends masked autoencoders to video propagation by processing pairs or groups of frames with a Siamese encoder and asymmetric masking.The initial frame is left unmasked as a complete reference while another frame is heavily masked.
3 Method
CropMAE creates two cropped views of an image, heavily masks one view, and reconstructs it using a Siamese ViT encoder and transformer decoder. Its tractable crop-based task supports extremely high masking and faster representation learning.
- Method overview: CropMAE generates two augmented views, patchifies both, heavily masks the second, and reconstructs it with a Siamese ViT encoder and transformer decoder.The decoder is trained with an L2 reconstruction loss and discarded after pre-training, leaving the encoder for downstream feature extraction.
- Cropping: The four cropping strategies are Same Views, Random Views, Local-to-Global, and Global-to-Local, differing in whether views are identical, independent, nested, or inversely nested.The strategies are designed to create distinct spatial relationships between the unmasked and masked views.
- Cropping: Global-to-Local achieved the best performance among the tested cropping strategies.The reported comparison places Global-to-Local above Random Views, Local-to-Global, and Same Views.
- Patch encoding and decoding: CropMAE uses positional embeddings with N × N patches and a [CLS] token, following the original ViT patch representation.The encoder processes both views using the same Siamese architecture.
- Masking: Because the local target is always contained in the global input, CropMAE can reconstruct it without conceptual object knowledge and performs best with only 1 or 2 of 196 patches visible.This corresponds to a masking ratio between 98% and 99%, with 98.5% reducing visible patches by a factor of 4.5 relative to 95%.
4 Experiments
CropMAE is evaluated against prior methods and SiamMAE across propagation tasks, reconstruction quality, attention maps, learning speed, training time, and component ablations. Under a fixed 400-epoch budget, it improves DAVIS-2017 performance, learns object boundaries without explicit motion, and trains substantially faster, while performance varies by downstream task and configuration.
- 4.2 Results: +2.5% on DAVIS-2017 versus SiamMAE when trained on the ImageNet Subset, while CropMAE also improves by +0.7% on K400.On VIP, the gains are smaller (+0.1 on ImageNet and +0.5 on K400); on JHMDB, CropMAE only outperforms VideoMAE.
- 4.2 Results: At a 98.5% masking ratio, reconstructions are produced from ImageNet inputs using a ViT-S/16 and only the visible information in the masked view.Figure 3 orders the visualized stages as input image, random resized crop, masked image, and reconstruction.
- 4.3 Attention Maps: CropMAE learns object boundaries without explicit motion, indicating that transformations and deformations between two views can support object-centric representations.Self-attention maps from an unsupervised [CLS] token head show boundary-focused representations after training on the ImageNet Subset.
- 4.4 Learning Speed: 58.0 J &Fm is reached after 150 ImageNet-Subset epochs or 250 K400 epochs, whereas SiamMAE requires 350 epochs.The comparison is reported for DAVIS-2017 during pretraining.
- 4.5 Training time: 2380% speedup is achieved on the ImageNet Subset, while CropMAE is almost 30% faster than SiamMAE on K400.The speed advantage is attributed to using fewer visible patches and frames; the comparison covers the whole 400-epoch training process.
- Ablation Study: Ablations find that CropMAE works best with a small decoder depth, a small decoder embedding dimension, and random horizontal flips applied to both crops.The ablations use 400 ImageNet-Subset epochs and evaluate DAVIS-2017 performance.
5 Conclusion
CropMAE reconstructs heavily masked image crops to learn useful video-propagation features without explicit video motion. It trains faster than existing methods and achieves on-par performance with state-of-the-art methods across three video propagation tasks.
- CropMAE achieves on-par performance with state-of-the-art methods on three video propagation downstream tasks.
- CropMAE learns useful video-task features from still images without requiring explicit video motion.The method is also applicable to video frames and still images.
- Over 98.5% masking enables CropMAE to reconstruct image crops while exposing only a very small visible portion.
- CropMAE trains faster than existing methods because its pretext task is tractable.
6 Appendix
The appendix reports 2,000-epoch comparisons showing CropMAE outperforming SiamMAE while training on either K400 video frames or ImageNet images. CropMAE reaches strong propagation scores earlier, although none of the methods scales well to very long pre-training.
- CropMAE trains faster and consistently outperforms SiamMAE during 2,000-epoch DAVIS pre-training on both K400 frames and ImageNet Subset images.
- The appendix documents experiments at 2,000 epochs and compares CropMAE configurations trained on K400 and ImageNet with SiamMAE trained on K400.
- At 100 epochs, CropMAE reaches J &Fm 56.5 on ImageNet Subset, versus SiamMAE's 52.0.
- At 400 epochs, CropMAE reaches J &Fm 60.0 with both video and image training, while SiamMAE reaches 58.9.
- SiamMAE's later peak of 59.4 remains below CropMAE's performance.
- None of the three training methods scales well with very long pre-training.