Source-linked AI summary
Siamese Masked Autoencoders
Agrim Gupta, Jiajun Wu, Jia Deng, Li Fei-Fei
TL;DR
Visual correspondence remains challenging under occlusions, viewpoint changes, and appearance variation. SiamMAE extends masked autoencoders by independently encoding an intact past frame and a heavily masked future frame, then reconstructing missing future patches with cross-attention. It outperforms state-of-the-art self-supervised methods across video segmentation, pose propagation, and semantic part propagation.
Problem
Visual correspondence across scenes is difficult under occlusions, viewpoint changes, and object transformations, yet it supports tracking, segmentation, flow estimation, and 3D reconstruction [2] [4] [6] [7].
Method
SiamMAE independently encodes randomly sampled past and future frames, masks 95% of future patches while leaving the past intact, and predicts the missing patches with cross-attention decoding.
Results
SiamMAE outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation.
Takeaways & Limitations
The method provides a simple correspondence-learning baseline without data augmentation, handcrafted tracking-based pretext tasks, multi-crop training, or extra collapse-prevention techniques.
Takeaways & Limitations
The study operates on pairs of video frames, leaving prediction of multiple future frames and scalability across data and model size for future investigation.
Abstract
from arXiv · showhide
Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. By masking a large fraction ($95\%$) of patches in the future frame while leaving the past frame unchanged, SiamMAE encourages the network to focus on object motion and learn object-centric representations. Despite its conceptual simplicity, features learned via SiamMAE outperform state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks. SiamMAE achieves competitive results without relying on data augmentation, handcrafted tracking-based pretext tasks, or other techniques to prevent representational collapse.
1 Introduction
SiamMAE extends masked autoencoders to learn visual correspondence from videos by predicting heavily masked future frames from intact past frames. It outperforms state-of-the-art self-supervised methods across three correspondence-related tasks without several common training additions.
- Motivation: Visual correspondence remains difficult under occlusions, viewpoint changes, and object transformations, despite supporting applications including tracking, segmentation, flow estimation, and 3D reconstruction [2] [4] [6] [7].
- Motivation: Extending masked visual modeling to video is nontrivial because MAE features specialize in pixel reconstruction and existing video extensions use symmetric masking.
- Method: The method randomly samples two video frames, masks 95% of patches in the future frame, and leaves the past frame unchanged for prediction.The frames are encoded independently, while a cross-attention decoder predicts the missing future patches.
- Method: Asymmetric masking encourages the network to model object motion and propagate patches from the past frame to corresponding future locations.
- Contribution and results: SiamMAE outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation.The method achieves competitive results without data augmentation, handcrafted tracking-based pretext tasks, multi-crop training, or additional collapse-prevention techniques.
2 Related Work
Prior correspondence learning spans optical flow, tracking, predictive learning, contrastive learning, and masked autoencoders. SiamMAE builds on these lines while addressing their reliance on annotations, augmentations, or extra mechanisms.
- Temporal correspondence: Temporal correspondence is studied through optical flow, motion estimation, and object-level tracking, but fine-grained methods often rely on costly human-annotated or synthetic pixel-level ground truth [43].
- Temporal correspondence: Predictive learning has been used to learn correspondence from future-frame color prediction, although such methods have trailed contrastive approaches [10] [11] [12].
- Self-supervised visual representation learning: Contrastive self-supervised methods dominate correspondence learning, but depend on carefully selected augmentations and additional components to prevent representational collapse [18] [19] [20] [21].
- Masked autoencoders: Masked autoencoders learn representations by reconstructing masked inputs and have been extended from images to videos [24] [25] [26] [27] [28].
3 Method
SiamMAE samples two frames, independently encodes them, asymmetrically masks only the future frame, and uses cross-attention decoding to reconstruct its missing patches. These choices make the pretext task emphasize temporal correspondence and object motion.
- Patchify: SiamMAE samples two frames, patchifies them, adds spatial position embeddings, and omits temporal position embeddings.The frame gap is randomly selected from a predetermined range.
- Masking: Masking 0% of the past frame and 95% of the future frame forces the model to propagate past-frame patches to future locations and encourages object-motion modeling.Large temporal gaps make prediction harder and can produce multiple plausible futures.
- Encoder: A siamese encoder processes the two frames independently, matching the independent-frame testing setup and avoiding the poorer performance of joint encoders.
- Decoder: A joint decoder concatenates tokens from both frames but substantially increases GPU memory requirements, especially with smaller patches.
- Decoder: The cross-self decoder lets future-frame tokens attend to past-frame tokens and then to one another, functioning similarly to an affinity matrix used for correspondence learning.
- Decoder: The decoder predicts normalized pixel values for masked patches using an l2 reconstruction loss.
4 Experiments
SiamMAE is evaluated on three dense correspondence tasks against prior methods and through ablations of its architecture, masking, sampling, augmentation, and training choices. It consistently benefits from asymmetric masking, siamese encoding with cross-self decoding, smaller patches, longer training, and wider frame gaps.
- Comparison with Prior Work: SiamMAE outperforms prior contrastive and self-supervised correspondence methods across video object segmentation, semantic part propagation, and pose keypoint propagation.On DAVIS, ViT-S/8 outperforms all prior approaches; on VIP and pose propagation, SiamMAE likewise achieves the strongest reported performance.
- Comparison with Prior Work: 62.0% versus 39.3% for VideoMAE on video object segmentation, while ViT-S/8 gains +9.4% over the corresponding SiamMAE model.The results also exceed larger MAE-ST models and show that smaller patches improve segmentation performance.
- Comparison with Prior Work: 45.9 mIoU versus 39.5 for DINO on semantic part propagation, with SiamMAE gaining +8.6 mIoU from smaller patches compared with DINO’s +3.3 mIoU.VIP contains videos up to 120 seconds, making it especially challenging relative to the other evaluated datasets.
- Ablation Studies: The best architecture combines a siamese encoder with a cross-self decoder, while joint encoders perform worse because testing processes frames independently.The cross-attention operation resembles affinity-based correspondence learning and label propagation, aligning the decoder with the evaluation protocol.
- Qualitative Analysis: SiamMAE self-attention maps attend to object boundaries despite having no explicit loss on the [CLS] token.The paper attributes this emergent boundary sensitivity to asymmetric masking, which encourages learning object boundaries from object motion.
5 Conclusion
SiamMAE learns video correspondence by asymmetrically masking most future-frame patches while keeping the past frame unchanged, and its representations outperform state-of-the-art self-supervised methods across several tasks. The study is limited to frame pairs and leaves multi-frame prediction, scalability, and video-domain effects for future work.
- SiamMAE outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation.Its competitive results require neither data augmentation, handcrafted tracking-based pretext tasks, nor other anti-collapse techniques.
- Asymmetric masking of most future-frame patches while keeping the past frame unchanged effectively learns correspondence and object boundaries.The approach predicts a majority fraction of the future frame, encouraging representations to model object boundaries.
- The study operates on pairs of video frames, leaving prediction of multiple future frames and systematic scaling with data and model size for future investigation.
- The impact of pre-training video type remains open, including differences between egocentric and in-the-wild internet videos.
A Implementation Details
The experiments follow the original MAE implementation and prior evaluation methodology, with reported training and evaluation hyperparameters summarized in Table 4.
- Training follows the original MAE settings and open-source implementation, with deviations specified in Table 4a.ViT-S/16 models use 4 Nvidia Titan RTX GPUs, while ViT-S/8 models use 8.
- Evaluation follows prior work, while recent self-supervised methods receive a comprehensive grid search over the hyperparameters listed in Table 4b.The reported results for these methods are their optimal values from that search.
- Table 4 reports the training and evaluation hyperparameters used in the experiments.