Source-linked AI summary
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Zhan Tong, Yibing Song, Jue Wang, Limin Wang
TL;DR
VideoMAE targets the difficulty of pre-training video transformers effectively on relatively small datasets without relying on extra data. It uses extremely high-ratio tube masking in a masked autoencoder, making reconstruction more challenging while addressing temporal information leakage. The method achieves favorable performance with only thousands of videos and reports strong results across multiple video benchmarks, while remaining limited to RGB video and leaving larger data, models, resolutions, audio, and text for future work.
Problem
Training effective vanilla video transformers is difficult because video datasets are relatively small and prior models rely heavily on large-scale image pre-training.
Method
VideoMAE uses masked video autoencoding with extremely high-ratio tube masking and an asymmetric encoder-decoder architecture to make reconstruction more challenging and reduce information leakage.
Results
VideoMAE is data-efficient, works with only thousands of video clips, and vanilla ViT models significantly outperform models trained from scratch or with contrastive learning.
Takeaways & Limitations
VideoMAE provides a practical self-supervised video pre-training approach for scenarios with limited data available.
Takeaways & Limitations
VideoMAE uses only the RGB video stream and does not incorporate additional audio or text streams.
Abstract
from arXiv · showhide
Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables a higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets is an important issue. Notably, our VideoMAE with the vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2, 91.3% on UCF101, and 62.6% on HMDB51, without using any extra data. Code is available at https://github.com/MCG-NJU/VideoMAE.
1 Introduction
VideoMAE addresses the difficulty of training vanilla video transformers on limited video data by adapting masked autoencoding to videos. Its high-ratio tube masking makes reconstruction harder, reduces leakage from temporal redundancy, and supports effective self-supervised pre-training.
- Motivation: Training video transformers from scratch remains difficult because video datasets are relatively small and prior models depend heavily on large-scale image pre-training.The paper frames effective and efficient training of vanilla vision transformers using video data alone as an open challenge.
- Method: VideoMAE adapts masked autoencoding by masking random video cubes and reconstructing the missing ones with an asymmetric encoder-decoder architecture.The method is designed specifically for self-supervised video pre-training.
- Method: 90% to 95% masking remains favorable for VideoMAE, using video temporal redundancy to make reconstruction more challenging and meaningful.The high masking ratio also helps address information leakage caused by corresponding content in adjacent frames.
- Results: VideoMAE successfully trains vanilla ViT backbones on relatively small datasets including Something-Something, UCF101, and HMDB51 without extra data.The paper reports that these models significantly outperform previous state-of-the-art results under the same no-extra-data setting.
- Results: VideoMAE models significantly outperform models trained from scratch or pre-trained with contrastive learning methods.This supports masked reconstruction as a solution for self-supervised video pre-training.
- Findings: VideoMAE is reported as data-efficient, achieving successful training with only 3.5k videos and showing that data quality matters more than quantity under domain shift.These findings are presented as important observations about masked modeling and self-supervised video pre-training.
2 Related Work
Prior video representation learning uses supervised image backbones, direct supervised video training, temporal pretext tasks, contrastive learning, and autoencoding. VideoMAE extends masked visual modeling from images to self-supervised video pre-training with video-specific implementation choices.
- Video representation learning: Supervised video representation methods commonly initialize video encoders from image-pre-trained backbones, while some methods train video backbones directly from videos.The related work contrasts image-based initialization with direct supervised video training.
- Self-supervised learning: Self-supervised video methods exploit temporal information through pretext tasks, while contrastive methods rely heavily on strong augmentation and large batch sizes.Pixel-space video prediction has also been explored with CNN and LSTM backbones.
- Masked visual modeling: Masked visual modeling learns representations through masking and reconstruction, with prior work spanning image and video token prediction and asymmetric encoder-decoder architectures.VideoMAE is specifically inspired by ImageMAE and introduces video-specific implementation designs for self-supervised video pre-training.
3 Proposed Method
VideoMAE adapts masked autoencoding to video by addressing temporal redundancy and correlation with downsampling, tube masking, and a vanilla ViT encoder. Its design uses extremely high masking ratios to make reconstruction harder while reducing encoder computation.
- Video characteristics: Temporal redundancy makes consecutive video frames highly similar, allowing missing pixels to be recovered under extremely high masking ratios.It also makes retaining the original temporal frame rate less efficient and can dilute motion representations.
- Video characteristics: Temporal correlation can leak information because masked patches are recoverable from corresponding unmasked patches in adjacent frames.Plain random or frame masking may therefore encourage low-level temporal correspondence rather than high-level representations.
- VideoMAE framework: VideoMAE extends masked autoencoding to video with an asymmetric encoder-decoder pipeline that masks and reconstructs video tokens.The method uses downsampled frames, cube embeddings, and a vanilla ViT with joint space-time attention.
- VideoMAE framework: Temporal downsampling compresses sampled clips before pre-training to make video processing more efficient.The paper samples a clip of t consecutive frames and compresses it to T frames using a temporal stride.
- VideoMAE framework: Cube embedding represents each 2 × 16 × 16 spatiotemporal cube as one token, reducing the input's spatial and temporal dimensions.This design helps alleviate spatiotemporal redundancy in videos.
- Tube masking: 90% to 95% masking ratios increase reconstruction difficulty, mitigate information leakage, and make masked video reconstruction more meaningful.VideoMAE favors substantially higher masking ratios than ImageMAE because video information density is lower than images.
- Tube masking: Tube masking applies the same masking map across all frames, ensuring temporal neighbors of masked cubes are also masked.This mechanism alleviates leakage for cubes with no or negligible motion.
- Backbone: Joint space-time attention lets remaining tokens interact globally, while the high masking ratio reduces the encoder's quadratic computational burden.Only the unmasked tokens, such as 10%, are input to the encoder during pre-training.
4 Experiments
Experiments show that VideoMAE’s high-ratio tube masking supports effective, efficient, and transferable video representation learning across datasets and downstream tasks. The method performs strongly without extra data, including on small datasets, while domain shift affects transfer performance.
- Masking strategy: Increasing tube masking from 75% to 90% raises SSV2 accuracy from 68.0% to 69.6%, outperforming plain random and frame masking.The authors attribute the result to redundancy and temporal correlation in videos.
- Pre-training strategy: VideoMAE achieves 69.6% on SSV2 and 80.0% on K400 without using any extra data.These results outperform the reported training-from-scratch and ImageNet-21K-based alternatives in the ablation.
- Main results: VideoMAE pre-training outperforms competing settings increasingly as training sets shrink, achieving around 61% accuracy with only 3.5k HMDB51 clips.On Kinetics-400, it exceeds training from scratch by around 10% and MoCo v3 pre-training by around 5%.
- Efficiency: VideoMAE pre-training takes 19.5 hours versus 61.7 hours for MoCo v3, despite using 800 versus 300 training epochs.The reported efficiency is attributed to the asymmetric encoder-decoder and extremely high masking ratio.
- Masking strategy: 90% masking achieves the best efficiency-effectiveness trade-off, while even 95% masking maintains good performance on SSV2 and K400.The authors relate this tolerance to temporal redundancy and correlation across video frames.
- Transfer learning: VideoMAE representations transfer strongly, reaching 26.7 mAP on AVA and improving to 31.8 mAP after supervised Kinetics-400 fine-tuning.Larger datasets or backbones further raise performance, including 39.3 mAP with ViT-L on Kinetics-700 and 39.5 mAP with ViT-H on Kinetics-400.
5 Conclusion
VideoMAE is presented as a simple, data-efficient self-supervised method for video transformer pre-training, built around extremely high masking ratios and tube masking. The conclusion also identifies future improvements and potential societal concerns.
- 5 Conclusion: VideoMAE introduces extremely high masking ratios and tube masking to make video reconstruction more challenging.The harder reconstruction task is intended to encourage more representative features and relieve information leakage.
- 5 Conclusion: VideoMAE can learn effective representations using only thousands of video clips, supporting scenarios with limited available data.The paper identifies this property as having practical value for data-limited settings.
- 5 Conclusion: VideoMAE could be improved with larger webly datasets, larger models, higher input resolutions, and additional audio or text streams.The paper currently uses only the RGB video stream.
- 5 Conclusion: Potential negative societal impacts include pre-training energy consumption and misuse of video understanding models, such as for unauthorized surveillance.The paper notes that one pre-trained model can be shared across downstream tasks through additional fine-tuning.
Appendix
The appendix organizes additional material on VideoMAE architecture, implementation, experiments, analysis, visualizations, and dataset licensing.
- Appendix: The appendix provides a detailed architecture illustration and implementation details for VideoMAE.These materials are located in Sections 6 and 7.
- Appendix: The appendix reports experimental results, including ablations on Something-Something V2 and Kinetics-400.It also includes downstream action detection and further comparisons on UCF101 and HMDB51.
- Appendix: The appendix includes results analysis, reconstructed-sample visualizations, and dataset-license information.These materials are provided in Sections 9, 10, and 11.
6 Architectures
The architecture appendix describes VideoMAE using a 16-frame vanilla ViT-Base example and identifies joint space-time self-attention as a named component.
- 6 Architectures: The architecture details use a 16-frame vanilla ViT-Base as an example.The cited table presents architecture details for this configuration.
- 6 Architectures: MHA denotes the joint space-time self-attention in the architecture description.The abbreviation is explicitly defined in the table caption.
- 6 Architectures: The architecture table denotes output sizes with the format {C×T×S}.The cited passage identifies channel, temporal, and spatial sizes as the relevant dimensions.
7 Implementation Details
The implementation details specify distributed training, learning-rate scaling, dataset-specific pre-training and fine-tuning protocols, evaluation settings, and configuration tables.
- 7 Implementation Details: Experiments use 64 GPUs for Something-Something V2 and Kinetics-400, 8 GPUs for UCF101 and HMDB51, and 32 GPUs for AVA.The implementation also uses PyTorch and DeepSpeed for faster training.
- 7 Implementation Details: The learning rate is scaled linearly with overall batch size using lr = base learning rate × batch size / 256.This rule is stated for the distributed training setup.
- 7 Implementation Details: VideoMAE is pre-trained for 800 epochs by default on both Something-Something V2 and Kinetics-400.Dataset-specific pre-training and evaluation protocols are described separately for the two datasets.
- 7 Implementation Details: The appendix identifies separate tables for pre-training, linear probing, and end-to-end fine-tuning settings.The cited table references label these configuration groups as Tables 9, 11, and 12, with Table 10 covering additional fine-tuning settings.
- 7 Implementation Details: UCF101 uses 75% masking for 3200 pre-training epochs, while HMDB51 uses 75% masking for 4800 pre-training epochs.Both settings use 16 sampled frames, with different temporal strides and fine-tuning schedules.
8 Additional Results
VideoMAE remains effective with limited video data and benefits only slightly from longer pre-training schedules. On UCF101 and HMDB51, it outperforms scratch training and competing pre-training methods without extra data.
- Longer pre-training schedules bring slight gains on Something-Something V2 and Kinetics-400.The default VideoMAE pre-training schedule is 800 epochs.
- VideoMAE trains effectively without ImageNet pre-training using 9.5k clips on UCF101 and 3.5k clips on HMDB51.These small datasets test whether large ViT models can be trained from limited video data.
- On UCF101 and HMDB51, VideoMAE significantly outperforms training from scratch, MoCo v3 pre-training, and prior Vi2CLR results without extra data.The comparison covers small-scale video datasets where training large ViT models is more challenging.
9 Model result analysis
VideoMAE improves performance across most Something-Something V2 categories, suggesting stronger spatiotemporal representations than image-based alternatives. Its extreme masking can still miss motion from very small objects, while capturing broader object deformation and movement in other cases.
- VideoMAE brings significant gains for most Something-Something V2 categories over ImageMAE and ImageNet-21k supervised pre-training.The analysis interprets these gains as evidence of stronger spatiotemporal structure representations.
- The schedule analysis compares full training schedules on Something-Something V2 and Kinetics-400.The default ViT-B backbone is described in Table 8.
- VideoMAE performs slightly worse than the other two models on some Something-Something V2 categories.
- Extremely high masking may hide all tokens containing very small motion, limiting VideoMAE’s ability to reconstruct that pattern.The authors identify this as a suspected explanation for missed motion information from very small objects.
- VideoMAE captures object deformation and movement from hand squeezing that image pre-training cannot discriminate.
10 Visualization
VideoMAE reconstructs randomly selected validation videos under extremely high masking and produces satisfying visual results. These reconstructions are presented alongside the normalized-cube-pixel pre-training and fine-tuning evaluation setup used for UCF101 and HMDB51.
- VideoMAE produces satisfying reconstructions on randomly chosen validation videos even under an extremely high masking ratio.The examples imply that it learns representative features capturing holistic spatiotemporal structure.
- For UCF101 and HMDB51, VideoMAE reconstructs normalized cube pixels and uses masking ratio 75% during pre-training.It is pre-trained for 3200 epochs on UCF101 and 4800 epochs on HMDB51, with fine-tuning accuracy reported.
11 License of Data
The supplied passages identify dataset licensing information and provide several supplementary performance visualizations. They do not establish a substantive license comparison or interpretation of the plotted results.
- The Something-Something V2 and UCF101 datasets use custom licenses.
- Kinetics-400, HMDB51, and AVA use CC BY-NC 4.0 licenses.
- The supplied visualizations include comparisons with ImageMAE and ImageNet-21k supervised pre-training on Something-Something V2.
- Additional figures show prediction examples on Something-Something V2 and reconstructions for Kinetics-400 and Something-Something V2 validation videos.