Source-linked AI summary
Masked Autoencoders As Spatiotemporal Learners
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, Kaiming He
TL;DR
Video representation learning can be approached with masked autoencoding while using little spacetime-specific knowledge. The paper masks and reconstructs video patches with vanilla Vision Transformers, finding strong recognition results, a 90% optimal masking ratio, and substantial efficiency gains. It also reports encouraging results on uncurated Instagram videos, while noting that scaling high-dimensional video remains challenging.
Problem
The paper asks whether masked autoencoding can provide strong video representations without substantial domain-specific spacetime inductive bias.
Method
The method randomly masks spacetime patches, encodes visible patches with a vanilla ViT, and decodes the full set to reconstruct pixels.
Results
MAE pre-training improves recognition across video datasets, including a 13% absolute Kinetics-400 accuracy gain over training from scratch, while 90% masking gives a 4.1× wall-clock speedup.
Takeaways & Limitations
Masked autoencoding can support a unified, low-domain-knowledge framework for video representation learning, including pre-training on real-world uncurated data.
Takeaways & Limitations
The explored data scale is orders of magnitude smaller than language counterparts, and high-dimensional video remains a major scaling challenge.
Abstract
from arXiv · showhide
This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related to information redundancy of the data. A high masking ratio leads to a large speedup, e.g., > 4x in wall-clock time or even more. We report competitive results on several challenging video datasets using vanilla Vision Transformers. We observe that MAE can outperform supervised pre-training by large margins. We further report encouraging results of training on real-world, uncurated Instagram data. Our study suggests that the general framework of masked autoencoding (BERT, MAE, etc.) can be a unified methodology for representation learning with minimal domain knowledge.
1 Introduction
The paper extends masked autoencoding to video by reconstructing randomly masked spacetime patches with minimal spacetime-specific design. It finds that videos support very high masking, improving efficiency while producing strong recognition representations.
- Motivation and approach: The method randomly masks spacetime video patches and reconstructs them with an autoencoder whose components are largely agnostic to spacetime.Only patch and positional embeddings encode spacetime-specific information.
- Key observations: 90% is the optimal video masking ratio, compared with 75% for image MAE, supporting a link between masking and data redundancy.The paper relates the higher ratio to correlations in natural videos.
- Efficiency: 90% masking reduces encoder time and memory complexity to <1/10 and yields a theoretical 7.7× computation reduction.The reported practical wall-clock speedup is 4.1×, despite data loading becoming a bottleneck.
- Recognition results: MAE pre-training improves video recognition across datasets, including a 13% absolute Kinetics-400 accuracy gain for ViT-Large over training from scratch.It also outperforms supervised pre-training by large margins and remains competitive with methods using more domain knowledge.
- Data scale and realism: Pre-training on 1 million random, uncurated Instagram videos produces encouraging results, extending the approach beyond controlled curated data.The conclusion reports performance close to pre-training on curated data such as Kinetics.
2 Related Work
The paper places video MAE within the broader denoising and masked-autoencoding tradition while distinguishing its pixel-level, tokenizer-free approach. Its central video-specific observation is that structure-agnostic masking can exploit temporal redundancy efficiently.
- Masked autoencoding: Masked autoencoding generalizes denoising autoencoders and connects representation learning across language, images, and videos.BERT is presented as masked autoencoding over language tokens, while vision methods apply related ideas to pixels or patches.
- Prior video objectives: Video self-supervised learning has also explored temporal coherence, future prediction, object motion, temporal ordering, and spatiotemporal contrast.These related directions explicitly target different temporal or motion properties.
- Temporal coherence: The method uses temporal coherence implicitly through a high masking ratio, assuming videos are more information-redundant than images.Its largely spacetime-agnostic design leaves masking ratio as the main opportunity to exploit temporal coherence.
- Video self-supervision: Unlike approaches that tokenize prediction targets, this method reconstructs pixels directly and requires no extra tokenizer data or domain knowledge.The paper identifies this as a simpler alternative for video self-supervision.
- Concurrent work: The work was conducted independently and concurrently with another related method.The passage does not characterize the relationship or comparative results.
3 Method
The method applies vanilla Vision Transformers to randomly masked spacetime patches, encoding only visible tokens and decoding the full sequence to reconstruct pixels. Structure-agnostic masking and sparse encoding provide the main design and efficiency choices.
- Method overview: The method extends MAE to spacetime data under a general framework with minimal domain knowledge.Its stated goal is to avoid specialized video assumptions wherever possible.
- Patch representation: Patch and positional embeddings are the only explicitly spacetime-aware process in the model.The patches form a regular non-overlapping spacetime grid before linear embedding.
- Masking: Random sampling without replacement masks embedded patches independently of spacetime structure.This is analogous to unstructured masking in BERT and image MAE.
- Masking strategy: Structure-agnostic masking can outperform space-only, time-only, and block-wise sampling because neighboring patches retain coherent information.At high masking ratios, time-only sampling can leave only one frame and make prediction overly difficult.
- Encoder: The vanilla ViT encoder processes only visible patches, reducing encoder complexity to <1/10 at a 90% masking ratio.The reduction follows from applying self-attention to the much smaller visible-token set.
- Decoder and objective: A smaller vanilla ViT decoder combines encoded patches with mask tokens and reconstructs pixel-space patches using mean squared error on unknown patches.In practice, it predicts one temporal slice of each spacetime patch to keep the prediction layer manageable.
- Architecture: Neither encoder nor decoder uses hierarchy or spacetime factorization; global self-attention learns from the data instead.This preserves the paper’s minimal-inductive-bias design.
4 Implementation
Implementation uses standard video preprocessing and vanilla ViT components with 2×16×16 spacetime patches. Training evaluates pre-training through end-to-end fine-tuning and uses multi-view inference.
- Data preprocessing: The default pre-training input contains 16 frames at 224×224 resolution, sampled with temporal stride 4.Spatial processing uses random resized cropping with scale range [0.5, 1] and random horizontal flipping.
- Data loading: Repeated sampling takes four samples per loaded and decompressed raw video by default to reduce data-loading time per sample.It changes sample order during training but not the number of samples seen.
- Architecture settings: A 2×16×16 temporal-spatial patch size produces 8×14×14 tokens from a 16×224×224 input.The same patch size is used for ViT-B, ViT-L, and ViT-H.
- Positional embeddings: Separable learnable positional embeddings sum separate spatial and temporal embeddings to avoid excessive three-dimensional size.The paper reports that sinusoidal embeddings work similarly.
- Evaluation: Pre-training quality is evaluated by end-to-end fine-tuning, while inference averages predictions across K temporal clips and 3 spatial views.The default Kinetics setting uses K=7, giving 7×3 views.
5 Experiments
Experiments show that MAE pre-training improves video recognition accuracy and training efficiency, with 90% masking and random sampling particularly effective. Transfer results extend across datasets, tasks, and real-world Instagram video data using largely vanilla ViT designs.
- Performance: 13.0% absolute accuracy gain raises vanilla ViT-L from 71.4% to 84.4% on Kinetics-400 versus training from scratch.The 800-epoch MAE pre-training plus 100-epoch fine-tuning can also finish sooner overall than 400-epoch training from scratch.
- Efficiency: 4.1× wall-clock speedup results from 90% masking, although data loading becomes the bottleneck after sparse encoding reduces computation.The sparse encoder reduces FLOPs by 7.7× and computation time by 5.8× before data-loading overhead is included.
- Masking ratio: 90% masking performs best, while 75% and 50% perform worse despite exposing more tokens and incurring greater encoder computation.At equal encoded-token budgets, 90% and 95% masking perform closely.
- Mask sampling strategy: Random spacetime-agnostic sampling works best; time-only sampling falls from 79.1% to 75.4% when masking increases from 75% to 87.5%.Space-only sampling reaches 83.5%, while block-wise sampling reaches 83.2% at 75% masking.
- Transfer and data scale: 16.4% improvement on SSv2 and 9.5% on AVA over supervised K400 pre-training demonstrate strong transfer, while larger and uncurated Instagram datasets also yield encouraging results.K700 adds 2.0% on AVA and 1.5% on SSv2 over K400 pre-training; vanilla ViT results are competitive with specialized video methods.
6 Conclusion
The study finds that video MAE can learn strong representations with minimal spacetime-specific inductive bias and supports a unified masked-autoencoding framework. It also identifies masking ratio as data-dependent and reports encouraging results on uncurated data, while noting substantial scaling challenges remain.
- Video MAE learns strong representations with minimal domain knowledge or spacetime-specific inductive biases.The encoder and decoder can remain largely agnostic to spacetime structure.
- Optimal masking ratios depend on data characteristics, with videos requiring particularly high masking ratios.The paper presents masking ratio as an important factor across language, images, and videos.
- Self-supervised pre-training on real-world, uncurated data achieves performance close to pre-training on controlled, curated data.The authors describe promising uncurated-data results as rare in the literature.
- The explored data scale is orders of magnitude smaller than language counterparts, and high-dimensional video remains difficult to scale.The authors identify scaling up video self-supervised learning as an open problem.
A Implementation Details
The implementation uses standard video fine-tuning setups across Kinetics, AVA, and SSv2, with specified optimization, augmentation, and model-configuration choices. Image-pretrained models are adapted by inflating 2D patch embeddings into 3D.
- Kinetics action classification: Kinetics fine-tuning adds a linear classifier to averaged encoder tokens and uses settings summarized for ViT-B, ViT-L, and ViT-H.The listed configurations vary learning rate, warmup, batch size, and regularization by model size.
- AVA action detection: AVA fine-tuning adapts Faster R-CNN for video action detection, with relative positions used only for the reported AVA results in Table 8.The settings mainly follow prior video-detection work.
- SSv2 action classification: SSv2 clips are formed by sampling one frame from each temporal segment after splitting every video into segments.The fine-tuning settings mainly follow prior work.
- Fine-tuning from image pre-training: Image-pretrained video models inflate the 2D patch-embedding kernel to 3D and initialize temporal position embeddings with zeros.This procedure is used when transferring ImageNet-based supervised or MAE pre-training to videos.
B.1 System-level Comparisons
Vanilla ViT systems achieve competitive video recognition and detection results across Kinetics-400, AVA, and SSv2, despite using simpler architectures and lower-resolution inputs than some comparisons.
- Kinetics-400: On Kinetics-400, the method is competitive with leading systems while using vanilla ViT and standard 224×224 resolution.Other leading entries use hierarchical or specialized video designs, and some use higher-resolution inputs or in-house supervision.
- AVA: On AVA, the method is close to MaskFeat using 16×224^2 inputs versus MaskFeat’s 40×312^2 inputs.The models remain plain ViTs without feature hierarchies.
- SSv2: On SSv2, vanilla ViT results at 16×224^2 compare favorably with MaskFeat using 40×312^2 inputs.The comparison concerns action-classification performance under different input resolutions.
B.2 Ablation on Intermediate Fine-tuning
Intermediate supervised fine-tuning after self-supervised pre-training improves AVA substantially but has only a marginal effect on SSv2.
- Intermediate fine-tuning has substantial improvements on AVA action detection.The experiment uses ViT-L with MAE pre-training on K400, K600, or K700 for 1600 epochs.
- Intermediate fine-tuning has a marginal effect on SSv2 action classification.The paper therefore does not use intermediate fine-tuning for the SSv2 system-level results.
B.3 Masking during fine-tuning
The fine-tuning ablation tests whether masking can reduce training cost while preserving accuracy. Annealing a 50% masking ratio to 0% achieves nearly baseline accuracy with a modest speedup.
- B.3 Masking during fine-tuning: 84.1% accuracy is achieved with a 1.2× speedup when masking starts at 50% and is cosine-annealed to 0%.Full fine-tuning without masking reaches 84.4%.
- B.3 Masking during fine-tuning: 83.8% accuracy is achieved with a 1.3× speedup under the same 50%-to-0% annealing schedule.
- B.3 Masking during fine-tuning: The fine-tuning experiment varies the starting masking ratio among 0%, 50%, and 75%, with cosine annealing toward 0%.The baseline uses a 0% starting ratio, corresponding to fine-tuning without masking.