PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, Yu Qiao

arXiv:2303.16727v2cs.CVcs.LG

TL;DR

Scaling video foundation models to billions of parameters is difficult because video pre-training has high computational demands and limited data. VideoMAE V2 combines dual masking, model and data scaling, and progressive training, producing a billion-level video transformer with strong performance across recognition and detection tasks. The authors report continued challenges from performance saturation and the much smaller explored data scale relative to image and language models.

  • Problem

    Scaling video masked autoencoders to billion-level models is limited by high computational and memory costs, smaller video datasets, and adaptation challenges.

  • Method

    VideoMAE V2 uses dual encoder-decoder masking, larger ViT and video-data scales, and progressive training from unlabeled to labeled hybrid datasets.

  • Results

    VideoMAE V2 trains the first billion-level video transformer and achieves strong results across action recognition, spatial action detection, and temporal action detection.

  • Takeaways & Limitations

    The framework supports scalable video representation learning across a variety of downstream tasks.

  • Takeaways & Limitations

    Performance gains shrink from ViT-H to ViT-g, while training VideoMAE on billions of videos remains extremely challenging.

Abstract

from arXiv · show

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner. The code and model is available at \url{https://github.com/OpenGVLab/VideoMAEv2}.

1. Introduction

Video foundation models offer broad transfer, but scaling video masked autoencoders to billion-level models is limited by computational, memory, data, and adaptation challenges. VideoMAE V2 addresses these challenges with dual masking and progressive training to scale model and data while supporting varied downstream tasks.

  • Video foundation models can provide generic representations adaptable across many downstream tasks, motivating scalable video pre-training.
  • Billion-level video masked autoencoder pre-training is especially difficult because video adds temporal dimension, variation, computational cost, and memory demands.
  • 0.24M videos in Kinetics400 contrasts with 14.2M images in ImageNet-22k, highlighting the limited scale of public video data.
  • Directly fine-tuning billion-level video models on small labeled datasets may be suboptimal because limited supervision can cause overfitting.
  • VideoMAE V2 scales model and data with dual masking, million-level video pre-training, and progressive training through a labeled hybrid dataset.The framework trains a billion-parameter video transformer and targets action recognition, spatial action detection, and temporal action detection.

2. Related Work

Prior vision foundation models use supervised, contrastive, siamese, masked-autoencoding, and vision-language pre-training, while video progress is constrained by smaller datasets and greater modeling complexity. Existing video foundation models also emphasize action recognition and have limited scale and task breadth.

  • Vision foundation models span supervised CNN and Transformer pre-training, self-supervised contrastive or siamese learning, masked autoencoding, and image-text alignment.
  • Video foundation model progress lags behind image models partly because video datasets are smaller and video modeling is more complex.
  • Self-supervised video masked-autoencoding methods extend image masked modeling frameworks to video.
  • Existing video foundation models often limit pre-training data and model scale, with downstream evaluation focused narrowly on action recognition rather than temporal localization.
  • Vision model scaling has enlarged CNN capacity and training data, while billion-level vision transformers have mainly been explored with large-scale supervised image pre-training.

3. VideoMAE V2

VideoMAE V2 scales masked video autoencoding through dual masking, larger models and datasets, and progressive training. The resulting framework combines efficient reconstruction with a billion-level ViT backbone for broad video downstream tasks.

  • 3.1. VideoMAE Revisited: VideoMAE tokenizes downsampled video cubes, applies high-ratio tube masking, and reconstructs masked pixels with an asymmetric encoder-decoder.The encoder processes only unmasked tokens, while reconstruction uses normalized masked pixels and MSE loss.
  • 3.2. Dual Masking for VideoMAE: VideoMAE V2 applies separate encoder and decoder masks so the decoder processes fewer tokens while retaining complementary information for reconstruction.The decoder combines encoder outputs with selected remaining tokens, and supervision applies to decoder outputs invisible to the encoder.
  • 3.2. Dual Masking for VideoMAE: Dual masking uses high-ratio random tube masking in the encoder and diverse running-cell masking in the decoder to reduce decoder input length.The encoder limits information leakage from temporal correlation, whereas decoder masking seeks information complementarity.
  • 3.3. Scaling VideoMAE: Model scaling builds VideoMAE encoders from ViT-B through ViT-g, with ViT-g providing a billion-level video backbone and a lightweight decoder.
  • 3.3. Scaling VideoMAE: Progressive training first pre-trains on an unlabeled hybrid video dataset and then uses a labeled hybrid dataset to adapt large models with multiple semantic supervision sources.The staged pipeline is intended to reduce overfitting risk from direct adaptation to limited target supervision.
  • 3.3. Scaling VideoMAE: VideoMAE V2 combines dual masking, data scaling, and progressive training into a billion-level masked autoencoding framework evaluated across recognition and detection tasks.

4. Experiments

Experiments scale VideoMAE V2 across model capacity, data, masking efficiency, progressive training, and diverse downstream tasks. The results show reduced resource use, improved transfer performance, and strong performance across action classification and detection benchmarks.

  • Implementation and Downstream Tasks: VideoMAE V2 scales from ViT-H to the 1,011M-parameter ViT-g backbone and builds million-level pre-training datasets for billion-level video modeling.The ViT-g uses 1,011M parameters, while the UnlabeledHybrid dataset contains around 1.35M videos and the labeled hybrid dataset contains 0.66M clips.
  • Main Results: 50% decoder masking provides a good accuracy–efficiency trade-off in the ViT-B Something-Something V2 ablation.The ablation uses 800-epoch pre-training on Something-Something V2 and evaluates fine-tuning accuracy.
  • Main Results: 2.0% higher Something-Something V2 performance is obtained by large-scale pre-training with ViT-H versus original VideoMAE pre-training on smaller datasets.With the same ViT-L backbone, VideoMAE V2 also exceeds MAE-ST on Kinetics-400 by 1%, suggesting data quality and diversity may matter.
  • Main Results: Consistent performance gains accompany model scaling from ViT-B through ViT-g, although the improvement from ViT-L to ViT-H is smaller.Performance appears to saturate around 87.0 on Kinetics-400 and 77.0 on Something-Something V2 without extra labeled data.
  • Main Results: 1.79× speedup and 48% lower memory result from dual masking for ViT-B, while ViT-g achieves a 1.48× speedup and similarly reduced memory.For ViT-B, FLOPs decrease from 35.48G to 25.87G and memory from 631M to 328M; for ViT-g, memory decreases from 1753M to 1050M.
  • Main Results: VideoMAE V2 achieves strong transfer results across action classification, spatial action detection, and temporal action detection benchmarks.It reports above 7% improvement on Something-Something V1, 3.1% and 3.4% gains on AVA and AVA-Kinetics, and a 5% gain over prior best performance on FineAction.

5. Conclusion and Discussion

VideoMAE V2 scales video masked autoencoding to billion-level models and million-level pre-training data through dual masking, while learning general video representations. The authors note that scaling to billions of videos remains challenging.

  • VideoMAE V2 scales to a billion-level video model on a million-level pre-training set using dual masking.The authors describe the approach as simple and efficient for large-scale video foundation modeling.
  • Dual masking supports VideoMAE as a general and scalable representation learner for video action understanding.
  • Performance gains are smaller when scaling from ViT-H to ViT-g, partly because video benchmarks approach saturation.
  • Training VideoMAE on billions of videos remains extremely challenging for current software and hardware.The explored video data scale is still several orders of magnitude smaller than those used in image and NLP research.

A. Appendix

The appendix provides additional architectural, dataset, implementation, experimental, and analytical details for VideoMAE V2.

  • The supplementary material details VideoMAE V2 architectures, pre-training datasets, experimental implementation, and additional results.

B. Model Architecture

VideoMAE V2 uses vanilla ViT backbones with joint space-time attention and keeps the decoder compact as the encoder scales.

  • VideoMAE V2 builds both encoder and decoder with vanilla ViT backbones using joint space-time attention.
  • The decoder remains fixed at 4 layers and 512 channels instead of growing with the encoder.This design is intended to ensure efficient computation.

C. Datasets

VideoMAE V2 uses a multi-source UnlabeledHybrid dataset and constructs labeled post-pre-training data by uniting Kinetics versions.

  • UnlabeledHybrid combines Kinetics, Something-Something, AVA, WebVid2M, and a self-collected Instagram dataset.
  • Kinetics data merge the training and validation sets of Kinetics-400, -600, and -700 after duplicate removal and validation cleanup.
  • Something-Something V2 contributes 169k training videos and 25k validation videos to the hybrid dataset.
  • AVA clips are cut from 15-minute movie segments into 21k video clips.
  • WebVid2M contributes a random sample of 250k training videos.
  • The self-collected Instagram dataset contributes 250k randomly selected videos scraped using thousands of category tags.Its videos average 34 seconds in duration.
  • The labeled post-pre-training dataset is constructed as the union of different Kinetics versions.

D. Implementation Details

The implementation is organized as a three-stage progressive training pipeline: pre-training, post-pre-training, and task-specific fine-tuning.

  • The pipeline comprises pre-training, post-pre-training, and specific fine-tuning stages.
  • Pre-training is followed by post-pre-training before task-specific models are produced.

D.1. Pre-training

VideoMAE V2 pre-trains ViT-huge and ViT-giant models on UnlabeledHybrid for 1200 epochs using 64 80G-A100 GPUs, with mixed-precision and checkpointing optimizations.

  • 1200 epochs of pre-training are performed for ViT-huge and ViT-giant on the UnlabeledHybrid dataset.
  • The pre-training uses 64 80G-A100 GPUs and engineering optimizations including mixed-precision training and checkpointing.
  • The encoder uses FP-16 mixed precision while the decoder uses FP32 precision to reduce precision-overflow risk.

D.2. Post-pre-training

Post-pre-training fine-tunes the pre-trained encoder on LabeledHybrid before task-specific models are produced, with settings adjusted for large-model optimization and downstream tasks.

  • D.2. Post-pre-training: The pre-trained encoder is fine-tuned on the LabeledHybrid (K710) dataset during supervised post-pre-training.
  • D.2. Post-pre-training: The dual-masked architecture computes reconstruction loss only on tokens invisible to the encoder rather than reconstructing the full video clip.
  • D.2. Post-pre-training: Specific fine-tuning produces models for action classification, action detection, and temporal action detection.
  • D.2. Post-pre-training: For action detection, the pipeline uses person detection followed by action classification, with ground-truth boxes during training and AIA-detected boxes during testing.
  • D.2. Post-pre-training: Temporal action detection uses a LabeledHybrid-trained backbone with ActionFormer on THUMOS14 and FineAction, applying Soft-NMS at inference.

E. More Results

VideoMAE V2 improves recognition with larger inputs, performs strongly on smaller benchmarks, and distillation yields a more efficient ViT-B foundation model.

  • E. More Results: Larger input resolution and more frames further improve the model’s recognition results on Kinetics.
  • E. More Results: VideoMAE V2 achieves state-of-the-art performance on UCF101 and HMDB51, substantially outperforming VideoMAE V1 on these smaller benchmarks.
  • E. More Results: Distillation compresses VideoMAE V2-g into a ViT-B student initialized with VideoMAE V2-B post-pre-training weights.
  • E. More Results: The distilled student is trained on K710 or SSv2 for 100 epochs by minimizing KL divergence between student and teacher logits.
  • E. More Results: The distilled ViT-B outperforms the original VideoMAE ViT-B models on K400, K600, and SSv2.
Loading 2303.16727v2…