PaperlayerView exact revision on arXiv

Source-linked AI summary

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, Yu Qiao

arXiv:2303.16058v2cs.CV

TL;DR

VFMs face computational, data, and image-to-video transfer challenges, while VideoMAE’s pixel reconstruction conflicts with cross-modal alignment. This paper uses an unmasked IFM teacher to guide masked video modeling and progressive multimodal pre-training, achieving state-of-the-art results across video tasks with public pre-training sources.

  • Problem

    VFMs are limited by high computational costs, data scarcity, difficult IFM-to-video transfer, and VideoMAE’s conflict between pixel reconstruction and cross-modal alignment.

  • Method

    The method masks low-semantic video tokens and aligns selected unmasked tokens with IFM representations while progressively pre-training a ViT from scratch on video and vision-language data.

  • Results

    State-of-the-art performance spans action recognition, spatiotemporal localization, video-text retrieval, and video question answering using public sources for pre-training.

  • Takeaways & Limitations

    The framework supports scene-related, temporal-related, and complex video-language understanding while reducing pre-training carbon emissions relative to CoCa.

  • Takeaways & Limitations

    VideoMAE-style pixel reconstruction requires long pre-training, conflicts with cross-modal alignment, and adds decoder memory costs.

Abstract

from arXiv · show

Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.

1. Introduction

VFMs remain underexplored because video training requires substantial computation and data, while adapting IFMs can weaken temporal representation learning. UMT-based progressive pre-training addresses these constraints and achieves strong performance across video-only and video-language tasks.

  • VFMs are underexplored because video training faces high computational costs and data scarcity.
  • Adapting IFMs to video can undermine inherited generality, encourage single-frame scene shortcuts, and limit spatiotemporal learning.
  • VideoMAE learns effective spatiotemporal features efficiently but requires long pre-training, conflicts with cross-modal alignment, and incurs decoder memory costs.
  • UMT masks low-semantic video tokens and aligns unmasked tokens with an IFM while training a vanilla ViT from scratch.
  • Progressive pre-training uses video-only masked modeling first, then vision-language data for video-text retrieval and video question answering.
  • 90.6% top-1 accuracy on K400, 39.8 mAP on AVA, 58.8 R@1 on MSRVTT, and 47.1% accuracy on MSRVTT accompany 70× lower carbon emissions than CoCa.

2. Related Works

Related VFM approaches inherit strong spatial representations but remain limited in temporal understanding, action localization, scalability, or video-language capability.

  • Existing VFMs primarily build on well-prepared IFMs, whose strong spatial pre-training restricts spatiotemporal representation learning.
  • Despite strong video-only results, prior models struggle with temporal-related actions and action localization.
  • InternVideo broadens VFM learning through generative and discriminative objectives, but CLIP dependence and high training costs hinder scaling.

3. Method

UMT trains a video encoder from scratch by semantically selecting and aligning unmasked tokens, followed by progressive video-only and multimodal pre-training.

  • Unmasked Teacher: UMT uses an IFM as a teacher while masked video modeling trains a VFM from scratch without an extra decoder.
  • Architecture: CLIP-ViT supplies language-guided semantic representations, while spatiotemporal attention lets unmasked tokens interact across frames.
  • Masking: An 80% masking ratio reduces redundancy, and semantic masking preferentially retains informative tokens in each frame.
  • Masking: Semantic token importance is computed from the teacher’s final self-attention, then used to sample unmasked tokens and encourage long-term spatiotemporal reasoning.
  • Target: The student receives only L(1−r)T unmasked tokens, whose projected representations are aligned with corresponding teacher tokens using normalized-pair MSE.
  • Progressive pre-training: Stage 1 trains on videos for video-only tasks, while Stage 2 adds a text encoder, cross-modal decoder, and large-scale vision-text pairs.
  • Progressive pre-training: Both stages use UTA; Stage 2 additionally applies video-text contrastive and matching objectives for multimodal learning.

4. Experiments

Experiments show that UMT’s design choices improve training efficiency and support scene, temporal, localization, and video-language tasks. Across benchmarks, the model achieves strong performance with public data and comparatively modest training resources.

  • Implementation: UMT pre-training uses Kinetics-710 for Stage 1 and image-text corpora for Stage 2, with ViT-B/16 and ViT-L/16 configurations guided by CLIP teachers.The default setup trains on 32 A100 GPUs, using 80% video masking and 200 Stage-1 epochs.
  • Ablation Study: Unmasked token alignment improves accuracy at only 36% memory cost, whereas combining it with pixel reconstruction produces poor K400 and MSRVTT results.The findings support using high-level alignment rather than jointly optimizing low-level pixel reconstruction.
  • Ablation Study: Semantic masking benefits scene recognition, while random masking suffices for motion-focused SthSth V2; sparse sampling without temporal downsampling is preferred.The paper attributes this difference to K400 requiring scene and object identification, whereas SthSth V2 emphasizes intricate motion.
  • Ablation Study: Aligning the last 6 layers and using an 80% masking ratio provide favorable design choices without materially changing GPU memory or running speed.A 95% ratio makes token alignment too difficult, while removing masks makes the task too easy.
  • Ablation Study: Longer schedules improve both benchmarks, yet UMT converges faster than VideoMAE; at 200 epochs it reaches 3.9% on SthSth V2 and 6.8% on Kinetics-400.On K400, UMT-L also achieves a 3.7% improvement over VideoMAE with 400 rather than 1600 pre-training epochs.
  • Ablation Study: Spatiotemporal attention and mask modeling are vital for UMT, while directly using CLIP-ST as teacher causes a performance drop.The student can outperform CLIP-ST after fine-tuning, which the paper attributes to masked video modeling and long-term object dependencies.
  • Single-modality Tasks: On K400, UMT-L exceeds MTV-H by 0.4% top-1 accuracy while using 1/10 of the FLOPs and 1/3 of the parameters.The model also reports 1.0%/1.7% higher top-1/top-5 accuracy than UniFormerV2-L on Moments in Time.

5. Conclusion

The paper proposes an unmasked-teacher approach and progressive pre-training framework for video foundation models. The framework targets scene-related, temporal-related, and complex video-language understanding while emphasizing scalability and reproducibility.

  • The method uses an image foundation model as the unmasked teacher for masked video modeling.
  • The progressive pre-training framework handles scene-related actions, temporal-related actions, and complex video-language understanding.
  • The authors characterize the framework as simple, scalable, and reproducible for further video foundation model research.

Supplementary Material

The supplementary material specifies the video-encoder architecture and identifies the associated Stage-1 pre-training settings. It also defines the notation used to describe the architecture.

  • Table 17 presents the video-encoder architecture using ViT-B with an 8-frame input as an example.
  • The architecture notation includes MHSA, MLP, LN, and K, where K denotes the layer number for unmasked-token alignment.
  • Table 18 provides the Stage-1 pre-training settings.

A.1. Model architecture and training details

The paper uses a two-stage pre-training design followed by task-specific fine-tuning. The supplementary details describe the encoder, multimodal components, and task-specific sampling and heads.

  • A.1. Model architecture and training details: Stage 1 trains a vanilla ViT video encoder from scratch without temporal downsampling, using 1×16×16 spatiotemporal patches for ViT-B and ViT-L.A linear projection with Layer Normalization aligns the encoder with the unmasked teacher.
  • A.1. Model architecture and training details: Stage 2 adds text encoding and cross-modal decoding, initializing components from BERTbase or BERTlarge and assigning all loss weights to 1.
  • Action Recognition: Action recognition fine-tunes the Stage-1 video encoder with an added classification layer; Stage-2 fine-tuning produced similar Kinetics results.
  • Action Detection: Action detection adds ROIAlign with MaxPooling and uses a frame span of 300 because the Kinetics pre-trained models use sparse sampling.
  • Video-text retrieval: Video-text retrieval retains the Stage-2 architecture but applies only VTC and VTM losses, with 12 sparsely sampled frames for training and testing.Flip augmentation is used for SSV2 retrieval for fair comparison with Singularity, although it may harm this temporal-related dataset.
  • Video question-answering: Video question-answering is formulated as text generation with an additional multimodal decoder initialized from the pre-trained cross-modal decoder.

B.1. Video-text retrieval

The supplementary material reports additional zero-shot and fine-tuned video-text retrieval results across several datasets. It also lists related multi-choice video question-answering fine-tuning settings.

  • B.1. Video-text retrieval: Tables 25 and 26 report additional zero-shot and fine-tuned retrieval results.
  • B.1. Video-text retrieval: The additional retrieval evaluations cover MARVTT, DiDeMo, ActivityNet, LSMDC, and MSVD.
  • B.1. Video-text retrieval: Table 23 provides multi-choice video question-answering fine-tuning settings.

B.2. Dataset descriptions

This section identifies the tables covering pre-training and downstream dataset statistics, along with video-text retrieval settings and results across five benchmarks.

  • Table 24 lists the fine-tuning settings for video-text retrieval.
  • Table 25 presents zero-shot retrieval results on MSRVTT, DiDeMo, AcitivityNet, LSMDC, and MSVD.
  • Table 26 presents fine-tuned retrieval results on MSRVTT, DiDeMo, AcitivityNet, LSMDC, and MSVD.
  • Table 27 reports statistics of the pre-training datasets.
  • Table 28 reports statistics of the downstream datasets.
Loading 2303.16058v2…