PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVT: BERT Pretraining of Video Transformers

Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, Lu Yuan

arXiv:2112.01529v3cs.CVcs.LG

TL;DR

BEVT addresses the unexplored use of BERT-style pretraining for video transformers, where spatial and temporal dependencies vary across videos. It decouples spatial representation learning from temporal dynamics learning and achieves strong results across three video benchmarks, including 71.4% Top-1 accuracy on Something-Something-v2.

  • Problem

    BERT-style pretraining had been explored for image representation learning, but its use for video understanding remained unexplored despite the computational difficulty of learning video representations from scratch.

  • Method

    BEVT uses two encoders based on Video Swin Transformer to decouple spatial representation learning from temporal dynamics learning through image- and video-stream pretraining.

  • Results

    BEVT achieves competitive performance on Kinetics-400 and clear-margin improvements over alternative baselines on Something-Something-v2 and Diving-48, reaching 71.4% Top-1 accuracy on Something-Something-v2.

  • Takeaways & Limitations

    Decoupling spatial and temporal representation learning is reported as both efficient and effective, enabling state-of-the-art performance across three video recognition datasets.

  • Takeaways & Limitations

    The reported setup uses the BEVT architecture without loading pretrained weights.

Abstract

from arXiv · show

This paper studies the BERT pretraining of video transformers. It is a straightforward but worth-studying extension given the recent success from BERT pretraining of image transformers. We introduce BEVT which decouples video representation learning into spatial representation learning and temporal dynamics learning. In particular, BEVT first performs masked image modeling on image data, and then conducts masked image modeling jointly with masked video modeling on video data. This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors that can ease the learning of video transformers, which are often times computationally-intensive if trained from scratch; 2) discriminative clues, i.e., spatial and temporal information, needed to make correct predictions vary among different videos due to large intra-class and inter-class variations. We conduct extensive experiments on three challenging video benchmarks where BEVT achieves very promising results. On Kinetics 400, for which recognition mostly relies on discriminative spatial representations, BEVT achieves comparable results to strong supervised baselines. On Something-Something-V2 and Diving 48, which contain videos relying on temporal dynamics, BEVT outperforms by clear margins all alternative baselines and achieves state-of-the-art performance with a 71.4\% and 87.2\% Top-1 accuracy respectively. Code will be made available at \url{https://github.com/xyzforever/BEVT}.

1. Introduction

BEVT extends BERT-style masked-token pretraining to video transformers by separating spatial representation learning from temporal dynamics learning. It targets the computational cost of video pretraining and the varying reliance of video samples on spatial versus temporal clues, achieving strong results across three benchmarks.

  • Motivation: BERT-style pretraining for video transformers had not previously been explored, despite its success for language and emerging use in image representation learning.The gap motivates studying masked-token objectives beyond image transformers.
  • Motivation: Video representation learning from scratch is computationally expensive and often requires extremely large-scale datasets, motivating reuse of image-derived spatial priors.Prior image-pretrained self-supervised models can benefit video recognition, but their spatial context may not be explicitly preserved.
  • Motivation: Different videos require different discriminative clues: spatial information can suffice for relatively static actions, whereas fine-grained diving sequences require temporal dynamics.The paper uses “applying lipstick” and fine-grained diving sequences as contrasting examples.
  • Approach: BEVT decouples video representation learning into spatial and temporal components using an image stream and a video stream that interact during video modeling.The framework is built on the Video Swin Transformer architecture and uses a BERT-style objective.
  • Results: 81.1% Top-1 accuracy on K400 exceeded the strong supervised baseline’s 80.6%, while SSV2 and DIVING48 reached 71.4% and 87.2% Top-1 accuracy.The paper reports comparable performance on K400 and clear-margin improvements on SSV2 and DIVING48.
  • Contributions: The authors introduce a two-stream design, demonstrate differing spatial-temporal preferences across videos, and report comparable or better results than state-of-the-art methods on three benchmarks.These are stated as the paper’s main contributions.

2. Related Work

Related work spans CNN-based video understanding, vision transformers, self-supervised representation learning, and BERT-style masked image modeling. BEVT builds on these lines by applying BERT pretraining to video transformers and decoupling spatial and temporal learning.

  • Video understanding with CNNs: CNN video methods use temporal aggregation or 3D convolutions, with 3D CNNs generally incurring substantial computational cost.Temporal aggregation combines frame-level features, while 3D CNNs jointly model spatial-temporal relationships.
  • Vision transformers: Vision transformers apply patch-based tokenization and attention to images, while recent work extends transformer architectures to video understanding.Examples include ViT for images, multi-scale spatial-temporal tokens, and Video Swin Transformers.
  • Self-supervised representation learning: Self-supervised representation learning replaces manual labels with surrogate tasks such as inpainting, colorization, jigsaw prediction, rotation prediction, and contrastive learning.Recent contrastive-learning methods also use transformers as vision backbones.
  • BERT pretraining: BERT pretraining predicts masked tokens in corrupted text, and related vision methods apply masked image modeling to recover masked patch pixels or latent codes.BEiT and ICT established this image-transformer direction, with PeCo and pixel-recovery methods extending it.
  • BERT pretraining: BEVT extends BERT-style pretraining to video transformers while decoupling spatial representation learning from temporal dynamics learning.This design accommodates different videos’ varying reliance on salient spatial and temporal clues.

3. Method

BEVT decouples video pretraining into spatial representation learning on images and temporal dynamics learning on videos, using two interacting streams and masked-token objectives. It initializes the video stream from image pretraining, then jointly optimizes masked image and video modeling to preserve spatial information while learning temporal dynamics.

  • BEVT targets efficient self-supervised learning for relatively static videos needing spatial representations and dynamic videos also requiring temporal dynamics.The design addresses the computational cost of pretraining video transformers from scratch.
  • BEVT uses separate image and video streams that decouple spatial representation learning from temporal dynamics learning.The streams operate on images and videos, respectively, and interact during video modeling.
  • The video stream converts clips into 3D patches, while the image stream converts images into 2D patches before transformer processing.Video patches have size 2×4×4×3, image patches have size 4 × 4 × 3, and both are projected into token embeddings.
  • Masked image modeling captures spatial priors, whereas masked video modeling captures temporal dynamics by predicting corrupted image and video tokens.The image stream masks image patches, and the video stream predicts masked 3D token content.
  • BEVT uses blockwise masking for images and tube masking for videos, applying the same 2D mask across a randomly selected contiguous frame range.Masked tube lengths range from 0.5T to T, with a 0.5 masking ratio per masked frame.
  • Both streams use hierarchical Video Swin Transformer encoders, with patch merging performing spatial downsampling between stages.The encoders contain four stages, and Swin attention blocks follow the downsampling operations.
  • Lightweight decoders reconstruct masked tokens from encoder features, using spatial and temporal upsampling for the video stream and a softmax classifier at each output position.The video decoder upsamples stage-4 features, fuses them with stage-3 features, then temporally upsamples the fused representation.
  • Training first learns spatial representations on ImageNet, then initializes the video stream and jointly optimizes masked image and video modeling.The combined objective weights the image and video losses with λ, preserving spatial information while learning temporal dynamics.

4. Experiments

Experiments evaluate BEVT across three video benchmarks, compare pretraining strategies and state-of-the-art methods, and test the roles of temporal modeling, image initialization, image data, tokenizers, and masking.

  • Main Results: BEVT outperforms Image Sup by 4.3% on SSV2 and 2.7% on DIVING48, but performs on par with Image Sup on K400.BEVT also provides comparable or better results than Image CL across all three datasets.
  • Main Results: BEVT’s video stream improves over BEVT-I by 1.4% on SSV2 and 5.5% on DIVING48, while producing similar results on K400.Video-stream pretraining from scratch under a similar computation budget performs much worse than BEVT.
  • Dataset Analysis: Removing temporal information causes 60% and 70% performance drops on SSV2 and DIVING48, respectively, but has relatively small impact on K400.The analysis indicates that K400 relies mainly on spatial clues, whereas temporal dynamics are particularly important for SSV2 and DIVING48.
  • Main Results: BEVT achieves the best reported performance by clear margins on SSV2 and DIVING48, and competitive K400 results with similar or less GFLOPs.On SSV2, this performance is achieved without manual labels during ImageNet and K400 pretraining.
  • Ablation Study: Image-stream pretrained weights benefit both BEVT-V and joint BEVT pretraining, while jointly training image and video streams remains necessary for desirable gains.The ablation initializes image transformers with the image stream pretrained on ImageNet-1K.
  • Ablation Study: ImageNet images are slightly better than K400 frames for joint training, with differences below 0.3% on all three datasets.This suggests the spatial-preserving image stream is not very sensitive to data domains.
  • Ablation Study: The PeCo tokenizer outperforms the DALL-E tokenizer on all three datasets and yields 71.4% SSV2 and 87.2% DIVING48 Top-1 accuracy.The tokenizer ablation links higher performance to the use of a better visual tokenizer.

5. Conclusion and Discussion

The paper positions BEVT as a step from BERT-style image pretraining to video transformers, learning spatial representations and temporal dynamics through decoupled pretraining. It reports state-of-the-art video recognition performance while acknowledging substantial computational costs for scaling.

  • 5. Conclusion and Discussion: BEVT extends BERT pretraining from images to video transformers and learns discriminative spatial representations alongside temporal dynamics.The approach decouples video pretraining into spatial and temporal representation learning.
  • 5. Conclusion and Discussion: Decoupling video pretraining is reported as both efficient and effective, with state-of-the-art performance on three video recognition datasets.
  • 5. Conclusion and Discussion: BEVT still requires more than one week on 32 V100 GPUs, limiting scaling to larger video datasets and models.Ignoring masked tokens during transformer computation is identified as future work that may help address this obstacle.

A. Details on Pretrained Tokenizer

The pretrained visual tokenizer converts images into discrete token maps using a VQ-VAE codebook, while its decoder reconstructs images from those tokens.

  • A. Details on Pretrained Tokenizer: The VQ-VAE tokenizer transforms each 256×256 image into a 32×32 image token map using a visual codebook.The visual-token vocabulary contains 8192 tokens.
  • A. Details on Pretrained Tokenizer: The VQ-VAE decoder reconstructs each input image from its tokens.
Loading 2112.01529v3…