Source-linked AI summary
ViViT: A Video Vision Transformer
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid
TL;DR
Video transformers face long token sequences and limited labeled data compared with image datasets. ViViT factorizes spatial and temporal processing, uses regularization and pretrained image models, and achieves state-of-the-art results across five video-classification datasets.
Problem
Video transformers must handle long spatio-temporal token sequences despite video datasets having far fewer labeled examples than image datasets.
Method
ViViT extracts spatio-temporal tokens, encodes them with pure transformers, factorizes spatial and temporal dimensions, and uses regularization plus pretrained image models.
Results
ViViT achieves state-of-the-art results across five popular video-classification datasets, including Kinetics, Epic Kitchens, Something-Something v2, and Moments in Time.
Takeaways & Limitations
Regularization and image-model pretraining enable these high-capacity video transformers to train effectively on comparatively small datasets.
Takeaways & Limitations
The authors identify capturing fine-grained motion patterns, especially on Something-Something v2, as an area for improvement and future work.
Abstract
from arXiv · showhide
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks. To facilitate further research, we release code at https://github.com/google-research/scenic/tree/main/scenic/projects/vivit
1. Introduction
ViViT develops pure-transformer models for video classification that apply self-attention to spatio-temporal tokens and factorise spatial and temporal processing for efficiency. The study evaluates design choices through ablations and reports state-of-the-art results across multiple video benchmarks.
- Architecture: ViViT applies self-attention to spatio-temporal tokens extracted from input videos for pure-transformer video classification.The architecture is inspired by Vision Transformer models for images.
- Architecture: Several ViViT variants factorise the transformer along spatial and temporal dimensions to process large token sequences efficiently and scalably.These factorisations correspond to different attention patterns over space and time.
- Training and analysis: The models address transformers’ limited inductive biases by investigating tokenisation, architecture, and regularisation choices through thorough ablation analysis.The introduction notes that transformers may require more data or stronger regularisation than convolutional models.
- Results: ViViT achieves state-of-the-art results on multiple standard video classification benchmarks.The introduction attributes these results to design choices informed by the ablation analysis.
2. Related Work
Video understanding architectures evolved from hand-crafted features and two-stream CNNs toward pure-transformer models, following transformers’ success in NLP and ViT’s image-classification results. ViViT extends this direction to video with factorised variants, regularisation, and pretrained models for comparatively smaller datasets.
- Early video methods used hand-crafted appearance and motion features, while two-stream CNNs separately processed RGB frames and optical flow before late fusion.
- Transformers replaced convolutions and recurrent networks in NLP with architectures based on self-attention, layer normalisation, and multilayer perceptrons.
- In computer vision, self-attention was usually added late or used to augment residual blocks rather than forming the entire architecture.
- ViT showed that pure-transformer networks can achieve state-of-the-art image classification, but their effectiveness depends on large-scale training because they lack convolutional inductive biases.
- ViViT develops pure-transformer video classifiers with variants that factorise spatial and temporal dimensions, using regularisation and pretrained models to address smaller video datasets.
3. Video Vision Transformers
The paper adapts Vision Transformer principles to video through two tokenisation strategies and several transformer architectures. These designs address video’s long token sequences by factorising spatial and temporal processing while enabling spatio-temporal information aggregation.
- Video tokenisation: Videos are converted into spatio-temporal tokens by uniformly embedding sampled frames or by linearly embedding non-overlapping 3D tubelets.Uniform sampling concatenates nt·nh·nw frame-patch tokens, whereas tubelet embedding fuses spatial and temporal information during tokenisation.
- Model 1: Spatio-temporal attention: The unfactorised spatio-temporal model captures all pairwise token interactions from its first layer but incurs quadratic attention complexity as frame count grows.Video increases the number of tokens linearly with the number of input frames, making this complexity a central motivation for more efficient models.
- Model 2: Factorised encoder: The factorised encoder separates spatial and temporal attention, first producing frame-level representations and then modelling interactions across time with a temporal transformer.This late-fusion design uses separate spatial and temporal encoder blocks and requires fewer FLOPs than the unfactorised model despite having more transformer layers.
- Efficient architectures: The paper also introduces factorised variants that retain efficient spatial-temporal computation, including factorised dot-product attention with Model 1’s parameter count.Factorised dot-product attention computes separate spatial and temporal attention weights using different heads; spatial-then-temporal and temporal-then-spatial ordering were observed to perform equivalently under the stated initialization.
- Training considerations: Because video datasets contain far fewer labelled examples than image datasets, the authors motivate regularisation and image-model pretraining to make large video transformers trainable.At initialization, the 3D convolutional filter behaves like uniform frame sampling while learning to aggregate information across multiple frames during training.
4. Empirical evaluation
Empirical evaluation shows that ViViT’s factorised designs improve efficiency and generalisation while achieving state-of-the-art video-classification results across multiple benchmarks. Performance increases with stronger regularisation, larger backbones, more tokens, higher resolution, and longer temporal context, though fine-grained motion remains challenging.
- Model variants: The unfactorised model performs best on Kinetics 400, whereas the Factorised Encoder performs best on smaller Epic Kitchens, where average pooling performs worst.Factorised variants use fewer FLOPs, and Model 4 adds no parameters while using the least compute.
- Model regularisation: Adding each regulariser produces progressive improvements, yielding a 5.3% overall improvement on Epic Kitchens and a 5% improvement on SSv2.The reported Epic Kitchens models use all regularisers when trained from Kinetics-pretrained models.
- Ablation studies: Larger backbones, more temporal tokens, higher spatial resolution, and more input frames consistently improve accuracy, while increasing computation.For Model 2, FLOPs increase approximately linearly with input frames because computation is dominated by the initial Spatial Transformer.
- Kinetics: 84.9% and 85.8% are achieved on Kinetics 400 and 600, respectively, by the best ViViT-H model pretrained on JFT.The spatio-temporal attention models outperform state-of-the-art methods while requiring significantly fewer views than previous CNN-based methods.
- Benchmark results: ViViT surpasses prior state-of-the-art results on Moments in Time and Epic Kitchens 100, while its Factorised Encoder achieves state-of-the-art Top-1 accuracy on SSv2.On SSv2, the Factorised Encoder outperforms TimeSformer by 2.9%.
- Limitations: SSv2 remains challenging because similar backgrounds and objects require recognition of fine-grained motion patterns, identifying an area for future improvement.The paper observes an inverse correlation between prior methods’ relative performance on SSv2 and Kinetics.
5. Conclusion and Future Work
The paper presents four pure-transformer video-classification models with different accuracy–efficiency profiles, achieving state-of-the-art results across five datasets while enabling training on smaller datasets through regularization. Future work includes removing dependence on image-pretrained models and extending beyond video classification.
- Conclusion: Four pure-transformer models achieve state-of-the-art results across five popular video-classification datasets, with different accuracy and efficiency profiles.The authors also thoroughly ablate their main design choices.
- Conclusion: Effective regularization enables high-capacity transformer models to be trained on smaller datasets.This addresses the challenge of training such models when large datasets are unavailable.
- Future Work: Future work will remove dependence on image-pretrained models and extend the approach to more complex tasks beyond video classification.The paper identifies both directions as clear next steps.
Appendix · A. Additional experimental details
The appendix provides additional experimental details, covering the regularisers used and the training hyperparameters for the experiments.
- A.1 Regularisers: Section A.1 gives additional details about the regularisers used in the experiments.
- A.2 Training hyperparameters: Section A.2 details the training hyperparameters used for the experiments.
A.1. Further details about regularisers
The paper details four training regularisers: stochastic depth, temporally consistent Random augment, label smoothing, and Mixup. These methods modify layer outputs, video augmentations, label distributions, or training examples to regularise video-model training.
- Stochastic depth: Stochastic depth drops a layer’s output with probability pdrop(ℓ) during training by replacing it with the layer input.The dropping probability increases linearly with layer depth, where ℓ indexes the layer and L is the total number of layers.
- Random augment: Random augment sequentially applies data-augmentation transformations while using the same transformation on every video frame.Its hyperparameters are the number of sequential augmentation layers and the shared transformation magnitude.
- Label smoothing: Label smoothing trains on a mixture of the one-hot ground-truth label and a uniform distribution to encourage less confident predictions.The mixture is controlled by a scalar λ ∈[0, 1].
- Mixup: Mixup constructs virtual training examples by taking convex combinations of pairs of training examples and their one-hot labels.The mixing coefficient λ ∈ [0, 1] is sampled from Beta(α, α), with α specified among the experiment hyperparameters.
A.2. Training hyperparameters
All experiments use synchronous momentum SGD, cosine learning-rate decay with linear warmup, and batch size 64. Additional regularization is used only for the smaller Epic Kitchens and Something-Something v2 datasets, with configurations detailed in Table 7.
- All experiments use synchronous SGD with momentum, cosine learning-rate scheduling with linear warmup, and batch size 64.
- Additional regularization is applied only when training on the smaller Epic Kitchens and Something-Something v2 datasets.
- Table 7 details the training hyperparameters for the main experiments and defines “–” as no regularization method used.