Source-linked AI summary
Video Swin Transformer
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, Han Hu
TL;DR
Video recognition Transformers commonly connect patches globally, creating a need for more efficient modeling. This paper adapts Swin Transformer with spatiotemporal local attention, achieving state-of-the-art results across action-recognition and temporal-modeling benchmarks. The authors also note an inconsistency with prior ViViT observations that they plan to analyze after ViViT code is released.
Problem
Global self-attention in video Transformers incurs enormous computation and memory costs because videos require many more tokens than images.
Method
The paper adapts the hierarchical Swin Transformer into a pure-transformer video backbone using 3D shifted-window self-attention and spatiotemporal locality.
Results
The model achieves state-of-the-art performance across Kinetics-400, Kinetics-600, and Something-Something v2, including 84.9% and 86.1% top-1 accuracy on Kinetics-400 and Kinetics-600, respectively.
Takeaways & Limitations
Spatiotemporal locality provides a better speed-accuracy trade-off for video recognition than globally computed self-attention approaches.
Takeaways & Limitations
The authors identify an observation inconsistent with ViViT and defer analysis until ViViT’s code is released.
Abstract
from arXiv · showhide
The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-1 accuracy on Kinetics-400 and 86.1 top-1 accuracy on Kinetics-600 with ~20x less pre-training data and ~3x smaller model size) and temporal modeling (69.6 top-1 accuracy on Something-Something v2). The code and models will be made publicly available at https://github.com/SwinTransformer/Video-Swin-Transformer.
1 Introduction
Video Swin Transformer introduces a pure-transformer video backbone that uses spatiotemporal locality and adapts the hierarchical Swin Transformer structure. It achieves strong results across action-recognition and temporal-modeling benchmarks while using smaller models or pre-training datasets than cited competitors.
- Motivation: Video Swin Transformer uses spatiotemporal locality to approximate full self-attention with lower computation and model size.The approach exploits the tendency of nearby pixels in spatiotemporal distance to be more correlated.
- Method: The architecture extends Swin Transformer’s local attention from the spatial domain to the spatiotemporal domain while retaining its hierarchical structure.This adaptation enables initialization from strong image models pre-trained on large-scale image datasets.
- Training: Using a backbone learning rate of 0.1× the randomly initialized head’s rate helps the pre-trained backbone fit new video inputs while forgetting its parameters and data slowly.The paper links this training observation to better generalization and further study of pre-trained weights.
- Results: 84.9% top-1 accuracy on Kinetics-400 and 86.1% top-1 accuracy on Kinetics-600 slightly surpass ViViT by +0.1/+0.3 points.Swin-L uses 200.0M parameters versus ViViT-H’s 647.5M parameters, with ImageNet-21K rather than JFT-300M pre-training data.
- Results: 69.6% top-1 accuracy on Something-Something v2 improves by +0.9 points over MViT.This result evaluates temporal modeling rather than action recognition on Kinetics.
2 Related Works
Related work traces video recognition from convolutional and complementary self-attention models to pure video Transformers. Earlier Transformer approaches generally use global self-attention, whereas Video Swin Transformer investigates spatiotemporal locality.
- CNN and variants: 3D convolution extends 2D convolution for joint spatial-temporal modeling, while factorized spatial-temporal convolution improves the speed-accuracy trade-off.The cited convolutional approaches are also limited by the small receptive field of convolution operators.
- Self-attention/Transformers to complement CNNs: Self-attention methods such as NLNet, GCNet, and DNL complement CNNs by modeling long-range dependencies through global context.The paper positions its pure-transformer approach as a fuller use of self-attention’s capabilities.
- Vision Transformers: Swin Transformer adds locality, hierarchy, and translation-invariance inductive biases to Transformer backbones for image recognition.These properties motivate adapting Swin Transformer to video recognition.
- Vision Transformers: Video Transformer methods including VTN, TimeSformer, ViViT, and MViT use global self-attention modules with different temporal or spatial factorization strategies.The paper reports that Video Swin Transformer with spatiotemporal locality surpasses these vision Transformers across various video recognition tasks.
3 Video Swin Transformer
Video Swin Transformer extends Swin’s hierarchical local-attention design from images to spatiotemporal video windows. It uses 3D shifted windows to connect neighboring regions while retaining efficient computation and compatibility with pretrained image models.
- Overall Architecture: Video Swin Transformer treats 3D patches as tokens and preserves Swin’s four-stage hierarchy with spatial downsampling but no temporal downsampling.Each token begins from a 2×4×4×3 video patch, and patch merging downsamples spatial dimensions by 2× per stage.
- Overall Architecture: The main video block replaces standard MSA with 3D shifted-window MSA while retaining the feed-forward network, normalization, and residual connections.The block alternates regular and shifted 3D window attention across consecutive layers.
- 3D Window Attention: Local attention limits computation to non-overlapping 3D windows, addressing the enormous cost of global self-attention for videos with many spatial-temporal tokens.For an 8×8×8 token input with 4×4×4 windows, attention is performed within 8 regular windows.
- 3D Shifted Windows: Shifted windows move by (P/2, M/2, M/2) across temporal and spatial axes, introducing connections between neighboring windows in successive layers.The shifted configuration preserves efficient batch computation despite increasing the nominal window count in the illustrative example.
- 3D Shifted Window based MSA Module: The model adds a 3D relative position bias whose compact parameterization covers temporal, height, and width offsets within each window.Queries, keys, and values operate over P M^2 tokens, while the bias table has shape (2P−1)×(2M−1)×(2M−1).
- Initialization from Pre-trained Model: Video Swin variants retain compatibility with pretrained Swin models by adapting the first-stage embedding and relative-position bias shapes for temporal input.The embedding weights are duplicated across the temporal dimension and scaled by 0.5 for initialization.
4 Experiments
Experiments evaluate Video Swin Transformer across action recognition, temporal modeling, and ablations of attention, temporal locality, shifted windows, and initialization. The model reaches strong benchmark performance while locality-based designs improve the speed-accuracy trade-off.
- Comparison to state-of-the-art: 84.9% top-1 accuracy on Kinetics-400 establishes state-of-the-art performance with ImageNet-21K pre-training and about half the computation of ViViT-L.Swin-L (384↑) also uses a smaller pre-training dataset than ViViT-H.
- Comparison to state-of-the-art: 86.1% top-1 accuracy on Kinetics-600 achieves state-of-the-art performance with ImageNet-21K pre-training and about half the computation of ViViT-L.The pre-training dataset is significantly smaller than ViViT-H’s JFT-300M.
- Comparison to state-of-the-art: 69.6% top-1 accuracy on Something-Something v2 surpasses the previous best approach by 0.9%.Swin-B uses a K400 pre-trained model as initialization, while the previous best approach uses K600 pre-training.
- Ablation Study: The joint spatiotemporal attention design provides the best speed-accuracy trade-off, while factorized attention achieves relatively high accuracy with many more parameters.Joint attention computes spatial and temporal attention in the same layer; factorized attention uses separate temporal-only layers.
- Ablation Study: A temporal window size of 8 reduces computation by 17% relative to size 16 while losing only 0.3 performance points.This supports temporal locality as a useful efficiency-accuracy compromise for video recognition.
- Ablation Study: 3D shifted windows improve top-1 accuracy by 0.7%, while temporally shifted windows alone improve it by 0.3%.The results indicate that shifted windows build connections among non-overlapping windows.
5 Conclusion
The paper presents a pure-Transformer video architecture based on spatiotemporal locality and reports state-of-the-art performance across three widely used benchmarks. It also identifies an unresolved discrepancy with prior work pending release of ViViT’s code.
- The proposed pure-Transformer architecture uses a spatiotemporal locality inductive bias.
- The architecture achieves state-of-the-art performance on Kinetics-400, Kinetics-600, and Something-Something v2.
- The model adapts Swin Transformer for video recognition and leverages strong pre-trained image models.
- The authors made the code publicly available to facilitate future research.
- The authors note that one observation conflicts with prior work and defer analysis until ViViT’s code is released.