Source-linked AI summary
Space-time Mixing Attention for Video Transformer
Adrian Bulat, Juan-Manuel Perez-Rua, Swathikiran Sudhakaran, Brais Martinez, Georgios Tzimiropoulos
TL;DR
Video Transformers must model temporal information without the high cost of full space-time attention. This paper uses local temporal attention and efficient space-time mixing to achieve spatial-only complexity, with strong recognition accuracy and efficiency relative to other Video Transformers.
Problem
Full space-time attention makes Video Transformers computationally expensive, while existing temporal modeling often adds substantial overhead over spatial-only attention.
Method
The model restricts attention to a local temporal window, mixes space-time information efficiently, and adds lightweight global temporal-only attention mechanisms.
Results
The model has complexity O(TS^2), matches the baseline computational cost, and significantly outperforms recent Video Transformers in accuracy/FLOP efficiency.
Takeaways & Limitations
Efficient local space-time attention can provide strong video recognition while preserving the computational cost of spatial-only attention.
Abstract
from arXiv · showhide
This paper is on video recognition using Transformers. Very recent attempts in this area have demonstrated promising results in terms of recognition accuracy, yet they have been also shown to induce, in many cases, significant computational overheads due to the additional modelling of the temporal information. In this work, we propose a Video Transformer model the complexity of which scales linearly with the number of frames in the video sequence and hence induces no overhead compared to an image-based Transformer model. To achieve this, our model makes two approximations to the full space-time attention used in Video Transformers: (a) It restricts time attention to a local temporal window and capitalizes on the Transformer's depth to obtain full temporal coverage of the video sequence. (b) It uses efficient space-time mixing to attend jointly spatial and temporal locations without inducing any additional cost on top of a spatial-only attention model. We also show how to integrate 2 very lightweight mechanisms for global temporal-only attention which provide additional accuracy improvements at minimal computational cost. We demonstrate that our model produces very high recognition accuracy on the most popular video recognition datasets while at the same time being significantly more efficient than other Video Transformer models. Code will be made available.
1 Introduction
Video recognition seeks to recognize events in video, but full space-time attention is computationally expensive. The paper proposes an efficient Video Transformer that approximates space-time attention while retaining temporal modeling.
- Video recognition targets events of interest in video sequences, such as human activities.
- Full space-time attention has complexity O(T^2S^2), making video Transformers computationally heavy.
- Prior approaches achieve promising recognition accuracy but often add significant computational overhead through temporal modeling.
- The proposed Video Transformer has complexity O(TS^2), matching spatial-only attention, and improves efficiency over recently proposed alternatives.
- It restricts temporal attention to a local window, uses efficient space-time mixing, and adds lightweight global temporal-only attention mechanisms.
2 Related work
Video recognition has evolved from CNN-based approaches toward Transformer architectures, motivated by their success in image recognition. Video Transformers reduce the cost of full space-time attention through factorization or alternative approximations.
- Traditional video recognition methods use 2D or 3D CNNs, with 2D approaches aggregating frame-based features temporally.
- Vision Transformers established convolution-free Transformer architectures as state-of-the-art models for ImageNet image classification.
- Video Transformers extend Vision Transformers to video but must reduce the prohibitive cost O(T^2S^2) of full space-time attention.
- TimeSformer factorizes attention into alternating spatial and temporal operations with complexity O(T^2S + TS^2).
- The proposed work instead uses local temporal attention and shift-trick-based joint space-time attention.
3 Method
The method approximates full space-time self-attention with local temporal aggregation and efficient space-time mixing while retaining spatial-only complexity. Lightweight temporal aggregation mechanisms further exchange information across frames at low cost.
- Video Transformer: The model represents video as TS visual tokens with separate spatial and temporal positional embeddings, alongside a classification token.Each frame is divided into patches, embedded linearly, and augmented with learned positional information.
- Computational objective: Full space-time attention has complexity O(T^2S^2), so the method targets the dominant attention cost while preserving the rest of the Transformer computation.The full model’s attention term is identified as the main computational bottleneck.
- Efficiency: The proposed attention has complexity O(TS^2), matching spatial-only attention and avoiding additional complexity relative to the baseline.The paper presents this as more efficient than previously proposed Video Transformers while providing a better approximation to full space-time attention.
- Local temporal attention: Local attention restricts each token’s temporal receptive field, while Transformer depth expands coverage from [−tw, tw] to [−ktw, ktw].For tw = 1 and k = 4, the receptive field spans [−4, 4], covering a typical T = 8 clip.
- Space-time mixing: Space-time mixing aggregates temporal information by indexing channels from adjacent frames, replacing multiple window attentions with a single attention computation.The shift trick performs dimensionality reduction, space-time mixing, and attention without the otherwise costly projection.
- Global temporal aggregation: X-ViT adds lightweight global temporal aggregation through temporal attention or summary tokens, with one temporal-attention layer costing O(T) and summary tokens adding O(TS).Temporal attention operates on classification tokens, while summary tokens exchange frame-level information at intermediate layers.
4 Results
Experiments evaluate the proposed model across multiple video datasets and ablations, showing that local space-time mixing and lightweight temporal aggregation improve accuracy while preserving efficiency. The model achieves competitive or state-of-the-art recognition with substantial speed advantages over other Video Transformers.
- Ablation studies: A local temporal window of tw = 1 performs best, while the proposed model substantially outperforms the spatial-only baseline.The authors attribute degradation at tw = 2 partly to boundary effects and use tw = 1 thereafter.
- Ablation studies: 62.5% to 64.4% Top-1 accuracy results from replacing temporal averaging with one Temporal Attention layer, whereas Temporal Attention alone scores 56.6%.Additional Temporal Attention layers provide no further benefit, indicating that both space-time mixing and global temporal aggregation contribute to the final result.
- Ablation studies: Key-value space-time mixing outperforms alternative mixing variants by up to 2%, and mixing 25%–50% of channels is optimal.Using 100% adjacent-frame channels degrades performance because same-frame attention is excluded.
- Scaling studies: Increasing frames from 8 to 16 improves SSv2 Top-1 accuracy by 1.8%, while increasing to 32 adds only 0.2% with linear complexity growth.Similar diminishing-return behavior is observed on Kinetics and Epic-100.
- Comparison to state-of-the-art: The model matches or sets state-of-the-art results while running 20× faster than ViViT and 8× faster than TimeSformer-L on Kinetics-400.On SSv2 it reaches 90.8% Top-5 accuracy versus ViViT's 89.8% while using models that are 14× or 9× faster; Epic-100 also shows state-of-the-art results with improved verb accuracy.
5 Conclusions
The paper presents an efficient approximation to full space-time attention for video recognition and compares it with state-of-the-art methods on Epic-100. Its approach matches the computational cost of spatial-only attention while remaining more efficient than recent Video Transformers.
- Table 8 compares the model with state-of-the-art methods on Epic-100.
- The paper applies a novel, efficiently implementable approximation to full space-time attention for video recognition.
- The approximation has the same computational cost as spatial-only attention while producing a more efficient Video Transformer than recent alternatives.
- The authors identify detection, segmentation, non-pretrained models, alternative architectures, and low-power deployment as directions for future work.