Source-linked AI summary
VidTr: Video Transformer Without Convolutions
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, Joseph Tighe
TL;DR
Video recognition needs efficient global spatiotemporal modeling because convolutional approaches have limited receptive fields and vanilla video transformers have heavy memory demands. VidTr addresses this with stacked separable spatial and temporal attention, plus temporal topK pooling, achieving state-of-the-art or comparable results on five datasets with lower computational requirements and stronger performance on long-term reasoning actions.
Problem
Convolutional video models have limited per-layer receptive fields, while vanilla transformers model raw-pixel videos with quadratic sequence-length memory costs that make them impractical on many devices.
Method
VidTr applies stacked separable spatial and temporal attentions directly to raw video pixels and uses standard-deviation-based topK pooling to remove redundant temporal features.
Results
VidTr achieves state-of-the-art or comparable performance on five datasets with lower computational requirements and latency, and performs especially well on long-term temporal reasoning actions.
Takeaways & Limitations
Global attention and temporal pooling provide an efficient video-classification design, while VidTr features complement local convolutional modeling in an ensemble.
Takeaways & Limitations
The method does not down-sample spatial attention because preliminary experiments found a significant performance drop.
Abstract
from arXiv · showhide
We introduce Video Transformer (VidTr) with separable-attention for video classification. Comparing with commonly used 3D networks, VidTr is able to aggregate spatio-temporal information via stacked attentions and provide better performance with higher efficiency. We first introduce the vanilla video transformer and show that transformer module is able to perform spatio-temporal modeling from raw pixels, but with heavy memory usage. We then present VidTr which reduces the memory cost by 3.3$\times$ while keeping the same performance. To further optimize the model, we propose the standard deviation based topK pooling for attention ($pool_{topK\_std}$), which reduces the computation by dropping non-informative features along temporal dimension. VidTr achieves state-of-the-art performance on five commonly used datasets with lower computational requirement, showing both the efficiency and effectiveness of our design. Finally, error analysis and visualization show that VidTr is especially good at predicting actions that require long-term temporal reasoning.
1 Amazon Web Service; 2 Rutgers University
The authors are affiliated with Amazon Web Service and Rutgers University.
- Amazon Web Service is represented by the listed authors.
- Rutgers University is represented by the listed authors.
- The passage lists author contact domains for both institutions.
1. Introduction
VidTr applies stacked separable attentions directly to raw video pixels for global spatiotemporal aggregation, addressing the efficiency limits of convolutional and vanilla-transformer approaches. It achieves strong performance across video-action benchmarks and particularly supports long-term temporal reasoning.
- VidTr performs global spatiotemporal feature aggregation with separable attention in a transformer-based video classifier.The design applies attention directly to raw video pixels rather than relying on convolutional feature extraction.
- Vanilla video transformers can model spatiotemporal features from raw pixels but require more than 16GB GPU memory for a 16-frame 224 × 224 clip at batch size 1.Their O(n^2) sequence-length complexity makes the approach impractical on many commercial devices.
- 3.3× lower memory consumption is achieved by separable spatial and temporal attention without an accuracy drop.The method further uses standard-deviation-based topK pooling to reduce computation by removing redundant temporal information.
- VidTr achieves state-of-the-art or comparable performance on five of six evaluated datasets with lower computational requirements and latency than previous state-of-the-art approaches.The evaluated datasets include Kinetics 400/700, Charades, Something-something V2, UCF-101, and HMDB-51.
- VidTr performs significantly better than I3D on activities requiring longer temporal reasoning, such as making a cake versus eating a cake.The authors also report roughly a 2% Kinetics 400 improvement when ensembling VidTr with an eight-frame I3D50 model.
- The paper contributes a vanilla video transformer, VidTr variants including compact-VidTr, detailed results on six datasets, and pretrained weights for downstream tasks.
2. Related Work
Prior video-recognition systems largely depend on convolutional architectures, whose efficiency and receptive-field limitations motivate global attention-based alternatives. VidTr differs by learning global features from raw videos without heavily stacked convolutions.
- Early approaches combined 2D convolutions with LSTMs, while segment-based methods such as TSN later achieved good performance without relying on recurrent networks.
- 3D convolutional networks enabled spatiotemporal modeling, but training them from scratch was difficult and later methods commonly inflated weights from 2D networks.
- More effective 3D networks often require larger kernels or deeper structures, while depth convolution reduces computation but increases inference latency.
- TSM and TAM improve temporal-modeling efficiency but do not achieve state-of-the-art Kinetics performance, while neural architecture search remains limited by latency and generalizability.
- VidTr learns global features through attention from the first layer instead of relying on heavily stacked convolutions for feature aggregation.
- Unlike prior transformer applications that aggregate convolutional features, VidTr aggregates spatiotemporal features directly from raw videos without convolutional features.
3. Video Transformer
VidTr applies transformer attention directly to video patches, separating temporal and spatial aggregation to reduce memory, then adds temporal pooling to remove redundant information.
- 3.1. Vanilla Video Transformer: The vanilla video transformer converts raw video frames into embedded spatial patches, adds positional embeddings and a class token, and feeds them to transformer encoders.The class token aggregates features from the sequence for classification.
- 3.2.1. Separable-Attention: VidTr decouples 3D self-attention into temporal and spatial attention, applying temporal attention independently at each spatial location before spatial attention.Temporal down-sampling can reduce the temporal dimension from T to τ, while the class-token intersection supports final classification.
- 3.2.1. Separable-Attention: O(τ^2 + W^2H^2) replaces O(T^2W^2H^2) for the affinity matrix, reducing transformer-layer memory and enabling longer temporal sequences.Spatial attention is not down-sampled because preliminary experiments showed a significant performance drop.
- 3.2.2. Temporal Down-sampling method: Temporal down-sampling removes redundant frames, but uniform pooling can miss informative frames because attention is often concentrated on a small temporal subset.The proposed topK std pooling selects frames with strong localized attention and removes frames with uniform attention; experiments report better performance than average pooling or convolution.
4.1. Datasets
The evaluation covers six widely used video action-recognition datasets, including Kinetics 400, Kinetics 700, and Something-Something V2.
- 4.1. Datasets: Kinetics 400 and Kinetics 700 contain approximately 240K and 650K training videos, with 20K and 35K validation videos across 400 and 700 action categories.Both datasets contain videos trimmed to 10 seconds, and evaluation reports top-1 and top-5 validation accuracy.
- 4.1. Datasets: Something-Something V2 contains 174 actions, with 168.9K training videos and 24.7K evaluation videos evaluated using top-1 accuracy.
4.2. Kinetics 400 Results
On Kinetics-400, VidTr matches or exceeds competing video models while reducing computation, latency, and memory through global separable attention and compact temporal down-sampling.
- Comparison To SOTA: VidTr achieves state-of-the-art performance against I3D-based architectures with lower GFLOPs and latency.At roughly 200 GFLOPs, VidTr-M exceeds I3D50 by 3.6%, NL50 by 2.1%, and TPN50 by 0.9%.
- Comparison To SOTA: At 78% top-1 accuracy, VidTr-S uses 6× fewer FLOPs than NL-101, 2× fewer than TPN, and 12% fewer than Slowfast-101.VidTr also outperforms I3D-based networks at higher sample rates; X3D-XXL is the only reported network that outperforms VidTr.
- Compact VidTr: Temporal down-sampling removes roughly 56% of VidTr’s computation with only a 2% accuracy drop.Compact VidTr spans models from 39 GFLOPs to 79.1% accuracy and achieves better or similar performance than compact baselines, including TEA and TEINet.
- Ablations: Spatio-temporal separable attention reduces memory by 3.3× relative to joint attention while retaining performance, and temporal modeling is critical for attention-based architectures.Smaller spatial patches achieve 77.7 top-1 accuracy versus 71.2 for larger patches, despite increasing sequence length and memory demands.
- Ablations: Starting temporal down-sampling after the first encoder layer provides the best performance–FLOPs trade-off.Down-sampling immediately causes a significant accuracy drop, while later down-sampling slightly improves performance at higher FLOPs.
- Efficiency: VidTr-S balances accuracy, FLOPs, and latency better than several competing models, while X3D-L has about 3.6× higher latency.VidTr-S uses 5× fewer FLOPs than I3D101-NL, 12% fewer than Slowfast101, 2× fewer than TPN101, and 20% fewer than CorrNet50 at similar performance.
4.3. More Results
VidTr shows strong results across Kinetics-700 and Charades, while its performance varies with dataset size, pretraining, and model complementarity. The figure visualizes spatial and temporal attention patterns for comparison with I3D activation.
- +9%: VidTr-S significantly outperformed the baseline I3D model on Kinetics-700.
- VidTr-M matched SlowFast101 8 × 8, while VidTr-L was comparable to previous SOTA SlowFast101-nonlocal on Kinetics-700.
- +0.6%: Ensembling VidTr-L with I3D improved performance on Kinetics-700.
- On Charades, VidTr-L outperformed LFB and NUTA101 and was comparable to SlowFast101-NL, but underperformed X3D-XL, likely because transformers overfit on the relatively small dataset.
- Figure 3 compares VidTr spatial and temporal attention with I3D activation, including attention over informative frames and regions.
5. Visualization and Understanding VidTr
Attention rollout indicates that VidTr selects informative spatial regions and removes redundant temporal information, with spatial attention becoming stronger in deeper layers.
- Spatial attention focuses on informative regions, while temporal attention skips duplicated or nonrepresentative information.
- Spatial attention strengthens in deeper VidTr layers, specifically across the visualizations of the 4th, 8th, and 12th layers.
- Early layers do not capture meaningful temporal instances, whereas later layers show stronger spatial attention.
6. Conclusion
The paper presents VidTr as a stacked separable-attention architecture for video action recognition. Experiments and error analysis report state-of-the-art or comparable performance on five public datasets, especially for actions requiring long-term reasoning.
- VidTr uses separable attention in a novel stacked-attention architecture for video action recognition.
- VidTr achieved state-of-the-art or comparable performance on five public action-recognition datasets.
- Error analysis shows VidTr is especially effective at modeling actions requiring long-term reasoning.
- Future work includes combining VidTr and convolution for local-global action modeling and applying self-supervised training to large-scale data.