VidTr: Video Transformer Without Convolutions

Introduction

• VidTr applies stacked attentions directly to raw video pixels, addressing convolution-based architectures’ limited receptive fields and slow information aggregation.The paper presents the vanilla video transformer as a proof of concept for pixel-level spatio-temporal modeling.Source: §1 · pp. 1–2
• Vanilla video transformers require over 16GB for a batch-size-1, 16-frame 224 × 224 clip; separable attention cuts memory by 3.3× without accuracy loss, while topK std pooling selects representative frames.The pooling operation reduces sequence length and is motivated by temporal redundancy in videos.Source: §1 · p. 1
• VidTr achieved state-of-the-art or comparable performance on five of six datasets with lower computational requirements and latency, and outperformed I3D on longer-temporal-reasoning activities.The evaluation covered Kinetics 400/700, Charades, Something-something V2, UCF-101, and HMDB-51, as well as a sixth commonly used dataset.Source: §1 · p. 1
• Ensembling VidTr with 8-frame I3D50 improved Kinetics 400 performance by roughly 2%; attention rollout [1] showed spatial attention focusing on informative patches and temporal attention reducing redundant instances.Source: §1 · p. 2
• The authors provide detailed results and pretrained model weights across six datasets for use as a reference and in downstream tasks.The proposed model family includes VidTr permutations, including compact-VidTr with reduced computational costs through standard-deviation-based pooling.Source: §1 · p. 2

2.1. Action Classification

• Early action recognition used 2D convolutions; LSTMs modeled ConvNet features without significantly better performance, while TSN and variants performed well.Source: §2.1 · p. 2
• 3D spatiotemporal convolution began with and C3D [48]; I3D [7] introduced 2D-weight inflation, enabling varied backbones [10] and many subsequent 3D designs [19] [31] [32] [27] [33] [39] [44].Source: §2.1 · p. 2
• 3D networks often require larger kernels or deeper structures for better performance; depth convolution reduces computation [49] but raises inference latency, and TSM [37], TAM [17], and NAS [18] [43] retain performance or generalizability limitations.TSM and TAM did not achieve state-of-the-art Kinetics performance; NAS methods still faced high latency and limited generalizability.Source: §2.1 · p. 2
• VidTr avoids heavily stacked, sliding, and depth convolutions, learning global features from the first layer with fewer FLOPs and lower latency than 3D convolutions [18].Earlier global spatiotemporal modeling efforts [31] remained limited by convolutional backbones.Source: §2.1 · p. 2

2.2. Vision Transformer

• Vision transformers were used to bridge modalities [20] [36], aggregate convolutional features [5] [11] [61] [15] [21], and learn features from raw pixels [14] [47].Source: §2.2 · p. 2
• Unlike [21], which attends over convolutional features, VidTr learns spatiotemporal features jointly from raw video, with lower computational cost and higher performance than framewise ViT [41].The paper describes this as pioneering spatiotemporal feature aggregation from raw video without convolutional features.Source: §2.2 · p. 2
• Against concurrent work [4], VidTr's split attention achieved better performance without larger video resolution or longer clips; later work explored multi-scale and other attention factorizations [2] [4] [16] [40] [42].Source: §2.2 · p. 2

3.1. Vanilla Video Transformer

• The vanilla transformer embeds raw-video spatial patches, adds learnable positional embeddings and a class token, and learns 3D motion features from local patch sequences.The class token aggregates features from the sequence for classification.Source: §3.1 · p. 3, §3.2 · p. 3

3.2.1 Separable-Attention

• VidTr decouples 3D self-attention into temporal attention at each spatial location and spatial attention, using the intersection of spatial and temporal class tokens for classification.The model does not down-sample spatial attention because preliminary experiments showed a significant performance drop.Source: §3.2.1 · pp. 3–4
• Separable attention reduces the layer’s affinity matrix from O(T^2W^2H^2) to O(τ^2 + W^2H^2), addressing the vanilla transformer’s quadratic memory cost in sequence length.The vanilla affinity map requires T^2× memory for training, making it impractical on most available GPUs.Source: §3.2 · p. 3, §3.2.1 · p. 4

3.2.2 Temporal Down-sampling method

• TopK standard-deviation pooling ranks temporal attention-matrix rows to select informative features for down-sampling, while preserving the aggregation token.It targets clips where attention concentrates on a small set of features rather than being distributed uniformly over time.Source: §3.2.2 · p. 4

3.3. Implementation Details

• VidTr uses ViT-B initialization, trains for 50 epochs with SGD, and evaluates with 30-crop testing across 10 temporal segments and three spatial crops per segment.Training uses random crops and horizontal flips except on Something-something; the learning rate starts at 0.01 and drops tenfold after epochs 25 and 40.Source: §3.3 · pp. 4–5

4.1. Datasets

• Kinetics-400 contains approximately 240K training and 20K validation videos across 400 categories, and evaluation reports top-1 and top-5 accuracy.Videos are trimmed to 10 seconds.Source: §4.1 · p. 5
• Kinetics-700 contains approximately 650K training and 35K validation videos across 700 categories, and evaluation reports top-1 and top-5 accuracy.Videos are trimmed to 10 seconds.Source: §4.1 · p. 5
• Something-Something V2 comprises 168.9K training and 24.7K evaluation videos across 174 actions, with top-1 accuracy reported following prior evaluation setup [37].Source: §4.1 · p. 5
• Charades has 9.8K training and 1.8K validation videos, 157 multilabel classes, and performance measured by mean Average Precision (mAP).Videos average about 30 seconds.Source: §4.1 · p. 5
• UCF-101 contains 13,320 videos averaging 180 frames across 101 action categories, with top-1 classification reported on validation videos using split 1.Source: §4.1 · p. 5
• HMDB-51 contains 6,766 videos across 51 action categories, with top-1 classification reported on validation videos using split 1.Source: §4.1 · p. 5

4.2.1 Comparison To SOTA

• At roughly 200 GFLOPs, VidTr-M outperforms I3D50 by 3.6%, NL50 by 2.1%, and TPN50 by 0.9%.At 78% top-1 accuracy, VidTr-S uses 6× fewer FLOPs than NL-101, 2× fewer than TPN, and 12% fewer than Slowfast-101; X3D-XXL is the only network reported to outperform VidTr.Source: Table 2 · p. 5, §4.2.1 · p. 5

4.2.2 Compact VidTr

• The compact VidTr’s temporal down-sampling removes roughly 56% of computation with only a 2% accuracy drop.The compact family spans models from 39 GFLOPs to 79.1% accuracy; at similar performance, VidTr improves on TEA by 0.6% with 16% fewer FLOPs and TEINet by 0.5% with 11% fewer FLOPs.Source: §4.2.2 · pp. 5–6, Table 3 · p. 6

4.2.3 Error and Ensemble Analysis

• VidTr-S improves accuracy on long-context actions but performs worse on fast local motions, while ensembling it with I3D50 improves performance by roughly 2%.VidTr-S gains 26% accuracy over I3D50 on “making a cake” but is 21% worse on “shaking head”; the ensemble adds 37G FLOPs and beats the improvement from combining two 3D networks.Source: §4.2.3 · p. 6

4.2.4 Ablations

• Square spatial patches outperform cubic spatio-temporal patches, and 16^2 patches outperform 32^2 patches despite the latter reducing sequence length by 4×.Cubic patches use fewer FLOPs but score 73.1 versus 75.5; 16^2 patches score 77.7 versus 71.2 for 32^2 patches, which reduce affinity-matrix memory by 16×.Source: §4.2.4 · pp. 6–7
• VidTr’s spatio-temporal separable attention uses 3.3× less memory than joint attention without accuracy loss, while axial attention drops performance.Spatial-only attention performs worse than temporal models, and STD-based topK pooling outperforms temporal average pooling and convolution-based down-sampling.Source: §4.2.4 · p. 7
• Across backbones, T2T has the lowest FLOPs but lowest accuracy, while ViT-L matches ViT-B at 3× the FLOPs; down-sampling after the first encoder layer gives the best trade-off.Starting down-sampling at the very beginning gives 72.9 versus 74.9; consecutive down-sampling layers give 73.9 versus 74.9, with skipping one layer offering the best trade-off.Source: §4.2.4 · p. 7

4.2.5 Run-time Analysis

• VidTr-S balances accuracy, FLOPs, and latency, using 5× fewer FLOPs than I3D101-NL and 20× fewer than CorrNet50.X3D-L has about 3.6× higher latency than VidTr-S, which the authors attribute to depth convolution.Source: Figure 2 · p. 7, §4.2.5 · p. 8

Kinetics-700 Results

• VidTr-S outperformed I3D by +9% on Kinetics-700, while VidTr-M and VidTr-L were comparable to Slowfast101 8 × 8 and Slowfast101-nonlocal, respectively.VidTr-L ensembled with I3D gained +0.6%; the small gap from Slowfast-NL [19] is attributed to its pretraining on Kinetics-400 and 600 versus VidTr’s Kinetics-400 only.Source: §4.3 · p. 8

Charades Results

• On Charades, VidTr-L outperformed LFB and NUTA101, matched Slowfast101-NL, and its I3D ensemble achieved SOTA at 40.3 mAP.VidTr remained below X3D-XL, likely due to overfitting on the relatively small dataset; the authors note transformer-based networks may overfit more easily than convolutional models [14].Source: §4.3 · p. 8

Something-something V2 Results

• VidTr performed poorly on Something-Something V2, which the authors attribute to pure transformers modeling local motion less effectively than convolutions.They identify improving local-motion modeling as future work.Source: §4.3 · p. 8

UCF and HMDB Results

• After 6 epochs, VidTr achieved 96.6% on UCF-101 and 74.4% on HMDB-51, comparable to state-of-the-art performance on both small datasets.The authors interpret these results as evidence that VidTr generalizes well to smaller datasets.Source: §4.3 · p. 8

5. Visualization and Understanding VidTr

• VidTr’s spatial attention focuses on informative regions, while temporal attention skips duplicated or nonrepresentative information.These patterns were visualized using attention roll-out [1].Source: §5 · p. 8
• Spatial attention is stronger in deeper VidTr layers, while early temporal attention does not capture meaningful instances because it relies on spatial information.The visualizations compare the 4th, 8th, and 12th layers.Source: §5 · p. 8
• For catching fish, I3D misclassified the action as sailing while focusing on people and water, whereas VidTr predicted correctly and attended to action-related regions across time.Source: §5 · p. 8

• VidTr achieves state-of-the-art or comparable performance on five public action recognition datasets and is especially good at modeling actions that require long-term reasoning.Source: §6 · p. 9
• VidTr is a novel stacked attention-based architecture for video action recognition that uses separable attention.Source: §6 · p. 9
• Future work includes combining VidTr with convolution for better local-global action modeling [38] [57] and using self-supervised training [9] on large-scale data.Source: §6 · p. 9