PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Space-Time Attention All You Need for Video Understanding?

Gedas Bertasius, Heng Wang, Lorenzo Torresani

arXiv:2102.05095v4cs.CV

TL;DR

Video understanding has largely relied on convolutional operators, raising whether self-attention alone can provide an effective alternative. TimeSformer adapts Vision Transformer to space-time video patches and finds divided attention effective, achieving state-of-the-art benchmark results with low inference cost and support for clips over one minute.

  • Problem

    Video modeling still relies on 2D or 3D convolutions, motivating whether a performant convolution-free architecture can instead use self-attention.

  • Method

    TimeSformer adapts Vision Transformer to video by applying self-attention to frame-level patches across space and time, including separate temporal and spatial attention.

  • Results

    TimeSformer achieves state-of-the-art action-recognition results, with 78.0% top-1 accuracy at low inference cost and applicability to clips over one minute.

  • Takeaways & Limitations

    Space-time self-attention provides an effective, scalable alternative to convolution-based video networks for long-term video modeling.

  • Takeaways & Limitations

    Training TimeSformer from scratch is difficult and achieved only 64.8% video-level accuracy on Kinetics-400 without ImageNet pretraining.

Abstract

from arXiv · show

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/facebookresearch/TimeSformer.

1. Introduction

The introduction motivates replacing video convolutions with self-attention, drawing on videos’ sequential, context-dependent structure and the potential advantages of Transformer architectures. It proposes TimeSformer, a Vision Transformer adaptation that represents videos as frame-level patches and evaluates scalable space-time attention designs.

  • Motivation: Videos and sentences are sequential, and short-term actions may require long-range contextualization for full disambiguation.This analogy motivates applying long-range self-attention models from NLP to video modeling.
  • Motivation: The work asks whether a performant convolution-free video architecture can replace convolution operators with self-attention.The authors argue this design could overcome inherent limitations of convolutional models for video analysis.
  • Motivation: Transformers are motivated by their faster training and inference than CNNs, especially for high-resolution and long videos.This could enable larger learning capacity within the same computational budget.
  • Method: TimeSformer adapts Vision Transformer to video by extending self-attention from image space to the space-time 3D volume.It views a video as patches extracted from individual frames, with each patch linearly mapped into an embedding.
  • Method: The authors evaluate several scalable self-attention designs over the space-time volume because all-pairs token similarity is costly for videos.Among the proposed schemes, the introduction identifies a “divided” design as the best.

2. Related Work

Prior work explored self-attention for image classification alongside or instead of convolution, while video recognition methods generally used Transformers on convolutional feature maps. The paper positions its approach as the first video recognition architecture using self-attention exclusively.

  • Image classification: Image-classification research has used self-attention both with convolution and as a full replacement for it.Non-Local Networks generalize Transformer self-attention through a non-local mean, while other work proposes 2D self-attention competitive with 2D convolution.
  • Image classification: Image networks replacing convolution restrict self-attention to local neighborhoods or apply it globally only after heavy image downsizing.These constraints manage computational cost and memory because individual pixels serve as queries.
  • Video recognition: Prior video applications used Transformers for generation or on top of convolutional feature maps, rather than as the exclusive building block for video recognition.Such feature-map-based uses included action localization and recognition, video classification, and group activity recognition.

3. The TimeSformer Model

TimeSformer processes video as a sequence of frame-level patches with positional and classification embeddings, then applies stacked Transformer blocks for self-attention-based classification. Its Divided Space-Time Attention factorizes temporal and spatial attention, reducing comparisons and improving classification accuracy over alternatives.

  • Input representation: The model takes F RGB frames, decomposes each frame into N non-overlapping P × P patches, and feeds their embeddings into a Transformer sequence.The patches span each frame, with N = HW/P^2.
  • Classification embedding: A learnable classification token is prepended, and the final token representation is passed through a 1-hidden-layer MLP to predict video classes.Patch embeddings include learnable positional embeddings encoding each patch’s spatiotemporal position.
  • Space-Time Self-Attention Models: Spatial-only attention neglects dependencies across frames and produces degraded classification accuracy compared with full spatiotemporal attention.The degradation is especially pronounced on benchmarks requiring strong temporal modeling.
  • Space-Time Self-Attention Models: Divided Space-Time Attention applies temporal attention followed by spatial attention within each Transformer block, rather than joint spatiotemporal attention.The temporally attended representation is reused to compute spatial attention before the MLP produces the block encoding.
  • Space-Time Self-Attention Models: (N+F +2) comparisons per patch are required by Divided Attention, compared with (NF + 1) for joint spatiotemporal attention.The factorization is reported as both more efficient and more accurate in experiments.

4. Experiments · 4.1. Analysis of Self-Attention Schemes · 4.2. Comparison to 3D CNNs

Experiments show that divided space-time attention offers the best accuracy among evaluated schemes while scaling more efficiently than joint attention. Compared with 3D CNNs, TimeSformer combines lower inference and training costs with performance that depends on pretraining and video-data scale.

  • 4. Experiments: TimeSformer is evaluated on Kinetics-400, Kinetics-600, Something-Something-V2, and Diving-48 using a Base ViT pretrained on ImageNet-1K or ImageNet-21K.Unless otherwise indicated, experiments use 8 × 224 × 224 clips with frames sampled at 1/32.
  • 4.1. Analysis of Self-Attention Schemes: Divided space-time attention achieves the best accuracy on both K400 and SSv2 among the five evaluated attention schemes.Its separate temporal and spatial parameters provide larger learning capacity than joint space-time attention.
  • 4.1. Analysis of Self-Attention Schemes: Divided space-time attention scales gracefully as spatial resolution or video length increases, whereas joint attention incurs dramatically higher computational cost.Joint attention can cause GPU memory overflow when spatial frame resolution increases.
  • 4.2. Comparison to 3D CNNs: 121.4M parameters and 0.59 TFLOPs characterize TimeSformer, compared with 34.6M parameters and 1.97 TFLOPs for SlowFast 8x8 R50.I3D 8x8 R50 has 28.0M parameters and 1.11 TFLOPs, indicating higher inference cost despite fewer parameters.
  • 4.2. Comparison to 3D CNNs: 3 840 Tesla V100 GPU hours are required by SlowFast 8 × 8 R50, versus 448 GPU hours for TimeSformer on Kinetics-400.With similar budgets, SlowFast reaches 70.0% accuracy and I3D reaches 71.0% accuracy.
  • 4.2. Comparison to 3D CNNs: ImageNet pretraining is needed because TimeSformer’s large parameter count makes training from scratch difficult, unlike SlowFast, which can train from video alone at high cost.The experiments initialize TimeSformer with ImageNet weights before video training.
  • 4.2. Comparison to 3D CNNs: ImageNet-21K pretraining consistently improves K400 accuracy over ImageNet-1K, while both pretraining choices produce similar SSv2 accuracy.The study includes default, high-resolution, and long-range TimeSformer variants operating on 8, 16, and 96 frames, respectively.
  • 4.2. Comparison to 3D CNNs: TimeSformer outperforms competing models on every K400 training subset, but is strongest on SSv2 only with 75% or 100% of the full data.The SSv2 pattern is attributed to its more complex temporal reasoning requirements.

4.3. Varying the Number of Tokens

TimeSformer’s scalability supports increasing spatial resolution and clip length, both of which increase the Transformer’s input-token count. Higher resolution and longer clips improve accuracy, with evaluation reaching 96 frames despite GPU memory limits.

  • Token count and scalability: Higher spatial resolution increases patches per frame, while more frames increase the Transformer’s input-token sequence length.The study separately varies spatial resolution and clip length to investigate these effects.
  • Accuracy trends: Increasing spatial resolution up to a certain point boosts performance, and increasing clip length produces consistent accuracy gains.Findings are reported in Figure 5 for Kinetics-400 clip-level accuracy.
  • Accuracy trends: 96 frames is the longest tested clip length because GPU memory constraints prevented evaluation on longer clips.Using 96-frame clips is described as a significant departure from current convolutional models, which are typically more limited.

4.4. The Importance of Positional Embeddings · 4.5. Comparison to the State-of-the-Art

Space-time positional embeddings yield the best reported accuracy on both Kinetics-400 and Something-Something-V2. Across benchmarks, TimeSformer combines competitive accuracy with low inference cost and runtime, while excelling on Kinetics-600 and Diving-48 but trailing the strongest Something-Something-V2 models.

  • 4.4. The Importance of Positional Embeddings: Space-time positional embeddings produce the best accuracy on both Kinetics-400 and Something-Something-V2 among the evaluated variants.The comparison includes no positional embedding, space-only positional embedding, and space-time positional embedding.
  • 4.5. Comparison to the State-of-the-Art: 3 views reduce inference cost, while TimeSformer-L reaches 80.7% top-1 accuracy and default TimeSformer reaches 78.0% at the lowest inference cost among recent state-of-the-art models.The standard comparison uses 10 temporal clips and 3 spatial crops, totaling 30 space-time views.
  • 4.5. Comparison to the State-of-the-Art: 36 minutes, 1.06 hours, and 2.6 hours are the measured runtimes of TimeSformer, TimeSformer-HR, and TimeSformer-L, versus 14.88 hours for SlowFast.The measurements use 20K Kinetics-400 validation videos and 8 Tesla V100 GPUs.
  • 4.5. Comparison to the State-of-the-Art: TimeSformer-L achieves its best performance without multiple temporal clips, whereas X3D and SlowFast require at least 5 clips to approach their top accuracy.The comparison evaluates K ∈{1, 3, 5, 10} temporal clips, each with a single spatial crop.
  • 4.5. Comparison to the State-of-the-Art: TimeSformer-L achieves its best performance on about 12 seconds of a Kinetics video using a single clip.This result follows the comparison of multiple temporal clips during inference.
  • 4.5. Comparison to the State-of-the-Art: TimeSformer achieves lower accuracy than the best models on Something-Something-V2, while its Diving-48 result outperforms reproduced SlowFast 16×8 R101 by a substantial margin.For Something-Something-V2 and Diving-48, models use ImageNet-1K pretraining; the Diving-48 comparison is restricted because of a label issue.

4.6. Long-Term Video Modeling

TimeSformer is evaluated for long-term task classification on a 120K-video, 1059-category subset of HowTo100M. It substantially outperforms matched SlowFast baselines, with longer-range variants performing best and strong results under direct ImageNet pretraining.

  • Dataset and setup: The evaluation uses 120K HowTo100M videos spanning 1059 task categories, split into 85K training videos and 35K testing videos.Only categories with at least 100 video examples are included.
  • Long-term classification results: TimeSformer outperforms corresponding SlowFast R101 baselines by 8 −11% at the same single clip coverage.Both model families use four configurations with 8, 32, 64, or 96 frames sampled at 1/32 frame rate, pretrained on Kinetics-400 before finetuning.
  • Long-term classification results: The longest-range TimeSformer variant achieves the best video-level classification accuracy, indicating suitability for long-term video modeling.Inference averages predictions from non-overlapping clips sampled to cover each video's full temporal extent.
  • ImageNet-only pretraining: With ImageNet-1K pretraining, TimeSformer reaches top-1 accuracies of 52.8, 58.4, 59.2, 59.4 for 8, 32, 64, 96 frame inputs, respectively.With ImageNet-21K pretraining, the corresponding top-1 accuracies are 56.0, 59.2, 60.2, 62.1.

4.7. Additional Ablations

Additional ablations show that the default Base ViT with P = 16 and the proposed temporal-then-spatial attention order outperform larger or smaller models, larger patches, and reversed attention order. The authors attribute these declines to insufficient dataset scale, reduced spatial granularity, and attention-order sensitivity, respectively.

  • Smaller & Larger Transformers: 1% worse results on both Kinetics-400 and Something-Something-V2 were obtained with the Large ViT compared with the Base model.The authors suspect current datasets are not large enough to justify increasing capacity beyond the Base model’s 121M parameters.
  • Smaller & Larger Transformers: About 5% worse accuracies than the default Base ViT were produced by the Small ViT variant.
  • Larger Patch Size: About 3% worse results than the default P = 16 variant were produced by using P = 32 patches.The authors conjecture that the decrease is due to reduced spatial granularity and did not test P values below 16 because of much higher computational cost.
  • The Order of Space and Time Self-Attention: 0.5% lower accuracy on both Kinetics-400 and Something-Something-V2 resulted when spatial attention preceded temporal attention.The proposed Divided Space-Time Attention applies temporal attention first, followed by spatial attention.

4.8. Qualitative Results

Qualitative visualizations indicate that TimeSformer learns attention focused on relevant video regions for complex spatiotemporal reasoning. Its divided space-time attention also produces more semantically separable features than space-only attention or ViT.

  • Visualizing Learned Space-Time Attention: Attention visualizations suggest that TimeSformer focuses on relevant video regions to perform complex spatiotemporal reasoning.The visualizations use Attention Rollout on Something-Something-V2 videos.
  • Visualizing Learned Feature Embeddings: t-SNE visualizations show that divided space-time attention learns more semantically separable features than space-only attention or ViT.Each point represents a video, with colors indicating action categories.

5. Conclusion

The paper introduces TimeSformer as a convolution-free video architecture built exclusively on space-time self-attention. It is presented as conceptually simple, accurate on major action-recognition benchmarks, and efficient in training and inference.

  • TimeSformer is a fundamentally different video-modeling approach from established convolution-based video networks.
  • TimeSformer provides an effective and scalable video architecture built exclusively on space-time self-attention.
  • TimeSformer achieves state-of-the-art results on major action-recognition benchmarks while maintaining low training and inference cost.

Appendix · A. Implementation Details

The TimeSformer implementation uses PySlowFast and pytorch-image-models, with specified training, initialization, inference, comparison-model, and dataset procedures. These details cover optimization schedules, crops, sampling, attention weights, prediction aggregation, baselines, and benchmark composition.

  • A. Implementation Details: TimeSformer is implemented with PySlowFast and pytorch-image-models packages.The appendix specifies training and inference procedures for the model.
  • A. Implementation Details: 15 epochs use an initial learning rate of 0.005, divided by 10 at epochs 11 and 14.Training resizes the shorter video side randomly to [256, 320] before sampling a 224 × 224 crop.
  • A. Implementation Details: TimeSformer-HR uses random shorter-side resizing to [448, 512] followed by a random 448 × 448 crop.This is the high-resolution training configuration.
  • A. Implementation Details: A frame rate of 1/32 samples clips from full-length videos, with batch size 16 and synchronized SGD across 32 GPUs.Momentum is 0.9 and weight decay is 0.0001.
  • A. Implementation Details: The default model is the “Base” ViT, and temporal and spatial attention layers share ViT-derived initial weights.Unless otherwise noted, experiments use the Base ViT model.
  • A. Implementation Details: Inference samples one middle temporal clip, takes three spatial crops, and averages their softmax scores for the final prediction.The shorter side is scaled to 224 pixels, or 448 for TimeSformer-HR; crop sizes are 224×224 or 448 × 448.
  • A. Implementation Details: Kinetics-400, Kinetics-600, Something-Something-V2, and Diving-48 span 400, 600, 174, and 48 action categories, respectively.Their training and validation or testing sets contain 240K/20K, 392K/30K, 170K/25K, and 16K/3K videos, respectively.
Loading 2102.05095v4…