Source-linked AI summary
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, João F. Henriques
TL;DR
Video transformers need temporal correspondences because fixed spatial locations may not track moving objects or camera motion. The paper introduces trajectory attention for implicit motion-path aggregation and Orthoformer for more efficient attention, achieving state-of-the-art results across video benchmarks. Its approach still has transformer data-efficiency and training-speed limitations, and prototype selection bottlenecks runtime.
Problem
Video transformers often treat time like space, despite moving physical points becoming unrelated across frames; modeling temporal correspondences matters for dynamic scenes.
Method
The paper introduces trajectory attention, which uses attention-derived correlations to aggregate information along implicitly determined motion paths, together with Orthoformer to approximate self-attention efficiently.
Results
The combined method achieves state-of-the-art results on four video action recognition benchmarks, with trajectory attention outperforming other approaches by +2% on SSv2.
Takeaways & Limitations
Motion-path aggregation provides a video-specific inductive bias, while the approximation reduces the memory and computation demands of trajectory attention.
Takeaways & Limitations
The approach retains transformer models’ poor data efficiency and slow training, while prototype selection remains a runtime bottleneck because it is not easily parallelized.
Abstract
from arXiv · showhide
In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame $t$ may be entirely unrelated to what is found at that location in frame $t+k$. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers -- trajectory attention -- that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something--Something V2, and Epic-Kitchens datasets. Code and models are available at: https://github.com/facebookresearch/Motionformer
1 Introduction
Video transformers often ignore that physical points move across frames, while trajectory attention models information along implicitly determined motion paths. The paper also introduces Orthoformer to reduce attention’s computational and memory costs.
- Video transformers’ generic attention can be statistically inefficient because videos contain rich temporal information alongside redundant spatial information.
- Spatial locality is useful for images because nearby pixels often belong to the same 3D object, but video points project along 2D trajectories as they move.
- Existing video transformers pool across the full space-time volume or axially over time, disregarding object and camera motion trajectories.
- Trajectory attention uses attention’s correlation volume to guide pooling along motion paths and assemble features from relevant regions.
- Orthoformer approximates self-attention to reduce computational and memory costs, complementing trajectory attention in video transformers.
- The combined contributions achieve state-of-the-art results on four video action recognition benchmarks.
2 Related Work
Prior work spans learned video representations, vision transformers, efficient attention, and explicit temporal correspondence methods. Trajectory attention differs by learning correspondences implicitly rather than explicitly establishing them.
- Video representations and 3D-CNNs: Video representation research progressed from hand-crafted features such as SIFT-3D, HOG3D, and IDT toward end-to-end deep architectures including 3D-CNNs.
- Vision transformers: Vision transformers adapt the transformer architecture to image patches, with methods such as DeiT improving data efficiency through distillation and augmentation.
- Attention for video recognition: Video recognition methods have applied self-attention uniformly across space and time, whereas trajectory attention treats these dimensions differently.
- Efficient attention: Efficient-attention research reduces self-attention’s quadratic time and memory costs through sparse, hashing-based, and linear mechanisms evaluated by Long Range Arena.
- Temporal correspondences and optical flow: Optical flow, tracking, and data association establish explicit temporal correspondences, while this work facilitates implicit correspondence learning through trajectory attention.
3 Trajectory Attention for Video Data
Trajectory attention models video dynamics by first forming spatially weighted trajectory tokens across frames, then pooling those trajectories temporally. Orthoformer approximates attention through prototypes to reduce quadratic computation and memory costs.
- Model setup: The model represents videos as ST tokens with positional encodings and combines trajectory attention with standard transformer components and a classification token.Its transformer layers use layer normalization, multi-head attention, residual connections, and feed-forward networks.
- Trajectory attention: Trajectory attention constructs probabilistic paths for each space-time reference point by spatially comparing a query with keys independently across frames.The resulting trajectory tokens represent information pooled according to trajectory probabilities.
- Trajectory attention: Unlike joint or divided space-time attention, trajectory attention follows motion paths rather than treating spatial and temporal dimensions independently.Divided attention reduces complexity but permits only independent analysis of time and space, whereas trajectory attention links locations across frames.
- Trajectory attention: A second 1D temporal attention operation pools the trajectory tokens across time to reason about connections along each trajectory.The updated reference query contains information spatially pooled from the reference frame before temporal pooling.
- Complexity: The full trajectory-attention operation has quadratic complexity O(S^2T^2), making it slower than divided space-time attention without approximation.Fast approximations are introduced to address this computational limitation.
- Orthoformer approximation: Orthoformer decomposes attention through prototype-key and query-prototype matrices, replacing full O(N^2) attention with O(N) complexity when the prototype count R is constant.The two smaller matrix products use R × N and N × R attention matrices instead of the full N × N matrix.
4 Experiments
Experiments evaluate Motionformer’s tokenization, positional encoding, trajectory attention, and Orthoformer approximation on action-recognition and long-range benchmarks. The results show stronger motion-sensitive recognition, efficient attention approximation, and benefits from temporally shared prototypes and larger prototype sets.
- Input encoding: Something–Something V2 accuracy improves 1% with cubic rather than square tokenization, using matched token counts over the same temporal range.The comparison uses 8×224×224 inputs for patch tokenization and 16×224×224 inputs for cubic tokenization.
- Input encoding: 4% higher Something–Something V2 accuracy results from separate rather than joint space-time positional encoding.Separating positional encodings helps distinguish spatial and temporal dimensions for joint space-time attention.
- Attention block comparisons: Trajectory attention outperforms joint and divided space-time attention, with a +2% advantage on the motion-focused Something–Something V2 dataset.The improvement is modest on appearance-oriented Kinetics-400 but significant on Something–Something V2.
- Orthoformer approximated attention: Orthoformer effectively approximates self-attention, outperforming state of the art on Long Range Arena with 64 prototypes and lower computational and memory requirements.It performs comparably with Nyströmformer while using a reduced memory footprint and extends beyond video to long text and high-resolution images.
- Orthoformer ablations: 4% average performance improvement occurs on both datasets when Orthoformer prototypes increase from 16 to 128.Orthoformer performance improves monotonically with the number of prototypes.
- Orthoformer ablations: Temporally shared prototypes improve Something–Something V2 performance by 2% while reducing memory usage 5×.The authors attribute these gains to a regularization effect from leveraging redundant information across frames.
- Comparison to the state of the art: Motionformer improves top-1 accuracy by 1.0% on Something–Something V2 and 2.3% on Epic-Kitchens Nouns against current methods.The comparison covers Kinetics-400, Kinetics-600, Something–Something V2, and Epic-Kitchens, including models much larger than Motionformer.
5 Conclusion
The paper introduces trajectory attention for motion-aware video representation and Orthoformer to reduce attention’s resource demands, reporting strong benchmark performance alongside important computational limitations.
- Trajectory attention aggregates video information along implicitly determined motion trajectories rather than treating space and time equivalently.The design supplies a motion-based inductive bias for video data.
- The method achieves state-of-the-art results on video action-recognition benchmarks including Kinetics-400/600, Something–Something V2, and Epic-Kitchens.Table 6 reports comparisons using GFLOPS and accuracy metrics across these datasets.
- Orthoformer approximates attention to reduce memory requirements, addressing the largest bottleneck for transformer models.The approximation is intended to mitigate trajectory attention’s higher computational complexity.
- The approach remains limited by transformer models’ poor data efficiency and slow training, while prototype selection bottlenecks runtime.These limitations persist despite reduced memory and computation from the approximation.
- Trajectory attention is proposed for future applications including tracking, temporal action localization, and online action detection.These settings are identified as avenues for future work beyond video action classification.
- Video transformers have quadratic computational scaling with sequence length, creating environmental and resource costs compared with 3D-CNNs.The proposed linear-complexity approximation is intended to mitigate these computational requirements, while video recognition also carries surveillance misuse risks.
6.1 Further experimental analysis and results
Further experiments examine motion reliance, class-specific effects, trajectory maps, and computational comparisons, showing that trajectory attention is especially useful when fine-grained motion matters.
- Motion cues: SSv2 is more motion-reliant than Kinetics: single-frame classification performs well on Kinetics but very poorly on SSv2.This makes SSv2 a stronger test of combined appearance and motion reasoning.
- Motion-sensitive performance: +2% is the trajectory-attention advantage on the motion cue-reliant SSv2 dataset over the other compared approaches.The comparison contrasts trajectory attention with joint and divided space-time attention as temporal stride increases.
- Trajectory maps: Trajectory attention maps appear to implicitly track query points across time along learned motion paths.The qualitative maps are shown for Kinetics-400 and Something-Something V2 sequences.
- Computational comparison: 416 GPU hours is the reported training cost for the baseline Mformer-B, matching the cited TimeSformer-B cost while outperforming it by over 1%.Larger Motionformer variants required 1334 and 1376 GPU hours, respectively.
- Transfer evaluation: Motionformer features obtain a competitive J&F-Mean of 60.6 on DAVIS 2017 video object segmentation.The evaluation uses frozen features and attention maps in a label-propagation setting.
6.2 Implementation details
The experiments use standardized video-transformer training, augmentation, evaluation, and computing pipelines, with implementation choices specified for datasets, schedules, and resource usage.
- Preprocessing: Training clips contain 16×224×224 inputs sampled at rate 1/4 from 30 FPS videos, giving an effective temporal resolution of just over 2 seconds.Inputs are normalized to [−1, 1] and augmented with scale jitter, flips, color jitter, and additional rand-augment for smaller datasets.
- Training: All datasets use AdamW with weight decay 5 × 10^-2, per-GPU batch size 4, label smoothing alpha 0.2, and mixed precision training.
- Evaluation: The DAVIS 2017 video object-segmentation evaluation reports mean region similarity Jm and mean contour-based accuracy Fm.
- Training: Kinetics-400/600 and Something-Something V2 use 35 epochs with learning-rate decay at epochs 20 and 30, while Epic-Kitchens uses 50 epochs with decay at 30 and 40.
- Long Range Arena: Long Range Arena experiments reuse Nyströmformer’s training, validation, testing code, and parameters, with Performer ported to PyTorch.
- Computing resources: Ablation experiments use 32 GPUs for about 12 hours, while state-of-the-art comparisons use 64 GPUs for about 7 hours.
- Libraries: The implementation uses timm for the base vision transformer and PySlowFast for training, data processing, and evaluation.