Source-linked AI summary
Anticipative Video Transformer
Rohit Girdhar, Kristen Grauman
TL;DR
Action anticipation requires modeling past action progression to predict future activities rather than recognizing only current visual patterns. AVT uses causal attention and joint future-feature and next-action prediction, achieving strong results across four benchmarks and first place in the EpicKitchens-100 CVPR’21 challenge. Its evaluation follows benchmark setups with dataset-specific anticipation times.
Problem
Action anticipation must predict future activities from sequential past context, which is more challenging than classifying instantaneous visual patterns.
Method
AVT is an end-to-end attention-based architecture using causal temporal attention, attention-based frame encoding, and joint prediction of next actions and future features.
Results
AVT outperforms published prior work on EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads, and wins the EpicKitchens-100 CVPR’21 challenge.
Takeaways & Limitations
Anticipative training objectives improve long-range temporal modeling, with performance gains increasing as more temporal context is incorporated.
Takeaways & Limitations
The evaluation follows challenge benchmark setups in which the anticipation time τa is usually fixed for each dataset.
Abstract
from arXiv · showhide
We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies--both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR'21 challenge.
1. Introduction
AVT addresses action anticipation by modeling the sequential progression of observed video while retaining long-range temporal dependencies. It combines causal temporal attention, predictive future-feature learning, and attention-based frame encoding for next-action prediction.
- 1. Introduction: Action anticipation requires predicting multimodal future activities from past action progression rather than classifying instantaneous visual patterns.This makes anticipation more challenging than traditional action recognition.
- 1. Introduction: Existing approaches often aggregate frame or clip features with limited modeling of sequential evolution, while recurrent models can struggle with long-range dependencies.AVT is motivated by the tradeoff between sequential modeling and long-range context.
- 1. Introduction: AVT replaces aggregation-based temporal modeling with a causal anticipative architecture that processes inputs in parallel, attends over long ranges, and can be rolled out for further predictions.Each frame attends only to preceding frames, while the model predicts future features and actions.
- 1. Introduction: AVT uses an attention-based frame encoder that can attend to spatial features and spontaneously focuses on hands and objects in egocentric video.The spatial and temporal attention mechanisms are unified within the proposed purely attention-based video architecture.
- 1. Introduction: AVT outperforms published prior work on EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads, and wins the EpicKitchens-100 CVPR’21 challenge.It is also ranked #1 on the EpicKitchens-55 seen test set and #2 on the unseen test set.
2. Related Work
Related work spans future-feature prediction, temporal aggregation, affordance and hand-motion modeling, and transformer-based video understanding. AVT differs by providing a causal, end-to-end transformer architecture for predictive anticipation rather than bidirectional recognition.
- 2. Related Work: Action anticipation predicts future actions from a video clip and has gained popularity for first-person video because of wearable-computing applications.Prior approaches learn future-predictive representations, aggregate past features, or use affordances and hand motion.
- 2. Related Work: Self-supervised video representation learning uses signals including temporal consistency, inter-frame predictability, and cross-modal correspondence.AVT incorporates losses encouraging features to predict future features and actions.
- 2. Related Work: Vision self-attention methods have expanded from non-local and gating mechanisms to transformers that replace convolutional architectures for image recognition.Prior video work mostly placed attention architectures on standard spatiotemporal convolutional backbones.
- 2. Related Work: AVT is an end-to-end transformer for video with causal structure aimed at predictive tasks, unlike concurrent bidirectional transformer methods targeting traditional action recognition.The paper reports best results to date on several anticipation benchmarks.
3. Anticipation Problem Setup
The benchmark setup recognizes an action segment using an observed video window that ends a fixed anticipation time before the action begins. The observed-window length may vary, while the anticipation time is typically dataset-specific.
- 3. Anticipation Problem Setup: For an action starting at τs, the model uses the segment from τs −(τa + τo) to τs −τa to recognize it.The observed segment has length τo and ends τa units before the action starts.
- 3. Anticipation Problem Setup: Methods are typically allowed to use any observed-segment length τo, whereas the anticipation time τa is usually fixed for each dataset.
4. Anticipative Video Transformer
AVT is a two-stage, causal attention architecture that predicts future features and actions from observed video while preserving sequential information and long-range dependencies.
- Architecture: AVT uses a backbone to extract frame or clip features, followed by a head that predicts future features and actions.The architecture supports frame-based and spatiotemporal backbones while preventing features at time t from using future information.
- Backbone Network: The purely attention-based AVT-b backbone applies shared Vision Transformer processing to spatial patches in each frame and uses the [class] token as a frame embedding.Temporal position information is incorporated later in the head rather than the frame encoder.
- Head Network: The Causal Transformer Decoder predicts each future feature after attending only to features from frames 1 through t, then maps predictions to action distributions.The final prediction is used for next-action anticipation, with frame sampling typically matched to the anticipation horizon τa.
- Training AVT: AVT jointly trains next-action prediction with intermediate future-feature regression and intermediate action-class supervision when labels are available.Future-feature prediction uses an L2 objective, while the final next-action prediction uses cross-entropy.
- Training AVT: The anticipative training setting uses the model’s causal structure through intermediate supervision, whereas the naive setting supervises only the final next-action prediction.The paper reports that the anticipative setting produces significant improvements.
5. Experiments
AVT is evaluated across four action-anticipation benchmarks, modalities, backbones, and training objectives. It achieves strong state-of-the-art results, with gains linked to end-to-end anticipative training, longer context, and autoregressive rollout.
- Experimental setup: AVT is evaluated on four benchmarks spanning first- and third-person action anticipation.The experiments cover EpicKitchens-100, EpicKitchens-55, EGTEA Gaze+, and 50-Salads.
- State-of-the-art comparisons: AVT improves over prior work on EK100 using matched RGB and object features, with further gains from the AVT-b backbone.The end-to-end AVT-b model outperforms RULSTM by 1.6% in the reported comparison.
- State-of-the-art comparisons: AVT+ outperforms reported EK100 work, including CVPR’21 challenge submissions, with especially large gains on tail classes.AVT+ late fuses RGB and object predictions, while AVT++ ensembles multiple variants.
- State-of-the-art comparisons: AVT outperforms prior work on 50-Salads by 7.3% absolute, demonstrating effectiveness beyond egocentric video.The comparison uses top-1 performance averaged over the standard five splits.
- Ablations and analysis: Anticipative future-prediction losses are imperative for strong performance, with classes requiring action-sequence understanding gaining most.The losses improve both TSN and AVT-b settings, while ‘cook’ shows especially large per-class gains.
- Ablations and analysis: Performance improves with longer temporal context, especially under anticipative training, whose gain is 3.5 points versus 2.1 for naive training.The reported changes are 11.2 → 14.9 for anticipative training and 11.0 → 13.1 for naive training.
- Long-term anticipation: AVT can roll out predictions autoregressively to anticipate sequences of future actions, reusing features computed for previously observed frames.The qualitative rollout predicts ‘wash spoon’ after ‘wash knife’, followed by ‘wash hand’ and ‘dry hand’.
6. Conclusion and Future Work
The paper presents AVT as an end-to-end attention-based architecture for anticipative video modeling and reports state-of-the-art results across four benchmarks. It also identifies future applications beyond anticipation, including self-supervised learning, action-schema discovery, and temporally ordered action recognition.
- AVT is an end-to-end attention-based architecture for anticipative video modeling.
- Experiments across four benchmarks show state-of-the-art results and the importance of anticipative training objectives.
- The authors identify self-supervised learning, action-schema and boundary discovery, and temporally ordered action recognition as future directions.
A. Dataset and Metrics
The evaluation uses established first-person and third-person anticipation datasets with dataset-specific anticipation times, splits, and metrics. The benchmarks include cooking videos, egocentric actions, and third-person activities.
- Datasets: EK100 contains 700 hours-long egocentric cooking videos totaling 100 hours, with 89,977 labeled segments and τa=1s.
- Datasets: EK55 is an earlier 55-hour version of EK100 with 39,596 segments and standard challenge-server evaluation.
- Baselines: Prior EK methods commonly use fixed RGB, optical-flow, and object-detector features extracted from pretrained models.
- Datasets: EGTEA Gaze+ provides 10,325 action annotations across 106 actions and is evaluated on split 1 at τa = 0.5s.
- Datasets: 50-Salads is a third-person dataset with fifty 40-second videos and 17 action classes, evaluated at τa = 1s.
C. EpicKitchens-55 Full Results
On EpicKitchens-55, AVT outperforms prior work on the reported validation and challenge-server comparisons, while remaining slightly behind concurrent work on unseen-test top-1 actions. The anticipative training losses are reported as important for strong AVT performance.
- AVT outperforms all reported work on the EpicKitchens-55 validation set and challenge-server test comparisons.The challenge-server comparison reports top-5 superiority and second place on unseen-test top-1 actions.
- Employing anticipative training losses is reported as imperative for obtaining strong AVT performance on EK55.
- AVT is only second to concurrent work on unseen-test (S2) top-1 actions, whose method uses transductive learning from the test set.
D.1. Per-class Gains
Anticipative training produces especially large gains for some verb classes, particularly “cook” and “choose.” The authors associate these gains with modeling the preceding action sequence, while visualizations also expose timing and object-recognition errors.
- Per-class gains: Predicting that a person will cook often requires understanding preceding actions such as preparing ingredients and turning on the stove.
- Per-class gains: ‘Cook’ and ‘choose’ show particularly significant gains with causal modeling across the TSN and AVT-b backbones.
- Failure cases: Qualitative failure cases often involve anticipating an action too soon or too late, or confusing similar objects such as a spoon and fork.
D.3. Long-term Anticipation
The paper includes visualizations of AVT’s long-term anticipation and evaluates alternative future-feature losses and weights. L2 performs as well as or better than InfoNCE and is used throughout the experiments.
- Long-term anticipation: Figure 9 visualizes long-term anticipation by rolling predictions out autoregressively using predicted future features.The displayed repeat count indicates how many time steps each rolled-out action persists.
- Future-feature loss: L2 is as effective as or better than InfoNCE for Lfeat with both AVT-b and TSN backbones on EK100 validation RGB experiments.
- Future-feature loss: The authors use L2 with weight=1.0 for all experiments.The comparison varies scalar weights for Lfeat during optimization.
D.5. Computational complexity
Anticipative training adds little computational overhead because future-feature matching reuses features already computed for self-attention. Supplementary visualizations show how the model attends spatially and temporally while generating long-term predictions, including reasonable but temporally misaligned failures.
- Computational overhead: Runtime is only 1% higher than a model that predicts only the next action, while GPU memory remains nearly unchanged.
- Computational overhead: Lfeat adds no feature-computation cost because it matches past features already needed by self-attention to predict the next action.
- Inference and qualitative behavior: Inference is unchanged by the additional losses, while qualitative rollouts visualize future predictions and reveal occasional timing errors.