Source-linked AI summary
OadTR: Online Action Detection with Transformers
Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, Nong Sang
TL;DR
Online action detection must model long-range temporal structure despite RNNs' nonparallelism and gradient vanishing. OadTR uses a Transformer encoder-decoder with a task token for historical relationships and a decoder for future context, and it reports faster training and inference with stronger performance across three datasets.
Problem
Online action detection requires recognizing current actions from streaming video with inadequate observations, while RNN-based temporal modeling suffers from nonparallelism and gradient vanishing.
Method
OadTR uses an encoder-decoder Transformer whose task token aggregates historical relationships and whose decoder predicts future information for current-action recognition.
Results
OadTR achieves higher training and inference speeds than RNN-based approaches and significantly outperforms state-of-the-art methods on HDD, TVSeries, and THUMOS14.
Takeaways & Limitations
Transformer-based parallel processing provides an effective alternative to recurrent online action detection architectures within the evaluated datasets and tasks.
Abstract
from arXiv · showhide
Most recent approaches for online action detection tend to apply Recurrent Neural Network (RNN) to capture long-range temporal structure. However, RNN suffers from non-parallelism and gradient vanishing, hence it is hard to be optimized. In this paper, we propose a new encoder-decoder framework based on Transformers, named OadTR, to tackle these problems. The encoder attached with a task token aims to capture the relationships and global interactions between historical observations. The decoder extracts auxiliary information by aggregating anticipated future clip representations. Therefore, OadTR can recognize current actions by encoding historical information and predicting future context simultaneously. We extensively evaluate the proposed OadTR on three challenging datasets: HDD, TVSeries, and THUMOS14. The experimental results show that OadTR achieves higher training and inference speeds than current RNN based approaches, and significantly outperforms the state-of-the-art methods in terms of both mAP and mcAP. Code is available at https://github.com/wangxiang1230/OadTR.
1. Introduction
Online action detection must recognize ongoing actions from streaming video with inadequate observations, making long-range temporal modeling essential. OadTR replaces difficult-to-optimize RNN-based processing with a Transformer encoder-decoder that combines historical relationships and predicted future information.
- Motivation: Online action detection identifies ongoing actions as video frames arrive, despite having no access to future observations.The task supports applications including autonomous driving, video surveillance, and anomaly detection.
- Motivation: RNN-based approaches model temporal dependencies but suffer from nonparallelism and gradient vanishing, making optimization difficult.The paper motivates Transformers as an alternative with long-range temporal modeling and better convergence properties.
- OadTR: OadTR uses an encoder-decoder Transformer to learn historical relationships and future information while classifying the current action.A task token summarizes global temporal relationships, while the decoder predicts future actions; their outputs are concatenated for recognition.
- Contributions: OadTR is presented as the first framework in this work to bring Transformers to online action detection.The contribution specifically targets efficient aggregation of long-range historical information and future anticipations.
- Evaluation: Experiments on HDD, TVSeries, and THUMOS14 report that OadTR significantly outperforms state-of-the-art methods, supported by extensive ablation studies.The paper also compares training and inference efficiency with RNN-based approaches.
2. Related Work
Related work covers online action detection methods that model temporal structure, temporal action detection methods that localize action boundaries, and Transformer architectures adapted to vision.
- Online Action Detection: Online action detection identifies actions in live video even when only part of an action has been observed.Prior work includes TVSeries, LSTM-based feedback networks, RED, action-start detection, IDN, and TRN.
- Online Action Detection: Prior online action detection methods use recurrent architectures to model past information, predict future information, or encourage earlier decisions.IDN manipulates a GRU cell, while TRN recursively predicts future information and combines it with current context.
- Temporal Action Detection: Temporal action detection differs by locating the start and end points of action instances in untrimmed videos.Existing approaches include one-stage feature-pyramid networks and two-stage proposal-generation and classification architectures.
- Transformers: Transformer-based vision methods adapt the architecture to tasks such as object detection and image recognition.Examples include DETR, which removes several hand-designed components, and ViT, which processes image patches with a Transformer encoder.
3. Methodology
OadTR is an encoder-decoder Transformer for online action detection that combines long-range historical modeling with parallel future-context prediction. A task token supplies task-related encoder features, while predicted future representations assist current-action classification.
- Overview: OadTR identifies current actions from streaming video by encoding historical observations and predicting future context in parallel.The model uses an encoder for current-frame recognition and a decoder for future prediction, whose outputs provide auxiliary classification information.
- Encoder: A learnable task token is appended to clip features so the encoder can learn global discriminative representations for online action detection.The task token interacts with other tokens and produces the encoder representation used for the task.
- Encoder: Positional encoding is added to the token sequence because self-attention does not inherently preserve frame order.The position encoding is added element-wise to retain temporal positional information.
- Encoder: Multi-head self-attention models long-range temporal dependencies by letting tokens interact and aggregate semantic information through attention weights.Queries, keys, and values are projected from the input, and multiple heads are concatenated before a linear projection.
- Decoder: The decoder processes learnable prediction queries in parallel and uses encoder-decoder cross-attention to extract anticipated future information.The predicted future features are pooled and concatenated with task-related encoder features before classification.
- Training: Training jointly optimizes current-action classification and future-feature prediction, with the balance coefficient λ set to 0.5.The joint objective combines cross-entropy terms for the current action and future-step predictions.
4. Experiments
Experiments on HDD, TVSeries, and THUMOS14 compare OadTR with prior methods and analyze its components and design choices. OadTR achieves strong performance across datasets, while ablations support the roles of task tokens, future prediction, position encoding, and encoder-decoder configuration.
- Compared with state-of-the-art methods: OadTR improves HDD mAP from 29.2% to 29.8% over compared state-of-the-art methods.
- Compared with state-of-the-art methods: On THUMOS14, OadTR improves mAP by 8.3% under TSN-Anet features and 4.9% under TSN-Kinetics features.
- Compared with state-of-the-art methods: OadTR outperforms prior methods at most action portions, including early stages, on the TVSeries dataset.
- Ablation studies: Adding the task token and decoder yields OadTR’s best results, improving over the baseline by 1.1%, 0.6%, and 2.5% on HDD, TVSeries, and THUMOS14.
- Ablation studies: Learned position encoding reaches 29.8% mAP, compared with 28.8% without position encoding and 29.3% with fixed encoding.
- Ablation studies: Sparse Transformer replacement maintains good performance but reduces computational consumption with slight performance degradation.
5. Conclusion
OadTR is a Transformer-based online action detection framework that recognizes current actions by encoding historical information and predicting future context simultaneously. It achieves higher training and inference speeds than RNN-based approaches and better performance than state-of-the-art methods.
- OadTR recognizes current actions by encoding historical information and predicting future context simultaneously.
- OadTR is designed as a direct end-to-end parallel network rather than recursively processing sequences one by one.
- OadTR achieves higher training and inference speeds than current RNN-based approaches.
- OadTR obtains significantly better performance than state-of-the-art methods.