Source-linked AI summary
Temporal Recurrent Networks for Online Action Detection
Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S. Davis, David J. Crandall
TL;DR
Online action detection must recognize actions as frames arrive despite lacking future observations. TRN jointly anticipates immediate future actions and combines predicted future information with historical evidence for present-frame recognition. Across HDD, TVSeries, and THUMOS’14, the method significantly outperforms state-of-the-art approaches, particularly early in actions.
Problem
Real-time applications require detecting actions as each frame arrives using current and historical observations, unlike offline methods that observe the entire video.
Method
TRN jointly predicts future actions and recognizes the current action, combining predicted future information with accumulated historical evidence in one framework.
Results
TRN significantly outperforms state-of-the-art methods across HDD, TVSeries, and THUMOS’14, with greater advantages at earlier action stages and in future-action prediction.
Takeaways & Limitations
Predicted future information can improve online action recognition and learned action representations, even though actual future frames are unavailable at test time.
Takeaways & Limitations
TRN is trained offline with labels for current and future frames, although testing uses predicted future information without accessing actual future frames.
Abstract
from arXiv · showhide
Most work on temporal action detection is formulated as an offline problem, in which the start and end times of actions are determined after the entire video is fully observed. However, important real-time applications including surveillance and driver assistance systems require identifying actions as soon as each video frame arrives, based only on current and historical observations. In this paper, we propose a novel framework, Temporal Recurrent Network (TRN), to model greater temporal context of a video frame by simultaneously performing online action detection and anticipation of the immediate future. At each moment in time, our approach makes use of both accumulated historical evidence and predicted future information to better recognize the action that is currently occurring, and integrates both of these into a unified end-to-end architecture. We evaluate our approach on two popular online action detection datasets, HDD and TVSeries, as well as another widely used dataset, THUMOS'14. The results show that TRN significantly outperforms the state-of-the-art.
1. Introduction
Online action recognition must update predictions from current and past observations, unlike offline recognition, which can use the complete video. TRN addresses this constraint by predicting immediate future actions and combining that information with historical evidence to improve present-frame detection.
- Real-time applications require action predictions to be updated as each frame arrives, using accumulated evidence over time.
- Offline action recognition uses all frames in a short clip, whereas online detection cannot access future observations.
- Jointly modeling current recognition and future anticipation is proposed to learn more discriminative action representations.
- TRN predicts future actions and uses those predictions with historical evidence to recognize the action in the current frame.
- TRN is evaluated on HDD, TVSeries, and THUMOS’14, with reported gains especially when only a fraction of an action is observed.
2. Related Work
Prior work covers offline localization, early recognition, and online detection, but existing online methods focus on current and past observations. TRN extends this setting by jointly learning online detection and immediate-future anticipation.
- Offline Action Detection: Offline action-detection methods observe entire videos to estimate each action’s start and end moments.
- Early Action Detection: Early action detection recognizes actions after observing only a fraction of an event, using methods including structured SVMs and LSTM-based objectives.
- Online Action Detection: Online action detection identifies actions in each arriving frame without using future context, with prior methods modeling current inputs and temporal dependencies.
- Online Action Detection: TRN simultaneously performs online action detection and immediate-future anticipation, using estimated future information to improve present detection.
3. Online Action Detection
Online action detection estimates the action in each arriving frame using only current and historical observations. TRN augments recurrent historical modeling with predicted future actions and jointly trains present detection with future anticipation.
- Problem definition: Online action detection estimates a probability distribution over candidate actions for each frame using only current and past frames.The distribution includes a background class indicating that no action is occurring.
- Temporal Recurrent Network: TRN predicts actions several frames ahead and uses those predictions to classify the action occurring in the present frame.The model processes current features and the previous hidden state while incorporating anticipated future information.
- TRN cell: The TRN cell combines a temporal decoder, future gate, and spatiotemporal accumulator to control internal information flow.The decoder predicts future actions, the future gate embeds decoder states as future context, and the accumulator updates the recurrent state.
- TRN cell: Future context is formed by averaging decoder hidden states and embedding the result with a fully connected layer.The default future gate uses average pooling followed by an FC layer, while other fusion operations are possible.
- Training objective: TRN outputs both the current action estimate and predictions for the next ℓd time steps.Training combines accumulator and decoder losses so online detection and future prediction are optimized jointly.
- Training and inference: At test time, TRN uses predicted future information without accessing actual future frames, preserving its online operation.Training uses labels from current and future frames, whereas inference relies on predicted future information.
4. Experiments
The evaluation compares the online action detector with state-of-the-art and baseline methods on three public datasets. These datasets cover driving, everyday activities, and sports actions in long, untrimmed videos.
- Evaluation setup: Experiments compare the proposed online action detector with multiple state-of-the-art and baseline methods on HDD, TVSeries, and THUMOS’14.The datasets were selected because they contain long, untrimmed videos across diverse perspectives and applications.
- Evaluation setup: HDD contains first-person dashboard-camera driving videos, TVSeries contains television activities, and THUMOS’14 contains sports-related actions.Together, the datasets span driving, everyday activities, and sports applications.
4.1. Datasets
The datasets provide frame-level action annotations across driving, everyday activities, and sports, with varied video conditions and train/test protocols. HDD additionally includes non-visual vehicle sensor readings.
- HDD: HDD contains nearly 104 hours from 137 driving sessions, with frame-level annotations for 11 goal-oriented actions.The recordings use a front-facing vehicle camera and also include Controller Area Network sensor readings.
- TVSeries: TVSeries contains 16 hours from 27 episodes across six television series, annotated frame-by-frame with 30 everyday actions.The dataset includes diverse actions, multiple actors, unconstrained viewpoints, heavy occlusions, and many non-action frames.
- THUMOS’14: THUMOS’14 contains over 20 hours of sports video annotated with 20 actions and uses 200 untrimmed validation videos for training and 213 test videos for evaluation.The trimmed training videos are unsuitable for temporal action detection, so the protocol follows prior work by training on the validation set.
4.2. Implementation Details
TRN was implemented in PyTorch and trained end-to-end with Adam. Training uses sequence sampling with beginning-frame truncation and fixed-length non-overlapping inputs.
- Implementation: The proposed TRN was implemented in PyTorch and experiments used Nvidia Quadro P6000 graphics cards.These details describe the software and hardware environment used for evaluation.
- Optimization and sampling: Training used Adam with learning rate 0.0005, weight decay 0.0005, batch size 32, and end-to-end optimization.Videos were augmented by randomly removing 1 to ℓe initial frames and divided into non-overlapping samples of ℓe consecutive frames.
4.3. Settings
The experiments follow prior state-of-the-art settings across datasets, using established feature inputs and hyperparameters, with dataset-specific sampling and feature extraction procedures.
- The experiments follow prior state-of-the-art settings, including input features and hyperparameters, for fair comparisons.
- Code was planned for public release upon publication.
- HDD: HDD inputs combine visual features from InceptionResNet-V2 with CAN bus sensor values sampled at 3 fps.
- TVSeries and THUMOS’14: TVSeries and THUMOS’14 use 24-fps videos, six-frame chunks, chunk-level decisions, and VGG-16 or two-stream CNN features.
4.4. Evaluation Protocols
Online action detection is evaluated with per-frame precision metrics and with separate measurements across successive action stages to assess early recognition.
- Per-frame mean average precision (mAP) is used to evaluate online action detection.
- Per-frame calibrated average precision (cAP) is additionally used for TVSeries evaluation.
- Calibrated precision adjusts false positives by the negative-to-positive frame ratio, correcting class imbalance.
- Performance is computed separately for each video decile, from the first 10% of action frames through the final interval.
4.5. Baselines
The evaluation compares TRN with frame-based, recurrent, encoder-decoder, and stronger online action-detection baselines across HDD, TVSeries, and THUMOS’14.
- The study compares TRN against multiple baselines to assess its effectiveness.
- CNN baselines: CNN baselines classify each video frame independently without modeling temporal information.
- LSTM baselines: LSTM baselines receive current image features and the previous hidden state to model dependencies between consecutive frames.
- Encoder-decoder baselines: Encoder-decoder baselines encode historical visual information and decode future representations from the encoded features.
- Stronger baselines: Stronger baselines include CDC, 2S-FN, and RED, with RED reported as the strongest existing baseline for online action detection.
- Benchmark tables: The benchmark tables report HDD mAP, TVSeries cAP, and THUMOS’14 mAP comparisons between TRN and baseline methods.
4.6. Results
TRN improves online action detection and anticipation by combining historical evidence with predicted future information. Across HDD, TVSeries, and THUMOS’14, it outperforms baselines, with stronger advantages at earlier action stages.
- Online action detection: TRN outperforms Ramanishka et al. on HDD by 5.4%, 2.8%, and 8.1% mAP with sensor, InceptionResnet-v2, and multimodal inputs, respectively.The performance gaps are larger when sensor data are included.
- Online action detection: TRN exceeds state-of-the-art baselines on TVSeries by 3.0% mcAP with VGG features and 4.5% mcAP with two-stream features.The comparisons are against 2S-FN and RED, respectively.
- Online action detection: TRN outperforms RED and CDC on THUMOS’14 by 1.9% and 2.8% mAP, respectively.
- Ablation studies: TRN-TS improves TVSeries performance to 83.7% versus 75.4%, showing the benefit of incorporating optical-flow motion features.Motion information provides 1.5% improvements in the reported comparison.
- Ablation studies: Predicted future information improves detection: TRN performs comparably to the offline oracle RNN-offline, whose results are 41.6%, 85.3%, and 47.3% on HDD, TVSeries, and THUMOS’14.RNN-offline uses actual future information and is not an online model.
- Ablation studies: Larger decoder horizons do not guarantee better performance because declining anticipation accuracy over longer sequences adds noise to detection features.For baseline comparisons, the decoder horizon is set to 2 video seconds.
- Different action stages: Compared with SVM-FV, TRN-TS gains increase from 6.5% at 100% observed action to 11.8% at 10% observed action on TVSeries.The results indicate a larger advantage when less of the action has been observed.
- Action anticipation: For anticipation, TRN achieves 75.7% mcAP on TVSeries, 38.9% mAP on THUMOS’14, and 32.2% per-frame mAP on HDD over the next 2 seconds.The TVSeries and THUMOS’14 results exceed RED and ED baselines.
5. Conclusion
TRN jointly models historical and predicted future temporal context for online action detection. Experiments on three datasets show improved representations, stronger performance than state-of-the-art methods, and greater advantages at early action stages and in future-action prediction.
- TRN jointly models historical and future temporal context while performing online action detection and future action anticipation.
- Experiments on three popular datasets show that predicted future information improves action representations and online detection performance.
- TRN has greater advantages at earlier action stages and in predicting future actions.
- The estimated-future-information approach may benefit other online tasks, including video object localization and tracking.The paper identifies these applications as future work.