Source-linked AI summary
Online Real-time Multiple Spatiotemporal Action Localisation and Prediction
Gurkirt Singh, Suman Saha, Michael Sapienza, Philip Torr, Fabio Cuzzolin
TL;DR
Existing spatio-temporal action systems are typically offline and too slow for real-world online use. This paper combines SSD frame detection with incremental action-tube construction, achieving state-of-the-art localisation and early prediction, including real-time operation up to 40fps.
Problem
Existing spatio-temporal action detectors assume complete videos and are too computationally expensive for real-time online processing.
Method
The framework uses SSD networks for frame-level box regression and classification, then incrementally builds and labels action tubes online.
Results
The method achieves superior spatio-temporal localisation and early action prediction performance while operating in real time at up to 40fps.
Takeaways & Limitations
The framework supports online localisation and early prediction for multiple actions in untrimmed videos, including UCF101-24.
Takeaways & Limitations
Performance on JHMDB may be constrained by the dataset’s relatively small size, which limits the expressive-power benefits of SSD models.
Abstract
from arXiv · showhide
We present a deep-learning framework for real-time multiple spatio-temporal (S/T) action localisation, classification and early prediction. Current state-of-the-art approaches work offline and are too slow to be useful in real- world settings. To overcome their limitations we introduce two major developments. Firstly, we adopt real-time SSD (Single Shot MultiBox Detector) convolutional neural networks to regress and classify detection boxes in each video frame potentially containing an action of interest. Secondly, we design an original and efficient online algorithm to incrementally construct and label `action tubes' from the SSD frame level detections. As a result, our system is not only capable of performing S/T detection in real time, but can also perform early action prediction in an online fashion. We achieve new state-of-the-art results in both S/T action localisation and early action prediction on the challenging UCF101-24 and J-HMDB-21 benchmarks, even when compared to the top offline competitors. To the best of our knowledge, ours is the first real-time (up to 40fps) system able to perform online S/T action localisation and early action prediction on the untrimmed videos of UCF101-24.
1. Introduction
The paper targets real-time, online spatio-temporal action localisation, where existing systems are offline, computationally expensive, and unsuitable for incremental video processing. It combines SSD-based frame detection with online tube construction to support localisation, classification, and early prediction.
- Existing spatio-temporal action detectors assume the entire video is available and cannot process frames incrementally at real-time speed.This limits their applicability to settings such as video surveillance and human-robot interaction.
- Earlier CNN-based systems rely on expensive optical flow, region proposals, or independently trained multi-stage components that limit deployment and can produce sub-optimal solutions.The cited methods include separate motion and appearance CNNs followed by one-vs-rest SVMs, or RPN-RCNN pipelines.
- The proposed framework uses SSD as a single-stage, end-to-end trainable detector for frame-level action boxes and class-specific confidence scores.SSD eliminates the region proposal generation step while addressing frame-level accuracy and speed.
- A novel single-pass online tube-building method updates multiple action tubes frame by frame, including their action-specific scores and labels.The incremental design supports online localisation and early action prediction from partially observed videos.
- Up to 40fps detection is reported, with performance exceeding offline methods at realistic detection thresholds and surpassing the state of the art for early action prediction.The framework also reports online prediction and localisation on partially observed untrimmed videos.
2. Related work
Prior work addresses spatial, temporal, or spatio-temporal action detection using diverse supervised and proposal-based methods, but real-time spatial-temporal localisation remains largely unresolved. The paper positions its framework as the first work to address real-time action localisation.
- Deep learning has been applied to action classification, spatial localisation, temporal localisation, spatio-temporal localisation, and event detection.
- Spatial localisation methods include segmentation and region-proposal approaches, with some restricted to temporally trimmed videos.The cited example combines Selective Search, CNN features, and one-vs-rest SVMs.
- Temporal detection methods use sliding windows, 3D CNNs, LSTMs, or dynamic programming, and some operate online without addressing spatial localisation.
- Spatio-temporal localisation has been studied under supervised, semi-supervised, and weakly supervised settings, including proposal methods based on dense trajectories.Dense-trajectory proposals can fail on actions characterised by small motions.
- Real-time action-classification systems may be fast but lack spatial localisation, while earlier online prediction work is limited by trimmed videos or non-real-time segmentation.
- The paper identifies its contribution as the first work to address real-time action localisation.
3. Methodology
The methodology combines two-stream frame-level detection with an efficient online procedure for incrementally building, matching, and temporally labelling multiple action tubes. The design supports online processing while trading optical-flow accuracy against real-time speed.
- Detection network: The framework processes appearance and optical-flow frames with an integrated single-stage CNN that predicts bounding boxes and class-specific confidence scores.The network unifies region proposal generation, box prediction, and class-score estimation in one end-to-end trainable component.
- Optical flow: Real-time optical flow is used at test time for speed, while a more accurate optical-flow alternative can be selected when accuracy is prioritised.Separate networks are trained for the two optical-flow algorithms, with transfer learning from accurate-flow results to initialise the real-time-flow network.
- Detection fusion: Appearance and flow detections are merged with fusion strategies to improve robustness and accuracy, while retaining unmatched flow boxes can help localise concurrent actions.The rationale is especially relevant for UCF-101 classes with multiple simultaneous action instances.
- Online action tube generation: The online tube algorithm builds multiple action tubes for each class in parallel by associating detections frame by frame and temporally trimming each tube.Tubes are treated as tracklets, with greedy matching followed by online temporal labelling.
- Temporal labelling: Temporal trimming assigns each tube box either its action class or background, then uses an online Viterbi dynamic program to obtain an optimal binary labelling.The pairwise potential penalises label changes, and the optimal labelling can be generated at any time in linear time.
4. Experiments
Experiments evaluate early prediction and online localisation on UCF101-24 and J-HMDB-21, showing strong accuracy, realistic-threshold gains, and real-time operation across input modes.
- Experimental setup: The framework is evaluated for early action prediction and online spatio-temporal localisation on UCF101-24 and J-HMDB-21.It follows prior experimental setups and reports performance against video observation percentage using AUC and mAP.
- Online localisation: The proposed method outperforms Soomro et al. on online action localisation across IoU thresholds δ = 0.2 and 0.5 and all video observation percentages.Its performance remains stable over observation time, whereas the competitor requires warm-up time and slightly declines at the end.
- Global performance: 70.2% mAP versus 66.6% for [33] at δ = 0.2 in online real-time UCF101-24 localisation.Using accurate flow in non-real-time mode reaches 73.5% mAP, comparable to.
- Global performance: 46.2% mAP at δ = 0.5 versus 32.1% for and 36.4% for [33], with the real-time mode also outperforming both competitors.The gain is reported at a more demanding detection threshold.
- Discussion: Real-time flow improves UCF101-24 performance by 2.1% at δ = 0.5 and J-HMDB-21 performance by 5%, while accurate flow adds further gains.Real-time optical flow provides less improvement than accurate flow but retains real-time speed.
- Discussion: Online tube generation performs better than the evaluated offline method, especially at higher detection thresholds, alongside higher-quality SSD detections.The authors attribute the increase to both SSD detections and the new online tube-generation method.
- Discussion: The SSD boost on J-HMDB-21 is limited by the dataset’s relatively small size, which may not exploit SSD models’ expressive power.The authors suggest that cross-validating CNN hyperparameters could improve performance.
5. Conclusions and future plans
The framework performs concurrent multiple-action localisation and prediction in real time, combining high accuracy with fast test-time detection. Future work includes faster motion representations, detectors, and online tracking methods.
- The framework addresses concurrent multiple human action recognition, spatial localisation, and temporal detection in real time.
- Its high accuracy and fast test-time detection speed support applications including autonomous driving, human-robot interaction, and surgical robotics.
- Future extensions include replacing optical flow with motion vectors, incorporating faster frame-level detectors such as YOLO, and exploring more sophisticated online tracking.