Source-linked AI summary
Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos
Suman Saha, Gurkirt Singh, Michael Sapienza, Philip H. S. Torr, Fabio Cuzzolin
TL;DR
The paper addresses accurate and efficient detection of multiple concurrent actions in temporally untrimmed videos, where existing pipelines rely on costly, task-independent region proposals and multi-stage processing. It introduces an end-to-end appearance-and-motion framework that links detections into temporally trimmed action tubes, achieving state-of-the-art benchmark performance with substantially faster detection.
Problem
Existing action detectors are limited by resource-demanding unsupervised region proposals, disconnected training objectives, and computationally expensive multi-stage classification and feature extraction.
Method
The framework jointly uses RGB and optical-flow detection networks, fuses their spatially overlapping class scores, and constructs temporally trimmed action tubes through two dynamic-programming passes.
Results
The method significantly outperforms prior state of the art across UCF-101, J-HMDB-21, and LIRIS-HARL while detecting at test time 10× faster than [8] and 5× faster than [34] on J-HMDB-21.
Takeaways & Limitations
The approach supports simultaneous detection, spatial localisation, and temporal detection of multiple concurrent actions in untrimmed videos within one coherent framework.
Abstract
from arXiv · showhide
In this work, we propose an approach to the spatiotemporal localisation (detection) and classification of multiple concurrent actions within temporally untrimmed videos. Our framework is composed of three stages. In stage 1, appearance and motion detection networks are employed to localise and score actions from colour images and optical flow. In stage 2, the appearance network detections are boosted by combining them with the motion detection scores, in proportion to their respective spatial overlap. In stage 3, sequences of detection boxes most likely to be associated with a single action instance, called action tubes, are constructed by solving two energy maximisation problems via dynamic programming. While in the first pass, action paths spanning the whole video are built by linking detection boxes over time using their class-specific scores and their spatial overlap, in the second pass, temporal trimming is performed by ensuring label consistency for all constituting detection boxes. We demonstrate the performance of our algorithm on the challenging UCF101, J-HMDB-21 and LIRIS-HARL datasets, achieving new state-of-the-art results across the board and significantly increasing detection speed at test time. We achieve a huge leap forward in action detection performance and report a 20% and 11% gain in mAP (mean average precision) on UCF-101 and J-HMDB-21 datasets respectively when compared to the state-of-the-art.
1 Introduction
The paper proposes an end-to-end action detection pipeline for multiple concurrent actions in temporally untrimmed videos, addressing limitations in proposal quality and computational cost. It fuses appearance and motion detections, then uses two-pass dynamic programming to construct temporally trimmed action tubes.
- Motivation: Existing detectors rely on unsupervised region proposals and expensive multi-stage classification, limiting accuracy and slowing training and test-time detection.On UCF-101, prior training and feature extraction takes a week on seven Nvidia Titan X GPUs, plus one day for SVM training.
- Approach: The framework uses single-stage deep architectures to predict frame-level detection boxes and action-class scores in an end-to-end fashion.Separate RGB and optical-flow pipelines use RPNs and Fast R-CNN detection networks.
- Approach: Two-pass dynamic programming links detections into class-specific paths spanning the video, then trims them temporally to enforce consistent action labels.The second pass prevents action tubes from being constrained to the full video duration.
- Approach: Appearance and motion detections are fused using class-specific softmax scores and spatial overlaps between detection boxes.The fusion strategy boosts appearance-based detection confidence using motion-based detections.
- Results: The method significantly outperforms the state of the art and detects actions 5 to 10 times faster at test time.Its two-pass tube generation also supports multiple co-occurring action instances in temporally untrimmed videos.
2 Related work
Prior work applies CNNs, video segmentation, and temporal sliding windows to action recognition and localisation, but these approaches have limitations in proposal quality, temporal precision, or computational efficiency. Joint spatial-temporal methods provide a closer precedent, while co-occurring-action coverage remains limited in reviewed approaches.
- Deep learning and localisation: Deep architectures have been applied to action classification, spatial and spatiotemporal localisation, and event detection.
- Video segmentation: Video segmentation methods generate space-time proposals, but reviewed approaches may lack CNN representation power, span long intervals, or fail on actions with small motions.
- Temporal detection: Temporal sliding-window methods address temporal detection but do not jointly solve spatial localisation and suffer from inefficient window processing.
- Joint localisation: Frame-level proposal linking has become prominent for joint spatial and temporal localisation, but prior evaluations may use single-action, temporally trimmed videos.
- Concurrent actions: Reviewed methods have limited co-occurring-action coverage, including caps of two detections per class or narrow datasets and action categories.
3 Methodology
The method combines RGB and optical-flow region detection with score fusion, then constructs and trims action tubes through dynamic programming. This design supports temporally bounded tubes and multiple co-occurring instances of the same action class.
- Detection networks: Two independent RGB and optical-flow pipelines use RPN proposals and Fast R-CNN to produce class-specific refined boxes and scores.The RPN generates proposals, while Fast R-CNN regresses class-specific boxes and assigns multi-class softmax scores.
- Late fusion: Motion detections boost appearance-box scores when their maximum IoU exceeds τ, adding a motion score weighted by spatial overlap.The method sets τ = 0.3 in its tests.
- Action path construction: Action paths maximize unary class scores plus pairwise temporal-overlap terms, favoring high-scoring boxes with substantial overlap between consecutive frames.The overlap potential is the boxes’ IoU, and the maximizing path is found with the Viterbi algorithm.
- Multiple instances: After finding an optimal path, its boxes are removed and the next best path is recursively extracted, enabling multiple co-occurring instances of one action class.This repeated extraction operates on the remaining detection boxes.
- Temporal trimming: A second dynamic-programming pass assigns foreground or background labels smoothly, trimming full-video paths to temporally consistent action segments.The resulting tubes can occupy arbitrary temporal intervals rather than spanning the entire video.
- Tube scoring: Each retained contiguous subsequence becomes an action tube and receives a global score equal to the mean of its top k augmented class scores.One or more tubes may be found per video and action class.
4 Experimental validation and discussion
Experiments on UCF-101, J-HMDB-21, and LIRIS-HARL show strong detection performance, including gains over prior work and benefits from fusion and temporal trimming. The method also substantially reduces test-time detection cost.
- Datasets and protocol: The evaluation covers challenging UCF-101, J-HMDB-21, and LIRIS-HARL datasets using the original papers’ metrics and data splits.UCF-101 includes multiple instances of one action class, J-HMDB-21 contains single trimmed action instances, and LIRIS-HARL includes human-human and human-object interactions.
- UCF-101: 66.75% mAP versus 46.77% from [34] on UCF-101 at δ = 0.2, a 20% gain over the reported baseline.The method also reports 46.35% mAP at δ = 0.4 and results up to δ = 0.6.
- Multiple instances: The method detects multiple concurrent instances and reports UCF-101 class-specific video APs of 88.0, 83.0, and 62.5 for Fencing, SalsaSpin, and IceDancing.These categories concern multiple inherently co-occurring action instances.
- J-HMDB-21: 18% and 11% mAP increases over [8] and [34], respectively, are reported on J-HMDB-21 at δ = .5.Combining motion- and appearance-based detections adds a further 4% at δ = .5, while classification accuracy improves by 8% over [8].
- Efficiency: 10× and 5× faster test-time detection than [8] and [34] are reported on J-HMDB-21, with 10.89 seconds per video versus 113.52 and 52.23 seconds.The comparison averages timing over all videos in J-HMDB-21 split1.
- Ablation and discussion: Fusion improves mAP by 9.4%, 3.6%, and 2.5% on UCF-101, J-HMDB-21, and LIRIS HARL, respectively.The second dynamic-programming pass improves mAP by 20% on LIRIS HARL and 6% on UCF-101.
5 Conclusions and future work
The approach jointly addresses concurrent action recognition, spatial localisation, and temporal detection, while outperforming prior state of the art on challenging benchmarks. Its accuracy and test-time speed support potential real-time applications, with online tube generation planned as future work.
- The framework jointly addresses concurrent multiple-action recognition, spatial localisation, and temporal detection.It combines simultaneous region-proposal detection and classification with improved action-tube generation.
- The method significantly outperforms previous state of the art on challenging benchmark datasets and handles temporally untrimmed videos.
- Its combination of high accuracy and fast test-time detection is promising for real-time applications such as smart car navigation.
- Future work will make tube generation and labelling fully incremental and online by updating dynamic programming as each frame arrives.