Source-linked AI summary
Cascaded Boundary Regression for Temporal Action Detection
Jiyang Gao, Zhenheng Yang, Ram Nevatia
TL;DR
Temporal action detection must localize action intervals despite sliding windows that may not cover complete actions or may include background. The paper introduces a two-stage detector with Cascaded Boundary Regression, which iteratively refines boundaries in both stages. CBR achieves state-of-the-art performance on THUMOS-14 and TVSeries, including an improvement from 19.0% to 31.0% for map@tIoU=0.5 on THUMOS-14.
Problem
Sliding-window action detectors can omit parts of action instances or include extraneous background, limiting temporal boundary precision.
Method
CBR uses temporal coordinate regression within a two-stage pipeline, feeding refined clips back for further boundary refinement in each stage.
Results
CBR achieves state-of-the-art performance on proposal generation and action detection on THUMOS-14 and TVSeries, improving map@tIoU=0.5 on THUMOS-14 from 19.0% to 31.0%.
Takeaways & Limitations
Cascaded boundary refinement is effective for both temporal proposal generation and action detection.
Takeaways & Limitations
Unit-level coordinate regression is easier to learn than frame-level regression but produces coarser temporal boundaries.
Abstract
from arXiv · showhide
Temporal action detection in long videos is an important problem. State-of-the-art methods address this problem by applying action classifiers on sliding windows. Although sliding windows may contain an identifiable portion of the actions, they may not necessarily cover the entire action instance, which would lead to inferior performance. We adapt a two-stage temporal action detection pipeline with Cascaded Boundary Regression (CBR) model. Class-agnostic proposals and specific actions are detected respectively in the first and the second stage. CBR uses temporal coordinate regression to refine the temporal boundaries of the sliding windows. The salient aspect of the refinement process is that, inside each stage, the temporal boundaries are adjusted in a cascaded way by feeding the refined windows back to the system for further boundary refinement. We test CBR on THUMOS-14 and TVSeries, and achieve state-of-the-art performance on both datasets. The performance gain is especially remarkable under high IoU thresholds, e.g. map@tIoU=0.5 on THUMOS-14 is improved from 19.0% to 31.0%.
1 Introduction
The paper addresses imprecise temporal localization from sliding-window action detection by introducing Cascaded Boundary Regression in a two-stage pipeline. CBR refines boundaries iteratively and achieves state-of-the-art results on THUMOS-14 and TVSeries.
- Sliding-window detectors may miss parts of actions or include background, producing imprecise temporal intervals.
- CBR feeds regressed clips back into each pipeline stage for further temporal boundary refinement.The cascade allows the system to observe different content during successive refinement rounds.
- The two-stage model generates class-agnostic temporal proposals first and detects specific actions from those proposals second.
- CBR is evaluated on both proposal generation and action detection using the THUMOS-14 and TVSeries datasets.
- map@tIoU=0.5 on THUMOS-14 improves from 19.0% to 31.0%.
2 Related Work
Related work situates temporal action detection alongside temporal proposal generation and object detection. Prior approaches use proposal networks, recurrent models, and object-proposal techniques to support localization.
- Temporal action detection methods include two-stage proposal-and-classification frameworks and recurrent networks for encoding temporal information.
- Temporal proposal generation seeks efficient, accurate class-agnostic proposals using visual features, convolutional networks, or LSTM video encoding.
- Object detection frameworks commonly build detection systems on object proposals generated by handcrafted features or spatial regression.
3 Methods
The method models video clips from unit-level features and refines temporal boundaries through a two-stage proposal-and-detection pipeline. Cascaded regression feeds refined clips back for further refinement, using coordinate regression, shared parameters, and multi-task training.
- Video Clip Modeling: Video clips are represented by concatenating mean-pooled internal-unit features with mean-pooled surrounding-context features.Context units before and after a clip provide temporal information for boundary inference.
- Video Clip Modeling: Multi-scale temporal sliding windows scan videos, while clip-level features reuse unit-level computations.Window length and overlap define the sliding-window scales.
- Temporal Coordinate Regression: Temporal coordinate regression predicts boundary offsets directly or through parameterized center-and-length coordinates.Non-parameterized regression can operate at frame or unit level; unit-level regression is easier to learn but produces coarser boundaries.
- Two-Stage Proposal and Detection Pipeline: The two-stage pipeline first generates class-agnostic proposals, then predicts action categories and category-specific boundary offsets.The proposal network outputs offsets and an actionness score; the detection network outputs category scores and offsets.
- Cascaded Boundary Regression: Each stage applies cascaded boundary regression by feeding refined clips back into the same process for further refinement.Proposal and detection cascades use Kp and Kd steps respectively, with shared network parameters across steps.
- Training: CBR is trained with a multi-task loss combining classification and temporal coordinate regression, using sliding-window samples and L1 regression distance.Training uses Adam, learning rate 0.005, batch size 128, and λ=2 for both proposal and detection networks.
4 Evaluation
CBR is evaluated on THUMOS-14 and TVSeries for proposal generation and action detection, with experiments examining regression offsets, cascade depth, and comparisons with state-of-the-art methods.
- Evaluation setup: CBR evaluation covers temporal action proposal generation and action detection on THUMOS-14 and TVSeries.The experiments use standard benchmarks and include both stages of the detection pipeline.
- Coordinate regression: All three coordinate-offset variants improve action localization over no boundary regression, while unit-level offsets outperform frame-level offsets.The compared variants are parameterized, non-parameterized frame-level, and non-parameterized unit-level offsets.
- THUMOS-14 cascade experiments: 42.7 to 45.0 for two-stream features and 38.6 to 39.6 for C3D features are reported as performance increases in the proposal cascade experiment.Two-stream CBR performs best at Kp_c = 3, after which performance drops slightly.
- THUMOS-14 cascade experiments: 28.4 to 31.0 for two-stream features and 21.5 to 22.7 for C3D features are reported in the detection cascade experiment.Performance becomes saturated after Kd_c = 2.
- State-of-the-art comparisons: CBR-FL outperforms TURN-FL for temporal proposal generation, while CBR-TS provides further improvement over CBR-FL.The comparison reports Average Recall at Proposal Frequency AR@F=1.0.
- State-of-the-art comparisons: CBR-C3D outperforms SCNN at all tIoU thresholds on THUMOS-14, and two-stream CBR exceeds state-of-the-art methods by 12% at tIoU=0.5.The advantage of CBR-C3D is described as especially strong at high tIoU thresholds.
- TVSeries experiments: On TVSeries, temporal coordinate regression brings a big improvement, proposal performance peaks at Kc = 3, and action-detection performance peaks at Kc = 2.After the action-detection peak, performance starts to decrease; the cascade-step distribution is consistent with THUMOS-14.
- TVSeries experiments: At tIoU=0.2 on TVSeries, CBR reaches 9.5 mAP, outperforming same-feature SVM classifiers by 3.5% and FV at 4.9.Reported mAP values at tIoU=0.1 and tIoU=0.3 are 11.0 and 7.9, respectively.
Conclusion
The paper presents a two-stage temporal action detection pipeline using Cascaded Boundary Regression in both proposal generation and action detection, achieving state-of-the-art performance on THUMOS-14 and TVSeries.
- CBR integrates cascaded boundary regression into both stages of a two-stage action detection pipeline.Temporal proposals are generated first, followed by action detection based on those proposals.
- CBR achieves state-of-the-art performance on both THUMOS-14 and TVSeries.
- Experiments analyze cascaded steps and temporal regression offset settings for proposal generation and action detection.