Source-linked AI summary
Temporal Action Detection with Structured Segment Networks
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, Dahua Lin
TL;DR
Temporal action detection must recognize and precisely localize actions in untrimmed videos, where incomplete fragments can resemble valid proposals. SSN models action stages with a structured temporal pyramid, jointly predicts activity and completeness, and uses TAG for proposal generation. It reports significant gains over state-of-the-art methods on ActivityNet and THUMOS14.
Problem
Temporal action detection must identify action categories and precise temporal boundaries in untrimmed videos, while incomplete fragments can resemble valid detections.
Method
SSN models each proposal with structured temporal pyramid pooling, decomposes prediction into activity and completeness classifiers, and uses TAG to generate action proposals.
Results
SSN achieves significant performance gains over state-of-the-art methods on both ActivityNet and THUMOS14.
Takeaways & Limitations
The framework can localize temporal boundaries precisely and work with activity classes having different temporal structures.
Abstract
from arXiv · showhide
Detecting actions in untrimmed videos is an important yet challenging task. In this paper, we present the structured segment network (SSN), a novel framework which models the temporal structure of each action instance via a structured temporal pyramid. On top of the pyramid, we further introduce a decomposed discriminative model comprising two classifiers, respectively for classifying actions and determining completeness. This allows the framework to effectively distinguish positive proposals from background or incomplete ones, thus leading to both accurate recognition and localization. These components are integrated into a unified network that can be efficiently trained in an end-to-end fashion. Additionally, a simple yet effective temporal action proposal scheme, dubbed temporal actionness grouping (TAG) is devised to generate high quality action proposals. On two challenging benchmarks, THUMOS14 and ActivityNet, our method remarkably outperforms previous state-of-the-art methods, demonstrating superior accuracy and strong adaptivity in handling actions with various temporal structures.
1. Introduction
Temporal action detection must identify both action categories and precise temporal boundaries in untrimmed videos, where incomplete fragments can resemble valid actions. SSN addresses this through explicit temporal-stage modeling, decomposed classification, and actionness-based proposal grouping.
- Motivation: Temporal action detection requires predicting both the action category and its precise starting and ending time points in untrimmed videos.The task is more challenging than action recognition because it must localize instances temporally.
- Motivation: Incomplete action fragments contain discriminative snippets, making them difficult for snippet-based classifiers to distinguish from complete detections.The paper identifies temporal structure analysis of starting, course, and ending stages as necessary for assessing completeness.
- SSN framework: SSN represents each complete activity as starting, course, and ending stages using structured temporal pyramid pooling.Stage-wise features are combined into a global proposal representation that explicitly models activity structure and surrounding context.
- SSN framework: SSN jointly classifies activity categories and proposal completeness, enabling it to distinguish complete action instances from background or incomplete proposals.The two classifiers are integrated into a unified network and trained end to end.
- Proposal generation: Temporal actionness grouping uses multiscale grouping of the actionness signal to generate proposals with higher temporal recall using fewer proposals.The proposal scheme is intended to further improve detection performance through higher-quality temporal candidates.
- Results: The framework achieves superior temporal action detection performance on standard benchmark datasets and can be efficiently learned end to end.The paper reports training times of 5 to 15 hours over a large dataset such as ActivityNet.
2. Related Work
Earlier action-recognition systems use CNN-based appearance and motion modeling, while temporal detection methods face limitations in handling long proposals and temporal structure. SSN’s framework combines proposal processing with structured temporal representations and classification.
- Action Recognition: CNN-based action-recognition methods evolved from hand-crafted features to two-stream and 3D-CNN architectures incorporating appearance and motion.The related work describes subsequent approaches including multi-stage CNNs and recurrent models for localization.
- Structured Segment Network: SSN takes video proposals of varying durations and outputs activity instances with category labels and temporal ranges through a structured framework.Its overview combines proposal processing with temporally organized representations and classification components.
- Action Recognition: S-CNN improves accuracy with a multi-stage CNN and localization network but relies on C3D features whose temporal stride is problematic for long action proposals.The limitation arises because C3D was initially designed for snippet-wise action classification.
3. Structured Segment Network
SSN represents each temporal proposal through three activity stages and uses structured temporal pooling plus separate activity and completeness classifiers. These components support classification, completeness assessment, temporal refinement, and end-to-end training.
- Three-Stage Structures: Each proposal is modeled as consecutive starting, course, and ending stages, with an augmented context surrounding the proposal.The augmented proposal extends beyond both temporal boundaries to capture how the activity starts and ends.
- Structured Temporal Pyramid Pooling: STPP builds stage-level temporal pyramid features and concatenates them into a global representation for the proposal.The course stage uses a two-level pyramid, while starting and ending stages use simpler one-level pyramids.
- Activity and Completeness Classifiers: The activity classifier predicts K activity classes plus background, while class-specific completeness classifiers determine whether proposals capture complete instances.Completeness uses the global STPP representation and surrounding context rather than the proposal alone.
- Activity and Completeness Classifiers: The classifiers form a joint distribution and are trained with a unified loss, using the completeness term only for non-background proposals.The classifiers, STPP, and loss are integrated into a single end-to-end network.
- Training Samples: Training includes positive, background, and incomplete proposals, with positive proposals requiring at least 0.7 IoU and incomplete proposals defined by specified containment and IoU conditions.Each mini-batch contains all three proposal types.
- Location Regression and Multi-Task Loss: Class-specific regressors refine the proposal center and span, and a multi-task loss combines classification with regression for positive complete proposals.The regression loss uses smooth L1 loss.
4. Efficient Training and Inference with SSN
SSN reduces computational cost for long proposals through sparse training-time sampling and reordered inference computation. The latter exploits shared features across overlapping proposals to accelerate post-feature processing.
- Training with Sparse Sampling: Dense snippet sampling is costly for long proposals, so SSN samples one snippet from each of nine evenly divided segments.This fixes the number of computed features regardless of proposal length.
- Inference with Reordered Computation: During inference, SSN samples snippets at a fixed six-frame interval and constructs the temporal pyramid from them.The fixed interval is used when testing.
- Inference with Reordered Computation: Because proposals overlap and share snippet features, SSN moves linear responses before pooling and pools network outputs separately for each proposal.Average pooling can be exchanged with matrix multiplication because it is linear.
- Inference with Reordered Computation: Reordered computation reduces post-feature-extraction processing time from around 10 seconds to less than 0.5 second per video on average.The reduction applies after network outputs have been extracted.
5. Temporal Region Proposals
Temporal actionness grouping generates proposals by identifying continuous regions with high snippet-level actionness. It applies watershed-style multi-scale grouping to merge basins into proposal regions.
- Temporal Actionness Grouping: TAG uses a binary actionness classifier to assign actionness probabilities to individual snippets.The resulting signal is used for temporal proposal generation.
- Watershed Grouping: TAG applies a one-dimensional watershed algorithm to the complemented actionness signal at different water levels γ.Flooding produces basins corresponding intuitively to temporal regions with high actionness.
- Figure 3: Figure 3 shows actionness probabilities, the complemented signal, regions from different flooding levels, and final proposals after merging.Final proposals are displayed in orange.
- Basin Grouping: The grouping scheme consecutively absorbs basins from a seed until the basin-duration fraction falls below threshold τ.The absorbed basins and intervening blank spaces form a single proposal region.
6. Experimental Results
Experiments evaluate SSN and its components on ActivityNet and THUMOS14 using proposal recall, ablations, and action-detection comparisons. TAG improves proposal quality, decomposed classifiers and location regression improve detection, end-to-end training is competitive, and SSN outperforms prior methods on both benchmarks.
- Experimental setup: Experiments use ActivityNet and THUMOS14, with dataset-specific mean average precision reported at multiple IoU thresholds.ActivityNet also uses the average mAP over thresholds from 0.5 to 0.95, while THUMOS14 reports thresholds from 0.1 to 0.5.
- Temporal action proposals: Improved TAG proposals generally produce improved action-detection performance compared with sliding-window proposals.The comparison evaluates proposal methods in the context of detection using mAP.
- Ablation studies: The decomposed activity and completeness classifiers outperform a single classifier, with mAP decreasing from 23.7% to 17.9% when the two negative types are combined.The paper attributes the gain to the different natures of background and incomplete proposals.
- Ablation studies: Location regression and multi-task learning always improve detection accuracy, while end-to-end training slightly outperforms stage-wise training under matched settings.End-to-end training also avoids storing extracted training features, which can become storage-intensive as the dataset grows.
- Comparison with the state of the art: On THUMOS14, SSN exceeds previous state-of-the-art methods by over 10% absolute mAP in most cases, while ActivityNet v1.3 reaches an average mAP of 28.28 with one model.The ActivityNet result also performs well at IoU thresholds 0.75 and 0.95.
7. Conclusion
The paper presents a generic temporal action detection framework that combines a structured temporal pyramid with separate classifiers for activity class and completeness. It achieves gains over state-of-the-art methods on ActivityNet and THUMOS14 while handling varied temporal structures and precise temporal boundaries.
- The framework combines a structured temporal pyramid with classifiers for activity class and completeness.
- The method achieves significant performance gains over state-of-the-art methods on ActivityNet and THUMOS14.
- The framework precisely localizes temporal boundaries and works across activity classes with different temporal structures.
- The work was partially supported by Sense-Time Group, Hong Kong’s General Research Fund, and the Early Career Scheme.