Source-linked AI summary
CTAP: Complementary Temporal Action Proposal Generation
Jiyang Gao, Kan Chen, Ram Nevatia
TL;DR
Temporal action proposal generation must capture action intervals while balancing coverage against boundary precision. CTAP combines sliding-window and actionness proposals, filters complementary windows with PATE, and refines candidates using temporal convolution. It outperforms state-of-the-art methods on THUMOS-14 and ActivityNet 1.3 and improves downstream action detection.
Problem
Temporal action proposal generation seeks high-recall action intervals with few proposals, but sliding windows have imprecise boundaries while actionness grouping may omit proposals when scores are low.
Method
CTAP combines actionness and sliding-window proposals, uses PATE to collect windows that actionness scores may miss, and applies temporal convolution for ranking and boundary adjustment.
Results
CTAP outperforms state-of-the-art proposal-generation methods by a large margin on THUMOS-14 and ActivityNet 1.3 and consistently improves action detection with a standard detector.
Takeaways & Limitations
Combining complementary proposal sources yields high-quality proposals that also improve performance when used in an existing action detector.
Abstract
from arXiv · showhide
Temporal action proposal generation is an important task, akin to object proposals, temporal action proposals are intended to capture "clips" or temporal intervals in videos that are likely to contain an action. Previous methods can be divided to two groups: sliding window ranking and actionness score grouping. Sliding windows uniformly cover all segments in videos, but the temporal boundaries are imprecise; grouping based method may have more precise boundaries but it may omit some proposals when the quality of actionness score is low. Based on the complementary characteristics of these two methods, we propose a novel Complementary Temporal Action Proposal (CTAP) generator. Specifically, we apply a Proposal-level Actionness Trustworthiness Estimator (PATE) on the sliding windows proposals to generate the probabilities indicating whether the actions can be correctly detected by actionness scores, the windows with high scores are collected. The collected sliding windows and actionness proposals are then processed by a temporal convolutional neural network for proposal ranking and boundary adjustment. CTAP outperforms state-of-the-art methods on average recall (AR) by a large margin on THUMOS-14 and ActivityNet 1.3 datasets. We further apply CTAP as a proposal generation method in an existing action detector, and show consistent significant improvements.
1 Introduction
Temporal action proposals must balance coverage and precise boundaries, but existing sliding-window and actionness-grouping methods fail in complementary ways. CTAP combines both proposal types with trustworthiness filtering and temporal convolution, achieving strong proposal-generation and detection results.
- Motivation: Temporal action proposals capture video intervals likely to contain actions, and high-quality proposals improve downstream action detection.Quality is evaluated by Average Recall with as few retrieved proposals as possible.
- Existing methods: Sliding windows cover every video segment but produce imprecise boundaries and require many proposals for high Average Recall.SW+R&A methods rank and adjust sliding-window proposals, but boundary precision remains limited.
- Existing methods: Actionness grouping predicts finer boundaries but can omit correct proposals when actionness scores are low.TAG groups continuous high-score units into proposals, so missing actionness evidence can prevent candidate generation.
- CTAP: CTAP uses complementary filtering to select sliding windows likely missed by actionness proposals, then ranks proposals and adjusts boundaries.Its Proposal-level Actionness Trustworthiness Estimator produces probabilities indicating whether actionness scores can correctly detect each proposal.
- CTAP: CTAP introduces temporal convolution for proposal ranking and boundary adjustment while preserving temporal ordering information.The architecture contains initial proposal generation, complementary filtering, and a final ranking and boundary-adjustment module.
- Results: CTAP outperforms state-of-the-art proposal-generation methods on THUMOS-14 and ActivityNet v1.3 and consistently improves action detection with a standard detector.The reported gains are described as large for proposal generation and significant for downstream detection.
2 Related Work
Related work divides temporal action proposal generation into sliding-window classification and actionness-based grouping, alongside broader temporal and online action-detection tasks.
- Temporal Action Proposal: Temporal action proposal generation is treated as an effective step for action detection and other high-level video-understanding tasks.
- Temporal Action Proposal: Sliding-window proposal methods formulate generation as binary classification over clips or windows.Examples include Sparse-prop, SCNN-prop, and TURN, which also uses unit-level features and temporal regression for boundary adjustment.
- Temporal Action Detection: Temporal action detection predicts action categories and their start and end times in untrimmed videos.Existing approaches include two-stage proposal-based detection and single-shot detection without proposal generation.
- Online Action Detection: Online action detection differs because the whole video is unavailable at detection time, requiring actions to be detected as the video unfolds.Related work includes datasets and models for online detection and action anticipation.
3 Complementary Temporal Action Proposal Generator
CTAP combines actionness proposals and sliding windows, then filters, ranks, and adjusts them through complementary stages. Its temporal-convolutional design preserves ordering information while supporting proposal scoring and boundary refinement.
- Pipeline: CTAP has three stages: initial proposal generation, complementary filtering, and proposal ranking with boundary adjustment.Initial proposals come from actionness scores with TAG and sliding windows.
- Initial proposal generation: Initial proposals combine actionness-based proposals from unit scores and TAG with uniformly sampled sliding windows.Actionness proposals depend on score distributions, whereas sliding windows are sampled to maintain high ground-truth overlap with few windows.
- Complementary filtering: PATE estimates whether sliding-window proposals can be correctly detected by actionness scores and TAG, using proposal-level features formed by mean pooling units inside each ground-truth segment.Training labels are positive when an actionness proposal overlaps a ground-truth segment above threshold θc; otherwise they are negative.
- Complementary filtering: During testing, CTAP collects sliding windows with PATE scores below θa, combines them with all actionness proposals, and forwards the result to TAR.The collected windows target segments where TAG may fail because actionness scores are not trustworthy.
- Proposal ranking and boundary adjustment: TAR uses temporal convolution over proposal and boundary-unit features to rank proposals and adjust their start and end boundaries.Unlike mean pooling, the temporal-convolutional design aggregates unit-level features while retaining temporal ordering information; outputs include start and end offsets and proposal action probability.
- Training and scoring: Sliding-window proposal scores are multiplied by PATE and TAR scores, while actionness proposals use the TAR proposal score as their final score.TAR is trained with Softmax cross-entropy for ranking and L1 distance loss for boundary adjustment.
4 Experiments
Experiments evaluate CTAP on THUMOS-14 and ActivityNet v1.3, including component comparisons, complementary filtering, state-of-the-art comparisons, generalization, and downstream detection. Across these evaluations, CTAP and its modules improve proposal recall, ranking, boundary adjustment, and detection performance.
- Component evaluation: TAR outperforms TURN at AN=50, 100, and 200 on THUMOS-14 when using the same flow-16 features and test sliding windows.The comparison evaluates proposal ranking and boundary adjustment under matched settings.
- Component evaluation: Complementary filtering achieves the best AR at AN=50, 100, and 200 compared with union, union+NMS, and tIoU-selection.The method preserves priority between actionness and sliding-window proposals while selecting complementary windows.
- Temporal action detection: On THUMOS-14, CTAP-TS6 achieves the best performance with the same SCNN detector and exceeds TURN and TAG by over 4%.The downstream comparison reports mean average precision at tIoU=0.5.
- Component evaluation: TAR improves SW-TURN by 18.29% in AR@100 and 6.86% in AUC, while its actionness-proposal AUC improves by 10.70%.These results support the effectiveness of temporal convolution for proposal ranking and boundary adjustment.
- Component evaluation: CTAP consistently outperforms SW-TAR and TAG-TAR on AR@100 and AUC, indicating that complementary proposals improve over either proposal source alone.The comparison attributes the gain to selecting sliding-window proposals that fill omissions in actionness proposals.
- ActivityNet evaluation: On ActivityNet v1.3, CTAP improves AR@100 over MSRA by 2.60% and over Prop-SSAD by 1.32%, while showing only a slight drop on unseen classes.The evaluation reports both state-of-the-art comparison and generalization across seen and unseen classes.
- Qualitative analysis: Visualizations show CTAP selecting suitable actionness or sliding-window proposals and adjusting their temporal boundaries more precisely across THUMOS-14 and ActivityNet v1.3.The examples include proposals from two videos in each dataset.
5 Conclusion
CTAP combines complementary proposal-generation strategies to produce high-quality temporal action proposals. It improves proposal and action-detection performance across both THUMOS-14 and ActivityNet 1.3.
- CTAP combines sliding-window ranking with actionness-score grouping to collect complementary proposals.Sliding windows provide coverage, while actionness proposals contribute more precise boundaries.
- The method uses a temporal convolutional network for proposal ranking and boundary adjustment.
- CTAP outperforms state-of-the-art methods by a large margin on THUMOS-14 and ActivityNet 1.3.
- Applying CTAP to action detection produces consistent large performance improvements.