Source-linked AI summary
Temporal Context Aggregation Network for Temporal Action Proposal Refinement
Zhiwu Qing, Haisheng Su, Weihao Gan, Dongliang Wang, Wei Wu, Xiang Wang, Yu Qiao, Junjie Yan, Changxin Gao, Nong Sang
TL;DR
Temporal action proposals often have inaccurate boundaries and weak retrieval confidence because temporal modeling and boundary-context use remain limited. TCANet combines local-global temporal context aggregation with complementary, progressive boundary refinement, and experiments report improved proposal and detection performance across several benchmarks.
Problem
Temporal action proposal generation must estimate action intervals in untrimmed videos, but existing proposals suffer from inaccurate boundaries and inferior retrieval confidence.
Method
TCANet combines channel-grouped local-global temporal encoding with frame-level boundary-context and segment-level internal-context regression through progressive refinement.
Results
Experiments on HACS, ActivityNet-v1.3, and THUMOS-14 report significant improvements in action proposal and action detection performance.
Takeaways & Limitations
TCANet generates high-quality proposals with precise boundaries and reliable confidence through complementary and progressive refinement.
Abstract
from arXiv · showhide
Temporal action proposal generation aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet important task in the video understanding field. The proposals generated by current methods still suffer from inaccurate temporal boundaries and inferior confidence used for retrieval owing to the lack of efficient temporal modeling and effective boundary context utilization. In this paper, we propose Temporal Context Aggregation Network (TCANet) to generate high-quality action proposals through "local and global" temporal context aggregation and complementary as well as progressive boundary refinement. Specifically, we first design a Local-Global Temporal Encoder (LGTE), which adopts the channel grouping strategy to efficiently encode both "local and global" temporal inter-dependencies. Furthermore, both the boundary and internal context of proposals are adopted for frame-level and segment-level boundary regressions, respectively. Temporal Boundary Regressor (TBR) is designed to combine these two regression granularities in an end-to-end fashion, which achieves the precise boundaries and reliable confidence of proposals through progressive refinement. Extensive experiments are conducted on three challenging datasets: HACS, ActivityNet-v1.3, and THUMOS-14, where TCANet can generate proposals with high precision and recall. By combining with the existing action classifier, TCANet can obtain remarkable temporal action detection performance compared with other methods. Not surprisingly, the proposed TCANet won the 1$^{st}$ place in the CVPR 2020 - HACS challenge leaderboard on temporal action localization task.
1. Introduction
Temporal action detection remains limited by weak temporal modeling and incomplete use of boundary context. TCANet addresses these issues with local-global temporal encoding and complementary, progressive boundary refinement.
- Temporal action detection locates and classifies action instances in long untrimmed videos, supporting applications including video content analysis and recommendation.
- Existing methods inadequately model temporal relationships because convolutions limit long-range flexibility while global fusion overlooks location-specific dependencies and local boundary details.
- TCANet’s Local-Global Temporal Encoder captures local and global temporal relationships simultaneously through channel grouping.
- Temporal Boundary Regressor combines local frame-level and global segment-level regressions for complementary, progressive boundary refinement.
- Experiments on HACS, ActivityNet-v1.3, and THUMOS-14 report convincing proposal performance and remarkable temporal action detection performance with an existing classifier.
2. Related Work
Prior work uses temporal convolutions, pooling, anchor-based refinement, and boundary-based refinement, while TCANet combines local-global temporal encoding with complementary boundary regressions.
- Action Recognition: Action recognition methods include 2stream, 3D convolution, and decoupled (2 + 1)D convolution approaches for modeling appearance, motion, or spatio-temporal information.
- Temporal Action Proposal Generation and Detection: Temporal action proposal methods include anchor-based and boundary-based approaches that refine sliding windows, predefined anchors, or proposal boundaries.
- Temporal Action Proposal Generation and Detection: TCANet uses LGTE to encode local and global temporal relationships and TBR to refine proposals using boundary and internal context.
- Self-Attention Mechanism: Self-attention is widely used in video understanding because it captures long-term dependencies more effectively than recurrent models and pooling methods.
3. TCANet
TCANet generates temporal action proposals by combining local-global temporal feature modeling with complementary, progressively fused boundary refinements. Its LGTE captures temporal dependencies efficiently, while TBR uses boundary and internal proposal context to improve localization.
- Overview: TCANet consists primarily of a Local-Global Temporal Encoder and a Temporal Boundary Regressor for proposal generation.LGTE encodes local and global temporal relationships, while TBR refines proposal boundaries using boundary and internal context.
- Local-Global Temporal Encoder: LTE dynamically models each location's local temporal relationships using regional similarity over a finite modeling window.The modeling-window size is denoted by w, and the local interaction uses projected feature representations.
- Local-Global Temporal Encoder: GTE models long-term dependencies by computing each temporal location's interaction with the global feature sequence.Together, LTE and GTE capture whole-video dependencies and local changes with less noise than global modeling alone.
- Local-Global Temporal Encoder: LGTE divides projected features into channel groups, assigning the first A groups to local encoders and the remaining N −A groups to global encoders.The channel grouping gives LTEs and GTEs separate capacity for local and global temporal modeling.
- Temporal Boundary Regressor: TBR fuses two refined proposals and applies progressive refinement to generate increasingly accurate candidate boundaries.The fusion parameter τ is set to 0.5 empirically, and multiple TBRs can be studied for proposal performance.
- Temporal Boundary Regressor: TBR combines frame-level boundary regression with segment-level center-and-duration regression using complementary proposal contexts.Starting and ending contexts refine boundary locations, while starting, internal, and ending contexts refine center and duration.
4. Training and Inference of TCANet
TCANet trains on candidate proposals selected from BMN outputs and combines proposal confidence scores during inference before removing redundancies with Soft-NMS.
- Proposal Selection: TCANet selects the top 100 BMN proposals after Soft-NMS preprocessing for efficient TBR training.Positive proposals are assigned using a temporal IoU threshold during TBR training.
- Training Objective: The training objective includes SmoothL1 loss and uses λ = 1.0 as its balance parameter.
- Evaluation: Table 1 evaluates HACS detection using mAP at multiple tIoU thresholds and average mAP, while Table 2 reports the same measures on ActivityNet-v1.3.
- Inference: The final proposal confidence fuses BMN and TCANet scores during inference.
- Inference: Soft-NMS removes redundant proposals from TCANet outputs.
5. Experiments
Experiments on HACS, ActivityNet-v1.3, and THUMOS14 evaluate proposal and detection quality, while ablations and visualizations examine TCANet’s components, refinement behavior, and efficiency.
- Datasets and Setup: The experiments use HACS, ActivityNet-v1.3, and THUMOS14, with AR, AUC, and mAP evaluated across specified tIoU thresholds.HACS has 200 action categories and 37.6k training videos; ActivityNet-v1.3 has 200 categories and 10k training videos; THUMOS14 has 20 categories.
- Comparison with State-of-the-art Results: TCANet improves state-of-the-art detection results across HACS, ActivityNet-v1.3, and THUMOS14.Its mAP improves over BMN by 4% on HACS and 1.67% on ActivityNet-v1.3; at tIoU 0.6 on THUMOS14, it exceeds BSN++ by 4.9%.
- Ablation Study: Three progressive TBR stages are retained because additional stages improve performance with diminishing gains.
- Ablation Study: LGTE performs best with WindowSize = 9, eight channel groups, and four stacked LGTE modules, while excessive modules cause over-fitting.
- Ablation Study: Combining frame-level and segment-level regression improves final average mAP over using either regression granularity alone.
- Efficiency Analysis: With 2000 BMN candidate proposals, recall reaches 91% at tIoU = 0.5, and LGTE encodes video features once while TBR processes multiple proposals.
- Visualization: Visualizations show LGTE produces smoother, more accurate boundaries than GTE, while fused TBR outputs refine proposals from coarse to fine with more reliable confidence for short actions.
6. Conclusion
TCANet is proposed for temporal action proposal generation, combining local and global temporal relationship modeling with complementary boundary regression. Experiments show improved action proposal and action detection performance.
- TCANet captures local and global temporal relationships simultaneously through the Local-Global Temporal Encoder.
- A complementary boundary regression mechanism produces more precise proposal boundaries and confidence scores.
- Extensive benchmark experiments demonstrate significant improvement in both action proposal and action detection performance.