Source-linked AI summary
Fast Learning of Temporal Action Proposal via Dense Boundary Generator
Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji
TL;DR
Precise temporal boundaries and reliable confidence remain difficult for proposals in long, untrimmed videos. DBG densely predicts boundary and completeness score maps using dual-stream, global proposal, and action-aware features. Experiments on ActivityNet-1.3 and THUMOS14 report superiority over state-of-the-art methods.
Problem
Temporal action proposal generation must predict precise boundaries and reliable confidence despite challenging variation and clutter in untrimmed videos.
Method
DBG uses dual-stream features with temporal boundary classification and action-aware completeness regression to produce dense proposal score maps.
Results
DBG outperforms state-of-the-art proposal methods on ActivityNet-1.3 and THUMOS14.
Takeaways & Limitations
Dense global boundary prediction and action-aware completeness estimation provide a unified approach for temporal action proposal generation.
Takeaways & Limitations
Prior boundary-matching methods motivating DBG cannot handle complex activities and cluttered backgrounds while relying only on low-level features.
Abstract
from arXiv · showhide
Generating temporal action proposals remains a very challenging problem, where the main issue lies in predicting precise temporal proposal boundaries and reliable action confidence in long and untrimmed real-world videos. In this paper, we propose an efficient and unified framework to generate temporal action proposals named Dense Boundary Generator (DBG), which draws inspiration from boundary-sensitive methods and implements boundary classification and action completeness regression for densely distributed proposals. In particular, the DBG consists of two modules: Temporal boundary classification (TBC) and Action-aware completeness regression (ACR). The TBC aims to provide two temporal boundary confidence maps by low-level two-stream features, while the ACR is designed to generate an action completeness score map by high-level action-aware features. Moreover, we introduce a dual stream BaseNet (DSB) to encode RGB and optical flow information, which helps to capture discriminative boundary and actionness features. Extensive experiments on popular benchmarks ActivityNet-1.3 and THUMOS14 demonstrate the superiority of DBG over the state-of-the-art proposal generator (e.g., MGG and BMN). Our code will be made available upon publication.
Introduction
Temporal action proposal generation must recover precise boundaries and reliable confidence in difficult untrimmed videos. DBG addresses these challenges with dense global boundary prediction, action-aware completeness estimation, and unified score-map fusion.
- Temporal proposal generation remains open under variable durations, complex activities, blurred boundaries, camera motion, clutter, and viewpoint changes.
- Anchor-based methods use fixed multiscale candidates, limiting boundary precision and flexibility across action durations.
- Boundary-based methods improve temporal localization but prior approaches may omit actionness and lack global coverage for blurred or variable-duration actions.
- DBG combines global proposal features for boundary maps with action-aware features for completeness analysis.
- DBG uses auxiliary actionness supervision, an efficient proposal feature generation layer, and dense boundary confidence maps for all proposals.
- Experiments on ActivityNet-1.3 and THUMOS14 report superiority over state-of-the-art methods.
Related Work
Video action recognition commonly uses two-stream or 3D convolutional networks. DBG builds on spatial and temporal video representations within a proposal-generation pipeline.
- Modern video action-recognition networks commonly follow two-stream or 3D-network designs.
- DBG encodes spatial and temporal visual contents before feeding RGB and flow features into its generator.
Approach
DBG processes untrimmed-video features through a dual-stream backbone and separate proposal-level regression and classification modules. It samples temporal regions to form differentiable proposal context features for dense proposal evaluation.
- Approach: DBG takes RGB and optical-flow feature sequences and uses three modules: DSB, ACR, and TBC.DSB serves as the backbone, producing actionness and dual-stream features for ACR and TBC.
- Approach: The DSB applies one-dimensional temporal convolutions to capture local semantic information and discriminative boundary and actionness features.Its outputs are routed separately to the action-aware completeness regression and temporal boundary classification branches.
- Proposal feature generation layer: PFG transforms an L × C input feature into an L × L × N × C proposal tensor containing context features for dense proposals.The layer is differentiable and supports end-to-end training.
- Proposal feature generation layer: For each proposal, PFG samples left, center, and right temporal regions by linear interpolation, then concatenates the sampled features into a context proposal feature.The regions use Nl, Nc, and Nr samples around the proposal boundaries and inside the proposal interval.
- Proposal feature generation layer: The experiments set Nl = Nr = 8 and Nc = 16, giving N = 32 sampled locations per proposal.Proposals with ts ≥ te receive zero proposal features.
- Approach: ACR receives actionness features and produces an action completeness map estimating overlap between candidate proposals and ground-truth action instances.It uses proposal-level processing with the PFG layer and two-dimensional convolutions.
Training and Inference
DBG jointly trains actionness, boundary classification, and completeness regression, then fuses score maps to generate dense proposals and suppresses redundancy with Soft-NMS.
- Training: DBG constructs actionness, starting-boundary, ending-boundary, and completeness labels for its three-task training objective.The labels supervise DSB actionness classification, TBC boundary classification, and ACR completeness regression.
- Training: DSB assigns actionness labels from maximum overlap ratios, using 0.5 as the positive-label threshold.Each temporal location is represented by a region whose overlap ratio with ground truth determines its actionness label.
- Training: TBC applies binary logistic regression separately to starting and ending boundary confidence maps.Boundary labels are computed for locations in the starting and ending maps before classification loss is applied.
- Training: ACR predicts an action completeness map for proposal regions and trains it with smooth L1 regression loss.The completeness label is based on the maximum IoU between each proposal region and ground-truth action instances.
- Inference: During inference, DBG fuses completeness with boundary probabilities, forms dense candidate proposals, averages shared-boundary scores, and applies Soft-NMS.A confidence threshold after Soft-NMS produces the final sparse proposal set.
Experiments
Experiments evaluate DBG on ActivityNet-1.3 and THUMOS14 using proposal recall, efficiency, ablations, generalization, and downstream detection. DBG is reported to improve proposal quality, inference speed, and temporal detection performance over competing methods.
- Evaluation setup: ActivityNet-1.3 evaluation uses AR@AN across IoU thresholds and AUC, while THUMOS14 uses AR@AN across proposal counts and IoU thresholds.ActivityNet-1.3 uses IoU thresholds [0.5:0.05:0.95] and also reports AUC; THUMOS14 uses [0.5:0.05:1.0].
- Proposal quality: DBG using C3D or two-stream features significantly outperforms other methods on THUMOS14 across proposal counts from 50 to 1000.The comparison uses the testing set and the same video features and post-processing for fairness.
- Efficiency: 13ms total inference time and 8ms proposal feature generation are reported for DBG on a 3-minute ActivityNet-1.3 video.The proposal feature generation time is reduced from 47ms to 8ms in the comparison with BMN and BSN.
- Ablation study: DBG outperforms its variants across AUC and IoU thresholds, while TBC provides more precise boundaries at the strict 0.9 IoU threshold.The ablations compare removing DSB, ACR, or TBC; the text also reports that action-aware features help completeness regression and dense boundary maps help all proposals.
- PFG analysis: Sampling 8, 16, and 8 locations from the left, center, and right proposal regions achieves the best PFG-layer performance.The reported analysis also indicates that context around proposals is necessary for better performance.
- Downstream detection: DBG-based temporal detection significantly outperforms state-of-the-art methods on THUMOS14 using shared action classifiers.At IoU 0.7, DBG improves mAP by 1.4% and 1.2% over BMN-based methods for the two classifiers.
Conclusion
The paper introduces DBG, a unified temporal action proposal generator that combines dual-stream features with boundary classification and action-aware completeness regression. Experiments on ActivityNet-1.3 and THUMOS14 demonstrate superiority over state-of-the-art methods.
- Conclusion: DBG uses a dual stream BaseNet, temporal boundary classification, and action-aware completeness regression to generate precise boundaries and reliable action completeness confidence.The framework generates two feature levels for boundary prediction and completeness estimation.
- Conclusion: Comprehensive experiments on ActivityNet-1.3 and THUMOS14 demonstrate DBG's superiority over state-of-the-art temporal action proposal methods.