Source-linked AI summary
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, Ming Yang
TL;DR
Temporal action proposal generation must provide precise, flexible boundaries and retrieve high-recall, high-overlap proposals efficiently in untrimmed videos. BSN uses local boundary detection and global proposal-level evaluation, outperforming existing proposal methods on THUMOS14 and ActivityNet-1.3 and improving full detection performance.
Problem
Temporal proposal methods need precise boundaries, flexible durations, and reliable retrieval to cover action instances with high recall and overlap using relatively few proposals.
Method
BSN detects local boundary and actionness probabilities, combines likely start and end locations into proposals, and evaluates proposal confidence using proposal-level features.
Results
BSN significantly outperforms state-of-the-art proposal-generation methods on THUMOS14 and ActivityNet-1.3 and improves detection performance in a full framework.
Takeaways & Limitations
BSN achieves high recall and temporal overlap with relatively few proposals and improves temporal action detection when used as the proposal stage.
Takeaways & Limitations
Predefined-duration and interval proposal methods have limited temporal precision and flexibility for variable-duration ground-truth actions.
Abstract
from arXiv · showhide
Temporal action proposal generation is an important yet challenging problem, since temporal proposals with rich action content are indispensable for analysing real-world videos with long duration and high proportion irrelevant content. This problem requires methods not only generating proposals with precise temporal boundaries, but also retrieving proposals to cover truth action instances with high recall and high overlap using relatively fewer proposals. To address these difficulties, we introduce an effective proposal generation method, named Boundary-Sensitive Network (BSN), which adopts "local to global" fashion. Locally, BSN first locates temporal boundaries with high probabilities, then directly combines these boundaries as proposals. Globally, with Boundary-Sensitive Proposal feature, BSN retrieves proposals by evaluating the confidence of whether a proposal contains an action within its region. We conduct experiments on two challenging datasets: ActivityNet-1.3 and THUMOS14, where BSN outperforms other state-of-the-art temporal action proposal generation methods with high recall and high temporal precision. Finally, further experiments demonstrate that by combining existing action classifiers, our method significantly improves the state-of-the-art temporal action detection performance.
1 Introduction
Temporal action proposal generation targets precise, flexible proposals and reliable retrieval for long, untrimmed videos. BSN addresses these needs through local boundary detection and global proposal evaluation, improving proposal and detection performance.
- Temporal action detection must identify both action boundaries and classes in long, untrimmed videos containing irrelevant content.
- High-quality proposals should achieve high recall and temporal overlap with fewer proposals to reduce subsequent computation.
- Pre-defined sliding-window durations and intervals can produce imprecise proposals and poorly cover action instances with variable durations.
- BSN evaluates temporal boundary and actionness probabilities, then combines high-probability starting and ending locations to generate proposals.
- BSN evaluates proposal confidence using actionness features within and around each proposal, adding global information for retrieval.
- BSN improves proposal quality over existing methods, supports unseen action classes, and improves temporal action detection when combined with an existing classifier.
2 Related work
Related work spans action-recognition features, object-detection proposal strategies, and temporal action detection methods. BSN is distinguished by generating proposals from boundary probabilities and combining local boundary information with global proposal evaluation.
- Action recognition: Action-recognition systems use handcrafted features, two-stream networks, or 3D convolutional networks to extract visual features from videos.
- Object detection and proposals: Object-detection proposal methods include sliding windows and approaches exploiting low-level visual cues before classification.
- Temporal action detection and proposals: BSN differs from LocNet by generating proposals rather than revising them, and by calculating boundary probabilities once per video instead of repeatedly for each box.
- Temporal action detection and proposals: Earlier temporal proposal methods used sliding windows, while newer methods used predefined durations and intervals or learned proposal confidence scores.
- Temporal action detection and proposals: Compared with a class-wise start-middle-end approach, BSN uses temporal convolution and local-to-global processing for boundary precision and proposal retrieval.
3 Our Approach
BSN encodes snippet-level visual features, predicts temporal boundaries and actionness, combines likely boundaries into proposals, and evaluates them with Boundary-Sensitive Proposal features before Soft-NMS post-processing.
- Video Features Encoding: Two-stream encoding concatenates spatial and temporal network outputs from RGB frames and stacked optical flow into the feature sequence consumed by BSN.Snippets are sampled at a regular frame interval to reduce computation.
- Boundary-Sensitive Network: BSN’s temporal evaluation module uses temporal convolutions to predict starting, ending, and actionness probabilities at each temporal location.Its three-layer architecture uses two 512-filter ReLU layers followed by three sigmoid outputs.
- Boundary-Sensitive Network: For each candidate, BSP features sample actionness sequences from its center, starting, and ending regions to represent proposal-level information.The center uses 16 interpolation points, while each boundary region uses 8 points.
- Boundary-Sensitive Network: Candidate proposals combine locations with high starting or ending probabilities, including probability peaks, subject to minimum and maximum duration constraints.Starting and ending locations are formed separately before pairwise combination.
- Boundary-Sensitive Network: The proposal evaluation module uses BSP features to predict whether each proposal contains an action instance and combines these scores with boundary probabilities.The evaluator is a one-hidden-layer multilayer perceptron.
- Prediction and Post-processing: Soft-NMS recursively decays scores of highly overlapping proposals to suppress redundancy while retaining rescored proposals.Proposals are processed in descending score order using overlap IoU with the highest-scoring proposal.
4 Experiments
Experiments on ActivityNet-1.3 and THUMOS14 evaluate BSN’s proposal recall, temporal overlap, generalization, module contributions, and downstream detection performance. BSN consistently improves proposal quality, including with relatively few proposals, and transfers effectively to unseen action classes and detection frameworks.
- Temporal Proposal Generation: BSN outperforms state-of-the-art proposal methods across ActivityNet-1.3 validation/testing sets and THUMOS14 over average proposal counts from 10 to 1000.Evaluation uses AR@AN and, for ActivityNet-1.3, AUC.
- Temporal Proposal Generation: 15.60%: BSN raises THUMOS14 average recall from 21.86% to 37.46% at an average of 50 proposals.
- Temporal Proposal Generation: With 100 proposals, BSN achieves higher recall across IoU thresholds from 0.5 to 1.0; with 1000, its largest gains occur above 0.8 IoU.
- Temporal Proposal Generation: Controlled experiments indicate BSN’s architecture, rather than C3D versus two-stream features or Greedy-NMS versus Soft-NMS, drives the performance gains.C3D-BSN remains stronger than other C3D-based methods, while Soft-NMS provides only a small promotion over Greedy-NMS.
- Generalization: BSN shows only a slight performance drop on semantically different unseen ActivityNet-1.3 classes, indicating generalization of temporal proposal learning.The comparison uses seen and unseen action subsets with C3D features.
- Action Detection with Our Proposals: Combining BSN proposals with existing classifiers improves state-of-the-art temporal action detection, especially at high overlap thresholds.On THUMOS14, BSN also outperforms other proposal methods with the same classifier and works with video-level and proposal-level classification settings.
5 Conclusion
BSN generates flexible-duration proposals with precise boundaries and accurate confidence-based retrieval, achieving high recall and temporal overlap with relatively few proposals. It outperforms state-of-the-art proposal methods on THUMOS14 and ActivityNet-1.3 and improves detection performance when paired with an action classifier.
- BSN combines high-probability boundary locations to generate proposals with flexible durations and precise temporal boundaries.
- BSN evaluates proposal-level confidence scores to retrieve proposals accurately.
- BSN achieves high recall and high temporal overlap using relatively few proposals.
- BSN significantly outperforms other state-of-the-art proposal generation methods on THUMOS14 and ActivityNet-1.3.
- Using BSN as the proposal stage in a full detection framework significantly improves detection performance.