Source-linked AI summary
BMN: Boundary-Matching Network for Temporal Action Proposal Generation
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, Shilei Wen
TL;DR
Temporal proposal generation needs precise boundaries and reliable confidence scores, while existing bottom-up methods do not efficiently provide both. BMN introduces Boundary-Matching to score dense boundary-pair proposals in a confidence map and jointly trains boundary and confidence branches. Across proposal generation and temporal detection tasks, BMN outperforms prior methods with reported efficiency and generalizability.
Problem
Proposal generation must cover ground-truth actions precisely and exhaustively while providing reliable confidence scores, but BSN uses inefficient per-proposal evaluation, limited context, and multiple stages.
Method
BMN represents proposals as matching starting-ending boundary pairs, forms a dense confidence map, and jointly generates boundary probabilities and proposal confidence scores.
Results
BMN outperforms state-of-the-art methods in proposal generation and temporal action detection, including a 0.93% AUC improvement on ActivityNet-1.3 validation from 66.17% to 67.10%.
Takeaways & Limitations
BMN provides precise boundaries, flexible durations, and reliable confidence scores with remarkable efficiency and generalizability.
Takeaways & Limitations
BSN’s limitations motivate BMN: per-proposal evaluation is inefficient, its proposal features capture insufficient temporal context, and its pipeline is not unified.
Abstract
from arXiv · showhide
Temporal action proposal generation is an challenging and promising task which aims to locate temporal regions in real-world videos where action or event may occur. Current bottom-up proposal generation methods can generate proposals with precise boundary, but cannot efficiently generate adequately reliable confidence scores for retrieving proposals. To address these difficulties, we introduce the Boundary-Matching (BM) mechanism to evaluate confidence scores of densely distributed proposals, which denote a proposal as a matching pair of starting and ending boundaries and combine all densely distributed BM pairs into the BM confidence map. Based on BM mechanism, we propose an effective, efficient and end-to-end proposal generation method, named Boundary-Matching Network (BMN), which generates proposals with precise temporal boundaries as well as reliable confidence scores simultaneously. The two-branches of BMN are jointly trained in an unified framework. We conduct experiments on two challenging datasets: THUMOS-14 and ActivityNet-1.3, where BMN shows significant performance improvement with remarkable efficiency and generalizability. Further, combining with existing action classifier, BMN can achieve state-of-the-art temporal action detection performance.
1. Introduction
Temporal action proposal generation must produce precise, flexible proposals and reliable confidence scores, but existing approaches face efficiency and context limitations. BMN addresses this with Boundary-Matching confidence maps and jointly trained boundary and proposal-evaluation branches.
- Temporal action detection requires locating action instances in untrimmed videos with both categories and temporal boundaries.
- Proposal generation should provide flexible durations, precise boundaries, and reliable confidence scores for proposal retrieval.
- BSN improves boundary precision and duration flexibility but evaluates proposals individually, uses limited context, and relies on multiple stages.
- The Boundary-Matching mechanism represents each proposal as a starting-ending boundary pair and combines dense pairs into a two-dimensional confidence map.
- BMN simultaneously generates boundary probabilities and proposal confidence maps in jointly trained branches within an efficient, effective, end-to-end framework.
2. Related Work
Prior work spans action-recognition feature extraction, correlation-based matching, and top-down or bottom-up proposal generation. BMN applies correlation-style boundary matching to address confidence evaluation for densely distributed proposals.
- Action Recognition: Action-recognition systems use appearance and motion information from RGB, optical flow, or raw video volumes to extract visual features.
- Correlation Matching: Correlation matching represents paired correspondences as a cost volume, inspiring BMN’s map of temporal starting-ending boundary pairs.
- Temporal Action Proposal Generation: Top-down proposal methods use predefined durations and intervals, limiting boundary precision and duration flexibility.
- Temporal Action Proposal Generation: Bottom-up methods improve boundary localization, but TAG lacks retrieval confidence and BSN uses a multi-stage confidence-evaluation pipeline.
- Temporal Action Proposal Generation: In BM confidence maps, rows share temporal duration while columns share starting time; proposals extending beyond the video are excluded.
3. Our Approach
BMN represents proposals as boundary-matching pairs, samples temporal features for all proposals simultaneously, and jointly predicts boundary probabilities and proposal confidence. Its shared architecture and training framework support efficient proposal evaluation with context-rich features.
- Problem formulation: A temporal proposal is represented by its starting and ending boundaries, while proposal generation seeks precise and exhaustive coverage of ground-truth action instances.
- Boundary-Matching mechanism: The BM confidence map assigns each densely distributed boundary-duration pair a confidence score for its corresponding proposal.For proposal ϕi,j, the map uses starting boundary ts = tj, duration d = ti, and ending boundary te = tj + ti.
- Boundary-Matching mechanism: The BM layer uniformly samples N points across each proposal’s expanded temporal region and forms proposal features by dot products with temporal feature sequences.The expanded region is [ts − 0.25d, te + 0.25d], and non-integer locations use a two-neighbor sampling mask.
- Boundary-Matching mechanism: Because the sampling mask is shared across videos and pre-generated, BMN produces context-rich BM features for all proposals efficiently.The resulting BM feature map has dimensions C × N × D × T and supports adjacent-proposal context.
- Boundary-Matching Network: BMN uses a shared base module, a Temporal Evaluation Module for boundary probabilities, and a Proposal Evaluation Module for the BM confidence map.The two evaluation branches are trained jointly in a unified framework rather than as separate stages.
- Training of BMN: Training constructs overlapping observation windows, assigns boundary labels from overlap ratios around ground-truth starts and ends, and assigns proposal labels from maximum IoU.The Temporal Evaluation Module uses weighted binary logistic losses for starting and ending probabilities.
4. Experiments
Experiments evaluate BMN for proposal generation, generalization, and action detection on ActivityNet-1.3 and THUMOS-14. BMN improves proposal quality and efficiency over prior methods and supports stronger detection performance.
- Dataset and Setup: Experiments use THUMOS-14 and ActivityNet-1.3 to evaluate proposal quality, efficiency, generalizability, and detection performance.ActivityNet-1.3 contains 19,994 temporally annotated videos across 200 action categories; THUMOS-14 contains 413 videos across 20 categories.
- Temporal Action Proposal Generation: 67.10% AUC on ActivityNet-1.3 validation improves over BSN's 66.17% by 0.93%.The comparison uses proposal-generation performance on the ActivityNet-1.3 validation set.
- Temporal Action Proposal Generation: BMN outperforms other methods on THUMOS-14 across C3D and two-stream features, proposal counts from 10 to 1000, and both Greedy-NMS and Soft-NMS.Soft-NMS especially improves average recall under small proposal numbers.
- Temporal Action Proposal Generation: Jointly trained BMN achieves higher recall and faster speed than separately trained BMN, while BM-based PEM is faster than BSN's proposal evaluation.The BM mechanism generates confidence scores for all proposals simultaneously rather than evaluating them one by one.
- Generalizability of Proposals: BMN shows only a slight performance drop on unseen ActivityNet-1.3 action categories, indicating generalization to unseen actions.The evaluation compares models trained on seen categories with models trained on seen and unseen categories.
- Action Detection with Our Proposals: BMN proposals combined with classifiers outperform other state-of-the-art proposal-generation methods in ActivityNet-1.3 and THUMOS-14 detection experiments.On THUMOS-14, BMN confidence scores are reliable enough for retrieving results with either video-level or proposal-level classifiers.
5. Conclusion
The paper introduces BM and BMN to generate precise proposals and reliable confidence scores efficiently. Experiments show improvements in proposal generation and temporal action detection, with efficiency and generalizability.
- 5. Conclusion: BM represents each proposal as a matching pair of starting and ending boundaries and combines densely distributed pairs into a BM confidence map.This mechanism evaluates confidence scores for densely distributed proposals.
- 5. Conclusion: BMN combines high-probability boundaries to produce proposals with precise boundaries and flexible duration while generating confidence scores simultaneously.The network jointly trains its branches in a unified framework.
- 5. Conclusion: Experiments show that BMN outperforms state-of-the-art methods in proposal generation and temporal action detection, with efficiency and generalizability.