Source-linked AI summary
Learning Salient Boundary Feature for Anchor-free Temporal Action Localization
Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Yanwei Fu
TL;DR
Temporal action localization requires both action classification and precise temporal boundaries, while existing approaches use redundant proposals or predefined anchors. AFSD introduces a purely anchor-free predictor with saliency-based boundary pooling and consistency learning. It reports a 3.7% THUMOS14 mAP@0.5 improvement over state-of-the-art methods and comparable ActivityNet1.3 results.
Problem
Existing actionness-guided and anchor-based methods produce redundant proposals, require extra classification stages or predefined anchors, and involve sensitive hyper-parameter tuning.
Method
AFSD uses an end-to-end anchor-free predictor with saliency-based boundary pooling and Boundary Consistency Learning for boundary refinement.
Results
3.7% improvement on THUMOS14 mAP@0.5 was reported against state-of-the-art methods, with comparable results on ActivityNet1.3.
Takeaways & Limitations
The results indicate that anchor-free modeling is a promising choice for temporal action localization.
Takeaways & Limitations
ActivityNet1.3 results are described as comparable, and temporal-boundary ambiguity partially accounts for slight improvement there.
Abstract
from arXiv · showhide
Temporal action localization is an important yet challenging task in video understanding. Typically, such a task aims at inferring both the action category and localization of the start and end frame for each action instance in a long, untrimmed video.While most current models achieve good results by using pre-defined anchors and numerous actionness, such methods could be bothered with both large number of outputs and heavy tuning of locations and sizes corresponding to different anchors. Instead, anchor-free methods is lighter, getting rid of redundant hyper-parameters, but gains few attention. In this paper, we propose the first purely anchor-free temporal localization method, which is both efficient and effective. Our model includes (i) an end-to-end trainable basic predictor, (ii) a saliency-based refinement module to gather more valuable boundary features for each proposal with a novel boundary pooling, and (iii) several consistency constraints to make sure our model can find the accurate boundary given arbitrary proposals. Extensive experiments show that our method beats all anchor-based and actionness-guided methods with a remarkable margin on THUMOS14, achieving state-of-the-art results, and comparable ones on ActivityNet v1.3. Code is available at https://github.com/TencentYoutuResearch/ActionDetection-AFSD.
1. Introduction
Temporal action localization must classify action instances and identify their temporal boundaries, but existing actionness-guided and anchor-based approaches incur redundant proposals, extra stages, or sensitive anchor settings. AFSD addresses these issues with a purely anchor-free predictor and saliency-based boundary refinement, achieving stronger THUMOS14 results and comparable ActivityNet1.3 results.
- Temporal action localization jointly predicts action categories and accurate temporal locations for each activity instance.
- Actionness-guided methods generate O(T^2) proposals, while anchor-based R-C3D generates C·T proposals, increasing training and testing costs.
- Actionness-guided methods require extra classifiers, whereas anchor-based methods depend on carefully tuned anchor numbers and sizes.
- Anchor-free localization predicts start and end distances from each temporal location, combining boundary regression and classification without predefined anchors.
- AFSD combines an end-to-end anchor-free predictor with boundary pooling and Boundary Consistency Learning to refine fine-grained boundary predictions.
- 3.7% improvement on THUMOS14 mAP@0.5 was reported against state-of-the-art methods, while ActivityNet1.3 results were comparable.
2. Related Work
Related work includes anchor-based localization, actionness-guided proposal generation, and the development of anchor-free methods in object detection. These approaches differ in how they represent and generate temporal proposals.
- Anchor-based localization adjusts predefined anchors to classify activities and regress temporal boundaries.
- Actionness-guided methods predict start, end, and actionness signals, then assemble candidate temporal locations into proposals.
- BSN and BMN improve actionness-guided localization by evaluating candidate confidence and adding Boundary-Matching confidence maps.
- Unlike actionness-guided enumeration of temporal-location combinations, anchor-free localization directly predicts boundaries for each time step.
- Anchor-free object detection directly predicts bounding-box coordinates, motivating analogous approaches for temporal action localization.
3. Method
AFSD builds coarse anchor-free proposals from temporal pyramid features, then refines each proposal using salient boundary features and consistency-guided learning. Its end-to-end pipeline combines boundary pooling, frame-level features, and proposal-quality prediction for fine-grained localization.
- Basic Prediction Module: AFSD processes video features with a backbone and feature pyramid network before generating coarse proposals through a basic anchor-free predictor.The predictor outputs action classes and start/end distances for each temporal location.
- Basic Prediction Module: The basic predictor produces one coarse proposal sequence from each pyramid level without requiring predefined anchors.Each level uses a simple regressor and classifier to estimate temporal boundaries and class scores.
- Saliency-based Refinement Module: Boundary pooling searches proposal boundary regions for the most salient moment-level features instead of aggregating all frames.The method uses maximization along each channel and supplements deep pyramid features with shared frame-level features when temporal resolution becomes too small.
- Saliency-based Refinement Module: The refinement module concatenates original, pooled, and frame-level boundary features to predict regression offsets and refined class scores.Offsets are added to coarse boundaries to obtain fine-grained temporal predictions.
- Boundary Consistency Learning: Boundary Consistency Learning regularizes pooled features because saliency alone may select background frames rather than true action boundaries.Its activation-guided component uses start and end signals to constrain features toward action occurrences and closures, while contrastive learning compares fragmented action and background regions.
- Training and Inference: The model predicts proposal quality to suppress low-quality proposals, using temporal intersection-over-union rather than directly applying object-detection centerness.The paper describes centerness as unsuitable for temporal actions because exact start and end signals are difficult to define.
4. Experiments
Experiments evaluate AFSD on THUMOS14 and ActivityNet1.3, compare its accuracy and efficiency with existing methods, and analyze its components through ablations.
- Main Results: AFSD outperforms the strongest THUMOS14 competitors across all thresholds, including a 7.7% gain on mAP@0.6.The comparison includes A2Net and G-TAD.
- Main Results: AFSD obtains comparable ActivityNet1.3 results, with the best mAP@0.75 and average mAP against GTAN.Actionness-guided methods achieve higher mAP@0.95, while AFSD produces fewer proposals and better overall performance across multiple thresholds.
- Ablation Study: Quality loss improves average mAP by 1.0%, whereas FCOS centerness causes a 1.3% drop on mAP@0.7.The ablation supports quality loss as a more suitable objective for suppressing low-quality action proposals in TAL.
- Ablation Study: Boundary pooling performs best with a larger proportion of the coarse action region than background.Max pooling exceeds mean, convolution, and stacking by 0.8%, 0.9%, and 0.6% average mAP, respectively.
- Ablation Study: Boundary information is more valuable for refinement than neighborhood features, while frame-level features complement rather than replace FPN features.Using only frame-level features causes a 1.1% average mAP drop relative to naive refinement.
- Ablation Study: Inference-speed experiments on THUMOS14 compare AFSD with other methods to assess its efficiency.The reported efficiency is attributed to lighter prediction and refinement modules enabled by the anchor-free mechanism.
5. Conclusion
The paper presents anchor-free temporal action localization as a promising alternative and combines an end-to-end predictor with boundary refinement, boundary pooling, and Boundary Consistency Learning.
- 5. Conclusion: AFSD explores a purely anchor-free formulation of temporal action localization.The paper argues that this formulation can provide a promising alternative to existing approaches.
- 5. Conclusion: The model combines an end-to-end basic predictor with a temporal refinement module.The refinement module uses novel boundary pooling and Boundary Consistency Learning to improve boundary features.
- 5. Conclusion: AFSD achieves remarkable results on THUMOS14 and comparable results on ActivityNet1.3.The conclusion identifies anchor-free modeling as a promising choice for temporal action localization.