PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Anchor Mechanisms for Temporal Action Localization

Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, Junwei Han

arXiv:2008.09837v1cs.CV

TL;DR

Predefined anchors constrain temporal action localization when action durations vary drastically, especially for extremely short or extremely long instances. The paper introduces a point-based anchor-free module and combines it with a conventional anchor-based module in A2Net; the combined framework reaches 45.5% mAP on THUMOS14 versus 42.8% for TAL.

  • Problem

    Predefined anchors impose location and duration priors that limit flexibility for action instances with drastic duration variation.

  • Method

    A2Net represents actions with temporal points and distances to their boundaries, then integrates anchor-free and conventional anchor-based localization modules.

  • Results

    45.5% mAP at threshold 0.5 on THUMOS14 surpasses TAL’s 42.8% by 2.7 points.

  • Takeaways & Limitations

    The anchor-free and anchor-based modules are complementary, enabling A2Net to localize actions with varied durations in a simple framework.

Abstract

from arXiv · show

Most of the current action localization methods follow an anchor-based pipeline: depicting action instances by pre-defined anchors, learning to select the anchors closest to the ground truth, and predicting the confidence of anchors with refinements. Pre-defined anchors set prior about the location and duration for action instances, which facilitates the localization for common action instances but limits the flexibility for tackling action instances with drastic varieties, especially for extremely short or extremely long ones. To address this problem, this paper proposes a novel anchor-free action localization module that assists action localization by temporal points. Specifically, this module represents an action instance as a point with its distances to the starting boundary and ending boundary, alleviating the pre-defined anchor restrictions in terms of action localization and duration. The proposed anchor-free module is capable of predicting the action instances whose duration is either extremely short or extremely long. By combining the proposed anchor-free module with a conventional anchor-based module, we propose a novel action localization framework, called A2Net. The cooperation between anchor-free and anchor-based modules achieves superior performance to the state-of-the-art on THUMOS14 (45.5% vs. 42.8%). Furthermore, comprehensive experiments demonstrate the complementarity between the anchor-free and the anchor-based module, making A2Net simple but effective.

I. INTRODUCTION

Anchor-based localization uses predefined temporal priors that work well for common durations but restrict flexibility for highly variable actions. A2Net combines an anchor-free point-based module with an anchor-based module, achieving complementary performance and 45.5% mAP on THUMOS14 versus 42.8% for TAL.

  • Motivation: Temporal action localization identifies action boundaries and category labels in untrimmed videos.The task supports applications including surveillance, highlight extraction, and video summarization.
  • Limitations of anchor-based methods: Predefined anchors encode temporal location and duration priors, limiting anchor-based methods when action lengths vary substantially.Anchor-based methods select the closest anchor and regress its parameters toward the ground truth.
  • Proposed approach: The anchor-free module represents each action as a temporal point with distances to its starting and ending boundaries.All temporal locations are treated equally, and every point within an action can contribute a training sample.
  • Complementarity: Anchor-free localization performs better for extremely short and extremely long actions, whereas anchor-based localization is more appropriate for medium or long actions.The paper attributes this pattern to anchor-free flexibility and anchor-based classification-and-detection stability.
  • Results: 45.5% mAP at threshold 0.5 on THUMOS14 exceeds TAL’s 42.8% by 2.7 points.A2Net integrates both modules using a shared backbone; their predictions are merged before standard NMS.

II. RELATED WORK

Related work includes one-stage and two-stage anchor-based pipelines, bottom-up actionness-guided methods, and other localization strategies. The paper positions A2Net’s top-down anchor-free branch as distinct from actionness-guided approaches while complementing anchor-based localization.

  • Anchor-based pipelines: Anchor-based localization methods are commonly divided into one-stage and two-stage pipelines.Both represent actions with predefined anchors, select anchors near ground truth, and regress boundary adjustments.
  • Anchor limitations: Predefined anchors work well when action-distribution priors are reliable but become inflexible for actions with drastic variation.Anchor-free localization instead predicts distances to action boundaries autonomously.
  • Actionness-guided methods: Bottom-up actionness-guided methods estimate frame-level or boundary probabilities before generating and scoring action proposals.Examples include CDC, SSN, BSN, and BMN.
  • Positioning A2Net: Actionness-based methods can produce background false positives, while A2Net’s anchor-free branch predicts a confidence score for each temporal point.The paper distinguishes A2Net’s top-down design from the multiple procedures used by methods such as BSN.
  • Other approaches: The related-work landscape also includes sliding-window, synthesized-data, reinforcement-learning, recurrent, and graph-convolutional approaches.

C. Object Detection using Anchor Mechanism

A2Net adapts anchor mechanisms from object detection to temporal sequences through hierarchical features and two complementary localization branches. Its anchor-free branch predicts boundary distances, while its anchor-based branch uses anchors, overlap, and regression parameters.

  • Anchor mechanisms: Object detection uses anchor-based representations or anchor-free boundary-distance and key-point representations.FCOS predicts distances from a point to object boundaries; A2Net adapts the boundary-distance idea to temporal action localization.
  • A2Net architecture: A2Net contains hierarchical feature, anchor-free localization, and anchor-based localization modules.The hierarchical network constructs temporal features before both localization branches make predictions.
  • Anchor-free localization: The anchor-free branch predicts classification scores and distances to action boundaries to determine potential action instances.
  • Anchor-based localization: The anchor-based branch predicts classification score, overlap value, and regression parameters after matching predefined anchors with ground truth.Predictions from both branches are merged and processed by standard NMS during evaluation.
  • Feature preprocessing: The Base Conv Layer processes I3D frame features by expanding the receptive field and reducing computation for subsequent layers.Its listed components include two convolutional layers and a max-pooling layer.
  • Hierarchical features: The hierarchical feature module uses cascaded temporal convolutions to generate features at multiple pyramid levels.The implementation uses six levels and stride 2 to reduce temporal length at each level.

C. Anchor-Based Localization Module

The anchor-based module predicts classification, overlap, and regression values from predefined anchors, while the anchor-free module regresses boundary distances from temporal points. The two mechanisms differ in their handling of action durations and localization flexibility.

  • The anchor-based module jointly predicts classification scores, overlap value, and regression parameters for predefined anchors.
  • Anchor-based localization represents an action instance using the default anchor’s center and width together with learned regression parameters.
  • Training makes predictions for anchors at each temporal location and uses only accurately predicted anchors to learn overlap and regression parameters.
  • Anchors are matched with ground truth by IoU, with anchors above 0.5 treated as positive samples and a 1:1 positive-to-negative training ratio.
  • The anchor-free module regresses distances from a temporal point to the starting and ending boundaries, providing greater flexibility for extremely short or long actions.
  • Action instances are assigned to hierarchical pyramid levels according to temporal length, and foreground points alone regress boundary distances.

E. Training

A2Net trains anchor-free and anchor-based localization losses jointly in an end-to-end framework. The training objective combines classification, regression, and overlap losses from the two modules.

  • A2Net simultaneously predicts with anchor-free and anchor-based localization modules and combines their losses for end-to-end training.
  • The anchor-free classification loss uses standard cross-entropy over all temporal locations.
  • The anchor-free regression loss uses smooth L1 loss over foreground points to regress distances to the starting and ending boundaries.
  • The anchor-based classification loss uses cross-entropy over selected positive and negative anchors.
  • The anchor-based overlap loss estimates proposal IoU and applies mean square error, while the regression loss is computed over selected anchors.
  • The anchor-free module is inserted into the Action Proposal Network to generate proposals alongside the conventional anchor-based module before later aggregation.

F. Inference

During inference, A2Net obtains localization predictions from both modules, assigns confidence scores, and merges all results before non-maximum suppression.

  • The anchor-free module converts predicted boundary distances into action boundaries and uses the maximum classification score as confidence.
  • The anchor-based module fuses its classification score and overlap estimate to form the detection confidence.
  • A2Net collects anchor-free and anchor-based results from all hierarchical points, applies NMS, and returns the final localization results.

G. Anchor-free Module for Generating Action Proposals

A2Net extends proposal generation by adding an anchor-free branch to the Action Proposal Network alongside the conventional anchor-based branch. Their proposals are combined before NMS, and the framework is presented as complementary across localization pipelines.

  • A2Net integrates anchor-based and anchor-free mechanisms into a unified action localization framework.
  • The authors state that complementarity between the anchor-free and anchor-based modules may also apply to two-stage action localization methods.
  • The anchor-free module generates proposals in parallel with the traditional anchor-based module inside the Action Proposal Network.
  • All proposals are gathered together and processed with NMS to generate candidate proposals.

IV. EXPERIMENTS

Experiments on THUMOS14 and ActivityNet v1.3 evaluate the complementarity of anchor-free and anchor-based modules, their fusion, and architectural choices in A2Net. Results show that joint learning and complementary feature patterns improve localization, while additional capacity and direct result fusion have limits.

  • Experimental setup: Experiments use THUMOS14 and ActivityNet v1.3, with sliding-window processing on THUMOS14 and fixed-length feature sequences on ActivityNet v1.3.THUMOS14 contains 20 labeled categories; ActivityNet v1.3 contains 200 labeled categories.
  • Complementarity: The anchor-free and anchor-based modules learn different localization cues: boundary distances and absolute distances versus offsets from default anchors and relative distances.The modules share the backbone but independently make predictions, supporting their complementary feature patterns.
  • Complementarity: Under mAP@0.5 on THUMOS14, A2Net exceeds the anchor-based module by 4.0 points, while direct prediction fusion falls behind A2Net by 3.4 points.These comparisons support jointly learning both modules rather than merely merging their localization results.
  • Module analysis: The anchor-free module recalls 74.87% of the top 200 proposals versus 73.75% for the anchor-based module, indicating that classification limits its standalone localization performance.The paper notes that an auxiliary classification branch could help but would make the anchor-free module more complicated.
  • Complementarity: On ActivityNet v1.3, A2Net reaches average mAP 27.75, compared with 25.97 for anchor-free, 25.68 for anchor-based, and 26.40 for direct fusion.The improvement under threshold 0.95 is not obvious, and anchor-based and fused methods perform better there.
  • Architecture: A2Net performance increases from 42.5 to 45.5 as the pyramid grows from 3 to 6 levels, whereas individual modules show limited improvement or fluctuation.A 7-level architecture provides no noticeable additional gain, so the 6-level architecture is selected.
  • Architecture: False positives increase when extra pyramid levels add temporal points, and neither individual module fully exploits the added capacity, causing performance fluctuations.The reported GTAN comparison also shows lower mAP@0.5 than the six-level anchor-based branch: 33.5 versus 36.6.
  • Feature fusion: Feature fusion improves performance over spatial-only and temporal-only inputs, while early fusion in A2Net exceeds late fusion by 2.0 points.A2Net therefore adopts concatenation-based early fusion of spatial and temporal features.

D. Comparison with state-of-the-art methods

A2Net is compared with early, anchor-based, and actionness-guided methods on THUMOS14 and ActivityNet v1.3. Its results are strongest at several overlap thresholds, while performance varies across datasets and competitors.

  • A2Net exceeds GTAN under thresholds 0.3:0.5 and TAL under thresholds 0.1:0.5 on THUMOS14.
  • A2Net exceeds BMN under thresholds 0.3:0.6, indicating flexibility for actions with different durations.
  • A2Net shows the best THUMOS14 performance under the official mAP@0.5 metric and the best average mAP across thresholds 0.1:0.5.
  • A2Net performs better than several ActivityNet methods but remains inferior to BSN, BMN, and GTAN.
  • Visualization shows that combining the modules precisely localizes instances that either anchor-free or anchor-based predictions localize inaccurately alone.

E. Model Efficiency

A2Net balances localization effectiveness with moderate computational and inference costs. Its efficiency differs from BMN and weakly supervised alternatives because computation and inference time do not align uniformly.

  • A2Net requires medium computation FLOPs and medium inference time while achieving superior localization performance among supervised methods.
  • BMN requires fewer computations than A2Net but has longer inference time because its boundary matching layer processes all proposals.
  • 3C-Net is computationally efficient but has an 18.9 performance gap from A2Net for mAP at threshold 0.5.
  • The authors characterize A2Net as a suitable choice when efficiency and effectiveness are considered together.

F. Anchor-free Module for Generating Action Proposals

The anchor-free module is integrated into R-C3D’s Action Proposal Network to test whether it improves proposal generation and action localization. It consistently improves both evaluation outcomes.

  • The anchor-free module is integrated into the Action Proposal Network and trained end-to-end with a reproduced R-C3D implementation.
  • 3 to 7 points of improvement are obtained under thresholds 0.1:0.7 over the reproduced R-C3D implementation.
  • The anchor-free module consistently improves proposal quality under AR@AN and AUC metrics.
  • The reported improvement is attributed to complementarity between the anchor-free and anchor-based modules.

V. CONCLUSION

The paper introduces an anchor-free mechanism for diverse-duration temporal action localization and combines it with an anchor-based module in A2Net. The framework achieves promising THUMOS14 performance while leaving end-to-end feature learning for future work.

  • A2Net represents actions with a temporal point and distances to both boundaries, avoiding prior bias about location and duration.
  • Comprehensive experiments demonstrate complementarity between the anchor-free and anchor-based modules.
  • A2Net is reported to precisely localize action instances with various durations and provide a simple, effective baseline on THUMOS14.
  • A2Net is only competitive on ActivityNet v1.3, although module complementarity remains verified there.
  • Most existing methods, including A2Net, extract video features before learning action localization; joint end-to-end learning is proposed as future research.
Loading 2008.09837v1…