PaperlayerView exact revision on arXiv

Source-linked AI summary

P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization

Yunchuan Ma, Laiyun Qing, Sai Xu, Guorong Li, Qingming Huang

arXiv:2610.34618v1cs.CV

TL;DR

Point-supervised temporal action localization uses sparse annotations, while existing methods mainly classify snippets rather than explicitly learning complete temporal localization. P2P-Former generates matched pseudo proposals and incorporates point-derived semantic cues and a point-coverage constraint; it outperforms several existing state-of-the-art approaches on THUMOS14 and ActivityNet-1.3.

  • Problem

    Point annotations are less complete than proposal annotations, and existing point-supervised methods mainly rely on snippet-level classification, leaving a gap from fully supervised models.

    Source: §1 · p. 2

  • Method

    P2P-Former matches point-aware proposals as pseudo labels, injects prototypes from point-annotated features, and constrains predicted proposals to cover annotated points.

    Source: §1 · pp. 3–4

  • Results

    On THUMOS14 and ActivityNet-1.3, P2P-Former outperforms several existing state-of-the-art approaches.

    Source: §1 · p. 4

  • Takeaways & Limitations

    P2P-Former provides instance-level learning intended to narrow the performance gap with fully supervised methods.

    Source: §1 · p. 4

Abstract

from arXiv · show

Recently, point-supervised temporal action localization has attracted increasing attention due to its low annotation cost (only one frame per action instance) and competitive detection performance. However, since point annotations are less complete than proposal annotations, existing methods mainly rely on multiple instance learning for snippet-level classification, which leads to a performance gap compared to fully supervised models. To bridge this gap, we propose P2P-Former, a novel point-to-proposal Transformer-based framework that effectively connects point-level supervision with proposal-level learning for temporal action localization. In this work, we identify two key challenges in transitioning from point-level supervision to full supervision: (1) how to generate reliable proposals as pseudo ground-truth, and (2) how to effectively integrate point-level cues into the fully supervised paradigm. To address the first challenge, we propose a Point-aware Proposal Generation (PPG) strategy that refines the confidence of proposals produced by a base locator using a point-aware importance estimation function, and then performs optimal matching between point annotations and candidate proposals to select reliable proposals as pseudo ground-truth. To tackle the second challenge, we introduce a Point-guided Semantic Enhancement (PSE) module that aggregates prototypes from point-annotated features and injects them into snippet features through Transformer-based interactions. Moreover, we design a Point-to-Proposal Constraint Loss (PPCL) that enforces each predicted proposal to include the annotated point, thereby encouraging more accurate localization. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our method outperforms several existing state-of-the-art approaches.

1. Introduction

Point supervision lowers annotation costs, but existing methods largely classify snippets and threshold class activation sequences without explicitly learning temporal localization. P2P-Former connects point annotations to proposal-level learning through pseudo proposals, point-derived semantic features, and a point-to-proposal constraint.

Motivation

  • Existing point-supervised methods use sparse labels for classification and threshold class activation sequences, without explicitly learning temporal localization.The paper argues that more complete action instances are needed to support temporal boundary regression alongside classification.Source: §1 · pp. 2–3

P2P-Former

  • P2P-Former uses point-aware proposal generation and optimal matching to select reliable proposals as supervisory signals.It frames proposal generation and integration of point-level information as two challenges in moving to proposal-based learning.Source: §1 · pp. 3–4
  • Point-guided semantic enhancement injects prototypes from point-annotated features, while PPCL enforces coverage of the annotated point.The constraint is intended to guide more accurate proposal generation.Source: §1 · p. 4

Experiments

  • Experiments on THUMOS14 and ActivityNet-1.3 show that P2P-Former outperforms several existing state-of-the-art approaches.The introduction reports the benchmark comparison without specifying numerical results.Source: §1 · p. 4

2. Related Work

Related work spans fully supervised localization, video-level weak supervision, and point-supervised methods positioned between them. The paper situates its proposal-based approach among point-supervised methods that use sparse annotations to learn action completeness and localization.

2.1. Fully-Supervised Temporal Action Localization

  • Fully supervised methods use precise temporal boundaries; one-stage methods jointly locate and classify, while two-stage methods generate proposals before category prediction and boundary adjustment.The paper notes that complete supervision provides accurate labels but has high annotation costs.Source: §2.1 · pp. 4–5

2.2. Weakly-Supervised Temporal Action Localization

  • Weakly supervised methods use video-level labels, including multiple-instance learning or attention, but lack frame-level annotations.The paper describes a notable performance gap from fully supervised methods.Source: §2.2 · p. 5

2.3. Point-Supervised Temporal Action Localization

  • Point supervision reduces annotation time to one-sixth of full supervision, and prior methods explore completeness, center priors, boundary detection, and proposal refinement.The cited comparison is 50 seconds versus 300 seconds per one-minute video; other work adds text refinement or temporal-consistency tasks.Source: §2.3 · pp. 5–6

3. Methodology

P2P-Former uses a Transformer-based temporal detector to connect point supervision with proposal-based learning. It generates matched pseudo labels, injects point-derived semantics, and constrains proposals with annotated points.

3.1. Preliminaries

  • Point annotations supervise snippet features in a Transformer-based temporal detector that predicts proposal boundaries, categories, and confidence scores.Video snippets are represented with features extracted by a pretrained I3D model.Source: §3.1 · pp. 6–7

3.2. Point-aware Reliable Proposal Generation

  • Point-aware scores combine proposal confidence with annotation proximity, then Hungarian matching assigns each point one globally optimal proposal as a pseudo label.The importance function favors points near proposal centers and suppresses annotations outside proposals or near their boundaries.Source: §3.2 · pp. 8–9

3.3. Point-guided Semantic Enhancement

  • PSE clusters point-labeled features by category into diverse prototypes and uses Transformer attention to inject their semantics into video features.Queries come from video features, while keys and values use concatenated video and prototype features.Source: §3.3 · pp. 9–11

3.4. Point-to-Proposal Constraint Loss

  • PPCL first penalizes proposals that exclude their annotated points, then activates center alignment to bring included points closer to proposal centers.Boundary regression jointly optimizes proposal extent.Source: §3.4 · pp. 11–12

3.5. Training and Inference

  • Training combines classification and boundary-regression losses with a weighted point-to-proposal constraint; inference uses P2P-Former alone and applies soft-NMS.The classification loss is focal loss, and the regression loss is IoU loss.Source: §3.5 · p. 12

4. Experiments

P2P-Former outperforms existing point-supervised methods on THUMOS14 and ActivityNet-1.3, with experiments examining its components, base locators, efficiency, and qualitative behavior.

4.1. Datasets and Evaluation

  • THUMOS14 uses 200/213 training/test videos and ActivityNet-1.3 reports validation results; evaluation uses benchmark-specific mAP IoU thresholds, with two-stream I3D features and dataset-specific training schedules.THUMOS14 has 20 categories; ActivityNet-1.3 has 200. The reported IoU ranges are [0.1:0.1:0.7] and [0.5:0.05:0.95], respectively.Source: §4.1 · p. 13, §4.2 · p. 13

4.3. Comparison with State-of-the-Art Methods

  • 65.8% average mAP@[0.1:0.7] on THUMOS14 exceeds all existing point-supervised methods, while 31.1% average mAP@[0.5:0.95] on ActivityNet-1.3 exceeds those methods there.On THUMOS14, the gains are 5.5% over HR-Pro and 24.6% over SF-Net; on ActivityNet-1.3, they are 3.4% over QROT and 7.1% over CRRC-Net.Source: Table 1 · p. 14, §4.3 · p. 14, Table 2 · p. 15

4.4. Ablation Study

  • Adding PPG raises average mAP@[0.1:0.7] from 62.6% to 64.1%; PSE reaches 64.6%, PPCL reaches 64.7%, and all three together reach 65.8%.The combined model improves 3.2% over the baseline; PPG uses more reliable pseudo proposals, while PSE and PPCL add semantic and point-position supervision.Source: §4.4 · pp. 15–16, Table 3 · p. 15
  • With P2P-Former, LACP's average mAP rises from 52.4% to 64.7% and HR-Pro's from 56.2% to 65.8%, narrowing their performance gap from 3.8% to 1.1%.The comparison evaluates P2P-Former with different PTAL base locators.Source: Table 4 · p. 16, §4.4 · p. 16
  • Two clusters per category outperform other tested counts by approximately 0.4%–0.7% mAP; λ=1 yields 65.8% average mAP@[0.1:0.7], while lower or higher weights degrade performance.The authors suggest too few clusters provide insufficient action-semantic information, while too many introduce noise.Source: Table 5 · p. 17, §4.4 · p. 17, Table 6 · p. 17
  • P2P-Former achieves 65.8% average mAP with 46.0M parameters and 49.3 ms inference latency, outperforming SF-Net, LACP, and HR-Pro by 25.6%, 13.4%, and 5.6%, respectively.The authors report higher computation from the proposal-level Transformer architecture but describe the latency as acceptable for offline video understanding.Source: Table 7 · p. 18, §4.4 · p. 18

4.5. Qualitative Results

  • ActivityNet-1.3 examples show more complete action localization and fewer redundant detections; THUMOS14 profiling shows more beneficial positives and fewer localization errors than the base locator.The qualitative comparison includes four action classes; false-positive analysis averages over tIoU thresholds 0.1–0.7.Source: §4.5 · pp. 18–19, Figure 3 · p. 19, Figure 4 · p. 20
  • Visualizations show one-to-one point/proposal matching selects pseudo-ground-truth proposals with better temporal alignment, while clustering yields two representative sub-prototypes per sampled category.The clustering visualization samples six THUMOS14 categories.Source: §4.5 · pp. 20–21, Figure 5 · p. 20, Figure 6 · p. 21

5. Conclusion

P2P-Former addresses the limits of snippet-level learning by using an instance-level framework for point-supervised temporal action localization. Experiments on two benchmarks demonstrate the proposed method’s effectiveness.

5. Conclusion

  • Experiments on two widely used benchmarks demonstrate the effectiveness of P2P-Former.Source: §5 · pp. 21–22
  • P2P-Former selects reliable pseudo proposals through point-aware evaluation and matching, then enhances snippet features with point-derived prototypes and enforces proposal–point consistency.Source: §5 · p. 21
  • Existing point-supervised methods mainly use snippet-level classification under sparse supervision, limiting their ability to model complete action instances.Source: §5 · p. 21

CRediT authorship contribution statement

The statement assigns writing, technical, administrative, and research-support contributions across four authors.

CRediT authorship contribution statement

Loading 2610.34618v1…