Source-linked AI summary
Learning High-Quality Pseudo-Labels for Point-Supervised Temporal Action Localization
Zhenying Fang
TL;DR
PTAL reduces annotation needs to points, but available temporal boundary supervision is absent and recovering complete intervals remains difficult with weaker labels. This paper combines point-guided pseudo-label initialization with consensus-guided refinement, and reports improved detection across four benchmarks.
Problem
Point-supervised localization must construct temporal pseudo-labels because training annotations provide instance timestamps and categories but not boundaries.
Source: §3.1 · p. 3Method
PGI uses point-derived priors to select initial pseudo-labels and retain alternatives, while CGR combines candidate consensus with detector predictions to refine boundary targets.
Source: §1 · p. 2, §3.3 · p. 5Results
Experiments on four benchmarks demonstrate improved detection performance, and analyses validate both modules and effectiveness across proposal generators and detector architectures.
Source: §1 · p. 2, §5 · p. 11Takeaways & Limitations
The findings highlight the value of retaining and reusing original proposal evidence throughout pseudo-label learning.
Source: §5 · p. 11Training annotations provide timestamps and categories, but instance boundaries are unavailable.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Point-supervised temporal action localization (PTAL) aims to recognize and localize action instances in untrimmed videos using only point-level annotations. Existing pseudo-label-based methods have made substantial progress in PTAL. However, confidence-based initialization can overlook well-localized proposals. Meanwhile, existing methods either keep pseudo-labels fixed during detector training or update them without fully exploiting alternative boundary evidence from the original proposal pool. These limitations constrain supervision quality, thereby hindering detection performance. To learn high-quality pseudo-labels, we propose a framework that integrates prior-guided initialization with consensus-guided refinement during detector training. Our framework comprises two modules: Prior-Guided Initialization (PGI) and Consensus-Guided Refinement (CGR). PGI combines proposal confidence with point-derived priors to establish instance-specific reliability for initial pseudo-label selection and candidate retention. CGR continually uses the retained original candidates to construct boundary targets, combining their consensus with evolving detector predictions to refine pseudo-labels throughout training. Experiments on four benchmarks demonstrate that our method achieves competitive detection performance compared with existing methods. In particular, our method achieves 68.5\% average mAP over tIoU thresholds 0.1--0.7 on THUMOS14, surpassing PseR by 3.7 percentage points. Code is available at https://github.com/zhenyingfang/LHQP
1. Introduction
The introduction frames pseudo-label quality in PTAL as a problem of both initialization and in-training evolution. It proposes PGI and CGR to select reliable labels, retain alternatives, and refine boundaries using candidate consensus and detector predictions.
1. Introduction
- Precise-boundary TAL annotations are costly, while video-level weak supervision lacks instance cues and makes complete action intervals difficult to recover.Source: §1 · p. 1
- PTAL uses one annotated frame and category per instance, but classification-derived activation sequences have a mismatch with localization objectives.Source: §1 · p. 1
- Confidence can overlook well-localized candidates, while fixed labels or narrowed pools underuse alternative boundaries during detector training.Figure 1(a) contrasts top-score and oracle proposals from the same point-compatible pool.Source: Figure 1 · p. 1, §1 · p. 2
- PGI combines confidence, geometric, and conflict priors to select initial pseudo-labels and retain alternative candidates with original boundaries and reliability scores.Source: §1 · p. 2
- CGR combines reliability-weighted candidate consensus with current detector predictions to construct boundary targets and guide consistency-controlled updates.It raises the fraction of labels with tIoU ≥0.7 over fixed PGI by 7.3 percentage points on THUMOS14 and 0.9 on ActivityNet1.3.Source: Figure 1 · p. 1, §1 · p. 2
- Experiments on four benchmarks show improved pseudo-label quality and competitive detection, while ablations validate the complementary roles of PGI and CGR.The benchmarks are THUMOS14, ActivityNet1.3, GTEA, and BEOID.Source: §1 · p. 2
2. Related Work
Related work moves from fully supervised temporal detectors to weakly supervised and point-supervised localization, where sparse cues motivate pseudo-label construction. This paper focuses on retaining and reusing original proposal boundaries after initialization.
2.1. Temporal Action Localization
- TAL detectors span anchor-based, anchor-free, hybrid, and query-based designs, but training still relies on costly temporal boundary annotations.Anchor-based methods use predefined temporal windows; anchor-free methods predict boundary distances directly, while query-based methods use set prediction.Source: §2.1 · p. 2
2.2. Weakly-Supervised Temporal Action Localization
- Weakly supervised methods derive intervals from video-level labels, but discriminative snippet scores can miss less distinctive action parts; pseudo-labels add direct boundary supervision despite absent instance positions.Point annotations provide sparse instance-level cues for proposal reliability and preservation of original boundary hypotheses.Source: §2.2 · pp. 2–3
2.3. Point-Supervised Temporal Action Localization
- PTAL methods either strengthen representations from points or construct pseudo-labels, while this work emphasizes continued use of original proposal evidence after initialization.Retained boundaries remain fixed, but their supervision weights evolve with detector predictions.Source: §2.3 · p. 3
3. Method
The method builds temporal pseudo-labels from point annotations and an original proposal pool, then improves them through prior-guided selection and consensus-guided refinement during detector training.
3.1. Problem Formulation and Overview
- Each training video has point annotations containing an action timestamp and category, while instance boundaries are unavailable; a proposal pool supplies candidate intervals, confidence scores, and predicted categories.The framework uses this pool and the point annotations to construct pseudo-labels and train a detector.Source: §3.1 · p. 3
3.2. Prior-Guided Initialization (PGI)
- PGI combines confidence, geometric, and conflict priors to assess candidate reliability and select an initial pseudo-label for each annotated point.Confidence scores alone do not fully capture instance-level localization quality; the geometric and conflict priors provide additional point-derived compatibility evidence.Source: §3.2 · pp. 3–4
- PGI retains the K highest-reliability candidates, keeping their original boundaries fixed so CGR can reuse alternatives to the initial pseudo-label.When fewer than K candidates are available, PGI retains all of them.Source: §3.2 · p. 5
3.3. Consensus-Guided Refinement (CGR)
- CGR combines reliability-weighted consensus from retained proposals with current detector predictions to construct boundary targets for pseudo-label refinement.Consensus fusion is gated by temporal overlap; pseudo-label updates use a consistency criterion and momentum, while ineligible labels remain unchanged.Source: §3.3 · pp. 5–6
3.4. Training and Inference
- Training begins with PGI-initialized pseudo-labels and replaces temporal supervision with refined labels after CGR updates; inference uses the trained detector alone.The original proposals and their reliability scores remain fixed during training, while PGI, CGR, point annotations, and the proposal pool are not used at inference.Source: §3.4 · p. 6
4. Experiment
Across four benchmarks, the framework achieves competitive detection results, and gains persist across proposal generators and detector architectures. Ablations show contributions from PGI, CGR, point-derived priors, and refinement choices; qualitative examples illustrate improved temporal alignment.
4.2. Comparison with State-of-The-Art Methods
- 68.5% average mAP over tIoU thresholds 0.1–0.7 on THUMOS14 surpasses PseR by 3.7 percentage points; ActivityNet1.3 reaches 30.9% average mAP over 0.5–0.95, 0.3 points above PseR.On THUMOS14, the method also improves over PseR by 5.4 and 5.9 points at tIoU 0.5 and 0.7, respectively.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
- 62.2% average mAP over tIoU thresholds 0.1–0.7 on GTEA surpasses TRA and PseR by 8.1 and 9.9 points, while BEOID reaches 74.2%, 5.3 points above PseR.On BEOID, mAP at tIoU 0.7 rises from PseR’s 28.9% to 42.3%.Source: §4.2 · pp. 6, 8, Table 3 · p. 7, Table 4 · p. 8
4.3. Generalization across Proposal Generators and Detectors
- Across four matched generator–detector combinations on THUMOS14, PGI and CGR raise average mAP over tIoU thresholds 0.3–0.7 by 4.6–5.4 percentage points.The combinations use LACP or HR-Pro proposals with ActionFormer or TriDet detectors.Source: Figure 3 · p. 8, §4.3 · p. 8
4.4. Ablation Studies
- Adding PGI and then CGR improves average mAP on THUMOS14, ActivityNet1.3, GTEA, and BEOID, with respective gains of 2.9/1.5, 0.9/0.9, 0.5/5.2, and 2.7/2.0 points.The ablation uses HR-Pro proposals and ActionFormer.Source: §4.4 · p. 8, Table 5 · p. 8
- CGR reaches mean pseudo-label tIoU 0.708 and a 62.4% high-quality label fraction; progressive consistency control also reaches 68.5% average mAP, 4.2 points above unconstrained updates.CGR improves mean tIoU by 0.030 and the high-quality fraction by 7.3 points over fixed PGI labels; full CGR reaches 68.5% average mAP versus 67.4% for detector-only refinement.Source: Figure 4 · p. 9, §4.4 · pp. 9–10, Table 7 · p. 9
- Combining geometric and conflict priors yields initial pseudo-labels with mean tIoU 0.678 and a 55.1% fraction at tIoU ≥0.7, improving on confidence-only selection.The conflict prior gives a higher mean tIoU than the geometric prior alone, while the geometric prior yields a higher fraction of well-localized labels.Source: §4.4 · pp. 8–9, Table 6 · p. 9
- On THUMOS14, retaining 10 proposals raises AVG mAP from 68.0% at K = 1 to 68.5%, which remains unchanged at K = 20.The study identifies K = 10 as a compact default; consensus thresholds 0.3 and 0.5 yield comparable AVG mAP on THUMOS14 and ActivityNet1.3.Source: Figure 5 · p. 10, §4.4 · p. 10
4.5. Qualitative Analysis
- In two THUMOS14 training clips, mean tIoU rises from 0.424 to 0.854 to 0.911 for HammerThrow and from 0.525 to 0.839 to 0.903 for LongJump across top-score selection, PGI, and CGR.On two unseen test videos, predictions achieve tIoU values of 0.9 and 1.0 for CleanAndJerk and 0.7 and 0.8 for BasketballDunk.Source: §4.5 · pp. 10–11, Figure 6 · p. 11
5. Conclusion
The framework learns pseudo-labels in point-supervised temporal action localization through prior-guided initialization and consensus-guided refinement. Experiments show improved detection performance across four benchmarks, with ablations and analyses supporting both modules and broader applicability.
5. Conclusion
- The framework combines prior-guided initialization with consensus-guided refinement to learn high-quality pseudo-labels for point-supervised temporal action localization.PGI retains alternative candidates, while CGR combines their consensus with evolving detector predictions to refine supervision.Source: §5 · p. 11
- Experiments across four benchmarks demonstrate improved detection performance.Ablations and pseudo-label analyses validate both modules.Source: §5 · p. 11
- The framework remains effective with different proposal generators and detector architectures, highlighting the value of reusing original proposal evidence.Source: §5 · p. 11