Source-linked AI summary
Boundary Voting Network for Ambiguity-Aware Timestamp-Supervised Action Segmentation
Runzhong Zhang, Yueqi Duan, Yang Chen, Weipeng Hu, Chen Cai, Suchen Wang, Yap-Peng Tan
TL;DR
Timestamp supervision makes boundary localization difficult because ambiguous action-transiting features destabilize pseudo-label generation. BVN propagates global action representations as votes into these local regions, and experiments report state-of-the-art performance across three public datasets and multiple metrics.
Problem
Feature ambiguity in action-transiting regions creates uncertainty in boundary localization, reducing the stability of pseudo-labels used for timestamp-supervised segmentation.
Method
BVN hierarchically propagates global prior knowledge by generating key action representations as votes and aggregating them into local action-transiting regions.
Results
BVN achieves state-of-the-art action segmentation performance across three public datasets and multiple evaluation metrics.
Takeaways & Limitations
The voting mechanism refines boundary localization and improves the quality of generated framewise pseudo-labels.
Abstract
from arXiv · showhide
Timestamp-supervised action segmentation aims to segment and classify actions in untrimmed videos with a random frame annotated per action. Precisely localizing action boundaries from timestamp annotations is crucial for this setting, as it enables generating framewise pseudo-labels and applying the well-explored fully-supervised training. However, prevailing methods struggle with intrinsic uncertainty in boundary localization due to less discriminative features in action-transiting regions. This imprecise boundary estimation significantly reduces the stability and reliability of the generated pseudo-labels in ambiguous action-transiting regions, consequently resulting in performance deterioration of the trained segmentation models. In our paper, we introduce the boundary voting network that mitigates feature ambiguity by hierarchically propagating video-level global prior knowledge into local action-transiting regions. By generating key action representations as votes throughout the video and targeting action-transiting regions, all votes collaboratively contribute to action-transiting feature enhancement and boundary localization refinement. Extensive experiments demonstrate the effectiveness of our method on GTEA, 50Salads, and Breakfast datasets.
I. INTRODUCTION
Timestamp-supervised segmentation uses one random frame annotation per action, making precise boundary localization essential for generating framewise pseudo-labels. BVN addresses ambiguity near action transitions by propagating global action representations into local regions, with experiments reporting effectiveness across three datasets.
- Timestamp-supervised action segmentation annotates only a single random frame for each action segment in the training video.
- Precise boundary localization converts timestamp annotations into framewise pseudo-labels for fully-supervised training.
- Feature ambiguity in action-transiting regions makes boundaries uncertain and destabilizes pseudo-labels, degrading subsequent segmentation training.
- BVN uses global-to-local voting, generating action representations and aggregating them into action-transiting regions to enhance features and refine boundaries.
- The method hierarchically propagates global prior knowledge, suppresses ambiguous regions, and separates adjacent actions during training and testing.
- Experiments evaluate BVN across three real-world datasets and multiple metrics, with ablations validating the voting mechanism.
A. Fully-supervised action segmentation
Action segmentation assigns action classes to video frames, while timestamp supervision provides only sparse temporal annotations. Existing timestamp-supervised approaches estimate boundaries to create pseudo-labels before applying fully-supervised learning.
- Fully-supervised action segmentation assigns a predefined action class to every frame of an untrimmed video.
- Timestamp-supervised action segmentation annotates each action segment with a single arbitrary frame rather than dense framewise labels.
- Prior methods generate framewise pseudo-labels through boundary estimation and then train segmentation models in a fully-supervised manner.
- A training video is represented by T frames and N annotated actions, with N significantly lower than T.
- The learning objective is to predict an action label for each video frame from sparse timestamp annotations.
A. Overview
BVN is an encoder-decoder action segmentation network that propagates global action knowledge toward ambiguous local transition regions. Its voting blocks produce start- and end-oriented votes, while later stages aggregate them into enhanced video features.
- BVN uses a transformer-based encoder-decoder network with voting and aggregation blocks inserted before decoder layers.
- Each frame produces separate votes toward action starts and ends, allowing action knowledge to target different temporal regions.
- Normalized temporal indexes record frame positions, while independent start and end networks generate feature and index offsets.
- The voting block combines original features and offsets to propagate key action representations to targeted temporal positions.
- Generated votes rearrange evenly distributed frame features and cluster around boundaries, providing global prior knowledge for suppressing less discriminative regions.
C. Votes aggregation
The aggregation block groups generated votes, selects representative ones with temporal farthest point sampling, and merges their information into video features. Repeated voting and aggregation suppress transition regions and refine boundary localization.
- C. Votes aggregation: The aggregation block first groups votes and then aggregates them back into the original video features.
- C. Votes aggregation: Temporal farthest point sampling selects representative key votes from the unevenly distributed vote set while limiting noise in grouping.
- C. Votes aggregation: Each key vote group encodes action content through vote features and temporal location through its centroid.
- C. Votes aggregation: Max-pooled group features update video features within a temporal window before subsequent decoder processing.
- D. Generating action-transiting regions: Boundary localization generates forward and backward pseudo-labels, whose conflicts identify action-transiting regions between consecutive actions.
- D. Generating action-transiting regions: Repeated voting and aggregation progressively suppress the temporal duration of action-transiting regions and reduce boundary-localization uncertainty.
E. Loss functions and training process
The training objective combines classification, smoothing, confidence, and voting losses, while the two-stage process first learns from timestamps and then iteratively refines pseudo-label supervision.
- Loss functions: The final objective combines framewise classification, smoothing, confidence, and voting losses, balanced by hyperparameters α, β, and γ.The voting loss is added to direct votes toward neighboring action-transiting regions.
- Loss functions: The classification loss uses cross-entropy on the model’s predicted probability for the ground-truth action at each frame.
- Loss functions: The smoothing loss uses truncated mean square error to improve neighboring-frame consistency and produce smoother prediction transitions.The loss is designed to alleviate over-segmentation.
- Loss functions: The confidence loss reflects decreasing confidence in an action label as frames move farther from its annotated timestamp.
- Loss functions: The voting loss supervises start- and end-net offsets so votes target the corresponding action-transiting regions.For each region between consecutive actions, the method computes a center and regresses temporal offsets toward it.
- Training process: Training uses initialization and iterative stages: timestamps supervise initialization, generated framewise pseudo-labels supervise iterative refinement, and voting loss begins afterward.
F. Discussions with related TCSVT papers
The paper distinguishes BVN from related work through its timestamp supervision, global-to-local voting framework, and focus on feature-level ambiguity in conventional RGB features.
- Comparison with related papers: Unlike transcript- or dense-annotation-based studies, this paper uses timestamp supervision and addresses feature-level ambiguity in conventional RGB features.
- Qualitative comparison: Figure 3 compares GTEA, 50Salads, and Breakfast visualizations, with GT marking ground-truth segmentation and red dashed rectangles highlighting boundary alignment.
- Comparison with related papers: BVN uses a global-to-local voting mechanism, distinguishing it from related papers in supervision format, methodological framework, and problem formulation.
IV. EXPERIMENTS
The experiments include quantitative and qualitative comparisons, additional quantitative analyses, and ablation studies assessing the proposed method.
- Experimental design: The experimental section covers quantitative comparisons, qualitative comparisons, additional quantitative analysis, and detailed ablation studies.
A. Experimental setup
Experiments use GTEA, 50Salads, and Breakfast with standard action-segmentation metrics, shared backbones, and comparisons of segmentation and pseudo-label quality. BVN improves both evaluation targets across datasets and backbones.
- Datasets and metrics: Experiments cover GTEA, 50Salads, and Breakfast, using framewise accuracy, segmental edit score, and F1@{10, 25, 50}.The datasets contain instructional videos spanning egocentric, salad-preparation, and breakfast activities.
- Implementation and comparison: BVN is implemented with MS-TCN++ and ASFormer backbones for fair comparison with prior timestamp-supervised methods.The voting and aggregation blocks match the number of decoders and are integrated before decoder layers.
- Action segmentation results: BVN improves framewise accuracy by +1.0% on GTEA, +1.4% on 50Salads, and +0.8% on Breakfast with MS-TCN++.With ASFormer, the corresponding improvements are +1.0%, +1.2%, and +2.6%.
- Pseudo-label results: +7.5% F1@50 improvement on GTEA with MS-TCN++ and +9.1% on 50Salads with ASFormer demonstrate stronger pseudo-label generation.The paper reports improvements across all datasets, evaluation metrics, and backbones.
- Pseudo-label results: Higher-quality pseudo-labels enhance training stability and improve action-segmentation testing performance.The reported relationship connects boundary-localization refinement with the downstream segmentation results in Table I.
C. Qualitative comparisons
Qualitative analyses show BVN refining difficult boundaries and transferring action information from explicit frames to ambiguous transition frames. These visualizations support more precise segmentation and improved probability scores.
- Action segmentation visualizations: BVN captures challenging actions missed by existing methods and provides more precise boundaries, especially in long videos with complex actions.The comparison includes Li et al., Du et al., and BVN visualizations.
- Boundary localization refinement: Across training stages, voting and aggregation progressively suppress action-transiting regions in Breakfast samples.Each stage combines the voting block, aggregation block, and decoder before producing new action-transiting regions.
- Voting visualizations: Votes from explicit “pour,” “cut tomato,” and “fry egg” frames are directed toward ambiguous transition frames, improving their probability scores.The improvement is reported relative to a baseline without voting, aggregation, and voting loss.
D. Additional quantitative analysis
Additional analyses examine feature quality, boundary-offset accuracy, annotation-budget comparisons, and computational cost. BVN is reported to outperform reduced-data fully supervised baselines, while requiring more training time.
- Analysis scope: The additional analysis evaluates action-transiting features, index offsets, fully supervised comparisons, and computational complexity.These analyses are intended to clarify the voting mechanism and its contribution to segmentation performance.
- Feature quality: Action-transiting feature quality is assessed by comparing final-decoder features for key action frames and adjacent ambiguous boundary frames.The evaluation uses 10 adjacent frames near each ground-truth boundary.
- Fully supervised comparison: Under an approximate equal annotation-time comparison, BVN outperforms fully supervised methods trained on 50% of videos while using timestamp supervision for less than 0.5% of frames.The comparison is approximate because fully supervised methods cannot use unlabeled videos.
- Computational complexity: ASFormer-based BVN requires approximately 20 hours to train on Breakfast versus 11 hours for Li et al., reflecting increased computational complexity.The paper characterizes the additional cost as arising from the voting mechanism.
E. Ablation studies
Ablations show that BVN’s voting, aggregation, loss, and bidirectional design jointly improve pseudo-label quality and segmentation, while overly broad propagation can introduce noise. The method remains reliable across varied action lengths.
- BVN significantly improves GTEA framewise pseudo-label quality over the baseline by refining boundary localization and reducing action-transiting feature ambiguity.
- The MLP-plus-max-pooling aggregation design significantly outperforms direct max pooling and average pooling, whereas voting alone remains comparable to the baseline.
- Both start and end networks outperform single-direction variants, although using both inevitably increases model parameters.
- Removing the voting loss causes significant performance declines across all datasets because voting becomes unguided and introduces noise into video features.
- A low distance threshold limits propagated prior knowledge, while a high threshold introduces irrelevant votes and noise; both extremes degrade performance.
- Increasing key vote groups improves GTEA performance initially, but raising P from 16 to 32 degrades performance because excessive noise enters the groups.
- BVN maintains reliable performance for action lengths ranging from 0.25 to 2 times the original scale, indicating generalization across varied action durations.
V. CONCLUSIONS
BVN uses global-to-local voting and aggregation to rearrange global temporal representations into local action-transiting regions. Across three public datasets and multiple metrics, it achieves state-of-the-art action segmentation performance.
- BVN rearranges globally distributed temporal features and aggregates key action representations into local action-transiting regions through hierarchical propagation.
- Across three public datasets and multiple evaluation metrics, BVN achieves state-of-the-art action segmentation performance.