Source-linked AI summary
Alleviating Over-segmentation Errors by Detecting Action Boundaries
Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka
TL;DR
Temporal action segmentation must identify action segments in untrimmed videos, where over-segmentation remains a critical problem. ASRF combines long-term shared features with separate action-classification and class-agnostic boundary-regression branches, plus a smoothing loss, to refine predictions. It outperforms state-of-the-art methods across three challenging datasets, with gains of up to 16.1% in segmental F1 score and 13.7% in segmental edit distance.
Problem
Temporal action segmentation must recognize and classify action segments in untrimmed videos, but existing methods can produce fluctuating predictions and over-segmentation errors.
Method
ASRF uses a long-term feature extractor with separate Action Segmentation and class-agnostic Boundary Regression branches, refining frame-wise predictions with predicted boundaries and a smoothing loss.
Results
Up to 16.1% improvement in segmental F1 score and 13.7% improvement in segmental edit distance are reported over state-of-the-art methods on three challenging datasets.
Takeaways & Limitations
The framework alleviates over-segmentation errors and performs especially strongly on the Breakfast dataset, which has more videos and action classes than the other datasets.
Takeaways & Limitations
Boundary precision is low because precise action boundaries are difficult to predict under ambiguous annotations and a strict five-frame threshold.
Abstract
from arXiv · showhide
We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code will be publicly available soon.
1. Introduction
Temporal action segmentation targets frame-level recognition of action segments in untrimmed videos, where fluctuating predictions create over-segmentation errors. ASRF addresses this by combining frame-wise classification with class-agnostic boundary detection and reports improvements across challenging datasets.
- Motivation: Untrimmed videos contain multiple action instances, so temporal action segmentation must recognize and classify each action segment.The task has potential applications in robotics, surveillance, and human-activity analysis.
- Motivation: Existing feature-extraction and temporal-classification methods can produce fluctuating action predictions and over-segmentation errors, especially with diverse action classes.These errors are particularly difficult for methods evaluated on larger datasets with many action categories.
- Motivation: Over-segmentation is critical because it detects extra steps in action sequences; reducing it is essential when actions occur as stable chunks.The paper illustrates this issue using cooking videos, where extra predicted segments correspond to nonexistent steps.
- Proposed framework: ASRF uses a long-term feature extractor, an Action Segmentation Branch, and a Boundary Regression Branch to refine frame-level action predictions with predicted boundaries.The ASB predicts frame-wise action classes, while the BRB predicts action boundaries independently of action class.
- Results: 16.1% improvement of segmental F1 score, 13.7% improvement of segmental edit distance, and 2.6% improvement for frame-wise accuracy are reported against state-of-the-art methods.These results are reported across three challenging action-segmentation datasets.
2. Related Work
Related work extracts spatiotemporal features and models temporal action sequences with sliding windows, probabilistic models, recurrent networks, and temporal convolutional architectures. Existing methods increasingly capture long-range dependencies and reduce over-segmentation, while ASRF is evaluated against these approaches under a distinct setting.
- Action segmentation: Earlier approaches detect action segments with sliding windows and remove redundant hypotheses using non-maximum suppression.
- Action segmentation: Other methods model temporal action sequences with Markov models, recurrent networks, or semi-Markovian models for frame-wise classification and joint segmentation.
- Temporal modeling: Temporal convolutional networks and deformable convolutions capture long-range dependencies between actions, although temporal pooling may lose temporal information.
- Temporal modeling: MS-TCN stacks dilated temporal convolutional networks with residual connections and uses a loss that penalizes over-segmentation errors.Related work also includes bilinear pooling combined with MS-TCN and graph-based approaches.
- Comparison scope: Domain-adaptation methods are not compared because their experimental setting differs from the setting used in this work.
- Action proposals: Action proposal generation methods use anchor-based or anchor-free approaches to produce proposals and confidence scores.
3. Our Proposed Method
ASRF separates frame-wise action classification from class-agnostic boundary regression, using shared long-term features and predicted boundaries to refine action segments. It also introduces GS-TMSE to smooth action probabilities while reducing penalties at true transitions.
- Framework: ASRF uses a long-term feature extractor with two decoupled branches: ASB predicts frame-wise actions, while BRB regresses action-boundary probabilities.The extractor expands the temporal receptive field and supplies shared features to both branches.
- Boundary Regression Branch: The class-agnostic BRB predicts general action boundaries rather than modeling transitions between every pair of action classes.This design requires less training data than class-aware methods and is intended to improve robustness.
- Refining Action Segmentation Results: During inference, local maxima of BRB boundary probabilities above θp divide the video into segments, whose labels are assigned by ASB majority voting.The refinement process is applied only during inference.
- Loss Function: The joint loss combines ASB and BRB losses as L = Lasb + λLbrb, with λ set to 0.2 for GTEA and 0.1 for 50 Salads and Breakfast.The framework averages losses across predictions in each branch; both ASB and BRB use four predictions in the described configuration.
- Loss Function for ASB: The ASB evaluates cross entropy, class-weighted cross entropy, Focal Loss, TMSE, and GS-TMSE, including combinations of these losses.The loss comparisons address class-frequency imbalance and temporal-transition smoothing.
- Loss Function for ASB: GS-TMSE applies a Gaussian similarity weight so adjacent frames with large feature differences receive smaller smoothing penalties than similar neighboring frames.It addresses TMSE's penalty on frames where actions actually transition; the paper uses frame-level input features for similarity, σ = 1.0, and τ = 4.
4. Experiments
Experiments evaluate ASRF on three challenging datasets using segmentation, boundary, and refinement analyses. ASRF improves segment-sensitive metrics, reduces over-segmentation, and shows limitations when predicted boundaries cannot correct incorrect ASB segments.
- Experimental setup: Evaluation uses 50 Salads, GTEA, and Breakfast with frame-wise accuracy, segmental edit distance, and segmental F1 metrics.Boundary regression is evaluated with boundary F1, using thresholded local-maximum boundary probabilities.
- Experimental setup: Oracle-ASB experiments provide an upper bound for evaluating BRB efficacy, while oracle-BRB refinement is not assumed achievable.The authors use a ground-truth frame-labeling ASB with a single BRB, but note that even an oracle BRB cannot provide an achievable refinement upper bound.
- Comparison with state of the art: Up to 8.3% improvement in segmental edit distance and 10.6% improvement in segmental F1 are reported on 50 Salads and GTEA, with competitive frame-wise accuracy.On Breakfast, ASRF outperforms existing methods by a large margin across all evaluation metrics.
- Comparison with state of the art: 7.1% improvement on GTEA at F1@50 over MS-TCN with bilinear pooling shows stronger performance at stricter overlap thresholds.The methods perform similarly at F1@10, whereas the larger-overlap criterion favors ASRF.
- Refinement analysis: Action-boundary refinement improves every segmental metric by over 8.8% on 50 Salads while maintaining comparable frame-wise accuracy.ASRF outperforms smoothing, similarity-based refinement, and relabeling; unlike relabeling, it captures segments irrespective of temporal length.
- Architecture and boundary analysis: Four stages do not improve 50 Salads predictions, which the authors attribute to overfitting given the dataset size.The ASB and BRB can mutually support refinement: false-positive BRB boundaries may be ignored when ASB predictions lack over-segmentation.
- Architecture and boundary analysis: BRB precision is low because boundary locations and annotations are ambiguous, yet overdetected boundaries can still assist refinement.Segmental metrics tolerate boundary shifts, producing little to no correlation between boundary scores and segmentation scores.
- Loss-function analysis: Up to 3.6% improvement in segmental metrics comes from the proposed smoothing loss, while focal loss negatively influences performance.The smoothing loss is designed to smooth action-probability transitions without penalizing genuine action transitions.
5. Conclusions
The framework combines action segmentation with boundary regression and loss-function analysis to recognize action segments and alleviate over-segmentation errors. It outperforms state-of-the-art methods on three challenging datasets, especially Breakfast.
- ASRF combines an action segmentation network with an action boundary regression network to refine segmentation results using predicted boundaries.
- The framework evaluates various loss functions and their combinations for action segmentation.
- ASRF recognizes action segments and alleviates over-segmentation errors.
- ASRF outperforms state-of-the-art methods on three challenging datasets, especially the larger, more diverse Breakfast dataset.Breakfast contains more videos and action classes than the other datasets.