Source-linked AI summary
Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
Li Ding, Chenliang Xu
TL;DR
Weakly supervised action segmentation needs scalable alternatives to computationally expensive sequence models and precise temporal annotations. The paper combines TCFPN for frame-wise prediction with ISBA for iterative transcript alignment and soft boundary assignment, and reports state-of-the-art performance on both weakly supervised segmentation and alignment.
Problem
Weakly supervised action segmentation seeks scalable frame-wise labeling from coarse action transcripts because dense temporal annotations are tedious and existing RNN- and HMM-based methods are computationally expensive.
Method
The framework combines TCFPN, a temporal convolutional network for frame-wise labels, with ISBA, which iteratively refines transcripts, assigns soft boundaries, and updates the network.
Results
The complete TCFPN+ISBA system outperforms state-of-the-art methods on weakly supervised action segmentation and alignment.
Takeaways & Limitations
ISBA is presented as a general training strategy that can integrate with other segmentation work or be used in other video action-understanding tasks.
Takeaways & Limitations
Iterative transcript refinement can overfit network behavior, so the method requires a stop criterion based on video-level recognition loss.
Abstract
from arXiv · showhide
In this work, we address the task of weakly-supervised human action segmentation in long, untrimmed videos. Recent methods have relied on expensive learning models, such as Recurrent Neural Networks (RNN) and Hidden Markov Models (HMM). However, these methods suffer from expensive computational cost, thus are unable to be deployed in large scale. To overcome the limitations, the keys to our design are efficiency and scalability. We propose a novel action modeling framework, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion. The proposed framework is evaluated on two benchmark datasets, Breakfast and Hollywood Extended, with four different evaluation metrics. Extensive experimental results show that our methods achieve competitive or superior performance to state-of-the-art methods.
1. Introduction
The paper targets weakly supervised action segmentation using only action transcripts, motivated by the cost and inconsistency of dense temporal annotations and the computational burden of existing RNN- and HMM-based methods. It introduces TCFPN and ISBA, combining efficient temporal modeling with iterative transcript alignment and soft boundary supervision.
- Dense frame-level annotations are tedious to obtain, difficult to temporally align consistently, and therefore limit scalable action-segmentation systems.
- Action transcripts provide ordered action units without precise boundaries and are easier to obtain, so the framework assumes transcripts during training.
- Existing RNN-, ECTC-, and HMM-based approaches incur expensive computation, including video chunking and iterative training until convergence.
- TCFPN predicts frame-wise action labels, while ISBA iteratively aligns action sequences, updates the network, and avoids recurrency or Markovian processing.
- ISBA refines transcript boundaries from current network inference, applies weaker supervision at boundary frames, and uses video-level recognition loss to stop training before overfitting.
- The complete system achieves state-of-the-art performance on weakly supervised action segmentation and alignment with efficiency and scalability on Breakfast and Hollywood Extended.
2. Related Work
The related work organizes video action modeling into fully supervised and weakly supervised approaches, with existing methods modeling temporal dependencies using recurrent architectures and related sequence-learning techniques.
- Fully-Supervised Methods: Fully supervised methods commonly use frame-level features followed by temporal models over video sequences, including attention LSTMs and bidirectional recurrent architectures.
3. Temporal Convolutional Feature Pyramid Network (TCFPN)
TCFPN improves an encoder-decoder temporal convolutional network with lateral connections that combine semantic and dense features across scales. Its pyramid produces frame-wise predictions by averaging outputs from multiple decoder layers, while soft boundaries make coarse targets more robust.
- Architecture: TCFPN extends the ED-TCN encoder-decoder baseline with lateral connections between encoder and decoder layers.The lateral connection mechanism is adapted from object detection for action segmentation.
- Architecture: TCFPN uses K encoder and decoder feature layers, with K > 0 dataset-dependent and K = 3 in all experiments.The architecture retains an encoder-decoder structure throughout the pyramid.
- Encoder: The encoder applies temporal convolution, batch normalization, ReLU, and temporal max pooling to progressively transform frame-level video features.The first encoder layer contains the extracted video feature from each frame.
- Decoder: The decoder uses same-width 1 × 1 convolutions for element-wise lateral fusion, injecting high-level semantic information into low-level dense feature maps.The 1 × 1 convolutions also reduce the dimension of high-level features.
- Prediction: Each decoder layer applies temporal convolution after upsampling, and frame-wise softmax outputs are averaged across all K layers.This combines coarse, semantically strong features with fine, semantically weak features while adding little computation expense.
- Soft boundary mechanism: Linear interpolation of action probabilities makes temporal boundaries coarse, producing generated targets that are more robust and reliable.The mechanism is designed for boundaries between two different actions.
4. Iterative Soft Boundary Assignment (ISBA)
ISBA iteratively converts weak action transcripts into increasingly refined training targets by combining soft boundary assignment, model inference, and transcript updates. It uses video-level recognition loss to select a stopping point and supports alignment on training videos and segmentation on unseen videos.
- Overall procedure: ISBA generates training targets from action transcripts, trains a temporal segmentation model, infers on training videos, and refines the transcripts iteratively.The procedure repeats until its stopping condition, after which the model performs alignment on training videos and segmentation on testing videos.
- Target Generation with Soft Boundary Assignment: Soft boundary assignment replaces hard uniform labels near action transitions with linearly interpolated probabilities for adjacent actions.This accommodates variable action durations while preserving the transcript’s ordering information.
- Transcript Refinement with Iterative Training and Inference: Transcript refinement adjusts action positions and lengths while preserving the weakly supervised action ordering.The resulting transcript becomes progressively less coarse and provides stronger supervision for subsequent target generation.
- Transcript Refinement with Iterative Training and Inference: The model’s inferred probabilities determine transcript updates at estimated boundaries between consecutive, different actions.At each boundary, the method inserts the higher-probability action when the probability difference exceeds threshold ρ.
- Transcript Refinement with Iterative Training and Inference: A Bernoulli randomness parameter θ can make boundary-label insertion more robust before the next training iteration begins.For the two candidate labels, the insertion choice is sampled according to the specified Bernoulli distribution.
- Stop Criteria: Video-level recognition loss monitors iterative training because continued refinement can overfit network behavior rather than the unknown ground truth.Training stops when recognition loss fails to decrease for three iterations, and the minimum-loss iteration is selected as the final result.
5. Experiments
The experiments evaluate TCFPN and ISBA across three action-segmentation and alignment tasks on Breakfast and Hollywood Extended using four metrics. Results generally favor the proposed methods, while ablations and iterative-training analysis examine efficiency, soft boundaries, and stopping.
- Experimental Setup: Experiments cover fully-supervised segmentation, weakly-supervised segmentation, and weakly-supervised alignment on Breakfast and Hollywood Extended.Four metrics are used across the evaluations, with additional ablation analysis.
- Fully-Supervised Action Segmentation: TCFPN outperforms ED-TCN substantially and is competitive with state-of-the-art methods in fully-supervised action segmentation.Its non-recurrent design supports efficient, highly parallelized training.
- Weakly-Supervised Action Segmentation: TCFPN+ISBA outperforms competing methods on most weakly-supervised segmentation metrics across both datasets, with Hollywood Extended frame-wise accuracy as the stated exception.On Hollywood Extended, background frames explain the exception, while HTK’s higher frame-wise accuracy is accompanied by lower IoU.
- Weakly-Supervised Action Segmentation: On Breakfast, TCFPN+ISBA is 5.1% better than GRU and HMM with reestimation in weakly-supervised segmentation, despite TCFPN being 5.8% worse than GRU under full supervision.The authors attribute the stronger weakly-supervised result mainly to ISBA.
- Weakly-Supervised Action Alignment: TCFPN+ISBA achieves superior weakly-supervised alignment results on Breakfast, while Hollywood Extended shows better IoU but lower IoD under the transcript assumption.The authors relate this trade-off to the network being less likely to label most frames as background.
- Analysis and Ablation Study: Soft boundary assignment improves most metrics and makes ISBA converge faster, while the video-level recognition loss provides a useful stopping signal.Using an alternative stopping criterion required 21 iterations and achieved 38.7 Acc., versus 10 iterations and 39.8 Acc. with the proposed criterion.
6. Conclusion
The paper introduces ISBA for weakly-supervised action segmentation and alignment and TCFPN for supervised action segmentation. TCFPN+ISBA outperforms state-of-the-art methods on both weakly-supervised tasks, while the training strategy is designed for broader integration.
- Contribution: ISBA combines iterative transcript refinement, soft boundary assignment, and a video-level loss used as the stopping criterion.It is proposed for weakly-supervised action segmentation and alignment.
- Contribution: TCFPN is a temporal convolutional network for supervised action segmentation that trains quickly and achieves competitive state-of-the-art performance.The conclusion emphasizes its efficiency without recurrent connections.
- Results: TCFPN+ISBA outperforms state-of-the-art methods on weakly-supervised action segmentation and alignment.The stated outcome covers both weakly-supervised tasks.
- Implication: ISBA is presented as a general training strategy that can integrate with other methods or be used in other video action-understanding tasks.The proposed generality is stated as a direction for facilitating future research.