A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation

1 INTRODUCTION

• Manual frame-boundary annotation is costly and time-consuming, limiting the practicality of collecting large-scale training data for long-term temporal models.Ordered action labels can instead be easier to collect or derived from scripts or subtitles.Source: §1 · p. 1
• The model pairs an RNN that recognizes fine-grained subactions with coarse probabilistic inference for temporal alignment of long sequences.This coarse-to-fine structure supports recognition of fine-grained movements while capturing mid- and long-term temporal relations.Source: §1 · p. 1
• Dividing actions into subactions distributes heterogeneous action information across subclasses, and automatically adapting each class’s subaction count improves performance.The subactions are modeled by the RNN and combined during inference.Source: §1 · p. 1
• Training begins with a uniform action and subaction alignment, then alternates RNN training, boundary inference, subaction-count updates, and realignment until convergence.The procedure uses the weakly supervised transcripts to initialize the alignment.Source: §1 · p. 1
• A length prior regularizes temporal inference by enforcing meaningful state lengths, addressing degenerate states that can arise when observation priors dominate.The authors report that this regularization improves recognition accuracy.Source: §1 · pp. 1–2
• 0.25% of Breakfast frames annotated (about one per action instance) improves accuracy; sparse supervision reaches full-supervision-comparable results, and all evaluated cases outperform prior state of the art.The authors also report that the length model improves weakly, semi-, and fully supervised recognition.Source: §1 · p. 2

2 RELATED WORK

• Classical action-recognition benchmarks are described as nearing a performance ceiling, while recent work explores temporal learning without full supervision.The section situates this work after a progression from hand-crafted features toward learned features and temporal connections.Source: §2 · p. 2

2.1 Weakly supervised learning from structured sequences

• Weakly supervised sequence methods use ordered labels for temporal assignment, including clustering, CTC-inspired alignment, HMMs, and temporal convolutional feature pyramids.Prior work includes methods that infer action boundaries from ordered action lists and iteratively refine alignments.Source: §2.1 · pp. 2–3

2.2 Weakly supervised fine-tuning

• Weakly supervised fine-tuning detects action instances in untrimmed videos using pretrained networks, where pretraining classes may overlap with target classes.Training data may provide video-level class presence without specifying when or how often a class occurs.Source: §2.2 · p. 3

2.3 Weakly supervised approaches in other domains

• Related weak-supervision work in other domains combines visual models with HMMs, aligns recipe steps to speech transcripts, or localizes actions from weak labels.Other approaches use web imagery, noisy labels, video-level tags, or RGB-D data.Source: §2.3 · p. 3

2.4 Length modeling for temporal sequences

• Prior duration models include class-relative segment-length distributions that penalize overly short or long sequences, but one such first-order formulation has quadratic inference runtime.Explicit-state-duration HMMs and other duration models are also discussed in the context of temporal sequences.Source: §2.4 · p. 3

3.1 Learning from transcripts only

• Transcript-only learning assumes the training videos’ ordered action lists are known and each action appears with at least two different predecessors and successors.The authors explain that repeated fixed action pairings leave the boundary between them indeterminate.Source: §3.1 · p. 3

3.2 Learning from transcripts including sparse frame-level annotation

• Sparse frame-level supervision adds action labels for a fraction of frames; these indicate action positions and constrain alignment but do not mark boundaries.The task remains to infer action boundaries while matching the annotated frames.Source: §3.2 · pp. 3–4

4 SYSTEM OVERVIEW

• The hierarchical model represents activities as ordered actions, each decomposed into latent subactions that capture finer-grained movement phases.Subaction counts are initialized heuristically and refined during optimization; the feed-forward state graph permits only self-transitions or transitions to the next state.Source: §4 · p. 4, §4.1 · p. 4

4.1 Hierarchical action model

• Training examples pair framewise video features with ordered action transcripts, and each action is modeled as an HMM over its ordered subactions.Subaction boundaries are latent; their number is initialized by a heuristic and refined during optimization.Source: §4.1 · p. 4

4.2 Fine-grained Subaction Model

• A single-hidden-layer GRU RNN predicts framewise probabilities over subactions while using recurrent connections to incorporate local temporal context.For long videos, training processes chunks x[t−20, t] around each frame rather than forwarding whole sequences at once.Source: §4.2 · pp. 4–5

4.3 Inference

• Inference uses Viterbi decoding over subaction alignments consistent with a context-free grammar built from training transcripts.For sparse frame labels, dynamic warping assigns labels to compatible transcript segments with boundary shifts kept as small as possible.Source: §4.3 · pp. 5–6

4.4 Length prior

• The length prior regularizes state duration by rewarding a state initially and penalizing it when it persists too long.It addresses the tendency to skip states by assigning them only one or two frames.Source: §4.4 · p. 6
• 74.6% of processed states were skip states without a length prior, versus 61.9% with the prior in the best-performing configuration.The prior rewards remaining in a new state for some duration and penalizes staying too long.Source: §4.4 · p. 6

4.5 Decoding with Length prior

• Unlike context-dependent duration models, the proposed monotonically decreasing prior uses current state duration and retains linear O(T) decoding.Monotonicity avoids penalizing a newly entered state before its eventual duration is known.Source: §4.5 · p. 7

4.6 Training

• Training alternates between RNN and HMM learning and HMM-based frame-to-subaction alignment, starting from a linear segmentation and repeating until convergence.After inference produces new boundaries, the HMM states are redistributed according to the updated segmentation.Source: §4.6 · p. 7
• Training iteratively refits the RNN and HMM, realigns frames to subactions, and reestimates subaction counts from updated action lengths.Initialization uses equal action segments and targets an average of 10 frames per subaction; realignment uses Viterbi inference with length regularization, and a new randomly initialized RNN is trained from scratch each iteration.Source: §4.6 · pp. 7–8, Figure 5 · p. 8

4.7 Stop criterion

• Training stops when fewer than 5% of frames change labels between iterations or the process reaches 15 iterations.Source: §4.7 · p. 8

5.1 Datasets

• Breakfast spans 1,712 clips and 66.7 hours across 48 action classes; Hollywood extended has 937 clips and 16 classes, with different temporal heterogeneity.Source: §5.1 · p. 8

5.2 Features

• Both datasets use Fisher vectors computed over 20-frame sliding windows, with PCA reducing the final representation from 8,192 to 64 dimensions.Source: §5.2 · p. 8

5.3 Alignment vs. Segmentation

• Alignment provides the transcript and evaluates boundary recovery with MoF or Jaccard/IoD; segmentation withholds the transcript and evaluates inferred actions and boundaries by MoF.Source: §5.3 · pp. 8–9
• Across five iterations, the GRU-based model outperforms MLP and GMM; the MLP quickly overfits, while the GRU oscillates at a higher level.Source: Table 1 · p. 9

6.1 Evaluation of GRU-based model

• On Breakfast, GRUs reach 29.3% after the fourth iteration, compared with a maximum of 27.0% for GMMs, while the MLP remains below the GRU.The comparison uses a simplified system without subaction reestimation or a length prior; the MLP peaks after iteration two and then declines.Source: §6.1 · p. 9

6.2 Analysis of the subaction modeling

• Subaction reestimation performs 5% better than the model without reestimation, while using ground-truth action lengths raises performance by almost 20% over the same system with different subaction counts.The system without subactions performs significantly below the other configurations; reestimated state counts increase early and converge after five to ten iterations.Source: §6.2 · pp. 9–10, Table 2 · p. 9, Figure 7 · p. 10

6.3 Analysis of length prior

• On Breakfast, the linear decay length model reaches 37.0%; all four evaluated length models outperform the system without a length prior.The half Gaussian gives the most consistent improvement across both datasets; the impact is smaller on Hollywood Extended, where actions have lower temporal variance.Source: §6.3 · p. 10, Table 3 · p. 10, Figure 8 · p. 11

6.4 Semi supervised learning including sparse frame-level annotation

• On Breakfast, 0.25% annotated frames improves accuracy by almost 20% to 56.0%, while 1% brings results close to the 61.0% fully supervised setting.With frame-level annotation, convergence takes two to four iterations rather than the usual 12–15; on Hollywood Extended, gains are smaller and mainly appear with at least 1% labeled frames.Source: §6.4 · pp. 10–11, Table 4 · p. 12

6 EVALUATION

• The evaluation covers temporal action segmentation and action alignment on Breakfast and Hollywood Extended, with segmentation measured by mean accuracy over frames on Breakfast and Jaccard index on Hollywood Extended.The component evaluations report mean accuracy over frames on the Breakfast test set.Source: §6 · p. 9, §6.5 · p. 11

6.5 Comparison to State-of-the-Art

• The proposed system outperforms most current approaches on the evaluated datasets; only TCFPN achieves better temporal segmentation results, and the approach also outperforms current state-of-the-art methods for action alignment.Fully supervised segmentation on Breakfast also outperforms previous approaches; the authors attribute much of the improvement to GRU models and report an additional benefit from the length model.Source: §6.5 · p. 12, Table 5 · p. 12, Table 6 · p. 12

• The approach combines RNN-modeled subactions with a coarse probabilistic model for temporal alignment and inference over long sequences.The method is presented for weakly, semi-, and fully supervised learning of human actions.Source: §7 · pp. 12–13
• Iterative realignment and subaction-count reestimation are combined with a length model to improve performance.The system achieves competitive results on weak learning tasks, including temporal action segmentation and action alignment, on Breakfast and Hollywood Extended.Source: §7 · pp. 12–13

• The four length models are normalized so their maximum value is one, and each uses ε = 0.001.The text notes that the normalized functions are not strict probability distributions, but the normalization adds only a constant in Viterbi decoding and does not affect the outcome.Source: Appendix A · p. 13

• The box length prior leaves inference unaffected through twice the state's mean length, then multiplies the probability by ε to marginalize the state.The displayed definition assigns 1 when lt ≤ 2 · len(st) and ε otherwise.Source: Box Function · p. 13

Linear Decay

• The prior stays fixed until the mean length of the state, then decreases linearly, penalizing longer states more than shorter ones.It reaches ε after twice the mean length, so the state is no longer used.Source: Linear Decay · p. 13

Half Poisson

• The prior is fixed through the mean state length, then follows the right half of a Poisson distribution.This combination is chosen to model discrete state lengths while ensuring a monotonically decreasing function.Source: Half Poisson · p. 13

Half Gaussian

• Setting μ = 0 and σ = len(st) implicitly ensures a monotonically decreasing function.The Half Gaussian model is closely related to the Half Poisson model.Source: Half Gaussian · p. 13