Source-linked AI summary
Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling
Alexander Richard, Hilde Kuehne, Juergen Gall
TL;DR
The paper tackles temporal action localization and classifier learning from videos annotated only with ordered action lists, avoiding hand-labeled frame boundaries. It combines recurrent subaction recognition with coarse temporal inference and iteratively reestimates subaction counts, outperforming state-of-the-art methods on two weak-learning tasks across Breakfast and Hollywood extended.
Problem
Temporal localization and classification in long, untrimmed videos usually require costly frame-level start and end annotations, motivating learning from ordered action lists instead.
Method
The method combines an RNN-based discriminative representation of subactions with a coarse probabilistic model, iteratively realigning frames and reestimating subaction counts.
Results
The approach outperforms current state-of-the-art methods on temporal action segmentation and action alignment on the Breakfast and Hollywood extended datasets.
Takeaways & Limitations
Automatically adapting the number of subactions to action-class characteristics significantly improves the system's weakly supervised action-learning performance.
Takeaways & Limitations
Training long videos is computationally demanding because backpropagation through time processes and stores outputs for every frame, while recurrent models capture limited temporal context.
Abstract
from arXiv · showhide
We present an approach for weakly supervised learning of human actions. Given a set of videos and an ordered list of the occurring actions, the goal is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. To address this task, we propose a combination of a discriminative representation of subactions, modeled by a recurrent neural network, and a coarse probabilistic model to allow for a temporal alignment and inference over long sequences. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes. To this end, we adapt the number of subaction classes by iterating realignment and reestimation during training. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.
1. Introduction
The paper addresses weakly supervised action learning from ordered action lists rather than frame boundaries, using a fine-to-coarse model to align frames and learn action classifiers. Iterative subaction modeling and reestimation support temporal inference and improve performance across two weak-learning tasks.
- Motivation: Weak supervision replaces exact action boundaries with an ordered list of actions occurring in each video.This annotation is easier to obtain manually or derive from scripts and subtitles.
- Training: Training begins with uniform segmentation and iteratively retrains the RNN, infers new action boundaries, and updates the alignment.The initial segmentation uniformly distributes the ordered actions across each video.
- Method: The proposed system combines an RNN-based discriminative subaction model with a coarse probabilistic model for long-sequence temporal alignment.The RNN captures local temporal information, while the coarse model supports alignment and inference over long sequences.
- Method: Subactions distribute heterogeneous information across subclasses and represent characteristics such as the overall action length.The paper reports that automatically learning the number of subactions for each action class notably improves performance.
- Evaluation: The approach outperforms current state-of-the-art methods on temporal action segmentation and action alignment on Breakfast and Hollywood extended.Temporal action segmentation combines segmentation and classification when the test video has no further annotation.
2. Related Work
Prior work spans fully supervised temporal modeling and several weak-learning formulations for human actions. These approaches use sources such as scripts, ordered labels, web data, or sign-language corpora, with differing alignment and supervision assumptions.
- Fully supervised learning: Fully supervised approaches require annotated temporal structure and include neural, hidden Markov, and semi-Markov models.The cited methods combine deep learning with temporal modeling in different ways.
- Weak action learning: Early weak-learning methods mined training samples from movie scripts without dedicated temporal alignment within extracted clips.Later work addressed action-class learning and temporal alignment from weak annotations.
- Weak action learning: Ordered action labels have been formulated as temporal assignment problems for aligning actions in videos.This line of work includes the Hollywood extended dataset.
- Other weak-learning settings: Other weak-learning scenarios use sign-language hand shapes, web images, retrieved videos, or noisy labels to learn or localize actions.These settings differ from weak supervision based on temporally ordered action labels.
3. Technical Details
The framework combines recurrent subaction recognition with coarse probabilistic temporal modeling to align weakly labeled videos and infer action boundaries. Training alternates between model estimation and realignment while adapting subaction counts to action lengths.
- Weakly Supervised Learning: Weak supervision provides ordered action labels rather than framewise boundaries, requiring the model to infer temporal segments.The segmentation initially maps frames to transcript actions, with framewise likelihoods defined from action-conditioned probabilities.
- Subaction Representation: Long, heterogeneous actions are represented as sequential combinations of latent subactions whose counts are refined during optimization.Subactions distribute heterogeneous information across smaller units and are initially estimated heuristically.
- Fine-to-Coarse Model: A GRU-based RNN predicts framewise subaction probabilities using local temporal context, while a coarse HMM enforces valid subaction ordering.The HMM combines fine-grained probabilities with transition frequencies learned from training alignments.
- Efficiency: Long videos create computation and memory problems for full-sequence BPTT, so the method processes short frame-centered chunks and limits temporal scope.The implementation uses chunks over x[t −20, t], increasing processed data while enabling parallel minibatch training.
- Inference: Inference searches for frame-to-subaction alignments whose extracted action sequences are generated by a context-free grammar.The extractor maps alignments to action sequences, and the resulting optimization can be solved efficiently with Viterbi decoding.
- Training: Training alternates RNN and HMM estimation with alignment updates, beginning from uniform segmentation and reestimating action lengths and subaction counts until convergence.The number of subactions for action a is reestimated from its updated average length using len(a)/m.
4. Experiments
Experiments evaluate the iterative fine-to-coarse model on Breakfast and Hollywood extended datasets, focusing on temporal segmentation, model components, convergence, and action alignment. Subaction modeling and reestimation improve performance, while GRUs provide stronger low-level recognition than frame-level baselines.
- Experimental setup: The model is evaluated on Breakfast and Hollywood extended datasets using temporal action segmentation and action alignment tasks.Breakfast contains 1,712 clips and 66.7 hours; Hollywood extended contains 937 clips with 16 action classes.
- Training and convergence: Training initializes uniform action and subaction alignments, alternates RNN training with HMM-based realignment, and reestimates subaction counts until convergence.The stop criterion uses the frame change rate and threshold ϑ = 0.02; results use the last iteration before crossing the threshold.
- Coarse-model analysis: Subaction modeling improves recognition over a single action-class model, while subaction reestimation performs 5% better than fixed subaction counts.Using different subaction numbers with the same system increases performance by almost 20%.
- Fine-grained analysis: GRUs outperform GMMs and MLPs for low-level recognition, rising from 25.5% initially to 29.3% after the fourth iteration.The MLP remains below the GRU and begins decreasing after its second iteration, while the GRU oscillates around 29%.
- Comparison to state of the art: Subaction-based GRU systems outperform current approaches on both evaluated datasets, with reestimation providing a larger boost on Breakfast than Hollywood extended.The paper relates this difference to greater variation in action-class lengths in Breakfast.
- Comparison to state of the art: With a given action order, the reestimated GRU shows a clear improvement over current systems for action alignment.Alignment is evaluated using the Jaccard index computed as intersection over detection (IoD).
5. Conclusion
The proposed weakly supervised action-learning system combines recurrent subaction representations with coarse probabilistic inference and improves performance by adapting subaction counts through iterative realignment and reestimation.
- The system combines an RNN-based discriminative subaction representation with a coarse probabilistic model for temporal alignment and long-sequence inference.
- Adapting the number of subaction classes through iterative realignment and reestimation significantly improves performance across action classes.
- The resulting models outperform state-of-the-art methods on weak learning tasks including temporal action segmentation and action alignment.