Source-linked AI summary
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Alexander Richard, Hilde Kuehne, Ahsan Iqbal, Juergen Gall
TL;DR
Video learning needs methods that avoid frame-level annotation for large, continuously growing video collections. The paper introduces direct Viterbi-based weak supervision with context and length modeling, achieving up to 10% higher accuracy and outperforming the state of the art.
Problem
Growing video collections and continuous-stream applications create demand for learning methods that avoid costly frame-level annotations, while prior approaches require heuristic pseudo-ground-truth and can oscillate between iterations.
Method
The paper integrates Viterbi decoding into the loss for direct incremental learning from videos and ordered action classes, adding explicit context and length models.
Results
The method improves accuracy by up to 10% over existing methods and outperforms the current state of the art by a large margin across action-segmentation evaluations.
Takeaways & Limitations
Explicit length and context modeling, together with direct optimization of video classes, is reported as crucial for strong performance and robust, stable convergence.
Takeaways & Limitations
The prior pseudo-ground-truth approach processes the entire dataset at each iteration, preventing incremental learning; the paper’s framework is limited to models whose best segmentation is Viterbi-inferable.
Abstract
from arXiv · showhide
Video learning is an important task in computer vision and has experienced increasing interest over the recent years. Since even a small amount of videos easily comprises several million frames, methods that do not rely on a frame-level annotation are of special importance. In this work, we propose a novel learning algorithm with a Viterbi-based loss that allows for online and incremental learning of weakly annotated video data. We moreover show that explicit context and length modeling leads to huge improvements in video segmentation and labeling tasks andinclude these models into our framework. On several action segmentation benchmarks, we obtain an improvement of up to 10% compared to current state-of-the-art methods.
1. Introduction
The paper addresses weakly supervised temporal action segmentation by integrating Viterbi decoding into neural-network training, avoiding frame-level labels and pseudo-ground-truth initialization. It combines direct action-class learning with explicit length modeling and reports improvements over existing methods.
- 1. Introduction: Weak supervision is important because untrimmed videos contain varied actions and lengths, while frame-level annotation is expensive.Transcripts provide ordered action classes but not their frame boundaries.
- 1. Introduction: The method is motivated by limitations of iterative Viterbi-based pseudo-labeling, whose accuracy can oscillate and whose training depends on heuristic ground truth.CTC does not provide the same explicit context and temporal-progression modeling.
- 1. Introduction: The proposed algorithm trains directly from input videos and ordered action classes by incorporating Viterbi decoding into the loss.It avoids framewise labeling for initialization and pseudo-ground-truth generation.
- 1. Introduction: Up to 10% higher accuracy is reported than existing methods through direct action-class learning with an explicit length model.The length model replaces widely used HMMs and avoids learning intermediate HMM states.
- 1. Introduction: Explicit length modeling is presented as a second contribution alongside the Viterbi-based direct-learning procedure.The approach learns action classes directly rather than intermediate HMM states.
2. Related Work
Prior weakly supervised video-learning methods adapt speech-recognition and sequence-learning ideas to infer temporal action structure without frame annotations. Their limitations include two-step optimization and the inability of CTC-based approaches to incorporate explicit context and length models.
- 2. Related Work: Most fully supervised CNN and LSTM approaches require framewise annotated training data, motivating weaker supervision for temporal segmentation.These methods target action segmentation but rely on dense labels.
- 2. Related Work: Earlier weakly supervised approaches also use movie scripts, background assumptions, or web images to obtain training information.These lines of work precede methods that infer exact temporal boundaries for multiple actions.
- 2. Related Work: Weakly supervised action segmentation has been formulated as speech recognition, mapping videos to audio signals and action classes to words.HMM-GMM systems and neural-network variants use ordered action transcripts as supervision.
- 2. Related Work: Existing HMM-based approaches use two-step optimization rather than direct sequence-wise training.They first infer a segmentation and then train the neural network from that inferred segmentation.
- 2. Related Work: CTC-based methods can support weak supervision but do not allow explicit context and length models in the same way as the proposed method.Some extensions address degenerate segmentations using visual similarities.
3. Temporal Action Segmentation
Weakly supervised temporal action segmentation seeks the most likely segmentation and class labeling of a video when frame-level boundaries are unknown but action order is given. Existing Viterbi-based formulations support this setting, while the paper focuses on improving their modeling and learning procedure.
- The task is to segment a T-frame video into an unknown number of segments and assign each segment an action label and length.
- Weak supervision provides only the ordered action classes in each training video, leaving segment lengths and frame-level annotations unknown.
- State-of-the-art probabilistic models formulate video labeling so its most likely segmentation can be computed efficiently with a Viterbi-like algorithm.
- The paper proposes explicit action-length modeling and a more principled weakly supervised learning approach that incorporates Viterbi decoding into training.
4. NeuralNetwork-Viterbi
The framework integrates Viterbi decoding into the loss, enabling direct online learning from transcripts without frame-level pseudo-ground truth. A buffer-based update reduces sequence bias during incremental optimization.
- NeuralNetwork-Viterbi: The prior two-stage procedure separates transcript decoding from model learning, making it sensitive to pseudo-ground-truth initialization and prone to oscillating accuracy.It also processes the entire dataset at each iteration, preventing incremental learning.
- NeuralNetwork-Viterbi: The method trains directly from videos and ordered transcripts by incorporating Viterbi decoding into the loss, rather than generating pseudo-ground truth first.The loss is computed for individual training sequences and does not require frame-wise initialization.
- NeuralNetwork-Viterbi: Viterbi segmentations are converted into framewise labels for cross-entropy optimization, and stochastic gradient updates operate on one randomly drawn sequence at a time.The framework is not restricted to cross-entropy; other losses can be used after segmentation.
- Enhancing the Robustness: Processing all frames from a sequence creates correlated updates biased toward that sequence’s frames and classes, especially after splitting long sequences into mini-batches.This bias conflicts with the independent-observation assumption commonly used in online learning.
- Enhancing the Robustness: A buffer stores recently processed sequences and inferred labels, while randomly sampled buffer frames are added to the loss to lessen sequence bias and improve optimization robustness.Because updates are gradual, many buffered frame-label pairs remain consistent with the current model.
5. The Model
The model combines neural visual predictions with explicit length and context models, and uses dynamic programming to find the highest-scoring segmentation. Its bounded implementation makes decoding linear in video length and grammar size.
- The Model: The model factorizes segmentation probability into visual, length, and context terms, represented respectively by a neural network, a class-dependent length model, and a grammar.The visual model uses a recurrent network with 256 gated recurrent units and softmax outputs.
- The Model: A class-dependent Poisson distribution models segment lengths, with λc updated as the mean segment length for class c after each iteration.For an unseen class, the initialization uses λc = N/T.
- The Model: A right-regular stochastic grammar represents context and generalizes both n-gram language models and finite allowed-sequence grammars.Training grammars come from transcripts, while segmentation evaluation estimates the grammar from training annotations.
- Viterbi Algorithm Revisited: The Viterbi procedure recursively scores segmentations using the current frame, segment length, class, and grammar context, while considering transitions and possible predecessor lengths.The auxiliary function tracks the best probability up to each frame under these conditions.
- Viterbi Algorithm Revisited: Without a length bound, decoding has quadratic complexity in video length; limiting the maximum segment length to L = 2,000 makes runtime linear in video length and grammar size.The bound is used in practice to control the cost of Viterbi decoding.
6. Experiments
Experiments evaluate NN-Viterbi across three action datasets, robustness settings, temporal models, direct learning, incremental input orders, and state-of-the-art comparisons. Buffered sampling, explicit length modeling, and direct optimization improve accuracy and convergence, while sequence ordering affects later activity learning.
- 6.1. Robustness: 27.2% frame accuracy is obtained without buffered sampling, while sampling buffered frames greatly increases accuracy by broadening the data and class distributions.
- 6.1. Robustness: NN-Viterbi converges smoothly after 2,000 to 3,000 iterations, and robustness enhancements provide a large early frame-accuracy margin over pure online learning.
- 6.2.1. Temporal Modeling: HMM vs. Length Model: +6% accuracy is gained by an explicit length model over an HMM, after HMM temporal modeling already improves accuracy by nearly +10%.The length model trains directly on action classes and can adapt dynamically during training.
- 6.2.2. Pseudo Ground-Truth vs. Direct Learning: 43.0% accuracy is obtained with direct NN-Viterbi learning, avoiding the pseudo-ground-truth errors of the two-step scheme.
- 6.3. Incremental Learning: Later coarse Breakfast activities lose about 50% accuracy when sequences are ordered by activity, whereas ordering by actor still gives very good performance.
- 6.4. Comparison to State of the Art: 55.5% unit accuracy is achieved with NN-Viterbi, compared with 43.4% for Lin et al.'s CTC-based approach on Breakfast.
7. Conclusion
The paper presents direct Viterbi-based learning for weakly labeled videos, supporting explicit context and length models. It reports improved state-of-the-art performance and robust, stable convergence.
- NN-Viterbi directly learns from weak labels, supports explicit context and length models, and can apply to models decoded by a Viterbi-like algorithm.
- The method outperforms related HMM-based pseudo-ground-truth methods and the current state of the art by a large margin while converging robustly and stably.