Source-linked AI summary
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
De-An Huang, Li Fei-Fei, Juan Carlos Niebles
TL;DR
The paper addresses action labeling when training provides only the order of actions, leaving frame-level alignments unknown. It introduces ECTC, which uses dynamic programming and visual similarity to evaluate and constrain alignments, and extends it with sparse frame annotations. With less than 1% of labeled frames per video, the method outperforms existing semi-supervised approaches and achieves comparable performance to fully supervised approaches.
Problem
Weakly supervised action labeling lacks the per-frame alignments needed to localize ordered actions in video during training.
Method
ECTC evaluates possible frame-to-label alignments with dynamic programming while enforcing consistency with frame-to-frame visual similarities and sparse frame-level constraints.
Results
Less than 1% of labeled frames per video enables the method to outperform existing semi-supervised approaches and achieve comparable performance to fully supervised approaches.
Takeaways & Limitations
Visual-similarity-weighted alignment and sparse frame supervision support temporal action labeling without full temporal annotation.
Abstract
from arXiv · showhide
We propose a weakly-supervised framework for action labeling in video, where only the order of occurring actions is required during training time. The key challenge is that the per-frame alignments between the input (video) and label (action) sequences are unknown during training. We address this by introducing the Extended Connectionist Temporal Classification (ECTC) framework to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities. This protects the model from distractions of visually inconsistent or degenerated alignments without the need of temporal supervision. We further extend our framework to the semi-supervised case when a few frames are sparsely annotated in a video. With less than 1% of labeled frames per video, our method is able to outperform existing semi-supervised approaches and achieve comparable performance to that of fully supervised approaches.
1 Introduction
The paper targets action labeling from training data that specifies only action order, avoiding costly frame-level temporal annotation. It introduces CTC and ECTC to evaluate possible alignments efficiently while favoring visually consistent ones, and extends the framework to sparse frame-level supervision.
- Problem: Weakly supervised action labeling trains models from action order and applies them to annotate each frame in unseen videos.The task requires reasoning about both action occurrence and exact temporal location without fully annotated training videos.
- CTC: CTC evaluates all possible frame-to-label alignments with dynamic programming instead of naively searching the alignment space.This addresses the large number of correspondences between video frames and ordered action labels.
- ECTC: ECTC enforces alignment consistency with frame-to-frame visual similarities to discourage visually inconsistent and degenerated paths.The method extends the forward-backward algorithm to incorporate visual similarity while efficiently evaluating alignments.
- Semi-supervision: Frame-level semi-supervision incorporates labels from a few annotated frames as alignment constraints that prune inconsistent paths.This supervision is less costly than precisely annotating all action boundaries and reduces the alignment space.
- Results: Less than 1% supervision supports state-of-the-art performance on long and complex activity videos.The evaluation uses the Breakfast Actions Dataset and a subset of Hollywood2.
2 Related Work
The related work spans video classification, video-to-action alignment, text-based video parsing, temporal modeling, and unsupervised video parsing. The paper distinguishes its setting by learning a temporal action-labeling model for unseen test videos without associated action information or text.
- Action understanding: Video classification assigns one discrete action class to an entire video, whereas this paper predicts the occurring action at every frame.The paper positions frame-level labeling as a use of video temporal structure rather than whole-video categorization.
- Video-to-action alignment: OCDC aligns frames to an ordered action list but can assume action ordering at both training and testing.In the paper’s unseen-video setting, OCDC becomes an implicitly learned frame-by-frame classifier and does not fully use temporal information at test time.
- Text-based supervision: Instructional-video and movie-script approaches often use text for alignment at testing time, while this paper targets unseen videos without associated text.Text processing may provide the paper’s supervision, but it is not the focus of the proposed method.
- Unsupervised parsing: The paper’s use of visual similarities relates to unsupervised video parsing, where frames are grouped into segments.Its stated goal remains training an action-labeling model that predicts frame-level actions in unseen videos.
- Temporal modeling: RNN-based prior work captures temporal dependencies, but accurate action localization is often ignored or requires pre-segmented training data.ECTC is presented as enabling recurrent temporal modeling without that temporal localization requirement.
3 Ordering Constrained Video Action Labeling
The framework adapts CTC to weakly supervised video action labeling by summing over ordered frame-to-action alignments and using visual similarity to favor consistent paths. ECTC extends this dynamic-programming formulation with binary frame-correlation terms and supports sparse frame-level constraints.
- 3 Ordering Constrained Video Action Labeling: The model uses an RNN to assign action labels to every frame in unseen test videos while training from an ordered action list.The exact per-frame labels are unavailable during training.
- 3 Ordering Constrained Video Action Labeling: CTC computes the probability of an ordered label sequence by summing probabilities over all compatible per-frame paths.The operator B removes consecutive repetitions from a path to recover the ordered label sequence.
- 3 Ordering Constrained Video Action Labeling: ECTC adds a binary term based on consecutive-frame visual similarity, rewarding paths that retain the same action across similar frames.The similarity is compared with a predefined minimum threshold θ; low-similarity frames are not penalized for retaining the same action.
- 3 Ordering Constrained Video Action Labeling: Original CTC is insufficient because long videos paired with short action sequences create many alignments, including degenerated paths.This setting may involve thousands of frames but only dozens of actions.
- 3 Ordering Constrained Video Action Labeling: ECTC's forward-backward algorithm incorporates the binary term into dynamic programming while efficiently evaluating all possible paths.The recursion introduces variables that summarize partial paths and model dependence between consecutive frames.
- 3 Ordering Constrained Video Action Labeling: Sparse frame-level labels can constrain alignments, substantially reducing the path space and pruning inconsistent paths.This extends ECTC beyond action ordering without requiring complete temporal boundary annotations.
4 Extension to Frame-level Semi-Supervised Learning
The frame-level semi-supervised extension incorporates sparse frame-action annotations as alignment constraints, pruning degenerate paths while preserving efficient inference.
- Sparse temporal supervision provides a few annotated frames with ground-truth actions rather than complete temporal boundaries.These annotations can come from movie scripts or manual labeling of selected frames.
- The method combines frame-level labels with action ordering to restrict the set of consistent alignment paths.Each constraint pairs a frame index with its corresponding action label.
- For example, two labeled frames in a six-frame video reduce the consistent paths to two and exclude an undesirable late-transition path.The example uses J = [(a, 2), (b, 4)] and ℓ = [a, b].
- An extra recursion check sets incompatible paths to zero when a constrained frame receives the wrong action label.This extends the dynamic-programming recursion with hard consistency checks.
- With less than 1% of frames labeled, the approach can perform comparably to a fully supervised model.
5 Experiments
Experiments evaluate the framework on Breakfast activity segmentation and Hollywood2 action detection, using temporal modeling and visual similarity to improve alignment and labeling. Results show stronger qualitative localization, improved performance over baselines, and benefits from sparse supervision, while difficult visual conditions remain challenging.
- Datasets and tasks: The experiments cover Breakfast activity segmentation and Hollywood2 action detection, including unseen test videos and alignment evaluation.Breakfast contains approximately 77 hours from 52 participants, 18 kitchens, and 10 cooking activities.
- Metrics: Frame accuracy measures per-frame labels, whereas unit accuracy evaluates the action sequence after dynamic-time-warping alignment.For weak supervision, frame accuracy more directly reflects temporal localization quality.
- Complex activity segmentation: ECTC produces better action localization and ordering than weakly supervised alternatives, while CTC can generate paths dominated by a single action.The qualitative comparison includes fully supervised BLSTM results as an upper-bound reference.
- Complex activity segmentation: Visual grouping and cosine similarity progressively improve ECTC accuracy by pruning inconsistent paths and rewarding visually consistent action assignments.The final model combines both similarity functions and outperforms fully supervised baselines on the reported Breakfast ablation.
- Error analysis: Hard cases include object-triggered false positives, similar motions, viewpoint and illumination changes, missing detections, and ambiguous action boundaries.
- Frame-level semi-supervision: A single frame annotation per segment substantially reduces possible alignments and provides a strong training cue; the semi-supervised method outperforms OCDC across labeled-frame fractions.Its frame accuracy declines more slowly than the Uniform baseline as supervision decreases.
- Training-set alignment: Training-set alignment results show good frame accuracy and Jaccard alignment quality, although OCDC performs closer to the proposed method in this setting.
- Action detection: On Hollywood2 detection and alignment, both proposed approaches outperform OCDC, while ECTC performs best for alignment because it evaluates alignments with visual similarity.The weakly supervised setting corresponds to zero annotated-video fraction.
6 Conclusions
ECTC learns temporal action models from weak supervision by combining dynamic programming over alignments with frame-to-frame visual similarity. It also supports sparse frame supervision and performs strongly on activity segmentation and action detection.
- ECTC learns temporal action models when only weak supervision is available.
- Visual similarity captures dependencies between consecutive frames while weighting possible alignments.
- Dynamic programming efficiently evaluates possible alignments, while frame-level semi-supervision reduces their search space.
- Less than 1% of supervision yields performance comparable to state-of-the-art fully supervised models and outperforms fully supervised baselines with weak supervision.