Source-linked AI summary
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, Juan Carlos Niebles
TL;DR
Weakly supervised action alignment and segmentation lack frame-wise targets and can suffer degenerate alignments under existing surrogate objectives. D3TW introduces discriminative modeling with a continuous relaxation of dynamic programming, enabling differentiable end-to-end training. It achieves state-of-the-art results for both tasks on two challenging datasets.
Problem
Weak ordering supervision provides no frame-wise action targets, while existing surrogate objectives can produce degenerate single-frame alignments.
Method
D3TW combines discriminative positive-versus-negative transcript modeling with a continuous relaxation of the dynamic-programming minimum for differentiable end-to-end training.
Results
D3TW achieves state-of-the-art results for weakly supervised action segmentation and alignment on two challenging real-world datasets.
Takeaways & Limitations
Discriminative modeling and differentiable relaxation together bypass degenerate sequences while directly optimizing alignment outputs.
Takeaways & Limitations
The alignment space is exponentially large, and the method assumes action class priors that are uniformly initialized.
Abstract
from arXiv · showhide
We address weakly supervised action alignment and segmentation in videos, where only the order of occurring actions is available during training. We propose Discriminative Differentiable Dynamic Time Warping (D3TW), the first discriminative model using weak ordering supervision. The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable. We address this challenge with a continuous relaxation of the min-operator in dynamic programming and extend the alignment loss to be differentiable. The proposed D3TW innovatively solves sequence alignment with discriminative modeling and end-to-end training, which substantially improves the performance in weakly supervised action alignment and segmentation tasks. We show that our model is able to bypass the degenerated sequence problem usually encountered in previous work and outperform the current state-of-the-art across three evaluation metrics in two challenging datasets.
1. Introduction
The paper targets weakly supervised video action alignment and segmentation using only action order, proposing a discriminative, differentiable framework to avoid degenerate alignments and improve both tasks.
- Motivation: Weak supervision uses an ordered list of actions rather than frame-level start and end annotations.This reduces annotation effort but leaves frame-wise targets unavailable during training.
- Challenge: Surrogate weak-label objectives can produce degenerate alignments that assign occurring actions to single frames.Such solutions conflict with the fact that actions typically span many frames.
- Approach: D3TW introduces discriminative modeling that makes positive-transcript alignments preferable to randomly sampled negative transcripts.The formulation is designed so degenerate alignment is no longer an obvious trivial solution.
- Approach: D3TW relaxes the dynamic-programming minimum operator continuously, making the discriminative alignment loss differentiable and enabling end-to-end optimization.This addresses the non-differentiability of structural prediction losses based on dynamic programming.
- Evaluation: The method is evaluated on weakly supervised action segmentation and alignment using the Breakfast Action and Hollywood Extended datasets.The paper reports significant improvement on both tasks and state-of-the-art performance.
2. Related Works
Related work covers dense action understanding, ordering-based weak supervision, language-derived video supervision, and continuous relaxation of discrete operations.
- Action Recognition and Segmentation: Action segmentation extends video understanding from trimmed clips to densely labeling every frame in longer, untrimmed videos.The paper shares this dense-labeling goal while removing dense training supervision.
- Weakly Supervised Learning in Vision: Ordering constraints are established weak supervision for videos, while related image work studies weak supervision for classification, segmentation, detection, and grounding.The paper positions its approach within broader weakly supervised vision research.
- Weakly Supervised Learning in Vision: NN-Viterbi combines a neural network with a non-differentiable Viterbi process, whereas D3TW is end-to-end differentiable and discriminatively optimizes alignment.This distinguishes D3TW from iterative approaches using hard structural inference.
- Using Language as Supervision for Videos: Language-based video supervision commonly uses movie scripts or instructional-video transcripts, while this paper assumes discrete action labels are already extracted.Its focus is leveraging action ordering rather than extracting labels from language.
- Continuous Relaxation: D3TW builds on continuous relaxation of discrete operations and applies that principle to discriminative dynamic-programming-based alignment.The paper extends relaxation methods to make weakly supervised alignment discriminative.
3. Method
D3TW learns weakly supervised video alignment and segmentation from action-order transcripts by combining discriminative modeling with differentiable dynamic programming. It uses relaxed DTW costs for end-to-end optimization and reduces segmentation to alignment over candidate transcripts.
- Weakly Supervised Action Learning: Training uses only an ordered action transcript and video, without frame-wise action labels.The model learns from inputs (X, ℓ+) while the frame-level ground-truth labels remain unavailable.
- Weakly Supervised Action Learning: At test time, alignment uses the given transcript, whereas segmentation searches candidate transcripts and selects the best alignment.This formulation converts action segmentation into an alignment problem over a set of possible transcripts.
- Video Alignment as Dynamic Time Warping: D3TW formulates transcript-video alignment as DTW, selecting a constrained path that minimizes the inner product with the transcript-video distance matrix.Dynamic programming efficiently finds the optimal path while preserving transcript action order.
- Video Alignment as Dynamic Time Warping: The distance function can use negative log-likelihood, d(ℓ_i, x_j) = −log p(x_j|ℓ_i), but latent alignments make prior optimization prone to degenerate solutions.Earlier likelihood-based approaches infer latent frame alignments with dynamic programming or hard EM, which can produce trivial alignments.
- Discriminative Modeling with Weak Supervision: D3TW imposes a discriminative constraint in which the positive transcript receives a better alignment cost than negative transcripts.This addresses the degenerated alignment problem by making the correct transcript preferable to randomly sampled alternatives.
- Differentiable Loss with Continuous Relaxation: Continuous relaxation of the DTW minimum makes the alignment cost differentiable, enabling gradients to optimize the distance function and the full model end to end.For γ > 0, the relaxed objective supports backpropagation through dynamic programming and combines alignment with transcript discrimination.
4. Experiments
The experiments evaluate D3TW for weakly supervised action segmentation and alignment, including semi-supervised segmentation, on Breakfast Action and Hollywood Extended. Across these settings, the results attribute improvements to combining differentiable relaxation with discriminative modeling.
- Experimental setup: D3TW is evaluated on Breakfast Action and Hollywood Extended for weakly supervised action segmentation and alignment, with additional analysis in frame-level semi-supervised segmentation.Breakfast Action contains 1,712 videos, 52 participants, 10 dishes, and 48 possible actions; Hollywood Extended contains 937 videos with 2 to 11 actions each.
- Weakly supervised action segmentation: D3TW significantly outperforms all baselines and achieves state-of-the-art results on every weakly supervised action segmentation metric.The ablations indicate that differentiable relaxation and discriminative modeling both contribute to the full model’s performance.
- Weakly supervised action segmentation: D3TW correctly captures all occurring actions in the qualitative comparison, while also producing more accurate action boundaries.The comparison contrasts D3TW with NN-Viterbi and the nondiscriminative ablation.
- Weakly supervised action segmentation: Discriminative modeling improves frame accuracy on almost all Breakfast activities, with gains exceeding 20% for distinct-step activities such as Sandwich and Cereals.The improvement is not significant for Pancake and Scrambled Egg, whose cooking steps are visually similar.
- Semi-supervised action segmentation: D3TW improves both frame and unit accuracy in the frame-level semi-supervised setting by incorporating sparse frame labels as path constraints in dynamic programming.The comparison includes ECTC and the Uniform baseline.
- Weakly supervised action alignment: In weakly supervised action alignment, D3TW significantly outperforms all baselines and achieves state-of-the-art results despite the smaller method gap caused by the given test-time transcript.Alignment performance is higher than segmentation because the ground-truth transcript is available at test time.
5. Conclusion
The paper concludes that D3TW combines discriminative modeling, differentiable dynamic alignment, and end-to-end training for weakly supervised action alignment and segmentation. It reports state-of-the-art results on both tasks and suggests that the framework can extend to other structured-output problems.
- Contributions: D3TW is a discriminative framework for weakly supervised action alignment and segmentation that addresses degenerate sequences through positive-versus-negative transcript modeling.The framework is designed for structural sequence prediction with weak supervision.
- Contributions: A continuous relaxation of the dynamic-programming minimum makes the discriminative loss differentiable and enables end-to-end optimization.The conclusion identifies differentiable relaxation and discriminative modeling as crucial components.
- Results: D3TW achieves state-of-the-art results for both segmentation and alignment on two challenging real-world datasets.The conclusion attributes the result to the combination of discriminative modeling and differentiable relaxation.
- Implications: The framework is presented as general and extendable to tasks requiring prior structure in the output and end-to-end differentiability.This is stated as a supported extension beyond the evaluated video tasks.