Source-linked AI summary
Weakly Supervised Action Labeling in Videos Under Ordering Constraints
Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, Josef Sivic
TL;DR
The paper asks how to localize actions and learn action classifiers when videos have ordered but not time-stamped action annotations. It jointly solves constrained temporal assignment and discriminative classifier learning, and evaluates the approach on a new action dataset with improved performance reported for temporal localization and classification.
Problem
Fully time-stamped action labels are costly to gather, whereas movie scripts provide weak annotations with poor temporal localization; the paper therefore studies how to use their action ordering.
Method
The method formulates action assignment to video intervals as discriminative clustering with ordering constraints, jointly learning action classifiers and temporal assignments, with a convex relaxation optimized by Frank-Wolfe.
Results
The model reports improved performance on temporal localization and action classification on a new action dataset.
Takeaways & Limitations
Temporal ordering in weak annotations can serve as supervision for learning individual action models while recovering their temporal locations.
Takeaways & Limitations
The optimization can produce trivial assignments dominated by the background label, motivating minimum-size constraints or a linear penalty.
Abstract
from arXiv · showhide
We are given a set of video clips, each one annotated with an {\em ordered} list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script. We seek to temporally localize the individual actions in each clip as well as to learn a discriminative classifier for each action. We formulate the problem as a weakly supervised temporal assignment with ordering constraints. Each video clip is divided into small time intervals and each time interval of each video clip is assigned one action label, while respecting the order in which the action labels appear in the given annotations. We show that the action label assignment can be determined together with learning a classifier for each action in a discriminative manner. We evaluate the proposed model on a new and challenging dataset of 937 video clips with a total of 787720 frames containing sequences of 16 different actions from 69 Hollywood movies.
1 Introduction
The paper addresses weakly supervised action recognition by using the temporal ordering of script-derived action annotations to jointly localize actions and learn action classifiers. It formulates this as discriminative clustering with ordering constraints and reports improved temporal localization and classification performance.
- Motivation: Fully annotated action videos are time-consuming to collect, while textual annotations from movie scripts provide weak supervision with poor temporal localization.The paper identifies temporal ordering as useful supervisory information that has been largely ignored in weakly supervised learning.
- Approach: The proposed discriminative clustering model jointly learns a classifier for each action and temporally assigns actions to intervals across clips.The model incorporates ordering constraints directly into the learning problem rather than treating localization and classification as separate stages.
- Problem formulation: Each clip is divided into time intervals, which are assigned action labels while preserving the order of the annotated action sequence.The annotations map ordered positions to action labels, even though the action-label set itself has no natural order.
- Optimization: The method uses a convex relaxation and the Frank-Wolfe conditional-gradient algorithm to optimize over the resulting constraint set.Frank-Wolfe is suitable because linear optimization over the constraint polytope is tractable even when projection operations are not.
- Evaluation: The paper evaluates the model on temporal localization and action classification using a new action dataset.The contributions state improved performance on both tasks, while related methods generally use weak annotations without jointly incorporating multiple action-order constraints.
2 Discriminative Clustering with Ordering Constraints
The model jointly learns action classifiers and temporally assigns weakly annotated video intervals while enforcing the annotated action order. It represents assignments with structured indicator matrices and uses a convex quadratic objective to learn shared classifiers implicitly.
- Joint discriminative learning: The model jointly solves temporal assignment and discriminative classifier learning across clips using a regularized loss.The classifiers are shared among all video clips, while each clip receives an ordering-consistent temporal assignment.
- Temporal assignment: Each clip is divided into time intervals assigned to annotated actions in sequence, with assignments constrained to remain at the current action or advance by one.Admissible assignments start with the first annotation, end with the last, and cannot skip annotation positions.
- Assignment-matrix parameterization: An assignment matrix Z encodes each interval’s action label with one-hot rows, and valid matrices are in bijection with admissible assignments.The valid set consists of matrices whose action columns form contiguous blocks in the predefined annotation order.
- Assignment-matrix parameterization: The valid assignment domain can be extremely large, with approximately T ≈ 100 intervals and K ≈ 10 annotated actions in the stated setting.Its size corresponds to choosing K − 1 transitions among T − 1 possible boundaries.
- Quadratic formulation: Choosing linear classifiers with square loss and Frobenius regularization yields a ridge-regression objective whose classifier parameters can be eliminated in closed form.The resulting problem is a quadratic optimization over assignment variables, with shared W and b recovered from the optimal assignment.
3 Convex Relaxation and the Frank-Wolfe Algorithm
The discrete assignment problem is relaxed to a convex-hull domain and optimized with Frank–Wolfe, which requires only linear minimization over that domain. Those linear subproblems are solved efficiently by dynamic programming, followed by geometric rounding to a valid discrete assignment.
- 3.1 The Frank-Wolfe Algorithm: Replacing the discrete valid-assignment set with its convex hull produces a continuous convex optimization problem over a complex polytope.The relaxation addresses the large finite domain of integer assignments while preserving the assignment constraints through the hull construction.
- 3.1 The Frank-Wolfe Algorithm: Frank–Wolfe iteratively minimizes an affine objective approximation over the polytope and moves toward the resulting boundary point.For quadratic objectives, the interpolation parameter is obtained by solving a univariate quadratic equation, and the algorithm provides a linearization duality gap for stopping.
- 3.2 Linear minimization: Each Frank–Wolfe linear subproblem is solved by dynamic programming over admissible temporal paths.After constructing the cost matrix D, the optimal path cost is computed through dynamic programming in O(TK), with back pointers recovering the assignment.
- Rounding: The converged relaxed solution is rounded by finding the closest valid assignment matrix in Frobenius distance.Because the norms of the relaxed and valid matrices are fixed, rounding reduces to another linear minimization over the valid set and can again use dynamic programming.
4 Practical Concerns
The model addresses practical issues in discriminative clustering by handling semi-supervised annotations, avoiding trivial assignments, and mitigating class imbalance while preserving efficient optimization.
- Semi-supervised setting: Semi-supervised annotations are incorporated by modifying the optimization problem without changing the optimization procedure.Fully annotated intervals contribute descriptors and class labels, while the existing procedure remains valid.
- Avoiding trivial solutions: Temporal constraints prevent constant assignment matrices from being trivial optima because the feasible assignment set is not permutation-symmetric.This removes one inherent discriminative-clustering difficulty.
- Avoiding trivial solutions: The centering matrix can still produce near-background assignments, with almost all intervals assigned to the background label ∅.The paper therefore introduces two remedies for trivial solutions.
- Class proportions: Hard class-proportion constraints would make dynamic programming intractable because it cannot enforce minimum and maximum proportions.The authors replace them with a validation-set-selected Lagrangian linear penalty while retaining Algorithm 1.
- Balancing the loss: Because the constraint set is heavily unbalanced toward ∅, the loss is reweighted by class rather than by instances.The class-weight matrix is validated, and only the Frank-Wolfe gradient computation changes.
5 Dataset and Features
The authors construct a new action dataset from Hollywood2 movies because script-based annotations are noisy and lack ground-truth temporal timestamps.
- Dataset: The dataset contains video clips annotated with sequences of actions, derived from the 69 movies used to create Hollywood2.The authors manually added full time-stamped annotations for 16 action classes.
6 Action Labeling Experiments
The action-labeling experiments divide data among supervised, evaluation, and validation roles and compare weakly supervised and semi-supervised methods using alignment performance.
- Experimental Setup: 90% of the dataset is split into Sup, Eval, and Val subsets for supervision, evaluation, and hyperparameter validation.Sup is used only in the semi-supervised setting, while Val adjusts λ, κ, and D.
- Experimental Setup: Figure 5 depicts the data split used for the action-labeling experiments.
- Baselines: The baseline partitioning objective is convex and uses Frank-Wolfe optimization to seek temporally coherent segments based on Chi-squared feature similarity.A weakly supervised baseline replaces ordering constraints with “at least one” occurrence constraints.
- Alignment Evaluation: Figure 6 compares alignment evaluation for weakly supervised models against SL across fully supervised-data fractions and reports semi-supervised results separately.Weak methods do not use the fully supervised data shown for comparison with SL.
- Results: Our semi-supervised method is consistently above the square-loss baseline, while the semi-supervised Bojanowski et al. baseline improves as full supervision increases.The comparison attributes the advantage to using weak annotations across the whole dataset together with full annotations on Sup.
7 Classification Experiments
Classification experiments compare the proposed classifiers with supervised and weakly supervised baselines, showing stronger performance for the proposed model in both settings.
- Results: Figure 7 reports alignment performance for various weakly supervised methods across all action classes.
- Weakly Supervised Setup: The proposed model outperforms the supervised square-loss classifier in weakly supervised experiments, despite using only weak annotations.The Bojanowski et al. method performs worst, which the authors relate to its lack of full supervision and ordering constraints.
- Semi-supervised Setup: In the semi-supervised setting, the proposed action model consistently performs better than the supervised SL baseline trained only on fully supervised data.