Weakly-Supervised Alignment of Video With Text

1. Introduction

• Manual temporal annotations are expensive, and action boundaries can be difficult to define in long videos.The introduction contrasts temporal localization with classifying short clips containing a single action.Source: §1 · p. 1
• The method aligns video frames with ordered sentences using weak supervision and an implicit linear mapping between their feature spaces.It formulates alignment as an integer quadratic program and solves a continuous convex relaxation with conditional gradient.Source: §1 · p. 1

1.1. Problem statement and approach

• Each video element is assigned to one text element while a linear map makes the assigned text feature approximate its video feature.Inserting zero vectors between text-feature columns allows some video frames to remain unassigned.Source: §1.1 · p. 2
• Separating text representation from assignment extends [7] from symbolic labels to continuous text features and permits data-independent assignment constraints.The earlier formulation combines labels and temporal assignments in one matrix.Source: §1.1 · p. 3
• The paper contributes an extension for continuous text, a method for adding assignment priors, and evaluations on video datasets with natural-language metadata.These are the three contributions stated by the authors.Source: §1.1 · p. 3

2.1. Basic model

• Each video element receives exactly one text assignment, and assignments preserve temporal order so consecutive video elements map to the same or next text element.This assignment structure also enables dynamic programming to minimize linear functions over feasible assignments.Source: §2.1 · p. 3
• The basic model fits the mapped text features to video features using squared L2 discrepancy and an L2-regularized linear map.For a fixed assignment, optimizing the map is a ridge-regression problem with a closed-form solution.Source: §2.1 · p. 3
• For multiple streams, the feature matrices are concatenated and the assignment matrix becomes block diagonal, with one assignment block per stream.The streams can be videos in this setting.Source: §2.1 · p. 3

2.2. Priors and constraints

• A duration prior penalizes sentence durations, measured by the number of video elements assigned to each text element, relative to a shared target and variance.The model obtains each duration from the corresponding row of the assignment matrix.Source: §2.2 · p. 3
• A path prior limits assignments outside a diagonal band to avoid degenerate solutions, while a penalty preserves the dynamic-programming structure.The authors select the penalty weight by grid search on a validation set.Source: §2.2 · p. 4

2.3. Full problem formulation

• The full problem minimizes the assignment cost together with duration and path-constraint penalties over the feasible assignment set.The objective combines q(Y), r(Y), and l(Y).Source: §2.2 · pp. 3–4, §2.3 · p. 4

3.1. Continuous relaxation

• The discrete positive-semidefinite quadratic objective over binary assignments is relaxed to a convex quadratic minimization over their convex hull.The paper avoids a general linear-programming solver to exploit the alignment structure.Source: §3.1 · p. 4
• Frank–Wolfe optimization is efficient when linear forms can be minimized over the convex domain, which dynamic programming does in O(IJ) steps before backtracking.The algorithm builds cumulative alignment costs recursively, then recovers the optimal assignment by backtracking.Source: §3.1 · p. 4

3.2. Rounding

• Three rounding procedures map the relaxed solution to a valid assignment: nearest in assignment space, feature-weighted distance, or prediction from the learned W.Each reduces to a linear problem over the valid assignment set.Source: §3.2 · pp. 4–5
• The W-based rounding procedure can also predict alignments for previously unseen data.Source: §3.2 · p. 5

4. Semi-supervised setting

• The weight κ balances supervised and unsupervised examples by scaling their feature representations.The scaling corresponds to a loss that weights the two types of examples.Source: §4 · p. 5
• With exact supervision, the known lower-right entries of Y are fixed to their ground-truth values during optimization.Source: §4 · p. 5
• Softer interval constraints let assignments select the most discriminative video portions within annotations and empirically yield much better performance.Source: §4 · p. 5

5. Experimental evaluation

• Experiments use [7]’s 843 videos and 16 labels, plus TACoS’s 127 cooking videos with crowd-sourced timed descriptions; sentence features include bag-of-words and continuous embeddings.TACoS also has fine-grained activity annotations, which this work does not use.Source: §5.1 · p. 5, §5.2 · p. 6

5.1. Comparison with Bojanowski et al. [7]

• With supervision, ΨY and W roundings improve over [7] by more than 5%; without supervision, our method is significantly better, while Y rounding generally performs poorly.Figure 5 compares performance against the amount of supervised data using the same evaluation metric as [7].Source: Figure 5 · p. 6, §5.1 · p. 6
• The initial performance drop seen in [7] is absent: less-coercive supervision accommodates annotations that need not match the unsupervised assignment, and ΨY is the best-performing rounding.The ΨY rounding corresponds to the Z = ΨY matrix used by [7].Source: §5.1 · p. 6

5.2. Results on the TACoS dataset

• 0.475 versus 0.441 (infinite σ): a carefully chosen duration prior improves performance, but an overly strong prior yields degenerate, lower-performing solutions.Figure 6 evaluates the priors and their parameter settings.Source: Figure 6 · p. 7, §5.2 · p. 7
• The band prior improves performance further; the best band width is 0.1, and narrow bands require careful α selection because the best path may not lie entirely inside them.Figure 6 plots performance against α and band width.Source: Figure 6 · p. 7, §5.2 · p. 7
• TACoS-trained continuous representations perform best; 200 dimensions do not help with its limited vocabulary, while Google News embeddings score 42.1.Table 1 compares representations trained on different corpora and in dimensions 100 and 200.Source: Table 1 · p. 7, §5.2 · pp. 7–8
• In Table 2’s unsupervised and semi-supervised comparisons, the diagonal baseline has reasonable performance; on ROOT, [7] exceeds diagonal but remains pretty far below our ROOT model.Source: Table 2 · p. 7, §5.2 · p. 8
• For bag-of-words, simple pooling works best: verbs alone lead without supervision, while ROOT+DOBJ performs better than ROOT as supervision increases.The authors suggest richer representations can mislead this weakly supervised method.Source: §5.2 · p. 8

• The paper presents a method for aligning videos with their natural-language descriptions.Source: §6 · p. 8
• Feature films and more complicated grammatical structures are future scenarios, and the authors plan to incorporate better grammatical reasoning.They note that their current use of natural-language processing tools is limited.Source: §6 · p. 8