Cross-task weakly supervised learning from instructional videos

Dimitri Zhukov; Jean-Baptiste Alayrac; Ramazan Gokberk Cinbis; David Fouhey; Ivan Laptev; Josef Sivic

Source-linked AI summary

Cross-task weakly supervised learning from instructional videos

Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, Josef Sivic

arXiv:1903.08225v2Mar 19, 2019cs.CV

TL;DR

The paper addresses learning step-level visual models from instructional videos using narrations and ordered step lists rather than temporal annotations. It introduces compositional component sharing, a weakly supervised learning framework, and CrossTask, finding improved cross-task performance and parsing of previously unseen tasks.

Problem
The paper asks how to learn visual models for multi-step tasks from instructional videos using weak narrations and ordered step lists instead of temporal annotations.
Method
The approach represents each step with shared component classifiers and learns their assignments and parameters under temporal constraints using alternating optimization.
Results
Sharing across tasks improves performance, especially at the component level, and the component model parses previously unseen tasks.
Takeaways & Limitations
Compositional representations let models exploit common parts across tasks and transfer to tasks not seen during training.
Takeaways & Limitations
The setting assumes an ordered list of steps and instructional narration, while CrossTask videos contain substantial background and omitted steps.

Abstract

from arXiv · show

In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations. At the heart of our approach is the observation that weakly supervised learning may be easier if a model shares components while learning different steps: `pour egg' should be trained jointly with other tasks involving `pour' and `egg'. We formalize this in a component model for recognizing steps and a weakly supervised learning framework that can learn this model under temporal constraints from narration and the list of steps. Past data does not permit systematic studying of sharing and so we also gather a new dataset, CrossTask, aimed at assessing cross-task sharing. Our experiments demonstrate that sharing across tasks improves performance, especially when done at the component level and that our component model can parse previously unseen tasks by virtue of its compositionality.

1. Introduction

The paper targets visual models for task steps using weak supervision from instructional narrations and ordered step lists rather than costly temporal annotations. It addresses localization and limited-data challenges by sharing information across related tasks through a compositional component model, evaluated on the new CrossTask dataset.

Temporal step annotations are time-consuming and unscalable, motivating weakly supervised learning from ordered steps and instructional narrations.
Weak supervision must jointly learn when steps occur and what they look like despite combinatorial localization choices and limited data for each visual model.
The component model shares information across steps by representing each step through constituent components such as verbs and nouns.
The framework combines compositional sharing with temporal constraints and alternates between updating classifier labels and classifier parameters.
CrossTask contains 4.7K instructional videos spanning 83 tasks and 376 hours, enabling evaluation of weakly supervised learning, related-task sharing, and unseen-task parsing.

2. Related Work

Prior instructional-video work commonly relies on strong annotations or weak supervision without compositional sharing. This paper addresses limitations in existing datasets and methods with a diverse, temporally annotated dataset and automatically learned component sharing.

Action-recognition approaches for instructional videos commonly use strong direct labels, whereas this method does not depend on lots of annotated data.
The proposed compositional model shares noun and verb components across steps instead of learning monolithic models independently per step.
Existing datasets are limited by small scale, narrow domains, or sparse temporal annotation, restricting systematic evaluation of cross-task sharing.
The collected dataset includes 83 tasks and 4.7K videos across car maintenance, cooking, and crafting, with temporal localization annotations.
The technical approach combines constrained discriminative clustering with an automatically learned compositional framework for weakly supervised learning.

3. Overview

The approach learns visual models for multi-step tasks by decomposing each step into shared component models. It uses only ordered step lists for manual supervision and training-time narration to support weakly supervised learning.

Each task is a multi-step process, and the goal is to learn a visual model for every step from instructional videos.
Instead of a monolithic classifier, each step is represented as a combination of classifiers for constituent components such as actions and objects.
Sharing component models transfers data between steps and supports parsing tasks that were not seen during training.
The ordered step list is the only manual supervision, while training videos additionally provide audio narrating the actions.

4. Modeling Instructional Videos

The framework jointly learns step labels and visual classifiers from ordered step lists and narrated videos, using temporal constraints and component-level sharing. Its component model represents each step by combining classifiers for its words, while optimization alternates between assignments and classifier parameters.

4. Modeling Instructional Videos: Training is weakly supervised because videos provide ordered step lists but not temporal locations, which the framework jointly infers with visual classifiers.Each video contains visual segments, narrations, and an ordered set of natural-language steps.
4. Modeling Instructional Videos: The model jointly learns binary temporal labels and step classifiers, using constraints to avoid trivial all-zero solutions.The labels indicate whether each video segment depicts a given task step.
4.1. Component Classifiers: The component model decomposes each step into stemmed-word components, enabling sharing across verbs, objects, and prepositions rather than only exact duplicate steps.A per-task matrix records which components occur in each step, while one classifier is learned per component.
4.1. Component Classifiers: Each step prediction averages the outputs of its component classifiers; for example, pour milk combines gpour and gmilk.This replaces a monolithic classifier for each step with compositional step models.
4.2. Objective and Constraints: The objective simultaneously optimizes temporal assignments Y and classifiers F across videos and tasks using a per-video cross-entropy loss.The classifier family F and temporal constraints C determine the optimization problem.
4.2. Objective and Constraints: Temporal constraints require every step to appear at least once, preserve the given order, and remain near matching narration intervals.These assumptions reduce the search space, while text-based intervals restrict possible step locations.
4.3. Optimization and Inference: Alternating optimization updates assignments independently per video and classifier parameters, with assignment updates solvable by dynamic programming.Fixing classifiers yields a constrained linear-cost problem in the labels; fixing labels yields the classifier update.
4.4. Implementation Details: The implementation uses a linear classifier with dropout, ADAM at 10^-5, and RGB I3D, ResNet-152, and audio features.The features are extracted over one-second temporal windows, with dimensions 1024D, 2048D, and 128D respectively.

5. CrossTask dataset

CrossTask was constructed to study cross-task sharing using diverse instructional videos and task relationships. Its videos are long and procedurally complex, with substantial background, omitted steps, and imperfect order consistency creating challenges for weakly supervised learning.

5. CrossTask dataset: CrossTask contains 83 tasks and 4.7K videos spanning cooking, car maintenance, crafting, and home repairs.The tasks derive from wikiHow, while the videos come from YouTube.
5. CrossTask dataset: The dataset separates fully annotated primary tasks from automatically gathered related tasks that share some components with the primary tasks.Related tasks are used to assess whether additional task videos improve learning on primary tasks.
5. CrossTask dataset: Task selection required physical object interactions, deterministic step order, and frequent YouTube appearance; filtering began from approximately 7K wikiHow candidates.Annotators reviewed wikiHow sections and manually filtered candidates for frequency.
5. CrossTask dataset: The collection selected 18 primary and 65 related tasks, with related tasks proposed using TF-IDF similarity between wikiHow descriptions and then filtered.Primary tasks span themes including auto repair, cooking, and DIY.
5. CrossTask dataset: CrossTask includes 2750 primary-task videos covering 212 hours and 1950 related-task videos covering 161 hours, with temporal annotations for primary-task steps.The primary videos were independently annotated for each step’s temporal extent.
5.2. Annotations and Statistics: Primary-task videos average 4min 57sec and 7.4 steps, while 72% of frames are background, 31% of steps are absent, and the best order-respecting parse misses 14%.The paper identifies fine-grained visual confusion and noun-driven overreaction as additional failure modes.

6. Experiments

The experiments evaluate cross-task sharing, compare component- and step-level sharing, and test transfer to unseen tasks. Sharing improves weakly supervised recall, component-based sharing extracts more benefit than step-based alternatives, and related-task training supports primary-task parsing.

6.1. Cross-task Learning: The proposed approach uses related tasks to improve performance on 18 primary tasks, with recall measuring correct step assignments across test videos.Each method makes one prediction per step, preventing assignment of all frames to every action.
6.1. Cross-task Learning: 22.4% average recall for the full sharing approach exceeded 18.6% for task-specific step training, improving on that baseline in 17 of 18 tasks.The comparison indicates a systematic advantage for cross-task sharing rather than an isolated task improvement.
6.1. Cross-task Learning: Sharing benefits depend on component compatibility: stirring gained 15% recall over independent training, whereas cut shelf fell from 47% to 28% in dissimilar contexts.The authors associate gains with components that are distinctive and frequent, while cross-context differences can reduce transfer.
6.2. Experimental Evaluation of Cross-task Sharing: In the sharing evaluation, increased sharing consistently improved performance, and component-based sharing outperformed step-based sharing.Step-based sharing requires exact matches, while component sharing captures variants such as cutting steak, tomato, or pickle.
6.3. Novel Task Transfer: Training only on related tasks enabled parsing of primary tasks, and increasing the number of related training tasks improved recall without plateauing at 65 tasks.The model parsed unseen combinations such as cutting strawberries by recombining knowledge from other cutting and cake-related steps.

7. Conclusion

The paper presents weakly supervised learning from instructional videos through compositional component sharing, temporal constraints, and a new dataset. Its framework supports efficient constrained optimization and recombination of components for unseen tasks.

7. Conclusion: Cross-task component sharing improves learning and allows the model to parse previously unseen tasks through compositionality.The paper identifies sharing and unseen-task parsing as demonstrated capabilities of the component model.
7. Conclusion: Weak supervision combines ordered step lists and narrated videos with temporal constraints to jointly learn step labels and visual models.Ordering and at-least-once constraints make the assignment problem efficiently solvable by dynamic programming.
7. Conclusion: The framework learns sentence representations by bringing semantically similar narrations closer and separating dissimilar ones, outperforming TF-IDF and max-pooled word vectors in precision and recall.The learned representation uses convolution, global max-pooling, and a linear mapping.
7. Conclusion: The constrained optimization uses dynamic programming with O(KT) complexity while enforcing ordering and at-least-once step coverage.The final state must indicate that the last step has been reached, with or without background at the end.

C.1. Video collection

The video collection procedure balances training quantity against task relevance and shows why cross-task sharing is needed when individual tasks have few positive examples. Video overlap between primary and related tasks is limited enough that transfer is nontrivial.

C.1. Video collection: Collecting more videos can improve task learning, but larger retrieval sets also contain more unrelated videos that may hurt performance.The number of retrieved videos is therefore treated as a purity-versus-quantity trade-off.
C.1. Video collection: 30 videos are insufficient to learn a task from scratch because they provide only a few positive examples per step.The paper identifies sharing knowledge across tasks as essential for overcoming this limitation.
C.1. Video collection: Primary and related tasks share about 2.6% of videos, while 70% of primary-task videos do not share channels with related-task videos.The dataset contains videos from 3007 YouTube channels, averaging 1.6 videos per channel.
C.1. Video collection: The limited overlap between primary and related videos means transferring related-task models to primary tasks is not trivial.Primary tasks share little with one another, whereas related tasks can be similar to primary tasks.

C.2. Tasks and statistics

CrossTask organizes tasks by shared step components and reports substantial variation in temporal ordering, background, and missing-step statistics. These properties motivate hard ordering constraints while exposing challenging tasks.

C.2. Tasks and statistics: Tasks mostly share components within the same domain, forming groups such as Car Maintenance, Drinks, and Woodworks.Task distances are based on the number of common step components and visualized with t-SNE.
C.2. Tasks and statistics: 72% average background motivates methods that leave frames unlabeled rather than requiring dense video segmentation.Background is the average fraction of frames not assigned to any step.
C.2. Tasks and statistics: 86% average order consistency supports using hard ordering constraints, although consistency is relatively low for some tasks.Make Kimchi Rice and Make Taco Salad are cited as examples with lower order consistency.
C.2. Tasks and statistics: Missing steps approach 50% for Grill Steak and Pickle Cucumber, making these tasks especially challenging.The statistic refers to the amount of annotated task steps absent from videos.

D.1. Comparison of evaluation metrics

Evaluation predicts one punctual temporal unit per step rather than an interval, reflecting the ill-posedness of exact boundaries under weak supervision. Narration constraints are noisy but still improve training recall.

D.1. Comparison of evaluation metrics: The evaluation counts a detection as correct when its predicted temporal unit falls within the ground-truth interval for that step.The model predicts one punctual unit per step because exact temporal extents are ill-posed without temporal training annotations.
D.1. Comparison of evaluation metrics: Recall is approximately twice mAP under the reported evaluation procedures.The two evaluation methods are described as highly correlated, with recall roughly equal to mAP × 2.
D.1. Comparison of evaluation metrics: Narration-derived constraints have 7.9% intersection over union with ground-truth steps, and 61% of ground-truth steps lie entirely outside them.This noise can prevent correct frame assignment even with a perfect classifier when constraints are enforced.
D.1. Comparison of evaluation metrics: 17% recall without text constraints rises to 22.4% with them, a 5.4% gain despite substantial temporal noise.The comparison uses the same training and evaluation setup with and without text constraints.

D.3. Additional qualitative results

Qualitative examples compare frame-level classifier outputs, inferred step localizations, and ground truth across several tasks, exposing both successful detections and assumption-driven errors.

Figures 4–9 compare classifier outputs, inferred solutions, and ground-truth intervals across videos for six tasks.Outputs are blue, correct localizations green, false detections red, and ground-truth intervals yellow.
A missing Top toast step produces a false detection because the method assumes every step is present.The Top Toast classifier also responds to an end-of-video final-product presentation, which resembles its positive training intervals.
Figure 4 shows false localizations caused by ordering constraints, including Pour milk, Whisk mixture, and Dip bread.Whisk Mixture is localized in a low-confidence region because Pouring Egg precedes it in that video.
Figures 5–9 provide additional qualitative examples of inferred solutions and ground-truth comparisons for shelves, cake, coffee, tire, and fish-curry tasks.Each figure uses the same visual encoding for classifier outputs, correct detections, false detections, and ground-truth intervals.
Figure 10 shows errors involving wrong objects and actions, especially confusing liquids with powders and static context with performed actions.The stated feature representation cannot reliably distinguish some object types or capture action dynamics.