Source-linked AI summary
Towards Automatic Learning of Procedures from Web Instructional Videos
Luowei Zhou, Chenliang Xu, Jason J. Corso
TL;DR
The paper asks whether human-consensus procedure structure can be learned from long, unconstrained videos using only visual evidence. It introduces a large annotated cooking-video dataset and ProcNets, a segment-level recurrent model, which achieves top performance against competitive baselines; performance drops substantially when temporal order is permuted.
Problem
Procedure learning commonly relies on subtitles, action labels, or fixed numbers of steps, motivating visual-only segmentation of unconstrained videos into category-independent procedure segments.
Method
ProcNets use context-aware video encoding, segment proposal, and sequential prediction to generate procedure segments while modeling dependencies among candidates.
Results
ProcNets achieve top performance against competitive baselines on standard procedure-segmentation metrics, with segment-level temporal modeling outperforming competing variants.
Takeaways & Limitations
The dataset and model support studying procedure structure from unconstrained instructional videos and provide segments usable for tasks such as dense video captioning and event parsing.
Takeaways & Limitations
When video segments are temporally permuted, ProcNets performance decreases by over a half, indicating reliance on learned temporal structure.
Abstract
from arXiv · showhide
The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.
Introduction
The paper frames procedure segmentation as learning human-consensus structure from long, unconstrained videos using visual evidence, then introduces YouCook2 and ProcNets to address it.
- Problem: Procedure segments are temporally bounded, conceptually compact units that may contain multiple actions and reflect human consensus about procedure structure.The paper illustrates this representation with eight segments for a BLT sandwich video.
- Problem: Procedure segmentation automatically divides a procedure video into category-independent segments that are closely coupled and usually have long-term dependencies.Unlike event proposals, the task identifies a necessary sequence of segments rather than maximizing recall over loosely coupled events.
- Motivation: Existing procedure-learning methods commonly assume concurrent language, a fixed number of procedure steps, or both.The paper identifies these assumptions as restrictive for the proposed visual-only setting.
- Dataset: YouCook21 contains 2000 videos from 89 recipes totaling 176 hours, with temporally bounded procedure steps described by imperative English sentences and checked by two annotators.The paper presents it as more than twice the size of the nearest comparable dataset and the only one combining both annotation types.
- Method: ProcNets segment long, unconstrained videos without subtitles or a known segment count using context-aware encoding, segment proposals, and sequential prediction.The sequential module finalizes boundaries by modeling dependencies among candidate segments.
- Results: ProcNets achieve top performance against competitive baselines on standard metrics, and a detailed study indicates that they learn procedure structure as expected.The generated segments are also positioned as preprocessing for tasks including dense video captioning and event parsing.
Related Work
The related work spans action proposals and action segmentation, while the paper distinguishes procedure segments by their semantic richness, coupling, and long-range dependencies.
- Action proposals: Action-proposal methods inspire segment-candidate generation, but ProcNets learn proposal locations through anchor offsets instead of transferring fixed clustered locations.The paper contrasts this with Deep Action Proposals, which use locations clustered from ground-truth segments.
- Action segmentation: Action segmentation and labeling focus on contiguous segments corresponding to sequences of actions, generally emphasizing transitions between adjacent action states.The paper notes that these methods leave long-range dependencies insufficiently captured and assume limited background activity.
- Datasets: YouCook2 addresses prior instructional-video dataset limitations involving video scale, background diversity, activity diversity, and missing boundary or description annotations.Its dataset statistics include recipe-level segment counts and duration distributions.
YouCook2 dataset
YouCook2 is a large, diverse dataset of untrimmed instructional cooking videos with temporally localized and imperatively described procedure segments, annotated to reflect human consensus.
- Dataset scope: YouCook2 contains 2,000 YouTube videos across 89 recipes, spanning four cuisine locales and varied cooking styles, methods, ingredients, and cookware.The videos include unconstrained recording conditions such as fast camera motion, zooms, defocus, and scene-type changes.
- Procedure representation: Procedure segments can contain multiple actions while remaining conceptually compact and capturing richer process information than fixed action labels.The paper notes that fixed action sets may miss attributes and fine-grained objects in instructional processes.
- Annotations: Two annotators label each video, with one providing the major annotation effort and another verifying it to reflect human consensus.The annotation process also uses restrictions intended to enforce consensus across annotators.
- Dataset use: The dataset provides temporal boundary annotations for the paper, while recipe descriptions are released for future research.The videos remain untrimmed, total 175.6 hours, and average 5.27 minutes per video.
Procedure Segmentation Networks
ProcNets segment untrimmed, unconstrained videos through context-aware encoding, candidate localization, and segment-level sequential prediction. The model avoids subtitles and a known segment count while modeling dependencies among proposed segments.
- Architecture: ProcNets combine context-aware video encoding, segment proposal, and sequential prediction to produce procedure segments from unseen videos.Training uses ground-truth boundaries without recipe categories or segment descriptions.
- Context-Aware Video Encoding: A bi-directional LSTM transforms frame-wise ResNet features into context-aware representations, outperforming context-free ResNet and LSTM-encoded features by a relative 9% on the evaluation metric.The encoder concatenates forward and backward outputs with the ResNet feature before reducing the feature dimension.
- Procedure Segment Proposal: The proposal module uses explicit anchors across video frames, predicting each candidate’s procedure likelihood and center and length offsets.Temporal convolution produces score and offset matrices, while offsets adjust anchor boundaries toward final locations during inference.
- Sequential Prediction: The sequential predictor models dependencies across candidate segments rather than enormous frame sequences, making segment-level dependency learning more tractable.Its LSTM input combines proposal scores, segment-location embeddings, and visual segment content.
- Sequential Prediction: During inference, the LSTM predicts the next proposal from the previous segment, with beam search used to sample segment-index sequences and greedy decoding performing best experimentally.The sequence begins with a special <start> segment token.
- Design rationale: ProcNets introduce segment-level sequential modeling for category-independent procedure segments without requiring hidden states or the number of procedure segments.The paper contrasts this with non-maximal suppression, which does not capture temporal structure or segment dependencies.
- Training objective: The training objective combines procedureness classification, boundary regression, and sequential prediction losses.The loss uses binary cross-entropy, smooth l1-loss, and cross-entropy, with αr = αs = 1 yielding good results empirically.
Experiments and Results
The experiments benchmark procedure segmentation with standard metrics, ablations, qualitative analyses, and comparisons against competitive baselines. ProcNets-LSTM achieves the strongest reported segmentation performance, while results also examine localization and sensitivity to temporal order.
- Baselines and metrics: The evaluation compares ProcNets variants with video-summarization, action-proposal, frame-level, and uniform baselines using Jaccard and mIoU.ProcNets-LSTM models segment-level temporal dependencies, whereas vsLSTM models frame-level dependencies.
- Temporal segmentation: ProcNets-LSTM outperforms all other methods by a huge margin on both Jaccard and mIoU across validation and testing sets.
- Ablation study: Removing the Proposal Vector, Location Embedding, or Segment Content tests the contributions of the three inputs to sequential prediction.The proposal scores are reported to play a significant role in determining final proposals.
- Segment localization: More than 9% higher recall and 7% higher precision than baselines are reported for the proposed model in segment localization evaluation.These localization results use an IoU threshold of 0.5 and evaluate ProcNets-NMS because the other metrics impose a fixed segment count.
- Qualitative results: Qualitative examples show that ProcNets can localize some segments, predict their lengths, and adapt the number of proposed segments to individual videos.The model proposes fewer segments near video beginnings and endings, where cooking processes usually do not occur.
- Temporal-order analysis: Performance decreases by over a half on temporally permuted videos, with only 10%-20% of videos predicting segments at the beginning of the second portion.The authors attribute this behavior to learned ending content and dependencies tied to the original temporal structure.
Conclusion
The paper introduces procedure segmentation for studying human consensus about procedure structure in unconstrained videos. ProcNets and the accompanying cooking-video dataset support visual procedure analysis and motivate extensions to captioning and weakly supervised segmentation.
- Conclusion: The paper introduces procedure segmentation to study human consensus about how procedures are structured in unconstrained videos.
- Conclusion: ProcNets uses frame-wise video features to predict procedure segments while modeling temporal dependencies among them.
- Conclusion: Experiments on a large-scale cooking-video dataset show significant improvements over competitive baselines under standard metrics.
- Future work: The paper identifies dense video captioning and weakly supervised segmentation as future extensions.The weakly supervised direction would align weak audio or subtitle signals with video before training.