Source-linked AI summary
What's Cookin'? Interpreting Cooking Videos using Text, Speech and Vision
Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, Kevin Murphy
TL;DR
The paper asks how procedural instructions can be aligned with videos, addressing the multimodal nature of procedural knowledge. It combines HMM-based speech alignment with visual refinement, producing large aligned recipe-video and clip corpora that support search and recipe illustration.
Problem
Procedural knowledge uses complementary language and perceptual information, motivating alignment between textual instructions and corresponding videos.
Method
The method aligns recipe steps to automatically generated speech with an HMM and refines the alignment using visual food detection.
Results
The system creates 180k aligned recipe-video pairs and 1.4M labeled video clips, with alignment improving object accuracy relative to keyword spotting and visual refinement further helping object accuracy.
Takeaways & Limitations
The aligned resources support within-video search and automatic recipe illustration while contributing toward a multimodal procedural knowledge base.
Takeaways & Limitations
Keyword spotting has higher recall, while HMM alignment can have lower recall and sometimes worse temporal localization accuracy.
Abstract
from arXiv · showhide
We present a novel method for aligning a sequence of instructions to a video of someone carrying out a task. In particular, we focus on the cooking domain, where the instructions correspond to the recipe. Our technique relies on an HMM to align the recipe steps to the (automatically generated) speech transcript. We then refine this alignment using a state-of-the-art visual food detector, based on a deep convolutional neural network. We show that our technique outperforms simpler techniques based on keyword spotting. It also enables interesting applications, such as automatically illustrating recipes with keyframes, and searching within a video for events of interest.
1 Introduction
The paper addresses alignment between procedural instructions and videos by combining textual, speech, and visual information, using cooking as a tractable domain. Its system aligns recipe steps to speech and refines them with vision, supporting large-scale procedural knowledge resources and applications.
- Procedural knowledge is multimodal, so organizing it requires extracting and integrating information from textual and visual sources.
- The paper studies alignment between video depictions of procedures and corresponding textual steps, focusing on cooking because recipes are linear and cooking videos are prevalent.
- The method first aligns instructional steps to speech with an HMM, then refines the alignment using a state-of-the-art computer vision system.
- The system combines text, speech, and vision to create 180k aligned recipe-video pairs and 1.4M labeled cooking clips.
- The resulting resources support applications including within-video search and automatic recipe illustration.
2 Data and pre-processing
The corpus pipeline collects cooking videos with recipes, extracts recipe and ingredient content, parses steps into structured actions and entities, and processes speech transcripts for alignment. The pipeline combines web-scale filtering and language processing with transcript handling, while facing recognition and parsing limitations.
- The corpus starts from 7.4M cooking videos with English transcripts and descriptive text, retaining 6.2M videos before recipe extraction.
- Recipe extraction retrieves linked documents and classifies sentences as recipe steps, ingredients, or background, leaving 180,000 videos with both ingredients and recipe sentences.
- The recipe classifier is a 3-class naïve Bayes model trained on 500k examples each of recipe, ingredient, and non-recipe sentences.
- Recipe text is parsed into micro steps, with missing-verb steps discarded and each step labeled using an optional action and noun chunks.
- Noun entities are canonicalized against recipe ingredients, while a recency heuristic resolves omitted objects such as the ingredients implicitly referenced by “Mix well.”
- The speech pipeline applies NLP to time-stamped YouTube ASR tokens, whose errors and missing punctuation can introduce additional processing errors.
3 Methods
The system aligns recipe steps with cooking videos by modeling their relationship to ASR transcripts with an HMM, then refining candidate segments using visual food detectors. It also compares this pipeline with keyword spotting and combines the two approaches for clip labeling.
- 3.1 HMM to align recipe with ASR transcript: The input-output HMM aligns recipe steps to sequences of ASR words while allowing unrelated speech to be assigned to a background model.The model represents recipe steps, transcript tokens, step assignments, and foreground/background generation; Viterbi decoding estimates the most likely sequence.
- 3.1 HMM to align recipe with ASR transcript: The HMM assumes recipe-order progression and clustered background or foreground tokens, with α set to the recipe-step-to-token ratio and γ selected by cross-validation.The transition structure encodes ordered recipe steps and contiguous background/foreground regions; γ is set to 0.7 on manually labeled recipes.
- 3.2 Keyword spotting: Keyword spotting extracts fixed windows around spoken action verbs, whereas the HMM labels segments using action and object information from aligned recipe steps.Keyword spotting uses a manually defined action whitelist and nearby noun tokens; HMM segmentation can leave background segments unlabeled.
- 3.3 Hybrid HMM + keyword spotting: The hybrid method uses keyword spotting for actions and HMM alignment for objects, falling back to transcript nouns when fewer than half the recipe steps align.This combines the higher recall of keyword spotting with HMM-based object inference while avoiding unreliable HMM output for poorly aligned videos.
- 3.4 Temporal refinement using vision: Visual refinement shifts candidate segments by up to 3 seconds toward the position maximizing the detector score for the labeled object.The procedure uses food detectors trained with a deep CNN and selects the temporal translation where the object is most visually present.
- 3.4 Temporal refinement using vision: The visual pipeline produces per-frame scores over 2809 food classes and matches ingredients to detector outputs using the maximum score when names do not correspond exactly.Videos are downsampled to 5 frames per second and 224 × 224 pixels before CNN processing.
- 3.5 Quantifying confidence via vision and affordances: Confidence for an action-object label combines the visual refinement score, which measures object visibility, with an affordance probability.The label is assigned to clips produced by keyword spotting and/or the HMM, enabling a precision-recall tradeoff.
4 Evaluation and applications
The evaluation shows that the hybrid alignment system produces higher-quality action/object clips than keyword spotting, while confidence filtering improves precision at a recall cost. The aligned segments also support recipe illustration and within-video search.
- 4.1 Evaluating the clip database: The hybrid method produces significantly better action and object clip quality than keyword spotting.Automatically generated transcripts perform nearly as well as manually curated transcripts, especially with visual refinement.
- 4.1 Evaluating the clip database: Keyword spotting is over 30% more accurate for actions than objects, whereas the hybrid method narrows this gap to 8%.Recipe-text alignment supplies cleaner object labels, and visual refinement is trained to detect ingredients.
- 4.1 Evaluating the clip database: Clip duration is a potential confound because HMM and hybrid clips vary from 2 to 10 seconds, while keyword-spotting clips are fixed at 8 seconds.Any bias favoring longer clips would benefit keyword spotting in this comparison.
- 4.1 Evaluating the clip database: Filtering segments by confidence improves clip-retrieval precision but reduces recall; the top 167,000 segments average above 1.75 in human ratings.Higher-confidence segments tend to receive the highest human quality ratings.
- 4.2 Recipe auto-illustration: Aligned recipe steps can be paired with automatically selected keyframes to illustrate recipes.A standard keyframe-selection algorithm chooses the best frame from each aligned video segment.
- 4.3 Search within a video: Within-video search returns confidence-ranked matching clips and automatically advances videos to the relevant section.This provides temporal localization within videos rather than returning only the entire video.
5 Related work
Prior work connects language and vision in instructional settings through keyword spotting, HMM alignment, visual recognition, and text illustration, but typically uses smaller or more controlled datasets. This paper differs by aligning recipe text to speech and incorporating video information at much larger scale.
- Keyword spotting labels instructional-video clips from speech transcripts, whereas this paper reports that its hybrid approach performs better, especially on automatically generated transcripts.
- Earlier HMM-based alignment generated images and used six controlled-lab videos, while this work generates ASR words and considers over 180k videos collected in the wild.
- Other recipe-alignment work used HMMs to connect recipe steps with online-forum comments, whereas this paper aligns steps to speech transcripts and adds video information.
- Prior automatic text illustration produced one image rather than a sequence, unlike this paper’s recipe auto-illustration application.
- Related cooking-video datasets and language-vision methods generally use standard visual features or controlled, smaller collections without speech, while this work targets instructional-text alignment at larger scale.
6 Discussion and future work
The paper presents a multimodal alignment method and applies it to large-scale recipe-video data. The resulting labeled corpus supports future visual-action modeling and broader multimodal language understanding.
- The method aligns instructional text to videos by leveraging speech recognition and visual object detection.
- The system aligns 180k recipe-video pairs and extracts 1.4M labeled video clips toward a multimodal procedural knowledge base.
- Future work will use the labeled corpus to train visual action detectors and combine visual and linguistic cues for language-understanding challenges.