Source-linked AI summary
RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis
TL;DR
RecipeQA addresses the challenge of enabling machines to understand procedural cooking knowledge through multimodal comprehension. It introduces a large recipe dataset with automatically generated tasks over aligned text and images, and baseline results characterize it as a challenging benchmark for multimodal systems.
Problem
Recipe comprehension requires systems to understand procedural text and connect information across cooking instructions and modalities.
Method
The paper introduces RecipeQA, a dataset of recipes with titles, descriptions, aligned images, and automatically constructed multiple-choice tasks covering four comprehension styles.
Results
Baseline results show that visual input improves textual cloze performance, while different models perform better on different visual tasks, demonstrating the dataset’s challenge.
Takeaways & Limitations
RecipeQA provides a publicly available benchmark for evaluating and developing multimodal machine-comprehension methods for procedural cooking knowledge.
Abstract
from arXiv · showhide
Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of cooking recipes. It comprises of approximately 20K instructional recipes with multiple modalities such as titles, descriptions and aligned set of images. With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge. Our preliminary results indicate that RecipeQA will serve as a challenging test bed and an ideal benchmark for evaluating machine comprehension systems. The data and leaderboard are available at http://hucvl.github.io/recipeqa.
1 Introduction
RecipeQA frames cooking recipes as multimodal procedural-comprehension problems and introduces a benchmark combining instructional text with aligned images. Its tasks target varied reasoning skills, including state tracking, cross-step integration, and joint image-text understanding.
- RecipeQA targets multimodal machine comprehension of cooking recipes using recipe descriptions and images aligned with particular procedural steps.
- Recipe understanding is treated as procedural reasoning because recipes encode ordered instructions, entities, actions, and state changes needed to complete a task.
- The dataset supports arbitrary numbers of recipe steps and step images, allowing questions to span multiple paragraphs and images.
- Its different question styles are designed to test distinct comprehension skills while requiring connections between modalities and procedural language.
- RecipeQA is presented as a challenging benchmark for evaluating multimodal comprehension systems, with statistical analyses and baseline results across introduced tasks.
2 RecipeQA Dataset
RecipeQA is a multimodal dataset of real-life cooking recipes, pairing step instructions with aligned titles, descriptions, and images. It provides automatically generated questions across recipe modalities and preserves category-balanced, non-overlapping dataset splits.
- RecipeQA contains approximately 20K recipes from 22 food categories and over 36K question-answer pairs.
- Each recipe contains an arbitrary number of steps with titles, descriptions, and illustrative images aligned to those steps.
- The recipes were collected from Instructables and filtered for English language, readable content, and normalized descriptions.
- Questions are generated automatically from structured step data using fixed procedures and modality-specific contexts and candidate answers.
- The dataset uses non-overlapping training, validation, and test splits with similar food-category distributions.
3 Tasks
RecipeQA defines four multimodal comprehension tasks that test missing text, missing images, visual consistency, and temporal ordering. Their contexts and answer choices draw on different combinations of recipe titles, descriptions, and images.
- RecipeQA includes textual cloze, visual cloze, visual coherence, and visual ordering tasks.
- Textual Cloze: Textual cloze questions hide a step title or description and require selecting the missing text using other modalities as context.
- Visual Cloze: Visual cloze questions hide a step image and require inferring it from a sequence of textual step titles and descriptions.
- Visual Coherence: Visual coherence questions replace one representative image with a distractor and require identifying the incoherent image in an ordered recipe sequence.
- Visual Ordering: Visual ordering questions require arranging jumbled recipe images according to the temporal order of the corresponding steps.
4 Experiments
The experiments compare simple and neural baselines using textual, visual, and multimodal representations for RecipeQA’s question types. Results indicate that visual information improves textual cloze comprehension, while simple baselines perform better on visual coherence and ordering tasks.
- Representations: Doc2Vec produces 100-dimensional embeddings for recipe titles and descriptions while encoding word order and semantic similarity.The model represents words, sentences, and paragraphs as fixed-sized vectors.
- Representations: ResNet-50 provides 2048-dimensional visual features that an autoencoder reduces to 100 dimensions for compatibility with text embeddings.
- Neural baselines: The neural baseline uses a three-layer unidirectional stacked LSTM that processes recipe steps sequentially to preserve temporal order.
- Neural baselines: The modular Impatient Reader adapts its query for different question types and trains with cosine similarity, hinge ranking loss, ADAM, and early stopping.The margin is 1.5, the learning rate is 1e-3, and early stopping uses validation accuracy with patience 10.
- Results: For textual cloze, multimodal Impatient Reader outperforms its text-only version, while Hasty Student performs better on visual coherence and visual ordering.Hasty Student ignores the context and selects answers using similarities or dissimilarities among question elements and candidates.
5 Related Work
RecipeQA is positioned among multimodal comprehension datasets while targeting procedural knowledge in cooking recipes. It differs from prior datasets through natural, unconstrained images and recipes with variable numbers of aligned steps, images, and sentences.
- Multimodal datasets: Prior multimodal datasets include COMICS for comic narratives and TQA for middle-school science lessons with diagrams and text.COMICS explores visual cloze questions, whereas TQA addresses science knowledge acquisition and requires limited reasoning.
- Distinctive setting: RecipeQA uses natural images taken by anonymous users in unconstrained environments, unlike the images in COMICS and TQA.
- Cooking-recipe research: Earlier cooking-recipe research addressed parsing, text-video alignment, recipe generation, cross-modal embeddings, and entity or action tracking.
- Distinctive setting: The paper states that no prior dataset focused on how-to instructions or recipes with arbitrary numbers of steps aligned to multiple images and sentences.
6 Conclusion
The paper presents RecipeQA as a benchmark for multimodal comprehension of procedural cooking knowledge. Its dataset and question styles are designed to connect modalities while evaluating distinct comprehension skills.
- Contribution: RecipeQA contains roughly 20K cooking recipes and over 36K context-question-answer triplets.
- Contribution: The authors describe RecipeQA as the first machine-comprehension dataset focused on procedural knowledge in a multimodal setting.
- Contribution: Its four question styles target particular skills and require connecting information across different modalities.
7 Supplementary Notes
The supplementary material illustrates RecipeQA examples across textual and visual question formats. Examples pair recipe contexts with questions and candidate answers, marking correct and incorrect predictions visually.
- Prediction examples: The supplementary section introduces prediction results from baseline models for each task.
- Textual example: A textual cloze example uses step descriptions and images as context while generating questions from recipe step titles.Correct answers and neural-model predictions are marked in green; an incorrect Hasty Student prediction is marked in red.
- Recipe example: The illustrated recipe is Peppermint-Patty-Pudding-Shot, with context modalities consisting of step titles and descriptions.
- Recipe example: The recipe example proceeds from gathering ingredients to mixing them and preparing the pudding for serving.The listed ingredients include chocolate pudding, milk, peppermint schnapps, cool whip, and crushed peppermint candy.
- Visual examples: Visual cloze, visual coherence, and visual ordering examples use step titles and descriptions as context while generating questions from recipe images.Correct answers appear with green frames or text, while wrong answers are marked red.