RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes

1 Introduction

• Reading-comprehension tests assess joint understanding of questions and context, while a performance gap between humans and deep neural models motivates continued research.The passage describes new datasets as a way to investigate model limitations and capabilities.Source: §1 · p. 1
• Recipes encode real-world knowledge and sequential instructions for achieving a goal, making recipe comprehension a challenging problem for machine reasoning.The paper likens each recipe to an algorithm for solving a task.Source: §1 · p. 2
• RecipeQA averages 12 images per recipe, compared with 3 per question in TQA; its images are aligned to recipe steps and are natural user photos from unconstrained environments.TQA primarily contains diagrams or textbook images.Source: §1 · p. 2
• RecipeQA allows variable numbers of recipe steps and images, varied question styles, lexical and syntactic divergence, procedural state tracking, cross-step evidence, and joint image-text reasoning.Answering may require tracking entities and actions as their states change, and drawing on multiple recipe steps.Source: §1 · p. 2
• RecipeQA is introduced as a challenging benchmark for multimodal comprehension of cooking recipes, with statistical analyses and baseline results for its tasks.The dataset pairs recipe instructions with related questions.Source: §1 · pp. 1–2

2 RecipeQA Dataset

• Each recipe has variable-length steps pairing titles, descriptions, and aligned images; four tasks probe textual cloze, visual cloze, visual coherency, and visual ordering.Source: §2 · p. 3

2.1 Data Collection

• Recipes came from Instructables; collection favored popular recipes as a quality heuristic, then removed non-English and unreadable items and normalized descriptions.Source: §2.1 · p. 3

2.2 Questions and Answers

• RecipeQA automatically generates fixed four-choice questions from structured step data, testing instruction semantics across task-specific modalities without relying on costly crowdsourcing.Source: §2.2 · p. 3
• Generation excludes recipes with fewer than 3 or more than 25 steps, omits the narrative first step, strips explicit step numbers, and uses sampled steps to create modality-based choices with remaining content as context.Distractors can be selected at random from steps in other recipes.Source: §2.2 · p. 3

2.3 Dataset Statistics

• RecipeQA contains approximately 20K recipes across 22 food categories and over 36K question-answer pairs across four question types, with non-overlapping training, validation, and test sets.Source: §2.3 · p. 4

3 Tasks

• Four template-generated tasks—textual cloze, visual cloze, visual coherence, and visual ordering—evaluate relationships between recipe steps through titles, descriptions, and images.Fixed templates enable question-answer generation without human intervention.Source: §3 · p. 5

3.1 Textual Cloze

• Textual cloze hides a randomly selected step’s title and description and asks systems to identify the missing text using context that includes illustrative images.Distractor answers are selected using the WMD-based strategy described in Section 2.2.Source: §3.1 · p. 5

3.2 Visual Cloze

• Visual cloze hides a randomly selected step image and asks systems to infer it from a context of step titles and descriptions.Distractor images are selected using Euclidean distances between 2048-d pool5 features from an ImageNet-pretrained ResNet-50.Source: §3.2 · p. 5

3.3 Visual Coherence

• Visual coherence asks systems to identify a distractor image replacing one representative step image, using recipe titles and descriptions as context.The task requires relating candidate steps and aligning modalities across the context and answers.Source: §3.3 · p. 5

3.4 Visual Ordering

• Visual ordering asks systems to arrange a jumbled set of recipe-step images into a sequence that matches the temporal relations described by the recipe.The context contains step titles and descriptions, and answer choices are random permutations of the recipe’s illustrative images.Source: §3.4 · pp. 5–6

4.1 Data Preparation

• Textual modalities used 100-d Doc2Vec word and document embeddings; ResNet-50 visual features were reduced from 2048-d to 100-d.The authors inspected nearest neighbors to assess semantic usefulness and detected ingredients with Salvador et al. (2017), converting compounds such as “olive oil” into hyphenated single-word ingredients.Source: §4.1 · p. 7

4.2 Baseline Models

• The neural baseline adapted the Impatient Reader with a three-layer unidirectional LSTM, feeding recipe steps and, for multimodal input, images in recipe order.A modular design adjusted queries by question type and used cosine similarity with hinge ranking loss; training used ADAM at 1e-3, margin 1.5, and early stopping with patience 10, without hyperparameter tuning.Source: §4.2 · p. 7
• The Hasty Student simple baseline ignored recipe context and ranked candidate answers using similarities or dissimilarities between question elements and answer candidates.Its task-specific procedures used text or visual features for cloze, average dissimilarity for visual coherence, and consecutive-image distances for visual ordering.Source: §4.2 · pp. 7–8

4.3 Baseline Results

• Multimodal input helped textual cloze answers; Impatient Reader outperformed Hasty Student on cloze, while Hasty Student performed better on visual coherence and ordering.Table 2 reports test-set results as the ratio of correct answers to total questions; the passage provides qualitative comparisons but no specific values.Source: Table 2 · p. 8, §4.3 · p. 8

5 Related Work

• Related multimodal datasets address comic narratives and visual-cloze tasks, movie-story questions, and middle-school science lessons using images and text (Iyyer et al., 2017; Tapaswi et al., 2016).COMICS contains 1.2M panels and 2.5M textboxes from 4K comic books and evaluates text cloze, visual cloze, and character coherence; MovieQA has 15K questions about 408 movies; TQA has 26K questions about 1K science lessons and mostly diagrams, with limited reasoning demands.Source: §5 · pp. 8–9
• RecipeQA uses natural images taken by anonymous users in unconstrained environments, unlike COMICS and TQA.Source: §5 · p. 9
• Prior cooking-recipe research includes recipe parsing, instructional-text/video alignment, recipe text generation, cross-modal embeddings, and tracking entities and action transformations.Examples cited are Malmaud et al. (2014), Jermsurawong and Habash (2015), Malmaud et al. (2015), Sener et al. (2015), Kiddon et al. (2016), Salvador et al. (2017), and Bosselut et al. (2018).Source: §5 · p. 9
• To the authors’ best knowledge, RecipeQA is the first dataset for multimodal comprehension of how-to instructions or recipes, aligning arbitrary numbers of steps with multiple images and sentences.Source: §5 · p. 9

• RecipeQA contains roughly 20K cooking recipes and over 36K context-question-answer triplets for multimodal comprehension of cooking recipes.Source: §6 · p. 9
• RecipeQA is, to the authors’ knowledge, the first machine-comprehension dataset for procedural knowledge in a multimodal setting.Its four question styles target particular skills and require connecting information across modalities.Source: §6 · p. 9
• Baseline results demonstrate RecipeQA is challenging, and its public release aims to promote new multimodal machine-comprehension methods.The authors intend to extend it with crowdsourced natural-language question-answer pairs and hope it will support related cooking-recipe research.Source: §6 · p. 9