Source-linked AI summary
Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, Antonio Torralba
TL;DR
Food understanding is limited by the scarcity of large, structured datasets linking recipes and images. The paper introduces Recipe1M+ and learns a semantically regularized joint embedding for image-recipe retrieval, achieving strong performance and supporting semantic vector arithmetic.
Problem
Existing food datasets were limited in scale, while comprehensive food understanding requires jointly analyzing images, ingredients, and cooking instructions.
Method
The paper builds Recipe1M+ and trains a multimodal neural model that embeds recipe ingredients, instructions, and images into a common space with semantic regularization.
Results
The embeddings achieve impressive image-recipe retrieval results, improving over baselines and performing comparably to humans.
Takeaways & Limitations
The dataset and embeddings support further exploration of food representations, including semantic vector arithmetic and potential recipe modification.
Takeaways & Limitations
Ingredient-level retrieval can miss ingredients that are not visible in the query image or are confused because of insufficient fine-grained visual features.
Abstract
from arXiv · showhide
In this paper, we introduce Recipe1M+, a new large-scale, structured corpus of over one million cooking recipes and 13 million food images. As the largest publicly available collection of recipe data, Recipe1M+ affords the ability to train high-capacity modelson aligned, multimodal data. Using these data, we train a neural network to learn a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Moreover, we demonstrate that regularization via the addition of a high-level classification objective both improves retrieval performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M+ dataset and food and cooking in general. Code, data and models are publicly available.
1 INTRODUCTION
Food understanding requires large, structured multimodal data linking images with ingredients and cooking instructions. The paper introduces Recipe1M+ and a jointly learned, semantically regularized embedding for image-recipe retrieval.
- Motivation: Food offers computer-vision challenges involving robust representations under occlusion, deformation, and ingredient-state variation.
- Motivation: Online recipe collections with user-submitted photos enable joint analysis of ingredients, instructions, and food images, but automated analysis requires large, well-curated datasets.
- Limitations of Existing Data: Medium-scale food datasets have limited categorization performance, with Food-101 improving from 50.8% to 77.4%, 79%, and 80.9% in subsequent work.
- Contributions: Recipe1M+ contains one million structured recipes and images, supporting the im2recipe task of retrieving a recipe for a dish visible but not necessarily describable.
- Contributions: A multimodal neural model embeds images and recipes in a common space with high-level classification regularization, outperforming baselines and performing comparably to humans.
2 DATASET
Recipe1M+ expands prior food datasets into a large structured corpus by combining recipe-site images with web image-search results. Cleaning, consolidation, and nutritional processing produce multimodal data with broad recipe and image coverage.
- Dataset Motivation: Existing food datasets remained limited in generality or size, motivating a publicly released corpus with over 1M recipes and 13M images.
- Collection: Recipe1M+ was built by collecting over 1M recipes and 800K website images, then augmenting them with web image-search results to exceed 13M images.
- Collection: Recipes were scraped from over two dozen cooking websites, extracted from raw HTML, normalized, deduplicated, and stored in a uniquely identified JSON schema.
- Image Extension: Image-search augmentation used recipe titles as queries, selected the top 50 results, and consolidated website and search images by removing corrupted files and duplicates.
- Nutritional Information: Nutritional processing isolated measurable ingredient units and quantities; 103,152 recipes had measurable units and numerical quantities for all ingredients.
- Dataset Analysis: After extension, only around 2% of recipes lacked associated images, compared with approximately half before extension.
- Dataset Analysis: Human workers selected original and queried images with similar frequencies, supporting the conclusion that the extended dataset was not much noisier than Recipe1M.
3 LEARNING EMBEDDINGS
The model learns a common recipe-image embedding from paired multimodal data. Recipes are represented through separate ingredient and instruction encoders, while images use pretrained convolutional representations.
- Joint Embedding: The joint embedding model uses paired recipe-image data to learn a common embedding space.
- Recipe Representation: Recipe representation covers ingredients and cooking instructions as distinct components.
- Recipe Representation: Ingredient names are extracted from ingredient text and represented with ingredient-level word2vec vectors, using a bidirectional LSTM extractor with 99.5% held-out accuracy.
- Image Representation: The image representation adopts VGG-16 and ResNet-50 convolutional networks.
4 JOINT NEURAL EMBEDDING
The joint neural embedding maps recipe and image representations into a shared space, training matched pairs to be close and non-matching pairs to be separated by a margin.
- 4 JOINT NEURAL EMBEDDING: Two recipe encoders represent ingredients and instructions before concatenating them into a recipe-level representation.Ingredients use a bidirectional LSTM, while instructions use a forward LSTM over skip-instructions vectors.
- 4 JOINT NEURAL EMBEDDING: The image representation is projected linearly into the same recipe-image embedding space.Both recipe and image representations are mapped into the joint space through learned transformations.
- 4 JOINT NEURAL EMBEDDING: The model maximizes cosine similarity for positive recipe-image pairs and minimizes it for non-matching pairs up to a specified margin.Training uses positive and negative pairs with a cosine similarity loss.
5 SEMANTIC REGULARIZATION
Semantic regularization adds a shared high-level classification objective to the joint embedding, aligning recipe and image representations through common semantic categories.
- 5 SEMANTIC REGULARIZATION: Semantic regularization trains recipe and image embeddings to solve the same high-level classification problem using shared weights.The classification objective is optimized together with the joint embedding loss.
- 5 SEMANTIC REGULARIZATION: The semantic categories combine Food-101 labels with recipe-title bigrams, producing 1,047 categories covering 50% of the dataset.Food-101 labels alone annotate only 13% of the dataset.
- 5 SEMANTIC REGULARIZATION: A shared fully connected classification layer maps recipe and image embeddings to semantic category probabilities.The same learned weight matrix is applied to both modalities before softmax activation.
- 5 SEMANTIC REGULARIZATION: The final objective combines cosine embedding loss with semantic regularization through the weighted sum L(φr, φv, cr, cv, y) = Lcos(φr, φv, y) + λLreg(φr, φv, cr, cv).The regularization weight is set to λ = 0.02 in all experiments.
- 5 SEMANTIC REGULARIZATION: Joint optimization uses a two-stage procedure because simultaneous updates of recipe and image networks become oscillatory and can diverge.The procedure first fixes image-network weights before later joint fine-tuning.
6 EXPERIMENTS
The experiments evaluate joint recipe-image embeddings for bidirectional retrieval, compare them with baselines and humans, and examine how dataset scale and semantic regularization affect performance. They also analyze whether the learned spaces support semantic arithmetic and interpolation, while identifying ingredient-level retrieval failures.
- Retrieval performance: The joint neural embedding models consistently outperform all baseline methods on Recipe1M retrieval.The evaluation reports median rank and recall at top K for im2recipe and recipe2im retrieval.
- Retrieval performance: 65% of evaluated queries retrieve the correct recipe from a food image, while semantic regularization lowers im2recipe medR from 7.2 to 5.2.The model also outperforms CCA baselines in all reported measures, with similar results in recipe2im retrieval.
- Human comparison: Semantic regularization surpasses human accuracy by 3.2 percentage points on all recipes and by 6.8 percentage points on course-specific tests.Accuracies decrease as the evaluation becomes more specific; dish-specific results show similar behavior, with humans better for some beverages and smoothies.
- Dataset comparison: Training on Recipe1M+ improves retrieval on the Recipe1M+ test set, including a 5-point lower medR in both tasks and increases in R@5 and R@10 of up to 10 percentage points.The two models perform similarly on the Recipe1M test set, while Recipe1M+ consistently outperforms Recipe1M in category-based evaluation.
- Embedding analysis: Embedding arithmetic and interpolation produce semantically meaningful food concepts, although retrieval can miss ingredients absent from the image or fine-grained visual distinctions.Recipe1M+ more often captures higher-level analogies, and increasing a soup coefficient makes curry images less chunky and more broth-like.
7 CONCLUSION
The paper presents Recipe1M+ and semantically regularized neural embeddings, showing strong retrieval, generalizability, and exploratory vector arithmetic. These representations also suggest applications such as recipe modification and cross-modal recipe generation.
- Recipe1M+ is presented as the largest structured recipe dataset to date, alongside the im2recipe task and semantically regularized neural embedding models.
- The extended dataset is not much noisier because AMT experiments and equal Recipe1M test performance support its data quality.
- Recipe1M+ greatly helps performance on Food-101, demonstrating the dataset’s value for model generalizability.
- Vector arithmetic on the learned representations hints at applications including recipe modification and cross-modal recipe generation.
- The methods could also be applied to other recipe-like materials, including assembly instructions and tutorials.
- Fractional interpolation between concept embeddings retrieves results along combinations of concepts such as salad and pasta.