Source-linked AI summary
Inverse Cooking: Recipe Generation from Food Images
Amaia Salvador, Michal Drozdzal, Xavier Giro-i-Nieto, Adriana Romero
TL;DR
Food images do not reveal the ingredients and preparation process needed to reconstruct a recipe. The paper addresses this with unordered ingredient-set prediction followed by instruction generation conditioned on both the image and inferred ingredients, achieving improved ingredient prediction, high-quality recipes, and stronger human judgments than retrieval-based approaches.
Problem
Food images provide limited access to the ingredients and preparation process, motivating systems that infer both from prepared meals.
Method
The system predicts ingredient sets while modeling dependencies without imposing order, then generates recipe titles and instructions from image and ingredient representations.
Results
On Recipe1M, the system improves ingredient prediction, produces high-quality recipes using image and ingredient information, and receives more compelling human judgments than retrieval-based approaches.
Takeaways & Limitations
Inverse cooking can generate structured recipes directly from food images while combining visual evidence with inferred ingredients during instruction generation.
Takeaways & Limitations
Generated recipes can contain ingredient-recognition errors, ingredient–instruction inconsistencies, and repeated ingredient mentions.
Abstract
from arXiv · showhide
People enjoy food photography because they appreciate food. Behind each meal there is a story described in a complex recipe and, unfortunately, by simply looking at a food image we do not have access to its preparation process. Therefore, in this paper we introduce an inverse cooking system that recreates cooking recipes given food images. Our system predicts ingredients as sets by means of a novel architecture, modeling their dependencies without imposing any order, and then generates cooking instructions by attending to both image and its inferred ingredients simultaneously. We extensively evaluate the whole system on the large-scale Recipe1M dataset and show that (1) we improve performance w.r.t. previous baselines for ingredient prediction; (2) we are able to obtain high quality recipes by leveraging both image and ingredients; (3) our system is able to produce more compelling recipes than retrieval-based approaches according to human judgment. We make code and models publicly available.
1. Introduction
The paper introduces inverse cooking: generating a recipe from a food image by predicting unordered ingredients and then generating instructions conditioned on both the image and ingredients. It addresses limitations of retrieval-based image-to-recipe systems and evaluates the approach on Recipe1M.
- Motivation: Inverse cooking infers ingredients and cooking instructions from a prepared meal because an image alone makes it difficult to know precisely what was eaten.The paper motivates a system that reconstructs preparation information from food images.
- Approach: The system generates a recipe containing a title, ingredients, and cooking instructions directly from an image.It first predicts ingredients, then conditions instruction generation on both the image and inferred ingredients.
- Approach: Ingredients are formulated as a set-prediction problem that models dependencies without penalizing prediction order.This design treats ingredient order as irrelevant while exploiting their underlying structure.
- Approach: Instruction generation is posed as sequence generation conditioned simultaneously on an image and its predicted ingredients.The architecture explores attention strategies for reasoning about both modalities.
- Evaluation: The paper evaluates the complete system on the large-scale Recipe1M dataset and compares it with prior ingredient-prediction and image-to-recipe retrieval approaches.The reported evaluation includes ingredient prediction and a user study of generated recipes.
2. Related Work
Related work spans visual food recognition, multi-label ingredient prediction, and conditional text generation. Prior approaches include classification and retrieval, while contemporary generation research provides sequence and attention-based modeling foundations.
- Food Understanding: Food understanding research uses large-scale datasets such as Food-101 and Recipe1M to benchmark image classification and other food-related tasks.The literature also addresses more challenging problems, including calorie estimation from food images.
- Multi-label classification: Multi-label classification methods model ingredient labels with neural architectures and specialized losses, but binary logistic approaches assume label independence.Label powersets capture dependencies but become intractable as the number of labels grows.
- Conditional text generation: Conditional text generation research includes recurrent, convolutional, and attention-based sequence models for text- and image-conditioned generation.These approaches have been applied to translation, image captioning, poetry, and story generation.
3. Generating recipes from images
The system generates cooking instructions from image features and inferred ingredients, while comparing ordered-list and unordered-set ingredient representations. Its set transformer models ingredient dependencies without penalizing prediction order, then trains the recipe transformer using image and ingredient information.
- Recipe generation: The recipe pipeline predicts ingredients from an image and generates cooking instructions conditioned on both visual features and the inferred ingredients.The instruction decoder receives an image embedding and an ingredient embedding before producing the instruction sequence.
- Recipe generation: The instruction decoder explores concatenated, independent, and sequential attention strategies for combining image and ingredient embeddings.Sequential attention is evaluated in both image-first and ingredients-first orderings.
- Ingredient representation: Ingredients are modeled both as ordered lists and as unordered sets because ingredient order may be irrelevant to the cooking outcome.The list formulation uses autoregressive conditionals, whereas the set formulation represents ingredients as a binary vector.
- Ingredient representation: The set transformer captures dependencies among ingredients with an autoregressive model while removing order penalties by pooling predictions across time.It uses max pooling, suppresses previously selected ingredients, and adds eos and cardinality losses during training.
- Ingredient representation: A feed-forward target-distribution model provides an alternative set representation by learning a joint distribution and greedily sampling ingredients until a probability threshold is reached.The model is trained with target-distribution cross-entropy and reconstructs a variable-cardinality set from sorted output probabilities.
- Training: Training proceeds in two stages: pretraining the image encoder and ingredient decoder, followed by training the ingredient encoder and instruction decoder.During recipe training, ground-truth ingredients are supplied to the instruction decoder and transformer models use teacher forcing.
4. Experiments
Experiments on Recipe1M evaluate ingredient-set prediction, multimodal recipe generation, retrieval comparisons, and human judgments. Modeling ingredient dependencies and conditioning instructions on both images and ingredients consistently improves the reported results.
- Experimental setup: The experiments use Recipe1M and compare ingredient prediction, instruction generation, retrieval-based systems, and human judgments.The dataset and evaluation include model selection, ingredient cardinality, test comparisons, and two user studies.
- Recipe generation: Concatenated attention achieves the best validation recipe perplexity among the compared multimodal attention strategies.Independent attention performs worst, while sequential attention lies between it and concatenated attention.
- Recipe generation: 8.51 test perplexity improves on both I2R and L2R baselines when instructions use image and ingredient information together.L2R reaches 8.67 versus 9.66 for I2R, indicating the value of conditioning on inferred ingredients.
- Ingredient prediction: TFset outperforms previous ingredient-prediction methods on F1 and IoU while modeling ingredient co-occurrences without imposing an ingredient order.Dependency-aware models improve ingredient scores, and TFset is third-best for cardinality error.
- Generation vs retrieval: 12.26 IoU points and 15.48 F1 score points separate TFset from the RI2LR retrieval baseline on ingredient prediction.Both proposed ingredient-prediction methods outperform the retrieval baselines by a large margin.
- User studies: 49.08% F1 gives the proposed method higher ingredient scores than humans at 35.20% and retrieval at 30.55%.A separate study also finds generated recipes more successful than retrieved recipes according to human judgments.
5. Conclusion
The paper presents an image-to-recipe system that predicts ingredient sets and generates instructions using both images and inferred ingredients. Its experiments show that dependency modeling matters and that the system outperforms retrieval approaches in user studies.
- Conclusion: The system converts food images into recipes containing titles, ingredients, and cooking instructions.The pipeline first predicts ingredient sets and then generates instructions.
- Conclusion: Modeling ingredient dependencies improves set prediction, while instruction generation benefits from reasoning over images and inferred ingredients together.The conclusion identifies both design choices as central findings.
- Conclusion: User studies demonstrate superiority over state-of-the-art image-to-recipe retrieval approaches.The conclusion also emphasizes the difficulty of the inverse-cooking task.
7. Supplementary Material
The supplementary material provides implementation details, vocabulary analysis, recipe examples, user-study interfaces, and qualitative generalization results. It also documents plausible outputs alongside recurring recognition, consistency, and repetition errors.
- Implementation and training: Recipe generation uses specified batch sizes, learning rates, frozen image encoders, Adam optimization, decay, dropout, and image augmentation.Instruction generation uses batch size 256 and learning rate 0.001, with the image encoder frozen during training.
- Generated recipes: Generated recipes are generally plausible and structured, but errors include ingredient confusion, ingredient-instruction inconsistencies, and repeated enumerations.Examples include confusing stuffed tomatoes with stuffed peppers, predicting unused cucumber, and repeating tomato paste.
- User studies: The supplementary material documents interfaces for ingredient selection, freeform recipe writing, and human assessment of image-recipe matching.The forms support both data collection and evaluation in the user studies.
- Human-written recipes: Human-written recipes average 5.29 instructions of 9.03 words each, while generated recipes average 9.21 instructions of 9 words each.The generated instruction count closely matches the real average of 9.08 sentences, whose average sentence length is 12.79 words.
- Out-of-dataset images: Recipes were also generated for food images photographed by the authors at home or in restaurants to test out-of-dataset generalization.These examples are presented as a qualitative generalization experiment.
- Ingredient vocabulary: Ingredient preprocessing changes vocabulary frequencies and clusters related ingredients, including oils and cheese varieties.The original vocabulary contains many narrowly defined ingredients; preprocessing merges or clusters overlapping terms.