Source-linked AI summary
CookGAN: Meal Image Synthesis from Ingredients
Fangda Han, Ricardo Guerrero, Vladimir Pavlovic
TL;DR
Meal images vary with ingredients and cooking, making photo-realistic generation from ingredient lists a challenging text-to-image task. CookGAN combines attention-based ingredient-image association with a generative model and cycle consistency; experiments report natural-looking meal images and improvements from cycle consistency on most subsets.
Problem
Meal images have variable appearances shaped by ingredients and cooking, while prior text-to-image work mainly targets well-structured object categories.
Source: §1 · p. 1Method
Results
CookGAN synthesizes natural-looking meal images, and cycle consistency improves IS and FID on most subsets and outperforms StackGAN-v2 [27] on MedR for most subsets.
Source: §4.3 · p. 7, §5 · p. 8Takeaways & Limitations
Processing ingredients into a canonical vocabulary is a critical step in the synthesis process.
Source: §5 · p. 8The ingredient encoder assumes a chain graph can approximate arbitrary ingredient dependencies within a recipe.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
In this work we propose a new computational framework, based on generative deep models, for synthesis of photo-realistic food meal images from textual list of its ingredients. Previous works on synthesis of images from text typically rely on pre-trained text models to extract text features, followed by generative neural networks (GAN) aimed to generate realistic images conditioned on the text features. These works mainly focus on generating spatially compact and well-defined categories of objects, such as birds or flowers, but meal images are significantly more complex, consisting of multiple ingredients whose appearance and spatial qualities are further modified by cooking methods. To generate real-like meal images from ingredients, we propose Cook Generative Adversarial Networks (CookGAN), CookGAN first builds an attention-based ingredients-image association model, which is then used to condition a generative neural network tasked with synthesizing meal images. Furthermore, a cycle-consistent constraint is added to further improve image quality and control appearance. Experiments show our model is able to generate meal images corresponding to the ingredients.
1. Introduction
Computational food analysis connects meal images and text with food-related information for applications including nutrition and meal planning. This work tackles photo-realistic meal-image generation from ingredients with CookGAN, combining attention-based association and cycle consistency.
1. Introduction
- Computational food analysis uses food information from text and meal images for applications including nutrient monitoring, meal preferences, and meal planning.The paper describes extracting ingredients and calories from images as one example.Source: §1 · p. 1
- Generating photo-realistic meals from ingredients is challenging because appearance varies with cooking methods, preparation styles, and visual dissolution.The model must infer these factors implicitly.Source: §1 · p. 1
- CookGAN generates photo-realistic meal images conditioned on ingredient lists by combining attention-based recipe association with StackGAN [27].The paper also proposes cycle consistency to improve image quality and control appearance as ingredients change.Source: §1 · p. 1
2. Related Work
Prior text-to-image work extracts text features to condition GANs, while meal-image generation faces variable appearances and ingredient roles. CookGAN builds on attention-based recipe association and adds cycle consistency to reduce semantic discrepancy and improve image quality.
2. Related Work
- GANs model real-image distributions using generator and discriminator networks, and text-conditioned systems use extracted text features to guide generation [7, 20, 28, 27].StackGAN [27] generates images at multiple resolutions by stacking three GANs.Source: §2 · pp. 1–2
- Because ingredient lists are sparse and ingredients differ in visual importance, an attention mechanism with a bidirectional LSTM helps identify key ingredients.Earlier text-conditioned systems use RNNs that process words sequentially and with equal importance.Source: §2 · p. 2
- Prior meal-image generators either produce low-resolution images for retrieval regularization or are tested only on pizza with predefined procedures and ingredients [19].CookGAN instead aims to generate meal images across food types and ingredients.Source: §2 · p. 2
- CookGAN adds cycle consistency to related approaches [25, 5], minimizing semantic discrepancy between fake and real images and improving image quality qualitatively and quantitatively.The intuition is that a high-quality generated image capturing the ingredients should yield features similar to those of a real image.Source: §2 · p. 2
3. Methodology
CookGAN learns an ingredient–image association and uses the ingredient representation to condition a multiscale GAN. Its methodology adds ingredient attention and a cycle-consistency constraint.
3.1. Attention-based Association Model
- The association model maps ingredient lists and matched or mismatched meal images into FoodSpace, maximizing positive-pair and minimizing negative-pair association.Source: §3.1 · p. 2
- Word2vec ingredient vectors feed a bidirectional LSTM whose attended hidden states form the ingredient representation; the authors assume a chain graph can approximate within-recipe dependencies.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Generative Meal Image Network
- A fixed ingredient encoder feeds conditional augmentation to sample ingredient-conditioned appearance factors; three generator branches use Gaussian noise and prior-branch features to produce low-, medium-, and high-resolution images.Source: §3.2 · p. 3
- At each scale, discriminators classify correctly paired real images as real and wrongly paired or generated images as fake; generator and discriminator branches train jointly with an unconditional image loss.Source: §3.2 · pp. 3–4
3.3. Cycle-consistency constraint
- Cycle consistency maximizes FoodSpace cosine similarity between real and corresponding generated images at multiple scales, with its loss weight cross-validated to 1.0.Source: §3.3 · p. 4
4. Experiments
The experiments evaluate canonical ingredients and attention for recipe-image association, then test CookGAN’s meal synthesis and the distinct roles of ingredient features and random input. Across these evaluations, cycle consistency improves reported generation and retrieval results on most subsets, while canonical ingredients consistently improve recipe retrieval from synthesized images.
4. Experiments
- Using Recipe1M, the study focuses on 402,760 recipes with at least one image, no more than 20 ingredients or instructions, and at least one ingredient and instruction.The data are split 70%/15%/15% into train/validation/test, using at most five images per recipe. The final list has 1,989 canonical ingredients, covering more than 95% of recipes.Source: §4 · p. 4
4.1. Effect of Canonical Ingredients
- Canonical ingredients greatly improve association-model performance on 5K samples, and retrieved images generally share the query’s food type.The correct image does not always rank first among the retrieved results.Source: §4.1 · p. 5, Table 1 · p. 5, Figure 3 · p. 5
4.2. Effect of Attention
- Attention does not improve retrieval scores, but its maps focus on visually prominent ingredients rather than frequent ingredients such as olive oil, water, or butter.The authors suggest the RNN may learn ingredient importance implicitly; attention can still locate important ingredients without supervision.Source: §4.2 · p. 5, Figure 4 · p. 6
4.3. Meal Image Generation
- Meal generation is tested on salad, cookie, and muffin subsets, with 17,209/3,784, 9,546/2,063, and 4,312/900 train/test samples, respectively.The study narrows categories to reduce preparation-method diversity; IS and FID are used to evaluate synthesized images.Source: §4.3 · p. 6
- Cycle consistency improves IS and FID on most subsets; for recipe retrieval using synthesized images as queries, canonical ingredients always improve MedR, and cycle consistency beats StackGAN-v2 [27] on most subsets.Table 2 compares salad, cookie, and muffin subsets; retrieval uses a range of 900.Source: Table 2 · p. 6, §4.3 · p. 7
- Compared with StackGAN-v2, CookGAN images with cycle consistency usually show clearer ingredient appearance and look more photo-realistic.Figure 5 shows generated examples alongside target ingredients and real images for reference.Source: §4.3 · p. 7, Figure 5 · p. 7
4.4. Components Analysis
- With random vector z fixed, changing ingredient features c changes the generated ingredients; with c fixed, changing z changes viewpoints while ingredients remain shared.These examples distinguish ingredient-conditioned appearance from variation associated with the random vector.Source: §4.4 · p. 8
- During feature-space interpolation between recipes with at least 70% ingredient overlap, the model gradually removes the target ingredient when the other recipe lacks it.The examples use tomato for salad and blueberry for muffin.Source: §4.4 · p. 8, Figure 8 · p. 8
5. Conclusion
The paper develops a model for generating photo-realistic meal images from ingredient sets and reports natural-looking images corresponding to desired ingredients. It identifies canonical ingredient vocabulary as critical and outlines richer recipe context and ingredient quantities as future directions.
5. Conclusion
- The model integrates an attention-based recipe association model with StackGAN-v2 and adds ingredient cycle-consistency regularization in a shared space.The association model aims to place ingredient features close to real meal images in FoodSpace, while StackGAN-v2 attempts to reproduce the image class from that encoding.Source: §5 · p. 8
- Processing ingredients into a canonical vocabulary is identified as a critical step in the synthesis process.Source: §5 · p. 8
- Experiments demonstrate natural-looking meal images corresponding to desired ingredients, visually and quantitatively through retrieval metrics.Source: §5 · p. 8
- Future work aims to add recipe instructions and titles for context and ingredient amounts for synthesizing images with arbitrary quantities.Source: §5 · p. 8