Source-linked AI summary
How to make a pizza: Learning a compositional layer-based GAN model
Dim P. Papadopoulos, Youssef Tamaazousti, Ferda Ofli, Ingmar Weber, Antonio Torralba
TL;DR
PizzaGAN addresses the challenge of reconstructing recipe steps and handling occlusion in layered foods. It learns composable GAN operators to add or remove ingredients from weakly labeled images, and experiments on synthetic and real pizzas demonstrate topping segmentation, inpainting, and ordering inference without depth-order supervision.
Problem
Reconstructing recipe steps and handling partially occluded ingredients remain challenges for mapping recipes to images, especially for foods composed of layers.
Source: §1 · p. 1Method
PizzaGAN learns add/remove GAN modules from image-level labels to generate ingredient layers or reveal occluded appearance, then applies removers sequentially.
Source: §1 · pp. 1–2Results
Experiments on synthetic and real pizzas demonstrate weakly supervised topping segmentation, inpainting of occluded areas, and ordering inference without depth-order supervision.
Source: §1 · p. 2Takeaways & Limitations
Overlaps between generated removal masks provide a cue for inferring which ingredient lies above another without supervision.
Source: §3.3 · p. 5The model was evaluated only on pizza; its application to other layered foods or domains remains to be explored.
Source: §7 · p. 8
Abstract
from arXiv · showhide
A food recipe is an ordered set of instructions for preparing a particular dish. From a visual perspective, every instruction step can be seen as a way to change the visual appearance of the dish by adding extra objects (e.g., adding an ingredient) or changing the appearance of the existing ones (e.g., cooking the dish). In this paper, we aim to teach a machine how to make a pizza by building a generative model that mirrors this step-by-step procedure. To do so, we learn composable module operations which are able to either add or remove a particular ingredient. Each operator is designed as a Generative Adversarial Network (GAN). Given only weak image-level supervision, the operators are trained to generate a visual layer that needs to be added to or removed from the existing image. The proposed model is able to decompose an image into an ordered sequence of layers by applying sequentially in the right order the corresponding removing modules. Experimental results on synthetic and real pizza images demonstrate that our proposed model is able to: (1) segment pizza toppings in a weaklysupervised fashion, (2) remove them by revealing what is occluded underneath them (i.e., inpainting), and (3) infer the ordering of the toppings without any depth ordering supervision. Code, data, and models are available online.
1. Introduction
PizzaGAN models pizza preparation as ordered ingredient-layer operations, addressing recipe-step reconstruction and occlusion in layered foods. Its add/remove GAN modules use image-level labels to decompose pizza images, segment toppings, inpaint hidden areas, and infer topping order without depth-order supervision.
1. Introduction
- Layered pizza images encode recipe order: adding ingredients creates overlaps, while reverse-order removal decomposes an image and must reveal what lies underneath.The introduction frames this as both ingredient-instance detection and recovery of the occluded appearance.Source: §1 · p. 1
- PizzaGAN learns add and remove operators for each object class from training images with image-level labels, rather than generating a complete replacement image.Adding modules generate a layer's appearance and visible-pixel mask; removing modules generate the occluded appearance and a mask for the removed layer.Source: §1 · pp. 1–2
- Experiments on synthetic and real pizzas show weakly supervised topping segmentation, inpainting of occluded areas, and ordering inference without depth-order supervision.The reported tasks require no pixel-wise supervision for segmentation.Source: §1 · p. 2
2. Related work
Related work spans conditional image translation, layered and amodal representations, residual-image generation, and modular GANs. PizzaGAN combines unpaired object-manipulation translations with composable, order-aware operations that generate layers and predict object ordering.
Generative Adversarial Networks (GANs)
- GANs pair a generator, which creates realistic samples, with a discriminator trained to distinguish generated samples from real ones.Source: §2 · p. 2
Image-to-image translation
- Conditional GANs support image-to-image translation, while CycleGAN [55] uses cycle consistency to avoid aligned training pairs and alleviate mode collapse.Earlier cGAN approaches include Isola et al. [23], which uses aligned training images.Source: §2 · p. 2
- PizzaGAN formulates object manipulation as unpaired image-to-image translation and composes modules for different operations, layered representations, and object-depth prediction.Its modules handle both segmentation and generation of pixels beneath the desired objects.Source: §2 · pp. 2–3
Image layers
Generating residual images
3. Method
PizzaGAN learns composable topping-addition and removal modules from image-level labels, then uses them to infer an ordered sequence of removals and decompose a test pizza into layers.
3. Method
- Training uses image-level topping labels and separates examples into images with and without each topping class.Source: §3 · p. 3
3.1. Architecture of modules
- Each topping class has adding and removing generators that composite a generated layer appearance and mask with the input image.The removal appearance represents parts occluded by the removed layer.Source: §3.1 · p. 3
3.2. Learning the model
- Joint training combines adversarial realism, classification, cycle consistency, and mask regularization losses.Classification preserves nontarget class labels, while mask regularization counters masks converging to zero.Source: §3.2 · pp. 3–4
3.3. Test time inference
- At test time, the discriminator identifies toppings, then pairwise mask-overlap comparisons infer removal order without supervision, resolving cycles by discarding the weakest pairwise prediction.The ordering with greater mask overlap is preferred for a pair of layers.Source: §3.3 · p. 5
4. Collecting pizzas
Section 4 describes creating synthetic pizzas and collecting annotated real pizza images. Synthetic data provides ground truth for evaluating ordering and multilayer segmentation, while real images receive crowdsourced topping labels.
4.1. Creating synthetic pizzas
- Synthetic pizzas provide annotation-free examples at arbitrary scale, plus ground-truth ordering and multilayer segmentation for quantitative evaluation.A pixel may carry multiple object labels when toppings overlap.Source: §4.1 · p. 5
- Background textures, plain pizza images, and topping clip art form varied synthetic pizzas with diverse topping configurations.The varied configurations include differences in topping number, quantity, position, and layer order.Source: §4.1 · pp. 5–6, Figure 6 · p. 6
4.2. Collecting real pizzas
- A CNN filtered half a million Instagram downloads; five AMT annotators labeled visible toppings per image, with majority vote determining final labels.Annotators first completed a qualification test using five simple pizza images.Source: §4.2 · p. 6
- The real dataset contains 9,213 annotated pizza images, averaging 2.9 toppings including cheese per pizza (standard deviation 1.1).Figure 7 shows the distribution of labeled toppings.Source: §4.2 · p. 6, Figure 7 · p. 6
- Figure 8 illustrates sequential predicted removals: each step removes the current top ingredient and reveals occluded ingredient parts.Source: Figure 8 · p. 6
5. Implementation Details
The implementation uses GAN modules for image manipulation, with a mask-producing generator and a multi-label PatchGAN discriminator. Training uses Adam, a staged learning-rate schedule, and specified loss weights.
5. Implementation Details
- Each generator adds a mask output alongside its image output; a modified PatchGAN discriminator also performs multi-label classification.The generator and discriminator architectures build on CycleGAN [55] and PatchGAN [23] [55], respectively.Source: §5 · p. 6
- Training uses Adam at 0.0002 for 100 epochs, then linearly decays the learning rate to zero over 100 more epochs.All modules train from scratch; λcls = 1, λcyc = 10, and λreg = 0.01, and real images are cropped and resized to 256×256.Source: §5 · p. 6
6. Experimental results
On synthetic pizzas, sequential removal improves segmentation over parallel removal and outperforms weakly supervised baselines; on real pizzas, predicted-order segmentation also exceeds CAMs with dense CRF. The experiments additionally assess classification, ordering, and cooking operations.
6.1. Results on synthetic pizzas
- Synthetic-pizza classification reaches 99.9% mAP versus 99.3% for a ResNet18 reference, and the average normalized ordering distance is 0.33.For images with exactly two toppings, predicted ordering is correct 88% of the time.Source: §6.1 · p. 7
- On synthetic pizzas, sequential removal adds +2% mIoU over parallel removal, and PizzaGAN outperforms AffinityNet by ∼10%.CAMs and AffinityNet with denseCRFs reach 51.5% mIoU, below PizzaGAN; comparison methods are pretrained on ILSVRC while PizzaGAN is trained from scratch.Source: §6.1 · p. 7, Table 1 · p. 7
- On synthetic pizzas, predicted ordering raises occluded-region mIoU from 0% to 18.2%, while non-occluded-region performance is 70.5%.Without ordering prediction, non-occluded-region mIoU is 70.4%.Source: §6.1 · pp. 7–8
6.2. Experiments on real pizzas
- On real pizzas, classification reaches 77.4% mAP, compared with 77.6% for a ResNet18 reference.The real-pizza experiments use 9,213 images across 12 topping classes and assess segmentation on 50 manually annotated images.Source: §6.2 · p. 8
- On real pizzas, predicted-order segmentation achieves 29.3% mIoU and outperforms CAMs with dense CRF by +6.6%.The real-pizza evaluation uses 50 manually annotated images; parallel removal achieves 28.2% mIoU.Source: §6.2 · p. 8
- Cooking and uncooking modules are trained using a manually labeled subset of 932 pizzas.Qualitative examples are shown alongside adding and removing operators.Source: Figure 9 · p. 7, §6.2 · p. 8
7. Conclusions
PizzaGAN mirrors pizza preparation with composable ingredient operations and is evaluated on synthetic and real pizza images. The experiments show topping segmentation, inpainting, and ordering inference without pixel-wise or depth-order supervision; evaluation is limited to pizza.
7. Conclusions
- On synthetic and real pizzas, PizzaGAN segments toppings without pixel-wise supervision, inpaints occluded regions, and infers topping order without depth-order supervision.Source: §7 · p. 8
- PizzaGAN models pizza-making with composable GAN operations for adding, removing, cooking, or uncooking ingredients, formulated as sequential unpaired image-to-image translations.Source: §7 · p. 8
- The model was evaluated only on pizza, while the authors identify other layered foods and digital fashion as potential future domains.Source: §7 · p. 8