Source-linked AI summary
Learning Cross-Modal Embeddings with Adversarial Networks for Cooking Recipes and Food Images
Hao Wang, Doyen Sahoo, Chenghao Liu, Ee-peng Lim, Steven C. H. Hoi
TL;DR
Food-image retrieval can provide recipes and related information, but cross-modal retrieval must address challenges including varied images for one recipe and dissimilar modality distributions. ACME combines hard-sample mining, adversarial alignment, and cross-modal translation consistency, and achieves state-of-the-art cross-modal retrieval results on Recipe1M.
Problem
Cross-modal recipe–image retrieval must learn a shared space despite image variation and differing modality distributions, while preserving relevant information across modalities.
Source: §3.1 · p. 3Method
Results
ACME achieves state-of-the-art cross-modal retrieval results on Recipe1M.
Source: §5 · p. 8Takeaways & Limitations
Food retrieval can connect an image to its recipe, nutrition content, or calorie information.
Source: §1 · p. 1The authors do not claim robust translation: noisy food categories and difficult image generation and classification tasks constrain these components.
Source: §4.5.3 · p. 8
Abstract
from arXiv · showhide
Food computing is playing an increasingly important role in human daily life, and has found tremendous applications in guiding human behavior towards smart food consumption and healthy lifestyle. An important task under the food-computing umbrella is retrieval, which is particularly helpful for health related applications, where we are interested in retrieving important information about food (e.g., ingredients, nutrition, etc.). In this paper, we investigate an open research task of cross-modal retrieval between cooking recipes and food images, and propose a novel framework Adversarial Cross-Modal Embedding (ACME) to resolve the cross-modal retrieval task in food domains. Specifically, the goal is to learn a common embedding feature space between the two modalities, in which our approach consists of several novel ideas: (i) learning by using a new triplet loss scheme together with an effective sampling strategy, (ii) imposing modality alignment using an adversarial learning strategy, and (iii) imposing cross-modal translation consistency such that the embedding of one modality is able to recover some important information of corresponding instances in the other modality. ACME achieves the state-of-the-art performance on the benchmark Recipe1M dataset, validating the efficacy of the proposed technique.
1. Introduction
Food retrieval can connect a food image to recipes and information such as nutrition or calories, but recipe–image retrieval faces challenges from image variation, differing modality distributions, and potentially lost semantic information. ACME addresses these challenges with hard-sample mining, adversarial modality alignment, and cross-modal translation consistency, and is reported to outperform state-of-the-art baselines on Recipe1M.
1. Introduction
- Food retrieval aims to retrieve relevant information, such as a recipe, nutrition, or calorie information, for a specific food image.The paper studies cross-modal retrieval between cooking recipes and food images.Source: §1 · p. 1
- Prior recipe–image retrieval methods face high image variation, unaligned modality distributions, and uncertainty about whether embeddings retain the other modality’s relevant semantics.The text notes that high variation among images for one recipe can hinder convergence of triplet loss with naive sampling.Source: §1 · p. 2
- ACME combines hard-sample mining with an improved triplet loss, adversarial alignment of recipe and image embeddings, and cross-modal translation consistency.The translation objectives generate an image from a recipe embedding and predict ingredients from a food-image embedding.Source: §1 · p. 2
- ACME is end-to-end trainable and is reported to significantly outperform state-of-the-art cross-modal retrieval baselines on Recipe1M.Figure 1 presents an overview of the framework.Source: §1 · p. 2
2. Related Work
Food computing includes tasks such as food recognition, recommendation, and retrieval; retrieval can connect a food item to relevant information. Cross-modal retrieval addresses the media gap between modalities, and prior recipe–image work motivates ACME’s combination of improved sampling, alignment, and translation consistency.
2.1. Food Computing
- Food computing research includes food recognition and food recommendation, alongside other ways to analyze food.The passage describes recognition methods ranging from kernel-based models to deep CNNs and mentions chatbot-based and dietary recommendation efforts.Source: §2.1 · p. 2
- Food retrieval seeks relevant information for a food item, and cross-modal retrieval includes image-to-recipe and recipe-to-image tasks.Retrieving a recipe and ingredients from an image may simplify nutrition and calorie estimation.Source: §2.1 · p. 2
2.2. Cross-Modal Retrieval
- Cross-modal retrieval faces a media gap: features from different modalities can be inconsistent, complicating similarity measurement.Prior approaches include metric learning, global or local alignment, and adversarial loss.Source: §2.2 · p. 2
- For recipe–image retrieval, earlier work used pairwise cosine loss, triplet loss, or hierarchical attention, while ACME adds improved sampling, cross-modal alignment, and translation consistency.The passage identifies JE, AdaMine, and hierarchical attention as prior approaches.Source: §2.2 · pp. 2–3
3. Adversarial Cross-Modal Embedding
ACME learns a shared embedding for food images and recipes, addressing limitations of retrieval losses, modality distributions, and information loss across modalities. Its joint objective combines retrieval learning, modality alignment, and cross-modal translation consistency.
3.1. Overview
- The framework targets paired retrieval while allowing multiple images to correspond to one recipe, bringing those images close to that recipe's embedding.Source: §3.1 · p. 3
- ACME encodes images with a CNN and recipes with an LSTM, then uses a shared fully connected layer to produce their final representations.The joint objective combines retrieval, modality alignment, and translation-consistency losses, weighted by trade-off parameters.Source: §3.1 · p. 3
3.2. Cross-Modal Retrieval Learning
- Hard-sample mining prioritizes the most distant positive and closest negative examples when constructing triplets for retrieval learning.Each triplet uses an anchor in one modality and positive and negative examples from the other modality.Source: §3.2 · p. 4
3.3. Modality Alignment
- Adversarial modality alignment trains a discriminator not to distinguish image from recipe features, aligning their penultimate-layer distributions.The paper adopts WGAN-GP for this alignment objective.Source: §3.3 · p. 4
3.4. Translation Consistency
- Translation consistency trains recipe features to generate food images and visual features to predict recipe ingredients, aiming to preserve information across modalities.The paper motivates this component as supporting better generalization and semantic alignment.Source: §3.4 · pp. 4–5
3.4.1 Recipe2Image
- For recipe-to-image translation, a generator conditions image generation on recipe features, while a discriminator distinguishes real from generated images.A food-category classifier also encourages generated images to retain the recipe's corresponding food category.Source: §3.4.1 · p. 5
3.4.2 Image2Recipe
- For image-to-recipe translation, a multi-label classifier predicts ingredients from visual features, with a category-classification loss supporting translation consistency.Source: §3.4.2 · p. 5
4.1. Dataset and Evaluation Metrics
The evaluation uses Recipe1M, with retrieval measured on sampled test subsets using median retrieval rank and recall at top K.
4.1. Dataset and Evaluation Metrics
- Recipe1M contains over 1 million recipes, 800,000 food images, 1,047 food classes, and 4,102 frequently occurring unique ingredients.Source: §4.1 · p. 5
- The original data splits provide 238,999 image-recipe pairs for training, 51,119 for validation, and 51,303 for testing.Source: §4.1 · p. 5
- Evaluation samples ten test subsets each at 1k and 10k pairs, ranks cross-modal candidates by embedding L2 distance, and reports MedR and R@K.MedR is the median retrieval rank; R@K is the percentage of queries whose match appears among the top K results.Source: §4.1 · p. 5
4.2. Implementation Details
The implementation uses pretrained visual and language representations, with both modalities mapped to 1024-dimensional features.
4.2. Implementation Details
- The image encoder is an ImageNet-pretrained ResNet-50 that outputs a 1024-dimensional feature vector.Source: §4.2 · p. 5
- The recipe encoder uses a hierarchical LSTM for instructions and bidirectional embeddings for ingredients, concatenated into a 1024-dimensional vector.Instruction word embeddings use pretrained skip-thought representations, while ingredient embeddings use word2vec.Source: §4.2 · p. 5
- Recipe-to-image generation produces 128 × 128 × 3 images; training uses Adam with batch size 64, learning rate 0.0001, and momentum 0.999.The loss trade-off parameters λ1 and λ2 are set to 0.005 and 0.002.Source: §4.2 · p. 5
4.3. Baselines
The evaluated baselines span linear correlation, attention-based recipe-image embeddings, and triplet-loss approaches.
4.3. Baselines
- CCA learns linear projections that map text and image features into a shared space maximizing feature correlation.Source: §4.3 · pp. 5–6
- SAN learns an ingredient-only image embedding through two-layer attention, whereas JE uses pairwise cosine loss and predicts food categories from its joint embedding.Source: §4.3 · p. 6
- AM applies attention across recipe title, ingredients, and instructions, while AdaMine combines joint-embedding and food-category triplet losses with adaptive mining.Source: §4.3 · p. 6
4.4. Quantitative Results
ACME matches the best method’s median rank on both 1k retrieval directions and outperforms all baselines across metrics on 10k tests; ablations report gains from each component.
4.4.1 Main Results
- On 1k tests, ACME achieves a median rank of 1.0 for both retrieval directions, matching the current best method; on 10k tests, its median ranks are below 6.7 and 6.0, about 50% lower than the current state of the art.ACME also outperforms all baselines across metrics on both test-set sizes, with Recall@K substantially superior.Source: §4.4.1 · p. 6
4.4.2 Ablation Studies
- Ablations report positive value from hard-sample mining, modality alignment, and cross-modal translation, with all components contributing collaboratively to overall performance.The study tests R2I and I2R translation separately and together, and compares the full model with and without modality alignment.Source: §4.4.2 · p. 6
4.5. Qualitative Results
On the full 50k test set, ACME’s top retrieved items are presented as qualitative matches despite the difficulty of retrieving the correct instance from so many possibilities. The examples cover both retrieval directions and cross-modal translation outputs.
4.5. Qualitative Results
- Despite the difficulty of retrieving the correct instance among 50k possibilities, ACME’s top results are described as similar to the ground truth.The section frames these examples as qualitative evidence rather than a claim that the ground truth is always retrieved.Source: §4.5 · pp. 6–7
4.5.1 Recipe to Image Retrieval Results
- For coffee cake, sauce chicken, and fruit salad queries, ACME’s top-five images are visually similar to the ground-truth image.The examples include category-relevant imagery, such as cakes, cooked chicken, and fruit; Figure 3 shows the query recipe and retrieved images, with the ground truth boxed in red.Source: §4.5.1 · p. 7, Figure 3 · p. 7
4.5.2 Image to Recipe Retrieval Results
- For chicken soup and sugar-cookie image queries, retrieved recipes share ingredients with the true recipes, including chicken when it is difficult to see in the soup image.Figure 4 places each query image above its true recipe and top retrieved recipe.Source: Figure 4 · p. 7, §4.5.2 · p. 8
4.5.3 Cross-Modal Translation Consistency
- The translation components serve primarily as constraints for learning semantic embeddings; the authors do not claim robust or very good translations.They cite noisy food categories and the difficulty of generating images and classifying 1,047 food categories or over 4,102 ingredients.Source: §4.5.3 · p. 8
- For two ice-cream recipes, generated images are reported to resemble real food images; for a pork-chop image, several predicted ingredients are reported as correct.Figures 5 and 6 visualize the recipe-to-image and image-to-ingredient-prediction examples, respectively.Source: §4.5.3 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
5. Conclusion
The conclusion presents ACME as an end-to-end framework for a joint embedding of cooking recipes and food images, using adversarial networks to guide learning. It reports state-of-the-art cross-modal retrieval results on Recipe1M.
5. Conclusion
- ACME is an end-to-end framework for learning a joint embedding between cooking recipes and food images, and the authors identify it as the first use of adversarial networks to guide this learning.Source: §5 · p. 8
- The framework combines hard-sample mining, adversarial modality alignment, and cross-modal translation consistency, using recipe embeddings to generate food images and image embeddings to recover ingredients.Source: §5 · p. 8
- ACME achieves state-of-the-art cross-modal retrieval results on the Recipe1M dataset.Source: §5 · p. 8