Source-linked AI summary
Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings
Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, Matthieu Cord
TL;DR
Cooking-oriented retrieval must connect heterogeneous recipe text and images while organizing both exact matches and semantically related items. AdaMine learns a shared cross-modal space with joint retrieval and class-based losses, a double-triplet scheme, and adaptive triplet mining, and is evaluated on Recipe1M. The paper reports that AdaMine outperforms state-of-the-art models and supports direct semantic refinement with fewer learned parameters.
Problem
Heterogeneous cooking data and culturally intrinsic recipe semantics make it challenging to align text and images while supporting cross-modal retrieval.
Method
AdaMine jointly learns retrieval and class-based features in a shared latent space using double-triplet losses and adaptive mining of informative triplets.
Results
AdaMine outperforms state-of-the-art models on the large-scale Recipe1M crossmodal dataset.
Takeaways & Limitations
Directly injecting semantic information into cross-modal metric learning refines the latent space while limiting the number of parameters to be learned.
Takeaways & Limitations
The paper frames simultaneous retrieval and class-based learning as a conjectured approach whose effectiveness is evaluated in the Recipe1M cooking domain.
Abstract
from arXiv · showhide
Designing powerful tools that support cooking activities has rapidly gained popularity due to the massive amounts of available data, as well as recent advances in machine learning that are capable of analyzing them. In this paper, we propose a cross-modal retrieval model aligning visual and textual data (like pictures of dishes and their recipes) in a shared representation space. We describe an effective learning scheme, capable of tackling large-scale problems, and validate it on the Recipe1M dataset containing nearly 1 million picture-recipe pairs. We show the effectiveness of our approach regarding previous state-of-the-art models and present qualitative results over computational cooking use cases.
1 INTRODUCTION
The paper addresses cross-modal cooking applications by learning a shared space for heterogeneous recipe data, while replacing classification-augmented alignment with joint retrieval and semantic learning. It introduces double-triplet training and adaptive triplet mining, and evaluates the resulting model on Recipe1M.
- Cooking data span text, images, videos, and preferences, but heterogeneous structures and culturally intrinsic vocabulary, measurements, and flavor perceptions make recipe semantics difficult to homogenize.
- Existing matching-pair alignment can generalize poorly because it brings exact dish pairs together without necessarily placing similar dishes nearby.
- The proposed framework jointly leverages retrieval and class-guided features directly in a shared latent space.
- Its double-triplet scheme combines fine-grained cross-modal matching with class-based similarity directly in feature space, avoiding an added classification layer.
- Adaptive mining selects significant triplets to tune gradient updates beyond usual gradient averaging, producing better embeddings.
- AdaMine is evaluated on Recipe1M and reported to outperform state-of-the-art models.
2 RELATED WORK
Related work frames cross-modal retrieval as alignment or ranking across heterogeneous modalities, with food datasets enabling increasingly complex tasks. The paper differentiates AdaMine through direct semantic integration, double-triplet learning, and adaptive mining.
- 2.1 Computational cooking: Computational cooking research uses recipe and food data for tasks including ingredient identification, recommendation, and popularity prediction.
- 2.2 Cross-modal Retrieval: Cross-modal retrieval ranks items in one modality against queries from another, with the central challenge of measuring cross-modal similarity.
- 2.2 Cross-modal Retrieval: Global alignment methods such as CCA maximize correlation between relevant cross-modal pairs but do not account for dissimilar pairs.
- 2.2 Cross-modal Retrieval: Local metric-learning methods formulate retrieval as ranking, optimizing losses for violated inequalities between relevant and irrelevant items.
- 2.2 Cross-modal Retrieval: AdaMine directly integrates class-based information into the loss, uses double-triplet learning, and adaptively mines informative triplets in minibatches.
3 ADAMINE DEEP LEARNING MODEL
AdaMine jointly learns cross-modal retrieval and class-based semantic structure in a shared latent space, using double-triplet losses and adaptive sampling to preserve both fine-grained and high-level relationships.
- Model overview: AdaMine learns recipe-item embeddings by jointly optimizing instance-based retrieval and semantic class-based objectives in a shared latent space.The framework combines retrieval and classification learning rather than adding a separate classification stage.
- Network architecture: The architecture maps image and recipe modalities through separate deep-network branches into a common representation space where they can be compared.The image branch uses ResNet-50, while the recipe branch separately embeds ingredients and instructions before mapping their concatenation into the latent space.
- Retrieval loss: Instance-based triplets bring matching items from different modalities together while separating irrelevant items, producing fine-grained latent-space structure.Each triplet contains a query, its matching counterpart in the other modality, and a dissimilar item; similarity uses cosine distance.
- Semantic loss: Semantic-based triplets bring items sharing a class closer than items from different classes, adding high-level semantic structure without changing the neural-network architecture.The semantic loss directly optimizes relations in the latent space and encourages same-class instances to remain close.
- Adaptive learning schema: Adaptive sampling normalizes the two losses and progressively emphasizes violating triplets, avoiding vanishing updates while leaving λ as the effective trade-off control.The strategy begins with contributions from all triplets and ends by focusing on hard negatives, automatically transitioning from averaging toward hard-negative learning.
4 EVALUATION PROTOCOL
The evaluation measures AdaMine’s retrieval effectiveness, semantic and adaptive-learning components, and text-structure choices on Recipe1M using cross-modal retrieval benchmarks and ablations.
- Dataset and evaluation goals: The evaluation uses Recipe1M, a large-scale dataset of roughly 1 million image-recipe pairs containing recipes, images, and categories.Its semantic metadata makes it suitable for evaluating semantic structure in the shared representation space.
- Dataset and evaluation goals: Cross-modal retrieval ranks items in the opposite modality by cosine distance for image-to-text and text-to-image queries.The protocol uses 10 subsets of 1,000 pairs or 5 subsets of 10,000 pairs from the test set.
- Baselines and ablations: The study compares AdaMine with CCA, PWC, PWC*, and PWC++ to assess global alignment, classification-based regularization, and improved pairwise-loss baselines.The comparison includes reported state-of-the-art results and an implementation of the prior architecture and loss.
- Baselines and ablations: Ablations test semantic-loss design, adaptive learning, and text structure by comparing AdaMine_ins+cls, AdaMine_avg, AdaMine_ingr, and AdaMine_instr with the full model.Additional scenarios isolate the instance loss and semantic loss through AdaMine_ins and AdaMine_sem.
- Implementation details: Training uses Adam at a 10^-4 learning rate, initially freezes ResNet-50, then fine-tunes the full architecture after 20 epochs for 60 additional epochs.The selected model has the best validation MedR; AdaMine uses α = 0.3 and λ = 0.3 selected by cross-validation.
- Implementation details: Mini-batches contain 100 image-recipe pairs and construct double-triplets using matching cross-modal items as positives and remaining items as negative retrieval candidates.Training uses 2,383 training mini-batches and 513 validation mini-batches, with class metadata available for roughly half of the pairs.
5 EXPERIMENTS
Experiments show that directly integrating semantic information through AdaMine’s semantic loss improves cross-modal retrieval and structures the shared latent space. AdaMine consistently outperforms classification-head, pairwise, and state-of-the-art baselines across retrieval settings, while supporting ingredient-based retrieval.
- 5.1 Analysis of the semantic contribution: AdaMine’s semantic loss structures embeddings by clustering same-class items while reducing distances between matching recipe-image pairs.The t-SNE analysis uses 400 matching pairs across five Recipe1M classes and links matching instances with traces.
- 5.1 Analysis of the semantic contribution: Semantic weighting is robust at lower λ values, whereas values above 0.5 hinder retrieval as semantic grouping becomes too influential.The semantic cost Lsem is weighted by λ in the joint objective.
- 5.2 Testing the effectiveness of the model: AdaMine reduces MedR by −61.84% versus PWC++ in the 10k image-to-text setting, supporting triplet-based relative-distance learning over pairwise methods.The model also reduces 1k MedR by a factor of 5, from 5.2 and 5.1 to 1.0 and 1.0.
- 5.2 Testing the effectiveness of the model: Adaptive triplet mining improves MedR over average sampling, yielding improvements of roughly 46.34% and 49.17%.AdaMine reports MedRs of 13.2 and 12.2, compared with 24.6 and 24.0 for AdaMine_avg.
- 5.2 Testing the effectiveness of the model: Using both ingredients and instructions is necessary for strong recipe identification, since removing either produces substantially higher MedR values.AdaMine achieves MedRs of 13.2 and 12.2, compared with 52.8 and 53.8 without instructions and 39.0 and 39.2 without ingredients.
- 5.2 Testing the effectiveness of the model: Ingredient-to-image retrieval shows that the latent space can discriminate recipes according to ingredient content, including modified queries.Removing broccoli from a recipe query changes the retrieved images so they no longer contain broccoli.
6 CONCLUSION
The paper introduces AdaMine for large-scale image-to-recipe and recipe-to-image retrieval. Its joint semantic metric-learning framework and adaptive triplet mining outperform state-of-the-art models on Recipe1M and support downstream cooking-oriented retrieval tasks.
- 6 CONCLUSION: AdaMine learns cross-modal embeddings for large-scale image-to-recipe and recipe-to-image retrieval.The approach is evaluated on the Recipe1M crossmodal dataset.
- 6 CONCLUSION: The framework directly injects semantic information into joint retrieval and classification metric learning while limiting learned parameters.Its double-triplet scheme combines retrieval and class-based losses, and adaptive mining selects informative triplets.
- 6 CONCLUSION: AdaMine outperforms state-of-the-art models on Recipe1M and produces a useful latent space for downstream cooking-oriented tasks.The paper also identifies hierarchical object semantics as a direction for future refinement.