PaperlayerView exact revision on arXiv

Source-linked AI summary

What Constitutes Food Similarity? Visual Representations and Retrieval in Indian Cuisine

Saransh Kumar Gupta, Akshadha Sharma, Lipika Dey, Partha Pratim Das, Ramesh Jain

arXiv:2610.37867v1cs.AIcs.CVcs.IR

TL;DR

Identity-centered evaluation offers limited evidence about how food representations organize relationships among foods, which matter for food-computing tasks. The paper compares seven visual representations in Indian cuisine and relates their dish-level structures to recipe-derived similarity. Image-level neighborhoods vary across representations, while aggregated dish-level visual structures share a common pattern but correspond only partially with recipe similarity.

  • Problem

    Food-vision evaluation centers on preserving dish identity, providing limited insight into how representations organize food relationships needed for tasks such as retrieval and recommendation.

    Source: §1 · p. 2

  • Method

    The study compares seven visual representations under fixed data and retrieval settings, examining image and dish neighborhoods and comparing dish-level visual structures with recipe-derived similarity.

    Source: §1 · pp. 2–3

  • Results

    Across the same collection and retrieval protocol, representations change retrieval behavior and food neighborhoods; aggregated dish-level visual similarity has strong common structure across representations but only partial correspondence with recipe-derived similarity.

    Source: §7 · p. 14

  • Takeaways & Limitations

    Food retrieval depends on the relationships accessible in a representation’s neighborhoods, so evaluation should consider induced relational structures as well as identity preservation.

    Source: §7 · p. 14

    The recipe analysis uses one recipe and a fixed text encoder per dish, so its results should not be treated as ground-truth culinary similarity.

    Source: §6 · p. 14

Abstract

from arXiv · show

Food similarity underlies retrieval, recommendation, substitution, and other food computing tasks, yet food representations are typically evaluated by identity preservation. We ask what happens to food similarity beyond identity: how different visual representations organize foods into relational neighborhoods, what culinary relationships these neighborhoods reveal or obscure, and how consistently these structures emerge across representations. Indian cuisine provides a challenging setting in which visually similar foods may have different culinary identities, while foods within a culinary family can vary substantially in appearance. Using an expanded Khana dataset and seven pretrained visual representations spanning convolutional, transformer-based, self-supervised, and image-text aligned models, we evaluate image-to-image retrieval and image-to-dish neighborhoods, examining culinary-family structure, contrast cases, and cross-representation agreement. We also aggregate image representations to compare dish-level visual similarity across models and against recipe-derived similarity for the same dishes. Image-level neighborhoods vary substantially across representations, including similarities across culinary-family boundaries. At the dish level, aggregation reveals common visual structure despite local variation, but this structure corresponds less strongly with recipe-derived similarity. These findings show that recognition performance alone cannot characterize relational structure: food similarity depends on representation, modality, and abstraction level rather than being a fixed property of the food collection. Understanding these structures is therefore a prerequisite to combining multiple food information modalities into a coherent notion of similarity.

1 Introduction

Food’s appearance does not reliably indicate culinary identity, while identity-focused evaluation offers limited insight into how representations relate foods. This paper studies representation-induced neighborhoods in Indian cuisine using an expanded dataset and a retrieval-centered comparison of seven visual representations.

1 Introduction

  • Food appearance varies with preparation, ingredients, presentation, regional practice, and cultural context, and does not map cleanly onto culinary identity.Visually similar dishes may have different identities, while related foods may look substantially different.Source: §1 · p. 2
  • Identity-focused benchmarks show whether a representation preserves dish labels but provide limited insight into relationships among foods.Such relationships matter for retrieval, recommendation, substitution, and other food-computing applications.Source: §1 · p. 2
  • The study holds the dataset, query, and retrieval procedure fixed while comparing seven visual representations and examining culinary-family relationships.It also compares visual structures with recipe-derived similarity over the same dish concepts.Source: §1 · pp. 2–3
  • The expanded Khana collection grows from 80+ classes and 130,000+ images to 300+ classes and 200,000+ images.It adds Indian food categories to the existing collection.Source: §1 · p. 2
  • A retrieval-centric protocol examines image retrieval, image-to-dish neighborhoods, culinary-family composition, contrast cases, and agreement across representations.Nearest-neighbor neighborhoods and similarity scores differ across representations for a common query, as illustrated for biryani in Figure 1.Source: §1 · p. 3, Figure 1 · p. 3

2 Related Work

Prior food-vision work primarily evaluates identity recognition, while food similarity and retrieval work often evaluates neighborhoods against task-specific objectives. Diverse visual representations and Indian cuisine’s underexplored relational structure motivate examining neighborhoods directly.

2 Related Work

  • Food-vision benchmarks have primarily assessed whether images from predefined dish categories are recognized or retrieved together [6] [11].This establishes category discrimination but says less about how distinct food concepts are positioned relative to one another.Source: §2 · p. 3
  • Food similarity and multimodal retrieval systems relate images, recipes, and other food information, but evaluations remain tied to their construction objectives [31].Neighborhood structure itself is less often examined independently.Source: §2 · p. 4
  • Visual representations vary in architecture, training objectives, supervision, and pretraining data, so they can produce different embedding structures [7] [20] [25].Representation analysis therefore examines neighborhoods and feature-space relationships alongside recognition performance.Source: §2 · p. 4
  • Indian food remains comparatively underrepresented in large-scale benchmarks, and existing resources provide limited explicit relationships among food concepts [21].Khana’s primarily dish-level organization leaves regional variants, culinary families, and visually similar foods with distinct identities relatively unexplored [21].Source: §2 · p. 4

3 Food Similarity is Representation-Dependent

Food has no single canonical similarity relation: its similarity structure depends on the information and representation used. The paper distinguishes visual similarity from culinary relatedness and probes where these relationships diverge.

3.1 Food Lacks a Canonical Similarity Structure

  • Foods sharing a dish identity can vary in filling, preparation, form, presentation, role, and regional expression, while distinct culinary identities can look alike.The samosa tradition illustrates how shared identity need not imply uniformity along any single dimension.Source: §3.1 · p. 5
  • Similarity is computed over item representations, so different representation functions can induce different structures over the same food collection.Here, visual similarity is a neighborhood structure induced by an image encoder and comparison mechanism.Source: §3.1 · p. 5

3.2 Visual Similarity, Culinary Relatedness, and Contrast Cases

  • Visual similarity draws on image-visible features, whereas culinary relatedness is independently operationalized through family annotations based on lineage, preparation tradition, culinary form, or regional variants.Family membership is not treated as semantic ground truth or as a consequence of visual proximity.Source: §3.2 · p. 5
  • Food similarity can also reflect ingredients, procedure, function, cultural or regional tradition, and shared preferences, rather than a single underlying relation.A universal score would require specifying which dimensions count and how they are combined.Source: §3.2 · pp. 5–6
  • Three contrast cases probe high visual similarity across families, low visual similarity within a family, and high visual similarity within a family.Cross-family visual dissimilarity is excluded because visual and culinary relations are aligned in that case.Source: §3.2 · p. 6

4 Methodology

The methodology varies visual representation while holding the food collection and retrieval procedure fixed, then examines similarity at image, image-to-dish, and dish-concept levels. It uses an expanded Indian-food benchmark, family annotations, and recipe-derived comparisons.

4 Methodology

  • The controlled experiment holds the collection, labels, family annotations, similarity function, and retrieval procedure fixed while varying representation across three similarity levels.The levels are image-to-image, query-image-to-dish, and dish-to-dish comparisons.Source: §4 · p. 6

4.1 Dataset and Benchmark Construction

  • The final benchmark contains 200,000+ images across more than 300 dish classes and 45 culinary families.It augments Khana with web-sourced images, cleans duplicates and irrelevant images, and assigns family relations independently of visual embeddings.Source: §4.1 · pp. 6–7

4.2 Visual Representations and Retrieval Structure

  • Seven pretrained representations span supervised convolutional, transformer-based, self-supervised, and image-text-aligned models; image embeddings are ranked by cosine similarity.Image-to-dish neighborhoods retain each dish’s highest-ranked image, with alternative rules aggregating maximum, top-five mean, or all-image mean similarities.Source: §4.2 · pp. 7–8

4.3 Dish-Level Visual and Recipe Similarity

  • Mean image embeddings produce one visual representation per dish, which is compared across models and with one recipe embedding per dish.Each recipe includes its title, ingredients, and preparation instructions and is encoded with all-mpnetbase-v2.Source: §4.3 · p. 8

4.4 Evaluation Metrics

  • Evaluation covers same-dish Precision@K, Recall@K, and MRR, culinary-family composition, dish-level Spearman correlation and Jaccard overlap, and neighborhood agreement.Query-level measures are averaged within dish class and then macro-averaged across classes.Source: §4.4 · pp. 8–9

4.5 Contrast-Case Operationalization

  • Contrast-case thresholds are calibrated separately for each representation using percentiles of its pairwise similarity distributions.Cases identify unusually high or low similarities relative to each model’s distribution rather than a shared absolute cosine threshold.Source: §4.5 · p. 9

5 Results and Discussion

Across the seven visual representations, image neighborhoods vary in identity retrieval and culinary-family composition, and visually similar foods often cross family boundaries. Dish-level visual structures share substantial agreement across models but correspond less strongly with recipe-derived similarity; the resulting notion of similarity depends on representation and aggregation.

5.1 Representation-dependent image retrieval and neighborhoods

  • Same-dish Precision@10 ranges from 0.258 to 0.523 and MRR from 0.509 to 0.707, while 0.758–0.850 of distinct-dish neighbors belong to different culinary families.Family@10 ranges from 0.150 to 0.242, separating identity preservation from broader culinary organization.Source: §5.1 · p. 10

5.2 Visual similarity crosses culinary boundaries

  • Highly similar cross-family foods occur at rates of 0.592–0.766, while highly similar within-family foods occur at 0.145–0.218.Extreme visual dissimilarity within a family is rare, so culinary relatedness constrains some visual extremes without preventing cross-family similarity.Source: §5.2 · p. 10, Figure 3 · p. 11

5.3 The induced neighborhoods differ across representations

  • Pairwise Top-10 neighborhood overlap ranges from 0.282 to 0.515, with representations differing both in neighbor membership and in the ordering of shared neighbors.Figure 4 reports mean shared distinct-dish neighbors; Figure 3 projects ViT-B/16 contrast cases defined in the original embedding space.Source: §5.3 · p. 11, Figure 4 · p. 11, Figure 3 · p. 11

5.4 Image-to-dish aggregation affects dish-level neighborhoods

  • Strongest-image and top-five-mean neighborhoods overlap by 0.541–0.599, recovering approximately 69–73% of strongest-image relationships, while strongest-image versus all-image-mean overlap is 0.277–0.345.The results indicate that a limited set of visually representative instances can substantially influence dish-level relationships.Source: §5.4 · p. 11, Figure 5 · p. 12, Figure 6 · p. 13

5.5 Visual and recipe-derived dish similarity

  • Dish-level visual structures correlate across representations at Spearman 0.749–0.964, but visual–recipe Spearman correlations are 0.223–0.326 and Top-10 Jaccard overlap is 0.069–0.095.Recipe-derived similarity is related to, but not uniformly reflected in, visual spaces and should not be treated as ground-truth culinary similarity.Source: §5.5 · p. 12, Figure 7 · p. 13, Table 2 · p. 13

5.6 Implications for food similarity

  • Food similarity is not an invariant structure: representation choice affects image neighborhoods, and image-evidence aggregation affects dish-level relationships.Its appropriateness depends on the intended retrieval context and comparison procedure.Source: §5.6 · pp. 12–13

6 Limitations and Future Work

The analysis is conditioned on the image collection and annotations, and its recipe comparison uses one recipe and a fixed text encoder per dish. Future work proposes richer multimodal and query-conditioned similarity, evaluated by the relational structures they induce.

Limitations

  • Image quality, composition, and source can affect absolute retrieval metrics, so results are primarily a controlled comparison of representation-induced structure rather than a benchmark of absolute recognition.The analysis is conditioned on the image collection and its annotations.Source: §6 · p. 13
  • Culinary-family assignments involve abstraction and judgment and capture only selected forms of culinary relatedness; one recipe and a fixed text encoder per dish also limit representation of within-dish recipe variation.The recipe results should not be interpreted as ground-truth culinary similarity.Source: §6 · p. 14

Future Work

  • Future work proposes richer multimodal representations, tests aggregation and alternative relatedness dimensions, and explores query-conditioned similarity across food aspects and purposes.Additional modalities should be evaluated by the relational structures they induce rather than assumed to resolve similarity ambiguity.Source: §6 · p. 14

7 Conclusion

Food similarity changes with the visual representation, and visual proximity is related to but distinct from culinary relatedness. Aggregated dish-level visual similarity shares common structure across representations but corresponds only partially with recipe-derived similarity.

7 Conclusion

  • Changing the visual representation changes retrieval behavior and food neighborhoods, which intersect culinary structure in different ways.Visual proximity and culinary relatedness are related but distinct.Source: §7 · p. 14
  • At the dish level, aggregated visual similarity has strong common structure across representations but only partial correspondence with recipe-derived similarity.Source: §7 · p. 14
  • Food retrieval depends on which relationships a representation makes accessible, rather than on a single similarity notion inherent to the food collection.Representations should be evaluated for whether their induced relational structures suit the intended retrieval context.Source: §7 · p. 14

A Samosa Variants

Examples of samosa-concept variation span different fillings and preparations, with forms ranging from flat squares and triangles to layered or crescent shapes.

A Samosa Variants

  • The listed samosa variants include minced meat, meat with egg and vegetables, and potato with onion and peas.Source: A Samosa Variants · p. 17
  • Lukhmi Samosa is described as deep-fried and flat-square or rectangular, while Warqi Samosa is deep-fried and layered, flaky, and triangular.Source: A Samosa Variants · p. 17
  • The examples include Burmese Samusa as a flat triangle, Goan Chamuça as triangular, and Sambousek in crescent, triangular, or tubular forms.The table also lists meat, cheese, chickpeas, and vegetables for Middle Eastern Sambousek.Source: A Samosa Variants · p. 17

Related papers

Loading 2610.37867v1…