PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Food Computing

Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain

arXiv:1808.07202v5cs.CYcs.MM

TL;DR

Food computing addresses open questions in an interdisciplinary area with few systematic reviews by acquiring and analyzing heterogeneous food data using computational approaches. This survey formalizes the field, reviews its datasets, tasks, applications, concepts, and methods, and summarizes key challenges and future directions.

  • Problem

    Food computing is interdisciplinary with many open questions, but few systematic reviews provide comprehensive coverage of the area and its open problems.

  • Method

    The survey formalizes food computing and reviews food-data acquisition and analysis, datasets, tasks, applications, and computational approaches.

  • Results

    The survey provides an extensive review of notable food-computing work spanning food-oriented data acquisition and analysis, perception, recognition, retrieval, recommendation, and monitoring.

  • Takeaways & Limitations

    The survey offers a collection of research studies and technologies for researchers and practitioners working in different food-related fields.

  • Takeaways & Limitations

    The survey identifies ongoing challenges including robust and accurate food-image recognition, vision-based dietary management, and health-aware personalized food systems.

Abstract

from arXiv · show

Food is very essential for human life and it is fundamental to the human experience. Food-related study may support multifarious applications and services, such as guiding the human behavior, improving the human health and understanding the culinary culture. With the rapid development of social networks, mobile networks, and Internet of Things (IoT), people commonly upload, share, and record food images, recipes, cooking videos, and food diaries, leading to large-scale food data. Large-scale food data offers rich knowledge about food and can help tackle many central issues of human society. Therefore, it is time to group several disparate issues related to food computing. Food computing acquires and analyzes heterogenous food data from disparate sources for perception, recognition, retrieval, recommendation, and monitoring of food. In food computing, computational approaches are applied to address food related issues in medicine, biology, gastronomy and agronomy. Both large-scale food data and recent breakthroughs in computer science are transforming the way we analyze food data. Therefore, vast amounts of work has been conducted in the food area, targeting different food-oriented tasks and applications. However, there are very few systematic reviews, which shape this area well and provide a comprehensive and in-depth summary of current efforts or detail open problems in this area. In this paper, we formalize food computing and present such a comprehensive overview of various emerging concepts, methods, and tasks. We summarize key challenges and future directions ahead for food computing. This is the first comprehensive survey that targets the study of computing technology for the food area and also offers a collection of research studies and technologies to benefit researchers and practitioners working in different food-related fields.

1 INTRODUCTION

Food computing emerges from the growth of large-scale, heterogeneous food data and advances in computational methods. This survey formalizes the field, reviews its studies and applications, and identifies open problems and future directions.

  • Traditional food studies often relied on small-scale data, including questionnaires, cookbooks, and recipes, before the web transformed research practices.
  • Social, mobile, and IoT networks enable people to share food images, recipes, cooking videos, and food diaries, producing large-scale food datasets.
  • These data support studies of food perception, culinary habits, and diet monitoring, while computer vision, machine learning, data mining, and network analysis provide analytical tools.
  • Food computing applies heterogeneous food data and computer science to issues spanning human behavior, health, agriculture, culture, medicine, food security, and food science.
  • Because few systematic reviews comprehensively organize the field, this survey defines food computing, introduces its framework, tasks, and applications, and classifies about 300 studies.

2 FOOD COMPUTING

Food computing uses computer science to acquire and analyze heterogeneous food data for interconnected perception, recognition, retrieval, recommendation, prediction, and monitoring tasks. It connects food data with human-centric services across multiple disciplines.

  • Its main tasks include food-oriented perception, recognition, retrieval, recommendation, prediction, and monitoring.
  • The field is interdisciplinary, connecting computer science with food science, medicine, biology, agronomy, sociology, gastronomy, neuroscience, cognitive science, and chemistry.
  • Food computing acquires and analyzes heterogeneous food data from disparate sources for perception, recognition, retrieval, recommendation, and monitoring.
  • Its framework collects food data from cookbooks, social networks, sensors, IoT devices, recipe websites, and specialized databases before computational analysis.
  • These tasks are mutually dependent: recognized results can support retrieval and recommendation, while predicted food preferences can inform personalized recommendation.

3 FOOD DATA ACQUISITION AND ANALYSIS

Food computing draws on data from official sources, recipe websites, social media, IoT devices, and crowdsourcing. The survey compares diverse food datasets and summarizes how their modalities and metadata support research tasks.

  • Data sources: The survey groups food-data sources into official organizations and experts, recipe-sharing websites, social media, IoT devices, and crowdsourcing platforms.
  • Data sources: Official and expert-based acquisition can be time-consuming, laborious, and difficult to scale.
  • Recipe-sharing websites: Recipe-sharing websites provide ingredients, images, cuisine, course, flavor, macronutrient, ratings, and comments for food-related analysis and recommendation.
  • Other sources: Social media, IoT devices, and crowdsourcing contribute textual, demographic, image, video, and nutritional data for studying health, perception, cooking, and food understanding.
  • Food-data types: Food datasets include recipes, dish images, cooking videos, and food attributes, supporting tasks such as recommendation, recognition, retrieval, and activity recognition.
  • Dataset comparison: Recipes contain richer attribute and metadata information than food images, while food-image recognition benchmarks are released frequently.

4 APPLICATIONS IN FOOD COMPUTING

Food computing applications span health, dietary management, social-media health analysis, and related food-data resources. These applications use food images and records to support assessment, monitoring, and healthier choices.

  • Food computing applications are organized around health, agriculture, culture, and food science.
  • Health: Food perception research examines how sensory cues influence food identification, choice, and intake.
  • Health: Food-related social-media data can support population health analysis, including associations between food choices and regional health.
  • Health: Dietary assessment supports disease prevention, while traditional questionnaires and self-reports can suffer from underreporting and miscalculation.
  • Health: Vision-based systems analyze photographs of food selection and plate waste to estimate portions, grams, and kilocalories.

generally need 3D reconstruction from food images.

Food computing combines image, recipe, sensor, and social-media data with computational methods for food recognition, monitoring, recommendation, and cultural analysis. The surveyed work also identifies recognition and recommendation challenges requiring specialized methods and richer personal information.

  • Im2Calories segments meal photos, labels segmented regions, and estimates food volume before calorie estimation.
  • Food-monitoring systems use images, mobile devices, depth cameras, and wearable sensors to recognize food, estimate intake, or detect eating events.
  • Food recommendation research balances user preferences with nutritional appropriateness and seeks to incorporate intake, portion size, and personal factors.
  • Recipe and social-media analyses examine culinary cultures, ingredient patterns, regional cuisines, and cultural relations across communities.
  • Food differs from ordinary objects in recognition settings, motivating new recognition paradigms for dishes and ingredients.

5 TASKS IN FOOD COMPUTING

Food computing includes six main task categories covering food perception, recognition, retrieval, recommendation, prediction, and monitoring.

  • The main food-computing tasks are perception, recognition, retrieval, recommendation, prediction, and monitoring.

5.1 Perception

Food perception is multimodal and influences food choice, acceptance, and intake. The survey highlights growing research interest while noting that computer-science methods and multimodal integration remain underused challenges.

  • Food perception research connects sensory cues with food choice, acceptance, flavor, and intake.
  • Studies examine neural responses to food attributes and the influence of visual and odor cues on recognition and selection.
  • Image recognition has been used to study gaps between human food perception and machine labeling.
  • Food perception integrates visual, auditory, taste, smell, and tactile signals rather than relying on one sense.
  • Most food-perception methods are domain-specific, and computer-vision and machine-learning methods remain incompletely exploited.
  • Effective integration of heterogeneous multimodal food features remains an important problem, with deep-learning fusion proposed as one feasible approach.

5.2 Recognition

Food recognition supports dietary assessment, health analysis, monitoring, and service automation. The surveyed approaches span single-label, multi-label, mobile, and restaurant-specific recognition, increasingly using deep visual features.

  • Motivation: Diet management is important for addressing obesity, diabetes, and cardiovascular disease, but manually recorded food diaries have difficult-to-guarantee accuracy.Smartphones and computer vision enable image-based food recognition for dietary assessment.
  • Applications: Recognized food categories or ingredients support calorie-intake estimation, nutrition analysis, and eating-habits analysis.Food recognition can also monitor consumption and automate billing in self-service restaurants.
  • Task Taxonomy: Food-recognition algorithms are organized into single-label, multi-label detection, mobile, and restaurant-specific recognition.Single-label methods target images containing one food item, whereas real-world images may contain multiple items.
  • Methods: WISeR combines a residual-network branch for general visual features with a slice branch for specific vertical food layers, then concatenates their outputs for classification.The fused representation is fed to two fully connected layers, and the method is reported as state-of-the-art on benchmark datasets.
  • Multi-label Recognition: Multi-food recognition detects candidate regions, classifies them, and can exploit co-occurrence relations between food items.Detection and segmentation are widely used when images contain multiple food items.
  • Methods: CNN-based deep networks improve food-recognition performance compared with handcrafted features and support diverse architectures and feature-fusion strategies.Examples include AlexNet, GoogLeNet, ResNet, Inception, and ensemble or multimodal representations.

5.3 Retrieval

Food retrieval covers visual food retrieval, recipe retrieval, and cross-modal recipe-image retrieval. Methods represent images, recipes, ingredients, and cooking procedures to support search and image-based recipe access.

  • Applications: Prepared-food images can mix and obscure ingredients, making direct ingredient recognition challenging and motivating image-query recipe retrieval.Recipes additionally provide ingredients, macronutrient composition, and cooking methods useful for calorie and nutrition estimation.
  • Retrieval Types: Food-relevant retrieval comprises visual food retrieval, recipe retrieval, and cross-modal recipe-image retrieval.These retrieval types use food-related data including text recipes, images, videos, and user preferences.
  • Visual Food Retrieval: Visual food retrieval systems distinguish food images, support image- and text-based queries, and compare feature-based image representations.Representations include BoF, SIFT, Bag of Textons, and CNN-based features.
  • Recipe Retrieval: Recipe retrieval models recipes using ingredient, cooking-flow, eating, nutrition, graph, or tree-structured representations for similarity search.Some approaches use frequent patterns and filtering algorithms for efficient online searching.
  • Cross-modal Retrieval: Cross-modal retrieval learns relationships between dish images and recipes using ingredients, semantic attributes, attention, multimodal embeddings, or adversarial alignment.One hybrid architecture learns visual features with CNNs and recipe features with LSTMs in a shared space.

5.4 Recommendation

Food recommendation incorporates recipe content, collaborative signals, context, culinary culture, and health information. The surveyed systems use these factors to personalize food and recipe suggestions.

  • Scope: Food recommendation is an important domain for individuals and society and involves complex, multifaceted, context-dependent information.The survey distinguishes content-based, collaborative, hybrid, context-aware, and health-aware recommendation.
  • Recommendation Methods: Content-based systems represent recipes through ingredients, topics, structure, or multimodal attributes, while collaborative systems use ratings and user tags.Matrix factorization and singular value decomposition are common collaborative approaches.
  • Recommendation Methods: A matrix-factorization approach combining ratings and user-supplied tags achieves significantly better prediction accuracy than content-based and standard matrix-factorization baselines.
  • Context-aware Recommendation: Context-aware recommenders use factors such as gender, time, hobbies, location, lifestyle preferences, and culinary culture.Analyses of social-media and recipe data identify temporal, spatial, and ingredient-combination patterns relevant to recommendation.
  • Health-aware Recommendation: Health-aware systems incorporate nutrition, calorie balance, personalized health data streams, environmental context, and recipe features to rank or filter recommendations.Applications include healthier recipe estimation, real-time nutrition recommendation, diabetic-food substitution, and patient-specific food lists.
  • Health-aware Recommendation: Nutrition and health labels help people make better food choices, supporting the use of health information in food recommendation.

5.5 Prediction and Monitoring

Food-related prediction and monitoring use large-scale social-media records to study consumption, nutrition, population characteristics, and public health. These approaches complement traditional questionnaires and food diaries.

  • Data Sources: Social networks provide large-scale food data that support food-related prediction and monitoring.Twitter, Instagram, and Facebook contain records of dietary choices and other food-related behavior.
  • Prediction: Studies use social-media data to predict geographic location, overweight and diabetes rates, recipe popularity, income, and food-production patterns.Methods include ingredient matching with the USDA National Nutrient Database and multimodal deep features.
  • Motivation: Traditional large-scale dietary studies rely on questionnaires and food diaries that can be inaccurate and expensive.Social-media records offer an alternative source for analyzing food consumption at scale.
  • Public-health Monitoring: Machine-learning systems mine Twitter to detect venues likely to pose public-health hazards and identify personal health mentions.These applications include foodborne-illness prevention and public-health monitoring.

6 CHALLENGES

Food computing faces challenges in recognizing diverse food images, estimating food volume, integrating heterogeneous data, and delivering personalized health-aware recommendations. These issues limit robust dietary management and broader food-related applications.

  • Food Image Recognition: Food images often lack distinctive spatial layouts, contain deformable appearances, randomly distributed ingredients, and vary with cooking methods.These properties make direct reuse of object- or scene-recognition methods difficult.
  • Food Image Recognition: Food recognition is a fine-grained classification problem with subtle category differences, but food lacks the fixed semantic parts used in conventional fine-grained methods.The paper calls for a new categorization paradigm suited to food recognition.
  • Food Dataset Construction: Large-scale benchmark food datasets remain insufficient, while regional naming differences and identical names for different dishes complicate standard ontology construction.Food101 and UEC Food256 are described as smaller than ImageNet and insufficient in category coverage and scale.
  • Dietary Management: Vision-based dietary management remains difficult because real-world systems recognize only limited standard dishes and must estimate food volume from a single image.Recognition errors can propagate to incorrect calorie estimates; user interaction may alleviate the problems but affect user experience.
  • Multimodal Fusion: Effective health-aware personalization requires fusing signals across social networks, mobile devices, and IoT, despite unbalanced distributions and heterogeneous modalities.Useful data sources may differ by task, making source selection itself difficult.
  • Health-aware Recommendation: Personalized food recommendation must account for temporal and spatial context, culture, gender, preferences, and changing physical states while monitoring wellness continuously.Existing work has integrated wearable and social-network data, but heterogeneous modality fusion remains difficult.

7 FUTURE DIRECTIONS

The survey identifies future directions centered on larger food datasets, robust recognition, multimodal learning, knowledge graphs, and personal-health models. These directions target the field’s data, modeling, and application challenges.

  • 7.1 Large-scale Standard Food Dataset Construction: An ImageNet-level ontology of food images is needed for large-scale food search, classification, understanding, training, and benchmarking.The proposed construction combines social-media image crawling with manual annotation and should cover geographically diverse cuisines.
  • 7.2 Large-scale Robust Food Recognition System: A large-scale robust food recognition system is a first priority, using deep models designed to capture food-specific structures and properties.The survey highlights CNN variants and food-specific architectures such as slice convolution networks.
  • 7.2 Large-scale Robust Food Recognition System: Ingredient recognition is proposed as an alternative route because worldwide dish diversity exceeds the number of ingredients.Multi-label ingredient prediction may support nutrition tracking and more accurate dietary assessment.
  • 7.3 Joint Deep and Broad Learning for Food Computing: Joint deep and broad learning should combine heterogeneous food modalities and data sources within a unified multimedia food-data fusion framework.The framework is intended to transfer into food medicine and food science.
  • 7.4 Food-oriented Multimodal Knowledge Graph: A food-oriented multimodal knowledge graph can represent visual, textual, structured, contextual, and relational information for retrieval, QA, reasoning, inference, and multimodal dialogue.The graph is proposed to learn from large-scale multimodal food data and support inference engines.
  • 7.5 Food Computing for Personal Health: Food computing can construct personal models for health-aware recommendation by integrating food data with attitudes, beliefs, preferences, lifestyles, and hobbies.The paper connects this direction to quantifying lifestyle and navigating personal health.

8 CONCLUSIONS

The survey reviews food-computing datasets, tasks, applications, and services, then identifies major challenges and future research directions. It frames food computing as a way to connect food and human concerns across health, behavior, and culture.

  • Scope and Contributions: The survey extensively reviews food-computing work spanning data acquisition and analysis, perception, recognition, retrieval, recommendation, prediction, monitoring, and applications.It presents the review as covering the field’s datasets, tasks, and applications.
  • Challenges: The paper highlights robust food image recognition, vision-based dietary management, multi-network food-data fusion, and health-aware personalized recommendation as key challenges.These challenges are presented alongside food perception for health and food recognition for diet management.
  • Future Directions: Future directions include standard food datasets, robust recognition, joint deep and broad learning, multimodal knowledge graphs, and food computing for emerging vertical fields.The survey states that these promising directions need further research.
Loading 1808.07202v5…