Source-linked AI summary
Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food
Quin Thames, Arjun Karpur, Wade Norris, Fangting Xia, Liviu Panait, Tobias Weyand, Jack Sim
TL;DR
Estimating nutrition from food images is difficult because existing datasets often lack diverse dishes and accurate portion annotations, while generic food requires both ingredient recognition and portion estimation. The paper introduces Nutrition5k, a richly annotated dataset with visual, depth, weight, and nutritional data, and evaluates image-based nutritional prediction. Its models outperform professional nutritionists on caloric and macronutrient estimation, while depth information improves direct nutritional prediction; the dataset may still contain collection-related biases.
Problem
Existing nutrition datasets often lack sufficient diversity or accurate portion annotations, limiting visual understanding of generic food, which requires ingredient recognition and portion estimation.
Method
Nutrition5k provides 5,000 real-world dishes with RGB videos, depth images, component weights, and high-accuracy nutritional annotations for training nutritional prediction models.
Results
The trained models outperform professional nutritionists in caloric and macronutrient estimation, and depth data significantly improves direct nutritional prediction from RGB images.
Takeaways & Limitations
Nutrition5k supplies a large, diverse, deeply labeled benchmark intended to accelerate research on visual nutritional understanding.
Takeaways & Limitations
Robotic data collection may introduce biases from limited viewing distances and visible sensor supports that models could exploit.
Abstract
from arXiv · showhide
Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dataset by training a computer vision algorithm capable of predicting the caloric and macronutrient values of a complex, real world dish at an accuracy that outperforms professional nutritionists. Further we present a baseline for incorporating depth sensor data to improve nutrition predictions. We will publicly release Nutrition5k in the hope that it will accelerate innovation in the space of nutritional understanding.
1. Introduction
Visual nutrition tracking is difficult because existing tools require tedious ingredient and portion logging, while generic dishes require both ingredient recognition and portion estimation. Nutrition5k addresses this gap with richly annotated real-world dishes and demonstrates visual nutritional prediction, including depth-based improvements.
- Limitations of current tools: Existing nutrition tracking often requires users to log each ingredient and exact portion size, and visual portion estimation can be highly error-prone.Camera-based logging could reduce this effort for existing users and potentially expand adoption.
- Problem definition: Generic food requires estimating portion sizes as well as recognizing ingredients, making it substantially harder than classification alone.The term refers to dishes outside a predetermined set with known nutritional content.
- Data gap: Internet food datasets often lack accurate portion-size annotations, limiting their usefulness for learning absolute nutritional content.Many online images are styled for appeal rather than realistic representation, and recipe datasets commonly omit portion sizes.
- Dataset motivation: Nutrition5k contains 5,000 unique real-world dishes with videos, depth images, component weights, and high-accuracy nutritional annotations.Ingredient breakdowns are logged through recipes, enabling near-exact nutrition annotations.
- Proposed direction: Nutrition5k trains models to predict calories and macronutrients from images, with depth data used as an additional signal for portion-size and nutrition prediction.The dataset is intended to support nutritional understanding beyond known-dish classification.
2. Related Work
Prior work commonly classifies known dishes or retrieves recipe information but generally cannot estimate absolute nutrition for arbitrary mixed dishes. Other approaches estimate volume or mass, yet often operate on limited food variation or narrow food categories.
- Dish classification: Known-dish classifiers assume menu items have approximately stable nutritional profiles but cannot represent arbitrary generic dishes.Some systems scale to thousands of dishes, while multi-dish images require additional processing.
- Recipe-based methods: Recipe-web approaches use labels, retrieval, or embeddings but omit portion estimation, so they predict relative rather than absolute nutritional information.Without portion estimation, these methods cannot compute a dish’s absolute nutritional content from its image.
- Portion estimation: Geometric and segmentation-based methods estimate food volume or mass, but evaluations often use limited variation, whole foods, or few food categories.Examples include shape fitting, voxel-based volume estimation, and mass prediction for mostly single whole foods.
- Dataset comparison: Nutrition datasets are summarized by unique dish combinations and whether dishes contain mixed ingredients, distinguishing nutritional datasets by ingredient diversity and composition.Unique dishes differ in their underlying nutritional breakdown through ingredients or portion sizes.
3. Dataset
Nutrition5k is designed as a diverse, realistic, challenging, and highly annotated dataset for generic-food nutritional understanding, combining visual data with ingredient-level measurements. It contains roughly 5,000 dishes with nutritional labels, RGB and depth capture, and carefully separated training and test data.
- 3.1. Goals: Nutrition5k prioritizes diversity, realistic cafeteria conditions, challenging occlusions, and measurement accuracy to support generic-food estimation rather than fixed-menu classification.The dataset explicitly excludes replacing approaches that classify dishes from a known menu with known portion sizes.
- 3.2. Scale and Splits: Nutrition5k contains roughly 5,000 dishes spanning more than 250 ingredients, with calories ranging from a few to over 1,000 and up to 35 ingredients per plate.The dataset averages 5.7 ingredients per plate and includes substantial variation in portion size and dish complexity.
- 3.6. Challenges: Nutrition5k is the largest nutrition dataset with portion annotations, but collection at one cafeteria skews its food toward mostly Western-style dishes.The authors identify broader geographic and cultural coverage as future work, and note possible automation-related visual biases.
- 3.2. Scale and Splits: The dataset partitions 10% of dishes into Nutri-Test and keeps all dishes from each incremental scan in the same split to avoid overlapping plate images.This split respects the incremental scanning structure used during collection.
- 3.4. Supervision Labels: Each example provides ingredient-level supervision for total weight, macronutrients, and calories computed from ingredient weights and USDA nutrient data.The labels include total dish weight, carbohydrate, fat, protein, and calorie values derived from ingredient quantities.
- 3.3. Data Collection: The dataset combines RGB videos, overhead RGB-D images, incremental weight measurements, recipes, and ingredient annotations captured through a custom sensor array.Four side-angle cameras sweep around the dish while an overhead camera records RGB-D data; robotic automation triggers the sensors simultaneously.
4. Experiments
The experiments evaluate RGB, RGB-D, and volume-assisted models for predicting calories, mass, and macronutrients from food images. End-to-end nutrition prediction remains harder than portion-independent prediction, while depth-derived volume improves mass estimation.
- Experimental Setup: The study trains multi-task Inception-based models to predict calories, macronutrients, and optionally mass using mean absolute error.
- Nutrition Understanding from 2D Images: Direct prediction produced 26.1% calorie MAE and 31.9% aggregate macronutrient MAE, compared with 9.5% and 14.7% for portion-independent predictions.
- Depth and Volume for Portion Estimation: 18.8% calorie MAE and 20.9% aggregate macronutrient MAE resulted from directly augmenting RGB input with depth.
- Depth and Volume for Portion Estimation: 16.5% calorie MAE and 26.2% aggregate macronutrient MAE were achieved by combining volume-assisted mass prediction with portion-independent nutrition estimates.
- Depth and Volume for Portion Estimation: Volume estimation significantly improved mass regression, reducing relative error from 18.7% to 13.7% in the reported depth experiments.
5. Conclusion
Nutrition5k is presented as a large, diverse, richly labeled nutritional-understanding dataset, validated by models that estimate nutrition in generic food settings. The paper also positions depth data and public release as ways to advance future evaluation and innovation.
- Models trained on Nutrition5k can outperform professional nutritionists at caloric and macronutrient estimation for generic food.
- The paper introduces depth-based baselines that improve direct nutritional prediction over 2D images alone.
- The authors intend Nutrition5k's public release to encourage innovation and provide a benchmark for future nutritional-understanding techniques.
Appendix A. Additional Dataset Examples
The supplementary examples document Nutrition5k's diverse dish data, combining RGB video frames, depth data, and ground-truth annotations. Ingredient quantities are omitted from overview figures but shown in dedicated examples.
- Figures 9 and 10 show RGB frames, depth data, and ground-truth annotations for 12 unique dishes from roughly 5,000 dataset dishes.
- Ingredient-level annotations with quantities are omitted from the overview figures and illustrated separately in Figure 8 and Table 5.
Appendix B. Illustration Of Incremental Scanning
Nutrition5k collects data through incremental scans in which recipes are added one at a time and the full sensor suite records each resulting dish. Each scan supplies RGB video, depth, ingredient, and mass information while preserving scan relationships across the dataset splits.
- Each incremental scan adds one recipe, then captures RGB videos around the plate, depth data, and ingredient and mass annotations.
- Every incremental scan is represented as a unique dish in Nutrition5k, increasing dataset variety.
- Table 5 provides the ingredient annotations associated with the dish example shown in Figure 8.
- Figures 9–11 illustrate the dish-level RGB views, overhead depth imagery, and incremental scanning procedure used during collection.