PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Scale Visual Food Recognition

Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, Shuqiang Jiang

arXiv:2103.16107v3cs.CV

TL;DR

Food recognition lacks the large-scale datasets that have advanced generic visual recognition, limiting food-oriented representation learning. The paper introduces Food2K and a progressive region enhancement network, then reports effectiveness and broader generalization across food-related vision and multimodal tasks.

  • Problem

    Existing food recognition work mainly relies on medium- or small-scale datasets, while large-scale data are needed for advanced food visual representation learning.

  • Method

    The paper introduces Food2K and a deep progressive region enhancement network that learns local and global food features through progressive training and self-attention.

  • Results

    Food2K-trained networks show better generalization across food recognition, retrieval, cross-modal recipe retrieval, detection, and segmentation.

  • Takeaways & Limitations

    Food2K can serve as a large-scale fine-grained benchmark and as a resource for developing models for broader food-relevant tasks.

  • Takeaways & Limitations

    Food2K experiments still contain fine-grained confusions such as shrimp versus sausage pizza and pickle versus spicy stir-fried rice cakes.

Abstract

from arXiv · show

Food recognition plays an important role in food choice and intake, which is essential to the health and well-being of humans. It is thus of importance to the computer vision community, and can further support many food-oriented vision and multimodal tasks. Unfortunately, we have witnessed remarkable advancements in generic visual recognition for released large-scale datasets, yet largely lags in the food domain. In this paper, we introduce Food2K, which is the largest food recognition dataset with 2,000 categories and over 1 million images.Compared with existing food recognition datasets, Food2K bypasses them in both categories and images by one order of magnitude, and thus establishes a new challenging benchmark to develop advanced models for food visual representation learning. Furthermore, we propose a deep progressive region enhancement network for food recognition, which mainly consists of two components, namely progressive local feature learning and region feature enhancement. The former adopts improved progressive training to learn diverse and complementary local features, while the latter utilizes self-attention to incorporate richer context with multiple scales into local features for further local feature enhancement. Extensive experiments on Food2K demonstrate the effectiveness of our proposed method. More importantly, we have verified better generalization ability of Food2K in various tasks, including food recognition, food image retrieval, cross-modal recipe retrieval, food detection and segmentation. Food2K can be further explored to benefit more food-relevant tasks including emerging and more complex ones (e.g., nutritional understanding of food), and the trained models on Food2K can be expected as backbones to improve the performance of more food-relevant tasks. We also hope Food2K can serve as a large scale fine-grained visual recognition benchmark.

1 INTRODUCTION

Food2K addresses the shortage of large-scale food recognition data by introducing a high-quality benchmark with broad category and image coverage. The paper also proposes a progressive region enhancement network and evaluates Food2K models across food-related tasks.

  • Motivation: Existing food datasets are too small in categories and images for developing advanced food visual representation learning methods.Large-scale datasets have driven progress in generic image classification and understanding, motivating a comparable resource for food computing.
  • Food2K: Food2K contains 1,036,564 images across 2,000 categories and exceeds existing datasets by roughly one order of magnitude in both dimensions.The dataset includes categories spanning super-classes such as vegetables, meat, barbecue, and fried food.
  • Food2K: Food2K emphasizes quality through rigorous data cleaning, iterative annotation, and multiple professional inspections.The dataset is intended to support food image representation learning and understanding methods.
  • Method: The deep progressive region enhancement network learns diverse local and global features through progressive training and enhances local features with multi-scale self-attention.Progressive training targets complementary fine-grained information, while self-attention incorporates richer context into local representations.
  • Evaluation: Experiments report the effectiveness of the proposed method and better generalization of Food2K-trained networks across recognition, retrieval, recipe retrieval, detection, and segmentation.The paper also evaluates popular deep networks, fine-grained methods, and existing food recognition methods on Food2K.

2 RELATED WORK

Prior food datasets and food-computing studies span recognition, multimodal learning, retrieval, detection, segmentation, and generation, but existing recognition benchmarks remain limited in scale and coverage. The paper positions Food2K as a larger resource for scalable visual feature learning and fine-grained recognition.

  • Food-centric datasets: ETH Food-101, Vireo Food-172, and FoodX-251 provide substantially fewer images and categories than Food2K.Food-101 has 101,000 images and 101 categories; Vireo Food-172 has 110,241 images and 172 categories; FoodX-251 has 158,846 images and 251 categories.
  • Food-centric datasets: Existing food recognition datasets lack comprehensive category coverage and large image quantities, while some larger-category datasets have few images per category.The text identifies both category coverage and per-category image quantity as limitations of prior benchmarks.
  • Multimodal food learning: Recipe1M targets cross-modal embedding and retrieval between recipes and images, whereas Food2K targets scalable food visual feature learning.The two datasets therefore serve different learning objectives despite both being large-scale food-related resources.
  • Food computing: Food recognition supports applications including nutrient assessment, food logging, and self-service settlement, while food computing also includes detection, segmentation, retrieval, and generation.Recognition is described as a basic task supporting more complex food-relevant vision and multimodal tasks.
  • Food recognition challenges: Food images pose distinctive fine-grained challenges because ingredients can overlap, vary in appearance, and lack rigid spatial layouts.These properties distinguish food recognition from tasks built around more rigid visual structures.
  • Food recognition methods: Prior methods include handcrafted color, texture, and SIFT features as well as deep networks and discriminative-region approaches.The related work traces food recognition methods from handcrafted descriptors to deep learning and region-mining models.
  • Food analysis modalities: RGB-only food analysis may be insufficient for nutritional content prediction because it lacks volume information, motivating RGB-D benchmarks.Depth images can help estimate food volume for calorie and macronutrient prediction.

3 FOOD-2K DATASET

Food2K is constructed as a large, diverse, hierarchical food dataset through vocabulary building, image aggregation, retrieval, and professional quality inspection. It contains 2,000 categories and 1,036,564 images, exceeding prior food recognition datasets in scale.

  • Vocabulary and image collection: Food2K was collected from Meituan’s large raw image pool and organized into a vocabulary of 2,000 frequently used food categories.The construction process began with noisy and redundant labels, which were normalized, synonym-aggregated, manually verified, and used to retrieve additional images.
  • Dataset diversity: Food2K combines merchant and user-generated images and aggregates images from multiple original labels for the same food to increase diversity and preserve fine-grained distinctions.The dataset includes both eastern and western foods and uses label-image mappings together with visual retrieval to enlarge category coverage.
  • Annotation and quality control: 1,036,564 images across 2,000 classes were retained after duplicate removal, annotation, and professional quality inspection.Unqualified images included unclear images, occluded food, paintings, missing important ingredients, and images containing multiple categories.
  • Food ontology: Food2K uses a hierarchical ontology with 12 super-classes, sub-classes, dishes, and ingredient-level structure.Examples include Meat containing Beef and Pork, with dishes such as Curry Beef Brisket and Fillet steak.
  • Scale and coverage: Compared with ETH Food-101, Vireo Food-172, and ISIA Food-500, Food2K has an order-of-magnitude larger image scale and broader category distributions, including underrepresented foods such as barbecue.The comparison also reports more dishes per category than existing datasets.

4 OUR METHOD

PRENet combines global features with progressively learned multi-scale local features and self-attention-based region enhancement. Its training and inference procedures use stage-specific predictions and losses to improve food classification.

  • Global-Local Feature Learning: PRENet combines global superclass features, complementary multi-scale local features, and self-attention-based region enhancement for food recognition.Global and local predictions are fused for final classification.
  • Progressive Local Feature Learning: Progressive training moves from small receptive fields to larger surrounding regions, extracting fine-grained information such as ingredient-relevant features.Features from different stages provide multi-scale representations.
  • Progressive Local Feature Learning: KL divergence encourages different stages to attend to different areas, helping the network capture more visual details.Without this constraint, multi-scale features may focus on similar regions.
  • Region Feature Enhancement: Self-attention models relationships among local features, incorporating context across spatial positions and scales before producing enhanced local representations.The enhanced feature maps are concatenated and converted to a common dimension.
  • Optimization and Inference: The network is optimized end-to-end using cross-entropy losses and a KL-divergence term weighted by balance parameters α and β.The KL-divergence term increases differences between stage representations.
  • Optimization and Inference: At inference, scores from all stages, including the fused stage, are summed with equal weights to predict the output class.The stage predictions are treated as complementary.

5 EXPERIMENT

The experiments evaluate PRENet on Food2K and assess Food2K's generalization across five food-relevant vision and multimodal tasks. The study also discusses future research opportunities enabled by the dataset.

  • Evaluation Plan: The study first evaluates PRENet for food recognition on Food2K.This establishes the primary recognition experiment before broader task evaluations.
  • Evaluation Plan: Food2K's generalization is examined through food recognition, food image retrieval, cross-modal recipe retrieval, food detection, and food segmentation.The evaluation spans visual and cross-modal food-related tasks.
  • Future Research: The paper discusses potential research problems and techniques that could be investigated using Food2K.These discussions follow the recognition and generalization experiments.

5.1 Recognition on Food2K

On Food2K, PRENet is compared with deep, fine-grained, and food-recognition baselines, supported by ablations and visualizations. It also transfers to ETH Food-101 and exposes remaining fine-grained confusions.

  • Recognition on Food2K: PRENet outperforms ResNet50 by 2.24% in Top-1 and 1.47% in Top-5 classification accuracy on Food2K.It also exceeds PMG by 1.74% in Top-1 classification accuracy.
  • Recognition on Food2K: 2.24% and 1.47% are PRENet's reported gains over ResNet50 for Top-1 and Top-5 classification accuracy, respectively.PRENet also outperforms PMG by 1.74% in Top-1 classification accuracy.
  • Ablation Study: 81.45%, 82.11%, and 83.03% Top-1 classification accuracy are obtained as the learning stages increase from U = 1 to 3.The consecutive gains are 0.66% and 0.92%.
  • Ablation Study: The progressive learning strategy improves recognition, while combining it with region enhancement provides a further performance boost.The ablation compares progressive learning, region enhancement, their combination, and simple feature fusion.
  • Ablation Study: The cross-entropy-only setting reduces Top-1 classification accuracy by 3.18%, supporting the contribution of KL divergence.The reported explanation is that KL divergence makes multi-scale features focus on different areas.
  • Transfer Evaluation: 90.74% highest Top-1 classification accuracy is reported on ETH Food-101 against existing food-recognition methods, with further improvement using Food2K-pretrained PRENet.This evaluation uses the same setting for comparison.

5.2 Generalization Ability of Food2K

Food2K features generalization across food recognition, retrieval, cross-modal recipe retrieval, detection, and segmentation. Models initialized or fine-tuned with Food2K generally outperform counterparts trained on target datasets or ETH Food-101, with gains varying by task and dataset.

  • Food Recognition: Food2K features improved Top-1 accuracy across ETH Food-101, Vireo Food-172, and ISIA Food-500 compared with training on target datasets alone.Average gains were 1.68%, 3.51%, and 3.41%, respectively, for common neural networks.
  • Food Recognition: 7.33% Top-1 accuracy improvement was obtained for VGG16 on Vireo Food-172 after fine-tuning on Food2K.The corresponding gains for ResNet152, Inception V3, DenseNet161, and SENet154 were 2.58%, 2.31%, 2.61%, and 2.74%.
  • Food Image Retrieval: 5.28% average mAP improvement was achieved on Vireo Food-172 using Food2K-fine-tuned backbones for food image retrieval.Average improvements were 4.04% on ETH Food-101 and 4.16% on ISIA Food-500; gains were smaller when base performance approached saturation.
  • Cross-Modal Recipe Retrieval: Food2K backbones produced larger gains than ETH Food-101 backbones for cross-modal recipe retrieval on Recipe1M.Improvements were observed in both median retrieval rank and Recall@K.
  • Food Detection: 66.4% average mAP was reached on UNIMIB2016 with Food2K detector backbones, versus 65.5% with ETH Food-101.Food2K also improved mAP and AP75 across methods, while AP50 could be lower for some detectors because it magnifies certain proposal-labeling errors.
  • Food Segmentation: More than 2 points of improvement over ImageNet-pretrained models was obtained for DeepLabv3+ segmentation using Food2K.Visual comparisons on UEC-FoodPix Complete also reported more accurate food-region segmentation than competing models.

5.3 Discussion

The discussion positions Food2K as a platform for robust large-scale food recognition and transfer learning. It also identifies future directions spanning human evaluation, few-shot recognition, generation, richer annotation, and broader applications.

  • Large-Scale Robust Food Recognition: Existing fine-grained and food recognition methods that perform well on smaller datasets fail to achieve desired performance on Food2K.The authors associate this with more complex visual patterns from diverse ingredients, accessories, and arrangements, and use progressive training with self-attention as an initial approach.
  • Human Vision Evaluation: Human food-recognition evaluation may be complicated by regional and cultural bias among subjects.The discussion contrasts this with human-vision research on generic object recognition.
  • Cross-X Transfer Learning: Food2K supports proposed research on cross-cuisine and other cross-domain transfer learning because food has geographical and cultural attributes.The discussion specifically suggests transfer between eastern and western cuisines and across increasingly fine-grained scenarios.
  • Large-Scale Few-Shot Food Recognition: Large-Scale Few-Shot Food Recognition targets hundreds of novel food categories while retaining previously learned categories from only a few samples.Food2K provides a large-scale benchmark and a food ontology that can support method design for this task.
  • More Applications on Food2K: Food2K can support food image generation, including synthesis of realistic and semantically consistent images from ingredients and instructions.The discussion also mentions generation based on step-by-step cooking procedures and synthetic images produced with GANs.
  • Extension of Food2K: Future extensions include richer attribute, region-level, and pixel-level annotations, plus aesthetic labels for broader food-related tasks.The paper also encourages applying Food2K-trained models to additional food-relevant applications.

6 CONCLUSIONS

Food2K is presented as a large-scale, diverse benchmark for scalable food recognition and related vision and multimodal tasks. The conclusion highlights its potential for future few-shot recognition and transfer-learning research.

  • Conclusion: Food2K provides larger data volume, broader category coverage, and higher diversity than existing food datasets.The paper presents it as a benchmark for scalable food recognition and reports benefits for recognition, retrieval, detection, segmentation, and cross-modal recipe retrieval.
  • Conclusion: Food2K-trained models generalize across food vision and multimodal tasks, including recognition, retrieval, detection, segmentation, and cross-modal recipe retrieval.The conclusion also points to large-scale few-shot recognition and cross-scenario, cross-cuisine, and cross-super-class transfer learning as future directions.
Loading 2103.16107v3…