PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Benchmark for Food Image Segmentation

Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven C. H. Hoi, Qianru Sun

arXiv:2105.05409v1cs.CVcs.MM

TL;DR

Food segmentation must locate and classify individual ingredients, while existing datasets often lack fine-grained ingredient masks and food appearances vary substantially. The paper builds FoodSeg103 and FoodSeg154 and proposes recipe-based ReLeM; it reports applicability across segmentation frameworks and significant accuracy improvement on CNN-based CCNet.

  • Problem

    Food image segmentation must identify ingredient categories and pixel locations, but existing food datasets often lack ingredient-level masks and food appearances vary across and within ingredient classes.

    Source: §1 · p. 1

  • Method

    The paper constructs FoodSeg103 and FoodSeg154 with ingredient labels and masks, and proposes ReLeM, which integrates recipe language embeddings with food-image representations.

    Source: §1 · p. 2

  • Results

    ReLeM applies across CCNet, Sem-FPN and SeTR, and significantly improves accuracy when incorporated into CNN-based CCNet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FoodSeg103 and FoodSeg154 provide a challenging benchmark for semantic segmentation in food images.

    Source: §1 · p. 2

Abstract

from arXiv · show

Food image segmentation is a critical and indispensible task for developing health-related applications such as estimating food calories and nutrients. Existing food image segmentation models are underperforming due to two reasons: (1) there is a lack of high quality food image datasets with fine-grained ingredient labels and pixel-wise location masks -- the existing datasets either carry coarse ingredient labels or are small in size; and (2) the complex appearance of food makes it difficult to localize and recognize ingredients in food images, e.g., the ingredients may overlap one another in the same image, and the identical ingredient may appear distinctly in different food images. In this work, we build a new food image dataset FoodSeg103 (and its extension FoodSeg154) containing 9,490 images. We annotate these images with 154 ingredient classes and each image has an average of 6 ingredient labels and pixel-wise masks. In addition, we propose a multi-modality pre-training approach called ReLeM that explicitly equips a segmentation model with rich and semantic food knowledge. In experiments, we use three popular semantic segmentation methods (i.e., Dilated Convolution based, Feature Pyramid based, and Vision Transformer based) as baselines, and evaluate them as well as ReLeM on our new datasets. We believe that the FoodSeg103 (and its extension FoodSeg154) and the pre-trained models using ReLeM can serve as a benchmark to facilitate future works on fine-grained food image understanding. We make all these datasets and methods public at \url{https://xiongweiwu.github.io/foodseg103.html}.

1 INTRODUCTION

Food image segmentation must identify ingredients and their pixel locations, but food varies greatly in appearance and existing datasets lack fine-grained ingredient masks. The paper introduces FoodSeg103/FoodSeg154 and recipe-based ReLeM, reporting broad applicability and significant accuracy gains on CCNet.

1 INTRODUCTION

  • Food image segmentation recognizes each ingredient category and its pixel-wise location, unlike food image classification, which represents the image as a whole.The task is presented as relevant to food and health-related applications, including nutrition and calorie profiling.Source: §1 · p. 1
  • Food segmentation is difficult because the same ingredient can look different across dishes, different ingredients can look alike, and ingredient categories are long-tailed.The paper illustrates these appearance challenges with pineapple and potato examples.Source: §1 · p. 1, Figure 1 · p. 1
  • Existing food datasets often support dish classification or recipe generation rather than fine-grained ingredient segmentation, and the two public food-segmentation datasets use dish-level masks.Dish-level masks cover an entire dish rather than individual ingredients.Source: §1 · pp. 1–2
  • FoodSeg103 contains 7,118 western-food images with 103 ingredient classes; FoodSeg154 adds 2,372 Asian-food images and brings coverage to 154 classes and around 60,000 masks.The additional Asian-food set is used for out-domain testing; its 112 classes overlap 55% with FoodSeg103's classes.Source: §1 · p. 2
  • ReLeM transfers recipe knowledge by combining recipe language embeddings with food-image visual representations, connecting the representation of an ingredient across different dishes.The embeddings draw on ingredient labels and cooking instructions.Source: §1 · p. 2
  • ReLeM applies across CCNet, Sem-FPN and SeTR, and significantly improves accuracy when incorporated into CNN-based CCNet.The authors report that the approach works more efficiently on stronger models.Source: §1 · p. 2

2 RELATED WORKS

Earlier food datasets largely target recognition or recipe generation, while available segmentation datasets use dish-level rather than ingredient-level masks. This work situates FoodSeg alongside established segmentation approaches and evaluates CCNet, FPN, and SeTR.

Food Image Datasets

  • Prior food datasets mainly supported food recognition and recipe generation rather than segmentation into ingredient masks and labels.UECFoodPix and UECFoodPixComplete provide food segmentation data, but their masks are dish-wise.Source: §2 · p. 2
  • FoodSeg103 and FoodSeg154 add ingredient-wise segmentation to a field where prior food segmentation datasets used dish-wise masks.The paper describes FoodSeg154 as the first and largest ingredient-level dataset for fine-grained food image segmentation.Source: §2 · p. 2

Semantic Segmentation in Images

  • The study evaluates CCNet, FPN, and SeTR, and tests ReLeM with these representative segmentation methods.The paper situates these experiments among methods including FCN, DeepLab, PSPNet, and vision-transformer-based segmentation.Source: §2 · pp. 2–3

3 FOOD IMAGE SEGMENTATION DATASET

FoodSeg103 is built from recipe images and ingredient annotations, while FoodSeg154 adds an Asian-food subset. The dataset uses ingredient-level masks, refined labels, and comparisons with dish-level food segmentation datasets.

3 FOOD IMAGE SEGMENTATION DATASET

  • FoodSeg103 contains 7,118 Recipe1M images, while FoodSeg154 adds 2,372 Asian-food images used to evaluate domain adaptation.The Asian-food set is not currently public because of image confidentiality.Source: §3 · p. 3

3.1 Collecting Food Images

  • Image selection required at least two and no more than 16 ingredients, with ingredients visible and easy to annotate.The dataset construction process began with Recipe1M and narrowed its roughly 1.5k ingredient categories to 103 after refinement.Source: §3.1 · p. 3

3.2 Annotating Ingredient Labels and Masks

  • Annotators identified ingredients, assigned category labels, and drew pixel-wise masks; refinement corrected labels, removed categories used on fewer than five images, and merged visually similar categories.Tiny regions under 5% of an image were excluded.Source: §3.2 · p. 3
  • Examples include both clear ingredient boundaries and difficult images with overlapping regions and complex compositions.The complete category statistics were to be published with the dataset.Source: §3.2 · p. 3

3.3 Comparing with Food Image Datasets

  • FoodSeg103 and FoodSeg154 provide 40k and 60k pixel-wise masks, respectively, compared with 10k for the UEC datasets.FoodSeg masks label ingredients, whereas the UEC masks cover entire dishes.Source: §3.3 · p. 4, Figure 4 · p. 4
  • FoodSeg103 is a more challenging semantic-segmentation benchmark than UECFoodPix and UECFoodPixComplete, and its fine-grained annotations are more useful for nutrition and calorie analysis.Figure 3 presents category statistics for FoodSeg103 and the additional Asian-food set.Source: §3.3 · p. 4, Figure 3 · p. 4

4 FOOD IMAGE SEGMENTATION FRAMEWORK

The framework transfers recipe-derived knowledge into food-image features, then uses the trained vision encoder to initialize a segmentation model. ReLeM targets ingredient appearance variation, while the Segmenter predicts pixel-level masks.

4 FOOD IMAGE SEGMENTATION FRAMEWORK

  • The two-module pipeline trains ReLeM to incorporate recipe information into image features, then initializes the Segmenter encoder with its trained encoder and trains a randomly initialized decoder on masks.The framework describes this as multi-modality knowledge transfer.Source: §4 · pp. 4–5, Figure 6 · p. 5

4.1 Recipe Learning Module (ReLeM)

  • ReLeM uses recipe text and visual features to reduce differences between the same ingredients cooked in different ways, combining cosine-similarity and semantic losses.Recipe text includes ingredient labels and cooking instructions; the paper uses LSTM- or transformer-based text encoders.Source: §4.1 · pp. 5–6

4.2 Image Segmentation Module (Segmenter)

  • The Segmenter follows an encoder-decoder segmentation paradigm, represented by dilation-based CCNet, FPN, and transformer-based SeTR, with pixel-wise cross-entropy training.The framework describes dilation, feature-pyramid, and transformer designs as distinct encoder-decoder approaches.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate ReLeM with three segmentation frameworks on FoodSeg103 and test transfer to Asian food in FoodSeg154. ReLeM improves all three in-domain baselines, while results also compare task difficulty, qualitative predictions, and cross-domain performance.

5.1 Implementation Details

  • FoodSeg103 is split 7:3 for training and testing, and experiments use CCNet and FPN with ResNet-50 and SeTR with ViT-16/B.Recipe representations are trained on Recipe1M+ with FoodSeg103 test images hidden; the models use 80k segmentation iterations and ReLeM uses a two-stage optimization strategy.Source: §5.1 · pp. 6–7

5.2 Results and Observations

  • On FoodSeg103, ReLeM improves CCNet, FPN, and SeTR by 1.3%, 1.3%, and 2.6%, respectively; across all model configurations, LSTM-based ReLeM consistently outperforms transformer-based ReLeM.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Comparing FoodSeg103 with Cityscapes

  • All three baseline methods perform satisfactorily on Cityscapes but drop substantially on FoodSeg103, which the authors interpret as evidence that food segmentation is more difficult.Table 4 compares segmentation results on Cityscapes and FoodSeg103.Source: §5.3 · p. 7, Table 4 · p. 8

5.5 Cross-Domain Evaluation

  • On Asian-food evaluation, ReLeM outperforms the baselines both without fine-tuning on the Asian training split and after fine-tuning.Without fine-tuning, evaluation covers 62 classes overlapping FoodSeg103; after fine-tuning, it covers all 112 classes.Source: Table 5 · p. 8, §5.5 · p. 8

5.4 Qualitative Examples

  • ReLeM-CCNet produces more accurate and detailed predictions than vanilla CCNet in the shown examples, but fails on an example with unclear ingredient boundaries.Figure 10 presents qualitative results on the FoodSeg103 test set.Source: §5.4 · pp. 7–8, Figure 10 · p. 8

6 CONCLUSIONS

The paper introduces FoodSeg103 and its FoodSeg154 extension, and proposes ReLeM for recipe-informed pre-training. Experiments evaluate the method on in-domain and cross-domain food segmentation settings.

6 CONCLUSIONS

  • FoodSeg103 and FoodSeg154 comprise around 10,000 images and 60,000 masks covering 154 ingredients.The paper presents the datasets as a benchmark for food image segmentation.Source: §6 · p. 8
  • ReLeM transfers multimodal recipe information into segmentation-model pre-training and is evaluated with three baseline semantic segmentation methods.Experiments use FoodSeg103 in a typical setting and FoodSeg154 in a cross-domain setting.Source: §6 · p. 8
  • The experiments assess ReLeM across FoodSeg103 and FoodSeg154, including a challenging cross-domain setting.Source: §6 · p. 8

APPENDIX: MORE DETAILS OF FOODSEG103 AND FOODSEG154 7.1 Statistics

The appendix describes dataset splits and category structure, shows additional annotation examples, and examines transformer-based segmentation results. Those results vary by model and decoder, and larger backbones do not guarantee better performance.

APPENDIX: MORE DETAILS OF FOODSEG103 AND FOODSEG154 7.1 Statistics

  • FoodSeg103 has 103 ingredient categories grouped into 15 super categories, and the appendix describes in-domain and Asian-set splits for evaluation.FoodSeg103 is used for in-domain training and testing; the Asian set is used for out-domain evaluation.Source: §7.1 · p. 10, Table 6 · p. 10, Table 7 · p. 10

7.3 Analysis on Transformer-based Models

  • ReLeM improves PVT and ViT-Naive by 0.7% and 2.6%, respectively, but the baseline performs better with ViT-MLA.The authors note that larger backbones can hurt performance and identify food-aware decoders and multi-level recipe learning as future directions.Source: Table 8 · p. 10, §7.3 · p. 10

7.2 Visualization

  • Additional examples pair source images with their corresponding FoodSeg103 annotation masks.Figure 11 places source images on the left and masks on the right.Source: §7.2 · p. 10, Figure 11 · p. 15

Related papers

Loading 2105.05409v1…