Source-linked AI summary
Grab, Pay and Eat: Semantic Food Detection for Smart Restaurants
Eduardo Aguilar, Beatriz Remeseiro, Marc Bolaños, Petia Radeva
TL;DR
Self-service restaurants need food-tray analysis that can identify multiple foods for nutritional information and billing. Semantic Food Detection combines segmentation, localization, and recognition, and the paper reports improved state-of-the-art results in recall and mean average accuracy, with predictions taking under 0.5 seconds.
Problem
Food-tray analysis must identify multiple foods in images despite mixed foods, distortions, lighting changes, and non-food objects, supporting nutritional analysis and self-service restaurant billing.
Source: §I · pp. 1–2Method
Results
The method significantly outperforms the state of the art in recall and mean average accuracy, and predicts all foods in an image in less than 0.5 seconds on a low-performance GeForce 940MX GPU.
Source: §V · p. 9Takeaways & Limitations
In semi-automatic billing, the cashier would need only small changes to produce the final bill when the model does not recognize the whole tray well.
Source: §V · p. 9
Abstract
from arXiv · showhide
The increase in awareness of people towards their nutritional habits has drawn considerable attention to the field of automatic food analysis. Focusing on self-service restaurants environment, automatic food analysis is not only useful for extracting nutritional information from foods selected by customers, it is also of high interest to speed up the service solving the bottleneck produced at the cashiers in times of high demand. In this paper, we address the problem of automatic food tray analysis in canteens and restaurants environment, which consists in predicting multiple foods placed on a tray image. We propose a new approach for food analysis based on convolutional neural networks, we name Semantic Food Detection, which integrates in the same framework food localization, recognition and segmentation. We demonstrate that our method improves the state of the art food detection by a considerable margin on the public dataset UNIMIB2016 achieving about 90% in terms of F-measure, and thus provides a significant technological advance towards the automatic billing in restaurant environments.
I. INTRODUCTION
The section motivates food-tray analysis for nutrition monitoring and faster self-service billing, frames multi-food tray images as challenging, and introduces Semantic Food Detection as a unified approach.
I. INTRODUCTION
- Food-image analysis can support nutrition monitoring, while self-service restaurant recognition could enable automatic billing and faster service.Automatic billing would remove the need for manual selection of the dishes a customer chose.Source: §I · p. 1
- Tray analysis must handle multiple foods, mixed foods in one dish, visual distortions and lighting changes, and non-food objects; intraclass variability is less influential than in traditional analysis.Source: §I · p. 2
- Semantic Food Detection integrates food segmentation with food-object detection to combine pixel and bounding-box localization with food recognition, aiming to prevent false detections.Source: §I · p. 2
- About 90% F2-score is reported; the method outperforms state-of-the-art methods by more than 10% in recall and more than 20% in mean average accuracy.The reported comparison is with the state-of-the-art food-tray method.Source: §I · p. 2
II. RELATED WORK
Related work covers food detection and recognition, extensions using context, and localization and segmentation for multi-food images. The section contrasts these lines with the limited prior work on food-tray analysis.
II. RELATED WORK
- Binary food detection distinguishes food-present from food-absent images; CNN-based approaches have improved this task and have also served image indexing and recognition preprocessing.The passage reports an approximately 4% overall-accuracy improvement for a CNN method over an earlier approach.Source: §II · p. 2
- CNN-based food recognition reached about 90% accuracy on datasets with around 100 food types, with approaches differing across fine-grained and high-level categories.For high-level categories, combining GoogLeNet and ResNet50 outputs achieved a 10% accuracy improvement over the baseline.Source: §II · pp. 2–3
- Restaurant-oriented recognition has incorporated online restaurant information,, menus, locations, and user images, as well as contextual relations among elements.Source: §II · p. 3
- Most recognition methods and datasets focus on single-dish images [18],,, while localization extracts food regions and segmentation classifies food pixels.The cited review identifies as the only known localization approach that does not segment food before extracting boxes.Source: §II · p. 3
- The prior food-tray approach combined saturation and color-texture segmentation with SVM classification, whereas this method runs learned segmentation and detection in parallel rather than sequentially.The earlier approach used generic segmentation methods and required several sequential steps; parallel processing is presented as faster.Source: §II · p. 3
III. SEMANTIC FOOD DETECTION
Semantic Food Detection combines food-versus-background segmentation with food localization and recognition. Its segmentation branch predicts a binary food image and converts regions into bounding boxes.
III. SEMANTIC FOOD DETECTION
- The proposed food-tray pipeline integrates food-versus-non-food semantic segmentation with food localization and recognition.Source: §III · p. 3
A. Food Segmentation
- Food segmentation predicts a binary image separating food and food-related items from the tray and background, using supervised semantic segmentation rather than image-property methods.Directly assigning the many food categories to pixels is described as difficult, since recent methods perform well on datasets with relatively few classes.Source: A. Food Segmentation · p. 3
- FCNs provide same-size output for arbitrary-size input images; the Tiramisu model is described as an FCN with downsampling and upsampling paths, skip connections, and dense blocks.Source: A. Food Segmentation · p. 4
- The segmentation stage traces exterior food-region boundaries while discarding holes, then forms bounding boxes and removes small regions as likely false positives.The Moore-Neighbor tracing algorithm with Jacob’s stopping criteria is used for boundary tracing.Source: A. Food Segmentation · p. 4
- Figure 4 shows the segmentation outputs in sequence: an FCN-predicted binary image, region boundaries, and food bounding boxes.Source: Figure 4 · p. 4, A. Food Segmentation · p. 4
B. Food Detection
The food detector adapts YOLOv2 to predict food boxes and classes. Semantic Food Detection integrates detection with segmentation to address duplicate detections and confusion with background.
B. Food Detection
- YOLOv2 is retrained for food detection, predicting bounding boxes and class probabilities with one convolutional network in real time.The model uses a modified Darknet-19 network.Source: B. Food Detection · p. 4
- All dishes were detected at threshold 1/65, but YOLOv2 produced eight false detections, mostly duplicates around plates.One false detection was entirely in the background.Source: B. Food Detection · p. 4
C. Semantic Food Detection
- Semantic Food Detection integrates object detection with segmentation to target duplicate-detection errors and errors caused by confusion with background.The proposal focuses on reducing these two common object-detector errors.Source: C. Semantic Food Detection · p. 4
1) Background Removal:
Background Removal estimates whether detected boxes belong to the background using segmentation-derived boxes and contours, then removes detections above a probability threshold.
1) Background Removal:
- The procedure estimates each detection box’s background probability using segmentation boxes and contours.The passage describes the probability of a box being a false detection based on segmentation boxes and whether it intersects a contour.Source: 1) Background Removal · p. 4
- Boxes with a background probability above 50% are removed as false detections.The stated criterion is Pr(Bkg|bY_i) > 0.5.Source: 1) Background Removal · p. 5
- After Background Removal, false detections around mandarine objects were removed in the illustrated example.The result is shown in Fig. 5(c).Source: 1) Background Removal · p. 5, Figure 5 · p. 5
2) Non-Maximum Suppression:
Non-Maximum Suppression greedily removes duplicate detections after Background Removal while retaining the remaining well-localized and classified foods.
2) Non-Maximum Suppression:
- After Background Removal, detections are ranked by class-specific confidence and lower-ranked boxes are removed when their maximum overlap with a selected box exceeds 50%.The procedure is applied separately within each class.Source: 2) Non-Maximum Suppression · p. 5
- Non-Maximum Suppression removes the remaining false detections while keeping foods well localized and classified in the illustrated example.The outcome is shown in Fig. 5(d).Source: 2) Non-Maximum Suppression · p. 5
IV. EXPERIMENTAL RESULTS
The experimental section introduces the evaluation dataset, measures, methods, and model configurations.
IV. EXPERIMENTAL RESULTS
- The evaluation uses images from self-service restaurants and presents results for different methods and model configurations.The section also describes the evaluation measures.Source: §IV · p. 5
A. Dataset
UNIMIB2016 is a self-service-canteen dataset of food trays, with annotations for food-item boundaries, boxes, and labels. Experiments use 1,010 images and 65 categories, split into training and test sets.
A. Dataset
- 1,010 images and 65 food categories were used from UNIMIB2016, which contains 1,027 images and 73 categories.The excluded images and categories were omitted due to low sample counts, as suggested in.Source: A. Dataset · p. 5
- The experimental split contains 650 training images and 360 test images.These are approximately 64% and 36% of the 1,010-image experimental set, respectively.Source: A. Dataset · p. 5
- Each food item is annotated with a polygon boundary, bounding box, and food label.The dataset was collected in a self-service canteen, with images acquired in a semi-controlled setting using a Samsung Galaxy S3.Source: A. Dataset · p. 5
B. Food Segmentation
The food-segmentation experiments compare Tiramisu-based networks with alternative segmentation approaches using pixel-wise and region-based measures. The Tiramisu networks outperform Classic Upsampling, and all FCNs outperform the two methods reported in on region-based measures.
B. Food Segmentation
- The evaluation uses pixel accuracy and class-mean IoU alongside region-based covering, rank index, and variation of information.The region-based measures were included to enable comparison with; lower variation of information is better.Source: B. Food Segmentation · pp. 5–6
- The segmentation comparison includes Tiramisu56, Tiramisu67, Tiramisu103, Classic Upsampling, JSEG, and the pipeline in.Tiramisu models benefit from dense blocks and, in general, greater depth and parameter count; Tiramisu67 provides a favorable depth-performance trade-off.Source: B. Food Segmentation · p. 6
- All FCNs outperform the two approaches in on region-based measures, while Tiramisu networks outperform Classic Upsampling on pixel-wise measures.Tiramisu67 achieves the best results, while Tiramisu103 overfits on this binary task.Source: B. Food Segmentation · p. 6
C. Semantic Food Detection Performance
Semantic Food Detection is evaluated for recognition and localization using standard food-recognition measures and comparisons with YOLOv2 and Ciocca et al.. On the test set, it improves Recall and MAA over, while tray-level outcomes and threshold effects qualify the comparison.
C. Semantic Food Detection Performance
- The minimum confidence threshold was selected because it gives the best Recall, MAA, and TA and lets segmentation discard false positives.At that threshold, the training-set comparison reports about 40% higher precision with only 0.1%–0.2% decreases in the other indicators; at the maximum threshold, results are practically identical.Source: C. Semantic Food Detection Performance · p. 7, Table II · p. 7
- Against the state-of-the-art method (Ciocca et al.), the proposal reports 12.4% higher Recall and 20.9% higher MAA, alongside a 2.5% decrease in TA.The authors note that TA does not penalize false positives and that tray food quantities vary from 1 to 9.Source: C. Semantic Food Detection Performance · p. 7
- With ground-truth segmentation, the proposal remains about 16% better in MAA than and has lower bias toward dominant classes.Using ground-truth segmentation produces no significant improvement over the proposal's own semantic segmentation, which refines object-detection predictions.Source: C. Semantic Food Detection Performance · p. 7
- TA tends to fall as tray food count rises, but Recall has no clear trend across counts.For trays with six foods, TA is 0.478; 17 misclassified objects across 12 trays average 1.42 errors per incorrectly classified tray.Source: Figure 7 · p. 7, C. Semantic Food Detection Performance · p. 8
- 1.09 errors per incorrectly classified tray are reported for the proposal, compared with 3.33 for.The proposal misclassified 85 trays versus 76 for, whose incorrectly classified trays averaged about three times as many wrongly classified objects.Source: C. Semantic Food Detection Performance · p. 8
- Reported error sources include out-of-class or off-tray items, closely placed foods, ambiguous foods, and confusions among cakes, meats, and pastas.Examples include fresh cheese, pane, mandarins, and pudding.Source: C. Semantic Food Detection Performance · p. 8, Figure 8 · p. 8
V. CONCLUSION
Semantic Food Detection combines segmentation, localization, and recognition for food-tray analysis. It significantly outperforms the state of the art in recall and mean average accuracy, with reported speed and billing-relevant results.
V. CONCLUSION
- Semantic Food Detection combines semantic segmentation, localization, and recognition for food-tray analysis.The system also uses food/non-food segmentation through FCNs and non-maximum suppression to avoid false detections.Source: §V · p. 8
- The proposal significantly outperforms the state of the art in recall and mean average accuracy.Source: §V · p. 9
- The model is less sensitive to class imbalance, averages about one food error when it fails to recognize a whole tray, and predicts all foods in under 0.5 seconds on a low-performance GPU.The authors note the error level is relevant to semi-automatic billing, where a cashier would make small changes to produce the final bill.Source: §V · p. 9