Source-linked AI summary
Two-view 3D Reconstruction for Food Volume Estimation
Joachim Dehais, Marios Anthimopoulos, Sergey Shevchik, Stavroula Mougiakakou
TL;DR
Traditional meal records are often inaccurate, with portion-size estimation a prominent source of error. The paper presents an automatic dense two-view method for estimating food volume. Across the evaluated datasets, it reports 8.2%–9.8% overall mean absolute percentage error in roughly 5.5 seconds.
Problem
Traditional meal records have questionable accuracy, and food portion-size estimation is their main reported source of error, averaging above 20%.
Source: §I · p. 1Method
The paper estimates food volume using a highly automatic dense two-view reconstruction method with minimal assumptions about food shape or type.
Source: B. Food Volume Estimation · p. 2Results
Overall mean absolute percentage error ranged from 8.2% to 9.8% across datasets, with over 90% of volume-estimate errors under 20%, in roughly 5.5 seconds.
Source: §VI · p. 9Takeaways & Limitations
The system is currently part of a carbohydrate-counting system for individuals with diabetes and may also be used for general food assessment.
Source: §VI · p. 9
Abstract
from arXiv · showhide
The increasing prevalence of diet-related chronic diseases coupled with the ineffectiveness of traditional diet management methods have resulted in a need for novel tools to accurately and automatically assess meals. Recently, computer vision based systems that use meal images to assess their content have been proposed. Food portion estimation is the most difficult task for individuals assessing their meals and it is also the least studied area. The present paper proposes a three-stage system to calculate portion sizes using two images of a dish acquired by mobile devices. The first stage consists in understanding the configuration of the different views, after which a dense 3D model is built from the two images; finally, this 3D model serves to extract the volume of the different items. The system was extensively tested on 77 real dishes of known volume, and achieved an average error of less than 10% in 5.5 seconds per dish. The proposed pipeline is computationally tractable and requires no user input, making it a viable option for fully automated dietary assessment.
I. INTRODUCTION
Diet-related disease makes accurate dietary assessment important, but traditional meal records are often inaccurate, especially in portion-size estimates. The paper situates its two-view food-volume method within image-based dietary assessment and passive 3D reconstruction.
I. INTRODUCTION
- Traditional meal records have questionable accuracy, with food portion-size estimation identified as their main reported source of error and averaging above 20%.The paper motivates computer-vision tools for dietary support amid the prevalence of diet-related chronic diseases.Source: §I · p. 1
- The proposed method addresses food portion estimation with an automatic image-based approach and evaluates algorithmic choices against state-of-the-art methods on real food data.Volume estimation is crucial in the four-stage image-analysis pipeline because detection and recognition may be semi-automatic or manual, while nutrient assessment is often a database lookup.Source: §I · p. 1
A. 3D Reconstruction from Images
- Passive multiview reconstruction can estimate depth from images using simple cameras, whereas single-view reconstruction requires strong assumptions about 3D shape.Active multiview methods emit a signal and process its reflection, but require special hardware.Source: A. 3D Reconstruction from Images · p. 2
- Extrinsic calibration establishes view geometry, producing sparse 3D points; dense reconstruction then uses image pixels, with stereo matching aided by rectification.Rectification aligns corresponding points onto the same image row to simplify matching.Source: A. 3D Reconstruction from Images · p. 2
B. Food Volume Estimation
Earlier food-volume methods used single-view shape models or multiview reconstruction, often with substantial user input or capture requirements. This work contributes a highly automatic dense two-view method and an investigation of its choices, while specifying input conditions that constrain its operating range.
B. Food Volume Estimation
- Earlier single-view methods relied on food shape models and user input, while other approaches required extensive images, difficult user manipulations, or careful video capture and a dish-sized reference card.The DietCam approach used three meal images and sparse reconstruction; Puri et al. used video, three extracted images, and two large reference patterns.Source: B. Food Volume Estimation · p. 2
- The study contributes an accurate, efficient, highly automatic dense two-view method with minimal assumptions about food shape or type, plus investigations of algorithmic choices and comparisons on real food data.The method estimates food volume for multi-food meals with unconstrained 3D shape.Source: B. Food Volume Estimation · pp. 2–3
III. METHODS
- The system estimates volume through extrinsic calibration, dense reconstruction, and volume extraction from two meal images, a reference card, and segmented food and dish.The dish must be elliptical with a flat bottom; segmentation may be performed automatically.Source: §III · p. 3
- Reliable operation requires well-exposed, focused images, widespread variable texture with limited specular reflection, and an appropriate range of motion between views.Textureless foods such as yoghurt may fail; phone motion sensors can guide users toward suitable capture angles.Source: §III · p. 3
A. Extrinsic Calibration
Extrinsic calibration estimates the geometry and scale of the two views through salient-point matching, relative-pose extraction, and scale extraction. The pose stage builds on RANSAC with local optimization and adaptive thresholding.
A. Extrinsic Calibration
- Calibration comprises salient-point matching, relative-pose extraction, and scale extraction because smartphone views lack fixed relative positions.The true scale is subsequently obtained from matches to a reference card.Source: A. Extrinsic Calibration · p. 3
1) Salient point matching
- SURF was selected for its balance of efficiency and accuracy, and symmetric top-match intersections are used to match salient points between images.The matches are obtained with symmetric k-nearest-neighbor search using hierarchical search on k-d trees.Source: 1) Salient point matching · p. 3
2) Relative pose extraction
- The relative pose is estimated with five-match RANSAC and refined by Levenberg–Marquardt, while the modified RANSAC adds local optimization and adaptive thresholds.Local optimization resamples inliers and scores models to favor high inlier counts and small distances; adaptive thresholds use a noise distribution from randomly matched points and an FDR bound.Source: 2) Relative pose extraction · pp. 3–4
3) Scale estimation
The reference card supplies the true scale for the sparse reconstruction.
3) Scale estimation
- The scale is estimated by matching image points to the flat reference card, comparing sparse-cloud distances with corresponding pattern distances, and selecting the modal ratio.A RANSAC homography links the first image to the reference pattern.Source: 3) Scale estimation · p. 4
B. Dense Reconstruction
Dense reconstruction rectifies the two images, matches pixels along corresponding rows, and converts those matches into a point cloud. The matching pipeline adapts the disparity search range and combines local costs into a smooth, multiscale solution.
B. Dense Reconstruction
- The reconstruction rectifies the images, performs stereo matching, then generates a point cloud from dense matches.Figure 5 depicts rectification, the disparity map, and the resulting unprojected point cloud.Source: B. Dense Reconstruction · p. 4, Figure 5 · p. 5
1) Rectification
- Rectification aligns the views in a consistent radial coordinate system and restricts dense matching to corresponding rows.If more than half of salient-point match pairs have inverted horizontal order, the second rectified image is mirrored.Source: 1) Rectification · p. 4
2) Stereo Matching
- Stereo matching compares image patches along corresponding rows to produce a disparity map, using Census descriptors and a dynamically extracted disparity range.The range uses the central 90% of sparse-match disparities, enlarged by a fixed amount; Census compares local binary patterns with Hamming distance and is robust to monotonic intensity changes.Source: 2) Stereo Matching · p. 5
- Matching uses fixed-size square windows, hierarchical dynamic programming with variance-linked smoothness, and median-filter refinement.The hierarchical search uses larger search spaces on smaller image pairs and searches at multiple scales; images are resized to a fixed dish size to address scale differences.Source: 2) Stereo Matching · p. 5
3) Point cloud generation
Dense matches are returned to original image coordinates and unprojected using the relative pose to form 3D points. The food surface is closed with a dish surface before volume is integrated.
3) Point cloud generation
- Dense matches are mapped back through the inverse rectification and unprojected using relative pose to generate 3D points and a depth map.The depths are integrated into a depth map for the first image.Source: 3) Point cloud generation · p. 5
C. Volume Estimation
- Before volume calculation, the food surface is closed into a watertight shape by generating a surface that lines the dish bottom.Figure 6 shows the food and plate surfaces and the food surface’s projection onto the dish surface.Source: C. Volume Estimation · p. 5, Figure 6 · p. 5
1) Food Surface Extraction
The method extracts separate food surfaces, constructs a dish surface using its rim and a reference card, then integrates food height above that surface. Evaluation uses several datasets of meals captured in smartphone image pairs with registered depth-based ground truth.
1) Food Surface Extraction
- The food surface is sampled on a regular grid, triangulated in image coordinates, and separated into distinct surfaces by discarding triangles that cross segments.The segmentation map first removes the background.Source: 1) Food Surface Extraction · p. 5
2) Dish Surface Extraction
- Because reflective dishes are difficult to reconstruct passively, the dish surface combines a densely matched rim plane with height information from a reference card.The rim plane is fit with RANSAC, and the reference card defines the table plane used to place the dish plane at a fixed height.Source: 2) Dish Surface Extraction · pp. 5–6
3) Volume Calculation
- Volume is the sum over food-surface triangles of horizontal area multiplied by the average height of their corners above the dish surface.The dish-surface normal defines the vertical direction for the integration.Source: 3) Volume Calculation · p. 6
A. Dataset
- The evaluation uses Meals-45, Angles-13, Plates-18, and Meals-14 to examine meal coverage, shooting angle, plate changes, food types, and generalization.Meals-45 contains 45 dishes and 113 items; the other datasets contain 13, 18, and 14 dishes, respectively.Source: A. Dataset · p. 6
IV. DATA AND EXPERIMENTAL SETUP
- Each dish has three smartphone image pairs captured at 40–60 cm and within 45 degrees of vertical, with registered depth-sensor point clouds used to derive ground-truth volume.Food items were manually segmented, and the table plane was extracted with RANSAC before ground-truth volume was obtained.Source: A. Dataset · p. 6
B. Evaluation Protocol for Food Volume Estimation
The protocol evaluates volume-estimation accuracy and stability over repeated runs and reports speed. The implementation uses C++ without parallelism on a specified Linux desktop system.
B. Evaluation Protocol for Food Volume Estimation
- Each food item receives 24 estimates from eight system runs over three image pairs; MAPE measures accuracy and CV measures precision or stability.Overall MAPE is calculated across all food items, and speed is also reported.Source: B. Evaluation Protocol for Food Volume… · p. 6
C. Implementation
- The methods run in C++ without parallelism under Linux on an Intel i7-3770K CPU with 8GB of RAM.Salient-point detection and matching use OpenCV 2.4.10 and openSURF.Source: C. Implementation · p. 6
- Table II compares performance for three major salient-point detector/descriptors.Source: Table II · p. 7
V. RESULTS AND DISCUSSION
The evaluation benchmarks choices across extrinsic calibration and reconstruction, then reviews the tuned system. SURF and background exclusion are among the reported choices that improve the balance of accuracy and speed.
V. RESULTS AND DISCUSSION
- The results section benchmarks algorithm and parameter choices on Meals-45, then evaluates shooting angle, plate effects, generalization, and comparisons with other methods.Changes below 0.2% in average error are treated as negligible because of RANSAC randomness.Source: §V · p. 7
A. Extrinsic Calibration
- The relative-pose experiments are reported as a separate calibration benchmark, with Table III presenting performance for different relative-pose choices.The supplied passages do not state the comparative outcomes.Source: A. Extrinsic Calibration · p. 7, Table III · p. 7
1) Salient Point Detection/Description
- SURF reduced error by 1.4% versus SIFT while running considerably faster, whereas ORB was fastest but had the highest error.Enforcing distinctiveness between consecutive matches improved system accuracy by 0.3%, but large distinctiveness filters removed valid matches.Source: 1) Salient Point Detection/Description · p. 7, Table I · p. 7
- Excluding the background from the matching region increased speed by 22% and reduced error from 9.6% to 8.2%; Census was the best-performing matching cost.SAD and ZNCC increased error by 4.5% and 5.5%, respectively.Source: 1) Salient Point Detection/Description · p. 8
C. Volume Extraction
The tuned system reports an 8.2% overall MAPE on Meals-45, with additional evaluation of error distributions, camera angle, and plate effects. The reported stable angle range is 15–25 degrees, and average error varies by less than 2.5% across tested plates.
C. Volume Extraction
- Mesh error decreased as reconstructed-surface resolution increased up to 212; the passage notes comparatively low performance at higher resolutions.Table IV compares different 3D mesh sizes.Source: C. Volume Extraction · p. 8
D. Evaluation of the Proposed System
- 8.2% MAPEoverall was achieved on Meals-45; 69% of signed errors were within ±10%, and the average per-item coefficient of variation was 7.1%.The error distribution and coefficient-of-variation distributions are shown in Fig. 8.Source: D. Evaluation of the Proposed System · p. 8, Figure 8 · p. 8
- 15–25 degrees was identified as the relative-angle range for stable executions; smaller angles ease matching but lower depth accuracy, while larger angles reverse that tradeoff.The authors note that enforcing camera-configuration properties could yield better results than those reported for Meals-45.Source: D. Evaluation of the Proposed System · p. 9
3) Plate effect
- Average error varied by less than 2.5% across the three tested plates, and performance was close to that on Meals-45.The authors interpret these results as robustness to plate style and transfer to the tested dataset.Source: 3) Plate effect · p. 9, Table VII · p. 9
4) Testing on a different dataset
On Meals-14, which contains foods from sources not represented in earlier datasets, the method achieved 9.8% MAPEoverall. The conclusion reports 8.2%–9.8% overall error across datasets and identifies improvements to food-container modeling as future work.
4) Testing on a different dataset
- 9.8% MAPEoverall on Meals-14 was a small increase from 8.2% on Meals-45, with results described as stable on foods from different sources.The authors say the method approximates these foods' volumes without being tuned on them.Source: 4) Testing on a different dataset · p. 9
5) Comparison to other methods
- The first two comparison approaches had average error more than twice the proposed method on Meals-45; the alternative dense method was also inferior.Single-view and sparse 3D methods were generally faster, but the authors describe their low accuracy as prohibitive for this application.Source: 5) Comparison to other methods · p. 9, Table VI · p. 9
VI. CONCLUSION
- 8.2%–9.8% overall mean absolute percentage error was reported across datasets, with over 90% of volume-estimate errors under 20% in roughly 5.5 seconds.The conclusion also reports that the system is part of a carbohydrate-counting system for people with diabetes.Source: §VI · p. 9
- The authors identify more complex food-container models, such as for bowls, or reconstructing the container from images as possible improvements.These are proposed extensions to the current system.Source: §VI · p. 9