Source-linked AI summary
FoodNet: Recognizing Foods Using Ensemble of Deep Networks
Paritosh Pandey, Akella Deepthi, Bappaditya Mandal, N. B. Puhan
TL;DR
Food recognition is challenging because food appearance varies with preparation, ingredients, lighting, and viewpoint, and food lacks a distinctive spatial layout. The paper combines features from fine-tuned deep CNNs in an ensemble, which outperforms the compared methods across both food databases and ranks. It also contributes an Indian food database for further evaluation and enrichment.
Problem
Food recognition is difficult because preparation, ingredients, lighting, and viewpoint produce substantial appearance variation, and food lacks a distinctive spatial layout.
Source: §I · p. 1Method
The paper uses a multi-layered ensemble of three fine-tuned deep CNN subnetworks, combining their features into robust discriminative representations.
Source: §IV · pp. 4–5Results
The Ensemble Net outperforms the other current state-of-the-art methods for all ranks on both ETH Food-101 and the Indian Food Database.
Source: §IV · p. 5Takeaways & Limitations
The Indian Food Database is contributed for further evaluation and enrichment.
Source: §IV · p. 5
Abstract
from arXiv · showhide
In this work we propose a methodology for an automatic food classification system which recognizes the contents of the meal from the images of the food. We developed a multi-layered deep convolutional neural network (CNN) architecture that takes advantages of the features from other deep networks and improves the efficiency. Numerous classical handcrafted features and approaches are explored, among which CNNs are chosen as the best performing features. Networks are trained and fine-tuned using preprocessed images and the filter outputs are fused to achieve higher accuracy. Experimental results on the largest real-world food recognition database ETH Food-101 and newly contributed Indian food image database demonstrate the effectiveness of the proposed methodology as compared to many other benchmark deep learned CNN frameworks.
I. INTRODUCTION AND CURRENT APPROACHES
Food recognition is difficult because food varies in appearance and lacks a distinctive spatial layout. The paper situates its CNN approach among earlier handcrafted-feature and food-recognition methods, and introduces a new Indian food database.
I. INTRODUCTION AND CURRENT APPROACHES
- Food varies in shape, size, texture, and color with preparation, ingredients, lighting, and viewpoint, while lacking a distinctive spatial layout,,.These variations make food classification challenging.Source: §I · p. 1
- Earlier handcrafted descriptors and multiple-kernel methods were used with SVMs to combine features and improve accuracy,,,.The descriptors included gradient, color correlogram, SIFT, local binary patterns, and SURF.Source: §I · p. 1
- Prior approaches used segmentation, object detection, or user-drawn boxes to localize food, but the semi-automated option was described as tedious and unsuitable for full automation [14],,.Automatic recognition was also presented as useful for organizing photo collections and improving access to online repositories.Source: §I · p. 1
- The paper extends CNN-based food classification by combining multiple networks, contributes a new Indian food image database, and reports experiments on two large databases.It also states that the developed models and Indian database would be made publicly available.Source: §I · p. 1
II. PROPOSED METHOD
Experiments comparing handcrafted features and CNN descriptors found CNNs performed substantially better. The section presents this comparison on ETH Food-101.
II. PROPOSED METHOD
- CNN-based methods (SELC and CNN features) perform much better than the other methods in the reported ETH Food-101 comparison.The authors describe CNNs as outperforming the explored handcrafted features by a huge margin.Source: §II · p. 2, Table I · p. 3
- The authors report extensive experiments with handcrafted features and CNN feature descriptors, selecting CNNs as the better-performing features.The comparison is presented as Table I, which reports accuracy percentages on ETH Food-101.Source: §II · p. 2, Table I · p. 3
- The comparison covers accuracy for SURF and bag-of-words variants, Fisher vectors, mid-level discriminative superpixels, random-forest discriminative components, SELC, and AlexNet trained from scratch.These are the methods listed for Table I on ETH Food-101.Source: Table I · p. 3
A. Proposed Ensemble Network Architecture
The architecture combines features from three fine-tuned CNNs in a weighted ensemble for food classification. Its design draws on AlexNet, GoogLeNet, and ResNet, with the stated aim of comparing accuracy against network depth and parameter count.
A. Proposed Ensemble Network Architecture
- AlexNet is the baseline for its lower computational time, while GoogLeNet uses concatenated 1×1, 3×3, and 5×5 filter banks in Inception modules.The paper notes that 1×1 convolutions are used to embed outputs from previous layers when deeper Inception modules raise computational cost.Source: A. Proposed Ensemble Network… · p. 2
- The design compares accuracy against network depth and parameter count, while ResNet learns residual functions of the form H(x) = Γ(x) + x.The authors describe the residual formulation as easier for the network to learn.Source: A. Proposed Ensemble Network… · p. 2
- Features from three differently configured subnetworks are weighted and combined for food classification, rather than compared as in a Siamese network.The authors propose that appropriately weighted features could improve classification accuracy.Source: A. Proposed Ensemble Network… · p. 2
- Each preprocessed color image is fed to all three fine-tuned networks; pooling is omitted in the ensemble but used with variable parameters in the subnetworks.The passage also defines the convolution, pooling, normalization, and fully connected layer notation used to describe the networks.Source: A. Proposed Ensemble Network… · p. 2
- The ensemble uses a weighted concatenation of fine-tuned network features, then applies SoftMax scores and selects the class with the maximum score.The feature weights sum to one.Source: A. Proposed Ensemble Network… · p. 3
B. Network Details
The network details cover pretrained feature baselines, dataset-specific fine-tuning, and training configurations. On ETH Food-101, pretrained GoogLeNet features had higher reported top-1 accuracy than pretrained AlexNet features.
B. Network Details
- Pretrained AlexNet and GoogLeNet fully connected and max-pool outputs were used as features for linear-kernel SVM classifiers.Images were resized and normalized for each network before feature extraction.Source: B. Network Details · p. 3
- On ETH Food-101, GoogLeNet achieved 44.06% top-1 accuracy, compared with AlexNet’s 39.6%, using features extracted from their selected layers; both networks were fine-tuned on this database.Features came from GoogLeNet’s last max-pool layer (cls3 pool) and AlexNet’s last fully connected layer (fc7).Source: B. Network Details · p. 3
- For ETH Food-101, the final filters were replaced with 101-channel outputs, and the three networks were fine-tuned before their normalized outputs were used by the ensemble.AlexNet was trained for 16 epochs; GoogLeNet and ResNet for 20 each, and the ensemble for 30.Source: B. Network Details · p. 3
- The Indian-dataset models retained the same network depths and parameters, used jitters to address the smaller image count, and produced 50-class outputs.The selected ResNet had 50 layers and 175 blocks; the authors say hardware capacity limited use of a deeper model.Source: B. Network Details · p. 3
III. EXPERIMENTAL SETUP AND RESULTS
The experiments used two food image databases and were run with the stated server, GPU, and software configuration.
III. EXPERIMENTAL SETUP AND RESULTS
- Experiments used ETH Food-101 and the authors' newly contributed Indian Food Database.Source: §III · p. 3
- The experiments ran on a server with 128GB RAM and an NVIDIA Quadro K4200 GPU with 4GB memory and 1344 CUDA cores.Source: §III · p. 3
- The implementation used MATLAB 14a with MatConvNet, importing Caffe pretrained network models.Source: §III · p. 3
A. Results on ETH Food-101 Database
On ETH Food-101, the proposed Ensemble Net consistently outperformed the other current state-of-the-art methods across retrieval ranks.
A. Results on ETH Food-101 Database
- ETH Food-101 contains 101 food classes and 101,000 images, with 750 training and 250 test images per class.Training images are intentionally left uncleaned, while test images are manually cleaned.Source: A. Results on ETH Food-101 Database · pp. 3–4
- The experiments convert RGB images to HSV, equalize only the intensity channel, and convert them back to RGB.This preprocessing is intended to preserve color characteristics while alleviating intensity or illumination variation bias.Source: A. Results on ETH Food-101 Database · p. 4
- The Ensemble Net consistently outperformed current state-of-the-art methods across retrieval ranks on ETH Food-101.Table II reports Top-1, Top-5, and Top-10 accuracy; Fig. 3(a) plots accuracy through rank 10.Source: A. Results on ETH Food-101 Database · p. 4, Table II · p. 4, Figure 3 · p. 4
B. Results on Indian Food Database
The Ensemble Net outperformed the other methods across ranks on the Indian food database, which contains 50 classes and 100 images per class.
B. Results on Indian Food Database
- The newly contributed Indian food database contains 50 classes with 100 images each, selected to represent varied Indian cuisine.The authors describe its varied colors, textures, shapes, sizes, and layouts as useful for testing classifier robustness and accuracy.Source: B. Results on Indian Food Database · p. 4
- The protocol randomly assigns 80 images per class to training and the remaining images to testing.Source: B. Results on Indian Food Database · p. 4
- The Ensemble Net outperformed all other methods consistently across ranks in both databases and outperformed individual networks on the Indian database.No single second-best method led across both databases; ResNet underperformed AlexNet and GoogLeNet, probably because training images were insufficient.Source: B. Results on Indian Food Database · p. 4
IV. CONCLUSIONS
The paper concludes that its three-subnetwork ensemble produces robust discriminative food-image features and consistently outperforms other methods across ranks on both databases.
IV. CONCLUSIONS
- The multi-layered ensemble uses three fine-tuned deep-CNN subnetworks to produce robust discriminative features compared with individual networks.Source: §IV · pp. 4–5
- The proposed Ensemble Net consistently outperformed other current state-of-the-art methods at all ranks on both databases.Source: §IV · p. 5
- The authors contributed a new Indian Food Database intended for public availability, evaluation, and enrichment.Source: §IV · p. 5