Source-linked AI summary
Wide-Slice Residual Networks for Food Recognition
Niki Martinel, Gian Luca Foresti, Christian Micheloni
TL;DR
Food recognition is challenging because dishes vary substantially within categories and may look similar across categories, while existing deep approaches generally use off-the-shelf architectures. WISeR combines residual learning with slice convolution to capture both generic food traits and vertical layers, and evaluations on three benchmark datasets report better performance than existing approaches. The authors also report that the architecture can self-identify relevant image portions, but its substantial memory and computational requirements prevent deployment on mobile devices.
Problem
Food recognition must handle large intra-class variation and inter-class confusion, while many existing deep approaches do not design architectures for food-specific structural challenges.
Method
WISeR combines a residual learning branch for generic food representation with a slice convolution branch that captures vertical food layers.
Results
WISeR performs better than existing approaches on three benchmark food-recognition datasets and self-identifies relevant image portions for classification.
Takeaways & Limitations
Combining generic residual features with structure-specific slice features provides the reported best achievements across the evaluated datasets.
Takeaways & Limitations
WISeR requires substantial memory and computational effort for each datum, preventing deployment on mobile devices.
Abstract
from arXiv · showhide
Food diary applications represent a tantalizing market. Such applications, based on image food recognition, opened to new challenges for computer vision and pattern recognition algorithms. Recent works in the field are focusing either on hand-crafted representations or on learning these by exploiting deep neural networks. Despite the success of such a last family of works, these generally exploit off-the shelf deep architectures to classify food dishes. Thus, the architectures are not cast to the specific problem. We believe that better results can be obtained if the deep architecture is defined with respect to an analysis of the food composition. Following such an intuition, this work introduces a new deep scheme that is designed to handle the food structure. Specifically, inspired by the recent success of residual deep network, we exploit such a learning scheme and introduce a slice convolution block to capture the vertical food layers. Outputs of the deep residual blocks are combined with the sliced convolution to produce the classification score for specific food categories. To evaluate our proposed architecture we have conducted experimental results on three benchmark datasets. Results demonstrate that our solution shows better performance with respect to existing approaches (e.g., a top-1 accuracy of 90.27% on the Food-101 challenging dataset).
1. Introduction
Food recognition is difficult because dishes vary greatly within categories and can resemble other foods, while some dishes exhibit exploitable vertical structure. WISeR addresses these challenges with a structure-aware residual architecture that combines slice convolution and residual features, outperforming existing approaches on three benchmarks.
- Motivation: Food diary applications could help address diseases related to excessive or incorrect food intake through food recognition and calorie estimation.
- Problem: Food recognition combines large intra-class variation with inter-class visual confusion, despite humans often recognizing dishes from partial views.Recipes, ingredients, presentation, and hidden ingredients contribute to the challenge.
- Research gap: Existing approaches use hand-crafted representations or learned features with off-the-shelf deep architectures that do not explicitly model food-specific structural challenges.This motivates designing the architecture from an analysis of food composition.
- Method: WISeR combines a slice convolution layer for vertical food layers with residual learning to provide a generic representation for dishes without that structure.The combined representations produce the food classification output.
- Method: WISeR significantly enlarges feature-map counts per convolution layer to address diminishing feature reuse in deep residual networks.
- Results: Three benchmark datasets show that combining the proposed ingredients performs better than existing approaches.
2. Related Work
Food-recognition research has progressed from hand-crafted representations toward learned features, but many deep approaches rely on standard architectures. WISeR differs by introducing food-structure-aware convolution, residual learning, and wider feature maps.
- Hand-crafted methods: Hand-crafted food-recognition methods use prior knowledge through features such as color, shape, spatial relationships, filters, textons, and feature combinations.
- Mobile-oriented methods: Some food-recognition systems were designed for mobile devices using bag-of-visual-words, nearest-neighbor, segmentation, HoG, color patches, or Fisher Vector features.
- Deep learning: Hand-crafted methods depend on a priori problem knowledge, while deep approaches learn food representations and generally report better performance.
- Deep learning: Food-recognition deep-learning studies commonly use ImageNet-pretrained networks fine-tuned for food categories because food images are scarce.
- WISeR distinction: WISeR introduces a structural convolutional layer, applies residual learning for food recognition, and uses many feature maps to address diminishing feature reuse.
3. Wide-Slice Residual Networks
WISeR combines residual learning with slice convolution to represent both generic visual content and vertical food-layer structure. Its wide residual blocks increase feature-map capacity, while pooling addresses variation in layer location.
- Architecture: WISeR combines residual and slice branches, concatenating their features before fully connected layers produce the food classification prediction.The residual branch encodes generic visual representations, while the slice branch captures vertical food layers.
- Residual Learning: Residual learning represents the output as the sum of the block input and a learned residual function F(x_l, W_l).The block uses identity mapping and learns parameters that approximate F.
- Wide Residual Blocks: The residual architecture widens convolutional layers by significantly increasing feature maps to improve the representational power of feature detectors.The paper links widening to the diminishing feature reuse issue in deep residual networks.
- Slice Convolution: Slice convolution uses a kernel with the input image’s full width to act as a detector for vertical food layers.Standard squared kernels do not directly consider these vertical traits.
- Food Structure: The architecture targets food recognition challenges involving strong intra-class variation and inter-class visual confusion.The same dish may vary with recipes, ingredients, location, and preparation, while different foods can look similar.
- Slice Pooling: Max pooling over vertically elongated windows helps detect a food layer despite variation in its vertical position.The pooling operation is intended to identify a specific layer within a vertical location.
4. Experimental Results
The paper evaluates WISeR through dataset descriptions, protocol choices, design analyses, and comparisons with existing methods. It then presents concluding remarks.
- Experimental Results: The experimental section describes selected datasets and the evaluation protocol before discussing experimental and design selections.These steps precede comparisons with existing methods.
- Experimental Results: The section compares WISeR with existing methods to demonstrate its reported performance.The comparison is followed by concluding remarks.
- Experimental Results: The experimental discussion is organized around dataset selection, evaluation, design choices, comparative results, and conclusions.
4.1. Datasets
WISeR is evaluated on three benchmark food-recognition datasets selected for different challenges. The supplied descriptions emphasize category scale, real-world images, multiple dishes, and labeling variation.
- Dataset Selection: WISeR is evaluated on three benchmark datasets selected according to the different challenges they present.
- UECFood100: UECFood100 contains 100 Japanese food categories and approximately 14’000 mobile-camera images.Images may contain more than one food dish, supporting evaluation of localization before classification.
- UEC-Food256: UEC-Food256 contains 256 food dishes represented in about 32’000 images, extending the number of categories beyond UECFood100.
- Food-101: Food-101 contains real pictures of 101 popular dishes, with 75750 training images that may include intense colors and wrong labels.The test set contains 250 manually cleaned images per class.
4.2. Evaluation Protocol
The evaluation reports Top-1 accuracy and additionally includes the Top-5 criterion used for deep neural network results. Dataset split protocols differ across the benchmarks.
- Metrics: Food-recognition approaches are generally evaluated using Top-1 recognition accuracy.
- Metrics: The study also reports Top-5 performance, a criterion generally considered for deep neural network results.
- Dataset Splits: UECFood256 and Food-101 use their provided dataset splits, while UECFood100 uses an 80% training and 20% testing partition.The UECFood100 split is random and follows the protocol used in prior works.
- Baselines: Performance values for existing methods were taken from their corresponding works or directly provided by their authors.
4.3. Experimental and Implementation Settings
The evaluation uses a generic framework without dataset-specific hyperparameter tuning and compares performance on benchmark tables organized by dataset, method family, and image-cropping condition. Training starts from ImageNet-pretrained residual weights, adds the slice branch, and uses augmentation and fixed optimization settings.
- The evaluation does not specifically tune network hyperparameters to each dataset, aiming instead to provide a generic framework.
- Initialization: The network is initialized from an ImageNet-pretrained WRN, augmented with the slice-convolution branch, and fine-tuned on the selected food datasets.
- UECFood100: Table 1 separates UECFood100 results into hand-crafted methods, deep methods on cropped images, and images containing multiple food classes without ground truth.
- Data: Training uses 224 × 224 random crops, flipping, scale and aspect-ratio augmentation, photometric and color distortions, and standard 10-crop testing.
- Optimization: Optimization uses mini-batch stochastic gradient descent with momentum 0.9, weight decay 0.0005, scheduled learning rates, and 100k training iterations.
- UECFood256: Table 2 organizes UECFood256 results by hand-crafted methods, deep methods on cropped images, and uncropped images containing more than one food class.
4.4. Performance Analysis
WISeR outperforms existing food-recognition approaches across the reported benchmark comparisons, including cropped and multi-dish settings. Ablations show that the residual branch contributes more than the slice branch, while the combined design supports strong representations.
- UECFood100: More than 7% improvement is achieved in Top–1 performance over food-specific methods on ground-truth-cropped UECFood100 images.The gap is about 2.5 percentage points relative to the top performer [40].
- UECFood100: About 20% improvement over [24] is reported on UECFood100 images containing multiple food dishes without ground-truth cropping.
- UECFood256: 83% and 95% recognition accuracies are surpassed at Top–1 and Top–5, respectively, on UECFood256.
- Food-101: WISeR exceeds the best hand-crafted and deep-learning results on Food-101, achieving more than 90% Top–1 accuracy.The authors state that this result indicates good representations can be learned from weakly labeled data.
- Ablation Analysis: For all three datasets, the residual-only branch largely outperforms the slice-only branch.The authors attribute this to ImageNet pretraining and to the slice branch excluding non-vertical food traits.
- Ablation Analysis: Training the complete architecture from scratch produces recognition rates of 78.12%, 68.37%, and 79.45% on the three considered datasets.
4.5. Visual Attention
Visual-attention analyses indicate that WISeR focuses classification on relevant food regions, including when images contain multiple dishes or distracting non-food objects. Guided Grad-CAM visualizations provide attention maps and input gradients for this inspection.
- Grad-CAM produces coarse localization maps of the image regions considered important for classification.
- When multiple dishes appear, WISeR focuses on the image portion containing the object of interest.
- Figure 6 ranks WISeR's five predictions per UECFood256 image and marks true matches in green and false matches in red.
- Figure 7 shows input images, color-coded visual attention, and gradients computed with respect to the input for two UECFood100 images.Blue denotes lower attention and red denotes higher attention.
- Features are not extracted from other non-relevant food or non-food objects such as plates with leaves, spoons, and paper glasses.
4.6. Discussion
WISeR performs better than existing approaches across three datasets, combining residual and slice branches while self-identifying relevant image regions. Its substantial memory and computational requirements currently prevent mobile deployment.
- WISeR performs better than existing hand-crafted and deep-learning approaches across three datasets.
- Combining residual features with vertical-layer traits from slice convolution produces the best achievements.
- The architecture self-discovers which image portions to use for feature extraction and classification.
- The results suggest WISeR addresses varied food-recognition challenges rather than problems specific to one dataset.
- Substantial memory loads and computational effort prevent deploying the solution on mobile devices.
5. Conclusion
WISeR is a task-specific food-recognition system that fuses residual and slice-convolution branches. Evaluations on three benchmark datasets show better performance than state-of-the-art approaches and self-identification of relevant image regions.
- WISeR fuses a residual branch capturing general food traits with a slice branch capturing vertical dish layers.
- Features from both branches are fused and used to produce the classification.
- Evaluations on three benchmark datasets show better performance than state-of-the-art approaches regardless of the dataset.
- Visual attention analysis shows that the network self-identifies relevant image portions for classification.