PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-grained Image Classification by Exploring Bipartite-Graph Labels

Feng Zhou, Yuanqing Lin

arXiv:1512.02665v2cs.CV

TL;DR

Ultra-fine-grained recognition must distinguish similar classes despite limited training data. The paper models fine-grained labels jointly with coarse classes using bipartite-graph labels (BGL) in CNNs. It reports improved results across several fine-grained datasets and introduces a food benchmark.

  • Problem

    Ultra-fine-grained recognition is difficult because classes can look similar while training labels are limited, and CNNs are more prone to overfitting with limited labels.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    BGL jointly models fine-grained classes with predefined coarse classes in bipartite graphs incorporated into CNN training, with a star-like structure for exact inference.

    Source: §1 · p. 1, §3.2 · pp. 3–4

  • Results

    BGL improves on previous work across a variety of datasets, with significantly better accuracy than GoogLeNet on the food benchmark and less significant gains where class relationships are less rich.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The bipartite structure supports efficient exact inference, while modeling class relationships yields benefits that are less significant when those relationships are less rich.

    Source: §1 · p. 2, §3.2 · p. 4

    The label structure is currently manually defined, though the authors note it could potentially be learned during training.

    Source: §5 · p. 8

Abstract

from arXiv · show

Given a food image, can a fine-grained object recognition engine tell "which restaurant which dish" the food belongs to? Such ultra-fine grained image recognition is the key for many applications like search by images, but it is very challenging because it needs to discern subtle difference between classes while dealing with the scarcity of training data. Fortunately, the ultra-fine granularity naturally brings rich relationships among object classes. This paper proposes a novel approach to exploit the rich relationships through bipartite-graph labels (BGL). We show how to model BGL in an overall convolutional neural networks and the resulting system can be optimized through back-propagation. We also show that it is computationally efficient in inference thanks to the bipartite structure. To facilitate the study, we construct a new food benchmark dataset, which consists of 37,885 food images collected from 6 restaurants and totally 975 menus. Experimental results on this new food and three other datasets demonstrates BGL advances previous works in fine-grained object recognition. An online demo is available at http://www.f-zhou.com/fg_demo/.

1. Introduction

Ultra-fine-grained recognition must distinguish visually similar classes despite scarce training images. The paper models class relationships with bipartite-graph labels (BGL) in CNN training and reports improved recognition on food and other fine-grained datasets.

1. Introduction

  • Ultra-fine-grained classes can look similar and have few training images, making it important to share information while preserving strong discriminativeness.Source: §1 · p. 1
  • BGL models relationships among ultra-fine-grained classes and incorporates the class bipartite graphs into CNN training with overall back-propagation.Source: §1 · pp. 1–2
  • BGL adds training constraints, supports knowledge sharing across related classes, learns discriminative features, and avoids exponential state enumeration in inference through its bipartite structure.Source: §1 · p. 2
  • BGL produces significantly better recognition accuracy than GoogLeNet on the new food benchmark, with less significant benefits on existing datasets with less rich class relationships.The benchmark contains 37,885 food images from 6 restaurants and 975 menus.Source: §1 · p. 2

2. Previous work

Previous work addresses fine-grained recognition through part-focused features and segmentation, while structural-label research models relationships among labels in several ways.

2.1. Fine-grained image classification

  • Fine-grained approaches pursue discriminative parts through pose normalization and alignment, segmentation, metric learning, augmentation, and human-guided localization; BGL can complement these approaches.Segmentation can improve features but adds computational cost.Source: §2.1 · p. 2

2.2. Structural label learning

  • Structural-label work spans binary, relative, and hierarchical attributes; BGL handles general graph-like attributes and exact inference, unlike ’s tree and ’s approximate inference.Source: §2.2 · pp. 2–3

3. CNN with Bipartite-Graph Labels

BGL augments a CNN to jointly model fine-grained labels and predefined coarse labels, using their relationships to guide recognition. Its star-like bipartite structure supports exact inference, and the model is optimized through back-propagation.

3.3. Optimization

  • The method modifies CNN output layers to model BGL and optimizes them with mini-batch stochastic gradient descent using closed-form gradients.The implementation jointly normalizes terms for numerical stability and reduces one computation from quadratic to linear complexity in the number of coarse classes.Source: §3 · p. 3, §3.3 · p. 4

3.1. Background

  • Ultra-fine-grained recognition is challenging because labels are limited and similar classes have large within-class variation, complicating CNN training and discrimination.The paper uses a final fully connected layer that maps image features to class scores as the softmax-classification background.Source: §3.2 · p. 3, §3.1 · p. 3

3.2. Objective function with BGL modeling

  • BGL connects fine-grained classes to predefined coarse classes in bipartite graphs, letting related classes share training data and guiding features to distinguish subtle class differences.Coarse classes may be larger categories, attributes, or tags; they can be defined without image-level annotations.Source: §3.2 · pp. 3–4
  • The star-like graph structure enables exact inference, with partition-function computation scaling as O(km) in the numbers of fine-grained classes and coarse-label types.This is contrasted with general CRF-based methods, which have exponentially many possible states.Source: §3.2 · p. 4
  • BGL uses a hierarchical prior that favors nearby weights for fine-grained classes connected to the same coarse class, extending beyond big classes to multiple attributes.The paper describes the prior as another way to regularize CNN training against overfitting when training data are limited.Source: §3.2 · p. 4

3.3. Optimization

  • BGL supports both a shared feature for fine and coarse scores and multiple CNN features for coarse-label scores.Figure 2 compares softmax, BGL with one feature, and BGL with multiple features.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

4. Experiments

Across four benchmarks, the experiments evaluate BGL with different CNN architectures and label hierarchies. BGL improves recognition results, with especially strong reported gains on Car-333 and Food-975.

4. Experiments

  • The four-benchmark evaluation spans datasets with one coarse-label level and richer ultra-fine class relationships, using AlexNet, GoogLeNet, and VGGNet with softmax and BGL variants.Training used ImageNet-pretrained models; tests evaluated top-1 accuracy with single-view or multi-view cropping, with or without bounding boxes on the first three datasets.Source: §4 · p. 5

4.1. Stanford car dataset

  • BGL improved all three tested CNN architectures on Stanford cars, and VGG-BGLm beat prior works except; GN-BGLm predicted car type at 95.7%.The figure compares training-label distributions, predictions, exemplars, and accuracy; the reported example corrects a confusion between similar car models.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2. CUB-200-2011 dataset

  • 76.9% accuracy without bounding boxes was reported on CUB-200-2011, 3% above the part-based method [62]; GN-BGLm also reached 89.3% and 93.3% on wing-color and wing-shape attributes.BGL improved AN-SM, GN-SM, and VGG-SM, but remained behind methods and.Source: §4.2 · p. 6, Figure 4 · p. 7

4.3. Car-333 dataset

  • BGL helped most when Car-333 training data were relatively limited, while its added computation was minor compared with GN.For a 128-image mini-batch, exact inference adds little loss-layer computation; the last fully connected layer adds backward-pass cost, but both layers form a small part of the pipeline.Source: §4.3 · pp. 6–7, Figure 6 · p. 7

4.4. Food-975 dataset

  • 6 ∼7% improvement from the GN baseline at the wild condition was achieved by BGL on Food-975.Using both dish and ingredient labels produced a larger improvement than using either alone.Source: Figure 7 · p. 8, §4.4 · p. 8
  • Food-975 contains 32,135 training photos of 975 menu items, with controlled testing on 4,951 photos and wild testing on 351 consumer photos.Its hierarchy comprises 975 fine-grained labels, 781 dishes, and 51 ingredient attributes; different restaurants may prepare similar dishes with different ingredients.Source: §4.4 · pp. 7–8

5. Conclusion

The conclusion presents BGL as a way to model relationships between fine-grained and coarse labels, with a bipartite structure for efficient inference. It also introduces Food-975 and outlines several directions for extending the work.

5. Conclusion

  • BGL jointly models fine-grained and coarse labels through bipartite-graph labels and uses the bipartite structure for efficient inference.Source: §5 · p. 8
  • Food-975 is an ultra-fine-grained food recognition benchmark dataset, and BGL improved previous work on a variety of datasets.Source: §5 · p. 8
  • Future directions include learning label structure during training, studying continuous labels, and extending BGL to image-level labels.The paper also discusses scaling to many more classes, including hundreds of thousands of food classes.Source: §5 · p. 8

6. Gradients of BGL’s objective

This section formulates BGL’s objective as a structural logistic likelihood and derives gradients for its components. A synthetic example illustrates fast gradient computation.

6. Gradients of BGL’s objective

  • The BGL objective is formulated as a structural logistic likelihood over the weights.Source: §6 · p. 10
  • The section decomposes the objective into components and derives gradients for each component.Source: §6 · p. 10

6.1. ∂log pi/∂fi′

  • Figure 8 illustrates fast gradient computation with two fine-grained labels connected to three coarse types of size 1.Source: Figure 8 · p. 11

7. Fast computation of gradient

The section reduces the computational cost of gradient calculation by accumulating scores and gradients rather than computing them directly. It gives the resulting complexity for the fast computation.

7. Fast computation of gradient

  • Precomputing k auxiliary variables that accumulate active scores pi/pj at each fine-grained label supports the alternative gradient calculation.Each training example activates a subset of m coarse labels.Source: §7 · p. 12
  • The method computes cumulative gradients for coarse labels after accumulating the scores, avoiding redundant gradient aggregation work.Source: §7 · p. 12
  • O(km + k sum_{j=1}^m k_j) is the total complexity after reducing the direct computation’s O(km sum_{j=1}^m k_j) cost.Source: §7 · p. 12

8. Car-333’s VGG-BGL result

Figure 9 summarizes the result of using BGL with VGG on the Car-333 dataset.

8. Car-333’s VGG-BGL result

  • Figure 9 summarizes the result of using BGL with VGG on Car-333.Source: §8 · p. 12

9. Food-975’s restaurant list and ingredient labels

Food-975 covers six Chinese restaurants in the Bay Area, and Figure 10 lists the dataset’s ingredient labels.

9. Food-975’s restaurant list and ingredient labels

  • Food-975 includes six Chinese restaurants in the Bay Area.They are Chef Yu, Golden Garlic, Nutrition Restaurant, Lei Garden, Shanghai Dumpling, and Shanghai Restaurant.Source: §9 · p. 12
  • The six restaurants are Chef Yu, Golden Garlic, Nutrition Restaurant, Lei Garden, Shanghai Dumpling, and Shanghai Restaurant.Source: §9 · p. 12
  • Figure 10 provides a detailed list of ingredient labels for Food-975.Source: §9 · p. 12

10. More results

The additional results show prediction examples and similar training exemplars across four datasets, alongside Car-333 accuracy and Food-975 ingredient-attribute distribution figures.

10. More results

Related papers

Loading 1512.02665v2…