PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding

arXiv:1806.05372v1cs.CV

TL;DR

Fine-grained recognition must distinguish subtle differences among visually similar categories. This paper combines OSME attention features with MAMC constraints and reports superior performance across bird, dog, and car recognition tasks.

  • Problem

    Fine-grained recognition is challenging because visually similar categories must be distinguished through subtle differences in key parts.

    Source: §1 · p. 1

  • Method

    The framework uses OSME to extract part features and MAMC to bring same-attention, same-class features closer than different-attention or different-class features.

    Source: §1 · p. 3

  • Results

    The method reports superior performance across fine-grained recognition tasks involving birds, dogs, and cars.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The CUB-200-2011 comparison indicates that good recognition results are not necessarily linked with high annotation costs.

    Source: §5.2 · p. 12

    Although MAMC can be combined with other triplet-based metric losses, the paper uses N-pair loss as its reference.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Attention-based learning for fine-grained image recognition remains a challenging task, where most of the existing methods treat each object part in isolation, while neglecting the correlations among them. In addition, the multi-stage or multi-scale mechanisms involved make the existing methods less efficient and hard to be trained end-to-end. In this paper, we propose a novel attention-based convolutional neural network (CNN) which regulates multiple object parts among different input images. Our method first learns multiple attention region features of each input image through the one-squeeze multi-excitation (OSME) module, and then apply the multi-attention multi-class constraint (MAMC) in a metric learning framework. For each anchor feature, the MAMC functions by pulling same-attention same-class features closer, while pushing different-attention or different-class features away. Our method can be easily trained end-to-end, and is highly efficient which requires only one training stage. Moreover, we introduce Dogs-in-the-Wild, a comprehensive dog species dataset that surpasses similar existing datasets by category coverage, data volume and annotation quality. This dataset will be released upon acceptance to facilitate the research of fine-grained image recognition. Extensive experiments are conducted to show the substantial improvements of our method on four benchmark datasets.

1 Introduction

Fine-grained recognition is difficult because similar classes differ in subtle parts, while existing attention methods can be costly, complex to train, and insensitive to correlations among parts. The paper proposes an end-to-end OSME–MAMC framework and a new dog-species dataset, reporting substantial improvements on four benchmarks.

1 Introduction

  • Fine-grained recognition is challenging because similar classes can differ in subtle features on key parts, such as ears, tails, and body length.Source: §1 · pp. 1–2
  • The method achieves substantial improvements on four benchmark datasets and requires only one training stage and one feedforward per training step.The authors contrast this efficiency with methods requiring multiple feedforward processes or alternative training stages.Source: §1 · p. 3
  • OSME directly extracts features from multiple attention regions, while MAMC pulls same-attention, same-class features closer and pushes different-attention or different-class features away.The framework is designed to produce noncorrelated, discriminative part features using lightweight extractors.Source: §1 · pp. 2–3
  • Dogs-in-the-Wild is introduced as a dog-species dataset with greater category coverage, data volume, and annotation quality than similar public datasets.Source: §1 · p. 3

2 Related Work

Related work spans part-based fine-grained recognition, visual attention, and metric learning; this method combines weakly supervised part extraction with metric constraints on parts and classes.

2.1 Fine-Grained Image Recognition

  • Manual part supervision has classified birds, dogs, and cars, but annotation costs limit its practicality.Source: §2.1 · p. 3
  • Weakly supervised methods use pose alignment and co-segmentation, spatial transformation, or parallel bilinear CNNs; ours extracts part features without cropping and scales efficiently.Source: §2.1 · p. 3

2.2 Visual Attention

  • Visual-attention work spans soft proposals, LSTM or reinforcement-learning attention [32], CAM, and multi-stage or recurrent refinement; ours localizes and captures features without ROI cropping.Source: §2.2 · p. 4

2.3 Metric Learning

  • Metric learning progresses from contrastive to triplet, N-pair, and angular loss; triplet avoids precise similarity labels but converges slowly.Source: §2.3 · p. 4
  • Unlike prior metric-learning work, this method treats object parts as instances and jointly considers each instance's part and class labels.Source: §2.3 · p. 5

3 Proposed Method

The method combines OSME attention-feature extraction with MAMC to regulate relationships among parts, and is trained end-to-end in one stage. The supplied passages describe its attention mechanism, feature constraints, and training loss.

3.1 One-Squeeze Multi-Excitation Attention Module

  • OSME extracts P attention-specific feature vectors by globally pooling channels once, then independently gating and reweighting the feature map for each attention.Adding a few layers atop the last residual block supports deep base networks and efficient one-stage training.Source: §3.1 · pp. 6–7

3.2 Multi-Attention Multi-Class Constraint

  • MAMC organizes attention features into triplets so same-attention same-class features are positives, while same-attention different-class and different-attention same-class features are closer to the anchor than different-attention different-class features.For each anchor, the constraints enforce that it is closer to a positive than to any negative by a margin.Source: §3.2 · p. 8

3.3 Training Loss

  • With P = 2 and N = 32, MAMC generates 130× more constraints than N-pair loss on the same data.MAMC jointly minimizes softmax and N-pair losses with weight λ, but can also be combined with other triplet-based losses.Source: §3.3 · p. 9

4 The Dogs-in-the-Wild Dataset

Dogs-in-the-Wild contains 299,458 images across 362 dog categories and was designed to provide broad species coverage, accurate annotations, and challenging real-world variation. Its curation includes repeated annotation checks and excludes ImageNet images for fair evaluation.

4 The Dogs-in-the-Wild Dataset

  • Dogs-in-the-Wild contains 299,458 images across 362 dog categories, making it 15× larger than Stanford Dogs.Species lists were combined from multiple sources, images were crawled and crowdsourced labels checked, small classes pruned, and similar classes merged after confusion-matrix analysis and manual validation; annotation was conducted three times, and ImageNet images were excluded for fair evaluation.Source: §4 · p. 10
  • Dogs-in-the-Wild covers greater within-category variation and more environment types than Stanford Dogs and Goldfinch, with manually examined images to minimize annotation errors.Goldfinch has comparable class number and data volume, but noisy images are common; examples in Dogs-in-the-Wild include varied Staffordshire Bull Terrier patterns and hair colors, diverse scenes, and dressed-up dogs.Source: §4 · p. 10
  • Dogs-in-the-Wild is more imbalanced in images per category than the other two datasets, while its higher average images per category contributes to intra-class variation and reduces overfitting vulnerability.The greater imbalance is described as more consistent with real-life situations and as increasing classification difficulty.Source: §4 · p. 10

5 Experimental Results

Across four fine-grained recognition datasets, the experiments compare the method with prior approaches and examine its components, efficiency, and learned attention regions. The method performs strongly across benchmarks, while ablations test the contributions of OSME and MAMC.

5 Experimental Results

  • Experiments use top-1 accuracy on CUB-200-2011, Stanford Dogs, Stanford Cars, and Dogs-in-the-Wild, with input images resized to 448×448 for training and testing.Each dataset is trained for 60 epochs with batch size 10 (N=5), base learning rate 0.001 decayed by 0.96 every 0.6 epoch, λ = 0.5, and FC channels C = 2048 and D = 1024; implementation uses Caffe and one Tesla P40 GPU.Source: §5 · pp. 10–11

5.1 Ablation Analysis

  • With ResNet-50 and no MAMC, OSME with P = 2 improves performance by 3.2% over baseline (84.9% vs. 81.7%); with MAMC, OSME adds 0.5% over independent FC layers (86.2% vs. 85.7%).Two attention regions yield promising results, while P = 3 gives slightly better performance.Source: §5.1 · p. 12
  • With ResNet-50 and OSME, the first MAMC constraint improves performance by 0.5% over baseline, and all three constraints add another 0.8%.The ablation supports the effectiveness of each of the three MAMC constraints.Source: §5.1 · p. 12
  • Compared with the ResNet-50 baseline, the method improves results by 4.5% with 30% more time, while similar method takes 3.6× more time and gives a less optimal result.Source: §5.1 · p. 12

5.2 Comparison with State-of-the-Art

  • On CUB-200-2011, ResNet-101 ties MACNN for best overall performance; ResNet-50 exceeds the second-best methods with extra and without extra annotation by 0.8% and 0.2%, respectively.The paper also reports that the method exceeds PDFR, MG-CNN, RACNN, B-CNN, and RAN by 2.0%, 4.8%, 1.2%, 2.4%, and 3.7%, respectively.Source: §5.2 · pp. 12–13, Table 2 · p. 13
  • On Stanford Dogs, the method exceeds all comparison methods except RACNN; on Stanford Cars, it achieves 93.0% accuracy, outperforming all comparison methods.The paper notes that, unlike multi-scale or multi-stage approaches such as DVAN and RAN, its multi-attention feature extraction uses one feedforward stage.Source: §5.2 · pp. 12–13, Table 2 · p. 13
  • On Dogs-in-the-Wild, the method achieves the best result with remarkable margins, and lower accuracies than on Stanford Dogs indicate the proposed dataset is more difficult.The paper presents these results as baselines for future exploration of the newly proposed dataset.Source: Table 2 · p. 13, §5.2 · pp. 13–14
  • OSME heatmaps highlight meaningful fine-grained parts, including bird heads and wings, dog heads and tails, and car headlights or grills and frames.The paper contrasts these attention-region heatmaps with the baseline ResNet-101 last-convolution-layer heatmaps.Source: Figure 6 · p. 14, §5.2 · p. 14

6 Conclusion

The paper proposes an end-to-end CNN for fine-grained recognition that uses OSME attention features supervised by the MAMC loss. It reports superior performance to state-of-the-art methods on birds, dogs, and cars, and introduces the Dogs-in-the-Wild dataset.

6 Conclusion

  • The CNN extracts attention-aware features with OSME, while MAMC pulls positive features toward the anchor and pushes negative features away.The method does not require bounding-box or part annotations and trains end-to-end in one stage.Source: §6 · p. 14
  • The method reports superior performance to state-of-the-art methods on fine-grained recognition tasks involving birds, dogs, and cars.Source: §6 · p. 14
  • The paper collected and plans to release Dogs-in-the-Wild, a dog-species dataset with the largest data volume, full category coverage, and accurate annotation among similar existing datasets.Source: §6 · p. 14

Related papers

Loading 1806.05372v1…