PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding

arXiv:1806.05372v1cs.CV

TL;DR

Fine-grained recognition remains difficult because existing attention methods often neglect correlations among object parts and rely on inefficient multi-stage or multi-scale mechanisms. The paper combines OSME attention features with MAMC metric constraints, achieving substantial improvements across four benchmark datasets while training end-to-end in one stage. It also introduces the Dogs-in-the-Wild dataset as a large, broadly covered, accurately annotated dog-species benchmark.

  • Problem

    Existing attention-based fine-grained methods often treat object parts independently and use inefficient multi-stage or multi-scale mechanisms.

  • Method

    The method uses OSME to extract multiple attention-region features and MAMC to pull same-attention same-class features closer while pushing different-attention or different-class features away.

  • Results

    The method achieves substantial improvements on four benchmark datasets and outperforms the ResNet-50 baseline by +4.5% with only 30% more time.

  • Takeaways & Limitations

    Attention-aware part features can be learned without bounding-box or part annotations in an end-to-end, one-stage framework, alongside the Dogs-in-the-Wild dataset contribution.

  • Takeaways & Limitations

    MAMC is presented as independent of N-pair loss, which is used as a reference for robustness and convergence rather than as a requirement of the framework.

Abstract

from arXiv · show

Attention-based learning for fine-grained image recognition remains a challenging task, where most of the existing methods treat each object part in isolation, while neglecting the correlations among them. In addition, the multi-stage or multi-scale mechanisms involved make the existing methods less efficient and hard to be trained end-to-end. In this paper, we propose a novel attention-based convolutional neural network (CNN) which regulates multiple object parts among different input images. Our method first learns multiple attention region features of each input image through the one-squeeze multi-excitation (OSME) module, and then apply the multi-attention multi-class constraint (MAMC) in a metric learning framework. For each anchor feature, the MAMC functions by pulling same-attention same-class features closer, while pushing different-attention or different-class features away. Our method can be easily trained end-to-end, and is highly efficient which requires only one training stage. Moreover, we introduce Dogs-in-the-Wild, a comprehensive dog species dataset that surpasses similar existing datasets by category coverage, data volume and annotation quality. This dataset will be released upon acceptance to facilitate the research of fine-grained image recognition. Extensive experiments are conducted to show the substantial improvements of our method on four benchmark datasets.

Fine-grained Image Recognition

The paper concerns fine-grained classification, metric learning, and visual attention, focusing on MAMC and OSME.

  • The work addresses fine-grained classification.
  • It uses metric learning to support recognition.
  • Its named mechanisms are Multi-Attention Multi-Class Constraint and One-Squeeze Multi-Excitation.

1 Introduction

Fine-grained recognition is difficult because classes can look globally similar while differing in subtle object-part details. The paper proposes OSME and MAMC to learn attention features without manual part annotations and reports substantial improvements across benchmark datasets.

  • Motivation: Fine-grained recognition must distinguish visually similar classes through subtle differences in key parts.
  • Motivation: Manual object-part annotations are costly, motivating weakly supervised approaches.
  • Contributions: The paper proposes OSME to localize different parts and directly extract their features with budgeted computational cost.
  • Contributions: MAMC enforces correlations among parts by making same-attention same-class features closer than different-attention or different-class features.
  • Contributions: The method requires no manual part annotations, trains end-to-end in one stage, and improves performance on four benchmark datasets.

2 Related Work

Related work combines part localization, visual attention, and metric learning to address subtle fine-grained differences, but often incurs annotation or computational costs. This paper instead captures attention-region features directly and treats object parts as metric-learning instances while incorporating class labels.

  • Part Localization: Supervised part-based methods can perform well but depend heavily on costly manual object-part annotations.
  • Part Localization: Weakly supervised methods avoid some annotation costs through pose alignment, co-segmentation, spatial transformation, or parallel CNNs.
  • Visual Attention: Part-based methods may require meaningful part definitions, which are difficult for non-structured objects such as flowers and food dishes.
  • Visual Attention: The proposed method automatically localizes attention regions and captures their features without explicitly cropping and reprocessing regions of interest.
  • Metric Learning: Metric learning learns similarity measurements, with contrastive learning minimizing positive-pair distances while separating negative pairs.
  • Architecture: The architecture illustrates multiple soft attention branches whose feature maps respond to regions such as a bird’s head or torso.
  • Metric Learning: Unlike whole-image metric learning, this method uses object parts as instances and jointly considers part and class labels.

3 Proposed Method

The proposed framework combines lightweight multi-region attention from OSME with MAMC metric constraints that organize attention features across classes and regions. It is designed as an end-to-end, one-stage feedforward network and exploits richer feature correlations than standard triplet or N-pair sampling.

  • 3.1 One-Squeeze Multi-Excitation Attention Module: The attention module is differentiable, lightweight, and directly extracts part features without explicitly cropping parts and running another feedforward pass.Adding a few layers above the last residual block supports relatively deep base networks and an efficient one-stage training pipeline.
  • 3.1 One-Squeeze Multi-Excitation Attention Module: OSME extends squeeze-and-excitation with one squeeze and multiple excitation branches to extract P attention-specific feature maps.Global average pooling produces a channel descriptor, and separate gating mechanisms reweight the original feature map for each attention region.
  • 3.2 Multi-Attention Multi-Class Constraint: MAMC groups OSME features relative to an anchor into same-attention same-class, same-attention different-class, and different-attention same-class relationships.Training batches contain 2N images, each producing P attention features, and the remaining features are partitioned according to class and attention identity.
  • 3.2 Multi-Attention Multi-Class Constraint: The three MAMC constraints select positives and negatives so the anchor is closer to positives than negatives by a distance margin.The constraints separately regulate same-attention same-class, same-attention different-class, and different-attention same-class comparisons.
  • 3.3 Training Loss: 130× more constraints than N-pair loss are generated when P = 2 and N = 32, while MAMC is reported to leverage richer correlations and obtain better convergence than triplet or N-pair loss.For a batch of N images and P parts, the method generates 2(PN − 1) + 4(N −1)2(P −1) + 4(N −1)(P −1)2 constraints, compared with N −1 from N-pair loss.

4 The Dogs-in-the-Wild Dataset

Dogs-in-the-Wild is a large, manually examined dog-species dataset designed to improve coverage, annotation quality, and realism for fine-grained recognition. Its greater variation and imbalance make classification more challenging while providing more images per category.

  • Dataset construction: Dogs-in-the-Wild contains 299,458 images across 362 dog categories, making it 15× larger than Stanford Dogs.Species lists combine multiple sources; images are crawled, crowdsourced labels are checked, small classes are pruned, and similar classes are merged.
  • Dataset construction: All images are manually examined, reducing annotation errors compared with noisy images reported in Goldfinch.The annotation process is conducted three times to guarantee quality.
  • Dataset characteristics: The dataset covers larger intra-class variation, diverse environments, and both natural and dressed-up dog appearances.Examples include natural, indoor, and artificial scenes, with broad variation in patterns and hair colors.
  • Dataset characteristics: Dogs-in-the-Wild has a more imbalanced images-per-category distribution than comparable datasets, reflecting real-life conditions and increasing classification difficulty.Its average images per category is nevertheless higher, supporting greater intra-class variation and reduced vulnerability to overfitting.

5 Experimental Results

Experiments evaluate the method on four fine-grained datasets using top-1 accuracy, ablations, efficiency comparisons, and state-of-the-art benchmarks. The results show gains from OSME and MAMC, strong performance across tasks, and one-stage efficiency, including on the challenging Dogs-in-the-Wild dataset.

  • Experimental setup: Experiments use CUB-200-2011, Stanford Dogs, Stanford Cars, and Dogs-in-the-Wild, with top-1 accuracy as the evaluation metric.Inputs are resized to 448×448 and each dataset is trained for 60 epochs.
  • Ablation Analysis: 84.9% vs. 81.7%: OSME with P = 2 improves the ResNet-50 baseline by 3.2% without MAMC.With MAMC, OSME adds a further 0.5% over using two independent FC layers, 86.2% vs. 85.7%.
  • Ablation Analysis: Applying the first MAMC constraint improves performance by 0.5%, while using all three constraints adds another 0.8%.The ablation attributes gains to the successive MAMC constraints.
  • Ablation Analysis: +4.5%: the proposed method outperforms the ResNet-50 baseline with only 30% more time, while a similar method takes 3.6× more time.This comparison evaluates the trade-off between performance and computational time.
  • Comparison with State-of-the-Art: On CUB-200-2011, the method achieves the best overall performance with ResNet-101 and exceeds second-best baselines by 0.8% and 0.2% with ResNet-50.It also surpasses B-CNN and RAN by 2.4% and 3.7%, respectively, using one-step multi-attention extraction.
  • Comparison with State-of-the-Art: 93.0%: the method achieves the highest reported accuracy on Stanford Cars, while using one feedforward stage for multi-attention multi-class feature extraction.On Stanford Dogs, it exceeds all comparison methods except RACNN, which requires multiple stages.
  • Comparison with State-of-the-Art: Dogs-in-the-Wild remains the most difficult evaluated setting, with lower accuracy than Stanford Dogs, while the proposed method achieves the best result on it.The reported results provide baselines for future exploration of the newly proposed dataset.

6 Conclusion

The paper concludes that OSME and MAMC form an efficient, end-to-end approach to fine-grained recognition, while Dogs-in-the-Wild expands available dog-species data with broad coverage and accurate annotation.

  • Conclusion: The proposed CNN combines OSME attention-aware features with MAMC, which pulls positive features closer and pushes negative features away.The method requires neither bounding-box nor part annotations and trains end-to-end in one stage.
  • Conclusion: Experiments report superior performance across fine-grained recognition tasks involving birds, dogs, and cars.The conclusion summarizes results across multiple task domains rather than a single dataset.
  • Conclusion: Dogs-in-the-Wild is presented as a comprehensive dog-species dataset with large data volume, full category coverage, and accurate annotation.The paper states that the dataset will be released to support future fine-grained recognition research.
Loading 1806.05372v1…