Source-linked AI summary
The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, Zheng Zhang
TL;DR
Fine-grained classification must distinguish categories using subtle local differences despite substantial pose, scale, rotation, and intra-class variation. The paper combines bottom-up proposals with object- and part-level top-down attention in CNNs using only image-level labels, achieving strong results across ILSVRC2012 and CUB200-2011, including 69.7% on CUB200-2011 and nearly 78% with VGGNet.
Problem
Fine-grained classification is difficult because discriminative differences are local while intra-class variation can be large, and existing attention methods often rely on costly object or part annotations.
Method
The pipeline combines bottom-up candidate patches, object-level top-down filtering, and part-level top-down localization to train domain-specific deep networks using image-level labels.
Results
69.7% accuracy was achieved on CUB200-2011 under weak supervision, with nearly 78% reported using VGGNet; the method also improved reported ILSVRC2012 dog and bird results to 28.1% and 11.0%.
Takeaways & Limitations
The attention levels provide complementary gains, and the method achieved the best CUB200-2011 accuracy reported under the weakest supervision setting.
Takeaways & Limitations
Part-level attention does not fully resolve ambiguities because robust part detection may require filters from multiple layers and closer integration with object-level attention.
Abstract
from arXiv · showhide
Fine-grained classification is challenging because categories can only be discriminated by subtle and local differences. Variances in the pose, scale or rotation usually make the problem more difficult. Most fine-grained classification systems follow the pipeline of finding foreground object or object parts (where) to extract discriminative features (what). In this paper, we propose to apply visual attention to fine-grained classification task using deep neural network. Our pipeline integrates three types of attention: the bottom-up attention that propose candidate patches, the object-level top-down attention that selects relevant patches to a certain object, and the part-level top-down attention that localizes discriminative parts. We combine these attentions to train domain-specific deep nets, then use it to improve both the what and where aspects. Importantly, we avoid using expensive annotations like bounding box or part information from end-to-end. The weak supervision constraint makes our work easier to generalize. We have verified the effectiveness of the method on the subsets of ILSVRC2012 dataset and CUB200_2011 dataset. Our pipeline delivered significant improvements and achieved the best accuracy under the weakest supervision condition. The performance is competitive against other methods that rely on additional annotations.
1. Introduction
Fine-grained classification distinguishes subordinate categories with small inter-class differences and large intra-class variation, making local feature localization difficult. The paper addresses this with weakly supervised, two-level attention that selects relevant object regions and discriminative parts for CNN classification.
- Fine-grained classification recognizes subordinate categories such as bird types, dog breeds, flower species, and aircraft models within a basic-level category.
- Large intra-class variance can exceed small inter-class variance, making fine-grained classification technically challenging.
- Discriminative features are localized on foreground objects and especially object parts, motivating a pipeline that first finds regions and then extracts features.
- Bottom-up region proposals offer high recall but low precision, so top-down object- and part-level attention filters noisy patches and selects relevant content.
- The method uses a FilterNet for object-level patch selection, a DomainNet for domain-specific classification, and CNN hidden representations as part detectors.
- 69.7% accuracy was reached on CUB200-2011, while reported ILSVRC2012 dog and bird results were 28.1% and 11.0% under weakest supervision.
2. Methods
The method combines bottom-up patch proposals with object-level filtering and part-level detection to classify fine-grained categories using complementary views of objects and discriminative parts.
- Object-Level Attention: Selective search generates multi-scale, multi-view candidate patches, but top-down attention filters the high-recall, low-precision proposals before classification.This reduces background noise while retaining varied object views.
- Object-Level Attention: An object-level FilterNet scores patches by the aggregated confidence of softmax neurons for the fine-grained category’s parent class and selects patches above a threshold.For Chihuahua, the parent class is dog; selected patches then drive DomainNet training and inference.
- Object-Level Attention: The selected patches train a DomainNet from scratch, producing domain-specific features and increasing relevant patch-based data augmentation.The DomainNet serves as a fine-grained classifier and supplies internal features for part detection.
- Part-Level Attention: Part detectors emerge by spectrally clustering similar mid-level DomainNet filters, with each cluster voting on warped candidate patches to produce detection scores.In the experiments, fourth-convolution-layer filters are grouped into three clusters.
- Part-Level Attention: The part detector’s selected patches produce DomainNet activations that are concatenated to train an SVM-based part classifier.This classifier focuses on discriminative local features rather than object-level views.
- Prediction Fusion: The final prediction merges object-level DomainNet and part-based classifier outputs to combine whole-object views with localized discriminative features.The two classifiers process different patch types and can provide complementary representations.
3. Experiment
The experiments evaluate the attention pipeline on weakly annotated ILSVRC2012 subsets and CUB200-2011, comparing attention strategies, feature extractors, and supervision levels. The combined attention method improves performance over baselines and remains competitive with methods using stronger annotations.
- Datasets: Experiments cover ILSVRC2012 Dog and Bird subsets and the CUB200-2011 bird benchmark.The ILSVRC2012 subsets provide only image-level class labels, whereas CUB200-2011 includes bounding boxes and part landmarks.
- Implementation Details: The evaluation uses AlexNet throughout most experiments, with VGGNet additionally tested to assess architecture sensitivity.The output layer is adjusted to the required number of categories, and VGGNet is used only for selected feature-extraction experiments.
- ILSVRC2012 Dog/Bird: Object-level attention lowers the ILSVRC2012 Dog top-1 error rate by 9.3% versus randomly cropped CNN training, while combined attention reaches 28.1%.Part-level attention alone performs worse than object-level attention but provides pose normalization and complements it.
- Feature Analysis: The feature-extractor comparison evaluates KDES, ILSVRC2012-pretrained CNN features, and DomainNet features under bounding-box and DPD pipelines.DomainNet features achieve the best results on both pipelines without CNN fine-tuning on CUB200-2011.
- CUB200-2011: With VGGNet features, combined attention reaches 77.9%, compared with 72.1% for the no-attention baseline using original images.Object-level and part-level attention individually reach 76.9% and 76.4%, respectively.
4. Related Work
Prior fine-grained classification work emphasizes object and part localization, feature representation, and varying degrees of human annotation. This paper relates its weakest-supervision attention pipeline to region-proposal, part-detection, and CNN-feature approaches.
- Fine-grained methods primarily address object or part localization and discriminative feature representation.
- Many systems use bounding boxes or part landmarks to implement object- and part-level attention.The strongest setting uses both annotation types during training and testing.
- R-CNN and Part-based R-CNN generate bottom-up region proposals before selecting or detecting informative object regions and parts.
- Feature-focused approaches include Kernel Descriptors, POOF, ImageNet-pretrained CNNs, and fine-tuned CNN feature extractors.
- This paper enriches object-level proposals with multiple views and scales while deriving attention, part detection, and features solely from CNN features under weakest supervision.
5. Conclusions
The paper combines bottom-up proposals with object-level and part-level top-down attention for fine-grained classification. It reports gains under weak supervision while identifying unresolved ambiguity in part detection and limited integration between attention levels.
- The proposed pipeline combines bottom-up proposals with object-level and part-level top-down attention, followed by late fusion.
- CNN classification features provide attention, part detection, and feature extraction using only class labels in the weakest supervision setting.
- The method achieves the best reported CUB200-2011 accuracy under the weakest supervision setting.
- The authors identify ambiguous part-level attention because part features may appear at different network layers due to scale variation.
- The current part-level pipeline does not exploit the large set of relevant patches produced by object-level attention for stronger pose normalization.