Source-linked AI summary
Interpretable and Accurate Fine-grained Recognition via Region Grouping
Zixuan Huang, Yin Li
TL;DR
Fine-grained recognition models are accurate but difficult to explain, motivating methods that discover meaningful parts and their classification contributions without part annotations. This paper integrates region-based part discovery and attribution into a deep network trained with image-level labels, using a U-shaped occurrence prior. Across CUB-200, CelebA, and iNaturalist, it reports favorable recognition performance and improved part localization, including a 5.7% gain over ResNet101 on iNaturalist.
Problem
Deep recognition models are difficult to explain, and discovering meaningful object parts without explicit annotations is challenging.
Method
The model combines region-based part discovery and attribution with a U-shaped prior for part occurrence, using only image-level object labels.
Results
Across CUB-200, CelebA, and iNaturalist, the model compares favorably in recognition and improves part localization, including a 5.7% gain over ResNet101 on iNaturalist.
Takeaways & Limitations
The model provides part segmentation and attribution while retaining strong object classification and part-localization results.
Takeaways & Limitations
The method assumes that object-part occurrence follows a U-shaped distribution.
Abstract
from arXiv · showhide
We present an interpretable deep model for fine-grained visual recognition. At the core of our method lies the integration of region-based part discovery and attribution within a deep neural network. Our model is trained using image-level object labels, and provides an interpretation of its results via the segmentation of object parts and the identification of their contributions towards classification. To facilitate the learning of object parts without direct supervision, we explore a simple prior of the occurrence of object parts. We demonstrate that this prior, when combined with our region-based part discovery and attribution, leads to an interpretable model that remains highly accurate. Our model is evaluated on major fine-grained recognition datasets, including CUB-200, CelebA and iNaturalist. Our results compare favorably to state-of-the-art methods on classification tasks, and our method outperforms previous approaches on the localization of object parts.
1. Introduction
The paper develops an interpretable fine-grained recognition model that discovers object parts and attributes their contributions using only image-level labels. A U-shaped prior for part occurrence helps produce meaningful regions while retaining strong recognition accuracy.
- Motivation: The paper asks how a deep model can discover meaningful object parts and estimate their importance for visual recognition.The proposed interpretation segments regions such as facial parts and identifies which regions contribute to decisions.
- Challenge: Part discovery is challenging without explicit part annotations, so the method exploits shared visual patterns within fine-grained object categories.Convolutional features are grouped into visually coherent regions, from which discriminative segments can be selected.
- Prior: The model applies a U-shaped prior because different object parts occur with different frequencies across images.For example, bird heads may appear in most images, whereas bird legs may appear less consistently.
- Contribution: The prior combined with region-based part discovery identifies meaningful object parts, while the resulting interpretable model remains highly accurate.The paper distinguishes this approach from prior part-discovery methods that did not consider the same occurrence prior.
- Model: The model learns part representations, groups feature maps into segments, pools region features, and uses attention to select segments for classification.Its interpretation comes from the resulting part segmentation and each part’s contribution to the decision.
- Evaluation: On CUB-200, CelebA, and iNaturalist, the model achieves low part-localization error or favorable accuracy, including a 5.7% improvement over ResNet101 on iNaturalist.The evaluation measures both interpretability through part localization and recognition through standard classification metrics.
2. Related Work
The paper builds on interpretable deep learning, part-based fine-grained recognition, weakly supervised part segmentation, and region-based recognition. It distinguishes its approach through explicit regularization of object-part occurrence and evaluation on a large-scale dataset.
- Interpretable Deep Learning: Prior interpretability work visualizes discriminative regions, benchmarks network concepts against human masks, or trains simpler models to mimic trained networks.These approaches provide qualitative or surrogate explanations rather than the paper’s integrated region-based model.
- Part-Based Recognition: Prototype-based methods encode object-part concepts inside the network, whereas this paper differs by adopting region grouping and an explicit occurrence regularization.The comparison identifies two distinctions from the most relevant prototype-based work.
- Part-Based Recognition: Other fine-grained methods use class-specific filters, reinforcement learning over region proposals, or grouped feature channels to discover parts and attention.These methods represent alternative strategies for selecting or encoding discriminative regions.
- Weakly-Supervised Segmentation: Weakly supervised and unsupervised segmentation methods derive parts from CNN activations using graph representations, matrix factorization, or limited part annotations.The paper is connected to these approaches through its focus on discovering object-part regions without full supervision.
- Region-Based Recognition: Region-based recognition methods combine region grouping or segmentation with recognition, including graph-based feature processing and compositional learning.The paper places its model within this broader effort to integrate region structure into deep recognition.
3. Method
The model learns a part dictionary that groups image features into regions, extracts and weights region features for classification, and regularizes part occurrence with a prior. Its outputs support both prediction and pixel-level attribution.
- Model overview: The model learns a part dictionary D and decision function from convolutional feature maps with only image-level fine-grained labels.Each dictionary vector represents an object-part concept.
- Part segmentation: A soft assignment map groups feature-map pixels into K part segments by comparing pixel features with the learned dictionary.The assignment probabilities are assembled into Q, with a learnable smoothing factor for each part.
- Region features and attribution: Region features are pooled from the assigned segments, and an attention vector assigns importance scores to the regions.The resulting region features are transformed before attention-based classification.
- Part occurrence regularization: Part occurrence is regularized by aligning its empirical distribution with a known U-shaped prior using 1D Wasserstein distance.During mini-batch training, the distance is approximated with a sample-based inverse-CDF comparison; logarithmic rescaling improves training stability.
- Attention-based classification: The classifier re-weights transformed region features by attention, while pixel contributions are back-tracked through the transposed assignment map and attention vector.Large attention values indicate more important regions for classification.
- Training objective: Training minimizes classification cross-entropy together with the 1D Wasserstein regularization loss.The relative loss weights and prior Beta distribution are varied during experiments.
4. Experiments and Results
Experiments evaluate the model’s accuracy and interpretability across CelebA, CUB-200-2011, and iNaturalist 2017. The model achieves strong recognition results while producing competitive or improved part localization and attribution, with limitations on the most challenging dataset.
- Experimental Setup: Experiments measure both recognition accuracy and interpretability across CelebA, CUB-200-2011, and iNaturalist 2017.Interpretability is assessed through landmark localization or Pointing Game error, depending on dataset.
- Visualizations: Visualizations show facial-part alignment on CelebA, discriminative facial-attribute regions, coherent bird parts across poses, and assignment or attention maps across iNaturalist categories.The iNaturalist visualizations are presented by super category, while the CUB-200-2011 examples demonstrate consistency under different poses.
- CelebA: CelebA results achieve state-of-the-art attribute-recognition accuracy and reduce landmark-localization error by 6.6% versus SCOPS and 21.9% versus DFF.The model performs on par with the strong ResNet101 baseline for attribute recognition.
- CUB-200-2011: On CUB-200-2011, the model discovers coherent bird parts and selects discriminative regions such as beak/legs and wings/tails for species recognition.The dataset includes bird landmarks and supports evaluation of both classification and localization.
- iNaturalist 2017: On iNaturalist 2017, the model improves baseline ResNet101 accuracy by at least 3.7%, gains another 2% with fully convolutional testing, and achieves the lowest localization error.Its localization error improves by 4.2% over CAM/Grad-CAM and 0.6% over Guided Grad-CAM, although accuracy remains 1.4% below TASN.
- Ablation and Limitations: The ablation shows that regularization improves localization accuracy by 3.9%, while removing attention sacrifices region and pixel attribution.The full model’s landmark localization errors range from 7.4% to 9.3% across the listed facial landmarks.
5. Conclusion
The model combines a prior on object-part occurrence with region-based discovery and attribution, using only image-level labels to support interpretable fine-grained recognition.
- The model integrates a prior on object-part occurrence with region-based part discovery and attribution in a deep network.
- Trained with only image-level labels, it predicts object-part assignment maps, attention maps for part regions, and object labels.
- The approach demonstrates strong results for both object classification and object-part localization.