Source-linked AI summary
Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding
Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, Liang Lin
TL;DR
Fine-grained recognition methods have generally focused on categories at a single hierarchy level, despite semantic correlations across levels. HSE predicts categories from higher to lower levels, using higher-level scores to guide representations and regularize labels; experiments on bird, butterfly, and VegFru datasets report superiority over baselines and existing competitors.
Problem
Fine-grained recognition methods primarily classify categories at one hierarchy level and usually overlook correlations among levels.
Source: §1 · p. 1Method
Results
Experiments on the bird, butterfly, and VegFru datasets demonstrate HSE's superiority over baseline methods and existing competitors.
Source: §6 · p. 8Takeaways & Limitations
HSE incorporates category hierarchy into fine-grained recognition through both representation guidance and label regularization.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Object categories inherently form a hierarchy with different levels of concept abstraction, especially for fine-grained categories. For example, birds (Aves) can be categorized according to a four-level hierarchy of order, family, genus, and species. This hierarchy encodes rich correlations among various categories across different levels, which can effectively regularize the semantic space and thus make prediction less ambiguous. However, previous studies of fine-grained image recognition primarily focus on categories of one certain level and usually overlook this correlation information. In this work, we investigate simultaneously predicting categories of different levels in the hierarchy and integrating this structured correlation information into the deep neural network by developing a novel Hierarchical Semantic Embedding (HSE) framework. Specifically, the HSE framework sequentially predicts the category score vector of each level in the hierarchy, from highest to lowest. At each level, it incorporates the predicted score vector of the higher level as prior knowledge to learn finer-grained feature representation. During training, the predicted score vector of the higher level is also employed to regularize label prediction by using it as soft targets of corresponding sub-categories. To evaluate the proposed framework, we organize the 200 bird species of the Caltech-UCSD birds dataset with the four-level category hierarchy and construct a large-scale butterfly dataset that also covers four level categories. Extensive experiments on these two and the newly-released VegFru datasets demonstrate the superiority of our HSE framework over the baseline methods and existing competitors.
1 INTRODUCTION
The paper presents Hierarchical Semantic Embedding (HSE) to use category hierarchies in fine-grained recognition, where prior methods commonly classified only one level. It also organizes bird categories and introduces a butterfly dataset with four-level hierarchies.
1 INTRODUCTION
- Fine-grained category hierarchies encode cross-level semantic correlations that can regularize semantic space and guide attention toward subtle, discriminative regions.The paper illustrates this with bird recognition, where identifying a superclass can narrow attention to its subordinate categories and their distinguishing parts.Source: §1 · p. 1
- HSE predicts category scores from higher to lower levels, using higher-level predictions to guide finer-grained representation learning and regularize lower-level labels.Its semantic-guided attention focuses on discriminative regions, while higher-level scores serve as soft targets during training.Source: §1 · pp. 1–2
- The Caltech-UCSD birds hierarchy groups 200 species into 13 orders, 37 families, and 122 genera.The categories are organized according to ornithological systematics.Source: §1 · p. 1
- Butterfly-200 contains 200 species, grouped into 116 genera, 23 sub-families, and 5 families, with 25,279 images and at least 30 images per species.The authors note that taxonomic hierarchies can be obtained from taxonomy literature or Wikipedia for adaptation to other domains.Source: §1 · p. 2
- Experiments on the bird, butterfly, and VegFru datasets show HSE outperforming baseline and existing state-of-the-art methods.The introduction also reports ablation studies of the framework's components.Source: §1 · p. 2
2 RELATED WORK
Prior fine-grained recognition work developed compact feature models and attention mechanisms, sometimes using extra annotations or semantic guidance. The paper distinguishes HSE by using hierarchy to guide both feature learning and label prediction, and situates its datasets among existing single- and multi-level benchmarks.
2.1 Fine-grained image recognition
- Earlier fine-grained methods used bilinear features and subsequent compression approaches, while some localized discriminative regions using manually annotated boxes or parts.Visual-attention methods also searched for informative regions automatically from image content.Source: §2.1 · p. 2
- Other methods used attributes or language descriptions to guide discriminative feature learning, whereas HSE uses category hierarchy to guide finer-grained representations and regularize predictions.The authors also characterize higher-level predictions used as lower-level soft targets as a way to distill higher-level knowledge.Source: §2.1 · p. 2
- Earlier hierarchy-related methods modeled label dependencies, while HSE also uses hierarchical information to guide feature learning.The cited approaches include transfer learning with class hierarchy, probabilistic classification with hierarchy and exclusion graphs, and RNNs for label co-occurrence.Source: §2.1 · p. 2
2.2 Fine-grained image datasets
- Existing multi-level datasets included vehicle and food categories, while the paper adds four-level bird and butterfly hierarchies to broaden dataset diversity.The authors note potential applications of the new datasets in wildlife recognition, protection, and discovery.Source: §2.2 · p. 2
3 HSE FRAMEWORK
HSE predicts category scores from the highest to lowest hierarchy level, using higher-level predictions to guide finer-grained representations and regularize lower-level labels. Its framework combines semantic embedding representation learning, semantic-guided label regularization, and staged optimization.
3 HSE FRAMEWORK
- HSE extracts image features with a trunk network, then predicts each hierarchy level in order, using higher-level scores to guide finer features and regularize lower-level predictions during training.The first level has no higher-level guidance or label regularization.Source: §3 · pp. 2–3, Figure 1 · p. 3
3.1 Semantic embedding representation learning
- At each finer level, a higher-level score vector guides soft attention over feature maps, while unguided features provide complementary contextual information.The model combines guided, unguided, and concatenated features through three classifiers, averaging their score vectors for the level's final prediction.Source: §3.1 · pp. 3–4
3.2 Semantic guided label regularization
- Semantic-guided label regularization duplicates each parent-category score across its child categories and pulls the finer-level predictions toward the resulting extended vector.The two vectors are normalized with a temperature-scaled softmax, and their Kullback–Leibler divergence defines the regularization term.Source: §3.2 · p. 4
3.3 Optimization
- Training proceeds from level-wise branch training to joint fine-tuning, combining classification and regularization losses where guidance is available.The first level uses classification loss alone; the trunk is kept fixed during level-wise training.Source: §3.3 · p. 5
4 DATASETS
The evaluation uses hierarchical versions of CUB and Butterfly-200, alongside the two-level VegFru dataset. Butterfly-200 is newly constructed, while the CUB hierarchy is added to an established bird benchmark.
4 DATASETS
- The evaluation compares HSE and existing competitors on CUB, Butterfly-200, and VegFru.Source: §4 · p. 5
4.1 Butterfly-200 dataset construction
- Butterfly-200 contains 25,279 images of 200 species organized into four levels, with at least 30 images per species and separate training, validation, and test sets.Four experts identify candidate images; the split assigns 20% to training, 20% to validation, and 60% to test.Source: §4.1 · p. 5
4.2 Caltech-UCSD birds dataset extenstion
- The CUB dataset's 200 bird species are organized into 122 genera, 37 families, and 13 orders, using the standard train/test split.CUB contains 11,788 images: 5,994 for training and 5,794 for testing.Source: §4.2 · p. 6
4.3 VegFru dataset introduction
- VegFru covers 25 upper-level categories and 292 subordinate classes, with the standard train, validation, and test splits used for evaluation.The dataset contains 160,731 images in total.Source: §4.3 · p. 6
5 EXPERIMENT
Across CUB and Butterfly-200, HSE improves recognition at finer hierarchy levels over two baselines; state-of-the-art comparisons report 88.1% CUB accuracy and superiority on VegFru.
5.1 Significance of semantic embedding
- On CUB, HSE accuracies of 95.7%, 92.7%, and 88.1% outperform baseline methods by 0.6%, 1.2%, and 2.9%, respectively.Across both datasets, HSE is comparable to the baselines at level l1 and consistently better at levels l2–l4.Source: §5.1 · p. 6
- On CUB level l4, HSE trains more stably and converges faster than the baseline, assessed through training-loss and test-accuracy curves across epochs.Figure 4 plots training-set loss and test-set accuracy against training epoch.Source: §5.1 · p. 6, Figure 4 · p. 6
- Removing SGLR lowers performance across levels on both datasets; with it, inter-superclass errors fall 17.5% on CUB and 13.5% on Butterfly-200.The reported error reductions are relative reductions at level l4.Source: §5.1 · pp. 6–7
- Removing SERL lowers performance at every level on both datasets, while HSE attends to discriminative regions among species sharing a genus.For the Bohemian and Cedar Waxwings, the reported attended regions are the throat and wing tail.Source: §5.1 · p. 7
5.2 Comparison with state-of-the-art methods
- HSE requires no bounding-box or part annotations and uses no multiple-CNN local/global feature extraction, but adds category-hierarchy guidance.The comparison covers finest-level recognition on CUB and VegFru.Source: §5.2 · pp. 7–8
- HSE achieves 88.1% accuracy, exceeding AGAL’s 85.5% and CVL’s 85.6%; on VegFru, it significantly outperforms all compared methods.Source: §5.2 · p. 8
6 CONCLUSION
Fine-grained categories form hierarchies that encode correlations across abstraction levels. HSE predicts categories across these levels, using higher-level scores to guide finer representations and regularize predictions; experiments across three datasets demonstrate superiority over baseline methods and existing competitors.
6 CONCLUSION
- Fine-grained category hierarchies encode rich correlations among categories at different levels of abstraction.Source: §6 · p. 8
- Experiments on the birds, butterfly, and VegFru datasets demonstrate HSE’s superiority over baseline methods and existing competitors.The supplied VegFru table caption identifies an accuracy comparison, but provides no values or winner.Source: §6 · p. 8, Table 3 · p. 8
- HSE orderly predicts score vectors across hierarchy levels, using higher-level scores to guide finer-grained feature learning and regularize label prediction.Source: §6 · p. 8