Source-linked AI summary
Embedding Label Structures for Fine-Grained Feature Representation
Xiaofan Zhang, Feng Zhou, Yuanqing Lin, Shaoting Zhang
TL;DR
The paper targets the lack of fine-grained feature representations that retrieve images at multiple relevance levels. It jointly learns classification and similarity constraints, embedding hierarchies or shared attributes through generalized triplets. Across three datasets, the method retains competitive classification accuracy and improves retrieval precision at different relevance levels.
Problem
Previous fine-grained methods rarely learn structured representations that retrieve related images at multiple relevance levels, despite the need for precise classification and retrieval.
Method
A multi-task CNN jointly optimizes softmax classification and triplet similarity losses, with generalized triplets embedding hierarchical labels or shared attributes.
Results
The method achieves competitive classification accuracy and significantly outperforms previous fine-grained representations for retrieval at different relevance levels across three datasets.
Takeaways & Limitations
Embedding label structures allows one feature-learning framework to support fine-grained recognition and retrieval across hierarchical or attribute-based relevance levels.
Takeaways & Limitations
Triplet-only optimization can converge slowly and provide inferior class differentiation because each triplet carries less information than an explicit classification constraint.
Abstract
from arXiv · showhide
Recent algorithms in convolutional neural networks (CNN) considerably advance the fine-grained image classification, which aims to differentiate subtle differences among subordinate classes. However, previous studies have rarely focused on learning a fined-grained and structured feature representation that is able to locate similar images at different levels of relevance, e.g., discovering cars from the same make or the same model, both of which require high precision. In this paper, we propose two main contributions to tackle this problem. 1) A multi-task learning framework is designed to effectively learn fine-grained feature representations by jointly optimizing both classification and similarity constraints. 2) To model the multi-level relevance, label structures such as hierarchy or shared attributes are seamlessly embedded into the framework by generalizing the triplet loss. Extensive and thorough experiments have been conducted on three fine-grained datasets, i.e., the Stanford car, the car-333, and the food datasets, which contain either hierarchical labels or shared attributes. Our proposed method has achieved very competitive performance, i.e., among state-of-the-art classification accuracy. More importantly, it significantly outperforms previous fine-grained feature representations for image retrieval at different levels of relevance.
1. Introduction
Fine-grained recognition has advanced, but feature representations remain underdeveloped for retrieving images with multiple levels of similarity. The paper addresses this gap by jointly learning classification and similarity while embedding label structures.
- 1. Introduction: Fine-grained retrieval must distinguish exact labels while also recognizing related images sharing a make, body type, or viewpoint.The figure illustrates relevance levels ranging from body type to model and viewpoint.
- 1. Introduction: Prior work largely optimizes classification accuracy or learns similarity constraints, rather than structured representations for multi-level relevance.Classification methods focus on discriminative parts, while metric-learning methods address intra-class variation.
- 1. Introduction: Triplet-based features can retrieve similar instances, but may reduce classification accuracy and converge slowly.The paper identifies both weaker recognition and slow convergence as limitations of similarity-only constraints.
- 1. Introduction: The proposed framework jointly optimizes softmax classification and triplet similarity losses to learn both category predictions and discriminative features.It is designed to retain classification accuracy while improving feature representation.
- 1. Introduction: Generalized triplets embed hierarchical labels or shared attributes, enabling retrieval at different relevance levels across three fine-grained datasets.The evaluated datasets are Stanford car, Car-333, and a fine-grained food dataset.
2. Related Work
Related work addresses fine-grained recognition through object parts, metric learning, and deep feature representations. The paper combines classification with similarity learning and extends the framework to label hierarchies and shared attributes.
- 2. Related Work: Fine-grained recognition is difficult because subordinate classes have small inter-class differences and large intra-class variation from pose and viewpoint.These challenges motivate both part-based models and embedding-based approaches.
- 2. Related Work: Part-based models capture subtle appearance differences in object regions and reduce variation caused by different poses or viewpoints.They use object parts to improve classification accuracy.
- 2. Related Work: Distance metric learning clusters same-class examples, separates different classes, and can flexibly group only some same-class neighbors.Deep feature representations such as DeCAF also provide discriminative embeddings for classification.
- 2. Related Work: Similarity constraints include pairwise siamese objectives and triplet-based methods for learning fine-grained feature representations.These methods complement classification constraints rather than relying on class labels alone.
- 2. Related Work: This paper jointly optimizes classification and triplet objectives while supporting hierarchy and shared-attribute structures in one framework.The experiments report better fine-grained feature representations than separate objectives.
3. Methodology
The method jointly learns classification and similarity representations, then embeds hierarchical or shared-attribute label structures to support relevance at multiple levels.
- Softmax classification learns fine-grained class distinctions, but can compress intra-class variation that helps retrieve visually and semantically similar instances.
- Triplet loss groups reference and positive images while separating the negative image using distances between ℓ2-normalized feature vectors.
- Jointly optimizing classification and triplet losses produces discriminative features, preserves intra-class variance, avoids sacrificing classification accuracy, and improves convergence over triplet-only training.
- Generalized Triplets for Hierarchical Labels: For hierarchical labels, quadruplets distinguish same fine-grained class, same coarse class, and different coarse classes through two distance margins.
- Generalized Triplets for Hierarchical Labels: The generalized hierarchical loss models richer coarse-to-fine relevance, while traditional triplet loss is its one-level special case.
- Generalized Triplets for Shared Attributes: For shared attributes, the margin adapts to Jaccard similarity, assigning smaller margins when positive and negative classes share more attributes.
4. Experiments
Experiments on three fine-grained datasets evaluate classification, multi-level retrieval, shared-attribute retrieval, convergence, and hyperparameter stability. The proposed joint framework with embedded label structures improves retrieval across relevance levels while maintaining strong classification accuracy and practical convergence.
- Experimental setup: The experiments compare triplet learning, post-softmax triplet fine-tuning, multi-task learning, and multi-task learning with embedded label structures across three datasets.Classification additionally includes traditional softmax CNNs.
- Stanford car: At least 13.5% higher fine-level and 12.4% higher coarse-level top-retrieval precision are obtained on Stanford car, with generalized triplets driving the coarse-level gain.Traditional and generalized triplets differ by at most 0.5% at the fine-grained level, while generalized triplets substantially improve coarse retrieval.
- Stanford car: 88.4% classification accuracy is achieved on Stanford car, exceeding fine-tuned GoogleNet at 86.9%, triplet-only learning at 78.7%, and post-softmax triplet fine-tuning at 83.0%.The joint optimization of classification and similarity losses reaches performance among state-of-the-art results without using parts.
- Car-333: At least 13.2% better middle-level and 12.8% better top-level top-2000 retrieval precision are reported on Car-333 than other methods.The label-structure method is also 7.2% better than the version without embedded structures at the top level.
- Food dataset: On the food dataset, embedding shared attributes improves precision by 5.5% at the fine-grained level and 4.2% at the attribute level.The attribute-level improvement is 3.1% over the authors’ method without embedded attributes, reducing errors by 16.9%.
- Convergence and stability: The methods converge after 800 Stanford-car epochs, 150 Car-333 epochs, and 600 food-dataset epochs, while remaining stable across tested hyperparameter ranges.Performance changes by within 0.8% for λs in [0.55, 0.85] and within 2% for feature dimensions from 128 to 512.
5. Conclusion
The paper presents a multi-task CNN framework that embeds hierarchical labels or shared attributes through generalized triplets. Across three fine-grained datasets, it retains classification accuracy while improving retrieval precision at multiple relevance levels.
- 5. Conclusion: The framework jointly learns fine-grained representations by embedding hierarchical labels or shared attributes through generalized triplets.The method incorporates similarity constraints at different relevance levels within CNNs.
- 5. Conclusion: Across three fine-grained datasets, the framework retains subordinate-class classification accuracy while considerably improving image retrieval precision at different label-structure levels.The conclusion also identifies a newly collected food benchmark among the evaluated datasets.