Source-linked AI summary
Learning Fine-grained Image Similarity with Deep Ranking
Jiang Wang, Yang song, Thomas Leung, Chuck Rosenberg, Jinbin Wang, James Philbin, Bo Chen, Ying Wu
TL;DR
Fine-grained image similarity must capture relative visual differences beyond category membership, including distinctions among images within one category. The paper learns this relationship directly from images with a triplet-based deep ranking model, a multiscale network, and scalable online sampling. Experiments report much better performance than state-of-the-art handcrafted-feature and deep classification models.
Problem
Existing category-level similarity and classification approaches do not adequately distinguish fine-grained differences among images within the same category.
Method
The paper uses a triplet-based hinge-loss ranking model with a multiscale neural network and efficient online triplet sampling, learning image embeddings directly from images.
Results
The deep ranking model achieves much better performance than state-of-the-art handcrafted-feature models and deep classification models.
Takeaways & Limitations
Relative similarity orderings in triplets provide fine-grained similarity information that category-level and classification models do not consider.
Takeaways & Limitations
Handcrafted-feature methods are limited by the representation power of their manually designed features.
Abstract
from arXiv · showhide
Learning fine-grained image similarity is a challenging task. It needs to capture between-class and within-class image differences. This paper proposes a deep ranking model that employs deep learning techniques to learn similarity metric directly from images.It has higher learning capability than models based on hand-crafted features. A novel multiscale network structure has been developed to describe the images effectively. An efficient triplet sampling algorithm is proposed to learn the model with distributed asynchronized stochastic gradient. Extensive experiments show that the proposed algorithm outperforms models based on hand-crafted visual features and deep classification models.
1. Introduction
Fine-grained image similarity must distinguish visual differences within the same category, not only category membership. The paper proposes deep ranking, multiscale representations, and efficient triplet-based training, with experiments showing substantial improvements over handcrafted-feature and classification approaches.
- Motivation: Search-by-example requires an image similarity metric that distinguishes differences between images within the same category.Category-level similarity alone is insufficient for this application.
- Motivation: Jointly learning image features and similarity models is presented as more promising for fine-grained similarity than relying on handcrafted features.The paper attributes the limitation of handcrafted approaches to their representation power.
- Approach: The proposed deep ranking model learns fine-grained similarity directly from images using relative triplet orderings.Each triplet contains a query, a more-similar positive, and a less-similar negative image.
- Approach: Bootstrapping generates virtually unlimited training data, while online triplet sampling produces meaningful and discriminative triplets for optimization.The sampling method supports asynchronous stochastic-gradient learning.
- Approach: A multiscale network captures both global visual properties and image semantics for similar-image ranking.Its architecture includes a convolutional neural network with two low-resolution paths.
- Evaluation: Experiments on a human-labeled dataset show the deep ranking model outperforms handcrafted visual-feature and deep classification models by a large margin.The dataset provides similarity-ranking information for images from the same category.
2. Related Work
Prior image-similarity research largely focuses on category-level similarity or learns fine-grained rankings from handcrafted features. This paper differs by learning a ranking model directly from image pixels and by enabling scalable triplet sampling.
- Category-level similarity: Most prior image-similarity work treats images as similar when they belong to the same category, emphasizing category-level or semantic similarity.Visual variability can remain substantial within a category, especially for broad semantic categories.
- Fine-grained ranking: Relative attribute, OASIS, and local distance learning address fine-grained ranking, but the latter methods build on handcrafted features.These approaches are not deep-learning based.
- Deep ranking: A prior deep-ranking approach learns from handcrafted features rather than directly from image pixels.The paper positions its Deep Ranking model as learning the representation and ranking jointly from images.
- Scalable sampling: The proposed online triplet sampler avoids loading the entire dataset into memory, supporting deep ranking on very large training sets.Earlier sampling methods assume the whole dataset can be loaded into memory.
3. Overview
The paper formulates image similarity learning as ranking images by distances in an embedding space. A triplet-based hinge-loss objective trains an embedding function to place more similar pairs closer than less similar pairs.
- Similarity is defined using squared Euclidean distance between image embeddings, with smaller distances indicating greater similarity.The embedding function maps each image to a point in Euclidean space, turning ranking into nearest-neighbor search.
- Pairwise relevance scores guide an embedding function to assign smaller distances to more similar image pairs.
- Each triplet contains a query, positive, and negative image that specifies a relative similarity order.
- The hinge loss penalizes triplets whose negative image is not separated from the positive image by the required distance gap.The gap parameter regularizes the learned ranker, while the regularization parameter controls the margin and generalization.
- The deep model learns the image embedding function directly from images instead of relying on hand-crafted visual features.
4. Network Architecture
The ranking model processes image triplets through three identical networks with shared parameters and learns embeddings using a triplet ranking loss. Its multiscale architecture combines semantic features from a strong-invariance ConvNet with appearance features from lower-resolution paths.
- Each triplet image is independently processed by identical neural networks with shared architecture and parameters to produce an embedding in R^d.
- A parameter-free ranking layer evaluates triplet hinge-loss violations and back-propagates gradients to the embedding networks.
- The multiscale network combines a ConvNet path with two shallower paths operating on down-sampled images.The ConvNet captures image semantics with strong invariance, while the lower-resolution paths capture visual appearance with less invariance.
- Features from the three paths are normalized and combined through a linear embedding layer into a 4096-dimensional representation.
- 45?
5. Optimization
Optimization uses distributed asynchronous stochastic gradients and online importance sampling to train on very large datasets without enumerating all triplets. Category buffers support relevance-weighted positive sampling and both in-class and out-of-class negatives.
- Distributed asynchronous stochastic gradient with momentum enables training when the deep-network dataset does not fit in one computer’s memory.The momentum method is described as a stochastic variant of Nesterov’s accelerated gradient method.
- All possible triplets are infeasible to enumerate: 12 million images yield approximately 1.728×10^21 triplets, while optimization converges with about 24 million samples.
- Online importance sampling prioritizes triplets relevant to top-ranked results rather than sampling uniformly.
- Triplets use query and positive images from the same category, while negatives are either out-of-class or less-relevant in-class images satisfying a relevance-margin constraint.Triplets that fail the margin condition are rejected, and examples with excessive failed trials may be discarded.
- Reservoir-style category buffers store images efficiently and make uniform buffer sampling equivalent to sampling proportional to total relevance score.
6. Experiments
The experiments evaluate training data, metrics, handcrafted-feature baselines, and neural architectures for fine-grained image similarity. DeepRanking benefits from direct image learning and multiscale modeling, while combining visual appearance with semantics.
- Training data: The evaluation uses ImageNet for semantic pretraining and bootstrapped relevance data for fine-grained similarity fine-tuning.The relevance data comes from search queries and uses a golden feature to compute within-query relevance.
- Evaluation metrics: Similarity precision measures correctly ranked triplets, while score-at-top-30 evaluates ranking quality among the 30 most relevant retrieval results.The latter is intended to reflect practical retrieval performance, where users focus on early results.
- Handcrafted-feature comparison: DeepRanking outperforms the evaluated handcrafted-feature models on similarity precision and score-at-top-30.The comparison includes individual features, L1HashKCPA, and OASIS; DeepRanking directly learns a ranking model from images.
- Architecture comparison: Ranking fine-tuning substantially improves performance over an ImageNet classification ConvNet with the same architecture.The ranking model is fine-tuned with relevance data and a ranking layer, whereas ConvNet is trained only for classification.
- Architecture comparison: The multiscale DeepRanking model improves both similarity precision and score-at-top-30 over single-scale ranking, while OASIS or linear embedding alternatives remain inferior.The multiscale design combines strong semantic representations with lower-invariance paths for visual appearance.
- Qualitative analysis: The learned filters capture more color information than filters learned in the referenced classification model, and Deep Ranking incorporates both appearance and semantics.The qualitative ranking comparison contrasts ConvNet, OASIS features, and Deep Ranking.
7. Conclusion
The paper presents a deep ranking model for fine-grained image similarity, combining triplet-based ranking, multiscale representations, and efficient online sampling. Empirically, it performs better than handcrafted-feature models and deep classification models.
- Conclusion: The proposed model learns fine-grained image similarity with a triplet-based hinge-loss ranking function and multiscale neural architecture.The architecture captures global visual properties and image semantics.
- Conclusion: Efficient online triplet sampling enables learning from very large amounts of training data.The sampling method supports training deep ranking models with online learning algorithms.
- Conclusion: Empirical evaluation shows better performance than state-of-the-art handcrafted-feature models and deep classification models.