Source-linked AI summary
Ranked List Loss for Deep Metric Learning
Xinshao Wang, Yang Hua, Elyor Kodirov, Neil M. Robertson
TL;DR
Deep metric learning needs losses that avoid trivial examples and retain useful similarity structure within classes. The paper proposes Ranked List Loss, which ranks all gallery examples with a margin and learns a hypersphere per class. RLL-Simpler achieves state-of-the-art performance on SOP and In-shop Clothes, while the paper also studies practical factors such as network depth.
Problem
Existing ranking-motivated losses use only a fraction of informative examples and tend to compress intraclass distributions, limiting the similarity structure they exploit.
Method
Ranked List Loss ranks all gallery examples for each query, separates positive and negative sets by a margin, and learns a hypersphere for each class.
Results
RLL-Simpler achieves state-of-the-art performance on the SOP and In-shop Clothes datasets.
Takeaways & Limitations
The paper provides a structured loss that uses informative examples broadly while preserving intraclass similarity structure, with a simpler two-hyper-parameter variant for practice.
Takeaways & Limitations
Proxy-based alternatives discussed in the paper lack a theoretical guarantee of scalability to extremely large datasets.
Abstract
from arXiv · showhide
The objective of deep metric learning (DML) is to learn embeddings that can capture semantic similarity and dissimilarity information among data points. Existing pairwise or tripletwise loss functions used in DML are known to suffer from slow convergence due to a large proportion of trivial pairs or triplets as the model improves. To improve this, ranking-motivated structured losses are proposed recently to incorporate multiple examples and exploit the structured information among them. They converge faster and achieve state-of-the-art performance. In this work, we unveil two limitations of existing ranking-motivated structured losses and propose a novel ranked list loss to solve both of them. First, given a query, only a fraction of data points is incorporated to build the similarity structure. Consequently, some useful examples are ignored and the structure is less informative. To address this, we propose to build a set-based similarity structure by exploiting all instances in the gallery. The learning setting can be interpreted as few-shot retrieval: given a mini-batch, every example is iteratively used as a query, and the rest ones compose the gallery to search, i.e., the support set in few-shot setting. The rest examples are split into a positive set and a negative set. For every mini-batch, the learning objective of ranked list loss is to make the query closer to the positive set than to the negative set by a margin. Second, previous methods aim to pull positive pairs as close as possible in the embedding space. As a result, the intraclass data distribution tends to be extremely compressed. In contrast, we propose to learn a hypersphere for each class in order to preserve useful similarity structure inside it, which functions as regularisation. Extensive experiments demonstrate the superiority of our proposal by comparing with the state-of-the-art methods.
1 INTRODUCTION
The paper proposes Ranked List Loss (RLL) to address wasted informative examples and compressed intraclass distributions in deep metric learning. RLL uses set-based ranking over all gallery examples and class hyperspheres to preserve intraclass structure.
- Limitations of Existing Losses: Existing pairwise, tripletwise, and structured losses can waste informative examples as training improves, slowing effective learning.Only a proportion of informative examples may be incorporated, leaving structured information underused.
- Ranked List Loss: RLL ranks all gallery examples for each query, forcing positive points ahead of negative points with a margin.The method interprets mini-batch learning as retrieval: each example queries the remaining gallery, which is divided into positive and negative sets.
- Preserving Intraclass Structure: RLL learns a hypersphere for each class instead of pulling same-class samples as close as possible.The class hypersphere preserves intraclass similarity structure and avoids collapsing each class toward one point.
- Experiments: RLL-Simpler achieves state-of-the-art performance on the SOP and In-shop Clothes datasets.The paper also presents ablations on factors including network depth and dynamic example weighting.
2 PRELIMINARIES AND RELATED WORK
Deep metric learning designs loss functions to learn discriminative embeddings from instance relationships. Related structured losses incorporate multiple examples, while this paper presents Ranked List Loss as a structured alternative addressing limitations of existing methods.
- Design of Loss Functions for Learning Discriminative Deep Representations: Metric learning learns an embedding function whose loss defines distances between instances and preserves desired neighborhood structure.In deep metric learning, a neural network provides nonlinear encoding and the loss supervises the embedding.
- Design of Loss Functions for Learning Discriminative Deep Representations: Instance-to-class and instance-to-proxy approaches constrain examples relative to class centres or learned proxies.Class-centre methods use logits and softmax-based cross entropy, while proxy methods represent classes with proxy points.
- Design of Loss Functions for Learning Discriminative Deep Representations: Instance-to-instance methods model pairwise or higher-order similarity relationships without requiring class-centre representations.This family includes contrastive, triplet, Lifted Struct, N-pair-mc, and other ranking-motivated losses.
- Ranking-Motivated Structured Losses: Triplet loss compares one anchor, one positive, and one negative, whereas N-pair-mc and Lifted Struct incorporate multiple negative examples or classes.N-pair-mc uses one negative example per class; Lifted Struct uses all negative examples while pulling one positive pair together.
- Ranking-Motivated Structured Losses: Proxy-NCA replaces original examples with proxies and selects the closest proxy for each point, but its scalability to extremely large datasets is not theoretically guaranteed.With static assignment, proxies are learned during training and resemble class vectors in classification.
- Ranking-Motivated Structured Losses: Ranked List Loss incorporates all non-trivial data points and explores their intrinsic structured similarity information.Figure 2 contrasts RLL with methods that use fewer positive or negative examples; RLL uses all positive and negative examples.
3 METHODOLOGY
RLL learns an embedding function that gives positive pairs higher similarity scores than negative pairs and ranks matching samples ahead of retrieved results.
- RLL learns a discriminative embedding function whose positive-pair similarity scores exceed those of negative pairs.
- For any query, RLL ranks matching samples before the query’s retrieved results.The method assumes at least two samples per class so all data points can be optimized.
- The learning objective supports optimization of all data points by selecting a query from any class.
3.1 Pairwise Constraint
RLL uses a pairwise margin constraint to pull positives within a distance boundary while pushing negatives beyond that boundary.
- The pairwise constraint pulls positive examples closer than boundary α and pushes negative examples farther than boundary α.It also separates positive and negative sets by margin m.
- The constraint labels pairs as positive when their class labels match and negative otherwise.yij = 1 when yi = yj, and yij = 0 otherwise.
- Pair distances are computed as Euclidean distances between their embedded representations.The distance is dij = ||f(xi) − f(xj)||2.
3.2 Ranked List Loss
Ranked List Loss iteratively treats each mini-batch image as a query, mines and weights informative gallery examples, and jointly optimizes positive and negative set constraints.
- Non-trivial examples are mined because they have non-zero losses and provide useful gradients for convergence.Trivial pairs have zero gradients and can weaken informative examples during gradient fusion.
- Negative examples are weighted according to their loss values, with temperature Tn controlling the weighting slope.Tn = 0 weights all non-trivial negatives equally, while Tn = +∞ yields hardest-negative mining.
- Positive examples can be weighted by distance, with temperature Tp emphasizing farther or closer positives depending on its sign.Tp > 0 emphasizes larger-distance positives, whereas Tp < 0 emphasizes closer positives to preserve local similarity structure.
- RLL jointly minimizes objectives that pull informative positives together and push informative negatives beyond boundary α.The combined objective balances the positive and negative sets using λ, fixed at 0.5 in the experiments.
- Each mini-batch image acts as a query and ranks all other images in the gallery by similarity.The remaining images form positive and negative sets for that query.
- RLL separates positive and negative sets by margin m and uses independent normalization to address their imbalanced counts.During each ranked-list optimization, other examples’ features are treated as constants, so only the query embedding is updated.
3.3 Hypersphere Regularisation by Distance Thresholds
Hypersphere regularisation preserves intraclass variation by placing each class within a bounded hypersphere rather than collapsing it to one point.
- RLL makes each query closer to its positive set than its negative set by margin m while forcing negative examples beyond boundary α.This places same-class samples inside a class hypersphere.
- Each class hypersphere has diameter α−m, preserving intraclass variance instead of suppressing it completely.The paper reports that this bounded distribution can yield better generalisation performance than collapsing each class to one point.
- The paper terms this bounded within-class embedding strategy hypersphere regularisation.
3.4 RLL-Simpler
RLL-Simpler reformulates ranked list loss with two active hyper-parameters while retaining hypersphere regularisation and a practical weighting scheme. It is designed to simplify application without requiring manual optimisation of many hyper-parameters.
- RLL-Simpler design: The full RLL uses α and m as distance hyper-parameters plus Tn and Tp as scaling parameters for negative and positive examples.RLL-Simpler removes the positive weighting parameter and links α to m.
- RLL-Simpler design: With L2-normalised embeddings, RLL-Simpler pushes negative-pair distances above 1 + m/2 and positive-pair distances below 1 − m/2.The Euclidean distance between data points ranges from 0 to 2, motivating a decision boundary at distance 1.
- Training setting: RLL-Simpler treats each mini-batch as an Nc-shot C-way retrieval problem with many more negative than positive examples.For this reason, Tp is set to 0 by default in practice.
- Algorithm: The method processes mini-batches through feedforward embedding, iterative retrieval, loss computation, gradient calculation, and back-propagation.The algorithm updates the embedding function using learning rate β.
- Empirical use: RLL-Simpler achieves state-of-the-art performance on SOP and In-shop Clothes, while the full version supports ablation studies including network-depth analysis.The authors recommend starting with RLL-Simpler and adopting full RLL when further performance improvement is needed.
3.5 Learning Deep Models with RLL
RLL trains deep embedding models with randomly sampled mini-batches interpreted as few-shot retrieval tasks. Each image becomes a query in turn, while the remaining images form its gallery.
- Training procedure: RLL-based training uses mini-batches with stochastic gradient descent to learn the deep embedding function.Randomly sampled classes and images create a smaller ranking problem than the full dataset.
- Mini-batch setting: Each mini-batch contains C classes and Nc images per class, forming an Nc-shot C-way few-shot retrieval task.The batch is treated as a smaller gallery for searching matching examples.
- Retrieval procedure: Every image acts iteratively as the query, and all other mini-batch images serve as the gallery.RLL is computed after obtaining embeddings for the batch.
3.6 Computational Complexity
RLL accepts random multiclass-labeled images rather than rigidly prepared tuples and performs ranking and loss computation online. Its computational complexity is quadratic in mini-batch size.
- Input and computation: RLL accepts random input images with multiclass labels instead of requiring triplets or n-pair tuplets.Ranking and loss computation are performed online after image embeddings are obtained.
- Complexity: O(N^2) is the computational complexity of RLL, matching existing ranking-motivated structured loss functions.N denotes the mini-batch size in the algorithmic setting.
3.7 Exploring The Critical Learning Periods of Deep Metric Learning
The paper examines whether early and later learning phases affect deep metric learning through dynamic example weighting. It varies T1 or T2 while holding the other parameter fixed and evaluates sensitivity in test performance.
- Motivation: The early learning phase is studied as an under-scrutinised but important part of deep neural-network training.The paper investigates this phase in deep metric learning using dynamic example weighting.
- Dynamic weighting: The weighting scheme is changed from a static parameter to a dynamic variant during training.The schedule depends on current and maximum training iterations.
- Experimental analysis: The empirical results of the dynamic-weighting investigations are presented and discussed in Section 5.The experiments compare optimisation outcomes under the two parameter schedules.
- Early phase study: The study fixes T2 and varies T1 to test whether test performance is sensitive to the early learning phase.Sensitivity to T1 is treated as evidence that the early phase is critical.
- Later phase study: The study fixes T1 and varies T2 to examine the effect of the later learning phase on the final optimisation solution.Sensitivity to T2 is used to assess whether the later phase is crucial.
4 EXPERIMENTS
Experiments evaluate RLL across benchmark datasets, architectural settings, embedding sizes, weighting choices, hypersphere regularisation, batch sizes, and qualitative retrieval behavior. RLL-Simpler achieves state-of-the-art performance on the two larger datasets, while ablations identify practical effects of network depth, weighting, hypersphere diameter, and embedding size.
- Comparison with Recent Baselines: RLL-Simpler achieves state-of-the-art performance on the two larger datasets, SOP and In-shop Clothes.
- Comparison with Recent Baselines: On the two smaller datasets, only Multi-Simi outperforms RLL among the compared technically scalable methods.The paper describes Multi-Simi's multiple-metric weighting scheme as more complex than RLL's.
- Network Depth and Embedding Concatenation: RLL-L and RLL-M use shallower network depth and are reported as highly competitive, sometimes outperforming deeper alternatives.The study examines concatenated low-, mid-, and high-level embeddings on SOP and In-shop Clothes.
- Example Weighting: 78.8% Recall@1 is obtained without negative-example weighting, only 1% below the best result with proper weighting.Performance changes only slightly for Tn from 0 to 20 and drops when Tn is too large.
- Hypersphere Regularisation: 79.1% Recall@1 with strong regularisation exceeds 76.4% without hypersphere regularisation when C = 30 and K = 6.The best performance in both tested settings occurs at a hypersphere diameter of 0.8.
- Embedding Size and Batch Size: Larger embedding sizes generally improve performance, but gains eventually become negligible, supporting smaller encodings when storage and speed matter.The experiments vary the number of classes per mini-batch while fixing images per class at K = 3.
- Qualitative Results: Qualitative retrieval results show robustness and invariance to rotation and viewpoint on the SOP test dataset.Each query is shown with its top four gallery images.
6 CONCLUSION
The paper proposes ranked list loss to exploit all nontrivial data points for more informative supervision and discriminative embeddings. RLL extends ranking-motivated losses by separating examples into positive and negative sets with a margin.
- Ranked list loss exploits all nontrivial data points to provide more informative supervision for learning discriminative embeddings.
- RLL is presented as a general extension of ranking-motivated losses following the authors’ CVPR 2019 conference version.
- Given a query, RLL splits other data points into positive and negative sets and forces a margin between them.
- The method also uses example mining and weighting to leverage available information.