Source-linked AI summary
Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, Matthew R. Scott
TL;DR
Pair-based deep metric learning must identify informative pairs amid redundant samples, but existing methods use incomplete views of pair similarity. The paper introduces GPW for gradient-based pair-weighting analysis and MS loss for iterative mining and weighting across three similarities. MS loss achieves state-of-the-art image-retrieval performance, including +5.0% Recall@1 over ABE on CUB200 and +7.1% over HTL on In-Shop Clothes Retrieval.
Problem
Pair-based metric learning produces many redundant pairs, while identifying informative pairs—especially quadratic-scale negative pairs—is challenging.
Method
The paper introduces GPW to unify pair-based losses through gradient-based pair weighting and proposes MS loss with iterative mining and weighting across self-similarity and relative similarities.
Results
+5.0% Recall@1 over ABE on CUB200 and +7.1% Recall@1 over HTL on In-Shop Clothes Retrieval were reported for MS loss.
Takeaways & Limitations
GPW provides a unified tool for comparing pair-based losses, while MS loss fully considers three similarities during pair weighting and achieves new state-of-the-art image-retrieval performance.
Takeaways & Limitations
Histogram loss may underperform because its weighting estimates pair weights using only Similarity-P, while BinLifted can neglect one of its two similarity values.
Abstract
from arXiv · showhide
A family of loss functions built on pair-based computation have been proposed in the literature which provide a myriad of solutions for deep metric learning. In this paper, we provide a general weighting framework for understanding recent pair-based loss functions. Our contributions are three-fold: (1) we establish a General Pair Weighting (GPW) framework, which casts the sampling problem of deep metric learning into a unified view of pair weighting through gradient analysis, providing a powerful tool for understanding recent pair-based loss functions; (2) we show that with GPW, various existing pair-based methods can be compared and discussed comprehensively, with clear differences and key limitations identified; (3) we propose a new loss called multi-similarity loss (MS loss) under the GPW, which is implemented in two iterative steps (i.e., mining and weighting). This allows it to fully consider three similarities for pair weighting, providing a more principled approach for collecting and weighting informative pairs. Finally, the proposed MS loss obtains new state-of-the-art performance on four image retrieval benchmarks, where it outperforms the most recent approaches, such as ABE\cite{Kim_2018_ECCV} and HTL by a large margin: 60.6% to 65.7% on CUB200, and 80.9% to 88.0% on In-Shop Clothes Retrieval dataset at Recall@1. Code is available at https://github.com/MalongTech/research-ms-loss.
1. Introduction
Deep metric learning with pair-based losses faces redundant, uninformative training pairs and difficult informative-pair identification. The paper introduces GPW to analyze pair weighting and MS loss to mine and weight pairs using multiple similarities, achieving strong image-retrieval results.
- Motivation: Random sampling can be overwhelmed by redundant pairs, slowing convergence and degrading pair-based metric-learning performance.The number of negative pairs grows quadratically with dataset size, making informative-pair identification difficult.
- Motivation: Existing sampling and weighting methods share the goal of learning from informative pairs, but identifying especially informative negative pairs remains challenging.Prior approaches use thresholds, margins, or smooth weighting functions to select or emphasize pairs.
- General Pair Weighting: GPW identifies self-similarity and two relative similarities as key factors in pair weighting, while existing methods typically explore only some of them.The relative similarities depend on comparisons with other pairs.
- General Pair Weighting: GPW unifies pair-based loss functions as pair-weighting schemes analyzed through gradient analysis.This framework casts sampling as a special case of pair weighting and supports comparison of existing methods.
- Multi-Similarity Loss: MS loss uses iterative sampling and weighting to consider self-similarity and relative similarities when collecting informative pairs.The method is designed to collect and weight informative pairs more efficiently and accurately.
- Results: +5.0% Recall@1 over ABE on CUB200 and +7.1% Recall@1 over HTL on In-Shop Clothes Retrieval were reported for MS loss.The method was evaluated on multiple image-retrieval benchmarks and outperformed current state-of-the-art approaches by a large margin.
2. Related Work
Related work develops pair-based losses, hard-sample mining strategies, and instance-weighting methods to address redundant or uninformative training samples in deep metric learning.
- Classical Pair-Based Losses: Classical pair-based losses include contrastive, triplet, quadruplet, lifted structure, N-pairs, binomial deviance, histogram, angular, and hierarchical triplet losses.These methods construct pairs, triplets, or quadruplets to learn relationships in the embedding space.
- Classical Pair-Based Losses: Lifted structure loss uses mini-batch pairwise relations but randomly samples approximately equal numbers of negative and positive pairs, discarding informative negatives.Binomial deviance loss instead evaluates costs between labels and similarities.
- Hard Sample Mining: Pair-based methods generate many redundant and uninformative samples, making sampling important for model capability and convergence.Random sampling can be overwhelmed by redundant samples and significantly degrade training.
- Hard Sample Mining: Semi-hard mining produces valid triplets only in limited numbers and can require very large batches, such as 1800.Other approaches collect hard samples offline, construct class hierarchies, or sample negatives according to distance.
- Instance Weighting: Instance weighting methods emphasize selected examples using criteria such as difficulty, variance, or confidence across related learning tasks.Examples include focal loss, active bias learning, and self-paced learning.
3. General Pair Weighting (GPW)
The General Pair Weighting framework recasts pair-based metric-learning losses as pair-weighting schemes derived through gradient analysis. It reveals how existing methods select and weight pairs, including their shared tendency to assign equal weights within selected sets or emphasize particular similarities.
- GPW formulation: GPW represents similarity through a neural embedding and an m × m matrix whose entries are pairwise dot products.Each input is projected onto a unit sphere, and Sij is the dot product of the projected vectors.
- GPW formulation: GPW formulates pair-based learning as designing weights wij for pairwise similarities, with sampling treated as a special case.The formulation is obtained by reformulating the loss gradient with respect to model parameters.
- Existing losses: Contrastive loss assigns equal weight to all positive pairs and hard negative pairs with Sij > λ, without distinguishing informativeness within the selected pairs.Its weighting is therefore a simple special case of pair weighting.
- Existing losses: Triplet loss equally weights valid triplets satisfying San + λ > Sap and discards triplets that fail this condition.It differs from contrastive loss in pair selection but shares the limitation of equal weighting among selected pairs.
- Existing losses: Lifted structure loss derives pair weights from all positive and negative pairs in a mini-batch, with separate expressions for positive and negative pairs.The weighting values are obtained by differentiating the loss with respect to pair similarity.
- Existing losses: Binomial deviance loss is a soft version of contrastive loss that assigns larger weights to negative pairs with higher similarity.Such pairs are treated as more informative because they distinguish similar samples from different classes.
4. Multi-Similarity Loss
Multi-Similarity Loss addresses the partial similarity information used by existing pair-based methods. It mines pairs using positive relative similarity, then weights them jointly using self-similarity and negative relative similarity.
- Multiple similarities: MS loss considers three pair similarities: self-similarity, positive relative similarity, and negative relative similarity.Existing methods generally use only subsets of these perspectives, whereas MS loss is designed to use all three.
- Multiple similarities: Self-similarity measures a negative pair’s cosine similarity to the anchor, with larger values identifying harder and more informative negative pairs.Contrastive and binomial deviance losses use this criterion.
- Multiple similarities: Positive relative similarity compares a pair with positive pairs sharing the same anchor, increasing when a negative pair is closer than a positive pair.Triplet and histogram losses use this type of comparison.
- Multiple similarities: Negative relative similarity compares a negative pair with other negative pairs, assigning greater importance when it carries more information than those negatives.Lifted structure, N-pairs, and NCA methods use this criterion.
- Multiple similarities: Existing methods rely on partial similarities, so a pair’s weight can remain unchanged when other relevant similarity relationships change.The paper argues that proper weighting should account for S-, P-, and N-similarity together.
- Multi-Similarity Loss: MS loss first mines informative pairs using positive relative similarity and then jointly weights selected pairs using self-similarity and negative relative similarity.The two-step design combines pair selection with soft weighting, and its derivative defines the resulting pair weights.
5. Experiments
Experiments evaluate MS loss through ablations, embedding-size studies, and comparisons with state-of-the-art retrieval methods. The results show benefits from multi-similarity mining and weighting across four benchmarks.
- Experimental Setup: MS loss is evaluated on CUB200, Cars-196, SOP, and In-Shop using standard image-retrieval Recall@K.The implementation uses an Inception network, 224 × 224 crops, random cropping and flipping for training, and center crops for testing.
- Ablation Study: Adding Similarity-N raises MS weighting Recall@1 from 71.9% to 73.2% over Binomial and improves LiftedStruct∗ from 67% to 72.2% over MS sampling.The latter improvement considers both Similarity-P and Similarity-N.
- Ablation Study: Adding Similarity-P mining consistently improves LiftedStruct∗, Binomial, and MS weighting; Binomial Recall@1 rises from 71.9% to 74.6%.The ablation uses Cars-196 with embedding size 64.
- Ablation Study: MS loss achieves the best performance among the compared pair-based methods by jointly exploring multi-similarities for pair mining and weighting.A direct BinLifted combination performs worse than single Binomial, showing that straightforward combination is insufficient.
- Embedding Size: Recall@1 increases consistently with embedding dimension through 512 on Cars-196, while 1024 does not improve performance.The evaluated dimensions are 64, 128, 256, 512, and 1024; sizes larger than 512 are unnecessary.
- Comparison with State-of-the-Art: MS loss achieves new state-of-the-art or comparable performance across CUB200, Cars-196, SOP, and In-Shop, including gains over ABE on CUB200, SOP, and In-Shop.Reported gains include +5% Recall@1 over ABE on CUB200, +1.9% on SOP, and +2.7% on In-Shop; on Cars-196, ABE remains best with a heavier ensemble model.
6. Conclusion
The paper establishes GPW as a unified gradient-based view of pair weighting and proposes MS loss to exploit three similarities jointly. MS loss achieves new state-of-the-art performance on multiple image-retrieval benchmarks.
- Conclusion: GPW unifies existing pair-based metric-learning approaches through gradient-based pair weighting analysis.It supports understanding and explaining differences and limitations among pair-based loss functions.
- Conclusion: MS loss considers all three similarities simultaneously and uses iterative pair mining and weighting to optimize the loss efficiently.The method is designed to fully exploit information from each pair.
- Conclusion: MS loss obtains new state-of-the-art performance on multiple image-retrieval benchmarks.
1. Introduction
The introduction positions GPW and MS loss within a broader effort to analyze pair-based metric-learning methods and their weighting choices.
- Introduction: The supplementary material expands the GPW framework and MS loss analysis by revisiting existing losses, direct BinLifted combination, and batch-size effects.The supplementary experiments concern CUB200 and SOP for batch-size analysis.
2. Revisit Pair-based Loss Functions
This section revisits several pair-based losses through GPW, deriving how their pair weights depend on self- or relative similarities. It also identifies histogram loss’s reliance on positive-relative similarity alone.
- N-pairs Loss: N-pairs loss is a special case of lifted structure loss that considers only a single positive pair.Its pair weighting can therefore be analyzed using the lifted-structure derivation.
- NCA Loss: NCA pair weights depend on relative similarities with neighboring pairs, emphasizing hard negatives and high-confidence positives around an anchor.
- Histogram Loss: Histogram loss estimates positive and negative pair distributions with cosine-similarity histograms divided into R bins.The positive and negative histogram values are denoted h+ and h−.
- Pair-weighting Limitations: Figure 1 illustrates BinLifted limitations: lower-similarity negatives can receive larger weights, and a negative pair’s weight can remain fixed as neighboring negatives move closer.
- Histogram Loss: Histogram loss weights a pair by comparing it with pairs of the opposite class, using only Similarity-P rather than all three similarity types.For a positive pair, the weight approximates the ratio of negative pairs with lower cosine similarity; the negative case is analogous.
3. BinLifted v.s. MS Weighting
BinLifted can overweight pairs based on only the larger of self- and relative-similarity terms, whereas MS weighting dynamically uses both to address these cases.
- A negative pair with large relative similarity can receive high weight while the method ignores or reduces the impact of Similarity-S.
- When Similarity-S is large, BinLifted can assign similar weights despite substantial differences in Similarity-N or self-similarity.
- BinLifted mainly considers the larger value from Similarity-S or Similarity-N, neglecting the smaller one.
- This weighting drawback may reduce performance considerably, even below that of a single binomial deviance loss.
- MS weighting dynamically computes negative-pair weights to address the cases illustrated in Fig. 1.
- MS weighting explores information from both Similarity-S and Similarity-N rather than focusing only on the larger value; positive pairs are analyzed analogously.
4. Effect of Batch Size
Batch size affects MS loss differently across CUB200 and SOP: CUB200 is less sensitive and declines with larger batches, while SOP benefits significantly from them.
- CUB200 is less sensitive to batch-size changes than SOP, while SOP performance benefits significantly from larger batches.
- CUB200 performance decreases as batch size increases, whereas SOP performance improves with larger batch sizes.
- Batch size impacts datasets with large inter-class variations more significantly than datasets with small inter-class variations.
- For SOP, batch sizes above 320 are required to collect enough hard negative pairs because small batches yielded no informative pair at over 20% of iterations.
- CUB200’s smaller inter-class variations make it easier to collect hard negative pairs with subtle differences.