PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Training Strategies and Generalization Performance in Deep Metric Learning

Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, Joseph Paul Cohen

arXiv:2002.08473v9cs.CV

TL;DR

Divergent DML protocols, architectures, and parameter choices make unbiased comparisons difficult. This paper systematically studies objectives and training choices, examines embedding compression and generalization, and reports that a compression regularizer boosts ranking-based models across benchmark datasets.

  • Problem

    Divergent DML architectures, objectives, training details, and sampling practices make unbiased comparison difficult and hinder reproducibility.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically analyzes DML objectives, training strategies, sampling, design choices, and embedding-space generalization, then proposes compression regularization for ranking-based methods.

    Source: §1 · p. 2

  • Results

    The proposed compression regularization boosts ranking-based DML performance across all benchmark datasets.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The study offers a consistent reference point for improving comparability of recent and future DML approaches.

    Source: §1 · p. 2, §6 · p. 8

    The study omits In-Shop experiments, stating that its data distribution resembles SOP and relative results are generally transferable.

    Source: Reviewer Comments · p. 9

Abstract

from arXiv · show

Deep Metric Learning (DML) is arguably one of the most influential lines of research for learning visual similarities with many proposed approaches every year. Although the field benefits from the rapid progress, the divergence in training protocols, architectures, and parameter choices make an unbiased comparison difficult. To provide a consistent reference point, we revisit the most widely used DML objective functions and conduct a study of the crucial parameter choices as well as the commonly neglected mini-batch sampling process. Under consistent comparison, DML objectives show much higher saturation than indicated by literature. Further based on our analysis, we uncover a correlation between the embedding space density and compression to the generalization performance of DML models. Exploiting these insights, we propose a simple, yet effective, training regularization to reliably boost the performance of ranking-based DML models on various standard benchmark datasets. Code and a publicly accessible WandB-repo are available at https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch.

1. Introduction

Deep metric learning learns distance-preserving embeddings for visual similarity, but varied methods and undisclosed training details make fair comparison difficult. This work benchmarks DML choices, studies generalization, and proposes a regularizer based on embedding-space compression.

1. Introduction

  • The study examines model-agnostic sampling of informative mini-batches, extending prior sampling work beyond triplet-mining strategies for ranking-based objectives.The authors identify data sampling as influential for DML optimization.Source: §1 · p. 1
  • Embedding-space compression, measured by spectral decay, strongly correlates with generalization, while large inter-class margins and intra-class variance do not reliably link to it.The paper analyzes compression alongside inter-class margins and intra-class variance.Source: §1 · p. 2
  • The paper provides an exhaustive analysis of recent DML objectives, training strategies, data sampling, and model design choices under a standard benchmark.The comparison uses identical training conditions on standard benchmark datasets.Source: §1 · pp. 1–2
  • The authors propose regularizing embedding-space compression and find that it boosts generalization performance for ranking-based DML approaches on standard benchmarks.The proposed technique is described as simple and effective.Source: §1 · p. 2

2. Related Works

Prior work spans diverse DML objectives and extensions, while mini-batch construction has received limited attention. This paper studies batch mining and embedding-space structure in relation to generalization, and proposes a regularizer for ranking-based objectives.

Deep Metric Learning

  • DML includes tuple-based ranking losses, such as pairs and triplets (Wu et al., 2017), as well as adapted classification methods.Sampling strategies and proxy-based approaches using a few virtual representatives address the computational cost of tuple-based methods (Qian et al., 2019).Source: §2 · p. 2
  • DML extensions include divide-and-conquer, ensembles, feature mining, self-supervision, and artificial-sample augmentation; the authors propose regularization to improve ranking-based objectives.These methods build on essential objectives and training parameters discussed in the study, so they can benefit from its findings.Source: §2 · p. 2

Mini-batch selection

  • Because mini-batch creation strategies have been little studied, the authors analyze diversity-maximizing mining strategies against standard DML heuristics (Wu et al., 2017; Roth et al., 2019; Sanakoyeu et al., 2019).Prior mini-batch research addressed classification convergence or informative core-set selection for generative models.Source: §2 · p. 2

Generalization in DML

  • Applying compression analysis to DML, the authors find strong compression hurts generalization and examine its correlation with learned embedding-space structure.Prior metric-learning generalization work has focused on robustness of linear or kernel distance metrics and generalization-error bounds.Source: §2 · p. 2

3. Training a Deep Metric Learning Model

The section frames a consistent comparison of deep metric learning by reviewing objective families, mini-batch sampling, and training choices that influence performance and complicate fair comparisons.

3.1. The objective function

  • DML objectives include ranking-based losses that separate same-class positives from different-class negatives, classification-based losses, and proxy-based methods that compare samples with learned class representatives.Proxy-based methods also alleviate tuple mining, an issue for ranking-based losses.Source: §3.1 · p. 3

3.2. Data sampling

  • The study examines label samplers that control class composition and embedded samplers that select a mini-batch to match statistics of a larger memory-bank sample.Label strategies include fixed samples per class or ensuring at least one same-label pair; embedded strategies include greedy coreset coverage and distance-distribution matching.Source: §3.2 · pp. 3–4

3.3. Training parameters, regularization and architecture

  • Architecture, weight decay, embedding dimensionality, image preprocessing, and batch size can affect DML performance, yet variation and incomplete reporting across studies impair fair comparison.The paper notes that architectures also differ in their initial Recall@1 after ImageNet pretraining.Source: §3.3 · p. 4

4. Analyzing DML training strategies

Under consistent training conditions, the study finds that DML performance is more saturated across methods than prior reports suggest, while parameter and batch-sampling choices materially affect comparisons.

4.1. Experimental Protocol

  • The protocol uses a ResNet50 with frozen Batch-Normalization, 128-dimensional embeddings, Adam at 10−5, SPC-2 sampling, and multiple-seed averages.Training runs for 150 epochs on CUB200-2011 and CARS196 and 100 epochs on SOP; benchmarks are evenly distributed across classes except SOP, where most classes have few instances.Source: §4.1 · pp. 4–5, §4 · p. 4

4.2. Studying DML parameters and architectures

  • Larger images improve performance across objectives and datasets, while augmentation effects depend on dataset; batch size, weight decay, preprocessing, architecture, and embedding dimension also affect results.Weight decay changes relative performance by up to 5%; objectives differ at a given embedding dimension but appear to converge at D = 512, and R50 and IBN-based setups show the highest consistency.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3. Batch sampling impacts DML training

  • Batch sampling changes mean performance by up to 1.5%; SPC-2 and FRD perform well and consistently outperform SPC-4, with no sampler winning across all datasets.The study attributes the benefit to mini-batch data diversity across objectives and finds simple heuristics such as SPC-2 sufficient, even if complex mining may perform better.Source: §4.3 · pp. 5–6, Figure 3 · p. 6

4.4. Comparing DML models

  • Across benchmarks, performance saturates across methods; margin and multisimilarity losses perform best on average without a notable margin, and many losses compete with or outperform more involved approaches on SOP.Ranking-based objectives outperform classification/NCE-based counterparts, though not significantly; carefully trained baselines can also outperform state-of-the-art approaches using stronger architectures.Source: §4.4 · p. 6

5. Generalization in Deep Metric Learning

The section links excessive embedding compression to weaker DML generalization under train–test shifts and finds that counteracting compression can improve ranking-based models’ recall. It also reports that embedding density relates to generalization on some datasets, but less consistently.

5.1. ρ-regularization for improved generalization

  • Ranking-based ρ-regularization switches negative samples with positives with probability pswitch, lowering spectral decay and improving benchmark recall with reasonable robustness to pswitch.The proposed tuple alteration pushes same-class samples apart, helping retain non-label-discriminative features; a toy example illustrates improved separation of unseen test classes, while higher compression is reported as detrimental.Source: Table 2 · p. 7, §5.1 · p. 8, Figure 7 · p. 8

5. Generalization in Deep Metric Learning

  • The study measures compression as ρ(Φ), the KL-divergence between a uniform distribution and the normalized singular-value spectrum; lower values indicate more directions of significant variance.The analysis uses the whole training embedding spectrum rather than separate class-conditioned spectra, given the train–test distribution shift.Source: §5 · p. 7
  • R@1 and spectral-decay ρ(Φ) show significant inverse correlation (≤−0.63), while distance-based measures are not consistently correlated with generalization across all three datasets.The analysis covers CUB200-2011, CARS196, and SOP.Source: §5 · p. 7
  • Higher embedding density (πratio) aligns with stronger generalization on CUB200-2011 and CARS196, but its estimate is likely too noisy on SOP.The authors interpret this as a weaker relationship than the inverse correlation with spectral decay.Source: §5 · p. 8

6. Conclusion

The study promotes more comparable Deep Metric Learning research and links embedding compression and density to generalization. It proposes a regularizer that reduces compression in ranking-based models and boosts performance across benchmark datasets.

  • A simple regularizer for ranking-based methods controls embedding-space compression and boosts performance across all benchmark datasets.The technique regularizes compression in the learned embedding space.Source: §6 · p. 8
  • A large study of DML training components and objectives addresses diverging protocols to improve comparability across recent and future approaches.Source: §6 · p. 8
  • Generalization strongly correlates with representation compression and embedding density; highly compressed representations discard helpful features that transfer to unknown test distributions.Source: §6 · p. 8

Reviewer Comments

The authors say they analyzed the effects of image augmentation and resolution, and explain that they omitted In-Shop experiments because its data distribution resembles SOP and relative results are generally transferable. They also report correcting typos and adding missing references.

Reviewer Comments

  • The authors analyzed how different image augmentations and resolutions affect performance.They describe data augmentation as an important factor for regularizing training.Source: Reviewer Comments · p. 9
  • In-Shop experiments were omitted because its data distribution resembles SOP, making relative results generally transferable, according to the authors.Source: Reviewer Comments · p. 9
  • The authors report correcting the identified grammar and typos and adding all mentioned references.Source: Reviewer Comments · p. 9

Supplementary: Revisiting Training Strategies and Generalization … A.2. Tuple Mining

The supplement describes the DML objectives, implementation choices, and tuple-mining strategies used in the study. It focuses on how ranking losses form training tuples and on alternative objective designs, including proxy-based methods.

A. Description of Methods

  • The methods use normalized image embeddings φ_i, denote their non-normalized versions φ*_i, and generally operate on mini-batches with embedding dimension D = 128.General training parameters and protocol details are covered in Sec. 4.1 of the main paper.Source: A. Description of Methods · p. 12

A.1. Training Criteria

  • Contrastive loss trains positive and negative pairs, triplet loss ranks a positive against a negative, and Generalized Lifted Structure uses all available anchor-positive and anchor-negative pairs.Quadruplet loss adds a higher-level ordering constraint using an anchor, a positive, and two exclusive negatives (Chen et al., 2017).Source: §A.1 · pp. 12, 14
  • N-Pair incorporates all mini-batch negatives, Angular adds angle-based constraints (Wang et al., 2017), Margin learns a boundary β (Wu et al., 2017), and MultiSimilarity evaluates several pair relations.The stated angular margin is α = π/4; MultiSimilarity uses cosine similarity and parameters α = 2, β = 40, λ = 0.5, and ϵ = 0.1.Source: §A.1 · pp. 13–14
  • Histogram Loss penalizes overlap between positive- and negative-pair distributions (Ustinova & Lempitsky, 2016), while SNR optimizes embeddings using a distance metric based on anchor variance and noise variance (Yuan et al., 2019).Histogram discretization R is set to 65 for CUB200-2011 and CARS196 and 11 for SOP; SNR uses γ = 0.2 and λ = 0.005.Source: §A.1 · pp. 13–14
  • ProxyNCA and SoftTriple use learnable class proxies, while Arcface and normalized softmax compare embeddings with class proxies in classification-based formulations.The passages identify ProxyNCA, SoftTriple (Qian et al., 2019), Arcface, and normalized softmax.Source: §A.1 · pp. 13–15

A.2. Tuple Mining

  • The study measures tuple sampling's effect on standard triplet loss and uses distance-weighted mining for ranking objectives other than N-Pair methods.Random mining selects a positive or negative sample for each mini-batch image.Source: §A.2 · p. 15
  • Semihard mining selects negatives that are reasonably hard but avoids overly hard candidates, while softhard mining probabilistically selects potentially hard positives and negatives to reduce collapse risk.Semihard mining is attributed to Schroff et al. (2015), and softhard mining to Roth & Brattoli (2019).Source: §A.2 · pp. 15–16
  • Distance-weighted mining samples negatives according to a distribution inverse to the analytical pairwise-distance distribution on the unit hypersphere (Wu et al., 2017).The study sets λ = 0.5 and limits distances to 1.4.Source: §A.2 · p. 16

A.3. Evaluation Metrics

The evaluation measures test-set embedding quality through nearest-neighbor retrieval and clustering metrics.

A.3. Evaluation Metrics

  • Recall@k averages whether each query has at least one same-class sample among its k nearest neighbors.The metric is evaluated over queries and their nearest-neighbor rankings.Source: §A.3 · p. 16
  • NMI measures clustering quality by comparing cluster assignments for embedded test samples with their true class labels.The passage describes clustering methods such as K-Means.Source: §A.3 · p. 16
  • F1-score is the harmonic mean of precision and recall, weighting both equally for nearest-neighbor retrieval.Precision and recall are defined over nearest-neighbor retrieval as for Recall@k.Source: §A.3 · p. 16
  • mAP@C averages class-wise average precision@kc, where kc is the number of samples in class c; mAP@1000 retrieves the 1,000 nearest samples.Both variants use nearest-neighbor rankings to determine recalled samples.Source: §A.3 · p. 17

B. Evaluating the robustness of pswitch choices … E. Analysis of per-class singular value spectra

Across these analyses, performance gains from ρ-regularization are associated with reduced spectral decay, while performance is reasonably robust to pswitch changes and class-conditioned spectra show a similar regularization effect.

B. Evaluating the robustness of pswitch choices

  • Performance increases steadily with pswitch until saturation, and the gains are reasonably robust to changes in pswitch.At saturation, the regulatory signal overshadows the discriminative one.Source: B. Evaluating the robustness of… · p. 17

C. Visual comparison of DML objectives on benchmarks

  • The full visual comparison qualitatively supports the reported saturation of DML objective performance.The passage identifies this as the full version of Fig. 1 and says it supports the saturation noted in Section 4 and Table 2.Source: C. Visual comparison of DML objectives… · p. 17

D. Correlation between performance and spectral decay ρ

  • Contrastive loss gains more than 5% on CARS196 with ρ-regularization, consistent with its unconstrained contraction of entire classes and potential for overly compressed embeddings.Unlike triplet-based formulations, which optimize relative positive and negative distances up to a fixed margin γ, contrastive loss controls only negative distances by γ.Source: D. Correlation between performance and… · pp. 17–18, Figure 9 · p. 18
  • Across ranking losses, performance gains from ρ-regularization are related to decreases in spectral decay ρ(Φ).The comparison includes training objectives with and without regularization and their corresponding spectral decay.Source: D. Correlation between performance and… · p. 17, Figure 9 · p. 18

E. Analysis of per-class singular value spectra

  • The class-conditioned average singular value spectrum also shows decreased decay with ρ-regularization.The analysis averages spectra from per-class embeddings and compares the first 35 sorted singular values for models trained with and without regularization.Source: E. Analysis of per-class singular… · p. 18
  • The per-class analysis complements whole-embedding analysis because unseen test classes may project anywhere in the embedding space, unlike class-conditioned analysis for i.i.d. classification.The paper contrasts its general compression analysis over all embedded samples with the class-conditioned analysis of Verma et al. (2018).Source: E. Analysis of per-class singular… · p. 18

F. Comparison to state-of-the-art approaches on SOP dataset · G. 2D Toy Examples · H. Influence of Manifold Mixup on DML

On SOP, the margin-loss baseline outperforms the reported approaches, while ManifoldMixup increases representation compression at the cost of DML generalization. In a 2D toy example, ρ-regularization enables separation of test classes that collapse without it.

F. Comparison to state-of-the-art approaches on SOP dataset

G. 2D Toy Examples

  • The toy experiment uses a two-layer, 30-neuron-per-layer network with 2D inputs and embeddings, trained for 200 iterations using contrastive loss and γ = 0.1.It uses batch size 24, learning rate 0.03, and regularization with pswitch = 0.001.Source: G. 2D Toy Examples · p. 19

H. Influence of Manifold Mixup on DML

  • The ManifoldMixup study trains Normalized Softmax, triplet loss with Distance Sampling, and margin loss with β = 0.6 and Distance Sampling on CUB200-2011.For ranking-based objectives, mixup is applied in ResNet50 with mixed class labels; the procedure follows ManifoldMixup.Source: H. Influence of Manifold Mixup on DML · p. 19
  • For ranking-based mixup, each mixed anchor can yield up to two class-label entries, whose possible triplet sets are loss-weighted by their interpolation coefficients.The method operates on mixup embeddings and samples interpolation values with mixup-α = 2.Source: H. Influence of Manifold Mixup on DML · pp. 19–20

G. 2D Toy Examples

  • In the 2D toy example, unregularized embeddings collapse test samples together, while ρ-regularization separates test classes and decreases spectral decay.Training lines are separable by x-coordinate, whereas test lines are separable by y-coordinate.Source: G. 2D Toy Examples · p. 19, Figure 12 · p. 20

H. Influence of Manifold Mixup on DML

I. Detailed Results

The section reports detailed DML setup comparisons across three benchmark datasets and evaluates batch-creation methods. It also notes the switch probability for each regularized method.

I. Detailed Results

  • DML setups are compared on CUB200-2011, CARS196, and Stanford Online Products using relevant performance metrics.The corresponding results are in Tables 4, 5, and 6; training runs for 150 epochs on CUB200-2011 and CARS196.Source: §I · p. 21, Table 4 · p. 22, Table 5 · p. 22
  • Batch-sampling methods are compared across loss functions on CUB200-2011, CARS196, and SOP.The detailed comparisons appear in Tables 7, 8, and 9, respectively.Source: §I · p. 21, Table 7 · p. 24, Table 8 · p. 25
  • The section also records the switch probability pswitch for each regularized method.Source: §I · p. 21

Related papers

Loading 2002.08473v9…