Revisiting Training Strategies and Generalization Performance in Deep Metric Learning

Karsten Roth; Timo Milbich; Samarth Sinha; Prateek Gupta; Björn Ommer; Joseph Paul Cohen

Source-linked AI summary

Revisiting Training Strategies and Generalization Performance in Deep Metric Learning

Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Björn Ommer, Joseph Paul Cohen

arXiv:2002.08473v9Feb 19, 2020cs.CV

TL;DR

DML research is difficult to compare because training protocols, architectures, and parameter choices diverge. The paper standardizes comparisons across objectives and training components, analyzes generalization through embedding compression, and proposes regularization for ranking-based models. Under this framework, performance differences saturate more than prior literature suggests, while the proposed regularization boosts ranking-based DML across benchmark datasets.

Problem
Divergent DML training protocols, architectures, and parameter choices make unbiased comparison difficult.
Method
The paper systematically evaluates DML objectives, training choices, mini-batch sampling, and embedding-space structure, then regularizes compression for ranking-based models.
Results
DML objectives show greater performance saturation under consistent comparison, and compression and embedding density strongly correlate with generalization.
Takeaways & Limitations
Regularizing embedding-space compression boosts ranking-based DML performance across standard benchmark datasets.
Takeaways & Limitations
The study omits In-Shop experiments because its data distribution is very similar to SOP and relative results are considered transferable.

Abstract

from arXiv · show

Deep Metric Learning (DML) is arguably one of the most influential lines of research for learning visual similarities with many proposed approaches every year. Although the field benefits from the rapid progress, the divergence in training protocols, architectures, and parameter choices make an unbiased comparison difficult. To provide a consistent reference point, we revisit the most widely used DML objective functions and conduct a study of the crucial parameter choices as well as the commonly neglected mini-batch sampling process. Under consistent comparison, DML objectives show much higher saturation than indicated by literature. Further based on our analysis, we uncover a correlation between the embedding space density and compression to the generalization performance of DML models. Exploiting these insights, we propose a simple, yet effective, training regularization to reliably boost the performance of ranking-based DML models on various standard benchmark datasets. Code and a publicly accessible WandB-repo are available at https://github.com/Confusezius/Revisiting_Deep_Metric_Learning_PyTorch.

1. Introduction

The paper addresses difficult-to-compare DML research by standardizing analyses of objectives, training choices, sampling, and architectures. It links generalization to embedding compression and proposes regularization that improves ranking-based models.

Figure 1 compares mean recall and standard deviation for objectives trained with and without the proposed regularization.
DML comparisons are difficult because studies vary in architectures, objectives, training tasks, and often undisclosed technical details.
Embedding compression shows a strong correlation with DML generalization, whereas targeted links to inter-class margins and intra-class variance are not reliably established.
The study analyzes DML objectives, training strategies, data sampling, and model-design choices to establish a consistent benchmark.
The proposed regularization targets embedding-space compression and boosts generalization performance for ranking-based DML approaches on standard benchmarks.

2. Related Works

Related DML work includes ranking-based objectives over tuples, proxy-based approaches that approximate class distributions, and more involved extensions to objectives, data, and embedding spaces.

Ranking-based DML methods operate on pairs, triplets, quadruplets, or higher-order sample sets.
The number of triplets scales as O(N^3) with dataset size, motivating methods that reduce tuple-mining costs.
Proxy-based approaches approximate full class distributions with one or more learned representatives.
Other extensions split and merge data or embedding spaces, use ensembles of specialized learners, or extend existing objectives.

3. Training a Deep Metric Learning Model

DML training combines embedding objectives, mini-batch sampling, and architectural and optimization choices. The paper studies these components because their variation affects comparisons and performance.

3.1. The objective function: DML learns an embedding function that maps data into a space where a predefined distance reflects visual similarity.
3.1. The objective function: Ranking objectives minimize anchor-positive distances and maximize anchor-negative distances, often subject to a margin.
3.1. The objective function: Classification-based objectives optimize discriminative predictions, while proxy-based methods compare samples directly with learned class representatives.
3.2. Data sampling: Mini-batch mining is treated as an objective-independent process for selecting informative batches using labels or embedded-data distributions.
3.2. Data sampling: DDM and FRD select candidate batches whose distance or embedding distributions most closely match a reference batch.
3.3. Training parameters, regularization and architecture: Training choices include architectures, batch size, weight decay, preprocessing, and embedding dimensionality, all of which can affect performance.

4. Analyzing DML training strategies

The study evaluates DML objectives under controlled architectures, parameters, preprocessing, and mini-batch sampling. It finds that training choices materially affect results, performance differences between objectives are often smaller than reported, and diverse sampling strategies can improve outcomes.

Experimental protocol: The protocol uses ResNet50 with frozen Batch-Normalization layers and 128-dimensional embeddings to support comparable DML evaluation.Training uses fixed learning rate 10^-5 without scheduling, with standardized image cropping and horizontal flipping.
Parameters and architectures: Larger batch sizes generally improve results, with particularly high relevance on the SOP dataset.The effect varies among training criteria.
Parameters and architectures: Preprocessing choices produce large performance differences: larger images improve performance across objectives and datasets, while advanced augmentation effects depend on the dataset.Random resized cropping is generally stronger than basic resizing and cropping.
Objective and mining comparisons: The study compares 14 DML objectives and four mining methods under fixed training conditions, using distance-based tuple mining for ranking-based models.The comparison includes random, semihard, and soft hard sampling for a triplet-loss study.
Objective comparisons: Under consistent training, performance saturates across objectives more than literature suggests, and standard deviations make improvements still less significant.Margin loss and multisimilarity loss perform best on average, but not by a notable margin; many losses compete with or outperform more involved methods on SOP.

5. Generalization in Deep Metric Learning

The study links DML generalization most strongly to spectral compression, while embedding density shows a weaker, dataset-dependent association. It uses these findings to motivate ρ-regularization for ranking-based approaches.

Spectral compression: Generalization is inversely correlated with spectral decay ρ(Φ), with correlations ≤−0.63 across CUB200-2011, CARS196 and SOP.Lower ρ indicates more directions of significant variance in the embedding space.
Embedding structure: Neither mean intra-class nor inter-class distance consistently correlates significantly with generalization across all three datasets.The analysis compares these distance measures with spectral decay and density-related ratios.
ρ-regularization: The toy example shows that regularization lowers spectral decay and enables separation of test classes that standard contrastive training fails to separate.The figure also compares singular-value spectra with and without regularization.
Embedding structure: For CUB200-2011 and CARS196, higher embedding density πratio is linked to stronger generalization, whereas SOP estimates are likely too noisy because of class imbalance.πratio is defined as πintra/πinter and is interpreted as an embedding-space density measure.
ρ-regularization: The authors propose ρ-regularization by randomly switching negative and positive samples in ranking-loss tuples with probability pswitch.This pushes same-class samples apart and counteracts compression induced by strong discriminative training signals.

6. Conclusion

The paper studies divergent DML training practices, relates generalization to representation compression and embedding density, and proposes regularization for ranking-based methods. The technique boosts performance across benchmark datasets.

Conclusion: The study improves DML comparability by comprehensively analyzing training components and objectives under divergent protocols.It examines generalization through compression and embedding density before proposing a ranking-based regularization technique.
Conclusion: Highly compressed representations disregard features useful for capturing data characteristics that transfer to unknown test distributions.The conclusion connects representation compression with generalization under shifted distributions.
Conclusion: The proposed technique regularizes learned embedding-space compression and boosts ranking-based performance across all benchmark datasets.The conclusion presents this as the practical outcome of the preceding analysis.

Reviewer Comments

Reviewer responses state that augmentation and resolution were analyzed, In-Shop was omitted because its distribution resembles SOP, and editorial corrections and references were addressed.

Data augmentation and resolution: The authors analyzed the impact of different image augmentations and input resolutions on performance.They identify data augmentation as an important factor for training regularization.
Dataset scope: In-Shop experiments were omitted because its data distribution is very similar to SOP, making relative results generally transferable.This is the stated scope boundary for the dataset experiments.
Editorial revisions: The authors corrected the reported grammar and typographical errors.
Editorial revisions: The authors added all mentioned references in response to the missing-reference comment.

Supplementary: Revisiting Training Strategies and Generalization

The supplement describes the DML objectives, tuple-mining strategies, proxies, and hyperparameter settings used in the study. It covers ranking, classification-style, proxy-based, and distribution-oriented formulations.

Training objectives: DML training objectives operate on pairs, triplets, or larger sets of samples to enforce similarity-based ordering or structure.The supplement discusses contrastive, triplet, quadruplet, lifted-structure, N-Pair, angular, margin, and MultiSimilarity formulations.
Tuple mining: Distance-weighted sampling is used for triplet-based training and for the first negative in the quadruplet implementation.The triplet implementation contrasts earlier random or semihard sampling with the more effective distance-weighted strategy.
Classification-style objectives: ArcFace adapts softmax classification to retrieval by enforcing an angular margin between embeddings and approximate class centers.The implementation uses γ = 0.5 and scaling s = 16, with class-center learning rate 0.0005.
Margin loss: Margin loss introduces a dynamic, learnable boundary β between positive and negative pairs.The boundary learning rate is 0.0005, with initial β values of 0.6 or 1.2; the implementation uses distance-weighted triplet sampling.
MultiSimilarity: MultiSimilarity loss jointly evaluates anchor-positive, anchor-negative, positive-positive, and negative-negative similarities.The study uses α = 2, β = 40, λ = 0.5, and ϵ = 0.1.
Proxy-based objectives: ProxyNCA and SoftTriple reduce tuple-sampling demands by comparing embeddings with learnable class or intra-class proxies.ProxyNCA uses class proxies, while SoftTriple learns multiple normalized proxies per class.

A.3. Evaluation Metrics

The evaluation uses nearest-neighbour retrieval, clustering, and precision–recall metrics on the test set. Recall-based mean average precision and NMI assess complementary retrieval and clustering quality.

Retrieval metrics: Recall@k measures whether a query has at least one same-class sample among its top k nearest neighbours.The metric averages this criterion across queries.
Clustering metrics: NMI evaluates clustering quality by comparing K-Means cluster assignments with true class-label partitions.The number of clusters is set to the number of classes.
Retrieval metrics: F1-score is the harmonic mean of precision and recall, giving equal importance to both retrieval quantities.Precision and recall are defined over nearest-neighbour retrieval for Recall@k.
Retrieval metrics: mAP@C averages class-wise average precision while recalling only as many samples as each class contains.The supplement also reports mAP@1000.

B. Evaluating the robustness of pswitch choices

Performance rises with pswitch until saturation, and the gains remain reasonably robust across pswitch choices. The broader analyses associate improved generalization with reduced embedding-space spectral decay.

pswitch robustness: Performance increases are reasonably robust against changes in pswitch.The figure provides mean recall performance and standard deviations for objectives trained with and without ρ-regularization.
Spectral decay: ρ-regularization is associated with higher performance and lower spectral decay across training objectives.The contrastive-loss gain exceeds 5% on CARS196, and the analysis links this to its unconstrained contraction of entire classes.
Spectral decay: Class-conditioned SVD shows that ρ-regularization decreases the average singular-value decay.The supplement averages sorted first-35 singular values across classes and reports a pattern similar to the total-spectrum analysis.

F. Comparison to state-of-the-art approaches on SOP dataset

On SOP, the carefully specified margin baseline outperforms the compared state-of-the-art implementations despite differences in architectures and ensembles. The supplement also illustrates regularization, spectral decay, and distribution-shift experiments.

SOP comparison: The margin-loss baseline outperforms each compared state-of-the-art model on SOP, including methods using varying architectures.The baseline uses a ResNet50 backbone and margin loss with β = 1.2.
SOP comparison: The baseline improves over MIC and D&C using the same backbone by at least 0.9%.It also remains competitive with Rank and ABE despite their more powerful network ensembles.
Toy generalization example: The toy example contrasts horizontal training discrimination with vertical test discrimination and shows regularization introducing additional features for separation.The example uses a normalized two-layer fully connected network trained with standard contrastive loss.
Distribution shift: ManifoldMixup produces more compressed representations and reduced generalization performance under the transfer-learning setting.The experiment evaluates Normalized Softmax, Triplet with Distance Sampling, and Margin loss on CUB200-2011.

I. Detailed Results

This section reports detailed method-comparison results across CUB200-2011, CARS196, and Stanford Online Products, alongside evaluations of batch-sampling methods.

Method comparisons: Method comparisons are reported separately for CUB200-2011, CARS196, and Stanford Online Products across relevant performance metrics.The corresponding results are presented in Tables 4, 5, and 6.
Regularization: The reported results also note the switch probability pswitch for each regularised method.This parameter is identified in the section overview of the detailed results.
Method comparisons: Training lasts 150 epochs for the CUB200-2011 and CARS196 method comparisons.The table captions specify the training duration for both datasets.
Method comparisons: The Stanford Online Products method comparison is conducted over 100 training epochs.This duration is stated in the Stanford Online Products table caption.
Batch sampling: Batch-sampling methods are compared across loss functions and sampling methods for CUB200-2011, CARS196, and Stanford Online Products.These evaluations are presented in Tables 7, 8, and 9.