PaperlayerView exact revision on arXiv

Source-linked AI summary

DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang

arXiv:1707.01220v2cs.CVcs.LGcs.NE

TL;DR

Powerful metric-learning networks are costly to deploy in real-time settings, motivating more efficient student models. DarkRank transfers cross-sample similarities through listwise rank matching, and experiments across metric-learning tasks report significant student improvements, further enhanced by combining transfer methods.

  • Problem

    Powerful networks improve metric learning but bring expensive computation, while existing knowledge-transfer methods miss relationships across different samples.

  • Method

    DarkRank transfers teacher-to-student cross-sample similarities by introducing listwise learning-to-rank into deep metric learning.

  • Results

    Across various metric-learning tasks, DarkRank significantly improves student networks and can be combined with existing methods for better transfer performance.

  • Takeaways & Limitations

    Cross-sample similarity transfer provides a complementary route to compressing and accelerating metric-learning models alongside methods that use intra-instance knowledge.

  • Takeaways & Limitations

    Soft transfer considers n! possible rankings and is feasible only when the list length n is not too large; hard transfer is faster with similar performance.

Abstract

from arXiv · show

We have witnessed rapid evolution of deep neural network architecture design in the past years. These latest progresses greatly facilitate the developments in various areas such as computer vision and natural language processing. However, along with the extraordinary performance, these state-of-the-art models also bring in expensive computational cost. Directly deploying these models into applications with real-time requirement is still infeasible. Recently, Hinton etal. have shown that the dark knowledge within a powerful teacher model can significantly help the training of a smaller and faster student network. These knowledge are vastly beneficial to improve the generalization ability of the student model. Inspired by their work, we introduce a new type of knowledge -- cross sample similarities for model compression and acceleration. This knowledge can be naturally derived from deep metric learning model. To transfer them, we bring the "learning to rank" technique into deep metric learning formulation. We test our proposed DarkRank method on various metric learning tasks including pedestrian re-identification, image retrieval and image clustering. The results are quite encouraging. Our method can improve over the baseline method by a large margin. Moreover, it is fully compatible with other existing methods. When combined, the performance can be further boosted.

Introduction

Deep metric learning benefits computer vision but powerful networks impose heavy computation and latency costs. DarkRank transfers cross-sample similarities from teacher to student networks through rank matching, complementing existing metric-learning losses and improving student performance.

  • Motivation: Powerful deep metric-learning architectures improve feature representations but impose heavy computation burdens in latency-critical applications.The paper highlights autonomous driving as an example where hardware resources are limited and real-time response is required.
  • Motivation: Existing knowledge-transfer methods supervise individual samples but miss similarities and distances between different samples.Cross-sample relations also encode the teacher’s embedded-space structure and fit metric learning’s instance-level supervision.
  • Contributions: DarkRank introduces cross-sample similarities as a new knowledge type for transfer in deep metric learning.The method is designed for model compression and acceleration.
  • Contributions: DarkRank formalizes teacher–student transfer as a rank-matching problem and adapts classical listwise learning-to-rank methods.The student architecture combines standard classification, contrastive, and triplet losses with the proposed similarity-transfer loss.
  • Results: Across various metric-learning tasks, DarkRank significantly improves student networks and can be combined with existing methods for better transfer performance.The paper reports compatibility with other methods rather than presenting a single task-specific result.

Related works

Related work covers deep metric learning, its loss functions, knowledge distillation, and learning-to-rank methods. DarkRank extends knowledge transfer to cross-sample similarities using listwise ranking.

  • Deep Metric Learning: Deep metric learning transforms raw features with neural networks and compares samples directly in Euclidean space.This distinguishes it from traditional approaches that learn Mahalanobis distances in Euclidean or high-dimensional kernel spaces.
  • Deep Metric Learning: Metric-learning losses aim to separate inter-class embeddings and reduce intra-class variance, but classification, verification, and triplet losses have distinct limitations.Classification can ignore intra-class variance, verification may impose overly strict margins, and triplet methods require careful sampling and training.
  • Knowledge Transfer: Knowledge distillation transfers softened classifier outputs that provide more information about each sample than one-hot labels.The related-work discussion presents distillation as approximating an ensemble or transferring teacher knowledge to a smaller network.
  • Learning to Rank: Learning to rank orders candidate samples for a query according to similarity using pointwise, pairwise, or listwise objectives.DarkRank introduces listwise ranking loss into deep metric learning to transfer soft candidate–query similarities into student models.

Background

Listwise ranking methods assign probabilities to permutations of candidate samples according to their relevance or similarity scores. ListNet and ListMLE use these probabilistic rankings in different objective formulations.

  • Listwise Ranking: ListNet and ListMLE associate probabilities with rank permutations using candidate–query relevance or similarity scores.This probabilistic treatment is the common foundation of the classical listwise methods reviewed here.
  • Permutation Representation: A permutation specifies the ordering of candidate indices, such as ranking the fourth sample first and the third sample second.The candidate matrix X contains samples as columns, with each column representing one candidate embedding.
  • ListNet: ListNet defines an objective after constructing the probability of a single permutation, using a score function based on distance between candidate and query.The score function connects embedding distances to ranking probabilities.
  • ListNet: The score function S(x) is based on the distance between candidate x and query q.This distance-based score supplies the relevance signal used by the ranking formulation.
  • ListMLE: ListMLE differs from ListNet by maximizing the likelihood of a ground-truth ranking.Its objective is defined over the target permutation rather than the full ranking distribution.

Our Method

DarkRank transfers cross-sample similarity knowledge from a teacher to a student by framing the task as ranking possible candidates. It supports soft and hard transfer variants and combines the resulting loss with ground-truth metric-learning losses.

  • Our Method: DarkRank transfers cross-sample similarities by framing teacher–student knowledge transfer as a ranking problem.The framework uses learning-to-rank ideas to preserve relationships among samples rather than only per-sample supervision.
  • Our Method: The method constructs similarity scores from Euclidean distances between embeddings, using scale and contrast parameters.A mini-batch supplies an anchor query and the remaining samples as candidates.
  • Our Method: DarkRank defines soft transfer by matching teacher and student distributions over all permutations with KL divergence.This transfers information from the teacher’s distribution over possible rankings.
  • Our Method: Hard transfer maximizes the student likelihood of the ranking assigned highest probability by the teacher.The hard-transfer objective is given as Lhard(Xs, Xt) = −log P(πy | Xs, Xt).
  • Our Method: Hard transfer is much faster than soft transfer on long lists while achieving similar performance in the experiments.Soft transfer considers n! possible rankings, limiting its feasibility when n is large.
  • Our Method: Student training combines ground-truth large margin softmax, verification, and triplet losses with either soft or hard DarkRank loss.The gradient derivation and the combined objective support optimization of the student network.

Experiments

The experiments evaluate DarkRank across person re-identification, image retrieval, and clustering using established datasets and protocols. The setup uses teacher and student networks, task-specific preprocessing, and visualization of Market1501 retrieval results before and after transfer.

  • Experiments: DarkRank is evaluated on person re-identification, image retrieval, and clustering tasks.The experiments compare the method with baselines and related work and include hyper-parameter ablations.
  • Datasets: CUHK03 experiments use detected images and report Rank-1, Rank-5, and Rank-10 performance on the first standard split.The dataset contains 13,164 images of 1,360 identities captured across two cameras.
  • Datasets: Market1501 experiments report mAP and Rank-1 accuracy under single- and multiple-query settings.The dataset contains 32,668 images of 1,501 identities from six camera views.
  • Datasets: CUB-200-2011 retrieval and clustering use the first 100 bird species for training and the remaining 100 for evaluation.The reported metrics are F1, NMI, and Recall@1.
  • Implementation Details: Market1501 visualizations show results before and after DarkRank transfer, with image border colors indicating each image’s relation to the query.The caption states that teacher knowledge helps the student learn a better distance metric for image similarities.

Compared Methods

The experiments compare DarkRank with knowledge distillation and direct distance matching baselines. These baselines represent single-sample teacher supervision and direct alignment of teacher and student cross-sample distances.

  • Compared Methods: The comparison includes soft and hard DarkRank variants, their combinations with other methods, and several baseline approaches.The study evaluates both standalone and combined transferring methods.
  • Compared Methods: Knowledge Distill uses softened softmax targets from the teacher, with temperature T = 4 and loss weight 42.This baseline is included because the model contains a classification loss.
  • Compared Methods: Direct Match matches the distances between a query and candidate samples produced by teacher and student models.It serves as a straightforward baseline for transferring cross-sample similarity knowledge.

Person ReID Results

DarkRank improves student-network performance across Market1501 and CUHK03 person re-identification experiments. Its gains complement knowledge distillation, while improvements are less significant on the stronger CUHK03 baseline.

  • Evaluation: Table 1 and Table 2 report Market1501 and CUHK03 person re-identification results using mAP, Rank-1, Rank-5, and Rank-10 accuracy.Market1501 uses mAP and Rank-1; CUHK03 reports Rank-1, Rank-5, and Rank-10.
  • Comparison: Directly matching teacher and student distances yields only marginal improvement over the original student model.The authors attribute this to the student’s limited capacity to match exact teacher distances.
  • Market1501: DarkRank’s soft and hard variants make significant improvements over the original student model on Market1501.The two variants achieve similar satisfactory results, while hard transfer is computationally preferable during training.
  • Complementarity: Combining DarkRank with KD further improves student performance, indicating complementary transfer of interinstance knowledge.The authors identify this interinstance knowledge as information ignored by KD.
  • CUHK03: CUHK03 shows similar trends, but its higher baseline performance makes the improvement less significant.The comparison concerns the observed trend relative to Market1501, not a reported numerical value.

Ablation Analysis

The ablation study examines contrast, embedding scaling, and transfer-loss weighting for soft DarkRank on CUHK03. Performance peaks at selected parameter values and remains stable across a broad loss-weight range.

  • Setup: Soft DarkRank ablations evaluate how hyper-parameters affect person re-identification performance.The analysis focuses on the CUHK03 validation set.
  • Contrast β: β = 3.0 produces the peak model performance on the CUHK03 validation set.The contrast parameter sharpens score differences when candidate distances are close.
  • Scaling factor α: α = 3.0 produces the peak model performance on the CUHK03 validation set.The scaling factor compensates for reduced representation power associated with small embedding norms.
  • Loss weight λ: Setting the transfer-loss weight λ to 2.0 balances the transfer loss and original training loss.Performance remains stable across a large range of λ values.

Transfer without Identity

DarkRank transfers useful embedding knowledge beyond identity-labeled re-identification settings. Experiments cover FitNet transfer on Market1501 and retrieval and clustering on CUB-200-2011.

  • FitNet comparison: FitNet alone achieves performance similar to DarkRank alone on Market1501, while combining them yields a significant improvement.The result supports complementarity between embedding matching and DarkRank’s transferred information.
  • Task scope: Image clustering groups images by visual similarity, whereas image retrieval finds the most similar gallery images for a query.Both tasks rely on embeddings and pairwise distances such as Euclidean or Mahalanobis distance.
  • Parameter evaluation: Figure 3 evaluates Rank-1, Rank-5, and Rank-10 performance while varying parameters on the CUHK03 validation set.The supplied figure caption identifies the metrics and validation setting but not the individual plotted outcomes.
  • CUB-200-2011: DarkRank achieves significant margins on F1, NMI, and Recall@1 for image clustering and retrieval on CUB-200-2011.F1 and NMI evaluate clustering, while Recall@1 evaluates retrieval.

Speedup

Table 5 compares teacher and student networks by complexity and performance. Using off-the-shelf architectures without further implementation optimization, the method achieves about 3X wall-time acceleration with minor performance loss.

  • Speedup: 3X wall-time acceleration is achieved with minor performance loss using off-the-shelf teacher and student architectures.Speed was tested on Pascal Titan X with MXNet, without further implementation optimization.

Conclusion

The paper introduces cross-sample similarities for compressing and accelerating deep metric learning models, transferring them through a modified listwise rank loss. The method improves student performance across metric-learning tasks, works with other transfer methods, and achieves about three times wall-clock speedup with minor performance loss.

  • Conclusion: Cross-sample similarities are introduced as a new knowledge type for deep metric learning model compression and acceleration.
  • Conclusion: A modified classical listwise rank loss bridges teacher and student networks to transfer cross-sample knowledge.
  • Conclusion: Student performance significantly improves across various metric-learning tasks through the proposed knowledge transfer.
  • Conclusion: Combining cross-sample transfer with methods exploiting intra-instance knowledge further narrows the performance gap between teachers and students.
  • Conclusion: About 3X wall-clock speedup is achieved with minor performance loss without deliberately tuning the network architecture.
Loading 1707.01220v2…