PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Metrics from Teachers: Compact Networks for Image Embedding

Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, Arnau Ramisa

arXiv:1904.03624v1cs.CV

TL;DR

Metric-learning distillation for compact image embeddings has been little explored, despite the cost of running large networks on constrained devices. The paper proposes absolute and relative teacher losses and reports retrieval gains over students trained without a teacher, including on a very small network.

  • Problem

    Large networks are often impractical for real-life applications, while network distillation has hardly been explored for metric-learning embeddings.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes absolute distillation that matches teacher and student embeddings and relative distillation that matches distances between data points.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Recall@1 gains over an equally sized student trained without a teacher were 6.3% on CUB-200-2011, 29.9% on Cars-196, and 6.3% on Stanford Online Products.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The relative teacher consistently outperforms the absolute teacher, and applying distillation to MobileNet-0.25 improves Recall@1 by 17.1%.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)

1. Introduction

Large networks can produce strong embeddings, but their computational cost limits deployment on devices with constrained power and battery life. The paper applies teacher–student distillation to metric learning and reports improved retrieval performance from small students across several datasets.

1. Introduction

  • Mobile devices’ limited computing power and battery life make it difficult to run large networks in real time.The introduction identifies reducing network traffic and server costs, and placing computation on end-user devices, as motivations for efficient networks.Source: §1 · p. 1
  • Feature embeddings must place items that users deem similar close together despite visual differences such as viewpoint, illumination, or image quality.Pairs or triplets of related and unrelated items are used to organize the embedding space.Source: §1 · p. 1
  • Network compression reduces a network’s parameter count, whereas distillation uses a typically large teacher to guide a small student without restricting the student’s architecture.The paper focuses on distillation for efficient feature-embedding computation.Source: §1 · pp. 1–2
  • Distillation improves retrieval over directly training the student on the data, and the relative teacher consistently outperforms the absolute teacher across the evaluated datasets.The paper also examines hint and attention layers, unlabeled training data, and transferring from high-quality teacher images to low-quality student images.Source: §1 · p. 2

2. Related work

Prior work on metric learning developed Siamese and Triplet networks, while distillation research largely targeted classification. Only two earlier works addressed distillation for embeddings, motivating the paper’s proposed relative-teacher comparison with DarkRank and PKT.

2. Related work

  • Metric-learning networks developed from Siamese contrastive-loss models to Triplet networks, while pair or triplet sampling and the cost of hard-negative mining remained challenges.Hard-negative mining can cause severe overfitting; semi-hard mining was introduced as a solution, and later approaches restricted mining to the current mini-batch to reduce cost.Source: §2 · p. 2
  • Most network-distillation work focused on image classification, with prior applications also including object detection and pedestrian detection.Earlier methods included teacher–student compression, temperature-scaled probability targets, intermediate hint losses, and multiple-teacher or student setups.Source: §2 · p. 2
  • Only two prior works addressed embedding distillation: Chen et al. used rank matching, and PKT modeled sample interactions as a probability distribution.The paper notes that Chen et al.’s list-wise loss can overflow when computing permutation probabilities, limiting batch size; it reports that its relative teacher outperforms DarkRank and PKT.Source: §2 · p. 2

3. Preliminaries

The preliminaries frame embeddings as representations whose distances should reflect task-relevant similarity, and introduce metric-learning and network-distillation concepts used by the paper. The paper considers transferring either embedding coordinates or pairwise distances from teacher to student.

3.1. Metric Learning

  • The paper’s two embedding teachers transfer either the teacher’s exact embedding coordinates or distances between data-point pairs to the student.The teacher network is frozen during student training.Source: Figure 1 · p. 3
  • Metric learning maps image features into an embedding space where L2 distance correlates with the desired notion of similarity.Feature extraction transforms pixels into a high-level representation of semantic characteristics relevant to the task.Source: §3.1 · pp. 2–3
  • Triplet loss uses an anchor, a same-category positive, and a different-category negative, requiring the negative to be farther from the anchor than the positive by a margin.Triplet networks impose local embedding constraints, and the paper evaluates distillation with triplet-loss embeddings.Source: §3.1 · p. 3

3.2. Network Distillation

  • For classification distillation, the student is trained with ground-truth and teacher-output cross-entropy losses, balanced by λ.The teacher’s softened output can convey which classes might be confused with the true label; temperature τ relaxes the signal from network outputs, and τ=1 gives the standard student output without temperature scaling.Source: §3.2 · p. 3

4. Distillation for Metric Learning

The section extends knowledge distillation to metric-learning embeddings, treating either embedding coordinates or pairwise distances as knowledge for a student. It also explores transferring intermediate hints and attention alongside the metric-learning and distillation losses.

4. Distillation for Metric Learning

  • The section extends knowledge distillation to embedding networks, which project inputs into embeddings used for distance computation.It also considers hint and attention transfer for these networks.Source: §4 · p. 3, §4.1 · p. 3

4.1. Knowledge distillation for embedding networks

  • An absolute teacher matches each student embedding to the teacher's coordinates, while a relative teacher matches distances between data-point pairs.The teacher remains frozen; the student is trained with both the standard metric-learning loss and a teacher loss weighted by a trade-off parameter.Source: §4.1 · p. 4
  • When two student embeddings preserve different relationships between samples, the relative loss favors the embedding equivalent to the teacher up to translation, whereas the absolute loss assigns both equal loss.The authors connect the relative loss to multidimensional scaling.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2. Learning from hints and attention

  • Hint transfer adds a loss on intermediate teacher representations, and the authors train that loss simultaneously with either absolute or relative distillation rather than in stages as in.The hint loss has a balancing weight, and the scheme applies hint and attention losses on multiple layers.Source: §4.2 · p. 5
  • Attention transfer matches student and teacher spatial attention maps, conveying which image locations the teacher treats as relevant.The attention-based objective includes a weight for the attention loss.Source: §4.2 · p. 5

5. Experimental Results

The experiments evaluate retrieval on CUB-200-2011, Cars-196, and Stanford Online Products, then examine semi-supervised training, very small students, and cross-quality distillation. Across these settings, the reported results show gains from the proposed distillation losses, especially the relative teacher.

5.1. Retrieval on Fine-grained Datasets

  • The relative teacher outperforms DarkRank and PKT on CUB-200-2011 and Cars-196, with reported gains of 1.8% and 4.9% on CUB and 2.3% and 29.7% on Cars.The teacher network also exceeds the directly trained student on CUB-200-2011 (58.9% versus 51.7% R@1); the teacher-student gap narrows at higher recall levels.Source: §5.1 · p. 6
  • Distillation improves student retrieval across the evaluated datasets; on Cars-196, the relative teacher even outperforms the teacher network by 1.8%.On CUB-200-2011, Recall@1 improves by 3.2% with the absolute teacher and 6.3% with the relative teacher; on Stanford Online Products, both distillation losses outperform direct student training by 6.0%.Source: §5.1 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
  • Hint layers were unstable and helped only with the absolute teacher, while attention layers gave small gains and relative-teacher performance remained stable across a broader range of λ values.The absolute teacher worked only within a narrow λ range.Source: §5.1 · p. 7

5.2. Semi-Supervised Learning

  • Adding unlabeled training images improved Recall@K with both distillation losses, especially the relative teacher; relative distillation also gained 4.5% over a student trained with 50% labeled data.The experiment used half the training images per class as labeled data and the rest as unlabeled data; the no-label setting used a trained teacher to train the student.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Very Small Student Networks

  • With MobileNet-0.25 on CUB-200-2011, relative distillation improves Recall@1 by 17.1% and Recall@16 by 15.5% over the original student model.MobileNet-0.25 has almost 40 times fewer parameters than ResNet-18 and 160 times fewer than ResNet-101.Source: §5.3 · p. 7, Table 5 · p. 8

5.4. Cross Quality Distillation

  • Cross-quality distillation improves results in most settings, with relative teachers consistently surpassing absolute teachers and more noticeable gains on Cars-196.The experiments transfer between high- and low-resolution images and between images cropped for the teacher and uncropped images for the student; the authors describe Cars-196 as more challenging because same-category cars can have different colors.Source: §5.4 · p. 8, Table 6 · p. 8

6. Conclusions

The paper distills teacher knowledge into two losses for efficient image-embedding networks. It reports Recall@1 gains across three datasets and for MobileNet-0.25, alongside benefits from unlabeled data and cross-quality distillation.

6. Conclusions

  • Recall@1 gains were 6.3% on CUB-200-2011, 29.9% on Cars-196, and 6.3% on Stanford Online Products versus same-capacity students trained without teachers.Source: §6 · p. 8
  • MobileNet-0.25 distillation improved Recall@1 by 17.1%.Source: §6 · p. 8
  • The method proposes two losses to communicate teacher-network knowledge to student networks.Source: §6 · p. 8
  • The experiments verify the benefit of adding unlabeled data for embedding learning.Source: §6 · p. 8
  • Embeddings learned from high-quality images can improve a student network that accesses only low-quality images.Source: §6 · p. 8

Related papers

Loading 1904.03624v1…