PaperlayerView exact revision on arXiv

Source-linked AI summary

A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses

Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, Ismail Ben Ayed

arXiv:2003.08983v3cs.LGcs.CVstat.ML

TL;DR

Deep Metric Learning has largely favored complex pairwise losses, while standard cross-entropy appears unrelated because it does not explicitly use pairwise distances. The paper connects cross-entropy and pairwise losses through bound-optimization and mutual-information analyses, and reports state-of-the-art benchmark results with cross-entropy.

  • Problem

    Deep Metric Learning commonly relies on complex pairwise-distance losses, whereas standard cross-entropy has been overlooked because its relevance to pairwise embedding properties is unclear.

  • Method

    The paper theoretically links cross-entropy and pairwise losses through an upper-bound optimization view and discriminative and generative mutual-information views.

  • Results

    Cross-entropy achieves state-of-the-art results on several standard DML benchmark datasets.

  • Takeaways & Limitations

    Cross-entropy serves as a proxy for maximizing the mutual-information objective associated with pairwise losses without complex sample-mining and optimization schemes.

  • Takeaways & Limitations

    Standard cross-entropy is difficult to use when only relative labels are available and samples may belong to multiple classes simultaneously.

Abstract

from arXiv · show

Recently, substantial research efforts in Deep Metric Learning (DML) focused on designing complex pairwise-distance losses, which require convoluted schemes to ease optimization, such as sample mining or pair weighting. The standard cross-entropy loss for classification has been largely overlooked in DML. On the surface, the cross-entropy may seem unrelated and irrelevant to metric learning as it does not explicitly involve pairwise distances. However, we provide a theoretical analysis that links the cross-entropy to several well-known and recent pairwise losses. Our connections are drawn from two different perspectives: one based on an explicit optimization insight; the other on discriminative and generative views of the mutual information between the labels and the learned features. First, we explicitly demonstrate that the cross-entropy is an upper bound on a new pairwise loss, which has a structure similar to various pairwise losses: it minimizes intra-class distances while maximizing inter-class distances. As a result, minimizing the cross-entropy can be seen as an approximate bound-optimization (or Majorize-Minimize) algorithm for minimizing this pairwise loss. Second, we show that, more generally, minimizing the cross-entropy is actually equivalent to maximizing the mutual information, to which we connect several well-known pairwise losses. Furthermore, we show that various standard pairwise losses can be explicitly related to one another via bound relationships. Our findings indicate that the cross-entropy represents a proxy for maximizing the mutual information -- as pairwise losses do -- without the need for convoluted sample-mining heuristics. Our experiments over four standard DML benchmarks strongly support our findings. We obtain state-of-the-art results, outperforming recent and complex DML methods.

1 Introduction

The paper addresses the apparent mismatch between cross-entropy and pairwise metric-learning losses by providing optimization and mutual-information connections. It reports that cross-entropy achieves state-of-the-art results across several DML benchmarks without complex sample-mining schemes.

  • Motivation: Deep Metric Learning commonly uses pairwise losses to pull same-class samples together and push different-class samples apart.These formulations often require practical optimization schemes that are difficult to implement.
  • Motivation: Cross-entropy has been largely overlooked in DML because it is a point-wise classification loss without explicit pairwise distances.Prior empirical evidence was limited and lacked theoretical explanation.
  • Contributions: The paper links cross-entropy and several pairwise losses through explicit optimization analysis and mutual-information arguments.The losses are shown to maximize a common mutual-information objective between learned embeddings and labels.
  • Contributions: Cross-entropy is an upper bound on an underlying pairwise loss, so minimizing it approximately minimizes intra-class distances while maximizing inter-class distances.This is framed as an approximate bound-optimization or Majorize-Minimize procedure.
  • Contributions: The paper establishes relations between pairwise DML losses and the generative view of mutual information, and shows cross-entropy maximizes its discriminative view.These results provide the paper’s two principal theoretical perspectives.
  • Results: Cross-entropy achieves state-of-the-art results on several standard DML benchmark datasets.The experiments support using cross-entropy as an alternative to complex pairwise-loss optimization schemes.

2 On the two views of the mutual information

Mutual information between learned features and labels has equivalent discriminative and generative views. The former emphasizes label identification from features, while the latter emphasizes feature-space structure.

  • Definition: Mutual information quantifies the amount of information shared by two random variables.The paper focuses on mutual information between learned features and labels.
  • Discriminative view: The discriminative view writes mutual information as H(Y) − H(Y | bZ), emphasizing label uncertainty remaining after observing learned features.Its maximization favors features from which labels are easily identified, while label balance is outside the model’s control.
  • Two views: The two views are analytically equivalent but provide complementary interpretations of the same mutual-information objective.The paper uses these interpretations to connect cross-entropy and pairwise losses.
  • Generative view: The generative view emphasizes spreading learned features through the feature space while keeping samples from the same class close.This view focuses on shaping the learned feature distribution rather than only identifying labels.

3 Pairwise losses and the generative view of the MI

The paper interprets pairwise DML losses through the generative view of mutual information, decomposing them into tightness and contrastive components. These components respectively compact same-class clusters and spread different classes apart, while bound relationships connect several losses.

  • Four DML losses—center, contrastive, SNCA, and MS—are interpreted as proxies for maximizing the generative view of mutual information.The analysis begins with contrastive loss and generalizes the connection to the other pairwise losses.
  • The tightness component minimizes intra-class distances so samples from each class form compact clusters.It is related to the conditional entropy H( bZ|Y ) and, for contrastive loss, is equivalent up to constants to the center-loss tightness term.
  • Contrastive tightness is modeled as a conditional cross-entropy involving a Gaussian centered at each class centroid, yielding an upper bound on conditional entropy.The bound is tight when bZ|Y follows the corresponding Gaussian distribution, so minimizing the term encourages low-entropy class clusters.
  • The contrastive component maximizes inter-class distances and prevents the trivial solution in which all embedded points collapse to one location.For contrastive loss, only negative pairs within margin m contribute to the contrastive cost.
  • Minimizing the full contrastive loss acts as a proxy for maximizing mutual information between labels and embedded features.The tightness and contrastive terms measure feature spread through different gradient dynamics while jointly connecting the loss to MI.
  • Center, contrastive, SNCA, and MS losses are explicitly related through bound relationships for their tightness and contrastive components.Under ℓ2-normalized features and balanced classes, the tightness terms satisfy the stated relations; the SNCA and MS contrastive terms share a lower bound related to H( ˆZ).

4 Cross-entropy does it all

The paper links cross-entropy to pairwise metric-learning losses through bound optimization and mutual information, showing that it can shape embeddings without complex sample-mining schemes.

  • 4 Cross-entropy does it all: Cross-entropy is connected to pairwise metric-learning losses through explicit optimization and mutual-information perspectives.The paper relates cross-entropy to several established and recent pairwise losses.
  • 4.1 The pairwise loss behind unary cross-entropy: Alternately minimizing cross-entropy over encoder parameters W and classifier weights θ approximates bound optimization of the Pairwise Cross-Entropy (PCE) loss.PCE is introduced as the underlying pairwise objective for this interpretation.
  • 4.1 The pairwise loss behind unary cross-entropy: The cross-entropy can be interpreted as constructing and minimizing an auxiliary upper bound on PCE after optimizing the classifier’s final-layer weights.The optimal classifier weights are related to class centroids in the embedded feature space.
  • 4.1 The pairwise loss behind unary cross-entropy: PCE combines tightness and contrastive parts, encouraging samples from the same class to remain close while separating samples and class soft means from the origin.Its contrastive component pushes samples apart and moves soft means away from the origin.
  • 4.1 The pairwise loss behind unary cross-entropy: The simplified SPCE removes PCE’s dependence on recomputing λ through eigenvalues, is easier to compute, and exhibits similar training behavior to cross-entropy.SPCE also has a form resembling contrastive losses used in unsupervised representation learning.
  • 4 Cross-entropy does it all: Minimizing conditional cross-entropy is equivalent to maximizing mutual information between learned features and labels, aligning cross-entropy with other DML losses.This information-theoretic view reinforces the bound-optimization connection.

5 Experiments

The experiments evaluate cross-entropy-based metric learning across four standard benchmarks using varied architectural, sampling, augmentation, and normalization choices. The approach achieves state-of-the-art or near-state-of-the-art recall, including with Euclidean distance on non-normalized features.

  • Model architecture and pre-training: The implementation uses standard architectures, including a ResNet-50 initialized with ImageNet-pretrained weights.The comparison limits architectural modifications and uses PyTorch.
  • Sampling: Training uses random sampling across all samples with mini-batches of 128 instead of class-balanced pairwise sampling.This differs from common DML schemes that fix the number of classes and samples per class in each mini-batch.
  • Data Augmentation: Data augmentation resizes images, extracts random patches, and resizes them to 224 × 224 before training.The resizing procedure differs across CUB and Cars versus SOP and In-Shop.
  • Evaluation setup: Experiments evaluate CUB200, Cars-196, SOP, and In-Shop, reporting the distance used to compute recall.CUB, Cars, and SOP use train and evaluation splits, while In-Shop uses query and gallery sets.
  • Results: On all datasets, the method reports state-of-the-art results except Cars, where a cross-entropy-trained method achieves similar recall.Euclidean distance on non-normalized features is also near state-of-the-art across all four datasets, while ℓ2-normalization during training can hinder final performance.

6 Conclusion

The conclusion links cross-entropy and pairwise DML losses through bound optimization and mutual information. It attributes observed performance differences mainly to training optimization, while noting that pairwise losses require more tuning than cross-entropy.

  • 6 Conclusion: Cross-entropy minimization is an approximate bound-optimization procedure for the Pairwise Cross-Entropy loss.The introduced pairwise loss has a structure similar to existing DML losses.
  • 6 Conclusion: Cross-entropy and pairwise losses maximize a common mutual-information objective between embedded features and labels.The connection is developed through generative and discriminative views of mutual information.
  • 6 Conclusion: Performance differences observed in previous work are attributed mainly to the optimization process during training.Cross-entropy uses unary terms, whereas traditional DML losses optimize pairwise terms requiring tuning of batch size, sampling, and pair weighting.

A Proofs

The proofs establish equivalence and bound relationships among several metric-learning losses under balanced classes and feature normalization assumptions. They connect center, contrastive, SNCA, and MS losses through Jensen-based arguments and related objectives.

  • A Proofs: The proofs assume balanced classes and use feature normalization to connect cosine and Euclidean distances.For balanced classes, each class contains the same number of features in the proof setting.
  • A Proofs: Center loss is linked to contrastive loss by expressing class-centered quantities through the hard mean of each class.The equivalence is obtained after summing the class-specific relations.
  • A Proofs: Contrastive loss is linked to SNCA loss using the convexity of −log(x) and Jensen’s inequality.The proof is completed by summing the class-wise inequalities.
  • A Proofs: MS loss is linked to contrastive loss using the concavity of log(x) and Jensen’s inequality.This establishes another bound relationship among standard pairwise losses.
  • A Proofs: The contrastive terms of MS and SNCA are upper bounds on a contrastive objective containing a feature-entropy proxy and a tightness term.The result connects these losses to differential entropy estimation.

Contrastive terms:

These derivations characterize cross-entropy as an upper bound on Pairwise Cross-Entropy and connect mutual-information maximization to conditional entropy and classifier divergence. The resulting optimization interpretations relate classifier weights to class means and cross-entropy training to pairwise structure.

  • Pairwise Cross-Entropy: The proof chooses λ so the relevant objective becomes convex in the classifier parameters through a positive-semidefinite Hessian.This permits searching for minima of the component functions.
  • Pairwise Cross-Entropy: The optimal linear-classifier weights are related to class centroids, supporting an auxiliary-function interpretation of cross-entropy optimization.The encoder update then acts as the second step of bound optimization on Pairwise Cross-Entropy.
  • Pairwise Cross-Entropy: Cross-entropy is an upper bound on Pairwise Cross-Entropy at every training iteration.Optimizing the classifier makes the bound nearly tight, so minimizing both losses with respect to the encoder becomes approximately equivalent.
  • Mutual information: Mutual information I(bZ; Y) equals label entropy minus conditional label entropy, with label entropy constant during optimization.Maximizing mutual information therefore acts through minimizing H(Y | bZ), which depends on the learned embeddings.
  • Mutual information: Cross-entropy simultaneously minimizes conditional label entropy and the KL divergence between labels and classifier predictions conditioned on embeddings.The decomposition suggests a two-step Maximize-Minimize view that separately adjusts embeddings and classifier predictions.

B Preliminary results with SPCE

On MNIST, the study tracks CE, SPCE, and validation accuracy during training to examine their empirical behavior.

  • Fig. 1 compares the training evolution of CE and SPCE alongside validation accuracy for both losses on MNIST.The experiment uses a small four-layer CNN with Adam and feature dimension d = 100.

C Analysis of ranking losses for Deep Metric Learning

The analysis extends the tightness/contrastive interpretation to ranking losses by decomposing them into terms that shape positive and overall distance distributions.

  • Ranking-loss setup: Ranking losses encourage points to be well ordered by distance rather than imposing manual margins between every positive and negative pair.
  • Random-variable formulation: For a query point, D represents distances to other embedded points, while R records whether each point is positive or negative relative to the query.
  • Bound decomposition: Jensen’s inequality yields a lower bound decomposed into TAP, a tightness term, and CAP, a contrastive term.
  • Intuitive interpretation: A single-positive Monte Carlo approximation uses a sampled positive distance d+ to interpret the tightness component.
  • Geometric interpretation: Maximizing TAP pulls positive points inside the radius-d+ hypersphere, whereas maximizing CAP pushes positive and negative points outside it.

D On the limitations of cross-entropy

Although CE is easier to optimize than pairwise losses, straightforward CE becomes prohibitive for relative labels and very large class counts.

  • Straightforward CE has scenarios in which its use becomes prohibitive despite being easier to optimize than pairwise losses.
  • Case of relative labels: Relative-label settings may assign each sample to multiple classes, making standard CE difficult to use.
  • Case of relative labels: SPCE can handle relative-label problems after reformulation, similarly to other pairwise losses.
  • Case of large number of classes: With millions of classes, storing the CE classifier weight matrix θ ∈ R^K×d can become prohibitive.
  • Case of large number of classes: Class-sampling heuristics can address large-class CE training, but they undermine the goal of avoiding additional training heuristics.
Loading 2003.08983v3…