Source-linked AI summary
Squared Earth Mover's Distance-based Loss for Training Deep Neural Networks
Le Hou, Chen-Ping Yu, Dimitris Samaras
TL;DR
The paper addresses the limitation that softmax cross-entropy ignores relationships among classes in single-label classification. It trains CNNs with exact squared Earth Mover’s Distance and learns the required ground distance matrix from CNN features. The method reports state-of-the-art performance on datasets with strong class relationships while avoiding adverse effects when those relationships are weak.
Problem
Softmax cross-entropy focuses on the ground-truth class probability and does not account for informative inter-class relationships.
Method
The paper uses exact squared EMD2 as a CNN loss or regularizer and can learn its ground distance matrix from CNN features without inter-class relationship priors.
Results
The method achieves state-of-the-art results on datasets with strong inter-class relationships, while learned matrices avoid spurious relationships on datasets without strong relationships.
Takeaways & Limitations
EMD2-based CNN training incorporates class relationships and can be applied across datasets without requiring prior relationship assumptions.
Takeaways & Limitations
Direct optimization of the EMD2 loss can converge to uniformly distributed predictions, so the formulation is modified for use as a regularizer.
Abstract
from arXiv · showhide
In the context of single-label classification, despite the huge success of deep learning, the commonly used cross-entropy loss function ignores the intricate inter-class relationships that often exist in real-life tasks such as age classification. In this work, we propose to leverage these relationships between classes by training deep nets with the exact squared Earth Mover's Distance (also known as Wasserstein distance) for single-label classification. The squared EMD loss uses the predicted probabilities of all classes and penalizes the miss-predictions according to a ground distance matrix that quantifies the dissimilarities between classes. We demonstrate that on datasets with strong inter-class relationships such as an ordering between classes, our exact squared EMD losses yield new state-of-the-art results. Furthermore, we propose a method to automatically learn this matrix using the CNN's own features during training. We show that our method can learn a ground distance matrix efficiently with no inter-class relationship priors and yield the same performance gain. Finally, we show that our method can be generalized to applications that lack strong inter-class relationships and still maintain state-of-the-art performance. Therefore, with limited computational overhead, one can always deploy the proposed loss function on any dataset over the conventional cross-entropy.
1. Introduction
The paper argues that cross-entropy overlooks informative relationships between classes and proposes exact squared Earth Mover’s Distance losses for CNN classification, including a learned ground distance matrix. Experiments report improved performance on datasets with strong inter-class relationships and protection against adverse effects when relationships are weak.
- Motivation: Cross-entropy considers the predicted probability of the ground-truth class but ignores informative inter-class relationships.For age-group classification, predictions with equal cross-entropy can differ in how plausibly their probability mass is distributed across classes.
- Method: The proposed exact squared EMD2 loss uses inter-class dissimilarities to train CNNs for multi-class classification.It can operate as a stand-alone loss or as a regularization term, with squaring chosen because it usually improves gradient-descent convergence.
- Motivation: Existing EMD approaches require a predefined ground distance matrix, motivating the paper’s learned-matrix approach.Prior matrices depend on assumptions such as ordered classes or external similarity information.
- Method: The paper proposes learning the ground distance matrix from CNN features during training without prior assumptions about inter-class relationships.The learned matrix is used to compute EMD2 as a regularization term with limited additional computational cost.
- Results: On datasets with weak inter-class relationships, the learned distance matrix does not capture spurious relationships that could adversely affect performance.This extends the method beyond settings with known ordered classes.
- Results: The authors report state-of-the-art results on datasets with strong inter-class relationships and performance improvements over softmax cross-entropy CNNs.The evaluation includes age estimation and image aesthetic assessment datasets.
2. EMD2-based Loss
The paper formulates exact EMD2 loss for CNN-based single-label classification, using inter-class distances to shape penalties across the predicted distribution. For ordered classes, the loss has a closed form based on cumulative distributions and backpropagates through all predicted probabilities.
- 2.2.1 Earth Mover’s Distance: EMD measures the minimum transport cost between prediction and target distributions using a ground distance matrix between classes.
- 2.2.2 Ordered-class classification: Ordered-class classification represents classes by ranks, allowing distances between classes to be defined from their rank differences.
- 2.2.3 EMD2 loss for ordered-class classification: For ordered classes, exact EMD is computed in closed form from cumulative distribution functions, and EMD2 is selected because it generally optimizes faster.
- The EMD2 formulation is differentiable with respect to network parameters and can be optimized using standard backpropagation.
- Softmax cross-entropy backpropagation depends only on the ground-truth probability, whereas EMD2 loss uses all prediction elements.
3. Self-Guided EMD
The self-guided method learns a ground distance matrix from the CNN’s features during training and uses exact EMD2 as a cross-entropy regularizer. The approach addresses unknown inter-class relationships with O(C) single-label EMD computation, while direct standalone EMD optimization can converge to an undesirable uniform-prediction minimum.
- 3. Self-Guided EMD: The method learns the ground distance matrix from CNN features during training and computes single-label EMD2 in O(C) time.It is designed for settings where inter-class relationships are not known beforehand.
- 3.1. Estimating ground distances: Class representations are initialized from centroids of L1-normalized second-to-last-layer CNN features.
- 3.1. Estimating ground distances: The initial distance matrix is row-wise percentile-mapped to spread poorly separated early-training distances across values between 0 and 1, then symmetrized.
- 3.2. Self-guided EMD2 regularization: For single-label targets, all predicted mass is transported to the ground-truth class, so the transportation flow from each predicted class equals its probability.
- 3.2. Self-guided EMD2 regularization: Directly optimizing exact EMD does not reach the desired local optimum because gradient descent can lower every class probability and produce uniform predictions.
- 3.2. Self-guided EMD2 regularization: The hybrid loss uses EMD2 as a regularizer, with λ controlling its weight, ω controlling distance sensitivity, and µ providing a distance bias.
4. Experiments and Results
Experiments across ordered-class, aesthetics, and general classification datasets show that EMD2-based losses generally outperform conventional losses, while self-guided training learns useful class relationships without requiring prior matrices. The method also adds limited training overhead and no test-time cost.
- Computational complexity of EMD: EMD2-based losses add less than 10% of CNN training time per iteration, require no additional test time, and reach cross-entropy performance in one-third as many epochs.The added cost comes from loss computation and storing features to estimate the ground distance matrix.
- Age estimation on Adience dataset: VGGF-XEMD2 outperforms the state-of-the-art VGGF-DEX method on Adience without using external face datasets.The comparison with VGGF-DEX plus IMDB-WIKI involves an external dataset ten times larger than Adience.
- Age estimation on Adience dataset: The learned Adience distance matrix assigns larger distances to farther age groups and captures semantic similarities such as 15–20s being closer to 25–32s than 8–13s.The matrix is learned by RESF-XEMD2 during training.
- Ordered-class datasets: EMD2-based losses outperform softmax cross-entropy on ordered-class datasets, while self-guided variants perform as well as prior-informed EMD.This pattern is reported for the Images of Groups dataset and the Adience experiments.
- Image aesthetics: On AADB, EMD2-based losses significantly outperform cross-entropy and L2 regression, and an eight-network image-only ensemble exceeds the prior attribute-augmented state of the art.The ensemble averages predictions from eight VGGF-EMD networks trained with different discretizations of aesthetic labels.
- Generalization on ImageNet: On ImageNet, the self-guided method can be applied without adverse performance effects because it finds weak inter-class relationships rather than imposing spurious structure.The reported standard deviation of pairwise class-centroid distances is 0.00614 for ImageNet, lower than the ordered-class datasets listed.
5. Conclusion
The study shows that exact squared EMD losses improve CNN classification by incorporating inter-class relationships, while learned relationships remove the need for prior knowledge. The approach also remains effective when those relationships are weak.
- Exact squared EMD losses incorporate inter-class relationships that conventional softmax cross-entropy ignores.
- The method significantly outperformed state-of-the-art regression-based and cross-entropy-based CNNs on two age-estimation and one image-aesthetic-assessment dataset.
- The approach efficiently discovers inter-class relationships without prior knowledge or external datasets, using only image information.
- The method also applies to datasets with weak inter-class relationships without adverse results.