Source-linked AI summary
Rethinking Feature Distribution for Loss Functions in Image Classification
Weitao Wan, Yuanyi Zhong, Tianpeng Li, Jiansheng Chen
TL;DR
Softmax-based classification does not explicitly formulate the likelihood of extracted features under the training distribution. The paper proposes a large-margin Gaussian Mixture loss with likelihood regularization, and reports improved classification across datasets while enabling likelihood-based abnormal-input detection. Its scope depends on the assumed Gaussian Mixture feature distribution and adversarial examples can be optimized toward high likelihood.
Problem
Softmax loss leaves the likelihood of an extracted feature under the training feature distribution insufficiently formulated, limiting distribution-based assessment of inputs.
Method
The L-GM loss assumes Gaussian Mixture-distributed deep features, combines posterior cross-entropy with likelihood regularization, and introduces a classification margin.
Results
The L-GM loss outperforms softmax loss and its variants across small- and large-scale datasets while producing GM-like feature spaces and more effective abnormal-input distinction.
Takeaways & Limitations
A trained L-GM model can both classify inputs and estimate their likelihood under the learned training-feature distribution, supporting refusal of decisions for sufficiently unlikely inputs.
Takeaways & Limitations
The method assumes Gaussian Mixture-distributed training features, and targeted FGSM can approximately jointly optimize classification and likelihood to generate high-likelihood adversarial examples.
Abstract
from arXiv · showhide
We propose a large-margin Gaussian Mixture (L-GM) loss for deep neural networks in classification tasks. Different from the softmax cross-entropy loss, our proposal is established on the assumption that the deep features of the training set follow a Gaussian Mixture distribution. By involving a classification margin and a likelihood regularization, the L-GM loss facilitates both a high classification performance and an accurate modeling of the training feature distribution. As such, the L-GM loss is superior to the softmax loss and its major variants in the sense that besides classification, it can be readily used to distinguish abnormal inputs, such as the adversarial examples, based on their features' likelihood to the training feature distribution. Extensive experiments on various recognition benchmarks like MNIST, CIFAR, ImageNet and LFW, as well as on adversarial examples demonstrate the effectiveness of our proposal.
1. Introduction
Softmax loss supports classification but does not explicitly model the likelihood of training features. The proposed Gaussian Mixture loss adds likelihood modeling and a classification margin, producing GM-like feature spaces and enabling likelihood-based abnormal-input detection.
- Softmax loss provides a probabilistic classification interpretation, but the likelihood of an extracted feature under the training distribution remains unspecified.
- The Gaussian Mixture loss assumes class-conditional GM-distributed features, computes posterior probabilities with Bayes’ rule, and adds likelihood regularization.
- Learned feature likelihood can identify abnormal inputs, with Figure 1 showing adversarial examples having extremely low likelihood under the learned GM distribution.
- GM-loss features approximately follow a Gaussian Mixture distribution, unlike features learned with softmax loss and its variants.
- The proposed loss combines a classification margin with likelihood modeling without requiring the complicated distance function used by large-margin softmax loss.
2. Related Work
Prior softmax-based approaches encourage feature compactness and class separation through pairwise, triplet, centroid, or angular constraints. Their limitations include combinatorial costs and inconsistent distance measurements, while adversarial examples motivate modeling feature distributions for detection.
- Contrastive and triplet losses encourage intra-class compactness and inter-class separation but require pairwise or triplet computations that grow combinatorially with dataset size.
- Center loss avoids pairwise or triplet computation by minimizing feature-to-centroid distances, but this formulation introduces inconsistency in feature-space distance measurements.
- Angular-margin softmax addresses distance-measurement concerns by introducing an angular margin through a sophisticated differentiable angular distance function.
- Because adversarial examples are expected to occupy distributions different from the learned training feature space, feature-distribution modeling offers a basis for distinguishing them.
3. Gaussian Mixture Loss
The GM loss models deep training features as a Gaussian mixture whose components represent classes, combining posterior-based classification with likelihood regularization. A classification margin further promotes separation, while the likelihood term connects feature modeling to center loss and supports likelihood-based abnormal-input detection.
- GM loss formulation: The GM loss assumes each class’s deep features form a Gaussian component, enabling posterior probabilities through Bayes’ rule.The model uses class means, covariances, and prior probabilities to represent the training feature distribution.
- GM loss formulation: Classification loss alone does not force training features to follow the assumed Gaussian mixture, so the method adds negative-log-likelihood regularization.The regularizer measures how well training samples fit the assumed distribution and is combined with classification loss in the GM loss.
- GM loss formulation: The classification and likelihood terms target discriminative capability and probabilistic distribution while sharing the same feature-space parameters.This couples class prediction with modeling of the training feature distribution.
- Large-Margin GM Loss: A non-negative margin can be added using squared Mahalanobis distances, requiring the correct class to be closer to its mean than competing classes by at least m.The margin formulation is straightforward because squared Mahalanobis distances are non-negative.
- Large-Margin GM Loss: The adaptive margin sets m = αd_zi, with α controlling the expected separation between class means in the training feature space.Figure 2 contrasts α = 0 without a margin against α > 0 with a large-margin GM loss.
- A Discussion on Llkd: Under identity covariance and uniform class priors, center loss is equivalent to likelihood regularization up to a constant.This identifies center loss as a special case of the proposed probabilistic regularization.
- A Discussion on Llkd: Likelihood regularization can estimate a sample feature’s likelihood under the learned Gaussian mixture, allowing classification refusal when likelihood is too low.The paper discusses this use for abnormal inputs such as adversarial examples and notes that softmax-plus-center-loss training may yield inaccurate likelihood estimates when features deviate from the Gaussian mixture.
4. Experiments
Experiments across image classification, face verification, and adversarial-example detection evaluate the L-GM loss against softmax-based alternatives. The results report consistently strong recognition performance and improved separation of adversarial inputs through feature-distribution likelihood.
- Experimental scope: The experiments evaluate L-GM on image classification, face verification, and adversarial-example detection, reporting mean and standard deviation over 3 tries.All experiments use the Caffe framework on NVIDIA TitanX GPUs.
- Image Classification: L-GM features roughly follow the assumed Gaussian Mixture distribution and exhibit larger inter-class margins as α increases on MNIST.The comparison uses softmax, center loss, large-margin softmax, and L-GM with 2D feature embeddings.
- Image Classification: L-GM outperforms softmax and its variants across CIFAR-10 ResNet models with different depths and consistently outperforms softmax-based losses on CIFAR-100.Increasing α improves augmented CIFAR-100 performance, but not consistently without augmentation, where each class has only 500 training samples.
- Image Classification: L-GM is effective on large-scale ImageNet classification using ResNet-101 under both 1-crop and 10-crop evaluation practices.The models are trained for 100 epochs on ILSVRC2012 using six Titan GPUs.
- Face Verification: Using only CASIA-WebFace for training, L-GM outperforms the other loss functions in single-model LFW face verification.Verification uses PCA-reduced 512-dimensional embeddings and Mahalanobis distance.
- Beyond Classification: 3.1% EER: L-GM distinguishes MNIST adversarial examples more effectively than center loss at 10.2% and softmax at 37.7%.With ε = 0.3, most adversarial examples remain distinguishable using feature likelihood; a separate setting reports 4.3% EER.
5. Conclusions
The proposed L-GM loss models deep training features with a Gaussian Mixture distribution while adding likelihood regularization and a classification margin. Experiments report improved classification over softmax losses and more effective detection of abnormal inputs.
- L-GM assumes a Gaussian Mixture distribution for deep training features and adds log-likelihood regularization to encourage that structure.
- A classification margin is introduced to improve the generalization capability of the trained model.
- Extensive experiments report that L-GM outperforms softmax loss and its variants on small- and large-scale datasets with different deep models.
- L-GM more effectively distinguishes abnormal inputs whose extracted features follow a distribution different from the learned training feature space.
- The authors identify improved robustness toward adversarial examples as one practical use of this abnormal-input distinction.