Source-linked AI summary
On Loss Functions for Deep Neural Networks in Classification
Katarzyna Janocha, Wojciech Marian Czarnecki
TL;DR
The paper examines the underexplored choice of classification loss in deep neural networks, where log loss is commonly used despite the models’ broader flexibility. It combines theoretical analysis with experiments and finds that L1 and L2 have expected-misclassification interpretations, while several alternatives can outperform or match log loss depending on the application. Further investigation remains necessary, especially for non-classical losses.
Problem
Classification research commonly uses log loss, leaving how alternative losses affect deep-model learning dynamics and robustness comparatively underexplored.
Method
The paper analyzes loss-function properties theoretically and evaluates 12 losses experimentally across toy and classical datasets, architectures, learning behavior, performance, robustness, and convergence.
Results
Depending on the application, losses other than log loss are preferable: squared hinge loss converges faster and performs better for accuracy-focused research, while Cauchy-Schwarz Divergence matches or outperforms log loss on MNIST and CIFAR10.
Takeaways & Limitations
L1 and L2 are valid deep-classification objectives with probabilistic interpretations tied to expected misclassification, and loss selection should depend on the application.
Takeaways & Limitations
The topic remains open, with non-classical losses such as Tanimoto loss and Cauchy-Schwarz Divergence requiring further investigation.
Abstract
from arXiv · showhide
Deep neural networks are currently among the most commonly used classifiers. Despite easily achieving very good performance, one of the best selling points of these models is their modular design - one can conveniently adapt their architecture to specific needs, change connectivity patterns, attach specialised layers, experiment with a large amount of activation functions, normalisation schemes and many others. While one can find impressively wide spread of various configurations of almost every aspect of the deep nets, one element is, in authors' opinion, underrepresented - while solving classification problems, vast majority of papers and applications simply use log loss. In this paper we try to investigate how particular choices of loss functions affect deep models and their learning dynamics, as well as resulting classifiers robustness to various effects. We perform experiments on classical datasets, as well as provide some additional, theoretical insights into the problem. In particular we show that L1 and L2 losses are, quite surprisingly, justified classification objectives for deep nets, by providing probabilistic interpretation in terms of expected misclassification. We also introduce two losses which are not typically used as deep nets objectives and show that they are viable alternatives to the existing ones.
1 Introduction
Deep learning offers extensive architectural flexibility, yet classification practice largely fixes the loss function to log loss. Prior alternatives exist, but they are isolated and rarely examine how loss choice affects learning or robustness.
- Deep learning’s modularity supports many architectural, activation, and learning-algorithm choices, while classification loss is usually kept fixed as log loss.
- Hinge, squared hinge, and L1 losses have achieved strong classification results, but prior studies appear disconnected and generally do not examine loss effects systematically.
- The paper analyzes 12 loss functions, combining theoretical analysis with experiments on learning speed, final performance, input and label-noise robustness, and convergence under limited resources.
2 Theory
The paper argues that L1 and L2 have valid probabilistic interpretations for classification, while also identifying optimization drawbacks and conditions favoring particular losses. It further relates Cauchy-Schwarz Divergence to cross entropy with an entropy regularizer.
- Probabilistic interpretation: L1 minimizes expected misclassification probability, while L2 minimizes the same quantity regularized by half the expected squared L2 norm of predicted probabilities.These objectives differ from log loss, which maximizes the probability of completely correct labelling.
- Robustness: The expectation-loss interpretation suggests greater robustness to outliers and label noise because a single heavily misclassified point has a smaller effect than under log loss.The paper names L1 and L2 as expectation and regularised expectation losses, respectively.
- Optimization properties: L1 and L2 applied after sigmoid or softmax have non-monotonic final-layer derivatives that vanish for heavily misclassified examples.The associated loss is neither convex nor concave with respect to final-layer weights, which can slow learning.
- Optimization properties: Piecewise-linear, nonconstant final-layer derivatives are proposed as favorable for first-order optimization, and among the analyzed losses this property holds only for L2 and squared hinge loss.The argument combines the piecewise-linear deep-learning hypothesis with the nonlinear probability output of sigmoid or softmax.
- Additional loss relation: Cauchy-Schwarz Divergence loss is equivalent to cross entropy regularized with half of the expected Rényi quadratic entropy of predictions.This relation provides the paper’s theoretical justification for considering the divergence as a neural-network objective.
3 Experiments
Experiments compare loss functions across toy, MNIST, and CIFAR10 settings, showing distinct trade-offs in fitting, learning speed, generalisation, and noise robustness. Margin-based losses generally train and generalise well, while expectation losses are slower but robust to input and label noise.
- Toy datasets: Toy experiments separate losses into those that fit dense low-dimensional data and those struggling to reduce error to zero.This distinction becomes clearer with five hidden layers and includes L1, Chebyshev, Tanimoto, and expectation loss.
- MNIST: Linear models behave qualitatively differently from networks with hidden nonlinearities, so conclusions from linear models do not transfer directly to deep nets.For example, regularised expectation loss is strongest for the linear MNIST model but is far from best after nonlinearities are introduced.
- Learning dynamics: Hinge2 and hinge3 are generally fastest, while L2 also becomes fast with more than one hidden layer; expectation losses train much more slowly.The slower expectation-loss training is attributed to vanishing partial derivatives on heavily misclassified points.
- Generalisation: Margin-based losses generally outperform other loss families in generalisation, despite strong dropout being used throughout the analysed networks.The authors also report that typical L1 or L2 regularisation penalties do not work well in deep networks.
- Cross-dataset comparison: Cauchy-Schwarz Divergence performs equally well or better than log loss on MNIST and CIFAR10 for both learning speed and final performance.The result is reported using a simple CIFAR10 convolutional architecture rather than a state-of-the-art model.
- Noise robustness: Expectation losses show high robustness to both input and label noise, while margin losses are usually slightly worse than expectation losses but better than log loss.For input noise, training examples receive Gaussian perturbations sampled from N(0, ϵI); label-noise results are described as analogous.
4 Conclusions
The paper analyzes how classification-loss choices affect deep-network training and final characteristics, challenging log loss’s dominance and identifying application-dependent alternatives. It also concludes that the topic remains open, especially for less conventional losses.
- Loss-function choice affects deep neural network training and the characteristics of the resulting models.The analysis examines effects beyond final accuracy, including training behavior and robustness.
- Log loss is not universally preferable: squared hinge loss suits accuracy-focused applications, while expectation losses suit highly noisy inputs and labels.Squared hinge loss is reported to converge faster, perform better, and offer greater robustness to label noise and slight input noise; expectation losses are favored under high noise in both spaces.
- Theoretical analysis gives regression-associated losses a valid probabilistic interpretation for deep-network classification.This supports considering losses beyond the usual classification objectives.
- The study cautions that intuitions from linear models rarely transfer to highly nonlinear deep networks.
- Further work is needed because the topic is not exhausted, particularly for Tanimoto loss and Cauchy-Schwarz Divergence.