PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

Ehsan Amid, Manfred K. Warmuth, Rohan Anil, Tomer Koren

arXiv:1906.03361v3cs.LGstat.ML

TL;DR

The paper addresses the vulnerability of standard logistic loss to outliers and noisy labels. It introduces a Bregman-based bi-tempered loss with a low-temperature logarithm and high-temperature softmax, obtaining improved robustness on noisy classification benchmarks. The method is proper and can be bounded and heavy-tailed, though its tempered exponential requires a restricted domain when t2 < 1.

  • Problem

    Standard logistic loss is convex but can be vulnerable to outliers and mislabeled examples because its loss is unbounded and softmax has an exponentially decaying tail.

  • Method

    The paper replaces the logarithm and softmax exponential with separately tempered functions and constructs the resulting loss using Bregman divergences.

  • Results

    At 40% label noise, bi-tempered loss reaches 98.56% versus 97.64% on MNIST and 62.55% versus 53.17% on CIFAR-100, and improves ImageNet-2012 results with ResNet18 and ResNet50.

  • Takeaways & Limitations

    Tuning the two temperatures yields a proper loss that can be bounded, heavy-tailed, and more robust to label noise than vanilla logistic loss.

  • Takeaways & Limitations

    When 0 ≤ t < 1, the tempered exponential’s inverse-property domain must be restricted to x ≥ -1/(1 - t).

Abstract

from arXiv · show

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non-convex already in the single layer case. When replacing the last layer of the neural nets by our bi-temperature generalization of logistic loss, the training becomes more robust to noise. We visualize the effect of tuning the two temperatures in a simple setting and show the efficacy of our method on large data sets. Our methodology is based on Bregman divergences and is superior to a related two-temperature method using the Tsallis divergence.

1 Introduction

The paper replaces logistic loss’s standard logarithm and softmax exponential with separately tempered functions, producing a bounded, heavy-tailed loss designed to improve robustness to noisy labels. Its Bregman-divergence construction supports non-convexity when the temperatures differ and yields strong results on noisy MNIST, CIFAR-100, and ImageNet-2012.

  • 1 Introduction: The standard logistic loss is convex in last-layer activations but can be sensitive to outliers and mislabeled examples because its loss grows without bound.The paper also identifies the exponentially decaying softmax tail as a limitation addressed by the tempered formulation.
  • 1 Introduction: The method replaces the logarithm and exponential in logistic loss with tempered versions, using a low-temperature logarithm and high-temperature exponential.The tempered logarithm supports bounded losses for t1 < 1, while the tempered exponential provides heavier tails for t2 > 1.
  • 1 Introduction: When t1 < t2, the bi-tempered loss becomes non-convex in output activations; t1 = t2 preserves convexity through a matching Bregman divergence.For 0 ≤ t1 < 1 the loss is bounded, and t2 > 1 makes the probability assignment heavy-tailed.
  • 1 Introduction: The proposed Bregman-based construction is proper, unlike the related two-temperature method based on Tsallis divergence.Properness is presented as a requirement for many real-world applications.
  • 1.1 Our replacement of the softmax output layer in neural nets: The neural-network replacement uses a tempered softmax with t2 > 1 and requires computing its normalization value by binary search or an iterative procedure.The loss reduces to vanilla logistic loss when t1 = t2 = 1.
  • 1.3 Summary of the experiments: 98.56% versus 97.64% on MNIST and 62.55% versus 53.17% on CIFAR-100 under 40% label noise favor bi-tempered loss over logistic loss.The method also improves over state-of-the-art results on ImageNet-2012 with ResNet18 and ResNet50.

2 Preliminaries

The preliminaries formulate classification losses using Bregman divergences and matching transfer functions on the probability simplex. Convex duality explains when matching losses are convex in activations, while temperature mismatch enables bounded non-convex losses.

  • Bregman divergences generalize familiar measures such as squared Euclidean and KL divergence, while remaining nonnegative but generally asymmetric.
  • Matching losses: A matching loss maps activations to probabilities with a transfer function and then evaluates a Bregman divergence against the target distribution.
  • Matching losses: When the transfer function is the gradient of the restricted convex dual, the matching loss is convex in the activations.
  • Matching losses: The standard logistic loss is recovered from the softmax transfer function paired with the relative-entropy KL divergence.
  • Matching losses: Using a larger transfer-function temperature than divergence temperature constructs bounded non-convex losses with heavy-tailed transfer functions.

3 Tempered Matching Loss

Tempered matching losses replace the logarithm-based convex construction with a temperature-indexed Bregman family. Temperatures below one yield bounded divergences, while simplex-restricted duality supplies the tempered probability transfer function.

  • The temperature-indexed convex function F_t has gradient log_t, producing a tempered Bregman divergence that includes several familiar divergence families.
  • For 0 ≤ t < 1, the tempered divergence is bounded on the probability simplex.
  • Restricting the convex function to the simplex defines a constrained dual whose transfer function produces tempered softmax probabilities.
  • The matching loss remains convex in activations for the matched tempered construction, with λ_t computed numerically when no closed form is available.

4 Robust Bi-Tempered Logistic Loss

The Bi-Tempered Logistic Loss combines a tempered divergence with a tempered softmax whose temperatures can differ. This construction yields a proper, Bayes-risk-consistent loss and an unbiased sample-based estimator.

  • The Bi-Tempered Logistic Loss introduces a mismatch between divergence and probability-assignment temperatures, using tempered softmax probabilities in the loss.
  • The prescribed temperatures make the loss bounded and the probability assignment heavier-tailed than vanilla softmax, properties linked to handling noisy examples.
  • 4.1 Properness and Monte-Carlo sampling: The sample-based loss is an unbiased approximation of the expected loss and is therefore proper.
  • The multiclass bi-tempered logistic loss is Bayes-risk consistent even in the non-convex case.

5 Experiments

Experiments compare the bi-tempered loss with logistic loss on noisy MNIST, CIFAR-100, and ImageNet-2012. The bi-tempered method generalizes better under label noise and improves ImageNet top-1 accuracy for both evaluated architectures.

  • 5 Experiments: The experiments evaluate the bi-tempered loss against logistic loss on corrupted-label MNIST and CIFAR-100, plus ImageNet-2012 with ResNet-18 and ResNet-50.MNIST and CIFAR-100 use artificially corrupted training labels, while ImageNet-2012 contains some inherent mislabeling.
  • 5.1 Corrupted labels experiments: The bi-tempered loss outperforms logistic loss on clean-test top-1 accuracy at every tested noise level, including noise-free CIFAR-100.Its accuracy remains strong at high label-noise levels, whereas logistic-loss accuracy drops with much smaller noise.
  • 5.1 Corrupted labels experiments: The bi-tempered loss overfits label noise later and less severely than logistic loss, yielding better final test accuracy and lower model variance on noisy MNIST.Figure 3 tracks noise-free training, noisy training, and noise-free test accuracy over optimization.
  • 5.2 Large scale experiments: The robust bi-tempered loss produces a significant top-1 accuracy gain over logistic loss for both ResNet-18 and ResNet-50 on ImageNet-2012.The large-scale experiments train both architectures for 180 epochs using the stated TPU setup.

6 Conclusion and Future Work

The paper concludes that tunable bi-tempered losses address robustness issues associated with logistic loss by combining bounded losses with heavy-tailed softmax alternatives. Temperature choices control convexity and robustness behavior in the last layer.

  • 6 Conclusion and Future Work: The construction combines bounded tempered losses for large-margin outliers with a heavy-tailed tempered softmax for small-margin mislabeled examples.The conclusion presents these as simultaneous remedies to two robustness issues.
  • 6 Conclusion and Future Work: Equal temperatures yield losses convex in the last layer, whereas a larger softmax temperature than logarithm temperature yields non-convex last-layer losses.The temperatures are treated as tunable loss-function parameters.
  • 6 Conclusion and Future Work: The paper proposes tunable loss functions as an additional variable for training neural networks alongside architecture, transfer function, optimizer, and label smoothing choices.The conclusion frames the method as a new training variant for benchmark networks and datasets.

A An Iterative Algorithm for Computing the Normalization

The appendix gives an iterative procedure for computing the normalization-related quantity λ_t(a) from an activation vector when t > 1. The procedure repeatedly updates shifted activations until convergence.

  • A An Iterative Algorithm for Computing the Normalization: The algorithm takes an activation vector a and temperature t > 1, then initializes a shift using the maximum activation.The shifted vector is formed by subtracting the maximum from a.
  • A An Iterative Algorithm for Computing the Normalization: It iteratively evaluates tempered exponentials, computes a normalization factor, and updates the shifted activations until convergence.The displayed loop defines the normalization update used before returning λ_t(a).
  • A An Iterative Algorithm for Computing the Normalization: After convergence, the procedure returns λ_t(a) computed from the converged shifted activations using the tempered logarithm.The return step follows the iterative normalization updates.

B Strong Convexity and Smoothness

The appendix develops strong-convexity and strong-smoothness tools for analyzing the relevant Bregman divergences. It uses duality between a function and its conjugate to relate divergence bounds to norm differences.

  • B Strong Convexity and Smoothness: Strong convexity is defined through a quadratic lower bound on a function over a convex set, using a specified norm and modulus σ.The appendix also states the corresponding second-derivative characterization for twice-differentiable functions.
  • B Strong Convexity and Smoothness: Strong convexity yields a lower bound on the associated Bregman divergence in terms of the squared norm distance between its arguments.This lemma supplies the basic inequality used in the appendix’s later estimates.
  • B Strong Convexity and Smoothness: For the tempered potential, the Hessian is diagonal with entries related to u^-t, and the appendix derives bounds by applying strong-convexity lemmas over a norm-bounded domain.The derivation identifies the boundary as the location of the minimum in the relevant optimization step.
  • B Strong Convexity and Smoothness: Strong smoothness is introduced as the dual smoothness property for differentiable functions with respect to a norm.The appendix uses strong smoothness of a dual function in subsequent divergence bounds.
  • B Strong Convexity and Smoothness: The appendix applies these results to F and its dual, using duality of Bregman divergences to bound divergences involving tempered logarithms and probability vectors.The resulting inequalities connect divergence values with norm differences under the relevant temperature-dependent norms.

C Other Tempered Convex Functions

The paper introduces an alternative strictly convex function family based on the curvature of the tempered logarithm and relates its Bregman divergence to the original construction through a temperature shift.

  • Alternative convex functions: The alternative family rF_t is constructed from the convexity or concavity of log_t and remains strictly convex.Its Hessian is described as diagonal with entries y^-(1+t).
  • Temperature-shift relation: The Bregman divergence induced by rF_t equals the original divergence with a shifted temperature, specifically ΔF_(t+1).The proposition states this relation explicitly for the alternative construction.
  • Relation to Tsallis forms: The alternative and original constructions are connected to generalized negative entropy functions, while their induced divergences relate to—but differ from—the Tsallis divergence over the simplex.The passages distinguish the Bregman divergence induced by negative Tsallis entropy from the Tsallis divergence itself.
  • Normalization and domain: For temperatures below one, the tempered exponential output is restricted by the range of log_t composed with exp_t, and normalization is invariant to adding a constant to all logits.The normalization function satisfies λ_t(a+b1)=λ_t(a)+b.

E Derivatives of Lagrangian and the Bi-tempered Matching Loss

The section derives derivatives of the normalization function and the bi-tempered matching loss, identifying an escort-distribution form and convexity properties of the normalization.

  • Normalization derivatives: The derivative of the normalization function produces the t-escort distribution associated with the tempered exponential probabilities.The escort distribution is defined from exp_t(a_i−λ_t(a)).
  • Second derivatives: The second derivative is obtained by differentiating the first-derivative relation again, and the normalization function is stated to be convex.The text notes that convexity is not immediate from the second derivative but can be shown.
  • Loss derivative: The derivative of the bi-tempered loss is expressed in terms of the target distribution and the tempered predicted probabilities.The displayed derivative uses y and y-hat=exp_t2(a−λ_t2(a)).

F Proof of Bayes-risk Consistency

The section formulates conditional risk for multiclass losses and proves Bayes-risk consistency for the bi-tempered loss using its explicit per-class expression and monotonicity.

  • Consistency criterion: Bayes-risk consistency requires the conditional-risk minimizer to satisfy the Bayes-optimal prediction condition.The definition introduces η_i as the conditional class probability and characterizes the required minimizer.
  • Proof setup: The proof substitutes the bi-tempered loss's explicit per-class form into the conditional-risk analysis.The loss combines a tempered logarithmic term with a class-independent second term repeated across classes.
  • Proof step: For 0≤t_1<1, monotonic decrease of −log_t1 is used in the proof of the consistency result.The monotonicity property supplies the ordering step in the argument.
Loading 1906.03361v3…