PaperlayerView exact revision on arXiv

Source-linked AI summary

An Alternative Probabilistic Interpretation of the Huber Loss

Gregory P. Meyer

arXiv:1911.02088v3stat.MLcs.CVcs.LG

TL;DR

Choosing the Huber loss transition parameter is difficult because the standard Huber-density interpretation offers limited intuition. The paper instead relates Huber minimization to an upper bound on KL divergence between Laplace label and prediction distributions, showing that their scales correspond to the transition point. The interpretation supports noise-based hyper-parameter selection and improves Faster R-CNN and RetinaNet without exhaustive search.

  • Problem

    The standard Huber-density interpretation provides limited intuition for selecting the transition point, so hyper-parameter search is often used.

  • Method

    The paper models label and prediction noise with Laplace distributions and relates Huber-loss minimization to an upper bound on their KL divergence.

  • Results

    The interpretation improved Faster R-CNN and RetinaNet performance without exhaustive hyper-parameter search.

  • Takeaways & Limitations

    Estimating label noise provides an intuitive basis for selecting well-suited Huber hyper-parameters.

  • Takeaways & Limitations

    The Faster R-CNN analysis relies on an assumption that anchor and ground-truth boxes have significant IoU.

Abstract

from arXiv · show

The Huber loss is a robust loss function used for a wide range of regression tasks. To utilize the Huber loss, a parameter that controls the transitions from a quadratic function to an absolute value function needs to be selected. We believe the standard probabilistic interpretation that relates the Huber loss to the Huber density fails to provide adequate intuition for identifying the transition point. As a result, a hyper-parameter search is often necessary to determine an appropriate value. In this work, we propose an alternative probabilistic interpretation of the Huber loss, which relates minimizing the loss to minimizing an upper-bound on the Kullback-Leibler divergence between Laplace distributions, where one distribution represents the noise in the ground-truth and the other represents the noise in the prediction. In addition, we show that the parameters of the Laplace distributions are directly related to the transition point of the Huber loss. We demonstrate, through a toy problem, that the optimal transition point of the Huber loss is closely related to the distribution of the noise in the ground-truth data. As a result, our interpretation provides an intuitive way to identify well-suited hyper-parameters by approximating the amount of noise in the data, which we demonstrate through a case study and experimentation on the Faster R-CNN and RetinaNet object detectors.

1. Introduction

The Huber loss is widely used for robust regression, but selecting its quadratic-to-linear transition point is difficult under the standard Huber-density interpretation. The paper proposes interpreting this choice through label and prediction noise modeled by Laplace distributions and KL divergence.

  • The Huber loss is robust, quadratic for small residuals, and approximately linear for large residuals.
  • Selecting the transition point between quadratic and linear behavior is a practical challenge when using the Huber loss.
  • The standard probabilistic interpretation links Huber-loss minimization to maximum likelihood under the Huber density, whose complexity often motivates hyper-parameter search.
  • The proposed interpretation assumes observed labels are noisy estimates of true values and relates Huber-loss minimization to an upper bound on KL divergence.
  • The Laplace-distribution scales represent label and prediction noise and are directly related to the Huber transition point.
  • The paper studies this interpretation through related-work review, a toy problem, object-detector analysis, and hyper-parameter experiments.

2. Related Work

Prior work noted similarities between the Huber loss and Laplace KL divergence, while other work developed related smooth potential functions. This paper extends the Laplace connection by identifying parameter relationships directly.

  • Noy and Crammer observed similarity between the Huber loss and Laplace KL divergence but did not explore the relationship beyond that observation.
  • This work further develops the Laplace-KL connection and identifies links between Huber-loss parameters and Laplace-distribution parameters.
  • Lange proposed Huber-like image-reconstruction potentials that are more than once differentiable and are derived through double integration rather than Laplace KL divergence.

3. Background

The Huber loss combines the robustness of L1 with the differentiability of L2, but its transition parameter must be selected. The standard likelihood interpretation uses a comparatively complicated hybrid Huber density.

  • L1 is robust to outliers but nondifferentiable at zero, whereas L2 is differentiable everywhere but highly sensitive to outliers.
  • The Huber loss combines L1 and L2 behavior, controlled by a positive parameter α, while remaining differentiable and robust to outliers.
  • Selecting α is a disadvantage of the Huber loss, motivating an intuitive probabilistic interpretation to ease hyper-parameter selection.
  • Maximum likelihood estimation chooses model parameters by maximizing the likelihood of observed targets given inputs, equivalently minimizing negative log likelihood.
  • Gaussian noise yields L2 minimization, while Laplace noise yields L1 minimization under the corresponding likelihood models.
  • The Huber density is a Gaussian-Laplace hybrid, and its complexity makes it difficult to use for selecting α.

4. Proposed Method

The paper models label and prediction uncertainty with Laplace distributions and derives a loss from their KL divergence. Its parameters encode the two noise scales, yielding bounds and an interpretation of the Huber transition parameter.

  • Because the true value is hidden, the method estimates model parameters by minimizing KL divergence between label and prediction uncertainty distributions.
  • The label and prediction uncertainties are modeled as Laplace distributions with separate positive scales b1 and b2.
  • The proposed loss is derived from Laplace KL divergence after removing constants and shifting its minimum to zero.
  • In the proposed loss, α corresponds to label-noise scale b1, β corresponds to prediction-noise scale b2, and x is the difference between distribution means.
  • Like the Huber loss, the proposed loss is quadratic for small residuals and linear for large residuals.
  • Huber minimization with parameter α corresponds to minimizing an upper bound on Laplace KL divergence when b1 = α and b2 = 1/α.
  • Scaling the Huber input inversely scales both noise distributions, while scaling the loss inversely scales the prediction distribution.

5. Toy Problem: Polynomial Fitting

The toy problem tests whether the Huber loss parameter α tracks controlled label noise. Across heavy-tailed noise distributions, the optimal α is closely related to noise scale, motivating noise-based hyper-parameter selection.

  • Results: The optimal α parameter is closely related to the scale of the label noise in the toy problem.The paper tests this relationship by controlling the amount of label noise across experiments.
  • Setup: The toy problem fits an overparameterized one-dimensional polynomial to samples with controlled label noise and evaluates predictions using test-set RMSE.The predicted polynomial uses K > D to account for possible model mismatch as a source of prediction noise.
  • Setup: Experiments use Laplace, Logistic, and Cauchy noise distributions because the Huber loss is designed to be robust to outliers.Each experiment samples N = 10000 points with δ = 2 and evaluates α through grid search.
  • Results: The results show a near linear relationship between label-noise scale and the optimal α for each tested distribution.This relationship suggests that knowing or estimating label noise can identify a suitable α parameter.
  • Implication: The authors next apply noise-based hyper-parameter selection to a real-world problem where label noise is unknown.The stated aim is to determine whether approximating label noise is an intuitive and effective selection method.

6. Case Study: Faster R-CNN

The Faster R-CNN case study interprets its bounding-box Huber losses through label and prediction noise scales. This analysis exposes mismatched uncertainty assumptions and motivates better-suited hyper-parameters.

  • Model and losses: Faster R-CNN uses a region proposal network and detection network, both applying the Huber loss to bounding-box regression.The detection network refines and classifies regions proposed by the proposal network.
  • Center prediction: For center regression, the implementation implies prediction noise of one-tenth the anchor width and label uncertainty equal to the full anchor width.The authors state that assuming labels contain uncertainty equal to the full anchor width is inappropriate.
  • Implementation analysis: The implementation differs significantly from the original Faster R-CNN paper, making its interpretation important because it underlies several other object detectors.The analysis is therefore applied to the implemented loss and target transformations.
  • Transformed targets: The proposal network uses α = 1/9 and σx = 1, while the detection network uses α = 1 and σx = 1/10, implying label-noise scales of a ninth and tenth of the anchor width.Under this interpretation, prediction uncertainty is much larger for the proposal network than for the detection network.
  • Width prediction: The width analysis uses a first-order logarithm approximation justified when the anchor and ground-truth box have significant intersection-over-union.The approximation assumes w*/wa ≈ 1, which follows from the matching requirement described by the authors.
  • Width prediction: For width regression, the proposal network uses σw = 1 and the detection network σw = 1/5, implying higher assumed label noise for the detection network.The authors note that this assumption could be less than optimal.
  • Implication: The authors argue that the peculiar hyper-parameters may have resulted from parameter sweeps and could be improved using their interpretation.Their stated goal is to eliminate inappropriate values rather than replace the Huber loss.

7. Experiments

The experiments evaluate whether the proposed interpretation can guide Huber-loss hyper-parameter selection in Faster R-CNN and RetinaNet. The interpretation improved detector performance without exhaustively searching hyper-parameters, with benefits observed across architectures and settings.

  • Experimental design: The experiments modify selected Huber-loss hyper-parameters in Faster R-CNN and RetinaNet to test whether the proposed interpretation yields task-suited settings.The study does not aim for state-of-the-art performance or to replace the Huber loss.
  • Faster R-CNN: Faster R-CNN experiments use a VGG-16 backbone on MS-COCO 2014 and evaluate mean average precision at multiple IoU thresholds.The dataset contains 80 object classes, over 80k training images, and 40k validation images.
  • Faster R-CNN: The published Faster R-CNN hyper-parameters performed worst, while reducing estimated label and prediction noise further improved performance at larger IoU thresholds.Experiments A and B traded off performance at 0.5 and 0.75 IoU, while Experiment C balanced both.
  • RetinaNet: RetinaNet repeats Experiments A and B using the Faster R-CNN proposal-network loss, with a ResNet-101 backbone on MS-COCO 2017.The RetinaNet setup uses a single-stage detector and the same evaluation metrics as the Faster R-CNN experiments.
  • RetinaNet: Experiment A achieved higher RetinaNet performance across the board, whereas Experiment B significantly degraded performance at 0.5 IoU.The authors report a similar trend in the Faster R-CNN experiments and conclude that the interpretation identifies suitable hyper-parameters across architectures, backbones, and datasets.

8. Conclusion

The paper concludes that its alternative interpretation connects the Huber loss to KL divergence between Laplace distributions and clarifies the meaning of its parameters. It reports improved Faster R-CNN and RetinaNet performance without exhaustive hyper-parameter search.

  • Conclusion: The proposed interpretation connects the Huber loss to the KL divergence of Laplace distributions and provides an intuitive understanding of its parameters.The paper identifies links between the Huber-loss parameters and the Laplace-distribution parameters.
  • Conclusion: The interpretation aided hyper-parameter selection and improved Faster R-CNN and RetinaNet performance without exhaustively searching over hyper-parameters.This is the paper's stated overall experimental conclusion.
  • Implications: The authors state that methods using the common Fast or Faster R-CNN Huber-loss formulation may benefit from the proposed interpretation when selecting task-specific hyper-parameters.The paper frames this as a potential improvement for existing and future methods.

B. Proof of Differentiability

This appendix proves differentiability properties of the proposed loss and establishes tight bounds relating it to the Huber loss. The proofs analyze limits and parameter constraints across the relevant regions.

  • Differentiability: The appendix derives the first and second derivatives of the proposed loss and proves their existence at x = 0.It establishes differentiability of both Dα,β(x) and its derivative at the origin under α > 0 and β > 0.
  • Differentiability: The differentiability proof evaluates right- and left-hand limits and shows that both sides agree at x = 0.The argument uses substitutions involving exponential terms and a standard limit identity.
  • Bounds: The appendix proves that the Huber loss Hα(x) is tightly bounded between Dα,1/α(x) and Dα/2,1/α(x) for all x.The stated bounds are symmetric and hold over the full real line.
  • Proof strategy: The inequality proofs use derivative behavior, Rolle’s theorem, the mean value theorem, and monotonicity arguments over the relevant intervals.These arguments establish nonnegativity of the auxiliary functions used in the bounds.

D. Experimental Validation of Target Approximation

The appendix validates an approximation for bounding-box width and height targets by comparing training results under the original and approximated targets. Replacing the targets caused only a very slight performance degradation.

  • Validation setup: The validation compares Faster R-CNN training with original targets against targets approximated by the percentage change between anchor and ground-truth width and height.No other implementation changes were made.
  • Result: Only a very slight degradation in performance was observed after replacing the original targets with their approximation.The authors interpret this result as validating the approximation in their loss-function interpretation.

E. Evaluation of Proposed Loss Function

The proposed loss function was evaluated as a direct replacement for the Huber loss in Faster R-CNN, without other training or evaluation modifications. The two loss functions achieved nearly identical performance.

  • The proposed loss function replaced Hα in Faster R-CNN without other modifications.The training and evaluation procedure remained unchanged.
  • The evaluation compares the Huber loss with the proposed loss function's performance.The results are reported in Table 6.
  • Nearly identical performance was observed for the two loss functions.The similarity was expected because the functions are similar.
Loading 1911.02088v3…