A General and Adaptive Robust Loss Function

Jonathan T. Barron

Source-linked AI summary

A General and Adaptive Robust Loss Function

Jonathan T. Barron

arXiv:1701.03077v10Jan 11, 2017cs.CVcs.LGstat.ML

TL;DR

The paper addresses the difficulty of choosing among fixed robust losses and manually tuning their robustness. It introduces a continuous two-parameter loss and a corresponding probability distribution whose robustness can adapt during optimization, improving performance across vision tasks and learning-based models.

Problem
Existing robust penalties have particular fixed forms, while adding robustness parameters complicates experimentation through manual tuning or cross-validation.
Method
The paper unifies several robust losses with a continuous shape parameter, interprets the loss as a probability distribution’s negative log-likelihood, and optimizes robustness adaptively.
Results
Across registration, clustering, image synthesis, and monocular depth estimation, the generalized loss or adaptive distribution improves reported performance relative to fixed-loss baselines or existing algorithms.
Takeaways & Limitations
The framework supports exploring and comparing robust penalties while generalizing existing vision algorithms and removing manual α tuning in adaptive neural-network training.
Takeaways & Limitations
The probability distribution is defined only for α ≥ 0 because its partition function diverges for α < 0.

Abstract

from arXiv · show

We present a generalization of the Cauchy/Lorentzian, Geman-McClure, Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2 loss functions. By introducing robustness as a continuous parameter, our loss function allows algorithms built around robust loss minimization to be generalized, which improves performance on basic vision tasks such as registration and clustering. Interpreting our loss as the negative log of a univariate density yields a general probability distribution that includes normal and Cauchy distributions as special cases. This probabilistic interpretation enables the training of neural networks in which the robustness of the loss automatically adapts itself during training, which improves performance on learning-based tasks such as generative image synthesis and unsupervised monocular depth estimation, without requiring any manual parameter tuning.

1. Loss Function

The paper introduces a smooth two-parameter loss whose shape parameter controls robustness and whose scale parameter controls the quadratic behavior near zero. By varying the shape parameter, the loss continuously generalizes several established robust penalties and supports optimization strategies such as graduated non-convexity.

The loss uses α to control robustness and c > 0 to control the quadratic bowl’s size near x = 0.
α = 2 approaches L2 loss, while α = 1 yields a smoothed L1 loss known as Charbonnier, pseudo-Huber, or L1-L2 loss.The smoothed L1 form behaves like L2 near the origin and like L1 elsewhere.
At nonpositive shape values, the loss includes Cauchy/Lorentzian and Geman-McClure losses, and α → −∞ gives Welsch/Leclerc loss.
The resulting loss is a superset of Welsch/Leclerc, Geman-McClure, Cauchy/Lorentzian, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and L2 losses.
The derivative is approximately linear for |x| < c across all α, while larger residuals receive increasingly robustness-dependent influence.
Because the loss is smooth in x, α, and c, it supports gradient-based optimization; reducing α can increase robustness during graduated non-convexity.The paper states that initializing with a convex loss and gradually reducing α can often help avoid local minima.

2. Probability Density Function

The loss defines a probability distribution whose negative log-likelihood is a shifted version of the loss and whose normalizing constant restricts the distribution to α ≥ 0. This probabilistic form enables adaptive robustness during training and supports efficient rejection sampling.

The distribution is p(x | µ, α, c) = 1/(cZ(α)) exp(−ρ(x − µ, α, c)), with NLL equal to the loss plus a normalization shift.
The density is defined only for α ≥ 0 because the partition function Z(α) diverges when α < 0.
At α = 2 the distribution becomes Gaussian, while at α = 0 it becomes Cauchy.
Using the NLL lets optimization treat α as a free parameter, balancing lower outlier costs against higher inlier costs as robustness changes.This tradeoff allows training to adapt robustness without manually tuning α as a fixed hyperparameter.
Rejection sampling uses a Cauchy proposal and has acceptance rates from approximately 45% at α = ∞ to 100% at α = 0.

3. Experiments

Across four vision tasks, the paper replaces fixed robust losses or distributions with a general, tunable formulation whose robustness can be adapted or optimized. These changes improve image synthesis, monocular depth estimation, registration, and clustering under the reported settings.

Experiment scope: The experiments evaluate the general loss and distribution on variational autoencoders, monocular depth estimation, registration, and clustering.Learning-based tasks use adaptive robustness, while registration and clustering introduce tunable or annealed shape parameters.
Variational Autoencoders: Per-dimension shape parameters let the VAE model image coefficients with a general distribution instead of fixed normal, Cauchy, or Student’s t distributions.The models also test DCT and CDF 9/7 wavelet representations in YUV space, motivated by heavy-tailed natural-image statistics.
Variational Autoencoders: The general distribution produces higher ELBOs than fixed distributions across representations and sharper, more detailed samples than normal-distribution VAEs.It performs similarly to Student’s t overall, while Wavelets + YUV gives the best validation ELBO among image representations.
Unsupervised Monocular Depth Estimation: Replacing the baseline depth-estimation loss with adaptive per-wavelet losses substantially improves KITTI depth estimates without changing the baseline network.The paper reports a substantial qualitative improvement when only the loss function is replaced.
Fast Global Registration: Generalized registration with shape annealing moderately improves FGR under high noise while behaving equivalently under low noise.The generalized formulation replaces Geman-McClure loss; FGR is recovered at α = −2.
Robust Continuous Clustering: Tuning α in generalized clustering improves adjusted mutual information by as much as 32% on some datasets, although performance is insensitive to α on others.The method replaces RCC’s Geman-McClure loss while keeping the provided implementation’s default hyperparameters fixed.

4. Conclusion

The paper unifies many robust losses in a continuous two-parameter family and uses it to generalize robust algorithms and adapt robustness during neural-network training. Its outlier-process formulation supports robust optimization, while the corresponding distribution includes normal and Cauchy cases.

The two-parameter loss generalizes Cauchy, Geman-McClure, Welsch, generalized Charbonnier, Charbonnier, and L2 losses.
The loss enables algorithms built around fixed robust penalties to explore and compare robustness continuously.
The corresponding probability distribution includes normal and Cauchy distributions as special cases.
The loss can be formulated as an outlier process for registration and clustering algorithms.
The Ψ-function is undefined for α ≥2 because the loss is no longer robust and cannot be described as rejecting outliers.

B. Practical Implementation

The implementation modifies the loss near removable singularities to improve numerical and framework-level behavior. This approximation has a specific accuracy limitation near α = 0.

The implementation guards against singularities at α = 0 and α = 2 using a small offset ϵ.The suggested offset can be as small as 10^-5.
The IRLS weight and Ψ-function vary with the shape parameter α and illustrate how robustness changes across settings.
The loss, gradient, and Ψ-function are adjusted with α + ϵ for α ≥0 and α −ϵ for α < 0.
Near α = 0, even small ϵ values can significantly distort the true partition function, so the approximation should be avoided when accurate Z(α) values are required.

C. Partition Function Approximation

Because the analytical partition function is difficult to evaluate and differentiate, the paper approximates log(Z(α)) with cubic Hermite splines after transforming α to concentrate knots where needed.

The analytical Z(α) is difficult to evaluate efficiently for arbitrary α and especially difficult to differentiate with respect to α.
The method transforms α with a continuously differentiable nonlinearity before spline interpolation.
A cubic Hermite spline uses evenly spaced knots in transformed space, with knot values matching the true log partition function.

D. Motivation and Derivation

The loss is derived by reparameterizing and rescaling generalized Charbonnier loss to obtain a scale-invariant family spanning familiar robust penalties. The construction addresses undesirable behavior while retaining smoothness, with a removable singularity at α = 2.

The proposed loss is derived from generalized Charbonnier loss and is reparameterized using α/2 to align with standard loss cases.Under this parameterization, α = 2 resembles L2 and α = 1 resembles L1.
Dropping the c^α scale factor makes the loss invariant to the scale parameter c.
After removing the scale factor, negative α values produce meaningful robust losses including Geman-McClure.
A shifted and scaled construction removes nonzero loss at x = 0 and the dependence of local quadratic curvature on c and α.
The resulting family generalizes L2, Cauchy, and Geman-McClure losses but originally flattened toward zero for α ≪0, limiting annealing strategies.
Choosing |α −2| preserves smoothness but introduces a removable singularity and reduces numerical stability near α = 2.

E. Additional Properties

The loss has bounded, locally quadratic behavior and its global shape depends on α. It is redescending and non-convex for α < 1, while remaining pseudoconvex for all α.

Local behavior: For all α, the loss is approximately quadratic when |x| is small relative to c.This follows from the loss’s local quadratic-bowl approximation.
Global behavior: When α is negative, the loss approaches a constant as |x| approaches infinity, providing a bound on the loss.
Monotonicity: The loss’s Ψ-function increases monotonically with α when α < 2 for every z in [0, 1].
Curvature and redescending: The loss is redescending if and only if α < 1, strictly convex if α ≥ 1, non-convex if α < 1, and pseudoconvex for all α.The second-derivative root identifies where redescending begins; it is undefined when α ≥ 1.

F. Wavelet Implementation

The image experiments use the CDF 9/7 wavelet decomposition with symmetric analysis filters and standard synthesis filters. The construction preserves image magnitude across scales and uses reflecting boundary conditions.

Wavelet decomposition: The wavelet experiments reparameterize images with the Cohen-Daubechies-Feauveau 9/7 wavelet decomposition.
Filter construction: The analysis filter lists its origin coefficient first, with the remaining coefficients defined symmetrically.
Filter construction: Synthesis filters are formed by reversing the signs of alternating wavelet coefficients in the analysis filters.
Normalization and boundaries: The lowpass filter sums to 2, doubling image intensities at each decomposition scale while preserving image magnitude in the wavelet representation.Boundary conditions are reflecting, or half-sample symmetric.

G. Variational Autoencoders

The VAE experiments adapt the general distribution to real-valued CelebA images across several output representations. Independently adapting coefficient-wise shape parameters outperforms fixed global choices and reveals representation-specific modeling behavior.

Data and setup: The VAE implementation changes the input and output resolution from (28, 28, 1) to (64, 64, 3).
Optimization: Training is extended from 5000 to 50000 steps, and cosine learning-rate decay begins during the final 10000 iterations.
Output representations: The models apply RGB-to-YUV conversion followed by per-channel DCT or wavelet transforms, with inverse transforms used for visualization.
Adaptive output model: Allowing each coefficient’s shape parameter to adapt automatically achieves higher validation ELBO than any single fixed α.No single global α setting matches the independently adapted model, including fixed settings between 0 and 2.
Adaptive output model: Each output coefficient receives a scale variable and, for the general distribution, a shape variable α.The scale is σ for normal distributions and c for the general distribution.
Data and setup: CelebA images are center-cropped from 178 × 218 to 160 × 160, downsampled to 64 × 64, and scaled to [0, 1].
Learned structure: The learned parameters assign normal-like behavior near consistent facial structures and Cauchy-like behavior to backgrounds and flat skin regions.The Pixels + RGB model estimates similar distributions across color channels, while Wavelets + YUV differs between luma and chroma.
Visualization: VAE figures distinguish decoded means from output-distribution samples, which tend to look noisy and irregular across representations.The paper’s usual samples draw from the latent space and decode the result without sampling the output space.

H. Unsupervised Monocular Depth Estimation

The monocular depth experiments replace the baseline reconstruction loss with the adaptive general loss and simplify the original training setup. The adaptive model shows substantial and consistent improvement across KITTI test scenes.

Experimental setup: The experiments replace the SfMLearner reconstruction loss, reduce training iterations from 200000 to 100000, and disable smoothness and multi-scale side-prediction terms.The loss change is reported to make training converge faster, while the disabled terms added little benefit with the new loss.
Learned robustness: The adaptive model learns coefficient-wise shape parameters that distinguish outlier-prone luma variation from more stable chroma variation.Low α values in luma indicate more expected outliers, whereas high α values in chroma indicate fewer.
Learned robustness: Horizontal luma variation tends to receive larger α values than vertical luma variation, possibly because horizontal motion provides more depth information in this dataset.
Results: The adaptive model’s improvement over the baseline is substantial and consistent across a variety of KITTI test scenes.Figures 16 and 17 compare adaptive outputs with the baseline and ground-truth depth.

I. Fast Global Registration

The paper evaluates vision applications using registration, image synthesis, and monocular depth estimation experiments. The reported setup includes repeated randomized registration evaluation and comparisons across image representations and loss distributions.

Fast Global Registration: Registration evaluation runs FGR 20 times with random initialization and reports the median error.The procedure reproduces the baseline using code corresponding to prior work.
Image synthesis: The variational autoencoder experiments compare image representations including pixels, RGB DCT, and YUV wavelets.The listed representations are arranged as alternative modeling choices.
Image synthesis: Reconstructions using the general distributions tend to be sharper and more detailed than reconstructions from the compared distribution choices.The comparison includes normal, Cauchy, Student’s t, and general distributions.
Monocular depth estimation: Replacing the baseline monocular-depth loss with an adaptive loss over wavelet coefficients significantly improves depth estimates on KITTI.The comparison changes only the loss function between the baseline network and the authors’ variant.
Monocular depth estimation: Additional monocular depth estimation results are presented in the same format as the preceding KITTI comparison.The passage identifies these as supplementary results rather than a separate evaluation format.