PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels

Erik Englesson, Hossein Azizpour

arXiv:2105.04522v4cs.LGcs.CVstat.ML

TL;DR

Noisy labels create a need for losses that avoid both noise overfitting and the underfitting associated with some robust objectives. The paper proposes JS and GJS, combining CE–MAE interpolation with consistency across perturbed inputs, and reports state-of-the-art results across synthetic and real-world noise settings.

  • Problem

    Existing noise-robust losses can underfit, while CE can overfit noisy labels; the paper addresses this trade-off and reduced consistency around noisy examples.

  • Method

    The paper proposes JS, which interpolates between CE and MAE, and GJS, which extends JS to multiple perturbed-input predictions to encourage consistency.

  • Results

    JS and GJS achieve state-of-the-art results across datasets, noise types, and rates, with large margins over CE on WebVision, especially in top-1 accuracy.

  • Takeaways & Limitations

    Consistency regularization within GJS is empirically associated with improved robustness to noisy labels across synthetic and naturally noisy datasets.

  • Takeaways & Limitations

    Theoretical robustness is established only in the π1 → 1 limit, leaving the robustness of GJS’s consistency term without a proof.

Abstract

from arXiv · show

Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g., WebVision) noise with varying noise rates.

1 Introduction

The paper targets noisy-label learning with two observations: robust losses can underfit, while standard networks lose consistency around noisy examples. It proposes JS and GJS losses to combine improved fitting, noise robustness, and consistency regularization.

  • Noisy labels occur even in systematically annotated datasets, making noise-robust learning algorithms important for real-world tasks.
  • Provably robust losses can underfit, whereas standard networks show reduced consistency around noisy data points.
  • JS interpolates between noise-robust MAE and faster-converging CE, while GJS encourages consistency across predictions for perturbed inputs.
  • The paper proposes JS and GJS loss functions, proves their relationships to CE, MAE, and consistency regularization, and evaluates them across datasets, noise types, and rates.
  • Consistency is measured on clean and noisy CIFAR-100 training examples alongside validation accuracy during CE training.

2 Generalized Jensen-Shannon Divergence

The section develops JS and GJS losses for noisy-label classification. JS connects CE and MAE through a weighting parameter, while GJS extends JS to perturbed predictions and adds consistency regularization.

  • Background & Motivation: Noisy-label learning seeks classifiers that remain effective when labels are changed with probability η under instance-dependent, asymmetric, or symmetric noise.
  • Background & Motivation: Consistency regularization encourages identical predictions for different perturbations of the same image.
  • Definitions: GJS uses the target distribution and predictions from randomly perturbed inputs, with equal weights assigned to the predictive distributions in the experiments.
  • Background & Motivation: CE converges quickly but can overfit noise, whereas theoretically noise-robust MAE and related losses may converge slowly and underfit.
  • JS’s Connection to Robust Losses: JS approaches CE as π1 approaches zero and MAE as π1 approaches one, thereby interpolating between the two losses.
  • Generalized Jensen-Shannon Divergence: GJS generalizes JS to multiple predictive distributions and decomposes into a label-to-mean-prediction term plus a consistency term across predictions.
  • Noise Robustness: JS and GJS share the same lower robustness bound, while GJS has an increasing upper bound as the number of distributions grows.

3 Related Works

The paper extends theoretically motivated noise-robust loss research by connecting divergence-based losses with consistency regularization. It distinguishes its training-time label-noise focus and multi-distribution generalization from related approaches.

  • Prior work constructs theoretically motivated robust losses, while peer prediction offers another family; this paper instead proposes DJS-based losses with ties to consistency regularization.The proposed direction aims to retain desirable properties of prior robust losses while introducing a consistency connection.
  • Loss correction reweights losses using a confusion matrix under class-conditional noise and is theoretically orthogonal to noise-robust losses.
  • GJS encourages consistency through a simple loss, whereas related methods use more complex pipelines; this work studies the approach for training-time label-noise robustness.Prior Jensen-Shannon consistency losses targeted test-time image corruptions and adversarial examples.
  • Compared with concurrent JS and f-divergence studies, this work generalizes to more than two distributions and studies π1 rather than treating it as constant.The authors associate these differences with improved performance and novel theoretical results.

4 Experiments

Experiments show that GJS performs competitively across synthetic and real-world label noise, while ablations clarify how its consistency mechanism, mixing parameter, distribution count, and perturbations affect learning.

  • Synthetic Noise Benchmarks: CIFAR: GJS achieves state-of-the-art results across noise rates, noise types, and datasets, with greater efficacy on challenging CIFAR-100.The benchmark reimplements competing losses in a shared setup with matched hyperparameter optimization and reports five-run means and standard deviations.
  • Synthetic Noise Benchmarks: CIFAR: 4.94 percentage points separate GJS from GCE under 60% uniform noise on CIFAR-100, whereas GJS trails GCE under 80% noise.The authors attribute the 80% result to sensitivity to hyperparameters at an unusually high noise rate.
  • Real-World Noise Benchmark: WebVision: On WebVision, JS and GJS substantially outperform CE, especially in top-1 accuracy, while top-5 accuracy shows smaller but significant gains.Top-5 accuracy can obscure susceptibility to noise-fitting because incorrect top predictions remain admissible.
  • Real-World Noise Benchmark: WebVision: GJS matches or exceeds state-of-the-art top-1 accuracy on WebVision and significantly outperforms ELR+ on ILSVRC12, reaching 74.33 versus 70.29.Averaging two independently trained GJS networks further improves results without changing the training code, and the paper reports state-of-the-art performance without pre-training.
  • Towards a Better Understanding of the Jensen-Shannon-based Loss Functions: GJS avoids JS’s low-π1 noise overfitting and high-π1 slow learning while converging to higher validation accuracy.JS behaves like CE at low π1 and MAE at high π1; GJS improves both learning dynamics and robustness across these settings.
  • Towards a Better Understanding of the Jensen-Shannon-based Loss Functions: Using three distributions improves accuracy over two at every noise rate, whereas more than three helps only at lower noise rates; experiments therefore use M = 3.At higher noise rates, increasing M beyond three degrades performance, which the authors connect to stronger consistency pressure with few clean examples.
  • Towards a Better Understanding of the Jensen-Shannon-based Loss Functions: The consistency term, rather than mean prediction alone, accounts for GJS’s improvement over JS.GJS improves performance across clean/noisy-example ablations, with noisy-only GJS reaching 74.1 versus 72.9 for clean-only and 74.7 when applied to both.
  • Towards a Better Understanding of the Jensen-Shannon-based Loss Functions: JS and GJS produce more consistent networks and higher accuracy than CE under 40% symmetric CIFAR-100 noise, while stronger augmentation improves all methods and GJS remains best.Weak augmentation still yields state-of-the-art results on WebVision, ANIMAL-10N, and Food-101N.

5 Limitations & Future Directions

The paper identifies unresolved theoretical and practical limitations of GJS, including missing robustness guarantees for its consistency term, degraded performance with many distributions under high noise, and added training computation.

  • Theoretical limitation: Theoretical robustness is proved only for GJS at π1 → 1, where the consistency term gradually vanishes.The robustness of the consistency term itself remains unproved.
  • Future analysis: GJS with M > 3 may degrade performance under high noise rates, although the mechanism requires systematic empirical and theoretical analysis.The authors offer initial conjectures but do not resolve the phenomenon.
  • Practical limitation: GJS adds forward-pass computations during training, although experiments use only one extra prediction with M = 3.The limitation applies to training time only.

6 Final Remarks

The paper presents JS and GJS as simple loss functions for noisy-label learning, combining theoretical connections with empirical evidence and achieving state-of-the-art results across synthetic and real-world noise settings.

  • Contributions: JS and GJS address robust-loss underfitting and reduced consistency around noisy data by interpolating between MAE and CE and encouraging consistency.The paper also connects these losses to CE, MAE, GCE, and consistency regularization.
  • Results: The proposed losses achieve state-of-the-art performance on synthetic and real-world noise, including comparisons with pipelines such as DivideMix and ELR+.The paper reports theoretical and empirical evidence supporting their usefulness.
  • Broader implications: The paper argues that robust methods could let less affluent organizations use labeled datasets despite imperfect or automatic labeling.This ethical consideration is presented alongside concerns about the resources required for manual annotation.

A Training Details

The experiments use shared training settings, controlled hyperparameter searches, and repeated runs across CIFAR and WebVision, with JS and GJS implemented through KL-divergence-based definitions.

  • Shared setup: All compared losses and baselines use the same training settings to support consistent evaluation.The CIFAR experiments use a PreActResNet-34, SGD with Nesterov momentum, batch size 128, and 400 training epochs.
  • Implementation: JS and GJS are implemented from KL-divergence definitions with gradients propagated through the target argument.The authors therefore use a custom implementation rather than PyTorch’s KL-divergence function.
  • Hyperparameter search: Learning rate and weight decay are searched on 40% CIFAR noise, followed by method-specific parameter searches for each noise rate and type.The search includes separate settings for symmetric and asymmetric noise.
  • Evaluation: The best parameters are used to retrain on the full training set with five different seeds before final evaluation.The resulting CIFAR parameters are reported in Table 7.
  • Additional settings: An appendix reports a less thorough search using a noisy validation set, while WebVision uses ResNet-50 with dataset-specific batch sizes and augmentation.For WebVision, GJS uses batch size 32, whereas CE and JS use batch size 64.
  • Parameterization: A parameter mapping using β = 1 − α produced worse performance than the reported search procedure.This comparison concerns the parameterization tested for the relevant methods.
  • Reported parameters: Table 7 reports learning rate, weight decay, and method-specific hyperparameters selected through the CIFAR search.For JS and GJS, the method-specific parameter is π1.

B Additional Experiments and Insights

Additional experiments evaluate GJS under instance-dependent CIFAR noise and compare it with CE and GCE, finding that its advantage is clearer on the more challenging CIFAR-100 dataset.

  • Instance-dependent noise: Instance-dependent noise varies the mislabeling risk by example and is intended to resemble real-world label noise more closely than simpler synthetic noise.The experiments consider three 35% instance-dependent noise types.
  • Results: On CIFAR-10, GCE and GJS perform similarly, whereas on CIFAR-100, GJS significantly outperforms GCE.The comparison uses three different types of 35% instance-dependent noise.

B.2 Real-World Noise: ANIMAL-10N & Food-101N

GJS outperforms other methods on both naturally noisy datasets, ANIMAL-10N and Food-101N, under matched training setups. These datasets contain estimated noise rates of 8% and 20%, respectively.

  • Food-101N: Food-101N contains 301k images across 101 food recipes, with an estimated 20% noise rate.The images were collected using Google, Bing, Yelp, and TripAdvisor.
  • Experimental setup: The experiments follow PLC’s training setup, including network architecture, augmentation strategy, optimizer, batch size, epochs, and learning-rate scheduling.The reported initial learning rate, weight decay, and π1 differ between the datasets.
  • ANIMAL-10N: ANIMAL-10N contains 55k images across 10 classes, grouped into five pairs of visually similar classes, with an estimated 8% noise rate.The images were collected using Google and Bing.
  • GJS outperforms all other methods on both ANIMAL-10N and Food-101N.Results report mean test accuracy and standard deviation from three runs, with baselines taken from Zhang et al..

B.3 Towards a better understanding of JS

The JS ablation connects noise robustness to boundedness and symmetry. On CIFAR-100 with 40% symmetric noise, unbounded KL and Jeffrey’s overfit, whereas JS does not.

  • The ablation decomposes JS to examine differences from CE and provides empirical evidence connecting bounded losses with robustness to noise.The analysis is framed around the construction of Jensen-Shannon divergence.
  • On CIFAR-100 with 40% symmetric noise, KL and Jeffrey’s are the only losses showing signs of overfitting.Both losses are unbounded.
  • JS does not overfit to noise, unlike its unbounded counterpart Jeffrey’s.The comparison is made during validation-accuracy training curves.
  • K, the bounded KL divergence, slows learning, while K′ considerably improves learning dynamics.The ablation examines KL-based divergences and their relation to boundedness and symmetry.
  • K′ may be robust to noise because K′ approaches MAE as π1 → 1, whereas K goes to zero.K′ is also described as a component of JS and reminiscent of label smoothing.

B.4 Comparison between JS and GCE

JS is compared with GCE through gradients, label-distribution requirements, consistency, and robustness experiments. JS generally penalizes low probability on the given class less than GCE, making it behave more like MAE.

  • Gradients: JS generally penalizes lower probability in the given class less than GCE, so JS behaves more like MAE.The comparison matches JS and GCE gradient scales at py = 1/2 for corresponding hyperparameters.
  • Label distributions: GCE requires onehot labels, making it harder to use with soft-label methods such as Mixup, co-training, and knowledge distillation.This requirement is presented as a limitation of GCE relative to methods accepting soft labels.
  • Consistency: GJS training produces significantly higher consistency than other methods when learning with noisy labels.This agrees with Proposition 2, which states that LGJS encourages consistency.
  • GJS construction: With onehot labels, GJS combines a term matching the mean prediction to the label with a term encouraging consistency among predictions.For M = 2, the consistency term is zero.
  • Noise robustness: The authors generalize noise-robustness theorems to bounded losses and multiple predictive distributions for GJS.The generalized results apply under specified conditions on the loss function and noise.

C.5 Robustness of Jensen-Shannon losses

The robustness analysis shows that JS and GJS become robust in the π1 → 1 limit because their lower and upper bounds coincide. The section also derives gradients of JS with respect to logits.

  • As π1 → 1, the lower and upper bounds of both JS and GJS become equal, establishing robustness in that limit.The result is stated as BL = BU for the limiting regime.
  • The GJS robustness proof decomposes the loss into a JS term and a consistency term, then shows the consistency contribution becomes constant as π1 approaches 1.The generalized divergence term is bounded and its scaling factor tends to zero.
  • Gradient derivation: The section derives the partial derivative of Jensen-Shannon divergence with respect to each logit of the predictive distribution.The derivation uses the mixture m = π1e(y) + π2p and the softmax-output derivative.
  • Gradient derivation: The softmax derivative uses an indicator that equals one when the logit and output indices match and zero otherwise.This derivative is used to obtain the logit gradients of JS.

D Extended Related Works

Related work addresses noisy labels through robust loss design, consistency regularization, noise correction, and training pipelines. This work positions JS/GJS as a principled loss-based alternative that connects learnability, robustness, and consistency.

  • Noise-robust losses: Prior robust-loss work includes MAE, GCE, symmetric cross entropy, and normalization mechanisms designed to balance noise robustness with data fitting.GCE generalizes MAE and CE, while SCE spans reverse CE and standard CE.
  • Alternative noise-handling methods: Other approaches identify or correct noisy labels, reweight losses using class-conditional noise models, or use label smoothing to alleviate fitting the noise.These directions differ from the proposed loss-based approach in how they address label noise.
  • Consistency regularization: Consistency regularization encourages minimal prediction shifts under input perturbations, but prior noisy-data methods often integrate it through complicated training pipelines.The paper presents multi-distribution JS as a way to incorporate this regularization directly into a loss.
  • Connection to test-time robustness: The paper contrasts its training-time noisy-label setting with AugMix, which uses GJS consistency loss for test-time uncertainty estimation and robustness to image corruptions.The authors describe the two settings as orthogonal and report that GJS improves robustness in both contexts.
  • Theoretical connections: The paper analyzes the JS mixing weight π1, showing CE and MAE as asymptotes as π1 approaches zero and one, respectively.This establishes a trade-off between learnability and robustness and differs from concurrent work treating π1 as constant.
  • Generalized Jensen-Shannon divergence: Unlike related work limited to two distributions, the paper studies a generalization to more than two distributions and uses it to incorporate consistency regularization.The generalized construction is presented as a principled extension of JS.
Loading 2105.04522v4…