PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Optimization Framework for Learning with Noisy Labels

Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, Kiyoharu Aizawa

arXiv:1803.11364v1cs.CVcs.LGstat.ML

TL;DR

DNNs can overfit noisy labels in large-scale web-collected datasets, creating a need for training without manual data cleaning. The paper jointly optimizes network parameters and labels by alternating their updates, and reports strong results on noisy CIFAR-10 and Clothing1M, including state-of-the-art performance in its conclusion.

  • Problem

    DNNs can memorize noisy training labels, degrading performance when large-scale web-collected datasets contain inaccurate annotations.

  • Method

    The framework jointly optimizes network parameters and labels, alternating updates so labels can be corrected during training.

  • Results

    The framework performs remarkably well on noisy CIFAR-10 and Clothing1M, outperforming the state-of-the-art methods.

  • Takeaways & Limitations

    Jointly updating labels and network parameters provides a framework for learning on noisy labeled datasets without treating noisy labels as fixed.

  • Takeaways & Limitations

    The method’s related-work scope notes that training-loss-based early stopping does not work well for measuring clean-label performance on noisy datasets.

Abstract

from arXiv · show

Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance degradation because DNNs easily overfit to noisy labels. To overcome this problem, we propose a joint optimization framework of learning DNN parameters and estimating true labels. Our framework can correct labels during training by alternating update of network parameters and labels. We conduct experiments on the noisy CIFAR-10 datasets and the Clothing1M dataset. The results indicate that our approach significantly outperforms other state-of-the-art methods.

1. Introduction

The paper addresses training DNNs on noisy web-collected labels without manual cleaning by jointly optimizing network parameters and labels. It reports strong performance on noisy CIFAR-10 and comparable performance on Clothing1M.

  • Motivation: Noisy web-collected labels can cause DNNs to overfit, motivating training methods that do not require manually cleaning datasets.The paper frames effective learning from noisy labeled datasets as its central question.
  • Proposed framework: The proposed framework jointly optimizes network parameters and labels through an alternating strategy.Labels are optimized rather than treated as fixed, enabling inaccurate labels to be corrected during training.
  • Observation: DNNs trained with a high learning rate can maintain high clean-data performance without memorizing noisy labels.This observation is presented as evidence that DNNs first learn simple patterns and later memorize noisy data.
  • Evaluation: The framework achieves state-of-the-art performance on noisy CIFAR-10 and comparable performance on Clothing1M.The evaluation covers both synthetic and real noisy datasets.

2. Related Works

Prior work addresses noisy labels through regularization, transition-matrix modeling, robust losses, and self-training. These approaches improve robustness but may rely on unavailable information, fail to address memorization explicitly, or retain noisy labels during training.

  • Regularization: Regularization methods such as weight decay, dropout, adversarial training, and mixup are used to address DNNs fitting noisy labels.They improve performance on clean labels but do not explicitly handle noisy labels.
  • Regularization: Long training can degrade performance, and noisy-dataset training loss is unreliable for early stopping based on clean-label performance.The last epoch is generally worse than the best epoch, while training loss cannot measure clean-label performance reliably.
  • Noise transition matrix: Noise-transition approaches either assume the ground-truth transition matrix or estimate it from noisy data.Estimated-transition performance can be slightly worse than using the true transition matrix.
  • Robust loss function: Robust-loss methods use losses such as ramp, unhinged, mean square, and mean absolute error without considering arbitrary-label learning by DNNs.The related work identifies memorization as an unaddressed issue for these methods.
  • Self-training and pseudo-labeling: Self-training and pseudo-labeling repeatedly retrain models while updating pseudo-labels, but noisy-label learning must treat all data equally because noisy examples are unknown.A related approach continues using original noisy labels until the end of training.

3. Notation and Problem Statements

The paper formulates c-class classification with training images and noisy labels, using network outputs in an empirical-risk objective. The task is to train CNNs to predict true labels when only noisy training labels are available.

  • Notation: Soft-label space S contains class-probability vectors with entries in [0, 1] that sum to one.The notation distinguishes soft labels from one-hot hard labels.
  • Notation: The dataset contains n training images X = [x_1, ..., x_n] with one-hot ground-truth labels represented in hard-label space H.Hard-label space consists of one-hot vectors whose entries sum to one.
  • Objective: The objective uses empirical risk, such as cross entropy, with network parameters θ and final-layer softmax output s.The softmax layer produces outputs for c-class prediction.
  • Problem statement: With only noisy labels Y available, the task is to train CNNs that predict the true labels.The noisy labels are denoted y_i, while the ground-truth label set is unavailable for training.

4. Classification with Label Optimization

The framework alternates updates to network parameters and labels to train on noisy data, using regularization terms to avoid degenerate or stalled solutions.

  • Objective: The loss combines classification and two regularization terms, with α and β controlling the regularization contributions.The classification term uses KL-divergence, while Lp and Le serve distinct regularization roles.
  • Alternating Optimization: The proposed method alternates SGD updates of network parameters with label updates for hard- or soft-label optimization.The two-step training procedure first obtains clean labels by updating labels, then trains the network with supervised learning using those labels.
  • Label Optimization: Updating labels minimizes the classification loss with network parameters fixed, separately for each training example.The framework supports both hard-label and soft-label updates.
  • Label Optimization: The soft-label method performed better than the hard-label method in experiments and was therefore used unless otherwise specified.
  • Regularization Terms: Lp prevents all examples from being assigned to one class by encouraging the mean predicted class distribution to follow a prior distribution.The mean probability is approximated over each mini-batch; this approximation does not handle many classes or extreme class imbalance well.
  • Regularization Terms: Le adds an entropy term for soft labels because updating parameters and labels without it can leave both stuck in local optima.The entropy term concentrates each soft-label probability distribution toward a single class.

5. Experiments

Experiments evaluate the framework on synthetic CIFAR-10 noise settings and Clothing1M, while examining learning rates, label-update strategies, and recovery accuracy. High learning rates and soft labels support robust training and label recovery, with strong results on noisy CIFAR-10.

  • Datasets: The experiments use SN-CIFAR, AN-CIFAR, and PL-CIFAR, plus Clothing1M as a real-world noisy-label dataset.PL-CIFAR pseudo labels have 62.50% accuracy; Clothing1M contains 1 million images across 14 classes.
  • Overall evaluation: The study reports state-of-the-art performance on noisy CIFAR-10 and comparable performance on Clothing1M.The experiments use soft labels in the reported later experimental sections.
  • Learning-rate effects: At lr=0.02, test accuracy gradually decreases as the network fits noisy labels, whereas lr=0.2 maintains high test accuracy during training.The high learning rate is reported to prevent memorizing and fitting noisy labels.
  • Learning-rate effects: With a low learning rate, training loss approaches 0 even at noise rate r=0.9; with a high learning rate, increasing noise raises training loss early in training.The authors interpret this pattern as allowing label optimization toward lower training loss at high learning rates.
  • Hard-label vs. soft-label: The soft-label method converges faster and achieves better recovery accuracy than the tested hard-label methods.The comparison uses SN-CIFAR with noise rate r=0.7 and 1500 training epochs in the first method step.
  • Hard-label vs. soft-label: 86.0% test accuracy for soft labels exceeds the optimal hard-label result of 85.7% by 0.3%.Soft labels retain class probabilities, whereas hard labels ignore confidence; the authors conclude that confidence matters for noisy-label training.

5.5. Experiment on SN-CIFAR

Experiments compare the proposed method with training on noisy labels across synthetic and pseudo-label settings. The method improves test and recovery accuracy, while optimized labels remain sensitive to noise type and may not be optimal.

  • SN-CIFAR: The method is evaluated on SN-CIFAR across noise rates r = 0.0, 0.1, 0.3, 0.5, 0.7, 0.9, using validation-selected α and β.Performance is compared with training directly on the initial noisy labels.
  • SN-CIFAR: Our method achieves overall better test accuracy and recovery accuracy on SN-CIFAR.With initial noisy labels, test accuracy decreases after approximately the 40th epoch, whereas accuracy remains high through training with optimized labels.
  • AN-CIFAR: The method achieves significantly better test accuracy and recovery accuracy on AN-CIFAR, except for the 50% noise-rate comparison.At 50% noise, CAT and DOG exchanges make the classes impossible to determine accurately.
  • AN-CIFAR: With a low learning rate, test accuracy decreases when training fits noisy labels, whereas accuracy remains high with the proposed method and with its optimized labels.The same decline is observed with the forward corrected loss.
  • PL-CIFAR: Pseudo-label training can have lower training loss than symmetric noise despite more inaccurate labels, because pseudo labels depend on both noise rate and noise type.The pseudo labels originate from an ImageNet-pretrained ResNet-50 and are already considered optimized by that network.
  • PL-CIFAR: Recovered labels can yield worse test accuracy than ground-truth labels, showing that labels optimized for the network are not necessarily optimal.The paper identifies this as a limitation of the proposed method.
  • Clothing1M: On Clothing1M, the method achieves better test accuracy than comparison #2 while requiring only the noisy-label distribution rather than a ground-truth noise transition matrix.Comparison #2 uses 50k clean examples and their noisy versions to estimate the transition matrix.
  • PL-CIFAR: Soft labels encode class probabilities, allowing incorrectly updated labels to receive lower importance during training.The paper states that this effect contributes to improving test accuracy.

6. Conclusion

The paper concludes that alternating optimization of network parameters and class labels provides a framework for noisy-label learning. It reports strong performance on noisy CIFAR-10 and Clothing1M, outperforming state-of-the-art methods.

  • Conclusion: The proposed framework alternately updates network parameters and class labels for learning from noisy datasets.The conclusion attributes its performance to high learning rates preventing memorization of noisy labels.
  • Conclusion: The framework performs remarkably well on noisy CIFAR-10 and Clothing1M.The reported experiments cover both datasets.
  • Conclusion: Updated labels can align with the network's training loss on ground-truth labels, indicating optimization toward the network's preferences.The corresponding figure caption states that the updated labels are completely optimized for the network.
  • Conclusion: The framework outperforms the cited state-of-the-art methods.The conclusion directly reports this comparison.

A. Detailed Architecture

The CIFAR-10 experiments use a network architecture based on PreAct ResNet-32.

  • Architecture: The CIFAR-10 experiments use a network architecture based on PreAct ResNet-32.The architecture is detailed in Table 4.

B. Dependency on Hyper Parameters

The experiments examine hyperparameter choices, learning-rate dependence, and label-update timing. Performance is reported as relatively robust to hyperparameter variation, while higher noise rates tend to favor higher optimal learning rates.

  • Learning rate: For SN-CIFAR, the optimal learning rate tends to be higher when the noise rate is high.The relationship is reported in the hyperparameter analysis.
  • Hyperparameter robustness: Prediction accuracy is not highly sensitive to the hyperparameters tested, and the method performs well with different hyperparameter sets.The appendix reports validation results across multiple settings.
  • Label-update timing: The label-updating schedule is evaluated by varying t1, the start epoch, and t2, the stop epoch.Tables 10 and 11 report validation accuracy for different update intervals.
  • Label-update timing: With a high learning rate, prediction accuracy retains a high value while label updating is scheduled.This observation motivates examining when label updating starts and stops.

C. Effect of Soft-Labeling

Soft-labeling represents each updated label as a class-probability distribution, allowing the network to downweight uncertain or incorrectly updated labels. Recovery accuracy is analyzed across probability ranges on AN-CIFAR and SN-CIFAR under specified noise rates.

  • Soft-labels encode probabilities for each class, allowing the network to treat incorrectly updated labels as less important.High-probability soft-labels are almost correct, whereas low-probability labels are more likely to be updated incorrectly.
  • Table 12 evaluates recovery accuracy across maximum-probability ranges on AN-CIFAR with noise rate = 0.4.
  • Table 13 evaluates recovery accuracy across maximum-probability ranges on SN-CIFAR with noise rate = 0.7.
Loading 1803.11364v1…