NLNL: Negative Learning for Noisy Labels

Youngdong Kim; Junho Yim; Juseung Yun; Junmo Kim

Source-linked AI summary

NLNL: Negative Learning for Noisy Labels

Youngdong Kim, Junho Yim, Juseung Yun, Junmo Kim

arXiv:1908.07387v1Aug 19, 2019cs.LG

TL;DR

Noisy labels can cause CNNs trained with conventional positive learning to overfit incorrect information and degrade image-classification performance. The paper proposes Negative Learning and its selective PL extension, SelNLPL, achieving state-of-the-art noisy-data classification through improved noisy-label filtering.

Problem
Noisy labels can cause CNNs trained with conventional positive learning to overfit incorrect information, degrading classification performance.
Method
The paper trains CNNs with complementary-label Negative Learning, then selectively applies Positive Learning to high-confidence data in SelNLPL before semi-supervised classification.
Results
The method achieves state-of-the-art performance in noisy-data classification through accurate filtering of noisy training data.
Takeaways & Limitations
Complementary-label training with selective positive learning can support noisy-data classification without tuning based on prior knowledge of noise type or ratio.
Takeaways & Limitations
The method assumes that data with confidence exceeding γ is clean during SelNLPL training.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) provide excellent performance when used for image classification. The classical method of training CNNs is by labeling images in a supervised manner as in "input image belongs to this label" (Positive Learning; PL), which is a fast and accurate method if the labels are assigned correctly to all images. However, if inaccurate labels, or noisy labels, exist, training with PL will provide wrong information, thus severely degrading performance. To address this issue, we start with an indirect learning method called Negative Learning (NL), in which the CNNs are trained using a complementary label as in "input image does not belong to this complementary label." Because the chances of selecting a true label as a complementary label are low, NL decreases the risk of providing incorrect information. Furthermore, to improve convergence, we extend our method by adopting PL selectively, termed as Selective Negative Learning and Positive Learning (SelNLPL). PL is used selectively to train upon expected-to-be-clean data, whose choices become possible as NL progresses, thus resulting in superior performance of filtering out noisy data. With simple semi-supervised training technique, our method achieves state-of-the-art accuracy for noisy data classification, proving the superiority of SelNLPL's noisy data filtering ability.

1. Introduction

Noisy or mismatched labels can severely degrade CNN image classification because Positive Learning (PL) supplies incorrect supervision. The paper introduces Negative Learning (NL) and Selective Negative Learning and Positive Learning (SelNLPL) to reduce overfitting and improve noisy-data filtering.

Introduction: Accurately labeling large image datasets is difficult and mismatched labels can degrade CNN training.CNNs perform well when corresponding labels are correct, but large-scale accurate labeling is daunting and time-consuming.
Introduction: Positive Learning trains CNNs with the statement that an input image belongs to its label, but mismatched labels provide wrong information.PL is described as a typical supervised method whose errors arise when images receive incorrect labels.
Introduction: Negative Learning trains CNNs with complementary statements that an input image does not belong to a label, reducing the frequency of incorrect information and preventing overfitting to noisy data.The paper illustrates NL with a dog image and the complementary label “bird,” and demonstrates its effectiveness against noisy-data overfitting.
Introduction: SelNLPL combines NL with selective PL, applying PL only to high-confidence training data after NL training to filter noisy data accurately.PL remains useful for clean data, while NL enables selecting data expected to be clean through classification confidence.
Introduction: After filtering, SelNLPL discards the noisy labels and treats those samples as unlabeled data for simple semi-supervised learning, achieving state-of-the-art noisy data classification performance.The paper attributes this result to SelNLPL’s superior noisy-data filtering ability.

2. Related works

Related work addresses noisy labels through robust losses, sample reweighting, label correction, and sample pruning, but many approaches depend on clean data or risk overfitting noisy labels. NL combines pruning and label correction while indirectly using noisy labels to avoid memorization and improve noisy-sample filtering.

Noise-robust losses: Noise-robust losses include MAE and generalized cross entropy, although MAE can reduce neural-network accuracy while generalized cross entropy remains robust and effective on deep networks.Ghosh et al. theoretically established MAE’s robustness, while Zhang et al. proposed generalized cross entropy for noisy-label robustness in deep networks.
Sample reweighting: Sample-reweighting methods assign reliability-based weights, but meta-learning approaches require clean data and CleanNet requires a label verified as correct.These requirements make the methods difficult to apply when clean data is scarce.
Label correction: Label-correction methods estimate or directly correct noisy labels, yet they commonly assume known noise transitions or require clean data to train cleaning and teacher networks.Some methods model the noise transition matrix with an additional layer, while others correct labels directly.
Pruning and alternative approaches: Other approaches model worker quality, address open-set noise, or prune samples; unreliable samples can instead be trained semi-supervised without label information.Ding et al. proposed pruning based on softmax outputs and semi-supervised training for samples deemed unreliable.
Positioning of NL: NL combines correct-sample pruning with label correction while indirectly using noisy labels, avoiding memorization that can arise when networks train directly on given noisy labels.The authors contrast NL with existing pruning and label-cleaning methods, which may overfit noisy labels even when pruning or cleaning is performed.

3. Method

The method trains CNNs with complementary-label supervision to reduce overfitting on noisy data, then applies confidence-based selective learning to improve convergence and filter noisy samples. SelNLPL combines NL, SelNL, and SelPL, enabling subsequent semi-supervised learning on the separated data.

3.1 Negative Learning: Negative Learning (NL) trains CNNs so inputs are assigned not to randomly selected complementary labels rather than directly to their given labels.The complementary label is selected randomly from all classes except the given label at each training iteration.
3.1 Negative Learning: NL optimizes the complementary label’s probability toward zero, increasing the probabilities of the other classes and reducing the risk of learning incorrect noisy labels.Unlike PL, which drives the given-label probability toward 1, NL drives the complementary-label probability toward 0.
3.2 Selective Negative Learning: Selective Negative Learning (SelNL) improves convergence after NL by training only on samples whose confidence exceeds 1/c, making the selected data less noisy.Confidence thresholding follows NL because NL separates clean and noisy data through lower confidence on noisy samples.
3.3 Selective Positive Learning: Selective Positive Learning (SelPL) then applies PL only to samples with confidence above γ, treating them as clean; this study sets γ to 0.5.After NL and SelNL, clean and noisy samples are separated by a large confidence margin, and clean samples reach confidence near 1.
Overall SelNLPL: The combined SelNLPL procedure improves convergence while preventing overfitting, maintains higher test accuracy than training accuracy, and separates clean from noisy data for filtering.The filtered data can then support semi-supervised learning with pseudo-labeling after discarding labels from filtered noisy samples.

4. Filtering ability

SelNLPL filters noisy training data by treating data not trained with PL as noisy and estimating the noise ratio from the filtered data. Its filtering performance improves across successive steps, surpassing PL, while the estimated noise ratio nearly matches the actual ratio.

Filtering results: Table 1 evaluates SelNLPL’s noisy-data filtering on CIFAR10 under 10%, 30%, and 50% symmetric-increasing noise settings using recall and precision.The corresponding actual noise ratios are 9%, 27%, and 45%.
Filtering process: SelNLPL assumes data with confidence exceeding γ are clean and filters data not trained with PL as noisy.The estimated noise ratio is defined by the amount of data not trained with PL.
Filtering results: SelNLPL can estimate the amount of noise even when the training-data noise level is unknown, providing an indicator of training-data quality.The estimated noise ratio is based on data not trained with PL.
Filtering performance: Each successive SelNLPL step increases filtering performance, and the resulting curve surpasses PL trained before overfitting to noisy data.Figure 6 compares PL, NL, NL→SelNL, and NL→SelNL→SelPL (SelNLPL).
Filtering performance: The estimated noise ratio from SelNLPL nearly matches the actual noise ratio, supporting its use when the actual ratio is unavailable.This makes it useful for indicating training-data quality in practical situations.

5. Experiments

Experiments evaluated SelNLPL across multiple CNN architectures, datasets, and noise settings by comparing its pseudo-labeling results with existing methods. The method achieved the best or near-best performance across most comparisons, with gains of up to 5% in some cases.

Experimental settings: Experiments compared SelNLPL with four baseline methods across CIFAR10, CIFAR100, FashionMNIST, and MNIST.The settings varied by CNN architecture, dataset, and training-label noise.
Results: SelNLPL achieved the best overall accuracy in almost all Table 3 cases across architectures, datasets, noise types, and noise ratios.Its maximum reported improvement was 5%, and it failed to converge only under 80% symm-exc noise.
Results: SelNLPL outperformed all comparable methods in Table 4 across noise types and ratios despite fixed hyper-parameters rather than settings tailored to each noise condition.This condition is more realistic because the noise type and ratio are unknown.
Results: SelNLPL surpassed most comparable results in Tables 5 and 6 across CNN architectures, datasets, noise types, and ratios.Its performance exceeded other methods by up to 4∼5% in some cases.

6. Analysis

The analysis shows that NL’s convergence slows as the number of classes or noise increases, while multiple complementary labels and selective learning improve convergence and noisy-data classification. SelNL is especially important under higher noise, and the complete SelNLPL sequence performs best among its ablations.

Convergence: NL training slows as the number of classes increases and failed to converge on CIFAR100 within the CIFAR10 training epoch budget.The analysis attributes this dependence to NL’s optimization process.
Classification performance: The extended method generally improved noisy-data classification and achieved state-of-the-art results for symm-exc noise.For asymm noise, Forward T [25] performed best, although the comparison was described as unfair because it relies on the priors.
Convergence: Providing multiple complementary labels per image generalizes NL to datasets with many class numbers.The extended method also enabled convergence on CIFAR100 with the same epoch count used for CIFAR10.
SelNLPL ablation: Removing either SelNL or SelPL from SelNLPL deteriorated performance, while removing both caused a further decrease.The analysis applied each ablated sequence followed by pseudo labeling.
SelNLPL ablation: SelNL becomes increasingly crucial as noise rises, with especially significant degradation after its removal under symm-inc 50% noise.NL convergence becomes less reliable as the training-data noise ratio increases.

7. Conclusion

The paper proposes Negative Learning (NL) to reduce the risk of wrong information when training CNNs with noisy data, and develops SelNLPL by combining NL with selectively faster, more accurate Positive Learning on clean data.

Negative Learning: NL trains CNNs with complementary labels stating that an input image does not belong to a selected label.This indirect learning approach targets noisy-data training.
Negative Learning: NL reduces the risk of incorrect training information because randomly chosen complementary labels are likely not ground-truth labels.The method relies on the high probability that a complementary label differs from the ground-truth label.
SelNLPL: SelNLPL combines NL with Positive Learning because PL is faster and more accurate than NL when learning from clean data.The proposed combination is intended to exploit the strengths of both learning modes.