PaperlayerView exact revision on arXiv

Source-linked AI summary

Early-Learning Regularization Prevents Memorization of Noisy Labels

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, Carlos Fernandez-Granda

arXiv:2007.00151v2cs.LGcs.CVstat.ML

TL;DR

Noisy annotations hinder classification when accurate labels are costly. The paper proves early learning and memorization in high-dimensional models and uses early-learning regularization with semi-supervised target probabilities, achieving strong noisy-label performance comparable to state-of-the-art methods.

  • Problem

    Classification methods need greater robustness to noisy annotations because accurately labeled datasets can be costly or difficult to obtain.

  • Method

    The framework regularizes models toward semi-supervised target probabilities estimated from model outputs, counteracting noisy-label gradients and memorization.

  • Results

    ELR consistently outperforms other loss-modifying methods, while ELR+ outperforms state-of-the-art methods except DivideMix across CIFAR-10 and CIFAR-100 noise settings.

  • Takeaways & Limitations

    The framework supports robust learning from noisy annotations in standard benchmarks and real-world datasets, including settings where accurate labels are costly to gather.

  • Takeaways & Limitations

    The theoretical analysis is limited to a linear generative model based on data drawn from a mixture of two Gaussians in R^p.

Abstract

from arXiv · show

We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase. In contrast with existing approaches, which use the model output during early learning to detect the examples with clean labels, and either ignore or attempt to correct the false labels, we take a different route and instead capitalize on early learning via regularization. There are two key elements to our approach. First, we leverage semi-supervised learning techniques to produce target probabilities based on the model outputs. Second, we design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels. The resulting framework is shown to provide robustness to noisy annotations on several standard benchmarks and real-world datasets, where it achieves results comparable to the state of the art.

1 Introduction

The paper studies classification with noisy annotations, where models first learn clean labels and later memorize false ones. It develops a regularization framework that exploits early learning through semi-supervised target probabilities to improve robustness.

  • Motivation: Manually verified datasets are often unavailable because labeling is costly or difficult, motivating classification methods that can handle lower-quality annotations.Examples of lower-quality annotations include labels obtained from online queries or crowdsourcing.
  • Early learning and memorization: Deep neural networks trained on noisy labels first fit cleanly labeled examples during early learning before eventually memorizing examples with false labels.The paper studies this observed transition as a basis for robust noisy-label classification.
  • Method: The resulting framework is presented as achieving robustness to noisy labels on standard benchmarks and real-world datasets, with results comparable to the state of the art.The framework’s two key elements are semi-supervised target probabilities and regularization toward those targets.
  • Theory: Early learning and memorization occur even in simple linear generative models, establishing them as fundamental phenomena in high-dimensional classification.This theoretical result is established in Section 3.
  • Method: The proposed technique counters noisy labels by using a regularization term that incorporates target probabilities estimated from model outputs with semi-supervised learning.The regularizer is designed to utilize early learning and counteract noisy labels’ influence on the cross-entropy gradient.

2 Related Work

Existing noisy-label methods include robust losses, loss correction, sample selection, and label correction. The proposed approach instead uses probability estimates in a novel regularizer that corrects cross-entropy gradients without modifying labels or selecting samples.

  • Robust-loss methods: Robust-loss methods design noise-resistant objectives, including MAE, Improved MAE, Generalized Cross Entropy, Symmetric Cross Entropy, and LDIM.Improved MAE reweights MAE; Generalized Cross Entropy generalizes MAE; Symmetric Cross Entropy adds reverse cross-entropy; LDIM uses information-theoretic considerations.
  • Loss-correction methods: Loss-correction methods explicitly adjust the loss using a transition matrix representing mislabeling probabilities.The transition matrix encodes the noise distribution.
  • Early-learning methods: Robust-loss and loss-correction methods do not exploit early learning, which prior work described and analyzed theoretically.The paper’s analysis differs by addressing noisy-label classification rather than least-squares regression and proving early learning and memorization in a linear model.
  • Early-learning methods: Early learning supports sample selection by using model outputs and losses to identify mislabeled examples, as in Co-teaching’s two-network procedure.Mislabeled examples tend to have higher loss, and each network trains on examples with small loss for the other network.
  • Early-learning methods: Label correction replaces labels with model-derived soft probabilities, hard one-hot predictions, or other corrected targets during early learning.The approach is related in spirit but instead uses probability estimates in a regularization term that corrects cross-entropy gradients, avoiding label modification and sample selection.

3 Early learning as a general phenomenon of high-dimensional classification

Early learning followed by memorization is intrinsic to high-dimensional classification, not peculiar to deep neural networks: a noisy-label linear model exhibits both stages. Theoretical analysis explains this transition and motivates regularization that preserves clean-label learning while neutralizing noisy-label influence.

  • A simple linear classifier with noisy labels first learns true labels, improving predictions even on noisy examples, before memorizing incorrect labels.Its qualitative behavior matches the early-learning and memorization pattern observed in deep-learning models.
  • The analysis considers two Gaussian clusters in R^p with fixed σ^2 and n, p →∞, where centers are separated by 2 units but cluster radii satisfy σ√p ≫2.The optimal separator is the hyperplane through the origin perpendicular to v.
  • As early learning proceeds, clean-example gradient coefficients decrease while wrong-label coefficients increase; once wrong-label effects dominate, memorization occurs and all noisy labels are eventually memorized.The proof attributes early progress to the majority of correctly labeled examples and later memorization to sufficiently many orthogonal directions in high dimension.
  • Under small enough σ and p/n ∈ (1 − ∆/2, 1), with probability 1 − o(1), gradient descent has an early-learning interval T = Ω(1/η).During this interval, the negative gradient correlates with the correct separator and accuracy on wrongly labeled examples exceeds initialization.
  • The analysis motivates regularization that keeps clean-label gradient contributions large and neutralizes wrong-label influence to prevent memorization.The proposed method is designed to achieve these two objectives in Section 4.

4 Methodology

The methodology connects label-noise memorization in linear and neural models to reversed gradient contributions, then introduces early-learning regularization (ELR) to exploit early model outputs and prevent memorization. ELR uses probability targets and a targeted regularizer whose gradient boosts clean examples while dampening mislabeled ones.

  • Neural-network effect of label noise: In neural networks, a noisy label reverses the gradient contributions for the true and impostor classes, eventually causing memorization.The effect is analogous to the linear model because label noise changes the cross-entropy gradient through p[i] − y[i].
  • Early-learning regularization: ELR assigns each training example a probability target computed from past model outputs and regularizes training toward those targets before they overfit noisy labels.The targets are assumed not to overfit during the early-learning phase, enabling regularization to exploit this transient reliability.
  • ELR gradient mechanism: When the true-class target dominates, ELR makes its regularization gradient negative, counteracting the later disappearance of clean-example cross-entropy gradients.This prevents mislabeled examples from dominating the optimization after the early-learning stage.
  • ELR gradient mechanism: ELR boosts clean-example gradients and neutralizes false-label gradients, thereby reducing the influence of mislabeled examples on optimization.These are the two desired properties identified for preventing memorization.
  • Target estimation: Running-average targets, temporal ensembling, averaged model weights, and two-network cross-targeting provide increasingly improved target-estimation options.The two-network variant computes each network’s target from the other network’s output.

5 Experiments

The experiments evaluate ELR on simulated-noise CIFAR benchmarks and real-world Clothing1M and WebVision datasets, using setups designed for comparison with prior work. They examine ELR and the more elaborate ELR+ variant, with results reported against state-of-the-art methods on CIFAR-10 and CIFAR-100.

  • Datasets and noise settings: The evaluation covers CIFAR-10, CIFAR-100, Clothing1M, and WebVision, combining simulated label noise with real-world noisy annotations.CIFAR noise uses both symmetric uniform flips and an asymmetric class-dependent distribution.
  • Experimental setup: The experiments prioritize comparability with prior work by using the same preprocessing and architectures whenever possible.Additional experimental details are provided in supplementary Section G.
  • Compared variants: The study evaluates ELR with temporal ensembling and ELR+, which additionally uses weight averaging, two networks, and mixup augmentation.ELR+ is the more comprehensive variant of the proposed approach.
  • Results: Table 2 compares ELR+ with state-of-the-art methods on CIFAR-10 and CIFAR-100 under symmetric and asymmetric noise.For ELR+, 10% of the training set is used for validation, with that validation set treated as a held-out test set.
  • Reproducibility: Code for reproducing the experiments is publicly available online.The repository is provided at https://github.com/shengliu66/ELR.

6 Results

ELR and ELR+ perform strongly across noisy-label benchmarks, achieving state-of-the-art or near-state-of-the-art results while retaining advantages from their training design. Ablations show that ELR+ components provide independent gains, especially under high noise.

  • CIFAR benchmarks: ELR consistently outperforms loss-based methods on CIFAR-10 and CIFAR-100 across symmetric and asymmetric label noise.The comparison uses the same ResNet34 architecture, batch size, and training procedure.
  • CIFAR benchmarks: ELR+ outperforms all compared sample-selection and augmentation methods except DivideMix on CIFAR-10 and CIFAR-100.Other methods report their best test performance during training, whereas ELR+ uses a separate validation set.
  • Real-world datasets: ELR+ achieves state-of-the-art performance on Clothing1M, slightly surpassing DivideMix.The methods use a ResNet-50 pretrained on ImageNet.
  • Real-world datasets: ELR+ achieves state-of-the-art performance on WebVision and slightly surpasses DivideMix, while DivideMix performs better on ILSVRC12, particularly for top1 accuracy.ELR also performs strongly on the mini WebVision benchmark despite its simpler design.
  • Ablation study: Each ELR+ component independently improves performance on CIFAR-10, with gains modest at 40% noise but very significant at 80% noise.The ablation evaluates weight averaging, two networks, and mixup under symmetric noise.

7 Discussion and Future Work

The work characterizes early learning and memorization theoretically for a linear generative model and uses these insights to develop a framework for learning with noisy annotations. The methodology performs strongly across benchmarks, real-world datasets, and multiple network architectures, while leaving open problems for future research.

  • The paper theoretically characterizes early-learning and memorization phenomena for a linear generative model.
  • The authors use these theoretical insights to propose a framework for learning from data with noisy annotations.
  • The methodology yields strong results on standard benchmarks and real-world datasets across several network architectures.
  • The discussion identifies multiple open problems for future research.

8 Broader Impact … A.2 Early-learning succeeds

The paper argues that noisy-label learning can support applications where accurate annotation is costly, and theoretically shows that linear classifiers make meaningful early progress before memorization. In the analyzed model, early training aligns gradients with the signal and sharply improves accuracy on mislabeled examples.

  • 8 Broader Impact: The method could advance machine-learning deployment where obtaining accurate annotations is costly, including high-impact applications such as medicine.
  • A Theoretical analysis of early learning and memorization in a linear model: The theoretical analysis formalizes Theorem 1 across the early-learning and memorization behavior of a linear classification model.
  • A Theoretical analysis of early learning and memorization in a linear model: The early progress eventually halts because gradient terms from correctly labeled examples begin to disappear.
  • A.1 Notation and setup: The model is softmax regression with two weight vectors, equivalent in the linear two-class case to logistic regression through the difference θ = Θ1 − Θ2.
  • A.1 Notation and setup: The analysis represents examples using true cluster assignments and Gaussian noise, with the gradient governed by tanh(θ⊤x[i]) − ε[i].
  • A.1 Notation and setup: Training starts from a random radius-2 initialization and uses gradient descent with fixed step size η < 1 in a high-dimensional asymptotic regime.
  • A.2 Early-learning succeeds: For the first T iterations, the negative gradient has constant correlation with v, unlike the negligible correlation typically expected from a random vector.
  • A.2 Early-learning succeeds: 1/2 to approximately 1: the mislabeled-example accuracy satisfies Â(θ0) ≈ 1/2 and Â(θT) ≈ 1 with high probability.

A.3 Vanishing gradients … D The Need for Early Learning Regularization

The paper’s theory explains early learning, eventual memorization, and the failure or success of regularization in noisy-label classification. Numerical examples and comparative analyses show why early-learning regularization is needed to prevent overfitting noisy annotations.

  • A.3 Vanishing gradients: During the first T iterations, coefficients for correctly labeled examples decrease, whereas coefficients for mislabeled examples increase.This establishes the contrasting early-learning dynamics underlying the theory.
  • A.4 Memorization: If p, n →∞ and lim infp,n→∞p/n > 1 −∆/2, the classes S+ and S− are linearly separable with probability tending to 1.Linear separability implies that gradient descent on logistic loss can perfectly memorize the labels.
  • A.5 Additional lemmas: Random initialization on the sphere of radius 2 yields |θ⊤0v| = oP(1).This lemma supports the high-dimensional initialization assumptions used in the early-learning analysis.
  • B Early Learning and Memorization in Linear and Deep-Learning Models: In both linear and deep-learning models, label noise flips the sign of the per-example cross-entropy gradient term p[i] −y[i].The comparison uses a linear model and a ResNet-34 trained with 40% symmetric noise.
  • C Regularization Based on Kullback-Leibler Divergence: KL-divergence regularization fails to provide robustness: increasing λ delays memorization but does not eliminate it.The model instead overfits initial estimates and eventually memorizes wrong labels.
  • C Regularization Based on Kullback-Leibler Divergence: ELR preserves learning on clean examples, whereas KL regularization can drive their true-class probabilities downward because its regularization sign depends on the targets.For clean examples, the cross-entropy gradient tends to vanish after early learning, allowing wrong-label examples to dominate.
  • D The Need for Early Learning Regularization: Target estimation alone is insufficient: temporal ensembling with cross-entropy eventually overfits noisy labels and decreases accuracy.The proposed framework therefore combines target estimation with an early-learning regularization term.

E Proof of Lemma 2 · F Algorithms

The paper formalizes the instance-level ELR regularizer and its gradient through softmax probability estimates. It then specifies ELR and ELR+ algorithms using temporal ensembling, with ELR+ additionally combining weight averaging, two networks, and mixup augmentation.

  • E Proof of Lemma 2: The proof simplifies notation by setting p := p[i] and t := t[i], then defines the instance-level ELR regularizer R.The supplied passage introduces this notation and states that the instance-level ELR is denoted by R.
  • E Proof of Lemma 2: The proof derives the gradient of the ELR regularizer and simplifies its resulting formula.The supplied passages explicitly present the gradient of R and a subsequent simplified formula.
  • E Proof of Lemma 2: The probability estimate p is expressed using the softmax function applied to the deep-learning mapping Nx(Θ).The softmax expression normalizes exponentiated network outputs across classes.
  • F Algorithms: Algorithm 2, denoted ELR+, augments ELR with weight averaging, two networks, and mixup data augmentation.The algorithm specifies β, γ, λ, and α, maintains two networks and averaged weights, and processes mixup minibatches.
  • F Algorithms: Temporal ensembling updates each target by combining its previous value with the current network probability using momentum β.Algorithm 1 initializes targets to zero and applies t[i] ← βt[i] + (1 − β)p[i].
  • F Algorithms: Algorithm 1 trains ELR with noisy labeled data, temporal ensembling, the proposed regularization component, and stochastic gradient descent.The pseudocode requires noisy training pairs, β, λ, and a trainable neural network, then updates network parameters using SGD.
  • F Algorithms: For each mixed example, ELR+ combines two examples’ data, labels, and targets using a convex combination whose ratio is sampled from a symmetric beta distribution.The mixing procedure samples a second example and uses α to select the symmetric beta distribution.

G.1 Dataset Information … H Sensitivity to Hyperparameters

The experiments cover four noisy-label image-classification datasets, with dataset-specific preprocessing and training configurations. Hyperparameters are selected by validation-based grid search, and ELR is generally robust to temporal-ensembling momentum when averaging is sufficiently strong.

  • G.1 Dataset Information: ELR and ELR+ are evaluated on CIFAR-10, CIFAR-100, Clothing-1M, and a WebVision subset.Because CIFAR-10 and CIFAR-100 lack predefined validation sets, 10% of each training set is retained for validation.
  • G.2 Data preprocessing: All datasets use normalization, random crops, and horizontal flips, with crop sizes matched to prior work.Crop sizes are 32 for CIFAR, 224 × 224 for Clothing-1M after resizing, and 227 × 227 for WebVision.
  • G.3 Training Procedure: CIFAR training uses ResNet-34 with SGD, momentum 0.9, weight decay 0.001, batch size 128, and dataset-specific epoch and learning-rate schedules.Training lasts 120 epochs for CIFAR-10 and 150 for CIFAR-100, with learning-rate reductions at 40 and 80, or 80 and 120, respectively.
  • G.3 Training Procedure: Clothing-1M uses ImageNet-pretrained ResNet-50, batch size 64, initial learning rate 0.001, and balanced noisy-label class sampling.The model trains for 10 epochs, reducing the learning rate by 1/100 after 5 epochs while sampling 2000 mini-batches per epoch.
  • G.3 Training Procedure: WebVision uses InceptionResNetV2, batch size 32, and weight decay 0.0005, while otherwise following the CIFAR-10 optimization details.This configuration follows the cited prior works’ backbone choice and optimization setup.
  • G.4 Hyperparameters selection: CIFAR hyperparameters are selected by grid search over β and λ using validation data, with different selected values for symmetric and asymmetric noise.For Clothing1M and WebVision, the paper reuses the CIFAR-10 values; γ is 0.997 except for Clothing1M, where it is 0.9999.
  • H Sensitivity to Hyperparameters: ELR performance is robust to β when the moving-average momentum is large, but drops to 38% without averaging at β = 0.The regularization coefficient λ must be sufficiently large to neutralize gradients, according to the reported sensitivity analysis.

I Training Time Analysis

On CIFAR-10 with 40% symmetric label noise, ELR trains faster than ELR+ and the two compared state-of-the-art methods.

  • Training time comparison: ELR+ is twice as slow as ELR when trained on a single Nvidia V100 GPU.Training times are compared in hours.
  • Training time comparison: DivideMix takes more than 2 times longer than ELR+, while Co-teaching+ is about twice as slow as ELR+.The comparison uses total training time on CIFAR-10 with 40% symmetric label noise.
Loading 2007.00151v2…