PaperlayerView exact revision on arXiv

Source-linked AI summary

Confident Learning: Estimating Uncertainty in Dataset Labels

Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang

arXiv:1911.00068v6stat.MLcs.LG

TL;DR

Noisy-label research often emphasizes model predictions rather than uncertainty in dataset labels. This paper generalizes confident learning by combining pruning, counting, and ranking under class-conditional noise to estimate the noisy–uncorrupted-label joint distribution. CL exactly finds label errors under sufficient conditions and outperforms recent methods across noisy-label tasks, while supporting dataset curation and improved learning after cleaning.

  • Problem

    The paper addresses how to identify label errors and learn with noisy labels when true labels and annotation information are unavailable.

  • Method

    CL combines pruning, counting, and ranking under a class-conditional noise assumption to estimate the joint distribution between noisy and latent labels.

  • Results

    CL exactly finds label errors under sufficient conditions and outperforms recent prior methods across noise estimation, label-error finding, and noisy-label learning tasks.

  • Takeaways & Limitations

    CL identifies label issues across datasets and modalities and can improve model performance by training on cleaned data.

  • Takeaways & Limitations

    The theoretical analysis includes conditions for robustness to erroneous predicted probabilities, while the simplified formulation can produce label collisions.

Abstract

from arXiv · show

Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amazon Reviews). We also employ CL on ImageNet to quantify ontological class overlap (e.g., estimating 645 "missile" images are mislabeled as their parent class "projectile"), and moderately increase model accuracy (e.g., for ResNet) by cleaning data prior to training. These results are replicable using the open-source cleanlab release.

1. Introduction

Confident learning addresses noisy-label datasets with a data-centric framework that estimates label uncertainty, finds label errors, and supports learning across modalities and models.

  • Noisy labels create the problems of identifying erroneous examples and learning effectively regardless of data modality or model.
  • The framework directly estimates the joint distribution between noisy and uncorrupted labels, whose marginals provide several label-noise statistics.
  • Confident learning combines pruning, counting, and ranking to characterize, identify, and learn with label errors.
  • CL provides sufficient conditions for exactly finding label errors and estimating the noisy-to-true-label joint distribution.
  • Experiments evaluate CL for noise estimation, label-error finding, and learning with noisy labels, including CIFAR comparisons and cleaned-ImageNet accuracy gains.
  • The approach is presented as reproducible through an open-source implementation and evaluated across CIFAR, ImageNet, WebVision, and MNIST.

2. CL Framework and Problem Set-up

The CL framework formulates noisy-label learning around latent true labels and class-conditional noise, estimating label uncertainty without annotation information or known latent quantities.

  • The problem setting includes multiclass examples with observed noisy labels paired to inputs, while true labels remain latent.
  • The framework generally assumes no annotation information beyond the observed noisy labels.
  • A class-conditional noise process maps each latent true label to an observed label with class-dependent flipping probabilities.
  • CL estimates the joint distribution of noisy and latent labels rather than estimating the noise transitions and latent-label prior independently.
  • Sparsity measures the fraction of zero off-diagonal entries in the noisy-to-true-label joint distribution, capturing non-uniform noise.
  • Self-confidence is the predicted probability assigned to an example's given label, with low self-confidence serving as a heuristic likelihood of label error.

3. CL Methods

Confident learning estimates label-noise structure from model probabilities and noisy labels, then uses that structure to identify errors, clean data, and train more reliably. Its thresholded confident joint is designed to handle class imbalance and heterogeneous probability distributions while remaining model-agnostic.

  • CL combines out-of-sample predicted probabilities with noisy labels to estimate the joint distribution of observed and latent labels.The model is decoupled from data cleaning, so any model producing predicted probabilities can supply the required outputs.
  • Prune and train: CL cleans datasets by filtering noisy examples and training on the remaining data with class-based reweighting.The overall procedure estimates noise, removes suspected errors, and trains with errors removed.
  • Count: The confident joint counts likely label errors by partitioning examples into noisy-label and estimated-true-label bins using per-class thresholds.Diagonal counts represent correct labels, while off-diagonal counts represent asymmetric label errors.
  • Robustness: Per-class thresholding and collision handling improve robustness to class imbalance and heterogeneous predicted-probability distributions.Unlike argmax-based confusion counting, thresholds adapt to class-specific confidence and reduce sensitivity to overconfident classes.
  • Estimate: CL estimates the joint distribution from calibrated counts, then derives noise transition and mixing matrices from that estimate.Count-based estimation is one step removed from predicted probabilities, supporting robustness to imperfect probability estimation.
  • Rank: The method’s default ranking strategy orders detected errors by normalized prediction margin before training.The normalized margin compares the probability of the observed label with the largest competing class probability.

4. Theory

The theory establishes when confident learning exactly identifies label errors and consistently estimates the noisy–true-label joint distribution. It extends ideal-probability guarantees to structured per-class and per-example prediction errors, while retaining assumptions about collision-free assignments and label-noise structure.

  • Assumptions: CL’s guarantees require at least one example from every class and interpret approximate equality as a consequence of finite-count discretization.For small classes, attainable count ratios may differ slightly from the underlying real-valued noise rates.
  • Ideal probabilities: Under ideal predicted probabilities and row-and-column diagonal dominance, CL exactly finds label errors and consistently estimates the joint distribution.The guarantee is approximate only up to discretization from count-based estimation.
  • Per-class robustness: Per-class diffracted probabilities preserve exact error finding and consistent joint estimation without requiring diagonal dominance across columns.This robustness covers linear combinations of class-conditional probability errors and contrasts with the confusion baseline.
  • Scope: The per-class robustness corollary is limited to structured probability alterations, whereas real-world models may produce up to n × m distinct erroneous probabilities.Theorem 2 addresses this broader per-example setting.
  • Per-example robustness: Theorem 2 extends robustness to per-example probability errors when predictions are collision-free and satisfy the stated diagonal condition.Within the residual ranges between ideal probabilities and thresholds, CL can identify errors despite probability perturbations.
  • Uncertainty: The theory treats joint-distribution estimation as epistemic label-noise uncertainty while accommodating aleatoric uncertainty in erroneous predicted probabilities.This separation is central to the paper’s robustness analysis.

5. Experiments

Experiments evaluate CL for joint estimation, label-error discovery, and learning with noisy labels across CIFAR, ImageNet, Amazon Reviews, and MNIST. CL generally remains robust across noise and sparsity, identifies diverse real-world label issues, and improves learning after data cleaning.

  • CIFAR-10: CL finds label errors with high recall and reasonable F1 across noise and sparsity settings in CIFAR-10.Table 4 compares five CL variants using precision, recall, and F1 averaged over ten trials.
  • CIFAR-10: Across CIFAR noise and sparsity settings, CL consistently performs well against prior methods, with significant improvements in high-noise and/or high-sparsity regimes.The results support directly modeling the joint noise distribution.
  • CIFAR-10: CL estimates the noisy–true label joint distribution on CIFAR, achieving RMSE .004 on average across 20% and 40% noise settings and all sparsities.The simplest Cconfusion variant achieves RMSE .006.
  • ImageNet: Human validation found 58% of 500 CL-identified ImageNet issues were multi-label cases, ontological issues, or errors.CL also identified ontological relationships and mixed issue types, including duplicated or semantically overlapping classes.
  • ImageNet: Cleaning ImageNet with CL reduced the training dataset by 10% while moderately improving validation accuracy and significantly improving accuracy on the erroneous maillot class.CL outperformed random removal in nearly all reported experiments and also improved average accuracy in the 1,000- and 20-class settings.
  • Amazon Reviews: On Amazon Reviews, all CL methods significantly outperformed the baseline, including with nearly half the training data pruned.With N = 1000K, baseline training began overfitting to noise while CL test accuracy continued increasing through 50 epochs.
  • MNIST: CL identified blatant label errors in the original unperturbed MNIST training dataset, which is widely presumed error-free.A two-layer CNN supplied cross-validated probabilities, and CL: PBNR ranked the top 24 errors by self-confidence.

6. Related work

Confident learning extends prior noise-estimation and robust-learning approaches with a theoretically grounded, model-agnostic method for multiclass label uncertainty. Unlike iterative or tightly coupled alternatives, CL directly estimates the noisy–true label joint distribution and modularly supports finding and removing label errors.

  • CL extends binary counting principles to multiclass label-noise estimation while addressing class imbalance and predicted-probability error.
  • CL provides a consistent estimator for the joint distribution of noisy and true labels under practical conditions.
  • Unlike INCV, CL uses theoretically supported thresholds rather than argmax disagreement to count errors robustly against class-level probability bias.
  • CL computes cross-validated probabilities once, directly estimates the joint distribution, and retrains in a single pass rather than iterating.
  • CL is modular: training, label-error finding, and error ordering are independent, whereas INCV ties them together in one iterative loop.

7. Conclusion and Future Work

The paper develops and evaluates confident learning for estimating label noise, finding label errors, and learning from noisy data. Across datasets and modalities, the findings emphasize practical dataset cleaning and improved model performance from training on cleaned data.

  • CL estimates joint label-noise distributions and evaluates their use for finding label errors and learning with noisy labels.
  • CL methods outperform recent prior art across uncertainty quantification, label-error finding, and noisy-label learning on CIFAR-10.
  • CL identifies pre-existing label issues in ImageNet, Amazon Reviews, MNIST, and other datasets while improving deep-network performance on cleaned data.

Appendix A. Theorems and proofs for confident learning

The appendix proves that confident learning can exactly recover label-error partitions and consistently estimate the noisy–true label joint distribution under ideal or bounded-error probability conditions. The results extend to per-class and per-example robustness when stated threshold and collision assumptions hold.

  • Theorem 1: Under ideal predicted probabilities and row-and-column-maximizing diagonal noise rates, CL exactly finds label errors and consistently estimates the noisy–true label joint distribution.
  • Proof strategy: The proof handles collisions by assigning examples through the confident-joint construction and uses partition arguments when no collisions occur.
  • Corollary 1.0: The exact-estimation corollary follows because the confident joint counts every noisy–true label partition, up to discretization rounding.
  • Corollary 1.1: Per-class robustness preserves exact label-error finding and consistent joint estimation for diffracted probabilities without label collisions when diagonal noise rates maximize their rows.
  • Robustness: CL relies on probability rankings for finding label errors, so valid calibrated probabilities are unnecessary for that task.
  • Theorem 2: Per-example robustness permits uniformly distributed probability errors while preserving exact label-error finding and joint estimation under the theorem’s collision and row-maximization conditions.

Appendix B. The confident joint and joint algorithms

The confident-joint and joint algorithms convert out-of-sample predicted probabilities and noisy labels into counts and an estimated latent class-conditional noise distribution. The procedure is computationally explicit and supports reproducible label-error analysis.

  • The confident-joint algorithm takes an n × m out-of-sample probability matrix and associated noisy labels as inputs.
  • The algorithm computes the unnormalized confident joint in O(m^2 + nm) steps.
  • Finding label errors in ImageNet takes 3 minutes on an i7 CPU, with results reproducible through the open-source cleanlab package.
  • Algorithm 2 calibrates confident-joint counts to estimate the latent true distribution of class-conditional label noise.

Appendix C. Extended Comparison of Confident Learning Methods on CIFAR-10

On CIFAR-10, confident learning accurately estimates label-noise joint distributions in moderate-noise regimes and improves over confusion-matrix estimation, while validation-set cleaning remains a limitation.

  • In moderate 20%–40% noise regimes, confident learning accurately estimates nearly every entry of CIFAR-10’s label-noise joint distribution.The evaluation spans 20%, 40%, and 70% noise with varying sparsity.
  • Validation-set label errors were not removed, potentially inducing distributional shift when training on the CL-cleaned training set.
  • C̃y,y∗ estimation improves over the confusion-matrix baseline, achieves low RMSE, and remains robust to sparsity under moderate noise.
  • Figure S1 measures absolute differences between the true and CL-estimated joint distributions across noise and sparsity settings.

C.1 Benchmarking INCV

The INCV benchmark required substantial computational resources and repeated restarts because implementation memory leaks caused frequent out-of-memory failures.

  • INCV was benchmarked using 128 GB of RAM and four RTX 2080 Ti GPUs.
  • Memory leaks in the February 2020 implementation frequently caused out-of-memory errors during training.
  • For fair comparison, training was restarted until every model completed at least 90 epochs, with time, epochs, and accuracy recorded.

Appendix D. Additional Figures

The appendix provides supplementary evidence for CL experiments, including ImageNet accuracy comparisons, synthetic-noise construction, and additional method-level benchmarks.

  • Figure S2 compares top-1 ILSVRC validation accuracy after removing increasing fractions of CL-estimated errors against random-example removal.
  • Figure S2 includes Clopper–Pearson 95% confidence intervals, a random-removal baseline, and an all-examples training reference.
  • Figure S3 supplies CIFAR-10 transition matrices used to generate synthetic noisy labels for the appendix experiments.
Loading 1911.00068v6…