PaperlayerView exact revision on arXiv

Source-linked AI summary

Active label cleaning for improved dataset quality under resource constraints

Melanie Bernhardt, Daniel C. Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C. Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew Lungren, Aditya Nori, Ben Glocker, Javier Alvarez-Valle, Ozan Oktay

arXiv:2109.00574v2cs.CVcs.LG

TL;DR

Large-scale expert relabelling is often infeasible, even though noisy labels can harm training and model evaluation. The paper proposes active label cleaning, ranking samples by estimated correctness and difficulty, and evaluates relabelling through simulation. Its experiments report that cleaning can mitigate label-noise effects across training, evaluation, and model selection, while noise-robust learning and self-supervision can complement the process.

  • Problem

    Label noise can impair model training and confound evaluation, while fully reviewing large datasets is resource-intensive and difficult cases may require multiple expert annotations.

    Source: Introduction · pp. 1–2

  • Method

    Active label cleaning prioritizes already-labelled samples by estimated label correctness and annotation difficulty, and the paper evaluates relabelling strategies with a simulation framework.

    Source: Introduction · p. 2, Active label cleaning · p. 2, Discussion · p. 7

  • Results

    Experiments show that cleaning noisy labels can mitigate their negative effects on training, evaluation, and model selection; noise-robust learning and self-supervision can also complement label acquisition.

    Source: Discussion · p. 7

  • Takeaways & Limitations

    Cleaning evaluation labels matters because noisy validation data can produce misleading performance metrics and model rankings that do not generalize to the test environment.

    Source: Discussion · p. 7

    Consistent label-error patterns may leave some mislabellings undetected, and random selection can be preferable when a dataset’s average incorrect-label rate exceeds its correct-label rate.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term "active label cleaning". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality.

Introduction

Label noise harms model training and evaluation, while fully reviewing large datasets is often too costly; re-annotation therefore needs to prioritize samples under limited budgets. Active label cleaning ranks already-labelled samples by likely correctness and difficulty, then revisits them sequentially to produce cleaner labels for training and evaluation.

Introduction

  • Noisy labels can impair model training and distort evaluation, potentially producing misleading model rankings and deployment choices.Fully reviewing large datasets is often infeasible, and difficult cases may require multiple expert opinions, increasing re-annotation cost.Source: Introduction · pp. 1–2
  • Active label cleaning targets label quality under a relabelling budget, unlike active learning’s goal of selecting unlabelled samples to improve downstream model performance.The paper also frames active learning and noise-robust learning as complementary approaches to coping with label noise.Source: Introduction · p. 2

Active label cleaning.

  • The work defines a setting for correcting already-labelled samples and proposes a priority score based on estimated label correctness and annotation difficulty.The intended clean labels can support both model training and benchmarking.Source: Active label cleaning · pp. 2–3
  • The procedure repeatedly ranks samples using predicted label posteriors, then re-annotates prioritized samples until a majority label forms or the budget is exhausted.Samples requiring more annotations to reach a majority incur greater relabelling cost.Source: Active label cleaning · pp. 2–3
  • The priority score combines estimated noisiness from label-count cross-entropy with an entropy penalty for ambiguous cases.The ambiguity term estimates aleatoric uncertainty and deprioritizes samples that may require more annotations.Source: Active label cleaning · p. 3
  • The sequential framework permits selector-model updates during cleaning using corrected labels collected so far.The paper reports that fine-tuning the selector this way improved cleaning performance.Source: Active label cleaning · p. 3

Datasets used in the experiments.

The experiments use CIFAR10H and a new chest-radiograph benchmark, NoisyCXR, both with multiple annotations per data point to model label distributions and relabelling costs.

Datasets used in the experiments.

  • CIFAR10H provides 10,000 natural images in 10 classes, with an average of 51 manual annotations per test-set image.The experiments use it to study noise rates, noise models, and active relabelling scenarios.Source: Datasets used in the experiments · p. 3
  • NoisyCXR is a new benchmark comprising 26,600 chest radiographs and multiple clinical-dataset labels indicating pneumonia-like opacities.It is presented as a more challenging imaging benchmark involving modelling difficulty as well as label noise and sample prioritization.Source: Datasets used in the experiments · p. 3
  • Both datasets provide multiple annotations per data point, supporting models of true label distributions and associated labelling costs.Source: Datasets used in the experiments · p. 3

Label noise undermines both model training and evaluation.

Label noise can reduce predictive performance and mislead validation-based model selection. The paper describes a sequential relabelling simulation that models annotator ambiguity to assess selection methods by corrected labels under re-annotation budgets.

Label noise undermines both model training and evaluation.

  • 8.5 percentage points: CIFAR10H test accuracy fell from 73.6% to 64.1% when a vanilla CNN was trained with 30% label noise.A self-supervised noise-robust classifier also fell from 80.7% to 78.7% accuracy under the same noisy training labels.Source: Label noise undermines both model… · p. 3
  • At a 40% instance-dependent noise rate, validation accuracy and model rankings were misleading: the self-supervised classifier scored 46.5% on noisy labels versus 65.8% on clean labels.Co-teaching scored 45.9% on the same noisy validation labels but achieved 69.9% on clean labels; the noisy metrics would select the lower-performing self-supervised model.Source: Label noise undermines both model… · p. 3

Simulation of sequential relabelling.

  • A sequential relabelling simulation evaluates corrected-label percentages against re-annotation count, drawing each new label from the sample’s true label distribution.This models both likely annotation errors and the differing effort needed for ambiguous samples.Source: Simulation of sequential relabelling · p. 4
  • The simulation compares vanilla CNN, co-teaching, and self-supervised selectors against random and oracle baselines, with a minimal sampler as an additional reference.Experiments use five random seeds; the budget is set to the oracle’s expected re-annotations to clean all mislabelled samples.Source: Simulation of sequential relabelling · p. 4

Cleaning noisy training labels improves predictive accuracy.

Cleaning noisy training labels improves predictive accuracy across training approaches, while active selectors correct labels with fewer re-annotations than random selection. The advantage holds across tested noise rates when average class-confusion matrices remain diagonally dominant.

Cleaning noisy training labels improves predictive accuracy.

  • Newly acquired labels benefit all tested training methods, including self-supervised initialisation and noise-robust learning.The reported comparison also finds label acquisition can outperform state-of-the-art noise-robust methods trained on noisy labels, approaching the all-true-label upper bound.Source: Cleaning noisy training labels… · p. 4

Label cleaning can be done in a resource-effective manner.

  • 3× fewer NoisyCXR or 2.5× fewer CIFAR10H re-annotations let the vanilla selector reach 90% correct labels than random selection.Across relabelling budgets, the simulations compare the number of collected labels with the number of correctly labelled samples.Source: Label cleaning can be done in a… · p. 4, Figure 3 · p. 5
  • Similar selection performance persists at larger noise rates while average class-confusion matrices remain diagonally dominant.The study tests both uniform and class-dependent noise models.Source: Label cleaning can be done in a… · p. 5

Sample scoring function impacts cleaning performance.

The scoring function’s ambiguity term favors clear label-noise cases, while BALD can improve test accuracy comparably but yields less improvement in label quality. Noisy evaluation labels can also alter measured performance and model rankings, and prioritised relabelling can alleviate selection bias.

Sample scoring function impacts cleaning performance.

  • Prioritising clear label-noise cases yields more corrected labels, with the scoring function’s entropy term further strengthening this effect.On CIFAR10H, including self-entropy slightly improves the final outcome: 91.85 vs 91.02.Source: Sample scoring function impacts… · p. 5
  • BALD yields comparable test-accuracy gains but significantly less label-quality improvement because disagreement may not identify label noise.The comparison uses the same relabelling budget.Source: Sample scoring function impacts… · p. 5
  • Fig. 4 ranks CIFAR10H examples by priority percentile and displays true label distributions alongside predicted label-noisiness and ambiguity scores.The bottom 10% contains ambiguous examples that may require many re-annotations to reach consensus.Source: Figure 4 · p. 6
  • Fig. 5 contrasts a correctly flagged pneumonia-like noise case, a correctly labelled sample wrongly flagged by the model, and a subtle abnormality diagnosed with medium confidence.The wrongly flagged case involves confusion between lung nodules and pneumonia-like opacities.Source: Figure 5 · p. 6
  • Evaluation-label noise conditions strongly affect standard performance metrics and model rankings, supporting the need for clean validation labels in practical model selection.The experiments use symmetric and instance-dependent noise models with 40% average noise.Source: Sample scoring function impacts… · pp. 5–6
  • Relabelling 10% of the evaluation set with sample prioritisation can alleviate bias in model selection.The described procedure applies SSL-Linear using the corresponding training data.Source: Sample scoring function impacts… · p. 6

Discussion

Label noise affects predictive performance, evaluation, and model selection, while cost-effective relabelling strategies are benchmarked in a simulation framework. The discussion highlights the value of label cleaning and the limits imposed by noise patterns, assumptions, and the study’s imaging scope.

Discussion

  • The work assesses label noise across predictive performance, evaluation, and model selection, and benchmarks cost-effective relabelling strategies in a simulation framework.The framework quantifies potential resource savings and downstream-use improvements.Source: Discussion · p. 7
  • Noisy evaluation labels may produce misleading performance metrics and model rankings that do not generalise to the test environment, risking overoptimistic decisions in high-stakes applications.The discussion identifies healthcare as one such application.Source: Discussion · p. 7
  • Label-error patterns can affect cleaning and robust-learning efficacy as much as average noise rates, so comparisons should consider the underlying noise mechanisms.This pattern was observed on both training and validation sets.Source: Discussion · p. 7
  • For test-set cleaning, selectors should ideally be trained only on the test set and not used for classification or evaluation; empirical validation beyond imaging remains future work.Selector-model biases may be mitigated with an ensemble of models with different formulations and inductive biases.Source: Discussion · p. 7
  • Under high noise, robust-learning approaches may not fully recover predictive performance, while SSL pre-training outperforms noise-robust models trained from scratch.Acquiring new labels can complement noise-robust learning by recycling noisy samples and can also handle biased labels.Source: Discussion · p. 7
  • Consistent label errors may remain undetected; under extreme noise where incorrect labels outnumber correct ones on average, random selection can become preferable.The discussion assumes majority vote represents ground truth and notes that it may be suboptimal when annotators have different experience levels.Source: Discussion · p. 7

Methods

The methods construct noisy natural-image and chest-radiograph datasets with varying label difficulty, then use noise-robust and self-supervised models to prioritize samples for relabelling.

CIFAR10H dataset.

  • CIFAR10H uses the 10,000-image CIFAR10 test set and crowdsourced labels; temperature scaling controls sampled-label noise, while normalized target-distribution entropy defines difficult samples.Samples above an entropy threshold of 0.3 are classified as difficult; Fig. 6 compares class confusion matrices for all samples and difficult samples at τ = 2.Source: CIFAR10H dataset · pp. 7–8, Figure 6 · p. 8

NoisyCXR: a medical benchmark dataset for label cleaning.

  • NoisyCXR maps NIH labels to binary pneumonia-like opacity labels, using RSNA challenge labels as correct labels and retaining ambiguous cases through probabilistic relabelling distributions.The resulting dataset contains 26,684 images with a 12.6% noise rate; opacity-box confidence and reader delineations distinguish difficult or ambiguous cases from easy ones.Source: NoisyCXR: a medical benchmark dataset… · p. 8

Learning noise-robust image representations.

  • Noise-robust learning models use regularization or sample exclusion to reduce noisy-label effects; meta-model approaches are excluded because clean validation data may be unavailable.The methods support the sample-selection algorithms by providing scoring functions intended to be unbiased.Source: Learning noise-robust image… · pp. 8–9

Sample selection algorithms.

  • The selection algorithms estimate class posteriors with noise-robust models, including co-teaching and self-supervised learning, to rank samples for relabelling.Co-teaching uses two networks to select training examples for one another; BYOL learns image embeddings without labels, and a linear classifier on fixed embeddings is intended to add robustness to mislabels.Source: Sample selection algorithms · p. 9

Data availability

The paper uses public, anonymized datasets and reports its data sources and ethics review.

Data availability

  • CIFAR10H, NoisyCXR, and NIH ChestXray data are publicly available through the cited dataset sources and repositories.The paper also points readers to challenge annotations, NIH labels, processed experiment labels, and acquisition and anonymization information.Source: Data availability · p. 9
  • The authors state that they collected no data themselves and used public, widely used anonymized datasets.The NIH Clinical Center released the NIH images used in the RSNA challenge dataset.Source: Data availability · p. 9
  • The study was reviewed by Microsoft Research’s ethics review program and classified as non-human subjects research.The passage identifies the review reference numbers and says required U.S. HHS Office for Human Research Protections considerations were met.Source: Data availability · p. 9

Code availability

The authors provide an open-source repository for the study’s code and experiment materials.

Code availability

  • The open-source repository contains code for the label-cleaning benchmarks, robust-learning models, and experiments.It also includes configuration files and library requirements for reproducing the reported experiments.Source: Code availability · p. 9
  • The repository provides configuration files for the reported experiments.These files are included alongside the code and library requirements.Source: Code availability · p. 9
  • The repository lists library requirements to support reproducibility of the experiments.The authors state that the repository contains all configuration files and library requirements.Source: Code availability · p. 9

Competing Interests

The authors declare no competing interests.

Competing Interests

Supplementary information

The supplementary experiments probe how active label cleaning behaves under higher noise, different scoring choices, and alternative uncertainty estimates. They also describe the noise models and evaluation setup used in these experiments.

Classification performance of noise-robust models

  • The robustness analysis separates clean and mislabelled training cases and evaluates on clean held-out labels, using the accuracy gap between clean and mislabelled cases as a proxy for selector performance.The intended behavior is high accuracy on correctly labelled cases and accuracy near chance on mislabelled cases.Source: Supplementary information · p. 11

Experiments with larger noise rates

  • With class-dependent noise above 50%, ranking quality degrades when some classes lose diagonal dominance, and is expected to approach random sampling as bias and noise increase.At 20% total NoisyCXR noise, 43% of positive initial labels are incorrect; co-teaching selection degrades toward vanilla, while self-supervised pre-training improves cleaning by separating noisy and clean cases earlier.Source: Supplementary information · pp. 11–12

Further analysis on the scoring function

  • On CIFAR10H with 30% and 50% noise, including self-entropy makes the SSL selector prioritize clear label-noise cases over difficult ones first, yielding a slight mislabel-correction-rate improvement.The best-performing sample selectors also favor correcting clear label-noise cases first.Source: Supplementary information · p. 12

Model ensembles

  • 0.823 versus 0.818 AUC is reported for Ensemble-Ent versus Single on CIFAR10H, while averaged ensemble posteriors do not yield a drastic improvement.The experiments use five differently initialized models and five label-sampling repetitions; marginalizing entropy over model weights is reported to improve uncertainty estimates.Source: Supplementary information · pp. 12–13

Instance-dependent noise model

  • The instance-dependent noise simulation derives class-flip probabilities from pretrained ResNet-50 image embeddings rather than raw pixels, associating transition models with higher-level image semantics.The probabilities are softmax-normalized across classes and scaled to the desired noise rate.Source: Supplementary information · p. 13

Supplementary Note 2: Implementation Details

The supplementary implementation details specify model training and augmentation choices, then assess how selected choices affect label cleaning and representation quality.

Implementation details

  • The CIFAR10H and NoisyCXR implementations use ResNet-50 models, with supervised models trained using cross-entropy and dataset-specific training configurations.CIFAR10H uses SGD for supervised training and BYOL for self-supervision; NoisyCXR uses Adam for supervised training and BYOL with a ResNet-50 encoder for self-supervision.Source: Supplementary Note 2: Implementation… · pp. 13, 15

Scoring-function analysis

  • In CIFAR10H noisy-label-cleaning simulations, including the self-entropy term in the SSL selector slightly improved the final correct-label percentage (91.85 vs 91.02).The best-performing methods prioritized clear-label noise cases over those with greater labeling difficulty.Source: Supplementary Note 2: Implementation… · p. 14

Alternative label-quality measure

  • Total variation between the true label distribution and sampled-label distribution shows a similar method ranking to majority-label accuracy on CIFAR10H at 30% noise.The figure plots total variation against relabelling budget.Source: Supplementary Note 2: Implementation… · p. 14

Ensemble comparison

  • Marginalizing the entropy term over model weights in the ensemble is reported to make uncertainty estimates more reliable on CIFAR10H at 30% noise.The figure compares selectors based on a single SSL model and an ensemble.Source: Supplementary Note 2: Implementation… · p. 14

Medical-image self-supervised training

  • The NoisyCXR BYOL augmentation ablation reports ROC-AUC values of 0.853 without cropping and 0.871 with all augmentations.Cropping contributed most to embedding quality, followed by contrast and brightness, then rotation and shear; using the full augmentation set often sustained improvement during longer training.Source: Supplementary Note 2: Implementation… · p. 15

Related papers

Loading 2109.00574v2…