PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning with noisy labels: exploring techniques and remedies in medical image analysis

Davood Karimi, Haoran Dou, Simon K. Warfield, Ali Gholipour

arXiv:1912.02911v4cs.CVcs.LGeess.IVstat.ML

TL;DR

Medical image analysis relies on small, costly-to-label datasets in which annotation variability and other imperfections can produce noisy labels. This paper reviews label-noise methods, surveys medical-imaging applications, and evaluates existing and new remedies across three noisy datasets. The review and experiments support application-specific method selection and recommendations for handling different noise types.

  • Problem

    Label noise in medical image analysis has received insufficient attention despite small datasets, expert-dependent labeling, and substantial inter- and intra-observer variability.

  • Method

    The paper reviews label-noise methods in machine learning, deep learning, and medical image analysis, then evaluates existing and newly developed strategies on three noisy medical imaging datasets.

  • Results

    The review finds that many label-noise methods were developed outside medical imaging, while experiments show that remedies should be evaluated across different noise types and applications.

  • Takeaways & Limitations

    Researchers should analyze label-noise effects carefully and experiment with multiple methods to identify suitable approaches for each medical image analysis application.

Abstract

from arXiv · show

Supervised training of deep learning models requires large labeled datasets. There is a growing interest in obtaining such datasets for medical image analysis applications. However, the impact of label noise has not received sufficient attention. Recent studies have shown that label noise can significantly impact the performance of deep learning models in many machine learning and computer vision applications. This is especially concerning for medical applications, where datasets are typically small, labeling requires domain expertise and suffers from high inter- and intra-observer variability, and erroneous predictions may influence decisions that directly impact human health. In this paper, we first review the state-of-the-art in handling label noise in deep learning. Then, we review studies that have dealt with label noise in deep learning for medical image analysis. Our review shows that recent progress on handling label noise in deep learning has gone largely unnoticed by the medical image analysis community. To help achieve a better understanding of the extent of the problem and its potential remedies, we conducted experiments with three medical imaging datasets with different types of label noise, where we investigated several existing strategies and developed new methods to combat the negative effect of label noise. Based on the results of these experiments and our review of the literature, we have made recommendations on methods that can be used to alleviate the effects of different types of label noise on deep models trained for medical image analysis. We hope that this article helps the medical image analysis researchers and developers in choosing and devising new techniques that effectively handle label noise in deep learning.

I. INTRODUCTION

Medical image analysis needs large labeled datasets, but small datasets, costly expert annotation, observer variability, and label imperfections make noisy labels common. This paper reviews label-noise remedies, surveys medical-imaging studies, and evaluates existing and new strategies across three datasets.

  • Medical imaging datasets are often small, difficult to access, and expensive to label because annotation requires domain experts.
  • Inter-observer variability can require consensus labels, multiple experts, and specialized aggregation methods.
  • Historical data and crowdsourcing can provide labels, but may omit desired annotations, contain more noise, or have limited applicability to medical tasks.
  • The authors review classical and deep-learning methods, survey medical-imaging studies, and test existing and new remedies on three noisy medical imaging datasets.
  • The paper defines its scope around label noise in labeled medical images, excluding measurement noise, weak labels, semi-supervised learning, and multiple-instance learning.

II. LABEL NOISE IN CLASSICAL MACHINE LEARNING

Label noise is a long-standing problem whose effects depend on its distribution, the data, the model, and training procedure. Deep models can fit random labels, yet initially learn dominant patterns before potentially memorizing incorrect labels.

  • Label noise can affect machine-learning performance more strongly than measurement or feature noise and may be class-independent, class-dependent, or class-and-feature-dependent.
  • Deep learning generally requires more data, which are often labeled by non-experts or automated systems and therefore may contain higher noise levels.
  • CNNs trained with standard stochastic gradient descent can fit very large datasets with completely random labels, demonstrating substantial memorization capacity.
  • On mostly correctly labeled data, deep models initially learn dominant patterns rather than memorizing individual samples.
  • The learning–memorization trade-off depends on data richness, noise amount, architecture, and training procedures, while its theory remains ongoing.

A. Label cleaning and pre-processing

Label-cleaning methods identify, correct, discard, reweight, or relabel samples likely to have incorrect annotations. Approaches use similarity, confidence, uncertainty, smoothed labels, generative models, or iterative filtering.

  • Label-cleaning methods identify and either fix or discard likely incorrect training samples before training or iteratively during model training.
  • CleanNet compares query-image features with class-representative features from a small clean dataset to assess label correctness or assign sample weights.
  • An iterative extension removes the clean-dataset requirement and uses multiple class prototypes; both studies improved classification accuracy by 1-5% depending on dataset and noise level.
  • Rank Pruning retains confident samples, while uncertainty-based filtering estimates prediction uncertainty and relabels samples likely to be incorrect.
  • Deep label distribution learning smooths noisy labels and trains models by minimizing KL divergence, while generative models and GANs can denoise noisy or synthetic labels.

B. Network architecture

Architecture- and loss-based methods incorporate label-noise modeling into training or modify objectives to reduce noise sensitivity. Reviewed approaches include noise layers, graphical models, robust losses, abstention, trimming, and transition-matrix corrections.

  • B. Network architecture: Noise layers model the transition between noisy and true labels and learn noise statistics alongside network weights.
  • B. Network architecture: Multiple-annotator models jointly estimate correct labels and annotator confusion matrices during CNN training, with confusion-matrix trace penalties resolving ambiguity.
  • B. Network architecture: Graphical-model approaches represent latent true labels, noise types, or label relationships and can incorporate domain-specific information.
  • C. Loss functions: Mean absolute error treats samples more equally than cross-entropy, but stochastic optimization can down-weight difficult correct samples and reduce accuracy while increasing training time.
  • C. Loss functions: Other loss methods enable abstention, ignore largest-loss samples, or correct losses and predictions using an estimated label-confusion transition matrix.

D. Data re-weighting

Data re-weighting methods reduce the influence of likely incorrect labels, while broader training strategies use curriculum learning, distillation, co-training, label updates, and interpolation to improve robustness.

  • D. Data re-weighting: Clean validation data can determine weights for noisy training samples by minimizing validation loss.Meta-learning assigns lower influence to samples when doing so improves performance on clean data.
  • D. Data re-weighting: Removing high-loss samples relies on the assumption that mislabeled examples increasingly produce unusually large losses during training.Empirical results support down-weighting large-loss samples for noisy labels, unlike the weighting pattern learned for class imbalance.
  • D. Data re-weighting: Feature clustering, auxiliary clean images, curriculum learning, and knowledge distillation provide alternative ways to reduce noisy-label effects.Reported examples include 3–20% CIFAR-10 accuracy gains from feature-based ensembles and up to 20% accuracy improvement from Mentor-Net on CIFAR-100.
  • D. Data re-weighting: Co-training methods update models using disagreement or selected examples, but co-teaching becomes less effective as label noise increases.Cross-validation-based selection was reported as more effective than co-teaching in the cited experiments.
  • D. Data re-weighting: Mixup reduced CIFAR-10 classification error by 6.5-12.5% compared with basic empirical risk minimization across different noise levels.The method interpolates labels and data points using a beta-distributed λ.
  • D. Data re-weighting: Alternating optimization can estimate latent object boundaries while training the segmentation model, improving F-measure over baseline CNNs.Reported improvements were 2-5% in one study and 1-17% in another, under different boundary-modeling assumptions.

IV. DEEP LEARNING WITH NOISY LABELS IN MEDICAL

This section reviews studies addressing label noise in deep learning for medical image analysis using the categorization introduced earlier.

  • IV. DEEP LEARNING WITH NOISY LABELS IN MEDICAL: Medical image analysis studies addressing label noise are organized using the same categorization as the preceding general deep-learning review.
  • IV. DEEP LEARNING WITH NOISY LABELS IN MEDICAL: The section focuses on training deep learning models for medical image analysis in the presence of label noise.
  • IV. DEEP LEARNING WITH NOISY LABELS IN MEDICAL: The reviewed medical applications are presented as studies that have addressed label noise during deep-model training.

A. Label cleaning and pre-processing

Medical image studies use label smoothing, noise modeling, uncertainty-aware training, re-weighting, and iterative pseudo-label refinement to address imperfect annotations.

  • A. Label cleaning and pre-processing: Label smoothing improved thoracic-disease classification by up to 0.08 AUC compared with simple approaches such as ignoring noisy samples.
  • A. Label cleaning and pre-processing: A noise layer slightly improved breast-lesion detection accuracy when applied to mammogram training.
  • A. Label cleaning and pre-processing: Dilated center annotations produced uncertain ring-shaped regions that were ignored when training a virus-particle segmentation model.
  • A. Label cleaning and pre-processing: Clean-label-derived weights improved pancreatic-cancer classification accuracy by 10% over training on clean and noisy patches without re-weighting.
  • A. Label cleaning and pre-processing: Teacher-student training uses uncertainty maps to constrain consistency on generated labels while preserving correct predictions on clean data.
  • A. Label cleaning and pre-processing: Confidence maps supplied additional MRI segmentation training data and yielded approximately 3% Dice similarity improvements.
  • A. Label cleaning and pre-processing: Iterative pseudo-label updates improved cystic-lesion segmentation accuracy over initial K-means segmentations.

V. EXPERIMENTS

The experiments examine label noise across detection, classification, and segmentation using three medical imaging datasets, including a TSC lesion dataset with systematically missed lesions.

  • V. EXPERIMENTS: Three medical imaging datasets represent detection, classification, and segmentation problems with three different types of label noise.
  • V. EXPERIMENTS: The TSC dataset contains 165 MRI scans from 88 subjects, with 12 accurately reannotated scans reserved for evaluation.
  • V. EXPERIMENTS: Approximately 18% of lesions were missed during first reading, especially lesions that were smaller, less dark on T1, or farther from another lesion.
  • V. EXPERIMENTS: The lesion experiment uses a 3D U-Net-like CNN for joint detection and segmentation.
  • V. EXPERIMENTS: The evaluated strategies include noisy-label and clean-data Faster-RCNN and 3D U-Net models, MAE and iMAE losses, and DSC loss.
  • V. EXPERIMENTS: Data re-weighting ignores samples whose losses exceed 1.5 standard deviations above the recent mean.
  • V. EXPERIMENTS: Iterative label cleaning adds CNN-detected lesions classified as true lesions by a random forest using six lesion and uncertainty features.

3) Results:

For brain-lesion detection, label-noise remedies affected performance differently across architectures, with iterative label cleaning producing the strongest reported gains. A small clean dataset could match a larger noisy dataset, although more clean scans might improve results.

  • Data re-weighting substantially improved lesion-count F1 score and DSC over the noisy-label baseline CNN.
  • Iterative label cleaning achieved much higher lesion-count F1 score and DSC than the baseline and outperformed data re-weighting.
  • MAE and iMAE produced lower lesion-count F1 score and DSC than the DSC-loss baseline for 3D U-Net.
  • MAE improved Faster-RCNN lesion-count F1 score by 0.041 over its cross-entropy baseline.
  • A small dataset with clean labels can be as good as a large dataset with noisy labels, while the clean dataset contained only 12 scans.

B. Prostate cancer digital pathology classification

The prostate classification experiment used six independently labeled pathologists’ annotations and evaluated methods for handling their disagreement. Ground truth was estimated with STAPLE, with classification assessed for cancer status, cancer grade, and large errors.

  • Data and labels: Six pathologists independently provided pixel-wise labels for benign tissue and Gleason grades 3, 4, and 5.Their Cohen’s kappa was approximately 0.40–0.60, indicating inter-observer variability in this task.
  • Model: The experiment used a MobileNet CNN with 768 × 768-pixel patches at 40X magnification.MobileNets use separable convolutional filters combining depth-wise and point-wise operations.
  • Evaluation: STAPLE combined six pathologists’ labels to estimate test-set ground truth because of high inter-observer variability.Five-fold cross-validation trained on 80% of cores and evaluated on the remaining 20%.
  • Metrics: Performance was measured using accuracy, AUC, and the percentage of large classification errors.Tasks distinguished cancerous from benign tissue and high-grade from low-grade cancer.
  • Compared methods: The study compared single-pathologist labels, majority vote, STAPLE, STAPLE with iMAE, minimum-loss labels, and annotator-confusion estimation.The minimum-loss method selected, for each training patch, the pathologist label producing the smallest loss.

3) Results:

Across the experiments, combining or modeling annotator information and down-weighting suspicious labels improved prostate classification relative to single-expert or baseline training. In lesion detection, data re-weighting and iterative label cleaning improved noisy-label CNNs, with iterative cleaning best overall.

  • Prostate classification: Using labels from multiple experts significantly improved prostate classification over labels from one pathologist.iMAE and the minimum-loss label method also improved classification accuracy.
  • Brain-lesion detection: For brain-lesion detection, the best DSC and lesion-count F1 score came from 3D U-Net with iterative label cleaning.Data re-weighting and iterative label cleaning substantially improved CNN performance with noisy labels.
  • Prostate classification: iMAE performed better for cancerous-versus-benign classification, whereas minimum-loss labels performed better for high-grade-versus-low-grade cancer.The authors relate this pattern to stronger inter-pathologist disagreement for high-grade versus low-grade annotation.
  • Prostate classification: Annotator confusion estimation achieved the best high-grade-versus-low-grade accuracy and close to the best cancerous-versus-benign accuracy.It also displayed the lowest reported large-error rate in the cited results passage.
  • Evaluation: With separate pathologists for training and testing, iMAE or annotator-accuracy modeling outperformed cross-entropy and single-expert training.Results were worse than using all six pathologists for training and test-truth estimation, especially for high-grade versus low-grade cancer.

1) Data and labels:

The experiments covered noisy labels in prostate pathology and fetal-brain segmentation, using expert annotations, synthetic or approximate segmentations, and CNN baselines. A dual-CNN iterative update method was introduced for fetal-brain segmentation.

  • Data and labels: The fetal-brain experiment used 2,562 diffusion-weighted MR images from 65 fetuses.One image per fetus was manually segmented by two experienced annotators for clean evaluation data; 2,497 remaining images received approximate segmentations.
  • Data and labels: Figure 4 organizes annotator confusion matrices with estimated true labels in rows and annotator labels in columns.The class order is benign followed by Gleason grades 3–5.
  • Model: The segmentation baseline used a CNN similar to 3D U-Net with four convolutional blocks in each contracting and expanding part.The first block extracted 10 feature maps, which doubled in subsequent blocks.
  • Method: The proposed dual-CNN method trained two diverse networks and iteratively replaced noisy masks when their predictions produced lower loss.If both predictions improved the loss, their average replaced the noisy segmentation mask.

3) Results:

Strong label noise substantially degrades baseline CNN segmentation, while MAE loss and Dual CNNs with iterative label updates reduce large errors and modestly improve worst-case performance.

  • Strong label noise significantly affects baseline CNN performance, with test DSC trailing the DSC of the training labels.
  • MAE loss and Dual CNNs with iterative label update reduce large segmentation errors under strong label noise, as measured by HD95.
  • Improvement in 5% DSC indicates some gain in worst-case performance, although outlier and difficult-sample effects remain distinct from label noise.

VI. DISCUSSION AND CONCLUSIONS

The paper finds that label-noise remedies must be matched to the noise type, annotation setting, dataset size, and application. Across reviews and experiments, modeling or correcting annotation errors often improved performance, while some strategies were less suitable for medical imaging.

  • Scope and motivation: The study reviewed label-noise methods in machine learning and medical image analysis, then evaluated existing and new remedies across detection, segmentation, and classification datasets.The authors emphasize that many general-purpose methods have not been widely employed in medical imaging.
  • Recommendations and limitations: The authors recommend selective label cleaning, architecture changes that estimate annotation-error statistics, noise-robust losses, and data re-weighting, while warning that feature-space consistency methods may suffer from the curse of dimensionality.Large medical images and relatively few samples make similarity-based methods less applicable; confidence-based cleaning also depends on calibration.
  • Method comparisons: Dual CNNs with iterative label updates were effective for fetal brain segmentation at small and medium noise levels, while MAE loss performed better at high noise levels.This result is reported in the comparison of methods across different label-noise levels.
  • Dataset size and strategy choice: Dataset size changed which noise-handling strategies worked: correcting missing annotations helped brain lesion segmentation, whereas ignoring high-loss labels helped pathology classification but failed for fetal brain segmentation.On pathology data, modeling annotator confusion performed better than ignoring high-loss labels.
  • Annotation-error modeling: Modeling annotation error improved performance in brain lesion segmentation and digital pathology, including gains over separate label-fusion procedures.The experiments used prediction uncertainty to fill missing annotations and estimated individual-expert error during CNN training.
  • Computer-generated labels: Computer-generated noisy fetal-brain annotations can outperform training on 65 manually segmented images, but very inaccurate annotations produced poor training.Approximate annotations may be available at little or no cost from other images, matched subjects, or atlases.
Loading 1912.02911v4…