PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion

Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, Nathan Silberman

arXiv:1902.03680v3cs.LGcs.CVstat.ML

TL;DR

Noisy annotator labels can impair supervised learning, particularly when disagreement is high and annotation is costly. The paper jointly estimates each annotator’s confusion matrix and the true-label distribution using a regularized loss, and reports competitive or superior performance while estimating noise from singly labeled examples.

  • Problem

    Noisy labels reflect annotators’ varying skills and biases, limiting predictive performance when treated as ground truth, especially in costly, variable medical-imaging annotation.

  • Method

    The method jointly estimates annotator confusion matrices and the true-label distribution by adding a regularization term to cross-entropy loss.

  • Results

    Experiments on synthetic and real image-classification data show better or comparable performance with EM-based methods and confusion-matrix recovery with one label per example.

  • Takeaways & Limitations

    Individual annotator modeling improves robustness to label noise while enabling estimation of annotator skills and annotation noise from sparse observations.

  • Takeaways & Limitations

    The method assumes one ground truth per input and image-independent annotator noise, limiting settings with ambiguous images or difficulty-dependent disagreement.

Abstract

from arXiv · show

The predictive performance of supervised learning algorithms depends on the quality of labels. In a typical label collection process, multiple annotators provide subjective noisy estimates of the "truth" under the influence of their varying skill-levels and biases. Blindly treating these noisy labels as the ground truth limits the accuracy of learning algorithms in the presence of strong disagreement. This problem is critical for applications in domains such as medical imaging where both the annotation cost and inter-observer variability are high. In this work, we present a method for simultaneously learning the individual annotator model and the underlying true label distribution, using only noisy observations. Each annotator is modeled by a confusion matrix that is jointly estimated along with the classifier predictions. We propose to add a regularization term to the loss function that encourages convergence to the true annotator confusion matrix. We provide a theoretical argument as to how the regularization is essential to our approach both for the case of single annotator and multiple annotators. Despite the simplicity of the idea, experiments on image classification tasks with both simulated and real labels show that our method either outperforms or performs on par with the state-of-the-art methods and is capable of estimating the skills of annotators even with a single label available per image.

1. Introduction

Noisy labels from annotators with differing skills and biases can limit predictive performance, especially when disagreement is substantial. The paper proposes regularized joint estimation of annotator confusion and true-label distributions, achieving strong performance with sparse annotations.

  • Motivation: Annotator disagreement and label noise can limit supervised-learning performance when noisy labels are treated as ground truth.This issue is especially relevant to medical imaging, where expert annotation is costly and inter-reader variability is high.
  • Motivation: Modeling annotator skills can support cleaner training labels and inform future decisions about which annotators label which examples.The paper links annotator modeling to both predictive accuracy and future label quality.
  • Prior work: Existing approaches either aggregate labels before classifier training or jointly model predictions and annotators, with simultaneous methods often requiring many labels per example.This label-redundancy requirement can be prohibitive in real-world applications.
  • Contribution: The proposed method adds trace regularization to cross-entropy, encouraging estimated annotators to be maximally unreliable while preserving fidelity to the data.A theoretical result states that the annotation noise can be recovered when the average annotator confusion matrix is diagonally dominant.
  • Results: Experiments on simulated MNIST and CIFAR10 labels and real cardiac-view labels show better or comparable performance with EM-based methods, including confusion-matrix recovery from one label per example.The experiments also compare against modern noise-robust methods under high inter-annotator variability.

2. Methods

The method jointly estimates true label distributions and annotator confusion matrices from noisy labels by combining cross-entropy fitting with trace regularization. Under diagonal-dominance conditions, minimizing the trace identifies the true annotator confusion matrices and separates annotation noise from the underlying labels.

  • Optimization: Joint optimization minimizes the sum of cross-entropy losses between estimated annotator distributions and their observed noisy labels.The parameters include the classifier and all annotator confusion matrices.
  • Model architecture: The model combines a base classifier for ground-truth class probabilities with one confusion-matrix estimator per annotator.Each annotator distribution is computed as A^(r)pθ(x), and the classifier output is used for final prediction.
  • Optimization: Cross-entropy alone cannot separate annotation noise from the true label distribution because infinitely many classifier–confusion-matrix combinations can match annotator label distributions.The ambiguity is represented by alternative factorizations of the observed annotator distributions.
  • Trace regularization: The proposed trace regularizer minimizes the trace of estimated confusion matrices while preserving data fit, encouraging annotator models to be maximally unreliable.The combined loss also supports missing-label settings through a sum over annotators who labeled each image.
  • Theoretical justification: With diagonally dominant true and estimated confusion matrices, minimal trace uniquely recovers the true annotator matrix for a single annotator.The proof establishes elementwise lower bounds on the estimated diagonal and equality only at the true matrix.
  • Theoretical justification: For multiple annotators, minimizing the mean trace uniquely matches each estimated confusion matrix to its true value when the mean confusion matrix is diagonally dominant.The method initializes estimated confusion matrices with identity matrices to encourage diagonal dominance.

3. Experiments

Experiments on MNIST, CIFAR-10, and cardiac-view ultrasound test the method against EM-based and noise-robust baselines under dense and sparse noisy-label settings. The method generally matches or outperforms baselines, while CM recovery and accuracy degrade when diagonal dominance fails.

  • Comparing with EM-based Approaches: MNIST comparisons evaluate classification accuracy and CM estimation with either all five annotator labels or one label per image.The benchmarks compare the proposed method with generalized EM and MBEM across mean skill levels.
  • Comparing with EM-based Approaches: With dense MNIST labels, the method consistently performs better or comparably to generalized EM and MBEM on classification accuracy and CM estimation.With one label per image, the same trend holds when the mean confusion matrices are diagonally dominant.
  • Comparing with EM-based Approaches: When diagonal dominance fails, all models suffer a steep accuracy drop because they cannot estimate annotator confusion matrices accurately.With one label per image, generalized EM predicts identity matrices and fails to recover the CMs.
  • Comparing with EM-based Approaches: On CIFAR-10 with dense labels, the method outperforms MBEM and generalized EM by a large margin in both classification accuracy and CM estimation.Removing the trace norm still yields reasonable performance, but adding it improves results, especially under larger noise on MNIST.
  • Value of Modelling Individual Annotators: Against methods that do not model individual annotators, the method is better or comparable when diagonal dominance holds, with gains pronounced for low skill and sparse labels.The baselines undergo a steep accuracy decline with one label per image, while a vanilla CNN with L2 weight decay overfits under high noise.
  • Value of Modelling Individual Annotators: On cardiac-view classification, estimated skills separate experts from two non-experts, capture characteristic mistakes, and outperform MBEM in accuracy and CM estimation.The task uses ultrasound images from six cardiac-view classes and includes labels from two nonexpert users.

4. Discussion and Conclusion

The method jointly recovers annotator label noise and the ground-truth distribution with a simple regularized objective. Its scope is limited by assumptions about class count, label ambiguity, and input-independent annotator noise.

  • The algorithm simultaneously recovers multiple annotators’ label noise and the ground-truth label distribution.
  • Adding a regularization term keeps the method simple while supporting classification and confusion-matrix estimation.
  • Experiments on synthetic and real datasets show superior performance over common EM-based methods for classification accuracy and confusion-matrix estimation.
  • The method can estimate annotation noise even when each image has only one label.
  • The work targets mainly medical-imaging problems with fewer than 10 classes and assumes one ground truth per input.
  • Future extensions should address ambiguous images, massively multi-class settings, and input-dependent annotator disagreement.

A. Data sets, training and architectures

The experiments use MNIST, CIFAR-10, and cardiac-view classification datasets, with a shared optimization protocol and CNN-based architectures. Training uses Adam for 200 epochs, while the proposed method applies trace regularization.

  • Data sets: Experiments use MNIST, CIFAR-10, and a cardiac view classification dataset from a handheld ultrasound probe.
  • Training: All experiments optimize parameters with Adam using an initial learning rate of 10^-3, minibatches of 50, and 200 epochs.
  • Training: The proposed method uses trace regularization with scale λ = 0.01.
  • Architectures: The MNIST classifier is a four-convolution-layer CNN with 3 × 3 kernels, ReLU activations, max-pooling, and global average pooling.
  • Architectures: The same CNN architecture is used for the cardiac-view dataset with increased input dimensions.

B. Confusion matrices of pairwise-flippers and hammer-spammers

MNIST simulations model two annotator types—pairwise-flippers and hammer-spammers—using groups of five annotators whose confusion matrices vary around a mean skill level.

  • Annotator types: The simulations consider pairwise-flipper and hammer-spammer annotator types.
  • Annotator groups: For each annotator type and skill level p, the experiments create groups of five annotators.
  • Confusion-matrix generation: Each confusion matrix is generated by perturbing the mean skill level p with Gaussian noise ε ∼ Normal(0, 0.01).
  • Confusion-matrix generation: Figure 8 shows example annotator groups whose diagonal entries are fixed for each annotator and drawn from Normal(p, 10^-2).

C. Additional experiments on MNIST

Additional MNIST experiments train models on noisy labels from groups of five hammer-spammers across a range of mean skill levels. The proposed method consistently matches or exceeds the baseline accuracies.

  • Experimental setup: The MNIST experiments use noisy labels from groups of five hammer-spammers across a range of mean skill levels p.
  • Results: The method consistently achieves comparable or better accuracy than the evaluated baselines.

D. Ablation study on trace regularization on MNIST

On MNIST, trace-norm regularization generally improves both classification accuracy and confusion-matrix estimation, especially when annotators are noisier. The study compares this ablation across annotator groups and labeling regimes.

  • Trace-norm regularization generally improves classification accuracy and confusion-matrix estimation.The comparison removes the trace norm from the proposed method.
  • The improvement is more pronounced at lower annotator skill levels, where label noise is larger.
  • Figure 9 compares the proposed method with generalized EM and MBEM under hammer-spammer noise, measuring both classification accuracy and confusion-matrix recovery error.It includes settings with all annotators labeling each example and with only one label per example.
  • Figure 10 evaluates accuracy as mean annotator skill varies when examples receive either all annotator labels or one randomly selected label.
  • Figure 11 compares the proposed method with and without trace norm for hammer-spammer and pairwise-flipper annotator groups.

E. Pseudo-codes of our method, generalized EM and MBEM

The pseudo-codes contrast the proposed joint optimization procedure with generalized EM and MBEM for learning classifier predictions and annotator confusion matrices. They also accommodate examples with missing annotator labels.

  • The pseudo-codes compare the proposed method, generalized EM, and MBEM for jointly learning true label distributions and annotator confusion matrices.
  • When examples are not labeled by every annotator, missing labels are filled with a pseudo class for notational convenience.
  • The comparison emphasizes the proposed method’s implementational simplicity despite comparable or superior performance on all three datasets.
  • The proposed method initializes confusion matrices near identity and jointly learns them with classifier parameters using minibatch SGD on a combined loss.The combined loss includes a trace-regularizer scale λ.
  • Generalized EM and MBEM iterate initialization, classifier updates, posterior estimation, and confusion-matrix estimation.

F. TensorFlow codes

The TensorFlow implementation jointly optimizes classifier parameters and annotator confusion matrices through a loss combining weighted cross-entropy with trace-norm regularization.

  • The implementation takes inputs and annotator labels, while a classifier estimates the true label distribution.
  • Annotator confusion matrices are included as model parameters in the joint optimization.
  • The final loss combines weighted cross-entropy with a scaled trace norm.The implementation identifies this as the probabilistic model’s proposed loss.

F.2. Defining confusion matrices of annotators

Each annotator is represented by a row-normalized confusion matrix whose entries give the probability of assigning an observed label conditional on the true label.

  • The implementation defines one confusion matrix for each annotator.
  • The (i,j) entry is P(label_annotator_a = j | label_true = i), the probability of assigning label j when the true label is i.
  • The matrices are initialized close to identity matrices before learning.
  • Confusion-matrix elements are constrained to be positive and each row is normalized to sum to one.

F.3. Cross-entropy loss with sparse and noisy labels

The section implements cross-entropy losses for noisy labels from one or multiple annotators by transforming classifier predictions through annotator confusion matrices. It supports sparse and missing labels while averaging losses across annotators and image examples.

  • Multi-annotator loss: The multi-annotator loss compares noisy labels with annotator-specific confusion-matrix models.The labels tensor represents multiple annotators, and missing labels are encoded as zero vectors.
  • Loss computation: The implementation computes cross-entropy separately for available annotator labels and stacks the resulting losses with shape (batch, num_annotators).Annotator networks without labels are filtered so they are trained only when labels are available.
  • Cross-entropy calculation: The loss uses one-hot label representations, clipped predicted probabilities, and summed negative log-likelihood.The single-annotator function returns a tensor of size [batch_size].
  • Confusion-matrix mapping: For each annotator, the classifier’s predicted label distribution is mapped into annotator-label space by multiplying it by that annotator’s confusion matrix.The confusion matrix entry gives P(label_annotator_a = j|label_true = i).
  • Missing labels: Missing labels are represented as zero vectors and incorporated through an availability mask when calculating losses across annotators.The mask multiplies the per-annotator losses before aggregation.
Loading 1902.03680v3…