PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Said What: Modeling Individual Labelers Improves Classification

Melody Y. Guan, Varun Gulshan, Andrew M. Dai, Geoffrey E. Hinton

arXiv:1703.08774v2cs.LGcs.CV

TL;DR

Existing aggregation methods use consensus or label distributions but ignore which expert supplied each label. This paper models experts individually and learns weights for combining their predictions, improving diabetic-retinopathy classification over baseline and competing methods.

  • Problem

    Consensus and label-distribution approaches discard potentially valuable information about expert identity, reliability, and specialized strengths.

  • Method

    A shared Inception-v3-based network models the opinions of individual doctors, then learns averaging weights for combining their predictions.

  • Results

    Five-class test error decreases from 23.83% to 20.58% versus the average-doctor baseline, while WDN also outperforms DN and the cited competing methods.

  • Takeaways & Limitations

    Using individual annotator identities can improve training from multiple noisy labels for diabetic-retinopathy diagnosis.

  • Takeaways & Limitations

    The label-noise model assumes an asymmetric binary noise distribution shared across pixels, and the evaluation excludes doctors who graded test images.

Abstract

from arXiv · show

Data are often labeled by many different experts with each expert only labeling a small fraction of the data and each data point being labeled by several experts. This reduces the workload on individual experts and also gives a better estimate of the unobserved ground truth. When experts disagree, the standard approaches are to treat the majority opinion as the correct label or to model the correct label as a distribution. These approaches, however, do not make any use of potentially valuable information about which expert produced which label. To make use of this extra information, we propose modeling the experts individually and then learning averaging weights for combining them, possibly in sample-specific ways. This allows us to give more weight to more reliable experts and take advantage of the unique strengths of individual experts at classifying certain types of data. Here we show that our approach leads to improvements in computer-aided diagnosis of diabetic retinopathy. We also show that our method performs better than competing algorithms by Welinder and Perona (2010), and by Mnih and Hinton (2012). Our work offers an innovative approach for dealing with the myriad real-world settings that use expert opinions to define labels for training.

Introduction

Multiple experts often disagree on medical-image labels, yet conventional aggregation discards which expert produced each label. The paper investigates modeling individual expert opinions to improve diabetic-retinopathy classification.

  • Motivation: Expert agreement on labels can be poor, challenging the assumption that training-label quality bounds neural-network performance.Reported agreement was 55.4% initially and 70.7% after some time.
  • Motivation: The paper argues that expert opinions contain information beyond consensus labels or label distributions.That information includes differences among experts in skill and area of expertise.
  • Beating the Teacher: A neural network can outperform unreliable teachers when label errors are random, as demonstrated using corrupted-label MNIST experiments.The paper relates this result to estimating how many noisily labeled examples equal one correctly labeled example via mutual information.
  • Application: The application is five-class diabetic-retinopathy grading from retinal fundus images, spanning No DR through Proliferative DR.The paper notes that its findings should also apply to diabetic macular-edema diagnosis, although that diagnosis is not studied here.
  • Approach: The proposed direction predicts individual doctors’ opinions while retaining expert-to-label assignments instead of collapsing multiple opinions into one.Evaluation uses an adjudicated reference standard created by three retinal specialists.

Related Works

Prior research estimates ground truth or annotator reliability through probabilistic, EM, message-passing, and related approaches. The paper distinguishes its contribution by modeling individual labelers directly.

  • Research areas: The broader literature connects multiple-annotator learning with noisy labels, crowdsourcing, weak supervision, semi-supervised learning, item response theory, and multi-view learning.
  • Ground-truth estimation: EM-based methods estimate annotator accuracies or infer accurate labels from multiple expert opinions.The cited foundational work includes Dawid and Skene and Smyth et al.
  • Probabilistic and crowdsourcing methods: Bayesian models, active-learning methods, and message-passing algorithms address image generation, annotation, annotator querying, sample selection, and crowdsourcing.
  • Extensions: Weak-supervision research generates labeled training sets programmatically, while crowdsourcing research also studies budget allocation during label sourcing.
  • Positioning: The paper states that prior biostatistics and epidemiology methods estimate ground truth without ground-truth data but do not model individual labelers as it does.

Methods

The paper models individual doctors with shared neural-network features and doctor-specific outputs, then learns weights to combine their predictions. It compares this approach with averaging-based and label-noise baselines for diabetic-retinopathy classification.

  • Model design: The proposed models retain which doctor supplied each label instead of collapsing doctor opinions into a single average.This motivation exploits differences in doctor reliability, expertise, and error patterns.
  • Model design: The model sequence progresses from Baseline Net (BN) to Doctor Net (DN), Weighted Doctor Net (WDN), Image-specific WDN (IWDN), and Bottlenecked IWDN (BIWDN).DN models 31 doctors with shared Inception-v3 weights and separate softmax outputs; later variants learn fixed or image-dependent averaging weights.
  • Training procedure: Averaging logits are trained against the opinions of doctors who labeled each image, while predictions from other doctor models are combined using the current weights.The parameters are updated with cross entropy between the target distribution and weighted-average prediction.
  • Training procedure: WDN trains doctor models first, freezes them, and then learns positive averaging weights through softmax-transformed averaging logits.Evaluation uses a linear combination of doctor-model predictions with the learned weights.
  • Baselines: Welinder and Perona (2010) is evaluated by using its updated labels to train the paper’s baseline network.Their method estimates annotator abilities and likely labels with representative online EM.
  • Baselines: The Mnih and Hinton (2012) baseline models an unobserved true label followed by corrupted observed labels, using an asymmetric binary noise distribution.The paper extends this method to multiclass labels and estimates its noise parameters from doctor-versus-average-opinion confusion matrices.

Experimental Setup

The experiments use Inception-v3-based diabetic-retinopathy models, multiple doctor-labeling setups, and evaluation procedures designed to handle binary and five-class outcomes and differing class distributions.

  • Tasks and metrics: The study trains a binary-output BN variant by thresholding five-class predictions at Moderate NPDR or above and validates it using area under the ROC curve.The threshold corresponds to a commonly used clinical definition of a referable eye condition.
  • Evaluation: Evaluation uses log prior correction, adding class-specific log ratios between evaluation and training label proportions to prediction logits.The assumed test distribution is the mean distribution of known training and validation images, and all reported results apply this correction.
  • Datasets: The training dataset contains 126,522 images, while validation and test sets contain 7,804 and 3,547 images, respectively.Training images come from four clinical partners; validation images come from EyePACS clinics, and test images come from EyePACS-1 and Messidor-2.
  • Expert labels: At least one of 54 ophthalmologists or ophthalmology trainees graded each training and validation image, with 30 sufficiently prolific ophthalmologists modeled individually.The remaining doctors are combined into a composite doctor to avoid doctor-specific parameters supported by fewer than 1,000 training cases.
  • Image processing: The models use 587×587 preprocessed fundus images with random brightness, saturation, hue, and contrast perturbations.Preprocessing detects the circular fundus disk and removes black borders.
  • Baseline design: The paper’s baseline comparisons use its own BN because it differs from Gulshan et al.’s setup in doctor exclusion, image resolution, preprocessing, and theoretical techniques.Doctors who graded test images are removed from training and validation, which excludes some of the most expert doctors.

Summary of Results

Modeling doctors individually improves diabetic-retinopathy classification over modeling averaged opinions, while learned averaging weights provide a further improvement; five-class training also benefits binary evaluation.

  • Loss-function comparison: Five-class loss raises BN test binary AUC by 1.53%, from 95.58% to 97.11%, compared with binary loss.All subsequent reported models use five-class loss.
  • Doctor modeling: DN reduces five-class test error by 1.97%, from 23.83% to 21.86%, compared with BN.DN models individual doctors rather than only the averaged doctor opinion.
  • Doctor modeling: DN improves binary AUC by 0.17%, binary error by 0.17%, and spec@97%sens by 2.21% over BN.The reported values are 97.28% versus 97.11%, 9.75% versus 9.92%, and 81.81% versus 79.60%, respectively.
  • Comparisons: DN is significantly better on all test metrics than BN trained with labels from Welinder and Perona (2010).The competing labels reduce five-class error by only 0.09% and worsen binary AUC and binary error relative to vanilla BN.
  • Learned weighting: WDN further decreases five-class test error by 1.28% relative to DN.WDN also increases binary AUC by 0.17%, decreases binary classification error by 0.68%, and increases spec@97%sens by 0.88%.
  • Comparisons: Results from IWDN and BIWDN are slightly worse than WDN, while the Mnih-and-Hinton extension worsens DN five-class test error by 0.90%.The Mnih-and-Hinton extension is also more computationally costly than (B)(I)WDN.

Conclusion

The paper models individual noisy annotators with shared neural-network outputs and learns averaging weights to combine their predictions, improving diabetic-retinopathy classification over averaged doctor opinions.

  • Conclusion: The method models multiple noisy annotators individually with a shared neural network that has separate outputs for each expert.It then learns averaging weights for combining the modeled predictions.
  • Conclusion: Compared with training on average doctor opinion, the method lowers five-class classification test error from 23.83% to 20.58%.The baseline is described as a strategy that had achieved state-of-the-art automated diabetic-retinopathy diagnosis results.
  • Conclusion: Training with five-class loss significantly outperforms binary loss on binary metrics in the published baseline comparison.The paper also reports superior performance to corresponding competing methods by Welinder and Perona (2010) and Mnih and Hinton (2012).
  • Conclusion: The authors present the methodology as generally applicable to supervised systems using datasets whose labels come from multiple annotators.The stated setting is one where each example is labeled by a subset of a larger expert pool.

A. Mutual Information for Noisy Labels

The paper uses mutual information to estimate how many noisy labels provide information comparable to correctly labeled examples. In the MNIST example, 60,000 noisy labels were empirically equivalent to about 1,000 clean labels.

  • Mutual information estimates the number of noisily labeled training cases equivalent to one correctly labeled case.The method assumes random noise and compares mutual information per noisy case with that of a perfect label.
  • For ten classes, perfect labels have mutual information Iperfect = 2.3 nats, while labels that are 20% correct have Inoisy = 0.044 nats.
  • 60,000 noisy labels were predicted to be worth approximately 1,148 clean labels.The calculation uses 60,000 × 0.044/2.3 ≈ 1,148.
  • In reality, about 1,000 clean labels produced similar results to the 60,000 noisy labels.
  • Mean class balancing upweighted rare classes and downweighted frequent classes, but lowered performance and was not used.The authors suggest this may be because balancing makes too many assumptions about the unknown test distribution.

B.2. Alternative Target Distribution for Training Averaging Logits

The authors tested an alternative target distribution for training averaging logits: the equally weighted average of predictions from the doctors who labeled each case. This alternative produced inferior results.

  • The alternative target distribution averages predictions from the doctor models corresponding to doctors who labeled the training case.
  • The alternative target distribution produced inferior results when used in the cross-entropy loss.
  • Table 5 lists the optimal hyperparameters used for diabetic-retinopathy classification.

B.3. An Alternative Noise Model

The alternative noise model assumes symmetric errors across incorrect classes while allowing its noise parameter to vary by doctor. It performed slightly worse than the five-class Mnih and Hinton variant.

  • The model assumes that an incorrect label has equal probability of belonging to any other class.
  • The noise parameter varies by doctor and is estimated separately for each doctor.
  • The alternative noise model performed slightly worse than the 5-class variant of Mnih and Hinton (2012).The authors note that other noise models with varying complexity are also possible.

C. Hyperparameter Search

The diabetic-retinopathy classifier’s hyperparameters were selected by grid search across dropout, learning rate, entropy weight, and weight-decay spaces.

  • Grid search tuned dropout for the Inception backbone, dropout for doctor models, learning rate, entropy weight, and weight decay.The searched learning-rate range was 1×10^-7 to 0.03, with additional spaces for the other parameters.

D. Dataset Details

The study uses a large training set combining selected images from Gulshan et al. (2016) with newly labeled images, plus separate validation and test datasets. Validation and test images were screened for gradability, leaving labeled subsets for evaluation.

  • The training set contains 119,589 images from Gulshan et al. (2016) and 6,933 newly acquired labeled images.Some images from the earlier dataset were reserved for validation or excluded because they were ungradable.
  • The validation dataset contains 7,963 EyePACS clinic images, including 7,805 with at least one label after 158 were deemed ungradable.It is a random subset of the 9,963-image EyePACS-1 test set used by Gulshan et al. (2016).
  • The test set combines 1,748 Messidor-2 images with 2,000 EyePACS-1 images.After adjudication, 1,744 Messidor-2 images and 1,803 EyePACS-1 images were considered gradable and assigned labels.
Loading 1703.08774v2…