Source-linked AI summary
Deep learning from crowds
Filipe Rodrigues, Francisco Pereira
TL;DR
Deep neural networks need extensive labeled data, while crowdsourcing supplies scalable but noisy labels from annotators with differing expertise. The paper introduces a crowd layer that trains networks end-to-end from those labels, and reports new state-of-the-art results across classification, regression, and sequence labeling. The approach captures annotator reliabilities and biases without EM’s additional computational overhead.
Problem
Deep learning requires large labeled datasets, but crowdsourced labels can be noisy and come from annotators with different expertise and reliability.
Method
The paper describes an EM baseline and proposes a general-purpose crowd layer that trains deep networks end-to-end directly from multiple annotators’ labels using backpropagation.
Results
The crowd layer achieved new state-of-the-art results across image classification, text regression, and named entity recognition, often outperforming EM-based and other state-of-the-art approaches.
Takeaways & Limitations
The crowd layer internally captures annotator reliabilities and biases while avoiding label aggregation before training and the additional computational overhead of EM.
Takeaways & Limitations
The EM approach is difficult to generalize to regression and sequence labeling, and the crowd layer assumes random or adversarial annotators are not a vast majority.
Abstract
from arXiv · showhide
Over the last few years, deep learning has revolutionized the field of machine learning by dramatically improving the state-of-the-art in various domains. However, as the size of supervised artificial neural networks grows, typically so does the need for larger labeled datasets. Recently, crowdsourcing has established itself as an efficient and cost-effective solution for labeling large sets of data in a scalable manner, but it often requires aggregating labels from multiple noisy contributors with different levels of expertise. In this paper, we address the problem of learning deep neural networks from crowds. We begin by describing an EM algorithm for jointly learning the parameters of the network and the reliabilities of the annotators. Then, a novel general-purpose crowd layer is proposed, which allows us to train deep neural networks end-to-end, directly from the noisy labels of multiple annotators, using only backpropagation. We empirically show that the proposed approach is able to internally capture the reliability and biases of different annotators and achieve new state-of-the-art results for various crowdsourced datasets across different settings, namely classification, regression and sequence labeling.
Introduction
Deep learning needs large labeled datasets, while crowdsourcing provides scalable labels that may be noisy and vary in contributor reliability. The paper addresses this challenge with an end-to-end crowd layer and reports strong results across multiple tasks.
- Motivation: Deep learning performs well on complex, high-dimensional tasks but typically requires large labeled datasets.Such data is often unavailable, requiring manual human labeling.
- Motivation: Crowdsourcing offers an efficient, cost-effective way to annotate large collections of data at scale.It has also been used for domains such as medical imaging.
- Motivation: Labels from multiple crowdsourced annotators can be noisy and compromise practical applications.Annotators may differ in expertise and reliability.
- Approach: The proposed crowd layer trains neural networks end-to-end directly from multiple annotators’ noisy labels using backpropagation.It avoids the additional computational overhead of EM.
- Results: Across image classification, text regression, and named entity recognition, the crowd layer captured annotator reliability and biases and achieved comparable or often superior results.The experiments used real Amazon Mechanical Turk data and reported new state-of-the-art results.
Related work
Prior work mainly aggregates noisy labels or jointly learns annotator expertise with a classifier. This paper instead trains deep networks directly from annotator labels, avoiding preprocessing and EM-based procedures.
- Label aggregation: Crowdsourcing research has developed methods to address label noise and annotator trustworthiness in large-scale annotation.Dawid–Skene and related models aggregate labels from contributors with different expertise.
- Label aggregation: Traditional label aggregation estimates latent truths or worker quality before downstream machine-learning training.Examples include majority voting, annotator error-rate models, item difficulty, and worker pruning.
- Joint learning: Joint-learning approaches combine annotator expertise estimation with classifier training, extending from logistic regression and Gaussian processes to deep models.These approaches are presented as preferable to standalone aggregation.
- This paper: The proposed method trains deep neural networks directly from multiple noisy annotators’ labels using pure backpropagation rather than label aggregation or EM.This is the paper’s contrast with earlier approaches.
- Comparison: A related two-stage method learns individual expert predictions and then averaging weights, whereas the proposed approach captures annotator biases.The related method also avoids EM but may require significantly higher computational time.
EM algorithm for deep learning from crowds
The paper formulates crowdsourced classification with latent true classes and annotator-specific confusion matrices, then derives an EM procedure to learn network and reliability parameters. Its limitations arise for continuous outputs and structured sequences, motivating a more general approach.
- Model formulation: Each input has multiple crowdsourced labels, modeled as noisy versions of a latent true class predicted by a deep network.The network parameters are Θ and annotator outputs are modeled conditionally on the latent class.
- Model formulation: Each annotator is represented by a confusion matrix that models their expertise, under an independence assumption across annotators.The number of classes is denoted by C.
- EM procedure: The EM algorithm jointly learns annotator reliabilities and neural-network parameters.The E-step estimates the posterior over latent variables, while the M-step updates model parameters.
- Optimization: Dirichlet priors on annotator reliability parameters are important because crowd annotators typically label only a small portion of the data.The network uses noise-adjusted ground-truth estimates for backpropagation with SGD or Adam.
- Limitations: The EM formulation is difficult to generalize beyond classification because continuous outputs lack the required probabilistic softmax interpretation.For sequence outputs, latent-variable marginalization becomes intractable as possible label sequences grow exponentially with sequence length.
Crowd layer
The crowd layer trains deep neural networks directly from multiple annotators’ noisy labels by learning annotator-specific mappings within a shared bottleneck. Backpropagation aggregates reliability- and bias-adjusted gradients, while the framework extends to sequence labeling and regression.
- The crowd layer is a network layer for training deep neural networks directly from multiple annotators’ noisy labels.
- It converts the usual output layer into a shared bottleneck and learns annotator-specific mappings that capture reliabilities and biases.
- For classification, each annotator can use a matrix mapping fr(σ) = Wrσ from the network output to annotator-specific activations.
- Backpropagation combines annotator-specific gradients as a weighted sum, adjusts for systematic class-confusion biases, and sets missing annotators’ contributions to zero.
- After training, the crowd layer can be removed so the bottleneck output σ predicts labels for unseen instances.
- The approach assumes random or adversarial annotators do not form a vast majority; otherwise, it would perform no better than a random predictor.
- The framework applies directly to sequence labeling and can be adapted to regression using scalar and bias parameters per annotator.
Experiments
Experiments on simulated and real crowdsourced data evaluate the crowd layer across image classification, regression, and sequence labeling. The results show that it can outperform aggregation and competing joint-learning methods while capturing annotator reliability and bias.
- Simulated image classification: The crowd layer was evaluated on simulated Dogs vs. Cats annotators with varying sensitivities and specificities using a CNN-based binary classification setup.The dataset contained 25,000 images, and annotator responses were sampled from Bernoulli distributions parameterized by individual sensitivities and specificities.
- Simulated image classification: Across 30 runs, EM-based and crowd-layer methods significantly outperformed majority voting and Dawid–Skene baselines on the Dogs vs. Cats task.The experiments reported average accuracies using separate training and testing splits.
- Simulated image classification: The best crowd-layer variant achieved higher accuracy than Doctor Net and Weighted Doctor Net, which could not correct annotator biases.The MW variant gave the best average accuracy, and its learned weight-matrix diagonals strongly correlated with annotator sensitivities and specificities.
- Real image classification: On the real LabelMe-derived AMT dataset, DL-CL again gave the best results and its learned weight matrices resembled annotators’ true confusion matrices.The dataset used 1,000 multiply annotated training images and 1,688 evaluation images; the small training set used pre-trained VGG-16 layers.
- Regression: On MovieReviews regression, the crowd layer—particularly the B variant—significantly outperformed all other methods and learned bias parameters highly correlated with true annotator biases.The B variant models each annotator with an additive bias, fr(µ) = µ + br.
- Sequence labeling: For CoNLL-2003 named entity recognition, the proposed approach significantly outperformed the other methods, matched CRF-MA similarly, and reduced training time by at least one order of magnitude.The comparison reports minutes rather than several hours on the stated hardware.
Conclusion
The crowd layer trains deep networks end-to-end from multiple annotators’ labels, capturing their reliabilities and biases while avoiding more complex EM-based approaches. Its generalization beyond classification is demonstrated on text regression and named entity recognition, although EM’s E-step can have O(NTR) complexity.
- Conclusion: The crowd layer trains deep neural networks end-to-end directly from multiple annotators’ labels using backpropagation.It adjusts backpropagated error gradients according to annotator reliabilities and biases.
- Conclusion: Empirical evaluation shows the crowd layer outperforms label-aggregation methods and more complex EM-based state-of-the-art approaches.
- Conclusion: The approach generalizes beyond classification to text regression and named entity recognition using real Amazon Mechanical Turk data.
- Conclusion: The EM baseline has an E-step computational complexity of O(NTR), making computational cost a relevant limitation of that approach.