Source-linked AI summary
Training Deep Neural Networks on Noisy Labels with Bootstrapping
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, Andrew Rabinovich
TL;DR
Purely supervised vision systems assume accurate, complete labels, although recognition, detection, and subjective tasks often violate that assumption. The paper augments prediction with perceptual consistency and reports robustness across noisy-label, emotion-recognition, and detection settings, while identifying tuning and data-scope directions for future work.
Problem
Visual learning must handle missing, subjective, noisy, and incomplete labels that are not accommodated by predominant purely supervised recognition and detection systems.
Method
The method augments the prediction objective with perceptual consistency and bootstraps training targets from labels and current predictions.
Results
Experiments report robustness across several label-noise types and datasets, including MNIST, Toronto Face Database emotion recognition, and ILSVRC2014 detection.
Takeaways & Limitations
The approach can be applied with little engineering effort to existing purely supervised networks and supports weakly supervised deep learning for multi-class and structured outputs.
Takeaways & Limitations
Future work is needed to learn a time-dependent policy for tuning β and to extend the approach to situated agents.
Abstract
from arXiv · showhide
Current state-of-the-art deep learning systems for visual object recognition and detection use purely supervised training with regularization such as dropout to avoid overfitting. The performance depends critically on the amount of labeled examples, and in current practice the labels are assumed to be unambiguous and accurate. However, this assumption often does not hold; e.g. in recognition, class labels may be missing; in detection, objects in the image may not be localized; and in general, the labeling may be subjective. In this work we propose a generic way to handle noisy and incomplete labeling by augmenting the prediction objective with a notion of consistency. We consider a prediction consistent if the same prediction is made given similar percepts, where the notion of similarity is between deep network features computed from the input data. In experiments we demonstrate that our approach yields substantial robustness to label noise on several datasets. On MNIST handwritten digits, we show that our model is robust to label corruption. On the Toronto Face Database, we show that our model handles well the case of subjective labels in emotion recognition, achieving state-of-the- art results, and can also benefit from unlabeled face images with no modification to our method. On the ILSVRC2014 detection challenge data, we show that our approach extends to very deep networks, high resolution images and structured outputs, and results in improved scalable detection.
1 INTRODUCTION
The paper addresses noisy, missing, subjective, and incomplete labels that limit purely supervised visual learning. It proposes perceptual consistency to make predictions agree across similar percepts and demonstrates robustness across recognition and detection settings.
- Motivation: Large-scale visual datasets often contain missing labels, subjective judgments, or incompletely localized objects.The paper argues that these problems become more acute as training sets grow and are fundamental to scaling vision.
- Approach: The method augments the usual prediction objective with perceptual consistency based on similarity between deep-network features.Consistency means producing the same prediction for similar percepts.
- Approach: Bootstrapping lets the learner use its representation to disagree with perceptually inconsistent labels and progressively clean the training data.The approach balances ordinary prediction against perceptual consistency to avoid excessive skepticism toward labels.
- Results: Experiments show robustness to multiple label-noise types across MNIST, Toronto Face Database emotion recognition, and ILSVRC2014 detection.The reported results include robustness to corrupted labels, subjective-label handling with state-of-the-art emotion recognition, and improved detection performance.
2 RELATED WORK
Prior work addressed semi-supervised learning, self-training, noise modeling, and task-specific robustness. This paper instead integrates a generic consistency objective directly into deep-network training and extends it to structured outputs.
- Earlier bootstrapping methods: Earlier bootstrapping and co-training methods iteratively classified unlabeled examples or generated additional labels using classifiers or graph propagation.These approaches included self-training, multiple views, graph-based propagation, and statistical identification of mislabeled data.
- Weakly supervised detection: Weakly supervised detection previously used self-training as a wrapper around an existing detection system.That work achieved performance comparable to systems trained with substantially more labels.
- This work: This work incorporates consistency directly into the model rather than generating labels and adding examples in an outer loop.It addresses unlabeled, noisy, and inexhaustively annotated examples within the training objective.
- Noise robustness: Unlike methods designed for specific noise types, the paper adds a generic consistency objective to the loss.The comparison is made with robust loss functions developed for label omission and registration errors in aerial-image labeling.
- Scope: The approach extends beyond shallow or fully generative models to deep networks and structured output prediction.The related-work comparison emphasizes applicability to high-dimensional sensory data and structured outputs.
3 METHOD
The method augments prediction training with perceptual consistency to handle noisy, incomplete, and structured labels. It includes reconstruction-based noise modeling, bootstrapping with model-generated targets, and an adaptation for MultiBox detection.
- 3.1 CONSISTENCY IN MULTI-CLASS PREDICTION VIA RECONSTRUCTION: The reconstruction approach models label noise explicitly and adds reconstruction loss to promote top-down consistency between predictions and observations.This allows the model to discover patterns in the noise distribution.
- 3.2 CONSISTENCY IN MULTI-CLASS PREDICTION VIA BOOTSTRAPPING: Bootstrapping instead combines observed labels with current model predictions to generate targets without directly modeling the noise distribution.This is especially suited to structured outputs where dense interactions among output units are impractical or unhelpful.
- 3.1 CONSISTENCY IN MULTI-CLASS PREDICTION VIA RECONSTRUCTION: The model introduces a latent true label and uses softmax distributions to represent predictions, noisy-label generation, and marginalization over the latent label.Purely discriminative training alone does not explicitly encourage the latent variable to represent the true label.
- 3.1 CONSISTENCY IN MULTI-CLASS PREDICTION VIA RECONSTRUCTION: The autoencoder formulation avoids the intractable partition-function gradients and MCMC complications of fully generative RBM training, enabling exact gradient descent with rectified-linear networks.Figure 1 compares the RBM and analogous feed-forward autoencoder architectures.
- 3.2 CONSISTENCY IN MULTI-CLASS PREDICTION VIA BOOTSTRAPPING: Bootstrapping dynamically updates mini-batch targets as a convex combination of noisy labels and model predictions, allowing increasingly trusted predictions to reduce the influence of inconsistent labels.Soft bootstrapping uses predicted probabilities; hard bootstrapping uses MAP estimates, yielding an EM-like update procedure.
- 3.3 CONSISTENCY WITH STRUCTURED OUTPUT PREDICTION: For structured detection outputs, the method modifies MultiBox training so model predictions reduce penalties caused by inexhaustive annotations and unlabeled positive objects.The adaptation uses confidence scores over object locations and can learn high-confidence predictions despite frequently missing labels.
4 EXPERIMENTS
Experiments evaluate bootstrapping across noisy-label recognition, subjective emotion recognition, and large-scale object detection. The method improves robustness and detection performance, with reconstruction- and hard-bootstrapping variants generally strongest in the reported comparisons.
- Experimental setup: The experiments cover MNIST digit recognition, Toronto Faces emotion recognition, and ILSVRC2014 object detection.Each task uses a deep neural network trained with one of the proposed consistency-based bootstrapping objectives.
- MNIST with noisy labels: Bootstrap-recon performs best on MNIST with permuted labels, while bootstrap-hard performs nearly as well.Bootstrap-soft helps in the high-noise regime but is only slightly better than the baseline overall.
- MNIST with noisy labels: Bootstrap-recon learns a non-identity predicted-to-noisy-label connection that models how predictions can diverge from corrupted labels during reconstruction.The reconstruction term permits predictions to vary from training labels when that improves reconstruction of the input.
- Toronto Faces Database emotion recognition: On Toronto Faces, bootstrap-recon and bootstrap-hard significantly outperform the softmax baseline, while bootstrap-soft provides a more modest improvement.The learned connection spreads probability toward commonly confused emotions, including “afraid” and “surprised.”
- ILSVRC2014 detection: For ILSVRC2014 detection, bootstrapping yields modest improvement on val2 data, mainly from MultiBox training.In fast person detection, the proposed methods outperform the prediction-only baseline at high precision, whereas the top-K heuristic is slightly better at high recall.
5 CONCLUSIONS
The approach works across multi-class and structured-output tasks with little engineering effort, while suggesting cheaper data collection through less exhaustive annotation. Future work includes learning how to tune β over time and extending experiments with unlabeled or weakly labeled detection data.
- The method was effective for multi-class prediction and structured-output prediction across several datasets.
- Its simplicity allows application to existing purely supervised networks with very little engineering effort.
- The results suggest that less exhaustive annotation and more inexpensive data could improve performance because mistaken labels are less harmful.
- Future work could learn a time-dependent policy for tuning β and extend training with unlabeled or weakly labeled detection images.
- The reported detection experiments were rerun only on the “val2” subset after context and post-classifier models were found to use “val1” data.