Source-linked AI summary
Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks
Arash Vahdat
TL;DR
High-quality annotation is costly, motivating methods that learn from cheaper noisy labels. The paper introduces a CRF-based semi-supervised framework with auxiliary-distribution regularization for tractable latent clean-label inference, and reports robust prediction and label cleaning on image-labeling benchmarks.
Problem
High-quality annotation is costly and time-consuming, while cheaper data-collection approaches introduce label noise and create difficult latent clean-label inference problems.
Method
The paper uses a CRF to model noisy and clean labels, trains it semi-supervised with noisy and small clean datasets, and regularizes inference with auxiliary information.
Results
The model predicts clean labels on unseen instances more accurately, recovers noisy training labels with higher precision, and reaches 69.4% mAP versus 49.4% initially on a COCO noisy-label set.
Takeaways & Limitations
The framework can both improve clean-label prediction and clean noisy annotations, including for image-labeling applications.
Abstract
from arXiv · showhide
Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming. This paper proposes a novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply. The problem is formulated using an undirected graphical model that represents the relationship between noisy and clean labels, trained in a semi-supervised setting. In our formulation, the inference over latent clean labels is tractable and is regularized during training using auxiliary sources of information. The proposed model is applied to the image labeling problem and is shown to be effective in labeling unseen images as well as reducing label noise in training on CIFAR-10 and MS COCO datasets.
1 Introduction
The paper addresses costly high-quality annotation by developing a CRF-based framework for training deep CNNs with noisy labels. It targets efficient latent clean-label inference through semi-supervised learning and auxiliary information.
- Motivation: Low-cost data collection methods reduce annotation expense but introduce label noise into training datasets.Examples include social-media mining, search engines, fewer annotators, and amateur annotators.
- Problem: Previous directed-model approaches struggle with multilabel classification because latent clean-label configurations grow exponentially and inference becomes difficult.The difficulty is linked to the explaining-away phenomenon.
- Approach: The proposed CRF models noisy–clean label relationships with latent clean labels that can be inferred efficiently, supporting multiclass and multilabel classification.The model is presented as a robust loss layer that can be plugged into existing networks.
- Approach: Latent variables may lack clean-label semantics when the joint model permits arbitrary latent configurations.The paper addresses this through a semi-supervised formulation using many noisy examples and a small clean set.
- Contributions: The framework proposes a generic robust CRF model, an auxiliary-information-based objective, and empirical evidence that it outperforms previous techniques.The stated application scope includes both multiclass and multilabel classification.
2 Previous Work
Prior work models label noise with class-conditional or class-and-instance-conditional assumptions, while related semi-supervised methods combine small clean datasets with noisy data. Deep structured models commonly rely on approximate or iterative inference to train CNN-CRFs.
- Learning from Noisy Labels: Label-noise methods are grouped into class-conditional models and class-and-instance-conditional models.The former model transitions from clean classes to noisy classes, while the latter conditions noise explicitly on each instance.
- Learning from Noisy Labels: Existing noisy-label techniques primarily address binary or multiclass classification, where marginalization over classes is possible.The cited approaches include transition modeling, mistaken-label variables, and neural-network bootstrapping.
- Semi-Supervised Learning: Semi-supervised approaches combine small clean datasets with noisy labeled data, often obtained from the web, using propagation or weighted-loss methods.Examples include pairwise similarity propagation, graph-based propagation, and weighted cross entropy.
- Deep Structured Models: Deep CNN-CRF training requires inference and gradient back-propagation through the structured model.Prior approaches use mean-field approximation, belief propagation, unrolled inference, or sampling.
3 Robust Discriminative Neural Network
The CNN-CRF models noisy and clean labels jointly, using tractable latent-label inference and semi-supervised training with auxiliary regularization. Training alternates variational inference and parameter updates, with auxiliary information guiding early inference.
- Model structure: The model represents noisy labels, latent clean labels, and hidden binary variables in a CRF conditioned on the image.The clean–noisy link captures label correlations, while hidden variables preserve a bipartite graph for simple inference.
- Model structure: The CNN supplies image-dependent bias terms, while label-interaction matrices remain input-independent for regularization.Together, the deep network and CRF form the CNN-CRF model.
- Semi-supervised training: Latent variables are inferred with an analytic conditional distribution, while marginal-likelihood optimization uses stochastic maximum likelihood, also called persistent contrastive divergence.The variational interpretation alternates an E step setting q to the analytic conditional and an M step equivalent to PCD updates.
- Semi-supervised training: Training combines marginal likelihood from a large noisy-labeled set with a small clean-labeled set in a semi-supervised objective.The noisy set contains image–noisy-label pairs, whereas the clean set additionally provides clean labels.
- Auxiliary regularization: An auxiliary distribution regularizes variational inference by penalizing divergence from auxiliary conditionals, interpolating between model-based and auxiliary inference through α.α = 0 recovers the original bound, α →∞ makes q ignore the model conditional, and intermediate values combine both sources.
- Auxiliary regularization: The auxiliary distribution is an RBM trained on the clean set, with fixed parameters during CNN-CRF training, and α is scheduled from large to smaller values.The schedule relies more on pretrained auxiliary information early, when the CNN-CRF conditional is inaccurate, then shifts toward the learned conditional.
4 Experiments
Experiments evaluate robust CNN-CRF variants on COCO and CIFAR-10, including noisy Flickr tags and synthesized label noise. The model improves clean-label inference and slightly improves CIFAR-10 prediction accuracy while substantially improving noisy-label recovery.
- COCO Dataset: COCO experiments compare robust CNN-CRF variants and several cross-entropy and structural baselines using mAP.The evaluated variants include models with or without hidden variables, the x-y link, and regularization.
- COCO Dataset: Adding each model component increases performance, while removing the x-y link generally improves recognition of clean labels.The authors attribute this to greater reliance on noisy labels and their correlations with clean labels.
- COCO Dataset: Removing regularization from the CRF without x-y connection performs very poorly, demonstrating the importance of the introduced regularization.This comparison uses the α = 0 variant.
- COCO Dataset with Flickr Tags: 49.4% mAP initially increases to 69.4% mAP for q on noisy COCO training images, a 20.0% mAP gain over training.The variational distribution q begins close to the fixed auxiliary distribution paux and becomes more accurate during CRF-CNN training.
- CIFAR-10 Dataset: CIFAR-10 evaluation uses synthesized label noise and compares the proposed model with baselines trained using noisy or clean labels.The experiment also includes forward and backward losses trained with the ground-truth noise transition matrix.
- CIFAR-10 Dataset: The proposed method achieves slightly better CIFAR-10 prediction accuracy and significantly higher recovery accuracy for clean labels in noisy training data.Recovery accuracy measures q’s clean-label predictions, while baseline recovery uses the trained network’s predictions mapped to the same set.
5 Conclusion
The paper presents a general undirected graphical model and semi-supervised objective for robust learning with noisy labels. Its framework improves prediction on unseen instances and recovery of clean labels in noisy training sets.
- Conclusion: The proposed framework models label noise with a general undirected graphical model for training deep neural networks.The formulation treats the problem as semi-supervised learning.
- Conclusion: A regularized objective helps the variational distribution infer latent clean labels more accurately using auxiliary information.The regularization term is part of the proposed objective function.
- Conclusion: The model predicts clean labels more accurately on unseen instances and recovers clean labels with higher precision on noisy training sets.The authors identify noisy-label cleaning as a valuable property of the framework.