Source-linked AI summary
Learning to Learn from Noisy Labeled Data
Junnan Li, Yongkang Wong, Qi Zhao, Mohan Kankanhalli
TL;DR
Noisy annotations offer cheaper data but can cause DNNs to overfit, motivating training methods that work without clean labels. The paper introduces a meta-learning update using synthetic noisy labels and teacher consistency before conventional training. Experiments on two datasets with synthetic and real-world noise report advantageous performance compared with state-of-the-art methods.
Problem
Low-cost labeled data often contains noisy annotations, and the paper asks how to train effectively on noisy labeled datasets without relying on expensive high-quality annotation.
Method
MLNT performs a meta-learning update before conventional training, using multiple synthetic noisy-label batches and teacher-model consistency to learn noise-tolerant parameters.
Results
Experiments on two datasets with synthetic and real-world label noise demonstrate advantageous image-classification performance compared with state-of-the-art methods.
Takeaways & Limitations
The proposed training algorithm learns from noisy labeled data without human supervision or access to clean labels and is applicable to gradient-based models.
Takeaways & Limitations
The iterative training scheme uses a threshold τ to balance the quality and quantity of data retained from mentor-model filtering.
Abstract
from arXiv · showhide
Despite the success of deep neural networks (DNNs) in image classification tasks, the human-level performance relies on massive training data with high-quality manual annotations, which are expensive and time-consuming to collect. There exist many inexpensive data sources on the web, but they tend to contain inaccurate labels. Training on noisy labeled datasets causes performance degradation because DNNs can easily overfit to the label noise. To overcome this problem, we propose a noise-tolerant training algorithm, where a meta-learning update is performed prior to conventional gradient update. The proposed meta-learning method simulates actual training by generating synthetic noisy labels, and train the model such that after one gradient update using each set of synthetic noisy labels, the model does not overfit to the specific noise. We conduct extensive experiments on the noisy CIFAR-10 dataset and the Clothing1M dataset. The results demonstrate the advantageous performance of the proposed method compared to several state-of-the-art baselines.
1. Introduction
The paper addresses training DNNs on noisy labeled data without human supervision or clean labels by placing a meta-learning update before conventional training. It uses synthetic label noise and teacher-consistency objectives to learn noise-tolerant parameters, with experiments showing advantageous performance on two datasets.
- Motivation: Low-cost annotation sources reduce data-collection expense but introduce label noise that can significantly affect classifier accuracy.The paper identifies training effectively on noisy labeled datasets as its central problem.
- Contribution: MLNT is a model-agnostic training algorithm that requires neither human supervision nor clean labels and applies to gradient-based models.It is presented as an alternative to methods that are either difficult to scale or heuristic.
- Contribution: The method optimizes a meta-objective before conventional training so parameters become less prone to overfitting across artificially generated label noise.For each mini-batch, synthetic noisy labels simulate noisy training before the conventional update.
- Contribution: The meta-objective trains models updated on varied synthetic noisy labels to produce predictions consistent with a teacher model.The teacher is constructed with self-ensembling to provide predictions less affected by synthetic noise.
- Evaluation: Experiments on two datasets containing synthetic and real-world label noise demonstrate advantageous image-classification performance compared with state-of-the-art methods.The study also includes extensive ablations of the proposed components.
2. Related Work
Prior noisy-label methods either depend on clean human supervision or use scalable but less effective and heuristic corrections. Figure 1 contrasts conventional updates that may overfit label noise with the proposed preceding meta-learning update using synthetic noise.
- Learning with label noise: Noise-modeling approaches infer true labels or downweight noisy samples, but often require clean labeled data or expensive estimation.These approaches use neural networks, graphical models, knowledge graphs, or conditional random fields to characterize label noise.
- Figure 1: Conventional cross-entropy gradient updates may overfit label noise, while a preceding meta-learning update with synthetic noise encourages noise-tolerant parameters.The figure presents the proposed update as reducing overfitting during subsequent conventional training.
- Self-ensembling: Self-ensembling uses a student and a teacher whose weights are an exponential moving average of the student, enforcing consistent predictions.This approach forms one basis of the paper’s method.
3. Method
MLNT prepares a model for noisy labels by applying a meta-learning update before conventional classification training. It generates synthetic noise, enforces consistency with a teacher, and extends this process through iterative filtering and prediction refinement.
- Meta-learning update: MLNT optimizes a meta-objective before the conventional classification update to reduce overfitting to label noise.The method is model-agnostic and applies to models trained with gradient-based rules.
- Synthetic noise generation: Synthetic noisy-label mini-batches are generated by transferring labels among nearest neighbors, preserving a noise distribution similar to the original data.Neighbors are selected from the top 10 within the mini-batch using pre-softmax feature representations from a pretrained DNN.
- Meta-test consistency: Each synthetic mini-batch receives one gradient update, and the resulting model is trained to match a teacher model’s predictions using KL-divergence.The meta loss averages consistency losses across all updated models, while optimization is performed over the student parameters.
- Training loop: After the meta-learning update, SGD minimizes classification loss on the original mini-batch, followed by an exponential-moving-average teacher update.The full training loop initializes student and teacher parameters, generates synthetic labels, and alternates meta-learning with conventional classification training.
- Iterative training: Iterative training filters samples using a mentor model and merges mentor and teacher predictions to improve consistency learning.The filtered set removes examples whose ground-truth labels receive low mentor probability, while a weight λ balances teacher and mentor predictions.
- Iterative training: The experiments use three training iterations, with the best model from the previous iteration serving as the mentor.Further iterations beyond three do not provide noticeable performance improvement.
4. Experiments
Experiments evaluate MLNT on synthetic and real-world noisy labels, comparing it with baselines and analyzing training behavior and hyperparameters. MLNT generally improves performance, with iterative training producing further gains and reduced overfitting.
- Datasets and settings: Experiments use CIFAR-10 with symmetric and asymmetric synthetic noise and Clothing1M with real-world label errors.CIFAR-10 uses artificial corruption, while Clothing1M labels come from surrounding seller-provided text and contain real-world errors.
- Baseline comparison: MLNT does not require prior knowledge of the noise transition matrix or class distribution, unlike several baselines.Forward and CNN-CRF require the ground-truth noise transition matrix, while Joint Optimization requires the ground-truth class distribution.
- CIFAR-10 results: MLNT with one training iteration significantly improves accuracy over reproduced Cross Entropy and achieves performance comparable to state-of-the-art methods on both noise types.After three iterations, MLNT-teacher significantly outperforms previous methods, except under 50% asymmetric CAT-DOG noise, which is theoretically indistinguishable without prior knowledge.
- CIFAR-10 results: +1.68 accuracy is achieved by MLNT over Cross Entropy with clean CIFAR-10 training data, indicating a regularization effect.The comparison is reported for CIFAR-10 with r = 0.
- Progressive comparison: MLNT learns faster initially and suffers less overfitting than Cross Entropy, while MLNT-teacher has the most stable training behavior.The progressive comparison uses asymmetric label noise with r = 0.4; after epoch 80, MLNT-student has lower training accuracy and higher test accuracy than Cross Entropy.
- Hyperparameters: Accuracy increases as the number of synthetic mini-batches M increases, with the largest gain from M = 0 to 5 and marginal improvement from M = 10 to 15.The experiments use M = 10 as a trade-off between training speed and performance.
- Hyperparameters: Performance is insensitive to ρ, with the optimal range generally [0.3k, 0.5k], while the optimal τ increases as the noise ratio increases.ρ controls how many labels change in each synthetic mini-batch; τ controls filtering during later training iterations.
- Optimization ablation: First-order optimization performs nearly the same as full second-order optimization, suggesting the main improvement comes from meta-loss gradients at updated parameters.The comparison averages test accuracy across five runs on four representative noisy-label settings.
5. Conclusion
The paper proposes a meta-learning method for training noise-tolerant models on noisy labeled data, combining synthetic-label meta-training with teacher-guided consistency optimization. It also introduces iterative training and reports advantageous performance on two datasets compared with state-of-the-art methods.
- The method performs a meta-learning update before the conventional gradient update to find parameters less prone to overfitting noisy labels.
- Meta-training uses multiple mini-batches with synthetic noisy labels, while meta-testing minimizes consistency loss between updated models and a teacher model.
- An iterative training scheme uses the previous iteration’s model to clean data and refine predictions.
- Experiments on two datasets validate advantageous performance compared with state-of-the-art methods.
- The authors propose extending the model-agnostic method to other domains and architectures, including recurrent neural networks for machine translation.