Source-linked AI summary
Meta Label Correction for Noisy Label Learning
Guoqing Zheng, Ahmed Hassan Awadallah, Susan Dumais
TL;DR
Learning from noisy labels is important for scaling machine-learning systems, but existing methods either reweight examples or rely on assumptions for label correction. MLC uses a meta-model to generate corrected labels while jointly training the main model through bi-level optimization, and it outperforms prior correction and reweighting methods across image and text tasks.
Problem
Learning with noisy labels seeks effective use of weak supervision when large-scale reliable annotations are scarce.
Method
MLC uses a label correction network as a meta-model to produce soft corrected labels while a main predictive model learns from them through bi-level optimization.
Results
MLC outperforms previous best methods for label correction and reweighting across 3 image-recognition and 4 large-scale text-classification tasks with varied noise.
Takeaways & Limitations
The correction framing lets MLC explore all classes in the label space rather than only up- or down-weighting a given noisy label.
Takeaways & Limitations
MLC is evaluated in a setting with a small clean dataset available alongside a large weakly labeled dataset.
Abstract
from arXiv · showhide
Leveraging weak or noisy supervision for building effective machine learning models has long been an important research problem. Its importance has further increased recently due to the growing need for large-scale datasets to train deep learning models. Weak or noisy supervision could originate from multiple sources including non-expert annotators or automatic labeling based on heuristics or user interaction signals. There is an extensive amount of previous work focusing on leveraging noisy labels. Most notably, recent work has shown impressive gains by using a meta-learned instance re-weighting approach where a meta-learning framework is used to assign instance weights to noisy labels. In this paper, we extend this approach via posing the problem as label correction problem within a meta-learning framework. We view the label correction procedure as a meta-process and propose a new meta-learning based framework termed MLC (Meta Label Correction) for learning with noisy labels. Specifically, a label correction network is adopted as a meta-model to produce corrected labels for noisy labels while the main model is trained to leverage the corrected labeled. Both models are jointly trained by solving a bi-level optimization problem. We run extensive experiments with different label noise levels and types on both image recognition and text classification tasks. We compare the reweighing and correction approaches showing that the correction framing addresses some of the limitation of reweighting. We also show that the proposed MLC approach achieves large improvements over previous methods in many settings.
Introduction
Weak or noisy labels provide scalable supervision but can come from corrupted, non-expert, heuristic, or interaction-based sources and challenge high-capacity models. MLC frames label correction as a meta-learning process, jointly optimizing corrected labels and the main model, and outperforms prior correction and reweighting methods across varied tasks and noise settings.
- Introduction: Weak supervision addresses the scarcity of large-scale annotated datasets by leveraging incomplete, inexact, or inaccurate supervision.Noisy labels may arise from corrupted annotations, non-expert annotators, heuristics, or user interaction signals.
- Introduction: High-capacity deep networks can fit and memorize noisy labels, motivating methods that select or re-weight likely reliable training examples.Prior approaches include co-teaching, curriculum learning, and instance reweighting.
- Introduction: Label correction can alter incorrectly assigned labels, whereas reweighting only increases or decreases an instance’s contribution during training.Earlier correction methods often estimate a corruption matrix under assumptions about the weak-label generation process.
- Introduction: MLC treats label correction as a meta-process that generates corrected labels while a main model learns from them through concurrent bi-level optimization.Clean labels serve as validation data for differentiably updating the correction process, without assuming the underlying noise mechanism.
- Introduction: MLC is evaluated on 3 image-recognition and 4 large-scale text-classification tasks across varying noise levels and types, including real-world noisy labels.The experiments compare label correction and instance reweighting methods.
- Introduction: Across these settings, the proposed method outperforms previous best methods for both label correction and reweighting.The paper presents this pattern as evidence for the proposed method’s performance across task and noise configurations.
Related Work
Prior noisy-label work either corrects labels using assumptions about a corruption process or selects and reweights likely useful examples. MLC instead co-optimizes label correction and model learning, outperforming state-of-the-art methods for both correction and reweighting.
- Related Work: Noisy-label research includes approaches for understanding, modeling, correcting, and learning from corrupted supervision.The paper focuses on settings with a small clean dataset and a large weakly labeled dataset.
- Learning with Label Correction: Label-correction methods estimate how noisy labels arise from true labels, often through a corruption matrix and a separate training step.Global corruption matrices can ignore data-dependent noise, while estimation may not receive feedback from the main model.
- Learning to Re-weight Training Instances: Sample-selection methods identify noisy examples likely to be correct, while reweighting assigns learnable weights instead of discarding examples.These approaches adjust each example’s contribution rather than changing its label.
- Related Work: MLC co-optimizes label correction and model learning through meta-learning rather than handling the two steps separately.The method is designed to outperform state-of-the-art approaches for both learning to correct and learning to re-weight.
Meta Label Correction
MLC frames noisy-label learning as joint meta-learning: a label correction network generates corrected labels while a main model learns from them, with both optimized bi-levelly using clean and weak data.
- Meta Label Correction: MLC uses a small clean dataset and a larger weakly labeled dataset to jointly learn label correction and prediction.The clean set provides trusted supervision, while the weak set supplies more training examples.
- Meta Label Correction: The label correction network maps a feature representation and weak label to a soft corrected-label distribution, while the main model predicts from the input.The LCN is parameterized by α, and the main model by w.
- Meta Label Correction: MLC links the two models through bi-level optimization: the main model trains on LCN-corrected labels, and clean-data loss guides updates to the LCN.The computation flow updates classifier parameters using weak data, evaluates the updated classifier on clean data, then updates the LCN.
- Meta Label Correction: One-step and k-step look-ahead SGD approximate the main model’s optimal parameters when computing meta-gradients for the LCN.Larger k can provide a less noisy estimate but requires caching k copies of the model parameters; k from 1 to 10 works empirically.
- Meta Label Correction: Soft labels enable gradient propagation to the meta-model, while splitting clean batches supplies direct clean guidance during training.The LCN output is constrained to a valid categorical distribution, and part of each clean batch is used in training the main model.
- Meta Label Correction: Unlike Meta-WN’s scalar weight on the observed noisy class, MLC assigns a distribution across all classes and directly uses features with the weak label.This gives MLC a less bottlenecked treatment of noisy examples than loss-based reweighting.
Experiments
MLC is evaluated across image and text classification tasks under simulated and real-world noisy labels, using a label-correction network and comparisons with reweighting and correction baselines. It consistently performs strongly, especially as noise becomes severe, while corrected labels can exploit information that reweighting alone cannot.
- Datasets and setup: MLC is evaluated on 3 image recognition and 4 large-scale text classification tasks with varying noise levels and types, including real-world noisy labels.
- Baselines and protocol: The experiments compare MLC with GLC label correction and MW-Net meta-learned instance reweighting using shared classifier architectures and benchmark protocols.
- Image recognition: MLC consistently outperforms other methods across image datasets and improves over GLC and MW-Net by over 2 points in accuracy on Clothing1M with real noisy labels.
- Text classification: On four text datasets, label correction achieves much higher performance than MW-Net, while MLC achieves the best results by combining correction with data-driven meta-learning.
- Noise-level analysis: As uniform noise increases from 0 to 1.0, Meta-WN performance decreases significantly, whereas GLC and MLC remain robust against severe label noise.
- Noise-level analysis: When noise exceeds 0.6, larger k produces higher MLC test accuracy, while k has little impact at low noise levels; similar trends occur for FLIP noise.
- Meta-model analysis: MLC outputs corrected-label distributions that can alter incorrect labels, whereas MW-Net only down-weights noisy instances.
- Training dynamics: Under increasing noise, MLC training becomes harder but still converges, with decreasing losses and increasing test accuracy.
Conclusions
The paper addresses noisy-label learning with a meta network that corrects labels and a main classifier trained on corrected or clean labels. Joint bi-level optimization and k-step look-ahead gradients support the method, which outperforms instance reweighting and previous noisy-label methods across image and text tasks.
- MLC uses a meta network to correct noisy labels, while a main classifier fits corrected labels for noisy examples and true labels for clean examples.
- The meta network and main classifier are jointly optimized through bi-level optimization using a k-step-ahead SGD update for the meta-gradient.
- Experiments across three image recognition and four text classification tasks show benefits of label correction over instance reweighting and strong performance over previous methods.