Source-linked AI summary
Multi-Objective Interpolation Training for Robustness to Label Noise
Diego Ortego, Eric Arazo, Paul Albert, Noel E. O'Connor, Kevin McGuinness
TL;DR
Label noise degrades deep-network training, while much prior work focuses on robust classification losses. MOIT combines interpolated contrastive learning, representation-based noise detection, and semi-supervised classification, with MOIT+ fine-tuning detected clean data; the methods achieve state-of-the-art results across several noisy-label benchmarks.
Problem
Label noise degrades deep neural network performance, motivating methods that robustly learn from synthetic and web-labeled data.
Method
MOIT jointly uses interpolated supervised contrastive learning, soft-label-based noise detection, and semi-supervised classification, while MOIT+ fine-tunes on detected clean data.
Results
MOIT/MOIT+ achieves state-of-the-art results across CIFAR-10/100, mini-ImageNet, and mini-WebVision with synthetic and real-world web label noise.
Takeaways & Limitations
Robust feature representations can support noise detection and semi-supervised classification within a unified interpolation-training framework.
Takeaways & Limitations
The evaluation omits other mini-ImageNet and WebVision frameworks because of computing-resource limitations.
Abstract
from arXiv · showhide
Deep neural networks trained with standard cross-entropy loss memorize noisy labels, which degrades their performance. Most research to mitigate this memorization proposes new robust classification loss functions. Conversely, we propose a Multi-Objective Interpolation Training (MOIT) approach that jointly exploits contrastive learning and classification to mutually help each other and boost performance against label noise. We show that standard supervised contrastive learning degrades in the presence of label noise and propose an interpolation training strategy to mitigate this behavior. We further propose a novel label noise detection method that exploits the robust feature representations learned via contrastive learning to estimate per-sample soft-labels whose disagreements with the original labels accurately identify noisy samples. This detection allows treating noisy samples as unlabeled and training a classifier in a semi-supervised manner to prevent noise memorization and improve representation learning. We further propose MOIT+, a refinement of MOIT by fine-tuning on detected clean samples. Hyperparameter and ablation studies verify the key components of our method. Experiments on synthetic and real-world noise benchmarks demonstrate that MOIT/MOIT+ achieves state-of-the-art results. Code is available at https://git.io/JI40X.
1. Introduction
Label noise limits deep-network performance, while existing defenses largely emphasize classification losses. MOIT instead combines contrastive and semi-supervised learning, using interpolation and representation-based noise detection, with MOIT+ fine-tuning detected clean data.
- Motivation: Label noise degrades deep neural network performance and complicates scalable dataset construction based on automatic annotation.Manual labeling is cumbersome, whereas web and user-tag annotation expands datasets but introduces incorrect labels.
- Motivation: Most label-noise defenses identify, correct, or downweight noisy samples, but predominantly rely on classification losses rather than similarity learning.The paper positions representation learning as an underused direction for robust classification.
- MOIT: MOIT jointly trains supervised contrastive and semi-supervised classification objectives so they mutually support robustness to synthetic and web label noise.The framework uses a single hyperparameter configuration across both noise settings.
- MOIT: Interpolated Contrastive Learning imposes linear relations on inputs and contrastive loss to mitigate supervised contrastive learning degradation under label noise.The strategy regularizes contrastive learning rather than relying only on a classification objective.
- Noise detection and refinement: MOIT detects noisy samples by comparing k-nearest-neighbor-inferred per-sample label distributions with original labels using noise-robust representations.Agreement identifies correctly labeled samples, enabling semi-supervised learning; MOIT+ then fine-tunes on detected clean data.
2. Related work
Prior work addresses label noise through sample rejection, label correction, interpolation, robust losses, and semi-supervised learning. MOIT is situated among these approaches while extending interpolation and contrastive representation learning within one framework.
- Label-noise defenses: Recent label-noise methods estimate transition matrices, reject samples, correct labels, or use clean data to reduce noisy-label influence.These strategies include sample weighting, iterative clean-set refinement, cross-network consistency, and label correction.
- Label-noise defenses: Semi-supervised approaches treat detected noisy samples as unlabeled, differing mainly in how they detect noise.The cited methods use network agreements, predictions, or related signals for detection.
- Other methods: Interpolation training such as mixup is widely used to prevent label-noise memorization, while other methods regularize memorized information, add abstention, or combine robust losses.DivideMix combines interpolation, cross-network agreements, semi-supervised learning, and label correction.
- MOIT positioning: MOIT’s figure combines identical interpolation in supervised contrastive and semi-supervised classification losses, repeated noise detection, and post-training clean-data fine-tuning.This connects contrastive representation learning with semi-supervised classification in a single training pipeline.
- Contrastive learning: Contrastive representation learning brings positive samples closer and negative samples farther apart, with augmentation and many negatives commonly supporting performance.These frameworks learn representations rather than only a class mapping.
3. Method
MOIT jointly trains robust representations and a classifier under label noise: interpolated contrastive learning supports noise detection, while semi-supervised classification uses detected clean and unlabeled samples. MOIT+ further fine-tunes on detected clean data.
- MOIT combines contrastive representation learning with semi-supervised classification so the two objectives mutually support robustness to label noise.Robust representations enable noise detection, while semi-supervised classification improves robust image classification.
- Interpolated Contrastive Learning: Interpolated Contrastive Learning (ICL) mixes training samples and imposes a corresponding linear relation on their contrastive losses.The mixing coefficient λ is sampled from a Beta distribution, and the dominant class guides positive and negative sampling.
- Interpolated Contrastive Learning: A memory bank extends contrastive learning beyond the minibatch by contrasting current samples with stored feature representations.The final ICL loss aggregates minibatch and memory losses; the paper reports benefits from both ICL and the memory bank.
- Label noise detection: Noise detection estimates class distributions from k-nearest-neighbor feature neighborhoods, corrects them using dominant neighborhood labels, and compares them with original labels.Higher disagreement indicates greater likelihood that a sample is noisy.
- Semi-Supervised Classification: MOIT selects clean samples using dynamically defined per-class thresholds, balances the clean set, and treats remaining samples as unlabeled for semi-supervised learning.The method avoids requiring a trusted clean set and initializes semi-supervised training with the original labels before accurate detection becomes available.
4. Experiments
Experiments evaluate MOIT/MOIT+ on synthetic and web label noise, with ablations examining representations, noise detection, balancing, and joint-training components. Across CIFAR, mini-ImageNet, and mini-WebVision, the methods achieve state-of-the-art results, especially under asymmetric noise.
- Representation analysis: Supervised contrastive learning degrades with label noise, while adding the classification objective stabilizes training and yields better representations than supervised contrastive or interpolated contrastive learning alone.Representations are evaluated with weighted k-NN using k = 200 on CIFAR-100 embeddings.
- Noise detection: 90.83 precision and 87.84 recall are achieved by corrected soft-labels, compared with 80.20 precision and 84.43 recall for the uncorrected soft-labels.The corrected distribution is estimated from nearest neighbors; K = 250 is used thereafter, and performance is not very sensitive once K is sufficiently large.
- Noise detection: Median balancing of per-class label agreements outperforms minimum, maximum, and unbalanced clean-set selection for classification.The median provides a trade-off between restricting classes with many agreements and extending classes with few agreements.
- Joint training ablation: Removing semi-supervised learning, memory, or clean-set balancing degrades performance, while classifier retraining and avoiding strong augmentation affect MOIT+ refinement.Semi-supervised learning is particularly important because omitting it leads to label-noise memorization; strong augmentation helps contrastive learning but harms classification accuracy.
- Benchmark evaluation: MOIT and MOIT+ achieve state-of-the-art results across CIFAR-10/100 and web-noise benchmarks, with especially robust performance under asymmetric noise.On mini-ImageNet and mini-WebVision, the methods remain robust without noise-level-specific re-parameterization; MOIT+ consistently outperforms MOIT, but gains are smaller than on CIFAR.
5. Conclusion
MOIT combines supervised contrastive learning and classification to robustly learn under synthetic and web label noise. MOIT+ further fine-tunes on detected clean data, and both achieve state-of-the-art results across evaluated datasets and noise settings.
- MOIT jointly combines supervised contrastive learning and classification to robustly handle synthetic and web label noise.Its contrastive component learns noise-robust representations used for noisy-sample detection and semi-supervised classification.
- Interpolated Contrastive Learning estimates per-sample soft-label distributions whose agreement with original labels identifies correctly labeled samples.Samples identified as noisy are treated as unlabeled during classifier training.
- MOIT+ refines the model by fine-tuning while retraining the image classifier.
- MOIT and MOIT+ achieve state-of-the-art results on CIFAR-10/100, mini-ImageNet, and mini-WebVision across different noise distributions and levels.