Source-linked AI summary
Learning from Noisy Labels with Distillation
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Li
TL;DR
Learning from noisy labels is important because large datasets are easy to obtain, but real-world noise can be multi-modal rather than statistical outliers. The paper uses distillation from a small clean dataset, guided by knowledge-graph label relations, and reports superior results on new real-world-noise benchmarks across three domains. Its scope includes settings with a small cleaned portion of the data, with broader noisy-label scenarios left for future work.
Problem
Real-world noisy labels can form multi-modal groups rather than independent statistical outliers, motivating methods that learn from abundant noisy data without relying only on outlier removal.
Method
The framework distills knowledge from a small clean dataset while using semantic relations in a knowledge graph to guide learning from the full noisy dataset.
Results
The proposed distillation methods achieve the best results against competing methods on new benchmarks spanning Sports, Species, and Artifacts.
Takeaways & Limitations
The study supports using clean-data knowledge and semantic label relations to correct noisy labels in practical visual-recognition benchmarks.
Takeaways & Limitations
The formulation assumes that the dataset contains a small clean subset and a larger remaining noisy subset, while application to other noisy-label scenarios is future work.
Abstract
from arXiv · showhide
The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Traditionally, the label noises have been treated as statistical outliers, and approaches such as importance re-weighting and bootstrap have been proposed to alleviate the problem. According to our observation, the real-world noisy labels exhibit multi-mode characteristics as the true labels, rather than behaving like independent random outliers. In this work, we propose a unified distillation framework to use side information, including a small clean dataset and label relations in knowledge graph, to "hedge the risk" of learning from noisy labels. Furthermore, unlike the traditional approaches evaluated based on simulated label noises, we propose a suite of new benchmark datasets, in Sports, Species and Artifacts domains, to evaluate the task of learning from noisy labels in the practical setting. The empirical study demonstrates the effectiveness of our proposed method in all the domains.
1. Introduction
Internet-scale visual datasets offer abundant but noisy labels, while real-world noise often forms meaningful multi-modal groups rather than statistical outliers. The paper proposes distillation using a small clean dataset and knowledge-graph relations, and evaluates it on new real-world-noise benchmarks.
- Internet-scale datasets provide abundant images and videos, but their labels can have weak visual correlation and challenge learning.
- Real-world noisy samples may form significant multi-modal masses, so statistical outlier detection can fail to identify the intended visual concept.Examples include polysemous search terms and Flickr images labeled basketball that depict surrounding events rather than the event itself.
- The proposed method distills knowledge from a small clean dataset into learning from the entire noisy dataset to improve visual representation and classification.The clean set may be public or labeled from part of the noisy data.
- The framework additionally uses a knowledge graph to encode relations among labels and guide the distillation process.
- The evaluation introduces 480K images across 780 categories in Sports, Species, and Artifacts, with real-world labeling noise and Wikipedia-based textual knowledge graphs.
- The distillation process and its knowledge-graph-guided version achieve the best results on the proposed datasets compared with competing methods.
2. Related Work
Related work addresses noisy labels through outlier handling, reweighting, side information, and specialized network layers, while distillation transfers information from a teacher to a student. This paper builds on soft distillation scores to study noisy-label learning.
- Prior methods treat noisy samples as outliers, use bootstrap label modification, or estimate noise for importance re-weighting.
- The related literature includes applications of noisy-label learning to text-based image search and fine-grained classification.
- Side-information approaches use graphs, small manually labeled datasets, and semantic distances to address data noise or support learning.
- Sukhbaatar and Fergus add a linear layer that estimates label-switching probabilities, but its performance in real-world scenarios is unclear.
- Network distillation transfers knowledge from a more complex or differently trained model to a student, with soft scores identified as better than hard labels.
3. Our method
The method distills predictions from a model trained on a small clean dataset into training on the full dataset, combining noisy labels with auxiliary soft targets. It further uses label relations from a knowledge graph and provides analyses comparing the resulting pseudo-label risk with alternatives.
- Knowledge Distillation: The framework trains an auxiliary model on the small clean dataset, then transfers its predictions to guide a primary model trained on the entire dataset.The primary loss uses noisy labels while an imitation loss encourages the model to match the auxiliary output.
- Knowledge Distillation: The framework is designed without a particular label-noise assumption, addressing noise described as diverse, non-stationary, and multi-mode.Its rationale is that the clean-data model supplies an independent variance source that can help offset variance introduced by label noise.
- Knowledge Distillation: For cross-entropy loss, the method uses a pseudo-label that linearly combines the noisy label yi and auxiliary prediction si.The balance parameter λ controls the contribution of the noisy label versus the auxiliary output.
- Risk Analysis: Under the squared-loss analysis, an appropriately weighted combination of noisy labels and auxiliary soft predictions has lower optimal risk than either source alone.The analysis treats the combined pseudo-label risk as smaller than the risks associated with the unreliable label and the auxiliary model output.
- Comparison with Related Methods: The analysis compares the method with label smoothing and bootstrap, arguing that clean-data auxiliary predictions can outperform a uniform prior and avoid the high correlation of bootstrap targets with noisy labels.The label-smoothing comparison is favorable when the auxiliary-model risk is smaller than the uniform-distribution risk.
- Distillation Guided by Knowledge Graph: Knowledge-graph guidance propagates confidence among related labels to reduce over-certainty from an auxiliary model trained on a small clean set.The graph is represented by a normalized matrix whose entries encode relationships between labels, and its guided soft label replaces the ordinary auxiliary soft label.
4. Datasets and Evaluation
The paper introduces real-world noisy-label benchmarks across Sports, Species, and Artifacts, built from YFCC100M with text-based entity linking and partially clean data. Evaluation compares distillation methods with established baselines using qualitative rankings and mAP results.
- Datasets: The benchmark suite uses real-world label noise across three visual domains rather than simulated corruption alone.The datasets cover Sports, Species, and Artifacts and include background images in evaluation.
- Datasets: Images are collected from YFCC100M and labeled by linking titles and tags to Wikipedia entities with DBpedia Spotlight.Ambiguous terms are disambiguated using textual context, but the resulting links introduce label noise.
- Datasets: Partial clean datasets come from crowdsourcing for Sports and Species and ImageNet cross-linking for Species and Artifacts.Species-Y and Species-I distinguish whether clean data comes from YFCC100M or ImageNet.
- Datasets: Qualitative examples show weak association and text ambiguity as distinct sources of incorrect labels.Examples include event-viewing photos labeled as Abseiling and texture patterns labeled as Tulip.
- Evaluation: Distillation and graph-guided distillation consistently outperform the evaluated baselines, with guided pseudo labels ranking more true positives higher.The comparisons include clean and noisy baselines, ensemble models, bootstrapping, label smoothing, and other noise-handling methods.
5. Conclusion
The paper introduces a noisy-label learning framework that combines a small clean dataset with a semantic knowledge graph, and evaluates it on real-world noisy-label benchmarks. The proposed methods achieve superior performance, while future work considers additional guidance sources and other noisy-label settings.
- The framework leverages knowledge from a small clean dataset and semantic knowledge graph to correct noisy labels.
- The benchmark suite covers Species, Sports, and Artifacts and reflects real-world labeling noise.
- Future work will explore guidance sources beyond knowledge graphs and applications to other noisy-label scenarios, including Web-scale photo search.
A. Proof for Proposition 1
The proof decomposes the distillation risk using assumptions about the auxiliary model's bias and the independence of label corruption from model variance. Under these conditions, the mixed noisy-label and auxiliary predictions have a weighted risk whose minimizing coefficient depends on the two component risks.
- The proof analyzes an auxiliary model trained from a small clean dataset by decomposing its expected prediction error into variance and bias.
- For a high-capacity CNN under squared loss, the proof assumes the auxiliary model's bias term is close to zero.
- The analysis assumes the unknown label-corruption process is independent of model variance.
- Under these assumptions, the cross term between label noise and model deviation becomes zero.
- The mixed prediction risk decomposes into λ^2R_y + (1 − λ)^2R_s, with the optimal λ set to R_s/(R_s + R_y).