Source-linked AI summary
CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R. Scott, Dinglong Huang
TL;DR
Large-scale web images provide cheap supervision but contain massive noisy labels, creating a challenge for training deep networks without manual annotation. CurriculumNet uses an unsupervised density-based curriculum to introduce data from easier to more complex subsets, achieving state-of-the-art results across four benchmarks and a 5.2% top-5 WebVision error with an ensemble.
Problem
Web images can scale training data without human annotation, but their text-query labels are highly unreliable and massively noisy, complicating deep-network training.
Method
CurriculumNet ranks image complexity using feature-space distribution density and gradually trains CNNs by adding increasingly noisy data.
Results
CurriculumNet achieves state-of-the-art performance on WebVision, ImageNet, Clothing-1M, and Food-101, including a 5.2% top-5 error rate on WebVision with an ensemble.
Takeaways & Limitations
Highly noisy images can improve model generalization when incorporated through the proposed curriculum, rather than merely being discarded.
Abstract
from arXiv · showhide
We present a simple yet efficient approach capable of training deep neural networks on large-scale weakly-supervised web images, which are crawled raw from the Internet by using text queries, without any human annotation. We develop a principled learning strategy by leveraging curriculum learning, with the goal of handling a massive amount of noisy labels and data imbalance effectively. We design a new learning curriculum by measuring the complexity of data using its distribution density in a feature space, and rank the complexity in an unsupervised manner. This allows for an efficient implementation of curriculum learning on large-scale web images, resulting in a high-performance CNN model, where the negative impact of noisy labels is reduced substantially. Importantly, we show by experiments that those images with highly noisy labels can surprisingly improve the generalization capability of the model, by serving as a manner of regularization. Our approaches obtain state-of-the-art performance on four benchmarks: WebVision, ImageNet, Clothing-1M and Food-101. With an ensemble of multiple models, we achieved a top-5 error rate of 5.2% on the WebVision challenge for 1000-category classification. This result was the top performance by a wide margin, outperforming second place by a nearly 50% relative error rate. Code and models are available at: https://github.com/MalongTech/CurriculumNet .
1 Introduction
CurriculumNet addresses the cost and unreliability of clean manual annotation by training standard CNNs on large-scale web images with a curriculum for noisy labels. It achieves state-of-the-art results across four benchmarks, including a 5.2% top-5 error rate on WebVision with an ensemble.
- Motivation: Clean, massive annotations are expensive and difficult to obtain, while web queries offer scalable but unreliable supervision with substantial label noise.The difficulty is especially pronounced for expert-dependent tasks and inconsistent annotator labels.
- Approach: CurriculumNet improves standard neural networks through a new training strategy rather than noise-cleaning, noise-robust, or semi-supervised algorithms.The approach is designed for large-scale images with massive noisy labels and data imbalance.
- Approach: The method uses curriculum learning to train high-performance CNNs from web images collected without human annotation.Its curriculum is designed specifically to handle massive noisy labels.
- Approach: The curriculum ranks data complexity using distribution density in feature space through an unsupervised procedure.This enables curriculum learning tailored to highly noisy web data.
- Results: State-of-the-art performance is reported on WebVision, ImageNet, Clothing1M, and Food101.With multiple models ensembled, CurriculumNet achieved a top-5 error rate of 5.2% on the WebVision Challenge.
2 Related work
Prior work addresses noisy web data through noise-robust learning, label cleansing, semi-supervised learning, or transfer learning. CurriculumNet instead changes the training strategy and uses an unsupervised curriculum that directly incorporates noisy examples.
- Noisy-label methods: Noise-robust and label-cleansing methods aim to learn from noisy labels or remove incorrect data, but identifying mislabeled samples separately from useful hard samples remains difficult.Hard samples can be important for improving model capability.
- Alternative approaches: Semi-supervised methods combine noisy labels with manually labeled subsets, while transfer learning transfers label correctness to other classes.These approaches use additional supervision or label-transfer mechanisms.
- CurriculumNet: CurriculumNet does not propose noise-cleansing, noise-robust, or semi-supervised learning; it improves standard neural networks through a new training strategy.The strategy is intended to alleviate the negative impact of noisy labels.
- CurriculumNet: The method differs from related CNN approaches by directly considering mislabeled samples in its training curriculum.Its curriculum is designed in a completely unsupervised manner.
3 Methodology
CurriculumNet ranks web images by feature-space density and trains CNNs progressively from cleaner to noisier subsets. This unsupervised curriculum combines density-based clustering, staged learning, and data-balancing strategies for large-scale noisy web data.
- Curriculum Design: The method first trains an initial model, projects images into a deep feature space, and uses their relationships to design the curriculum.The pipeline consists of initial feature generation, curriculum design, and curriculum learning.
- Curriculum Design: Local density estimates how many nearby samples surround each image, with dense groups interpreted as visually similar and likely cleaner.The distance threshold d_c is selected from sorted pairwise distances, using k = 60 in the experiments.
- Curriculum Design: Images are clustered by their distances to a high-density center, producing subsets ranked from easier, more reliable data to harder, noisier data.The clustering procedure uses the highest-density point as the category’s cluster center and applies k-means according to distances to that center.
- Curriculum Learning: Training proceeds through three stages, beginning with clean data and progressively adding noisy and highly noisy subsets.The first stage learns clear category features; later stages add harder samples while continuously mixing the training subsets.
- Curriculum Learning: Highly noisy data can improve generalization and regularize the model by reducing overfitting to clean data.The paper reports that the final highly noisy stage does not negatively affect the learned data structure.
- Implementation: The implementation addresses class imbalance with selective category-level sampling and uses multi-scale kernels that improve WebVision top-5 error by about 0.5%.Category-level balancing is applied only to the clean subset; applying it to noisier subsets reduced performance.
4 Experimental Results and Comparisons
Experiments evaluate CurriculumNet across noisy-data benchmarks, training curricula, noise levels, clustering choices, architectures, and validation settings. The method consistently improves performance, including a 5.2% WebVision Top-5 error with six-model ensembling.
- Datasets: CurriculumNet is evaluated on WebVision, ImageNet, Clothing1M, and Food-101, including web-scale and artificially noisy training settings.WebVision contains 2,439,574 noisily labeled training images without human annotation; Food-101 experiments add 20% randomly labeled noise.
- Training strategies: 30.16% →27.91% Top-1 error and 12.43% →10.82% Top-5 error result from using the 3-subset curriculum instead of training on all data.The curriculum models also show better convergence rates, and improvements occur in 668 of 1,000 categories.
- Label noise: 50% of the highly noisy subset produces the best Top-1 and Top-5 results, while adding more yields only very limited negative effect.The authors interpret highly noisy data as increasing training diversity and improving generalization under the proposed strategy.
- Label noise: 7.7% is the largest performance improvement for categories whose training-label correct rates fall in the 10%-20% interval.Categories with correct rates below 40% have performance gains greater than 4%.
- Benchmark comparisons: 5.2% Top-5 error is achieved on the WebVision challenge by an ensemble of six models, while CurriculumNet improves results across all four databases.Reported comparisons include CleanNet improvements of about 1.5% to 3.3% and a WebVision Top-5 reduction from 12.2% to 10.8%.
5 Conclusion
CurriculumNet trains CNNs on large-scale weakly supervised web images without human annotation by using curriculum learning based on cluster density. It reduces the negative impact of noisy labels, improves generalization, and achieves state-of-the-art results across four benchmarks.
- CurriculumNet trains CNN models more efficiently on weakly supervised web images without human annotation.
- The method measures data complexity using cluster density to construct a curriculum for training.
- CurriculumNet reduces the negative effect of massive noisy labels while improving model generalization through highly noisy data.
- The approach achieves state-of-the-art performance on WebVision, ImageNet, Clothing-1M, and Food-101.
- 5.2% Top 5 error was achieved on the WebVision Challenge 2017 using an ensemble of multiple models.