Source-linked AI summary
Learning From Noisy Large-Scale Datasets With Minimal Supervision
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, Serge Belongie
TL;DR
The paper studies how to use a small clean annotation set alongside massive noisy image data, arguing that direct fine-tuning does not fully exploit the clean labels. It learns to clean noisy annotations jointly with image classification, then trains using the clean set and the larger reduced-noise dataset. On Open Images, the approach outperforms direct fine-tuning across all major class categories and is especially effective for classes with 20-80% false positive annotations.
Problem
Massive image datasets often have noisy or missing annotations while only a small subset receives costly human verification, limiting how clean labels are used.
Method
A multi-task network jointly learns a visual- and label-conditioned mapping from noisy to clean annotations and a multi-label image classifier.
Results
The approach outperforms direct fine-tuning across all eight major Open Images categories and is most effective for classes with 20-80% false positive annotations.
Takeaways & Limitations
Using clean annotations to reduce noise before training with the full dataset is more effective than direct fine-tuning for leveraging limited clean data.
Takeaways & Limitations
Evaluation must account for Open Images validation labels that are human-verified only for a subset of classes in each image.
Abstract
from arXiv · showhide
We present an approach to effectively use millions of images with noisy annotations in conjunction with a small subset of cleanly-annotated images to learn powerful image representations. One common approach to combine clean and noisy data is to first pre-train a network using the large noisy dataset and then fine-tune with the clean dataset. We show this approach does not fully leverage the information contained in the clean set. Thus, we demonstrate how to use the clean annotations to reduce the noise in the large dataset before fine-tuning the network using both the clean set and the full set with reduced noise. The approach comprises a multi-task network that jointly learns to clean noisy annotations and to accurately classify images. We evaluate our approach on the recently released Open Images dataset, containing ~9 million images, multiple annotations per image and over 6000 unique classes. For the small clean set of annotations we use a quarter of the validation set with ~40k images. Our results demonstrate that the proposed approach clearly outperforms direct fine-tuning across all major categories of classes in the Open Image dataset. Further, our approach is particularly effective for a large number of classes with wide range of noise in annotations (20-80% false positive annotations).
1. Introduction
The paper addresses multi-label image classification when massive datasets have noisy annotations but only a small subset is cleanly verified. It uses clean labels to learn annotation cleaning, then trains with both clean data and the larger dataset after noise reduction.
- Motivation: Massive cleanly annotated image collections remain a bottleneck for scaling ConvNet learning.The paper notes that publicly available datasets have not provided an order of magnitude more clean labeled data than ImageNet.
- Motivation: Typical web-mining settings combine noisy or missing annotations with a small fraction of images sent for costly human verification.This motivates treating the problem as semi-supervised rather than assuming every annotation is noisy and no clean data exists.
- Approach: Clean labels supervise annotation cleaning, while the classifier uses clean data and the much larger noisy dataset after cleaning.The approach therefore differs from simply fine-tuning a noisy-data network on the small clean set.
- Approach: The proposed multi-task network learns a mapping from noisy to clean annotations while jointly learning an accurate image classifier.The mapping is conditioned on visual features and noisy labels, allowing it to model annotation noise and label-space structure.
- Results: The method improves performance over direct fine-tuning across all eight high-level Open Images categories and across the full range of label-noise levels.It is most effective for classes with 20-80% false positive annotations in the training set.
2. Related Work
Prior work includes noise-robust learning, label cleansing, semi-supervised learning, transfer learning, and student-teacher models. The proposed approach differs by jointly using image features, noisy labels, and a small clean set for multi-label annotation cleaning and classification.
- Noise and semi-supervised learning: Noise-learning methods either directly learn from noisy labels or cleanse mislabeled data, while semi-supervised methods combine labeled and unlabeled information.These approaches face challenges such as distinguishing difficult examples from mislabeled ones.
- Neural noise models: Several neural noise models assume noise is conditionally independent of the input image, limiting their ability to distinguish visual modes and related noise.The proposed model instead conditions cleaning on image features and supports multiple labels per image.
- Multi-label noise: Unlike methods focused only on missing labels, the proposed approach uses cleaned labels and can address both incorrect and missing annotations.This distinction is specific to the multi-label setting described by the paper.
- Transfer learning: Unlike conventional transfer learning, the approach trains from scratch with noisy labels and then uses clean labels to support fine-tuning.The clean set is used to learn annotation mappings rather than only visual representations directly.
- Student-teacher models: The framework resembles student-teacher and model-compression setups, but its cleaning network and image classifier are trained jointly in one model.The cleaning network also receives noisy labels alongside visual features as privileged information.
3. Our Approach
The approach jointly cleans noisy multi-label annotations using a small verified set and trains an image classifier on the cleaned labels and verified labels. A residual, image-conditioned cleaning network preserves noisy labels by default while adapting as verified supervision increases.
- 3. Our Approach: The method uses a large noisy dataset T and a much smaller verified dataset V to train a multi-label image classifier.The datasets differ significantly in size, with |T| ≫ |V|; in the experiments, T exceeds V by three orders of magnitude.
- 3.1. Multi-Task Label Cleaning Architecture: A multi-task network jointly learns to clean noisy labels and classify images accurately.The label-cleaning network g maps noisy labels and image features to cleaned labels, while classifier h predicts labels from images.
- 3.1. Multi-Task Label Cleaning Architecture: The cleaning network conditions on noisy labels and visual features, combining label-space structure with image information.Noisy labels are embedded separately from visual features, then the modalities are concatenated and projected back into the label space.
- 3.1. Multi-Task Label Cleaning Architecture: An identity-skip connection makes cleaning residual: the network learns differences between noisy and verified labels rather than regressing the full label vector.With no human-rated data, the cleaning network defaults to leaving noisy labels unchanged; outputs are clipped to the valid [0,1] label space.
- 3.2. Model Training: The classifier uses cleaned labels for noisy samples and verified labels for samples in V, with cross-entropy supporting multiple annotations per image.The jointly optimized losses include a cleaning loss on V and a classification loss over noisy and verified samples.
- 3.2. Model Training: Training batches use a 9:1 ratio of noisy-dataset samples to verified-dataset samples.This ratio uses the large number of samples in T while providing supervision to the cleaning network from V.
4. Experiments
The experiments evaluate noisy multi-label classification on Open Images using AP/MAP metrics and a small human-verified subset. The proposed approach consistently improves over baseline and direct fine-tuning, especially for frequent classes and medium annotation noise.
- Dataset and evaluation: Open Images contains 9,011,219 training images, highly uneven class frequencies, and substantial annotation noise estimated at 26.6% false positives.A quarter of the validation set, about 40 thousand images, supplies noisy and human-verified labels for cleaning; the remainder is held out for validation.
- Dataset and evaluation: Average precision is reported per class, with MAP averaging AP across 6,012 classes and APall weighting every annotation equally.Classes without human verification are ignored in the metrics because many highly ranked unverified annotations may be correct.
- Overall results: All proposed variants improve APall over the noisy-label baseline, and both proposed variants significantly outperform direct fine-tuning.The two proposed variants perform very similarly despite differing in whether the cleaning network is pre-trained.
- Overall results: Direct fine-tuning on clean data hurts MAP, while the proposed approach significantly improves over both baseline and direct fine-tuning across classes.Mixing clean and noisy annotations reduces overfitting but still incurs an overall-performance cost; the proposed approach avoids this overfitting problem.
- Label frequency and annotation quality: The approach improves performance across almost all class-frequency levels, with the largest gains for very common classes and improvement for most rare classes.Figure 5 groups the 6,012 classes into 10 equally sized frequency groups.
- Label frequency and annotation quality: Performance improves across all annotation-quality levels, with the largest gains for classes containing 20% to 80% false annotations.Very noisy classes provide little positive supervision, whereas medium-noise classes contain sufficient negative and positive human-rated examples for improvement.
- High-level categories: The method clearly improves over baseline and direct fine-tuning across all major Open Images categories.For products, improvements mainly occur among high-confidence labels; for animals, gains occur mainly in the lower-confidence regime.
5. Conclusion
The approach uses clean labels to reduce noise in a massive dataset before fine-tuning, outperforming direct fine-tuning across major class categories. Future work could increase cleaning-network capacity and map noisy labels across different domains.
- Clean labels reduce noise in the large dataset before fine-tuning with both clean labels and the reduced-noise dataset.
- The approach outperforms direct fine-tuning across all major categories of classes.
- Future work could explore higher-capacity interactions, such as bilinear pooling, in the cleaning network.
- Future work could map noisy labels in one domain into clean labels in another, such as Flickr tags to object categories.