PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, Yang Liu

arXiv:2110.12088v2cs.LGstat.ML

TL;DR

Learning with noisy labels is difficult to evaluate because synthetic noise may not reflect human annotations, while many real-world datasets lack clean verification or require large computational resources. The paper constructs CIFAR-10N and CIFAR-100N from Amazon Mechanical Turk annotations, finding feature-dependent human noise and consistent robustness gaps relative to synthetic noise. These benchmarks support more controlled evaluation of noisy-label methods.

  • Problem

    Real-world noisy-label datasets often lack clean-label verification or are large enough to complicate fair comparisons, while synthetic noise may not capture real-world noise patterns.

  • Method

    The paper builds CIFAR-10N and CIFAR-100N by adding human annotations to CIFAR training images and benchmarks popular noisy-label methods on these datasets.

  • Results

    Human noise is feature-dependent and differs substantially from synthetic class-dependent noise, with consistent performance gaps across the evaluated settings.

  • Takeaways & Limitations

    CIFAR-10N and CIFAR-100N provide accessible, verifiable benchmarks for evaluating and developing learning-with-noisy-label solutions.

  • Takeaways & Limitations

    Large-scale, high-resolution settings can make robust-method comparisons unfair because data complexity, augmentation, extra data, and computation jointly affect performance.

Abstract

from arXiv · show

Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at http://noisylabels.com.

1 INTRODUCTION

Existing noisy-label benchmarks rely heavily on synthetic noise or large datasets lacking clean verification, limiting how faithfully and fairly real-world noise can be studied. The paper introduces accessible CIFAR-based human-annotation benchmarks and reports that human noise differs from synthetic noise in distribution and method robustness.

  • Motivation: Existing noisy-label research commonly uses synthetic benchmarks, while real-world datasets often lack corresponding clean labels for controlled verification.This limits evaluation of noise levels and robust methods on many large-scale benchmarks.
  • Motivation: Large-scale, high-resolution real-world datasets can confound method comparisons through data complexity, augmentation, extra data, and computation differences.These factors jointly contribute to model performance and can make comparisons unfair under accessible computation budgets.
  • Contributions: CIFAR-10N and CIFAR-100N equip standardized CIFAR training datasets with human-annotated noisy labels as accessible, easy-to-use benchmarks.The datasets are intended to support validation of existing methods and future development.
  • Contributions: Human annotations exhibit imbalanced labels, flips among similar features, and co-existing clean labels in CIFAR-100, distinguishing them from synthetic class-dependent noise.The paper compares transition patterns qualitatively and quantitatively using hypothesis testing.
  • Contributions: Popular robust methods show consistent performance gaps between human and synthetic noise, while memorization behavior further distinguishes the two settings.The benchmark includes empirical comparisons on CIFAR-10N, CIFAR-100N, and synthetic noisy CIFAR datasets.

2 SYNTHETIC LABEL NOISE

The paper formalizes noisy labels through a transition matrix and reviews synthetic models that assume either feature-independent or instance-dependent label noise. Class-dependent models include symmetric and asymmetric transition structures, while newer models allow features to influence noise.

  • Transition-matrix formulation: A noise transition matrix T(X) models the probability of observing noisy label j given clean label i and feature X.Its elements are defined as Ti,j(X) := P(eY = j|Y = i, X).
  • Class-dependent label noise: Class-dependent noise assumes the transition matrix is independent of features, so T(X) ≡ T.Under this assumption, noise depends conditionally on the clean class rather than X.
  • Symmetric noise: Symmetric noise keeps Ti,i = 1 − ϵ and assigns every off-diagonal entry Ti,j = ϵ/(K−1).This models random flipping from a clean class to any other class with equal probability.
  • Asymmetric noise: Asymmetric-next noise flips class i to (i + 1) mod K with probability ϵ, whereas asymmetric-pair noise assigns ϵ to disjoint class pairs in both directions.For asymmetric-pair noise, diagonal entries are 1 − ϵ.
  • Instance-dependent noise: Instance-dependent noise is jointly determined by features and clean labels, including margin-, part-, and group-dependent synthetic models.These models represent cases where decision-boundary proximity, feature parts, or sub-populations affect transition behavior.

3 HUMAN ANNOTATED NOISY LABELS ON CIFAR-10, CIFAR-100

CIFAR-10N and CIFAR-100N add human annotations to CIFAR training images through Amazon Mechanical Turk while retaining the standardized CIFAR datasets. They provide multiple noisy-label configurations with measurable noise levels across fine and coarse labels.

  • Dataset construction: CIFAR-10N and CIFAR-100N are human-annotated noisy-label benchmarks built from CIFAR training images collected through Amazon Mechanical Turk.Workers completed paid annotation tasks for CIFAR-10 and CIFAR-100 images.
  • CIFAR-10N: CIFAR-10 contains 60k 32 × 32 color images across ten mutually exclusive classes, with 50k training and 10k test images.The test dataset remains unchanged during CIFAR-10N collection.
  • CIFAR-10N: Each CIFAR-10N training image has one clean label and three human annotations, yielding Aggregate, Random 1–3, and Worst noisy-label sets.Aggregate uses majority voting, while the random sets retain individual submissions; the Worst set selects a wrong label when available.
  • CIFAR-10N: 9.03% (Aggregate), 17.23% (Random 1), 18.12% (Random 2), 17.64% (Random 3), and 40.21% (Worst) are the CIFAR-10N noise rates.60.27% of training images received unanimous labels from three independent annotators.
  • CIFAR-100N: CIFAR-100N provides coarse and fine human labels for CIFAR-100 images, collected through super-class selection followed by fine-label selection.Workers receive example images and can re-select the super-class during annotation.
  • CIFAR-100N: CIFAR-100N has overall coarse-label and fine-label noise levels of 25.60% and 40.20%, respectively.Most batches contain approximately 25% noisy coarse labels and 40% noisy fine labels.

4 PRELIMINARY OBSERVATIONS ON CIFAR-10N, CIFAR-100N

Human annotations on CIFAR-10N and CIFAR-100N exhibit imbalanced, feature-related, and sometimes multi-label noise patterns that differ from simple class-dependent models. Qualitative visualization and hypothesis testing support feature dependence in human label noise.

  • Imbalanced annotations: Human annotators produce imbalanced label distributions despite equal clean-class image counts, with “Man” receiving ≥750 annotations and “Streetcar” approximately 200.CIFAR-10N also shows preferences for some classes, while 25% of CIFAR-100N noisy labels fall outside the clean coarse class.
  • Similar-feature label flips: CIFAR-100N errors usually concentrate on fewer than four fine classes, often between visually similar categories such as snake–worm or cockroach–beetle.Other confusions span related groups including boy, baby, girl, and man.
  • Noise transition patterns: CIFAR-10N transition matrices show frequent flips among similar classes, including approximately 25%–30% bidirectional truck–automobile confusion, while other classes follow low-rate symmetric noise.CIFAR-100N exhibits the same tendency for each class to flip disproportionately into a few misleading classes.
  • Co-existing labels: Some CIFAR-100N images contain multiple valid class concepts, so an annotation judged wrong against the clean label may nevertheless correspond to content present in the image.Figure 4 provides exemplary training images with clean labels and human noisy labels shown separately.
  • Feature-dependent noise analysis: The analysis estimates locally homogeneous transition behavior by clustering same-class representations into five k-means groups under an M-NN noise-clusterability assumption.Transition vectors are estimated within each cluster to compare human and synthetic noise.

5 LEARNING WITH CIFAR-10N AND CIFAR-100N

The paper benchmarks robust learning methods on human-annotated CIFAR-N and matched synthetic noise, finding substantial performance differences and distinct memorization behavior.

  • Performance comparisons: Popular robust methods are evaluated on collected human labels and synthetic noisy labels using standardized ResNet-34 training procedures.The comparison covers loss correction, loss re-weighting, and loss regularization methods, with identical training procedures and batch sizes for fairness.
  • Performance comparisons: Human noise creates a larger learning challenge than class-dependent synthetic noise for most methods, especially on CIFAR-10 at high noise levels.The gap is less obvious on CIFAR-100.
  • Performance comparisons: ELR performs slightly better on real-world human noise than on matched synthetic class-dependent noise, while Divide-Mix performs poorly in the low-noise regime.These method-specific patterns qualify the general performance-gap observation.
  • Memorization: Memorization is analyzed by separating training images whose annotations match clean labels from those with wrong labels.A feature is considered memorized when the classifier assigns some class probability above the confidence threshold η.
  • Memorization: Figure 6 compares memorized wrongly predicted samples with correctly predicted samples across three CIFAR-10N noisy label sets and synthetic noise with the same T.Red denotes wrongly predicted samples and blue denotes correctly predicted ones.

6 CONCLUSIONS

The paper concludes that CIFAR-10N and CIFAR-100N provide accessible human-noise benchmarks and reveal noise patterns and memorization behavior that differ from synthetic noise.

  • Conclusions: CIFAR-10N and CIFAR-100N provide accessible benchmarks with human-annotated noisy labels for weakly supervised learning.The datasets are built on CIFAR-10 and CIFAR-100 and are intended to support future development and evaluation.
  • Conclusions: Human annotations show imbalanced labels, flips among similar features, and co-existing labels in CIFAR-100N.The paper identifies these as observations from human annotations and as a new, largely neglected label-noise pattern.
  • Conclusions: Human noise is qualitatively and quantitatively distinguished from synthetic class-dependent noise through noise-transition analysis and hypothesis testing.The paper presents hypothesis testing as a template for testing simulated human-label noise patterns.
  • Conclusions: Benchmarking reveals a large performance gap between human and synthetic noise, accompanied by different memorization behavior on training samples.These observations are reported consistently across the benchmark comparisons.
  • Broader impacts: CIFAR-N may also support studies of human-annotation simulation in crowdsourced computer-vision labeling.The paper connects this use to the substantial expense of obtaining human annotations.

A.4 MORE DETAILED DATASET STATISTICS

Detailed CIFAR-10N statistics show batch difficulty variation and systematic annotator preferences, while CIFAR-100N annotation is harder because of fine-grained classes and limited background knowledge.

  • CIFAR-10N statistics: CIFAR-10N batches differ in annotation difficulty, with Batch3 showing a clearly higher noise rate than Batch4 and Batch5, especially for the Worst label set.The table summarizes consensus and noise levels for each noisy label set across ten batches.
  • CIFAR-10N statistics: Annotators prefer some visually similar CIFAR-10 classes, producing more automobile and ship labels and fewer deer and cat labels.The clean-label counts are equal across classes, so the frequency differences reflect annotation preferences.
  • CIFAR-100N statistics: CIFAR-100 annotation is hard and time-consuming because workers must distinguish among 100 possible fine labels.A case study reports an average of 5–6 minutes to find a label per image.
  • CIFAR-100N statistics: Fine-grained CIFAR-100 classes require background knowledge, making super-class errors and confusions among related labels common.Examples include aquatic mammals versus fish and fine distinctions among trees and medium-sized mammals.
  • CIFAR-100N statistics: Aggregating three independent CIFAR-100 labels reduces noise rates less than it does on CIFAR-10N.The paper reports that label aggregation has relatively few effects for CIFAR-100N.

B.2 DATASET COLLECTION

CIFAR-100N collection uses staged annotation: workers first choose among 20 super-classes, then select from corresponding fine labels with visual references.

  • Dataset collection: CIFAR-100 training images are divided into ten batches of 1,000 HITs, with five enlarged 96 × 96 images per HIT and one worker assigned to each HIT.The images are reshaped from the original 32 × 32 CIFAR-100 training images.
  • Dataset collection: The annotation interface groups 100 fine classes into 20 super-classes before directing workers to the corresponding 4–6 fine labels.This staged process addresses the difficulty of selecting directly from 100 classes.
  • Dataset collection: Workers receive example images for every fine label and can re-select a super-class when the initial choice is difficult to recognize.The interface also provides navigation controls to move efficiently to the next image.

B.3 MORE DETAILED DATASET STATISTICS

CIFAR-100N exhibits substantial, imbalanced, and feature-related annotation noise, while its collection procedure differs from CIFAR by assigning workers to label existing images. CIFAR-100N labels frequently move among visually similar classes and combine coarse and fine annotations.

  • Noise levels: CIFAR-100N has 25.60% noisy coarse labels and 40.20% noisy fine labels overall.Most batches contain approximately 25% noisy coarse labels and 40% noisy fine labels.
  • Noise levels: Human annotators place 25% of noisy labels outside the clean super-class and 15% inside it.These figures describe the distribution of noisy annotations relative to the clean coarse label.
  • Annotation distributions: Noisy annotations are imbalanced: “Man” appears at least 750 times, whereas “Streetcar” has approximately 200 annotations.Figure 9 compares noisy-label counts across clean fine classes, each of which has 500 training images.
  • Label transitions: Low-resolution images produce label flips among similar classes, including approximately 20% exchanges between “snake” and “worm.”Other examples include “cockroach”-“beetle” and groups such as “boy”-“baby”-“girl”-“man.”
  • Label transitions: CIFAR-100N transitions combine similar-class flips with low-rate symmetric noise among remaining classes.The transition matrices cover both coarse and fine noisy labels relative to clean labels.
  • Collection procedure: CIFAR-N labels existing CIFAR images through paid Mechanical Turk annotation, whereas CIFAR labels originated from class-conditioned image searches.CIFAR-N provides noisy-label sets with varied noise rates, unlike CIFAR’s negligible verified noise rate.

C.2 COMPARISONS BETWEEN CIFAR-10H AND CIFAR-N

CIFAR-10H was designed to study uncertainty in human annotations using controlled collection procedures, whereas CIFAR-N targets broader real-world annotation noise. The datasets therefore differ in noise rates, interventions, and suitability for evaluating noisy-label methods.

  • Collection controls: CIFAR-10H used no time limits, assigned 200 images per participant, removed participants below 75% accuracy, and collected 47–63 annotations per image.Label positions were shuffled, and payment was 0.0075 per 10 images.
  • Dataset purpose: CIFAR-10H targets the benefits of richer human label distributions, from hard labels to soft labels.Its central query concerns whether annotation uncertainty improves model generalization.
  • Interpretation: CIFAR-10H’s checks and removals may prevent it from fully revealing real-world human annotation noise.CIFAR-N rejects only uninformative or spam-like patterns and does not restrict worker accuracy or annotation volume in the same way.
  • Noise rates: CIFAR-10H contains approximately 5% wrong labels, compared with around 18% in CIFAR-10N and 40.20% in CIFAR-100N.The text attributes CIFAR-10H’s lower rate to collection controls and different collection objectives.
  • Evaluation scope: Training on CIFAR-10 test data yields about 83% generalization accuracy on training data, versus about 93% under standard training and testing.The reduced training-data size limits evaluation of competing noisy-label methods and their theoretical properties.
  • Hypothesis testing: For CIFAR-10N, human-noise comparisons reject the null for nearly every clean class, with most classes reaching p < 0.01.The automobile class is the stated exception, and three classes do not reach p < 0.01.

D.2 HYPOTHESIS TESTING OF CIFAR-100N

The CIFAR-100N tests compare human noise with feature-independent and synthetic-noise assumptions using transition vectors and t-tests. They reject feature independence overall, while indicating that roughly half the classes may still behave class-dependently.

  • Hypotheses: The CIFAR-100N null hypothesis treats human annotation noise as feature-independent, while the alternative treats it as feature-dependent.The testing setup explicitly contrasts these two hypotheses.
  • Testing procedure: The synthetic-noise comparison tests whether human transition vectors match corresponding feature-independent synthetic transitions.The method compares transition-vector distances across repeated augmented versions using two-sided t-tests at α = 0.05.
  • Results: Approximately 50 CIFAR-100N classes have p-values below α = 0.05, rejecting feature independence for the dataset overall.The reported conclusion is that human annotation noise in CIFAR-100N is feature-dependent.
  • Results: After removing the class constraint, the test reports p = 5.2e−16 for the whole data.The figure reports significance conventions from ns through **** for per-class results.
  • Beyond feature dependency: Around 50 classes can nevertheless be viewed as class-dependent according to their significance levels.Thus, feature dependence is not uniform across all CIFAR-100N classes.
  • Benchmark context: ELR+ and DivideMix substantially outperform other listed methods on CIFAR-100N, especially at high noise levels.Both methods use two networks with advanced strategies such as mix-up augmentation.

E.2 PERFORMANCE COMPARISONS ON SYNTHETIC CIFAR DATASETS

Synthetic CIFAR benchmarks use class-dependent transition matrices and a common ResNet-34 training procedure to compare methods. Most methods find this synthetic noise easier on CIFAR-10, while the human-versus-synthetic gap is less obvious on CIFAR-100.

  • Experimental setup: Synthetic CIFAR-10 and CIFAR-100 labels are generated with the same class-dependent noise transition matrices used for CIFAR-10N and CIFAR-100N.Test accuracies are compared under a common ResNet-34 procedure and batch size.
  • Results: Most methods learn class-dependent synthetic noise much more easily on CIFAR-10, particularly at high noise levels.The comparison concerns test accuracies on the synthetic CIFAR datasets.
  • Results: The performance difference between human noise and synthetic noise is less obvious for CIFAR-100.This is the reported cross-dataset contrast with CIFAR-10.
  • Implementation: Experiments use minibatches of 128, SGD, initial learning rate 0.1, momentum 0.9, weight decay 0.0005, and 100 epochs.The learning rate decays by 0.1 at 50 epochs, with standard data augmentation.
  • Implementation: Special treatments preserve default settings for Cores*, ELR+, and DivideMix and use a shared pretrained model where required.These treatments account for advanced augmentation, two-network methods, and warm-up or pretrained-model requirements.
Loading 2110.12088v2…