PaperlayerView exact revision on arXiv

Source-linked AI summary

UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning

Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, Mubarak Shah

arXiv:2203.14542v4cs.CVcs.LG

TL;DR

Noisy labels are difficult because existing selection methods disproportionately retain easy-class samples, creating class imbalance and weakening subsequent training. UNICON uses Jensen-Shannon-divergence-based uniform selection with unsupervised contrastive learning, and reports strong performance under severe noise, including an 11.4% CIFAR100 improvement at 90% noise. Its class-balance prior can be restrictive for extremely imbalanced datasets, and its advantage is less certain when label noise is low.

  • Problem

    Existing selection methods disproportionately select easy classes and reject harder ones, creating class imbalance that degrades noisy-label training, especially at high noise rates.

  • Method

    UNICON combines Jensen-Shannon-divergence-based uniform selection with unsupervised contrastive learning to balance selected clean samples and reduce noisy-label memorization.

  • Results

    UNICON improves CIFAR100 accuracy by 11.4% over the state of the art at a 90% noise rate and shows effectiveness across multiple noisy-label benchmarks.

  • Takeaways & Limitations

    Class-balanced selection and contrastive feature learning provide a practical approach that is particularly effective for training with severe label noise.

  • Takeaways & Limitations

    The class-balance prior can be restrictive for extremely imbalanced datasets, and UNICON may be outperformed when label noise is not substantial.

Abstract

from arXiv · show

Supervised deep learning methods require a large repository of annotated data; hence, label noise is inevitable. Training with such noisy data negatively impacts the generalization performance of deep neural networks. To combat label noise, recent state-of-the-art methods employ some sort of sample selection mechanism to select a possibly clean subset of data. Next, an off-the-shelf semi-supervised learning method is used for training where rejected samples are treated as unlabeled data. Our comprehensive analysis shows that current selection methods disproportionately select samples from easy (fast learnable) classes while rejecting those from relatively harder ones. This creates class imbalance in the selected clean set and in turn, deteriorates performance under high label noise. In this work, we propose UNICON, a simple yet effective sample selection method which is robust to high label noise. To address the disproportionate selection of easy and hard samples, we introduce a Jensen-Shannon divergence based uniform selection mechanism which does not require any probabilistic modeling and hyperparameter tuning. We complement our selection method with contrastive learning to further combat the memorization of noisy labels. Extensive experimentation on multiple benchmark datasets demonstrates the effectiveness of UNICON; we obtain an 11.4% improvement over the current state-of-the-art on CIFAR100 dataset with a 90% noise rate. Our code is publicly available

1. Introduction

UNICON addresses noisy-label learning by correcting class-imbalanced sample selection and adding contrastive feature learning to semi-supervised training. Its uniform selection and contrastive components target higher-quality pseudo-labels and improved robustness under severe noise.

  • Training Procedure: The method separates data into clean and noisy subsets before applying semi-supervised training, with two same-architecture networks repeating selection and training until convergence.Rejected samples are used as unlabeled data in the subsequent semi-supervised-learning stage.
  • Motivation: Label noise arises naturally from large-scale web annotation and can be memorized by deep networks, making noisy-label learning a fundamental challenge.Manual annotation is costly, while automatically annotated web data inevitably introduces noise.
  • UNICON: UNICON enforces class-balanced clean-sample selection and uses Jensen-Shannon divergence to simplify separation without requiring probabilistic modeling or extensive tuning.The method selects a fixed portion of samples from each class using a divergence-based cutoff.
  • UNICON: UNICON adds unsupervised contrastive feature learning to reduce noisy-label memorization and improve sample separation, especially under severe noise.Contrastive learning does not require labels and is therefore less exposed to label noise during feature learning.
  • Problem with Existing Selection: Existing sample-separation methods can produce severe class imbalance in the selected clean set, such as 1228 selected samples from one class versus 10 from another.False positives in the clean set can reduce precision and promote memorization during supervised training.

4. Proposed Method

UNICON combines Jensen-Shannon divergence-based uniform clean-sample selection with semi-supervised and contrastive learning. It balances selected samples across classes and uses contrastive objectives to improve separation under noisy labels.

  • Uniform Sample Selection: UNICON partitions data using JSD between ground-truth labels and predicted probabilities from two networks.JSD selection avoids normalization and probabilistic modeling, while its symmetric values range from 0 to 1.
  • Uniform Sample Selection: The method estimates a global filter rate R from a JSD-derived cutoff and selects the lowest-R portion within every class.This produces class-balanced clean and noisy partitions, with equal clean-sample allocation whenever each class has enough samples.
  • Uniform Sample Selection: The cutoff adapts to network prediction scores, eliminating manual per-dataset tuning and encouraging conservative clean-set selection when predictions are unreliable.High average divergence leads to a more conservative selection of clean samples early in training.
  • SSL-Training: Contrastive learning improves clean-sample detection precision by reducing memorization of noisy labels.The comparison evaluates ROC-AUC with and without contrastive learning on CIFAR10 with 90% noise.
  • SSL-Training: UNICON trains on separated sets with semi-supervised learning, MixUp, and contrastive loss using a projection head alongside classification.Noisy samples provide pseudo-label-based semi-supervised training, while contrastive learning uses differently augmented views and a cosine-similarity objective.

5. Experimental Settings

The experiments evaluate UNICON on synthetic and real-world noisy-label datasets using standard convolutional architectures and contrastive-learning modifications. Results are reported under symmetric and asymmetric noise settings.

  • Datasets: CIFAR10 and CIFAR100 contain 50K training and 10K test images, and experiments use symmetric and asymmetric synthetic noise models.Synthetic noise is used because noise characteristics are difficult to control in natural datasets.
  • Datasets: Tiny-ImageNet contains 200 classes with 500 images per class and 64 × 64 image resolution.It is described as a smaller version of ImageNet in classes and image resolution.
  • Datasets: Clothing1M contains 1M images across 14 clothing classes with substantial label confusion from seller-provided surrounding text.The paper identifies Knitwear and Sweater as examples of confusing classes.
  • Datasets: WebVision contains 2.4 million Flickr and Google images across 1,000 ImageNet classes, with experiments using the first 50 Google-image classes.The selected subset follows prior studies.
  • Training Details: PreAct ResNet18 is used for CIFAR10, CIFAR100, and Tiny-ImageNet, while Clothing1M and WebVision use ResNet50 or InceptionResNetV2.Architectures are modified with a 128-dimensional projection head for contrastive learning.
  • Training Details: CIFAR experiments use SGD with learning rate 0.02, weight decay 5e−4, momentum 0.9, batch size 64, and approximately 300 training epochs.The learning rate is decayed by 0.1 every 120 epochs, with dataset-specific warmup periods before selection and training stages.
  • Results: UNICON consistently improves CIFAR100 performance under different asymmetric noise settings.Table 3 reports experimental results on CIFAR10 and CIFAR100, with an asterisk marking runs performed by the authors.

6. Experimental Results

UNICON is evaluated across synthetic and real-world noisy-label datasets, noise levels, and ablations. Results show strong performance under severe noise, while selection precision, contrastive learning, class balancing, and ensembling materially affect outcomes.

  • Benchmark Results: UNICON is evaluated on CIFAR10, CIFAR100, TinyImageNet, Clothing1M, and WebVision under symmetric and asymmetric label noise.The experiments include symmetric noise rates of 20%, 50%, 80%, and 90%, and asymmetric rates of 10%, 30%, and 40%.
  • Benchmark Results: 11.4% accuracy improvement is obtained on CIFAR100 at a 90% noise rate.The improvement remains consistent across different noise settings, with superior asymmetric-noise performance against the listed baselines.
  • Benchmark Results: Around 1% performance improvement over state of the art is achieved on TinyImageNet across all noise rates.The comparison uses test accuracies under symmetric noise settings, including highest and average accuracy over the last 10 epochs.
  • Benchmark Results: 0.17% improvement over ELR is reported on Clothing1M, while WebVision gains around 1.5% over state of the art in Top-5 accuracy.On ILSVRC12, UNICON reports a 1.88% Top-5 improvement over DMix.
  • Ablation Studies: Improved selection precision increases classification accuracy by enabling better discriminative features and higher-quality pseudo-labels.Figure 5 links selection precision with ROC-AUC and test accuracy across noise settings.
  • Ablation Studies: Removing contrastive learning drops CIFAR10 and CIFAR100 test accuracy by 3.53% and 2.99%, respectively, at 90% noise.Contrastive learning is described as reducing noisy-label memorization, particularly under high noise.
  • Ablation Studies: Class balancing, contrastive learning, and ensembling outputs from both networks each significantly affect performance, especially at high noise rates.The ensemble procedure averages both networks’ predictions while alternating separation and training to limit confirmation bias.

7. Limitations of UNICON

UNICON’s class-balance prior can be restrictive for datasets with extreme class imbalance, and its advantage may be smaller when label noise is low.

  • Scope Boundaries: The class-balance prior can be restrictive when the dataset itself has extreme class imbalance.The authors note that the prior could be updated using the dataset’s class distribution, but do not explore that direction because the distribution may be unknown.
  • Scope Boundaries: UNICON is particularly effective under high label noise, so methods with more specialized training strategies may outperform it on datasets with little noise.The authors nevertheless characterize UNICON as a more general, simple solution that gives reasonable results at low noise rates.

8. Conclusion

UNICON combines uniform clean-sample selection with contrastive feature learning to combat noisy-label memorization and improve performance under severe noise.

  • UNICON reduces class disparity among selected clean samples through uniform selection, improving pseudo-label quality for noisy data.
  • UNICON uses contrastive feature learning to reduce memorization of noisy labels and improve sample separation.
  • UNICON achieves approximately 10% performance improvement over state-of-the-art methods on CIFAR10 and CIFAR100 with 90% noise.
  • Extensive empirical analysis evaluates UNICON across different noise scenarios.

9. Overview

The paper’s overview directs readers to the detailed SSL-training method, ablation findings, hyperparameter settings, and experimental results.

  • Section 10 details the semi-supervised learning training method.
  • Section 11 discusses the ablation studies and Section 13 presents hyperparameter settings and experimental results.

10. SSL-Training Details

UNICON separates labeled and unlabeled data, generates refined labels and pseudo-labels from weak augmentations, and trains with strong augmentations using semi-supervised and contrastive losses.

  • Uniform selection separates the training set into Dclean as labeled data and Dnoisy as unlabeled data before semi-supervised learning.
  • Weak augmentations support label refinement and pseudo-label generation, while strong augmentations update network parameters through backpropagation.
  • Label refinement combines the given label yi with prediction pi using coefficient wi, which is adjusted from the sample’s JSD value.
  • MixMatch combines strongly augmented labeled and unlabeled samples with ground-truth labels and pseudo-labels for semi-supervised training.
  • The semi-supervised objective combines labeled loss, unlabeled loss, and regularization, while contrastive loss is applied to Dnoisy samples.
  • Contrastive loss uses augmented pairs as positives and other mini-batch samples as negatives without requiring ground-truth or pseudo-labels.

11. Ablation Studies

Ablation and visualization analyses examine UNICON’s loss components, feature separation, class distributions, filtering behavior, and resistance to noisy-label memorization.

  • Removing any loss term decreases test accuracy, while the unlabeled loss LU has the largest impact on UNICON’s performance.
  • UNICON learns true class distributions under symmetric noise rates of 50%, 80%, and 90% and asymmetric noise of 40%.
  • UNICON’s loss-coefficient performance remains relatively stable across a broad range of λU and λC values.
  • The filtering rate increases as predictions become more confident and is selected from network predictions without manual tuning at each epoch.
  • Under 95% label noise, UNICON learns better class distributions than DMix, which the authors attribute to high-precision uniform clean-sample selection.
  • UNICON resists noisy-label memorization, with training accuracy saturating quickly instead of continuously improving as in standard training.

12. Baseline Methods

The paper compares UNICON with established noisy-label methods across CIFAR10, CIFAR100, and Clothing1M, while reporting largely dataset-shared hyperparameters.

  • UNICON is compared with LDMI, M-Up, PCIL, ELR, DMix, and MOIT on CIFAR10 and CIFAR100.
  • On Clothing1M, the comparison includes Joint-Optim, MetaCleaner, ELR, DMix, D2L, MentorNet, and Co-Teaching.
  • Most UNICON hyperparameters are shared across datasets, indicating general applicability of the method.

13. Training Details

Training details specify dataset-specific image preprocessing and note that UNICON is less competitive when label noise is relatively low, particularly on WebVision.

  • Most hyperparameters are kept the same across datasets.
  • Clothing1M images are resized to 256 × 256 and randomly cropped to 224 × 224.
  • WebVision images are resized to 320 × 320 and randomly cropped to 299 × 299, using 50 training and validation classes.
  • WebVision contains an estimated 20% noisy labels, and UNICON can underperform state-of-the-art methods in low-noise scenarios.
Loading 2203.14542v4…