PaperlayerView exact revision on arXiv

Source-linked AI summary

DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Junnan Li, Richard Socher, Steven C. H. Hoi

arXiv:2002.07394v1cs.CV

TL;DR

Deep networks require costly annotations, while noisy labels can cause overfitting and unlabeled-data methods have been insufficiently connected to noisy-label learning. DivideMix dynamically separates likely clean and noisy samples with a mixture model, trains two diverged networks using cross-network divisions, and adapts MixMatch with co-refinement and co-guessing. Across multiple benchmarks, it consistently achieves substantial improvements over state-of-the-art methods, including roughly 10% accuracy gains on highly noisy CIFAR-100 and over 12% top-1 improvement on WebVision.

  • Problem

    Expensive annotation and noisy labels make robust deep-network training difficult, while the connection between noisy-label learning and semi-supervised learning remains underexplored.

  • Method

    DivideMix fits a mixture model to per-sample losses, cross-trains two networks using dynamically divided clean and noisy data, and applies label co-refinement and co-guessing with MixMatch.

  • Results

    DivideMix consistently improves over state-of-the-art methods across benchmark datasets, with approximately 10% accuracy improvement on highly noisy CIFAR-100 and more than 12% top-1 improvement on WebVision.

  • Takeaways & Limitations

    Likely noisy samples can be used as unlabeled data, while cross-network division and ensemble inference support DivideMix’s reported robustness to label noise.

  • Takeaways & Limitations

    The loss-based clean/noisy separation depends on assumptions about loss behavior and requires warm-up modifications because asymmetric noise can make GMM separation ineffective.

Abstract

from arXiv · show

Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at https://github.com/LiJunnan1992/DivideMix .

1 INTRODUCTION

DivideMix connects learning with noisy labels and semi-supervised learning by treating likely noisy samples as unlabeled data. It dynamically separates samples and uses co-training and modified MixMatch to improve robustness to label noise.

  • Motivation: High-quality annotation is expensive, motivating methods that learn from noisily labeled or unlabeled data.Alternative data sources reduce annotation cost but inevitably introduce noisy labels, which deep networks can overfit.
  • Motivation: Existing noisy-label methods mainly correct losses, but transition-matrix estimation is difficult and prediction-based correction can overfit under high noise.These approaches rely on estimating noise structure or model predictions, both of which become unreliable in challenging noise settings.
  • Motivation: Semi-supervised learning leverages unlabeled samples through entropy minimization, consistency regularization, and unified methods such as MixMatch.MixMatch combines several dominant semi-supervised learning approaches and achieves strong performance.
  • DivideMix: DivideMix addresses the underexplored connection between noisy-label learning and semi-supervised learning by discarding likely noisy labels and using those samples as unlabeled data.The unlabeled samples regularize training against overfitting and support generalization.
  • DivideMix: Co-divide trains two networks that use Gaussian Mixture Models to separate mostly clean labeled samples from mostly noisy unlabeled samples for the other network.The framework also applies label co-refinement to labeled samples and label co-guessing to unlabeled samples during improved MixMatch training.

2 RELATED WORK

Prior work largely corrected losses or used semi-supervised regularization, while DivideMix instead repurposes likely noisy samples as unlabeled data. Its framework combines this strategy with two-network filtering and modified MixMatch.

  • Learning with Noisy Labels: Noisy-label methods commonly correct losses explicitly or implicitly, but some relabeling approaches require a small clean set.The related methods include transition-matrix correction, relabeling, graphical models, conditional random fields, knowledge graphs, and neural networks.
  • Learning with Noisy Labels: DivideMix discards labels likely to be noisy and uses the corresponding samples as unlabeled data for semi-supervised regularization.The authors report that earlier semi-supervised noisy-label methods perform poorly at high noise levels, whereas DivideMix better distinguishes and utilizes noisy samples.
  • DivideMix: DivideMix trains two networks simultaneously, with each network using the other network’s loss-based dataset division to filter errors and avoid confirmation bias.The procedure combines epoch-level implicit teaching with batch-level explicit teaching.
  • Semi-supervised Learning: Semi-supervised learning methods generally regularize models with unlabeled data through consistency or entropy-based objectives.These approaches add an additional loss term or otherwise encourage stable or confident predictions on unlabeled inputs.

3 METHOD

DivideMix models per-sample losses to separate likely clean and noisy examples, then trains two networks with cross-network data division and noise-aware semi-supervised learning. Confidence-penalized warm-up, label co-refinement, and label co-guessing support this pipeline.

  • Co-divide by loss modeling: A two-component GMM models per-sample loss and assigns each example a clean probability based on the lower-loss Gaussian component.The training data is split into labeled and unlabeled sets using a threshold on that probability.
  • Co-divide by loss modeling: Co-divide uses each network’s GMM-based data division to train the other network, keeping the networks diverged to reduce confirmation bias.The networks differ through separate initialization, data divisions, mini-batch sequences, and training targets.
  • Co-divide by loss modeling: During warm-up, a negative-entropy confidence penalty prevents asymmetric-noise overfitting and makes the loss distribution easier for the GMM to model.Standard warm-up can produce near-zero losses for most samples, limiting clean/noisy separation.
  • MixMatch with label co-refinement and co-guessing: Label co-refinement combines ground-truth labels with predictions using the other network’s clean probability, then sharpens the refined labels.This adapts labeled targets to possible label noise before MixMatch training.
  • MixMatch with label co-refinement and co-guessing: Label co-guessing averages predictions from both networks for unlabeled samples, after which augmented labeled and unlabeled data are mixed with MixMatch.MixMatch applies cross-entropy to transformed labeled data and mean squared error to transformed unlabeled data.
  • MixMatch with label co-refinement and co-guessing: The total objective combines labeled loss, weighted unsupervised loss, and regularization to discourage collapse toward a single predicted class.The regularizer uses a uniform class prior, while the unsupervised-loss weight controls the strength of the unlabeled-data term.

4 EXPERIMENTS

Experiments evaluate DivideMix across synthetic and real-world noisy-label datasets against state-of-the-art baselines, then examine its components through ablations. DivideMix consistently improves performance, with especially large gains under challenging noise conditions.

  • 4.1 Experimental Settings: Experiments cover CIFAR-10, CIFAR-100, Clothing1M, and WebVision using synthetic symmetric/asymmetric noise and real-world noisy labels.CIFAR datasets use 50K training and 10K test images; Clothing1M and WebVision contain large-scale web-collected data with noisy labels.
  • 4.2 Comparison with State-of-the-Art Methods: DivideMix is compared with recent state-of-the-art methods under matched network architectures and reported benchmark settings.Baselines include Meta-Learning, Joint-Optim, P-correction, and M-correction; some baseline results are reimplemented or copied from prior work depending on the dataset.
  • 4.2 Comparison with State-of-the-Art Methods: DivideMix outperforms state-of-the-art methods across CIFAR-10 and CIFAR-100 symmetric noise ratios from 20% to 90%.The comparison reports both best test accuracy across epochs and averaged test accuracy over the last 10 epochs.
  • 4.2 Comparison with State-of-the-Art Methods: ∼10% accuracy improvement is reported for CIFAR-100 at high symmetric noise ratios.CIFAR-10 asymmetric-noise experiments use 40%, because classes become theoretically indistinguishable above 50% asymmetric noise.
  • 4.2 Comparison with State-of-the-Art Methods: More than 12% improvement in top-1 accuracy is achieved on WebVision, while DivideMix consistently outperforms state-of-the-art methods across Clothing1M and WebVision.WebVision results report top-1 and top-5 accuracy on both WebVision and ImageNet ILSVRC12 validation sets.
  • 4.3 Ablation Study: Ablations show that two-network ensemble inference, co-training, label refinement, and input augmentation each contribute to DivideMix performance.Using one model instead of the two-network ensemble decreases accuracy, while self-divide decreases performance further; label refinement is especially important at high noise ratios.

5 CONCLUSION

The conclusion presents DivideMix as a semi-supervised approach to learning with noisy labels. It attributes robustness to dataset co-divide, label co-refinement, and co-guessing, supported by substantial improvements across multiple datasets.

  • 5 Conclusion: DivideMix leverages semi-supervised learning to address learning with noisy labels.The method trains two networks simultaneously and uses dataset co-divide, label co-refinement, and co-guessing to improve robustness to noise.
  • 5 Conclusion: Extensive experiments across multiple datasets show substantial performance improvements over state-of-the-art methods.The authors identify incorporating additional ideas between semi-supervised learning and noisy-label learning, and adapting DivideMix to NLP, as future directions.

APPENDIX A ADDITIONAL EXPERIMENT RESULTS

Additional experiments show that DivideMix outperforms prior methods, separates clean and noisy samples, filters noisy labels, and learns true class structure under severe noise.

  • DivideMix significantly outperforms previous methods using deeper or wider networks on CIFAR-10 and CIFAR-100 with symmetric noise.Table 6 reports test accuracy comparisons under symmetric noise.
  • DivideMix effectively separates clean and noisy CIFAR-10 training samples as training proceeds, even at high noise ratios.The separation is measured using the AUC of a GMM during the first 100 epochs.
  • The method discards noisy Clothing1M labels and uses co-guessed labels to regularize training.Figure 4 shows examples identified as noisy and contrasts ground-truth labels with co-guessed labels.
  • After 200 epochs on CIFAR-10 with 80% label noise, learned embeddings form 10 clusters corresponding to true rather than noisy labels.The t-SNE visualization indicates robustness to label noise.

APPENDIX B ADDITIONAL TRAINING DETAILS

CIFAR training uses a small set of shared settings, while the unsupervised loss weight increases when stronger regularization is needed under more severe noise.

  • For CIFAR experiments, the only per-experiment tuned hyperparameter is the unsupervised loss weight λu.The remaining listed settings are kept fixed across CIFAR experiments.
  • Clothing1M and WebVision share M = 2, T = 0.5, τ = 0.5, λu = 0, and α = 0.5, with SGD momentum 0.9 and batch size 32.Both experiments use a one-epoch warm-up period.
  • Higher noise ratios require stronger regularization from unlabeled samples through a larger λu.Table 7 reports the unsupervised loss weights used for CIFAR experiments.

APPENDIX C ADDITIONAL EXPLANATIONS FOR ABLATION STUDY

The ablation explanations attribute performance differences to co-training, label refinement, augmentation, and temperature sharpening, especially under high noise.

  • Removing co-training causes a performance drop, especially on CIFAR-100 with high noise ratios.The ablation compares DivideMix without co-training against the full method while retaining dataset division, refinement, and guessing.
  • Label refinement is important at high noise ratios because noisy samples can be mistakenly assigned to the labeled set.
  • Augmentation improves performance by producing more reliable predictions and providing consistency regularization.
  • Temperature sharpening is essential for the method to perform well.The authors report this observation alongside their MixMatch-based ablation findings.

APPENDIX D TRAINING TIME ANALYSIS

DivideMix has intermediate training cost: it is slower than Co-teaching+ but faster than P-correction and Meta-Learning in the reported CIFAR-10 comparison.

  • DivideMix is slower than Co-teaching+ in total CIFAR-10 training time.The comparison uses a single Nvidia V100 GPU.
  • DivideMix is faster than P-correction and Meta-Learning, which involve multiple training iterations.
  • Table 8 compares total training time, while Table 9 breaks DivideMix computation into per-operation times.
Loading 2002.07394v1…