DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Motivation

• High-quality annotation is expensive, motivating methods that learn from noisily labeled or unlabeled data.Alternative data sources reduce annotation cost but inevitably introduce noisy labels, which deep networks can overfit.Source: §1 · p. 1
• Existing noisy-label methods mainly correct losses, but transition-matrix estimation is difficult and prediction-based correction can overfit under high noise.These approaches rely on estimating noise structure or model predictions, both of which become unreliable in challenging noise settings.Source: §1 · p. 1
• Semi-supervised learning leverages unlabeled samples through entropy minimization, consistency regularization, and unified methods such as MixMatch.MixMatch combines several dominant semi-supervised learning approaches and achieves strong performance.Source: §1 · pp. 1–2

DivideMix

• DivideMix addresses the underexplored connection between noisy-label learning and semi-supervised learning by discarding likely noisy labels and using those samples as unlabeled data.The unlabeled samples regularize training against overfitting and support generalization.Source: §1 · p. 2
• Co-divide trains two networks that use Gaussian Mixture Models to separate mostly clean labeled samples from mostly noisy unlabeled samples for the other network.The framework also applies label co-refinement to labeled samples and label co-guessing to unlabeled samples during improved MixMatch training.Source: §1 · p. 2

Learning with Noisy Labels

• Noisy-label methods commonly correct losses explicitly or implicitly, but some relabeling approaches require a small clean set.The related methods include transition-matrix correction, relabeling, graphical models, conditional random fields, knowledge graphs, and neural networks.Source: §2.1 · p. 2
• DivideMix discards labels likely to be noisy and uses the corresponding samples as unlabeled data for semi-supervised regularization.The authors report that earlier semi-supervised noisy-label methods perform poorly at high noise levels, whereas DivideMix better distinguishes and utilizes noisy samples.Source: §2.1 · p. 2

DivideMix

• DivideMix trains two networks simultaneously, with each network using the other network’s loss-based dataset division to filter errors and avoid confirmation bias.The procedure combines epoch-level implicit teaching with batch-level explicit teaching.Source: Figure 1 · p. 3

Semi-supervised Learning

• Semi-supervised learning methods generally regularize models with unlabeled data through consistency or entropy-based objectives.These approaches add an additional loss term or otherwise encourage stable or confident predictions on unlabeled inputs.Source: §2.2 · p. 3

Co-divide by loss modeling

• A two-component GMM models per-sample loss and assigns each example a clean probability based on the lower-loss Gaussian component.The training data is split into labeled and unlabeled sets using a threshold on that probability.Source: §3.1 · p. 3
• Co-divide uses each network’s GMM-based data division to train the other network, keeping the networks diverged to reduce confirmation bias.The networks differ through separate initialization, data divisions, mini-batch sequences, and training targets.Source: §3 · p. 3, §3.1 · p. 4
• During warm-up, a negative-entropy confidence penalty prevents asymmetric-noise overfitting and makes the loss distribution easier for the GMM to model.Standard warm-up can produce near-zero losses for most samples, limiting clean/noisy separation.Source: §3.1 · p. 4, Figure 2 · p. 5

MixMatch with label co-refinement and co-guessing

• Label co-refinement combines ground-truth labels with predictions using the other network’s clean probability, then sharpens the refined labels.This adapts labeled targets to possible label noise before MixMatch training.Source: §3.1 · p. 4, §3.2 · p. 5
• Label co-guessing averages predictions from both networks for unlabeled samples, after which augmented labeled and unlabeled data are mixed with MixMatch.MixMatch applies cross-entropy to transformed labeled data and mean squared error to transformed unlabeled data.Source: §3.1 · p. 4, §3.2 · pp. 5–6
• The total objective combines labeled loss, weighted unsupervised loss, and regularization to discourage collapse toward a single predicted class.The regularizer uses a uniform class prior, while the unsupervised-loss weight controls the strength of the unlabeled-data term.Source: §3.2 · p. 6

4.1 Experimental Settings

• Experiments cover CIFAR-10, CIFAR-100, Clothing1M, and WebVision using synthetic symmetric/asymmetric noise and real-world noisy labels.CIFAR datasets use 50K training and 10K test images; Clothing1M and WebVision contain large-scale web-collected data with noisy labels.Source: §4.1 · p. 6

4.2 Comparison with State-of-the-Art Methods

• DivideMix is compared with recent state-of-the-art methods under matched network architectures and reported benchmark settings.Baselines include Meta-Learning, Joint-Optim, P-correction, and M-correction; some baseline results are reimplemented or copied from prior work depending on the dataset.Source: §4.2 · p. 6, Table 1 · p. 7, Table 3 · p. 8
• DivideMix outperforms state-of-the-art methods across CIFAR-10 and CIFAR-100 symmetric noise ratios from 20% to 90%.The comparison reports both best test accuracy across epochs and averaged test accuracy over the last 10 epochs.Source: §4.2 · p. 7, Table 1 · p. 7
• ∼10% accuracy improvement is reported for CIFAR-100 at high symmetric noise ratios.CIFAR-10 asymmetric-noise experiments use 40%, because classes become theoretically indistinguishable above 50% asymmetric noise.Source: §4.2 · p. 7
• More than 12% improvement in top-1 accuracy is achieved on WebVision, while DivideMix consistently outperforms state-of-the-art methods across Clothing1M and WebVision.WebVision results report top-1 and top-5 accuracy on both WebVision and ImageNet ILSVRC12 validation sets.Source: §4.2 · p. 7, Table 4 · p. 8

4.3 Ablation Study

• Ablations show that two-network ensemble inference, co-training, label refinement, and input augmentation each contribute to DivideMix performance.Using one model instead of the two-network ensemble decreases accuracy, while self-divide decreases performance further; label refinement is especially important at high noise ratios.Source: §4.3 · pp. 8–9

5 Conclusion

• DivideMix leverages semi-supervised learning to address learning with noisy labels.The method trains two networks simultaneously and uses dataset co-divide, label co-refinement, and co-guessing to improve robustness to noise.Source: §5 · p. 9
• Extensive experiments across multiple datasets show substantial performance improvements over state-of-the-art methods.The authors identify incorporating additional ideas between semi-supervised learning and noisy-label learning, and adapting DivideMix to NLP, as future directions.Source: §5 · p. 9

• DivideMix significantly outperforms previous methods using deeper or wider networks on CIFAR-10 and CIFAR-100 with symmetric noise.Table 6 reports test accuracy comparisons under symmetric noise.Source: Appendix A · p. 12, Table 6 · p. 12
• DivideMix effectively separates clean and noisy CIFAR-10 training samples as training proceeds, even at high noise ratios.The separation is measured using the AUC of a GMM during the first 100 epochs.Source: Appendix A · p. 12
• The method discards noisy Clothing1M labels and uses co-guessed labels to regularize training.Figure 4 shows examples identified as noisy and contrasts ground-truth labels with co-guessed labels.Source: Appendix A · p. 12, Figure 4 · p. 12
• After 200 epochs on CIFAR-10 with 80% label noise, learned embeddings form 10 clusters corresponding to true rather than noisy labels.The t-SNE visualization indicates robustness to label noise.Source: Appendix A · p. 13, Figure 5 · p. 13

• For CIFAR experiments, the only per-experiment tuned hyperparameter is the unsupervised loss weight λu.The remaining listed settings are kept fixed across CIFAR experiments.Source: Appendix B · p. 13
• Clothing1M and WebVision share M = 2, T = 0.5, τ = 0.5, λu = 0, and α = 0.5, with SGD momentum 0.9 and batch size 32.Both experiments use a one-epoch warm-up period.Source: Appendix B · p. 13
• Higher noise ratios require stronger regularization from unlabeled samples through a larger λu.Table 7 reports the unsupervised loss weights used for CIFAR experiments.Source: Table 7 · p. 13, Appendix B · p. 13

• Removing co-training causes a performance drop, especially on CIFAR-100 with high noise ratios.The ablation compares DivideMix without co-training against the full method while retaining dataset division, refinement, and guessing.Source: Appendix C · p. 13
• Label refinement is important at high noise ratios because noisy samples can be mistakenly assigned to the labeled set.Source: Appendix C · p. 13
• Augmentation improves performance by producing more reliable predictions and providing consistency regularization.Source: Appendix C · p. 13
• Temperature sharpening is essential for the method to perform well.The authors report this observation alongside their MixMatch-based ablation findings.Source: Appendix C · p. 13

• DivideMix is slower than Co-teaching+ in total CIFAR-10 training time.The comparison uses a single Nvidia V100 GPU.Source: Appendix D · p. 14
• DivideMix is faster than P-correction and Meta-Learning, which involve multiple training iterations.Source: Appendix D · p. 14
• Table 8 compares total training time, while Table 9 breaks DivideMix computation into per-operation times.Source: Table 8 · p. 14, Table 9 · p. 14, Appendix D · p. 14