PaperlayerView exact revision on arXiv

Source-linked AI summary

DMT: Dynamic Mutual Training for Semi-Supervised Learning

Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, Lizhuang Ma

arXiv:2004.08514v4cs.CV

TL;DR

DMT addresses unreliable pseudo labels by using disagreement between two models instead of relying only on one model’s confidence. Its dynamically re-weighted mutual-training loss achieves state-of-the-art performance in image classification and semantic segmentation, while retaining the limitations of offline self-training initialization.

  • Problem

    Pseudo-label self-training can retain high-confidence errors and discard low-confidence correct labels because it relies on a single model’s prediction confidence.

  • Method

    Dynamic Mutual Training compares predictions from two models and dynamically re-weights the mutual-training loss, assigning lower loss to larger disagreement.

  • Results

    DMT achieves state-of-the-art performance on both image classification and semantic segmentation.

  • Takeaways & Limitations

    Inter-model disagreement provides a basis for locating pseudo-label errors during semi-supervised training.

  • Takeaways & Limitations

    DMT shares offline self-training’s limitation that the initial model learned on the labeled subset may be insufficient when labels are limited.

Abstract

from arXiv · show

Recent semi-supervised learning methods use pseudo supervision as core idea, especially self-training methods that generate pseudo labels. However, pseudo labels are unreliable. Self-training methods usually rely on single model prediction confidence to filter low-confidence pseudo labels, thus remaining high-confidence errors and wasting many low-confidence correct labels. In this paper, we point out it is difficult for a model to counter its own errors. Instead, leveraging inter-model disagreement between different models is a key to locate pseudo label errors. With this new viewpoint, we propose mutual training between two different models by a dynamically re-weighted loss function, called Dynamic Mutual Training (DMT). We quantify inter-model disagreement by comparing predictions from two different models to dynamically re-weight loss in training, where a larger disagreement indicates a possible error and corresponds to a lower loss value. Extensive experiments show that DMT achieves state-of-the-art performance in both image classification and semantic segmentation. Our codes are released at https://github.com/voldemortX/DST-CBC .

1. Introduction

Semi-supervised learning can exploit abundant unlabeled data, but confidence-filtered pseudo labels retain errors and discard correct examples. DMT addresses this by using disagreement between two models to dynamically reduce the influence of likely erroneous pseudo supervision, achieving state-of-the-art results in classification and segmentation.

  • Motivation: Semi-supervised learning uses a small labeled subset and exploits the remaining unlabeled data, including for image classification and semantic segmentation.
  • Dynamic Mutual Training: DMT uses two different models whose prediction disagreement can identify each other’s errors more effectively than a model assessing its own predictions.The method assigns lower loss or gradient to examples with larger disagreement.
  • Dynamic Mutual Training: DMT trains models mutually with a dynamically weighted noise-robust loss, lowering the loss weight as inter-model disagreement increases.One model provides pseudo labels for the other on unlabeled data, with weighting based on the relation between their prediction confidences.
  • Dynamic Mutual Training: An iterative easy-to-hard framework applies DMT while progressively incorporating more unlabeled data to increase performance.
  • Results: Experiments on CIFAR-10, PASCAL VOC 2012, and Cityscapes show state-of-the-art performance in semi-supervised image classification and semantic segmentation.The authors report that DMT surpasses manual annotation in the harder semantic segmentation task.

2. Related Work

Related semi-supervised methods exploit multiple models through agreement, disagreement, or mutual example selection, but DMT instead uses inter-model disagreement to re-weight losses and targets both classification and segmentation.

  • Mutual training: Dual Student selects stable examples using one model alone, while DMT lets two models collaborate explicitly to re-weight loss.In DMT, the models jointly determine how training loss is weighted.
  • Disagreement-based methods: Deep Co-Training minimizes model disagreement and requires explicit weight-distance constraints to avoid online collapse.DMT’s dynamic loss is determined by both models and does not require such special constraints.
  • Disagreement-based methods: Co-training methods maximize agreement on unlabeled predictions, whereas DMT treats inter-model disagreement as a source of learning information.DMT uses disagreement to identify possible errors rather than simply penalizing disagreement.
  • Scope: DMT is presented as generally applicable to both image classification and semantic segmentation, addressing a scope limitation in prior methods.The related-work discussion notes that earlier methods often worked well only for a limited range of tasks.
  • Mutual training: Unlike Co-Teaching+, DMT addresses pseudo-label noise by loss re-weighting rather than using disagreement only to select update opportunities.For pseudo-label noise, disagreement signifies possible errors rather than merely indicating when to update parameters.
  • Scope: DMT’s training-phase loss re-weighting is complementary to graph-based pseudo-label rectification, which is applied during pseudo-labeling.The graph approach is described as impractical for segmentation datasets with millions of pixel-level data points per image.

3. Preliminaries

The preliminaries cover entropy minimization, consistency regularization, and disagreement-based learning, then frame pseudo-label noise as a central challenge for self-training.

  • Prior knowledge: Semi-supervised learning approaches are organized into entropy minimization, consistency regularization, and disagreement-based methods.These approaches respectively encourage confident predictions, perturbation consistency, or agreement between multiple classifiers.
  • Entropy minimization and self-training: Entropy minimization encourages confident decisions by minimizing predictive entropy, while self-training uses the most probable class as a pseudo label.The entropy reaches its minimum when one class probability is 1 and the others are 0.
  • Entropy minimization and self-training: Self-training may be online, generating pseudo labels after each forward pass, or offline, generating labels first and then retraining on labeled and pseudo-labeled data.Offline self-training can iteratively relabel unlabeled data with the most recently trained model.
  • Consistency regularization: Consistency regularization enforces similar predictions under perturbed inputs, commonly using mean squared error or a teacher model.Mean Teacher defines teacher weights as an exponential moving average of the student model.
  • Disagreement-based methods: Disagreement-based learning enforces prediction agreement between models and may require diversity losses to prevent model collapse.The unlabeled-data loss is described using Jensen-Shannon divergence.
  • Consistency regularization: General augmentation in offline self-training is described as an anchored form of explicit consistency regularization, with outcomes depending on the anchor’s quality.The authors report good performance from augmentation in offline self-training, equivalent to fully supervised training after pseudo-labeling.
  • Pseudo-label noise: Online self-training makes consistency regularization computationally costly and is more sensitive to pseudo-label noise, motivating the offline setting studied here.Multiple perturbations require multiple forward passes and induce higher computational cost.

4. Method

DMT trains two differently initialized models that exchange pseudo labels and dynamically weight their losses using inter-model disagreement. It extends this mutual training iteratively across image classification and semantic segmentation.

  • Dynamic Mutual Training: DMT trains two different models on labeled data, then alternates pseudo-label generation and dynamically weighted cross-entropy training between them.Each model can generate pseudo labels and confidence for the other model, using labeled and pseudo-labeled data together.
  • Dynamic loss: The dynamic weight uses the current model’s probability for the pseudo-labeled class when predictions agree or disagreement is lower-confidence.When the current model disagrees with higher confidence, the weight is set to zero because the pseudo label is probably incorrect.
  • Task adaptation: For semantic segmentation, the same disagreement-based re-weighting is applied independently to each pixel.The unlabeled input is represented as a pixel-wise map, while the re-weighting strategy remains unchanged.
  • Model initialization: DMT requires sufficiently different models; it uses random initialization for simpler classification tasks and different pre-trained datasets when segmentation needs pre-training.Difference-maximized sampling also creates labeled sub-subsets with the smallest intersection when suitable pre-trained weights are unavailable.
  • Iterative framework: The iterative framework repeatedly selects more pseudo labels and retrains models with dynamic loss, using five default iterations in experiments.Classification retrains randomly initialized models, while segmentation uses two separate fine-tunings between differently initialized models.
  • Iterative framework: In semantic segmentation, direct ranking replaces re-normalization because re-normalization slightly degrades final performance when using up to all data.The segmentation procedure performs equal mutual fine-tunings and uses more top-confident pseudo labels at each iteration.

5. Experiments

DMT is evaluated against state-of-the-art methods on CIFAR-10, PASCAL VOC 2012, and Cityscapes under standard small-labeled-set settings. It improves over comparison methods across classification and segmentation, with especially strong and stable segmentation results.

  • Experimental setup: DMT is evaluated on CIFAR-10, PASCAL VOC 2012, and Cityscapes using small labeled subsets and corresponding unlabeled data.The experiments cover image classification and semantic segmentation benchmarks with multiple labeled-data ratios.
  • Image classification: On CIFAR-10, DMT steadily improves over Curriculum Labeling, with larger gains in harder settings using fewer labeled examples.The comparison is controlled, and Curriculum Labeling is already close to Oracle performance in the 4,000-label setting.
  • Image classification: On CIFAR-10, DMT performs better than the other state-of-the-art methods included in the broader comparison.The comparison includes consistency-based, self-training, multi-model, graph-based, and hybrid methods.
  • Semantic segmentation: DMT exceeds the reported fully supervised Oracle performance on one semantic-segmentation evaluation.The paper reports Oracle performance as 96.35% before stating that DMT surpasses it.
  • Semantic segmentation: On PASCAL VOC 2012, DMT outperforms other methods with a clear margin and remains stable across different labeled ratios.It is the only method showing a clear improvement over Baseline on the challenging 100-label split besides CutMix.
  • Semantic segmentation: DMT outperforms competing methods by 2 ∼3% in the reported comparison.The cited comparison states that the competing methods still obtain good results, while DMT achieves the higher performance.

5.4. Ablations

The ablations compare self-training variants and show that DMT benefits from modeling inter-model disagreement, supports continued improvement across iterations, and balances performance with complexity.

  • CBST performance starts to degrade at iteration three due to too much pseudo label noise.
  • CBST consistently improves over Baseline, but its performance increase stops at iteration three while DMT benefits from all five iterations.
  • DST requires half the computing budget compared to DMT and performs well, while different model initializations naturally provide disagreement for DMT.
  • DMT-Naive is comparable to DMT on the 1/20 split but degrades significantly on the 1/50 split and generalizes poorly to CIFAR-10.
  • DMT-Flip performs similarly to DMT, while simplifying the three-case policy produces notable degradation; the DMT setup trades performance against complexity.

6. Conclusions and Discussions

The paper concludes that DMT addresses pseudo-supervision noise through inter-model disagreement and achieves strong results across classification and segmentation, with clearer promise for segmentation.

  • DMT uses a re-weighted loss function based on inter-model disagreement to counter pseudo-supervision noise.
  • DMT is adapted to an iterative frame-like semantic segmentation setting and is evaluated on CIFAR-10, PASCAL VOC 2012, and Cityscapes.
  • Experiments and ablations demonstrate state-of-the-art outcomes in both image classification and semantic segmentation.
  • DMT is more promising in semantic segmentation than image classification, possibly because dynamic weighting exploits high-quality pseudo labels across image pixels.
  • CIFAR-10 classification makes inter-model disagreement difficult to exploit because models require retraining and do not maintain equal classification ability.
  • Offline self-training remains constrained when the initial model learned from very few labeled examples is insufficient, such as 100 labels on Cityscapes.

Appendix A. Extra Qualitative Results

Qualitative Cityscapes comparisons show DMT reducing baseline confusion among similar classes, improving small-object detection, and producing more coherent predictions.

  • DMT does a better job than Baseline on Cityscapes qualitative comparisons against ground truth.
  • Baseline confuses similar classes including bus and car, wall, fence and building, and road and sidewalk.
  • DMT detects small objects better, including a motorcycle and a pole.
  • Baseline predicts nonexistent sky and produces chaotic results on a complex real-world fence, whereas DMT becomes consistent after training.

Appendix B. Proper Baseline Training

The baseline-training discussion explains that training choices affect reported performance, including epoch counts and augmentation alignment across semi-supervised comparisons.

  • The baselines unify augmentation schemes and training tricks, including strong augmentations and mixup for CIFAR-10.
  • For labeled subsets, scaling epochs directly with the inverse labeled ratio is too many or too few, so training uses a compromise schedule.
  • The Oracle training schedules use 300, 30, and 60 epochs for CIFAR-10, PASCAL VOC 2012, and Cityscapes, respectively.
  • Baseline performance is noticeably higher than in previous works and sometimes comparable to prior state-of-the-art methods on Cityscapes.

Appendix C. Realistic Validation

The appendix uses small validation sets and fine-grained testing to assess both prediction correctness and confidence. This testing helps distinguish similar setups but is less suitable for comparing mixup with non-mixup methods.

  • Realistic validation: The method uses a small validation set, including 200 images on CIFAR-10 instead of 5,000.The 200-image set is called valtiny and will be released with the source code.
  • Fine-grained testing: Fine-grained testing counts the probability assigned to the correct class, not only whether the prediction is correct.This measures how correct a prediction is according to the assigned probability.
  • Fine-grained testing: Fine-grained testing is helpful when normal testing cannot distinguish similar setups, such as nearby hyper-parameter choices.The appendix describes this as especially useful for comparisons involving a few different hyper-parameter values.
  • Limitations: Fine-grained testing does not work well for comparing mixup and non-mixup methods because mixup is better class-calibrated.The appendix also notes that 200 images cannot reliably reveal a 0.5% accuracy gap.
  • Hyper-parameter validation: Gamma is selected by grid search, with a best value of 5 validated on PASCAL VOC and trial grids of (3, 4, 5, 6) for CIFAR and Cityscapes.The appendix reports no obvious or notable improvement from additional trials.

Appendix D. Complexity

DMT leaves the model architecture and test-time complexity unchanged relative to the supervised baseline, while increasing training cost through repeated training or fine-tuning. The reported cost is roughly 2.5x Oracle training for segmentation and longer convergence for classification.

  • Classification: DMT has the same test-phase computational complexity as the supervised baseline because it makes no architecture modification.The difference from the baseline is confined to model training.
  • Segmentation: Segmentation DMT takes roughly 2.5x the time of fully supervised Oracle training across five iterations and pseudo-label generation.Each iteration is fine-tuning and therefore very fast.
  • Segmentation: Considering that DMT produces two models, its segmentation training cost is similar to Oracle training.This comparison follows the reported per-iteration cost and final number of models.
  • Classification: On CIFAR-10, DMT is still much faster than other semi-supervised learning methods.The stated comparison concerns overall training speed.

Appendix E. Extra Illustrations

The extra illustration describes difference-maximized sampling, where two equal-sized subsets are drawn with minimal overlap after initial random shuffling.

  • Extra illustrations: Difference-maximized sampling begins by randomly shuffling the complete set.The illustration concerns constructing subsets with maximized differences.
  • Extra illustrations: Subsets A and B have equal size and the fewest overlapped samples.This describes the sampling relationship between the two subsets.
Loading 2004.08514v4…