Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning

Zhanghan Ke; Daoye Wang; Qiong Yan; Jimmy Ren; Rynson W. H. Lau

Source-linked AI summary

Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning

Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, Rynson W. H. Lau

arXiv:1909.01804v1Sep 3, 2019cs.LGcs.CVstat.ML

TL;DR

Coupling an EMA teacher to its student limits consistency-based semi-supervised learning. Dual Student replaces the teacher with an independent student and stabilization constraint, improving results across SSL benchmarks and domain adaptation, including 91.50% top1 accuracy.

Problem
Existing teacher-student SSL methods tightly couple the teacher and student through an EMA, creating a performance bottleneck.
Method
Dual Student uses independently initialized student models and exchanges reliable knowledge through a stable-sample stabilization constraint.
Results
Performance improves across SSL benchmarks and domain adaptation, reaching 91.50% top1 accuracy in the reported domain-adaptation experiment.
Takeaways & Limitations
The reported gains are notable across datasets and tasks, while both discussed Dual Student variants achieve higher performance than the standard structure.
Takeaways & Limitations
Naively applying consistency between independent students can make them collapse by exchanging incorrect knowledge.

Abstract

from arXiv · show

Recently, consistency-based methods have achieved state-of-the-art results in semi-supervised learning (SSL). These methods always involve two roles, an explicit or implicit teacher model and a student model, and penalize predictions under different perturbations by a consistency constraint. However, the weights of these two roles are tightly coupled since the teacher is essentially an exponential moving average (EMA) of the student. In this work, we show that the coupled EMA teacher causes a performance bottleneck. To address this problem, we introduce Dual Student, which replaces the teacher with another student. We also define a novel concept, stable sample, following which a stabilization constraint is designed for our structure to be trainable. Further, we discuss two variants of our method, which produce even higher performance. Extensive experiments show that our method improves the classification performance significantly on several main SSL benchmarks. Specifically, it reduces the error rate of the 13-layer CNN from 16.84% to 12.39% on CIFAR-10 with 1k labels and from 34.10% to 31.56% on CIFAR-100 with 10k labels. In addition, our method also achieves a clear improvement in domain adaptation.

1 City University of Hong Kong 2 SenseTime Research · 1. Introduction

Deep semi-supervised learning addresses the cost of obtaining full labels, but existing Teacher-Student methods are bottlenecked by tightly coupled EMA teachers. Dual Student replaces the EMA teacher with an independently trained student and uses stable samples with stabilization constraints for knowledge exchange.

1. Introduction: Full labeling of huge computer-vision datasets is costly, motivating increased attention to deep semi-supervised learning.The introduction frames SSL as a response to the expense of obtaining complete labels for large datasets.
1. Introduction: Teacher-Student methods use teacher and student roles, with consistency constraints enforcing similar predictions under perturbations.Perturbations may arise from input noise or dropout, and the constraint encourages unlabeled-data smoothness.
1. Introduction: Existing Teacher-Student teachers are generated as exponential moving averages of student models, tightly coupling their weights.VAT and Π use shared weights, while Temporal Model and Mean Teacher apply EMA-based teacher constructions.
1. Introduction: Dual Student replaces the EMA teacher with another student initialized differently and optimized through an individual training path.The two students therefore learn their own knowledge without tightly coupled weights.
1. Introduction: Because naive consistency can make the students collapse into each other, Dual Student defines stable samples and introduces a stabilization constraint.The stabilization constraint enables effective knowledge exchange while preserving distinct student models.
1. Introduction: The work proposes two higher-performing Dual Student variants and evaluates the method extensively across several benchmarks and different tasks.The introduction specifically notes significant performance improvements and evaluation in multiple settings.
1. Introduction: The coupled EMA teacher causes a performance bottleneck in existing Teacher-Student methods.This limitation is identified as a central motivation for introducing an independent model.

2. Related Work

Recent SSL work commonly uses teacher–student consistency learning, varying how targets are generated and stabilized. Multi-model approaches address model collapse through structural constraints, motivating Dual Student’s reliable-knowledge exchange and stabilization constraint.

Consistency-Based SSL: Consistency-based SSL methods use network noise regularization, while target quality is also vital for the consistency constraint.Prior work explores random, adversarial, and dropout-based noise for unlabeled data.
Teacher–Student SSL: Teacher–Student methods apply a consistency constraint between teacher and student models to learn from unlabeled data.
Teacher–Student Variants: The Π Model uses two noisy forwards, whereas Temporal Model uses EMA-accumulated predictions to reduce computational overhead and stabilize training.
Teacher–Student Variants: Mean Teacher defines an explicit teacher as an EMA of the student and updates its weights each iteration before generating targets.
Multi-Model SSL: Deep Co-Training prevents collaborating models from collapsing by combining the Co-Training assumption with view differences and adversarial samples.Dual Student instead exchanges reliable knowledge through a stabilization constraint and does not require adversarial samples.

3. Limits of the EMA Teacher

Existing Teacher-Student SSL methods rely on consistency under perturbations, but their EMA teacher remains tightly coupled to the student. As training converges, this coupling limits the teacher’s independent knowledge and can reinforce biased predictions through confirmation bias.

Consistency-based SSL: Teacher-Student methods enforce consistent predictions for differently perturbed versions of the same sample under the smoothness assumption.The assumption links nearby points in high-density regions with similar outputs.
EMA coupling: Because the teacher is an EMA of the student, their weights are tightly coupled rather than independently learned.The paper explicitly identifies this coupling as a fundamental limitation of prior methods.
EMA coupling: As the student converges, its EMA teacher converges to the same limit, so the teacher cannot provide substantially more meaningful knowledge.Mean Teacher’s student updates shrink over time, and the EMA of a converging sequence shares its limit.
Empirical evidence: On CIFAR-10, the EMA teacher’s weight and prediction distances from the student approach zero, unlike the persistent separation between independent models.Figure 2 contrasts Sema, whose models become similar, with Ssplit, whose model weights and predictions remain more separated.
Confirmation bias: An EMA teacher can accumulate biased predictions and enforce them on the student, making misclassifications difficult to reverse through confirmation bias.Ramp-up operations used by prior methods are described as inadequate for fully solving this problem.

4. Dual Student

Dual Student replaces the coupled EMA teacher with two independently updated students that exchange only reliable knowledge through a stabilization constraint based on stable samples. The method combines classification, consistency, and stabilization constraints, with Multiple Student and Imbalanced Student variants extending the structure without increasing inference time.

4. Dual Student: Dual Student trains two independent models simultaneously to obtain loosely coupled targets, addressing the EMA teacher’s inadequacy when training steps are large.The two students may otherwise exchange incorrect knowledge and collapse, motivating selective knowledge exchange.
4.1 Stable Sample: Stable samples provide reliable model-specific knowledge by requiring locally invariant predictions and sufficient distance from the decision boundary.A sample may be stable for one model but not another, and neighborhood prediction consistency measures its stability degree.
4.2 Stabilization Constraint: The stabilization constraint dynamically transfers knowledge from a stable student to the other, applying no constraint when both students are unstable.When both students are stable, the constraint is directed from the more stable student to the less stable one.
4.2 Stabilization Constraint: Each student’s training objective combines a classification constraint, an individual consistency constraint, and a cross-model stabilization constraint.The consistency constraint enforces the smoothness assumption, while the stabilization constraint enables mutual knowledge exchange.
4.3 Variants: Multiple Student and Imbalanced Student achieve higher performance than standard Dual Student without increasing inference time, despite requiring more training computation.Multiple Student randomly pairs 2n students during training, while Imbalanced Student enhances one student’s capability without treating it as a teacher.

5. Experiments

Experiments show that Dual Student improves semi-supervised classification across CIFAR, SVHN, and ImageNet, while ablations support its stabilization constraint and domain adaptation results show reduced overfitting. The method also extends to multiple-student variants and combinations with existing consistency methods.

CIFAR benchmark: Dual Student boosts performance across semi-supervised settings on CIFAR-10, with larger improvements as the number of labeled samples decreases.Experiments use balanced labels from 1k, 2k, and 4k samples on CIFAR-10 and 10k on CIFAR-100.
SVHN and ImageNet: Dual Student is evaluated for generalization on SVHN and ImageNet using 250 or 500 labels on SVHN and 10% balanced labels on ImageNet.SVHN contains 73,257 training and 26,032 testing samples, while ImageNet contains more than 10 million images across 1k categories.
Dual Student variants: Multiple Student gains are limited beyond four students, while Imbalanced Student uses a strong ResNet with 3.53 million or 11.65 million parameters.The variants are compared with standard Dual Student using the same 13-layer CNN trained for 300 epochs.
Combining consistency methods: Combining Dual Student with Mean Teacher produces obvious improvement, with similar accuracy to Dual Student alone and insensitivity to the internal consistency constraint.This supports the method’s ability to break the performance limits of an EMA teacher.
Domain adaptation: 91.50% accuracy is achieved by Dual Student for USPS-to-MNIST adaptation, compared with 80.41% for Mean Teacher and 69.09% from USPS supervision.Mean Teacher avoids overfitting only partly, whereas Dual Student avoids overfitting and approaches target-domain supervision.

6. Conclusion

The paper identifies coupling in Teacher-Student methods as a performance bottleneck and proposes Dual Student with a stabilization constraint for training independent models.

6. Conclusion: Dual Student breaks the limits of the EMA teacher by replacing the coupled Teacher-Student structure with independent models.The models may use either the same architecture or different architectures.
6. Conclusion: The stabilization constraint is bidirectional overall but unidirectional for each stable sample.It provides an effective way to train the independent models.

Appendix A: Convergence of the EMA

Appendix A formalizes the coupling between the EMA teacher and student by proving that an EMA sequence converges to the same limit as the underlying student sequence. The proof uses geometric-series bounds and the fact that α^t tends to zero for 0 < α < 1.

Formal coupling claim: The appendix formalizes the claim that the EMA teacher is coupled with the student in existing Teacher-Student methods.It introduces a formal proposition and a simple proof.
Proposition 1: If the student sequence {s_t} converges to S ∈ R^m, its EMA sequence {s′_t} also converges to S when 0 < α < 1.The proposition assumes s′_0 ∈ R^m.
Proof: The proof bounds the finite geometric-series terms and uses lim_t→∞ α^t = 0 to establish convergence of the EMA sequence.For sufficiently large t, the proof applies the triangular inequality after substituting the geometric-series expression.

Appendix B: Model Architectures

The paper uses a 13-layer CNN for CIFAR-10, CIFAR-100, and SVHN, following FastSWA for comparison, and a 50-layer ResNeXt for ImageNet.

CIFAR and SVHN: A 13-layer CNN, implemented following FastSWA, is used for CIFAR-10, CIFAR-100, and SVHN experiments.This architecture matches previous works cited by the paper.
ImageNet: ImageNet experiments use a 50-layer ResNeXt with 3+4+6+3 residual blocks and 32-group convolution.

Appendix C: Semi-supervised Learning Setups

The experiments use SGD with Nesterov momentum and a cosine-based learning-rate schedule, with dataset-specific batch composition, optimization settings, stabilization coefficients, and confidence thresholds. ImageNet additionally uses ResNeXt-50 across 8 GPUs with specified augmentation, warm-up, training duration, and resolution.

Optimization: All experiments use SGD with Nesterov momentum 0.9 and a cosine-based learning-rate schedule from initial rate γ0.The schedule is γ = γ0 ∗(0.5 + cos((t −1) ∗π/N)), where t is the current training step and N is the total number of steps.
Dataset setups: CIFAR-10 uses batch size 100 with half labeled, initial learning rate 0.1, weight decay 1e−4, λ2 = 100, λ1 = 10, and confidence threshold 0.8.The stabilization coefficient λ2 ramps up during the first 5 epochs.
Dataset setups: CIFAR-100 uses minibatches of 128 with 31 labeled samples, initial learning rate 0.2, weight decay 2e−4, and confidence threshold ξ = 0.4.Other hyperparameters are the same as CIFAR-10.
Dataset setups: SVHN uses batch size 100 with 10 labeled samples, initial learning rate 0.1, weight decay 1e−4, stabilization scaling by 10, and confidence threshold ξ = 0.8.The stabilization constraint ramps up in 5 epochs.
Dataset setups: ImageNet uses ResNeXt-50 on 8 GPUs with batch size 320, half labeled, 224 × 224 inputs, 60 training epochs, and confidence threshold 0.01.The learning rate warms up from 0.08 to 0.2 in the first 2 epochs; weight decay is 5e−5 and the stabilization coefficient is 1000.

Appendix D: Domain Adaptation Setups

The appendix evaluates USPS-to-MNIST domain adaptation using a small convolutional network and specifies the training configuration, batch composition, and target-supervised sampling.

Domain adaptation setup: The USPS-to-MNIST setup uses a small convolutional network, 100 SGD epochs with Nesterov momentum 0.9, weight decay 1e−4, and cosine-decayed learning rate from 0.1 to 0.Each batch contains 256 samples, including 32 labeled samples; target-supervised experiments randomly extract 7000 balanced MNIST samples.