Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Antti Tarvainen; Harri Valpola

Source-linked AI summary

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Antti Tarvainen, Harri Valpola

arXiv:1703.01780v6Mar 6, 2017cs.NEcs.LGstat.ML

TL;DR

Semi-supervised learning needs effective use of unlabeled data, but Temporal Ensembling is unwieldy for large datasets because targets update only once per epoch. Mean Teacher averages model weights to generate consistency targets, improving learning speed and classification accuracy while scaling to large datasets and architectures.

Problem
Temporal Ensembling updates each example’s target only once per epoch, limiting its practicality as datasets grow.
Method
Mean Teacher averages model weights to form a teacher that generates targets for consistency regularization.
Results
Mean Teacher improves learning speed and classification accuracy, works with large datasets and online learning, and scales to large image sizes and state-of-the-art architectures.
Takeaways & Limitations
Weight-averaged teachers provide a practical consistency-regularization approach for semi-supervised learning on large datasets and modern architectures.
Takeaways & Limitations
The paper does not evaluate combining Mean Teacher with Virtual Adversarial Training, despite suggesting the approaches may be complementary.

Abstract

from arXiv · show

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

1 Introduction

Semi-supervised deep learning needs regularization that exploits unlabeled data because large models overfit and manual labeling is expensive. Consistency-based methods address this challenge, but teacher-generated targets can suffer from confirmation bias, motivating better teachers and improved target quality.

Motivation: Large deep learning models are prone to over-fitting, while manually adding high-quality labels is often expensive.The introduction motivates regularization methods that effectively exploit unlabeled data in semi-supervised learning.
Consistency regularization: Consistency regularization encourages models to produce similar outputs for slightly perturbed versions of the same data.Noise can be added to inputs or intermediate representations to help models learn abstract invariances.
Consistency regularization: The Γ model compares predictions with and without noise and applies a consistency cost, using the model simultaneously as teacher and student.This approach supplies targets for unlabeled examples, whose classification cost is otherwise undefined.
Target quality: Teacher-generated targets can create confirmation bias, so improving target quality is central to effective semi-supervised learning.The paper identifies both representation perturbations and teacher-model selection as routes to better targets.
Temporal Ensembling: Temporal Ensembling maintains an exponential moving average prediction for each training example, combining the model’s current and earlier versions that evaluated it.The EMA predictions are updated for examples in each minibatch and improve prediction quality.

2 Mean Teacher

Mean Teacher replaces prediction averaging with EMA averaging of student weights to construct targets updated at every training step. It defines consistency between noisy student and teacher predictions while treating the teacher as fixed during optimization.

Weight averaging: Mean Teacher averages model weights instead of predictions, using the EMA of consecutive student weights to construct better teacher targets.The teacher aggregates information after every step rather than every epoch.
Weight averaging: The teacher’s weight averaging improves all layer outputs, producing better intermediate representations in addition to improved top-level predictions.
Consistency cost: The consistency cost J is the expected distance between noisy student and teacher predictions, using weights θ and θ′ with noises η and η′.
Teacher generation: Unlike the Π model and Temporal Ensembling, Mean Teacher generates teacher predictions from EMA weights θ′ rather than shared weights or averaged successive predictions.The teacher weights are treated as constant with respect to optimization.
Consistency cost: Mean Teacher approximates the consistency cost by sampling independent noises at each training step with stochastic gradient descent.Mean squared error is used as the consistency cost in most experiments.

3 Experiments

Experiments show that Mean Teacher improves semi-supervised performance over the Π model and Temporal Ensembling, particularly when labels are scarce, while benefiting from EMA teacher feedback and efficient use of unlabeled data. Results also establish the importance of noise, hyperparameter choices, consistency design, and stronger network architectures, while noting that VAT performs better in two benchmark settings.

Benchmark comparisons: Mean Teacher improves test accuracy over the Π model and Temporal Ensembling on semi-supervised SVHN, and improves CIFAR-10 results over the Π baseline.Experiments used a 13-layer ConvNet with input and network noise and a consistency cost whose weight was ramped up during training.
Training dynamics: EMA-weighted teacher models make more accurate predictions than bare student models after an initial period.Using the EMA teacher creates a feedback cycle in which the teacher improves the student through consistency training and the student improves the teacher through exponential moving averaging.
Unlabeled-data efficiency: With 500 labels, Mean Teacher learns faster and continues improving after the Π model stops, whereas the methods behave virtually identically with all labels.Mean Teacher uses unlabeled data more efficiently, although 500k extra unlabeled examples allow the Π model to improve longer; Mean Teacher ultimately converges to a better result.
Ablation studies: Input augmentation or dropout is necessary for passable performance, while input noise adds no benefit when augmentation is already used.Dropout on the teacher side provides only a marginal benefit over the alternative tested configuration.
Ablation studies: Good consistency-cost weights and EMA decay values span roughly an order of magnitude, while performance degrades quickly outside those ranges; MSE outperforms the tested alternatives toward KL-divergence.The experiments also varied decoupling classification and consistency outputs to assess whether the two tasks should share an output.
Network architecture: A 26-layer ResNet with Shake-Shake regularization improves CIFAR-10 results remarkably, and a 152-layer ResNeXt shows clear improvement over the state of the art on ImageNet using 10% of labels.ImageNet results were evaluated on the validation set because its test set is not publicly available.

4 Related work

Related work connects consistency regularization to noise-based regularization, semi-supervised perturbation consistency, teacher–student distillation, and label propagation. These approaches differ in their mechanisms, training timing, and reliance on predefined input-space metrics.

Noise regularization: Noise regularization includes adversarial training, Dropout, and Dropconnect, which perturb inputs, activations, or individual weights.Adversarial Training changes inputs to maximize prediction differences; Dropout zeros random layer-output dimensions; Dropconnect zeros individual weights.
Consistency-based methods: Semi-supervised perturbation-consistency methods train predictions to remain consistent under noise, including DSS and the Γ variant of Ladder Network.The Γ variant produces noisy student and clean teacher predictions, then uses a denoising layer to predict teacher predictions from student predictions.
Distillation: Teacher–student consistency is related to model compression and distillation, where softmax outputs transfer task information from a complicated model to a simpler model.Representing softmax outputs rather than one-hot outputs provides more task information and regularizes the simpler model.
Distillation: Distillation differs from consistency regularization because distillation occurs after training, whereas consistency regularization occurs during training.Distillation can also be used to harden trained models against adversarial attacks.
Label propagation: Consistency regularization can be viewed as label propagation, pushing label information between nearby samples likely to share a class.Unlike ordinary label propagation, which requires a predefined input-space distance metric, consistency regularization is applied in a deep-learning setting.

5 Conclusion · Appendix

The conclusion presents Mean Teacher as a weight-averaging consistency-regularization method that scales to large datasets and online learning while improving learning speed and classification accuracy. It emphasizes that better teacher-generated targets drive consistency-regularization success and suggests combining Mean Teacher with Virtual Adversarial Training or developing further target-improvement methods.

5 Conclusion: Mean Teacher averages model weights to form a target-generating teacher model.It is presented alongside Temporal Ensembling, Virtual Adversarial Training, and other consistency-regularization approaches.
5 Conclusion: Mean Teacher works with large datasets and online learning, unlike Temporal Ensembling.This addresses a limitation identified for Temporal Ensembling.
5 Conclusion: Mean Teacher experiments suggest faster learning and higher classification accuracy.The conclusion reports improvements in both learning speed and the accuracy of the trained network.
5 Conclusion: Consistency-regularization success depends on the quality of teacher-generated targets.The conclusion states that targets should be improved whenever possible.
5 Conclusion: Mean Teacher and Virtual Adversarial Training exploit the principle of improving teacher-generated targets.They are described as two ways to apply this principle in semi-supervised learning.
5 Conclusion: Combining Mean Teacher with Virtual Adversarial Training may yield even better targets.The conclusion also suggests that additional methods could improve targets and trained models further.

A Results without input augmentation

This section presents SVHN and CIFAR-10 error-rate results without input augmentation. The evaluation uses exponential-moving-average weights, while comparison methods generally use similar architectures and augmentation.

A Results without input augmentation: All models are evaluated using exponential moving average weights.This evaluation procedure applies to all models in the reported results.
A Results without input augmentation: Comparison methods use a similar 13-layer ConvNet architecture and augmentation, except GAN, which uses no augmentation.The comparison setup is described relative to the authors’ architecture and augmentation.

B.1 Convolutional network models … B.2 Residual network models

The paper specifies convolutional and residual-network implementations for Mean Teacher and baseline comparisons, including distinct preprocessing, optimization, and training schedules across datasets. Mean Teacher differs from the Π model through EMA teacher weights and student-only gradient updates.

B.1 Convolutional network models: Mean Teacher uses an EMA of student weights for the teacher, while the Π model uses identical weights and backpropagates through both model sides.Mean Teacher applies gradients only to the student side.
B.1 Convolutional network models: The convolutional model applies mean-only batch normalization, weight normalization, and Leaky ReLU with α = 0.1.The nonlinearity is used on each convolutional layer.
B.1 Convolutional network models: The convolutional model combines cross-entropy classification cost with mean square error consistency cost using a weighted total objective.The classification-cost weight is the expected number of labeled examples per minibatch, subject to ramp-ups.
B.1.1 ConvNet on CIFAR-10: On CIFAR-10, inputs were ZCA-normalized and labeled and unlabeled examples were sampled equally, while the final 25000 steps used learning-rate and Adam β1 ramp-downs.The ramp-downs did not improve results but matched Laine & Aila settings.
B.1.2 ConvNet on SVHN: On SVHN, inputs were normalized to zero mean and unit variance, and each minibatch contained 1 labeled and 99 unlabeled examples.Labeled and unlabeled examples were shuffled and reused after exhaustion.
B.1.2 ConvNet on SVHN: For SVHN, Adam β2 and EMA decay were 0.99 during the first 40000 steps and 0.999 afterward, which helped the 250-label case converge reliably.Training lasted 180000, 400000, or 600000 steps depending on extra unlabeled data.
B.1.3 The baseline ConvNet models: Supervised-only baselines omitted unlabeled examples and consistency cost, while supervised-only and Π models used Mean Teacher hyperparameters but stopped earlier to prevent over-fitting.The supervised-only and Π training durations varied by dataset and label count.
B.2 Residual network models: Residual-network experiments were implemented in PyTorch1 with different architectures for CIFAR-10 and ImageNet.

B.2.1 ResNet on CIFAR-10

The CIFAR-10 experiments used a Shake-Shake-regularized ResNet with 4+4+4 residual blocks, trained with extensive augmentation, scheduled SGD, and consistency-based objectives.

The model replicated the 26-2x96d Shake-Shake-regularized architecture with 4+4+4 residual blocks.
Training used 4 GPUs and minibatches of 512 images, including 124 labeled examples, with random translations, horizontal flips, and channel-wise normalization.
SGD used an initial learning rate of 0.2, Nesterov momentum 0.9, and cosine annealing over 180 or 300 epochs for 1000 or 4000 labels, respectively.
The objective combined classification loss with dual-output logit MSE weighted 0.01 and consistency MSE ramped from 0 to 100.0 during the first 5 epochs.

B.2.2 ResNet on ImageNet

The ImageNet evaluation used a 152-layer ResNeXt with specified augmentation and optimization procedures. Training combined classification with logit MSE, KL-divergence consistency, L2 weight decay, and EMA averaging.

Architecture: The ImageNet runs used a 152-layer ResNeXt architecture with 3+8+36+3 residual blocks and 32 groups of 4 channels on the first block.
Training setup: Training used 10 GPUs with minibatches of 400 images, including 200 labeled images, and applied rotation, cropping, flipping, color jitter, and channel-wise normalization.
Optimization: Optimization used stochastic gradient descent with maximum learning rate 0.25, Nesterov momentum 0.9, a two-epoch linear ramp-up, 60 training epochs, and cosine annealing.
Objective: The cost combined classification with dual-output logit MSE weighted 0.01, KL-divergence consistency ramped to 10.0, L2 decay 5e-5, and EMA decay 0.9997.

B.3 Use of training, validation and test data

For CIFAR-10 and SVHN, development used a held-out validation split while retaining balanced labeled subsets, whereas final evaluation used the entire training set. ImageNet experiments retained balanced class labels from training and used the unmodified validation set for evaluation.

CIFAR-10 and SVHN: CIFAR-10 and SVHN development held out 10% of training data for validation and retained equal numbers of labels from each class in the remainder.Different labeled subsets were used for each evaluation run, while validation labels were retained for result exploration.
CIFAR-10 and SVHN: Final CIFAR-10 and SVHN evaluation used the entire training set, including the validation set.
Validation rationale: The validation-set setup supported more thorough research analysis, although a large fully labeled validation set would not be available in a real-world use case.Retaining previous hyperparameters where possible reduced the chance of over-fitting results to validation labels.
ImageNet: ImageNet training retained equal numbers of labels from each class after random label removal, while validation used the given set without modifications.Each evaluation run used a different training-label set and evaluated results against the validation set.

C Varying between mean squared error and KL-divergence

The consistency-cost experiment found that MSE outperformed KL-divergence and the temperature-adjusted Cτ across tested settings. The authors suggest confidence discounting in Cτ may help explain the difference, although the exact reason remains unclear.

C Varying between mean squared error and KL-divergence: The experiment varied the consistency cost function between MSE and KL-divergence, including the defined Cτ function.Cτ is formulated using a scaled KL-divergence between temperature-adjusted predictions pτ and qτ.
C Varying between mean squared error and KL-divergence: MSE performs better than KL-divergence or Cτ with any tested τ.Other KL-divergence consistency-cost weights also failed to match MSE accuracy.
C Varying between mean squared error and KL-divergence: The exact reason MSE outperforms KL-divergence remains unclear.The authors propose that Cτ may help explain the observed performance difference.
C Varying between mean squared error and KL-divergence: Using τ = 1 for classification and τ < 1 for consistency discounts the confidence of teacher predictions.This setup reflects the assumption that true labels are accurate while teacher predictions may be overly confident.