Source-linked AI summary
Temporal Ensembling for Semi-Supervised Learning
Samuli Laine, Timo Aila
TL;DR
Semi-supervised learning needs reliable targets when only a small portion of training data is labeled. The paper forms self-ensembles from predictions across epochs and stochastic conditions, achieving strong benchmark results and tolerance to incorrect labels. Its comparisons are qualified by differences in augmentation and pooling setups across methods.
Problem
Semi-supervised learning requires useful targets when only a small portion of training data is labeled.
Method
Self-ensembling uses predictions from one network across epochs, regularization, and input augmentations as targets for unlabeled examples.
Results
The methods set new semi-supervised benchmark records across SVHN, CIFAR-10, and CIFAR-100, with gains under non-augmented and augmented settings.
Takeaways & Limitations
Self-ensembling also improves fully supervised accuracy and provides tolerance against incorrect labels.
Takeaways & Limitations
Comparisons with prior work are limited by differing augmentation and fractional max-pooling configurations.
Abstract
from arXiv · showhide
In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.
1 INTRODUCTION
The paper extends ensemble-based prediction into semi-supervised training by using a network’s outputs across epochs, regularization conditions, and input augmentations as targets for unlabeled data. It introduces Π-model and temporal ensembling, reporting improved semi-supervised and fully supervised performance with tolerance to incorrect labels.
- Motivation: Self-ensembling averages predictions from one network across training epochs and different regularization or augmentation conditions.The ensemble prediction is intended to better approximate unknown labels than the latest network output.
- Semi-supervised objective: Unlabeled inputs receive inferred ensemble targets, while labeled inputs retain the supervised cross-entropy objective.The cross-entropy component is evaluated only for labeled inputs.
- Methods: The paper presents two self-ensembling implementations: the Π-model and temporal ensembling.These methods differ in how they obtain the multiple predictions used for consistency training.
- Reported contribution: Both approaches surpass prior state-of-the-art semi-supervised results by a considerable margin.The paper also reports improved accuracy in fully labeled settings and tolerance to incorrect labels.
- Relation to prior work: The methods relate to transform/stability loss, the ladder network’s Γ-model, and bootstrapping for noisy labels.The Π-model is described as a special case or simplification of related consistency-based approaches.
2 SELF-ENSEMBLING DURING TRAINING
Self-ensembling trains on consistency between stochastic network evaluations or predictions accumulated from earlier epochs, alongside supervised learning on labeled examples. Temporal ensembling reduces computation and target noise relative to the Π-model, but requires stored cross-epoch predictions and an additional momentum parameter.
- Π-model: The Π-model compares two stochastic evaluations of each input and penalizes differences between their prediction vectors.Its loss combines labeled cross-entropy with an unsupervised mean-square consistency term weighted by w(t).
- Π-model: The Π-model compares full output vectors rather than only final classifications, imposing a stronger consistency requirement.Dropout, Gaussian noise, and input augmentations create the variation between evaluations.
- Training schedule: The unsupervised loss weight ramps up slowly from zero during early training to avoid a degenerate solution.The implementation ramps the weight along a Gaussian curve during the first 80 epochs.
- Temporal ensembling: Temporal ensembling accumulates epoch-level predictions with momentum α and bias-corrects them into targets for the next training updates.Recent epochs receive larger weights, while the first epoch has zero targets because no prior predictions exist.
- Temporal ensembling: Temporal ensembling evaluates each input once per epoch, making training faster and producing less noisy targets than the Π-model.Its trade-offs are auxiliary storage across epochs and the additional hyperparameter α.
- Open direction: The paper leaves uncertainty-aware temporal ensembling based on higher-order prediction statistics for future work.Tracking prediction variance could support weighting more certain outputs more heavily.
3 RESULTS
Across CIFAR-10, SVHN, and CIFAR-100, self-ensembling methods substantially improve semi-supervised classification, with further gains from augmentation and unlabeled extra data. The methods also show tolerance to incorrect labels and improve fully supervised results.
- CIFAR-10: 2.1 percentage points lower CIFAR-10 classification error was achieved with 4000 labels using the non-augmented Π-model.The comparison is against earlier methods.
- CIFAR-10: 12.16% CIFAR-10 error was achieved with temporal ensembling and standard augmentations, versus 12.36% for the Π-model.Augmentation reduced the Π-model error by 4.2 percentage points to 12.36%, and temporal ensembling trained twice as fast.
- SVHN: 2.7 percentage points lower SVHN error was obtained with 1000 labels, from 8.11% to 5.43% without augmentation and to 4.42% with standard augmentations.With 500 labels, temporal ensembling further reduced the augmented error rate to 5.12%.
- CIFAR-100 and Tiny Images: 43.43% and 38.65% CIFAR-100 error rates were obtained without and with augmentation, improving supervised learning by 7.8 and 5.9 percentage points.The experiment used 10000 labels.
- CIFAR-100 and Tiny Images: 2.7 percentage points lower CIFAR-100 error resulted from adding 500k randomly selected unlabeled Tiny Images, from 26.30% to 23.63%.The result indicates that randomly selected natural images can provide useful unlabeled inputs.
- Tolerance to incorrect labels: Temporal ensembling retained over ninety percent SVHN classification accuracy when 80% of labels were randomized, while standard supervised training degraded rapidly.With half the labels randomized, temporal ensembling showed almost perfect resistance to disinformation.
4 RELATED WORK
The paper situates self-ensembling among semi-supervised, noisy-label, and ensemble-based approaches, while identifying related architectures and possible extensions.
- Prior semi-supervised work includes ladder-network variants, transform/stability loss, label propagation, and generative adversarial networks.
- The Π-model shares the transform/stability principle and simplifies the Γ-model by comparing network outputs directly.
- Self-ensembling derives variability from dropout and augmentation in one network rather than independently trained networks on different data subsets.
- Temporal ensembling connects to bootstrapping methods for training with noisy labels.
- The authors identify incorporating a generative component and applying the methods to regression as possible future directions.
A NETWORK ARCHITECTURE, TEST SETUP, AND TRAINING PARAMETERS
The experiments use a convolutional architecture, dataset-specific preprocessing, and scheduled optimization, with additional details concerning extra data and convergence constraints.
- The network uses weight normalization, mean-only batch normalization, leaky ReLU, and max pooling throughout the tested architecture.
- Training uses Adam, dataset-dependent maximum learning rates, and a Gaussian ramp-up for the unsupervised loss and learning rate.
- CIFAR-10 preprocessing applies ZCA, horizontal flips, and independent random translations to the Π-model branches.
- SVHN inputs are standardized, and experiments use only the official 73,257-image training set without horizontal flips.
- A slow unsupervised-cost ramp-up is important for convergence, while very low-label SVHN runs sometimes experienced exploding optimization.
- The training procedure does not guarantee stratified labeled examples across minibatches when label density is very low.
- Restricted Tiny Images extra data contains 237,203 category-selected images, while CIFAR-100 overlaps substantially with the broader Tiny Images source.