Temporal Ensembling for Semi-Supervised Learning

Samuli Laine; Timo Aila

Source-linked AI summary

Temporal Ensembling for Semi-Supervised Learning

Samuli Laine, Timo Aila

arXiv:1610.02242v3Oct 7, 2016cs.NEcs.LG

TL;DR

Semi-supervised learning needs reliable targets when only a small portion of training data is labeled. The paper forms self-ensembles from predictions across epochs and stochastic conditions, achieving strong benchmark results and tolerance to incorrect labels. Its comparisons are qualified by differences in augmentation and pooling setups across methods.

Problem
Semi-supervised learning requires useful targets when only a small portion of training data is labeled.
Source: §1 · p. 1
Method
Self-ensembling uses predictions from one network across epochs, regularization, and input augmentations as targets for unlabeled examples.
Source: §1 · p. 1, §2.1 · p. 3, §2.2 · p. 3
Results
The methods set new semi-supervised benchmark records across SVHN, CIFAR-10, and CIFAR-100, with gains under non-augmented and augmented settings.
Source: §1 · p. 1, §3.1 · p. 5, §3.2 · p. 6
Takeaways & Limitations
Self-ensembling also improves fully supervised accuracy and provides tolerance against incorrect labels.
Source: §1 · p. 1, §3.5 · p. 7, §3.4 · p. 7
Comparisons with prior work are limited by differing augmentation and fractional max-pooling configurations.
Source: §3.1 · p. 5, §3.2 · p. 6

Abstract

from arXiv · show

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

1 INTRODUCTION

The paper extends ensemble-based prediction into semi-supervised training by using a network’s outputs across epochs, regularization conditions, and input augmentations as targets for unlabeled data. It introduces Π-model and temporal ensembling, reporting improved semi-supervised and fully supervised performance with tolerance to incorrect labels.

Motivation

Self-ensembling averages predictions from one network across training epochs and different regularization or augmentation conditions.The ensemble prediction is intended to better approximate unknown labels than the latest network output.Source: §1 · p. 1

Semi-supervised objective

Unlabeled inputs receive inferred ensemble targets, while labeled inputs retain the supervised cross-entropy objective.The cross-entropy component is evaluated only for labeled inputs.Source: §1 · p. 1, Figure 1 · p. 2

Methods

The paper presents two self-ensembling implementations: the Π-model and temporal ensembling.These methods differ in how they obtain the multiple predictions used for consistency training.Source: §1 · p. 1

Reported contribution

Both approaches surpass prior state-of-the-art semi-supervised results by a considerable margin.The paper also reports improved accuracy in fully labeled settings and tolerance to incorrect labels.Source: §1 · p. 1

Relation to prior work

The methods relate to transform/stability loss, the ladder network’s Γ-model, and bootstrapping for noisy labels.The Π-model is described as a special case or simplification of related consistency-based approaches.Source: §1 · p. 1

2 SELF-ENSEMBLING DURING TRAINING

Self-ensembling trains on consistency between stochastic network evaluations or predictions accumulated from earlier epochs, alongside supervised learning on labeled examples. Temporal ensembling reduces computation and target noise relative to the Π-model, but requires stored cross-epoch predictions and an additional momentum parameter.

Π-model

The Π-model compares two stochastic evaluations of each input and penalizes differences between their prediction vectors.Its loss combines labeled cross-entropy with an unsupervised mean-square consistency term weighted by w(t).Source: §2.1 · pp. 2–3
The Π-model compares full output vectors rather than only final classifications, imposing a stronger consistency requirement.Dropout, Gaussian noise, and input augmentations create the variation between evaluations.Source: §2.1 · p. 3

Training schedule

The unsupervised loss weight ramps up slowly from zero during early training to avoid a degenerate solution.The implementation ramps the weight along a Gaussian curve during the first 80 epochs.Source: §2.1 · p. 3