PaperlayerView exact revision on arXiv

Source-linked AI summary

There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson

arXiv:1806.05594v3cs.LGcs.AIcs.CVstat.ML

TL;DR

Consistency regularization performs well in semi-supervised learning, but SGD continues exploring distant solutions with changing predictions. The paper analyzes this geometry and applies SWA and fast-SWA to average weights, achieving best-known results across several benchmarks and label settings.

  • Problem

    Consistency-based semi-supervised methods achieve strong results, but their loss geometry and SGD training trajectories require conceptual understanding.

  • Method

    The paper analyzes consistency-model trajectories and trains Π and Mean Teacher models with stochastic weight averaging, including fast-SWA for faster convergence.

  • Results

    Weight averaging improves reported semi-supervised results across CIFAR-10 and CIFAR-100 label settings; CIFAR-10 reaches 5.0% error with 4k labels.

  • Takeaways & Limitations

    Averaging weights or predictions exploits the diverse solutions explored by SGD and can improve generalization for consistency-based models.

  • Takeaways & Limitations

    The paper notes that its simplified isotropic perturbations do not generally lie along the data manifold, making natural-image perturbations more pertinent.

Abstract

from arXiv · show

Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

1 INTRODUCTION

Semi-supervised learning uses unlabeled data to improve performance where labels are scarce, with consistency-based methods currently leading results. This paper analyzes their training geometry and proposes weight averaging to improve optimization and benchmark performance.

  • Semi-supervised methods combine labeled and unlabeled data, which is valuable when labels are scarce and expensive.Medical imaging is given as an example application.
  • Consistency-based methods stabilize predictions under input or weight perturbations and currently achieve the best semi-supervised results.They can also be used with state-of-the-art architectures.
  • The simplified Π model implicitly regularizes output Jacobians with respect to inputs and weights, encouraging flatter solutions.Reduced Jacobian norms and solution flatness have been related to generalization.
  • Consistency-based models have larger distances between epoch weights, wider error curves, and substantially different predictions across SGD iterations.These observations are reported for Π and Mean Teacher models compared with supervised training.
  • SGD explores multiple distant solutions in consistency-based methods, motivating weight averaging to stabilize trajectories and center solutions in flat loss regions.The paper also reports diverse predictions among SGD iterates and improved generalization from averaging weights or predictions.
  • Fast-SWA averages multiple networks within longer learning-rate cycles and converges to a good solution much faster than SWA.The longer cycles increase the distance and prediction diversity among averaged weights.
  • 5.0% error on CIFAR-10 with 4k labels improved the best reported literature result by 1.3%.Weight averaging also improved results across several CIFAR-10 label counts and CIFAR-100 with 10k labels.

2 BACKGROUND

Consistency-based semi-supervised models train predictions to remain similar under perturbations of inputs or network parameters. Their consistency loss is combined with supervised cross-entropy, while different models use self-predictions, teacher predictions, or output averages.

  • Consistency-based models encourage similar predictions under small perturbations of inputs or network parameters.For example, translated versions of one image should produce similar predicted probabilities.
  • The consistency loss penalizes differences between predictions from perturbed inputs and weights, typically using mean squared error or KL divergence.The supplied passages identify the student predictions and perturbed inputs as the compared quantities.
  • The total training loss combines supervised cross-entropy with a consistency term weighted by λ > 0.For classification, LCE is the cross entropy between model predictions and supervised labels.
  • Π Model: The Π model uses its own student network as teacher with image augmentations, Gaussian noise, and binary dropout as perturbations.The listed data perturbations include translations, crops, and flips.
  • Mean Teacher Model: Mean Teacher uses the same data and weight perturbations as the Π model, while its teacher is formed from an exponential moving average of student weights.The decay rate α is usually set between 0.9 and 0.999.
  • Other Consistency-Based Models: Temporal Ensembling averages student outputs for teacher predictions, whereas Virtual Adversarial Training enforces consistency under adversarial input perturbations.VAT defines the perturbed input as x′ = x + ϵr_adv.

3 UNDERSTANDING CONSISTENCY-ENFORCING MODELS

Consistency-based models regularize sensitivity and explore broad, diverse regions of weight space late in training, making weight averaging especially effective.

  • Loss geometry: Consistency loss penalizes input-output Jacobian norms and Hessian eigenvalues, quantities associated with flatter solutions and generalization.For linear models, penalizing the input Jacobian norm corresponds exactly to L2 weight decay.
  • Loss geometry: Isotropic input perturbations need not follow the data manifold, whereas standard data augmentations can be interpreted as penalizing the manifold Jacobian norm.The simplified analysis uses small Gaussian perturbations to the input.
  • SGD trajectories: Consistency gradients remain high and dominate cross-entropy gradients through training, so Π and Mean Teacher make substantially larger optimization steps than supervised models.These steps suggest SGD continues exploring a set of solutions rather than converging to one minimizer.
  • SGD trajectories: Π and Mean Teacher have wider error surfaces and larger distances between late-training solutions than supervised training.The wider surfaces are consistent with the flatter solutions associated with consistency regularization.
  • Prediction diversity: 7.1% and 6.1% of test predictions differ between epochs 170 and 180 for Π and Mean Teacher, versus 3.9% for supervised learning.This prediction diversity supports continued exploration of plausible solutions late in training.
  • Averaging benefits: 1.2 ± 0.2% average error reduction from weight averaging over Π and Mean Teacher pairs matches or exceeds the 0.9 ± 0.2% gain from prediction ensembling.Weight averaging is emphasized because it has lower test-time cost and slightly higher performance in these comparisons.

4 SWA AND FAST-SWA

SWA averages SGD solutions under modified learning-rate schedules, while fast-SWA collects more points within cycles to accelerate averaging-based training.

  • Geometric motivation: SWA and fast-SWA average solutions explored across broad, locally flat regions, where separated SGD solutions can yield substantial error reduction.Figure 2 connects weight-averaging gains with distance between solutions and illustrates the averaged point within the error surface.
  • SWA: SWA averages weights traversed by SGD using a constant or cyclical learning-rate schedule to improve generalization.The method is applied to the student network in both Π and Mean Teacher models.
  • SWA: SWA collects weights at minimum learning rates after pretraining and uses their average for prediction without interfering with training.The cyclical schedule repeats learning rates over a specified cycle after pretraining.
  • fast-SWA: fast-SWA averages networks every k < c epochs, or multiple weights within one epoch when k < 1, instead of updating once per cycle.This design reduces the extra epochs needed to collect enough averaged models.
  • fast-SWA: fast-SWA includes higher-error, high-learning-rate models but converges substantially faster than SWA and has lower performance variance.The method is motivated by the larger gains observed when averaging points farther apart.

5 EXPERIMENTS

The experiments evaluate SWA and fast-SWA across datasets, architectures, label quantities, learning-rate schedules, and domain adaptation settings. Weight averaging consistently improves consistency-based models, while fast-SWA reaches comparable error faster and advances reported benchmarks.

  • 5.1 SETUP: Weight averaging improves Π and Mean Teacher performance across CIFAR-10, CIFAR-100, architectures, and learning-rate schedules.The study evaluates a 13-layer CNN and a Shake-Shake residual network under short and long schedules.
  • 5.2 CIFAR-10: For all quantities of labeled CIFAR-10 data, fast-SWA substantially improves test accuracy for both CNN and Shake-Shake architectures.The evaluated label settings are 1k, 2k, 4k, 10k, and 50k labels.
  • 5.2 CIFAR-10: 10.5% error is reached by fast-SWA at epoch 200 for CIFAR-10 with 4k labels, whereas SWA attains similar error at epoch 350.Fast-SWA continues improving while averaging weights collected at high learning rates.
  • 5.2 CIFAR-10: Π+fast-SWA outperforms MT+fast-SWA on CIFAR-10 with 4k, 10k, and 50k labeled examples using Shake-Shake.Weight averaging reduces the performance gap between Π and Mean Teacher.
  • 5.3 CIFAR-100: On CIFAR-100, fast-SWA reduces errors substantially faster than SWA in the evaluated 10k-label and 50k+500k settings.The additional unlabeled data come from Tiny Images, with a separate 50k+237k* setting using CIFAR-100-class images.
  • 5.5 PRELIMINARY RESULTS ON DOMAIN ADAPTATION: 16.8% error improves the previous domain-adaptation result of 19.9% for CIFAR-10 to STL after applying fast-SWA.The method is applied to a consistency-based domain-adaptation model.

6 DISCUSSION

The discussion interprets consistency-model training as exploration of diverse solutions rather than convergence to one point. It reports improved benchmark results from weight averaging and points to domain adaptation as a promising additional application.

  • 6 DISCUSSION: SGD continues exploring a diverse set of plausible solutions instead of converging to a single solution in Π and Mean Teacher models.This observation motivates averaging across the training trajectory.
  • 6 DISCUSSION: A variant of SWA advances the best known semi-supervised results on classification benchmarks.The discussion frames this as the principal benchmark consequence of the geometric analysis.
  • 6 DISCUSSION: Weight averaging shows promise in domain adaptation, although domain adaptation is not the paper’s primary focus.The discussion suggests application-specific analysis could extend these ideas to other areas.

A.1 ADDITIONAL PLOTS

Additional plots examine how unlabeled data, weight averaging, and training directions affect error geometry. They show centered fast-SWA solutions, wider consistency-model surfaces, higher gradient variance, and a relationship between prediction diversity and averaging gains.

  • Figure 6: Adding unlabeled data monotonically improves Π test accuracy while making solutions narrower along random rays.The comparison uses 4k labeled examples and varying quantities of unlabeled data.
  • Figure 6: Fast-SWA finds a centered solution, while SGD solutions lie near the boundary of a wide flat region for Π, Mean Teacher, and supervised training.The comparison follows directions connecting fast-SWA solutions with SGD iterates.
  • Figure 6: Consistency-model train and test error surfaces are wider along directions connecting SGD and averaged solutions than those for supervised training.The plotted methods are Π, Mean Teacher, and supervised training.
  • Figure 7: Gradient variance is much larger for Π and Mean Teacher than for supervised training.The left panel tracks the trace of gradient covariance during training.
  • Figure 7: Prediction diversity is highly correlated with the improvement from weight averaging, whereas weight distance has a less prominent correlation with diversity.The middle and right panels compare averaging gains, diversity, and pairwise weight distances.

A.2 DETAILED RESULTS

The experiments compare Π, Mean Teacher, supervised baselines, SWA, and fast-SWA on CIFAR-10 and CIFAR-100 using CNN and Shake-Shake architectures. Tables and training curves report test-error behavior across label settings and training epochs.

  • Evaluation scope: The evaluation covers CIFAR-10 and CIFAR-100 with both 13-layer CNN and Shake-Shake architectures.Tables 2–5 and Figures 8–9 summarize these comparisons.
  • Averaging comparison: Applying EMA to the Π student network without using it as a teacher typically yields a small test-error improvement.The paper distinguishes this student-weight averaging from Mean Teacher’s use of EMA as a teacher.
  • Tabulated results: The tables report semi-supervised test errors for multiple labeled-data settings and include prior published results where available.Table 2 covers CIFAR-10 with a 13-layer CNN, while Table 3 covers CIFAR-100 with the same architecture.
  • Architectures: The reported comparisons use the 13-layer CNN for Tables 2–3 and Shake-Shake with ResNet-26 for Tables 4–5.The Shake-Shake tables report results on both datasets using the regularized ResNet-26 architecture.
  • Training dynamics: Training-epoch curves compare baseline models, SWA, and fast-SWA for Π and Mean Teacher across several CIFAR-10 and CIFAR-100 label quantities.The CNN experiments include 1k, 2k, 4k, and 10k labels on CIFAR-10 and several larger settings on CIFAR-100.

A.3 EFFECT OF LEARNING RATE SCHEDULES

fast-SWA has one principal hyperparameter, the cycle length c, and its performance remains stable across a wide range of values. Cyclical learning rates generally accelerate convergence by collecting more varied weights.

  • Cycle length: fast-SWA performance is not sensitive to cycle length c over a wide range of values.The sensitivity study is shown in Figure 10a.
  • Learning-rate schedule: Cyclical learning rates generally make fast-SWA converge faster than a constant learning-rate schedule.The paper attributes this to greater variety in the collected weights.

A.4 EMA VERSUS SWA AS A TEACHER

The paper analyzes EMA and SWA as weight-averaging strategies and relates their behavior to consistency-model geometry, Jacobian regularization, and SGD trajectories. It reports that SWA can outperform EMA after convergence, while SWA as a teacher performs on par with EMA as a teacher.

  • Averaging methods: EMA emphasizes recent network weights more strongly than SWA.This distinction affects their behavior at different stages of training.
  • Training regime: EMA helps substantially early in training, whereas SWA provides greater gain once the student is near the optimum.The paper associates this difference with rapid early student changes and later convergence near the optimum.
  • Teacher comparison: Using SWA as a teacher after Mean Teacher pretraining performs on par with using EMA as a teacher.The switch to an SWA teacher occurs at epoch 150 in the reported experiment.
  • Jacobian interpretation: Consistency regularization with small Gaussian perturbations yields a stochastic estimator of the squared Frobenius norm of the network-output Jacobian.The estimator is interpreted through a Gaussian probe and extended to perturbations along the data-manifold tangent space.
  • Manifold perturbations: Consistency regularization with natural perturbations can be understood as penalizing Jacobian variation along the image-manifold tangent space.The perturbation distribution is projected onto the tangent space at each image.
  • Sharpness and averaging: Lower Jacobian norms reduce local random-ray sharpness, while averaging high-learning-rate samples can reduce estimator mean squared error when m > n.The latter comparison motivates including higher-variance points in fast-SWA averages.

A.8 NETWORK ARCHITECTURES

The experiments use two neural-network families: a 13-layer CNN and Shake-Shake with ResNet-26. Training configurations specify dataset-dependent batch sizes, learning rates, schedules, averaging, and consistency-loss settings.

  • Network architectures: The study uses a 13-layer CNN and a 26-2x96d Shake-Shake architecture with 12 residual blocks.The CNN follows prior consistency-learning architectures, while Shake-Shake uses Gastaldi’s regularized design.
  • CNN configuration: The 13-layer CNN architecture differs from the cited implementation by removing a Gaussian noise layer after horizontal flipping.This architecture is used for CIFAR-10 and CIFAR-100 CNN experiments.
  • Training schedules: The experiments use short and long cosine-annealing schedules, with the long schedule applied to Shake-Shake experiments.The short schedule has training length ℓ = 180, while the long schedule has ℓ = 1800.
  • Averaging and optimization: fast-SWA averages weights every third epoch, while Mean Teacher uses EMA with decay rate α = 0.97.The Π model back-propagates gradients through the student side only.
  • Consistency training: Consistency cost λ is ramped from 0 to 100 during the first five epochs, and cosine-annealing rates are used without learning-rate ramp-up.The same hyperparameters are used for Π and Mean Teacher within each experiment setting.
  • Dataset-specific settings: Batch sizes and labeled-batch sizes vary by architecture and dataset label setting, while the maximum learning rate is typically 0.1.The CIFAR-10 Shake-Shake experiments use η0 = 0.05; CIFAR-100 Shake-Shake uses η0 = 0.1.

A.10 DOMAIN ADAPTATION

For CIFAR-10-to-STL domain adaptation, fast-SWA compares averaging frequencies and applies confidence thresholding and geometric augmentation within a specific training setup. Averaging every iteration converges substantially faster and achieves better test accuracy than averaging once per epoch.

  • A.10 DOMAIN ADAPTATION: The domain-adaptation experiment uses the MT+CT+TFA setting, combining confidence thresholding with translation, flipping, and affine-transformation augmentation.This setting follows the best experiment configuration identified by French et al. (2018).
  • A.10 DOMAIN ADAPTATION: The implementation replaces Adam with SGD using Nesterov momentum and cosine annealing, with ℓ0 = 600, ℓ = 550, and c = 50.The passage describes two fast-SWA variants: averaging once per epoch or once every iteration.
  • A.10 DOMAIN ADAPTATION: 600 epochs instead of 3000 are needed when fast-SWA averages every iteration, and test accuracy is better.Averaging more frequently did not improve convergence or final results in the semi-supervised experiments, suggesting task-specific behavior.
  • A.10 DOMAIN ADAPTATION: The training code is based on French et al. (2018), with the original public implementation modified to apply fast-SWA.The base model uses public Pytorch code and is adapted from the Mean Teacher implementation for the Π model.
  • A.10 DOMAIN ADAPTATION: fast-SWA 1 averages weights every epoch and reports its final result at epoch 3000, whereas fast-SWA 2 averages every iteration and reports at epoch 600.These are the two averaging schedules compared in the domain-adaptation table.
Loading 1806.05594v3…