Source-linked AI summary
Snapshot Ensembles: Train 1, get M for free
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, Kilian Q. Weinberger
TL;DR
Deep-network ensembles improve robustness and accuracy but require costly multiple-model training. Snapshot Ensembling uses cyclic learning rates to save models from several local minima during one training run, achieving lower error without additional training cost.
Problem
Training multiple deep networks for ensemble averaging is computationally expensive because ensemble training cost increases linearly with the number of networks.
Method
Snapshot Ensembling trains one network through multiple local minima, saves its weights at each minimum, and uses cyclic learning-rate restarts to repeat convergence.
Results
Across four datasets and three architectures, Snapshot Ensembles almost always reduce error without increasing training costs, including 3.44% on CIFAR-10 and 17.41% on CIFAR-100.
Takeaways & Limitations
The ensemble’s training time is identical to that of a single traditional model, while its snapshots can be averaged at test time.
Abstract
from arXiv · showhide
Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.
1 INTRODUCTION
Snapshot Ensembling addresses the high cost of deep-network ensembles by reusing one optimization path to collect multiple models, while exploiting local-minimum diversity for accuracy.
- Motivation: Local minima may contain useful performance information, and flat basins are associated with better generalization than sharp basins.Learning-rate magnitude influences whether SGD avoids sharp minima or converges into a nearby minimum.
- Motivation: Ensembles can reduce error because independently optimized networks often make different mistakes, but training them separately scales cost linearly.Different local minima can have similar error rates while producing diverse errors; multiple networks are commonly combined by voting or averaging.
- Method: Snapshot Ensembling trains one network through M local minima, saving weights at each convergence and restarting with a large learning rate between minima.The method uses cyclic learning-rate schedules that abruptly raise and then cosine-anneal the learning rate.
- Method: The entire ensemble takes the same training time as one traditional model, although testing averages selected snapshots.The last m of M models can be used, and multiple Snapshot Ensembles can be combined when parallel training resources are available.
- Results: 3.44% and 17.41% error rates were obtained on CIFAR-10 and CIFAR-100, respectively, across evaluations using ResNet, Wide-ResNet, and DenseNet.Across four datasets, Snapshot Ensembles almost always reduced error without increasing training costs.
2 RELATED WORK
Prior work includes implicit ensembles, knowledge distillation, and learning-trajectory ensembles, while Snapshot Ensembling specifically targets ensemble training cost through cyclic learning rates.
- Implicit ensembles: Implicit ensembles create multiple stochastic subnetworks from one model, improving training and testing efficiency through dropout, dropped connections, or dropped layers.Examples include Dropout, DropConnect, Stochastic Depth, and Swapout.
- Distillation: Knowledge-distillation methods reduce ensemble test-time cost by transferring the knowledge of a cumbersome ensemble into a single model.Snapshot Ensembling is described as complementary because it reduces training cost rather than test-time cost.
- Trajectory ensembles: Several studies form ensembles from intermediate training stages or slices of a learning trajectory, including horizontal, vertical, and temporal ensembling.These methods combine outputs from models observed at different training epochs or trained with different regularization.
- Cyclic learning rates: Cyclic learning-rate research showed that each cycle can produce models nearly competitive with traditional schedules using fewer training iterations.Snapshot Ensembling builds on this result to obtain repeated convergences along one training process.
3 SNAPSHOT ENSEMBLING
Snapshot Ensembling repeatedly moves through local minima with cyclic cosine annealing, snapshots the resulting models, and averages their predictions at test time.
- Core idea: Snapshot Ensembling collects accurate and diverse models by visiting several local minima during one optimization process.The saved snapshots are combined by averaging their test-time predictions.
- Cyclic Cosine Annealing: Cyclic annealing rapidly lowers the learning rate to reach a minimum, then raises it to perturb the model away before repeating the process.The method is designed to obtain multiple convergences rather than one final solution.
- Cyclic Cosine Annealing: Each of M cycles starts with a large learning rate for escaping critical points and ends with a small rate that drives convergence toward a local minimum.The learning rate is represented using a monotonically decreasing function f over each cycle.
- Training process: M = 6 cosine annealing cycles produce intermediate DenseNet models that form an ensemble at the end of training.Figure 2 compares the cyclic schedule with a standard learning-rate schedule using training loss.
- Snapshot Ensembling: Snapshots are taken at each cycle’s training-loss minimum, yielding M models while keeping total training time equal to a standard single-model schedule.The last m snapshots are used because they tend to have the lowest test error.
4 EXPERIMENTS
Experiments evaluate Snapshot Ensembles across benchmark datasets, architectures, training budgets, ensemble sizes, learning-rate schedules, and model-diversity diagnostics. The method generally lowers error at the training cost of one model and can approach the performance of independently trained ensembles.
- Accuracy: Snapshot Ensembles generally achieve lower error than baseline methods across the evaluated architectures and datasets.The experiments use ResNet, Wide-ResNet, and DenseNet on four datasets.
- Accuracy: 17.41% CIFAR-100 error was achieved with a large DenseNet, compared with 19.25% for the same architecture and training cost.The result is reported as outperforming the record under the same training cost and architecture.
- Accuracy: NoCycle Snapshot Ensembles generally have little effect or increase test error, highlighting the importance of cyclic learning rates.SingleCycle performance is mixed and degrades for larger models, while Snapshot Ensembles benefit from information across previous cycles.
- Accuracy: 23.33% ImageNet validation error with M = 2 outperformed the single-model baseline's 24.01% validation error.The paper reports two cycles as the optimal choice for ImageNet under the tested 90-epoch budget.
- Ensemble Size: Adding ensemble members generally improves performance, with especially large error reductions when the second and third models are added.Later snapshots are selected because they have received more training and are likely to have converged to better minima.
- Training Budget: Snapshot Ensembles remain competitive at low training budgets and achieve performance comparable to two independently trained models with the training cost of one.With high budgets, the SingleCycle baseline eventually outperforms Snapshot Ensembles.
5 DISCUSSION
Snapshot Ensembling uses cyclical learning rates to save multiple converged model snapshots during one training run. These snapshots are accurate yet make different predictions, making them effective for test-time ensembles.
- Snapshot Ensembling saves model parameters each time SGD converges to a local minimum during one optimization path.Cyclical learning rates repeatedly lower and raise the learning rate to induce repeated convergence and escape.
- Cyclical snapshots are accurate while producing different predictions, supporting effective test-time ensembling.
- Snapshot Ensembles significantly improve state-of-the-art performance on CIFAR-10, CIFAR-100, and SVHN.
A. Single model and Snapshot Ensemble performance over time
Across almost all evaluated settings, complete Snapshot Ensembles outperform single-model baselines. Small ensembles of two or three snapshots can often match a conventionally trained single model.
- A. Single model and Snapshot Ensemble performance over time: Figures 7-9 compare individual snapshot test error with ensemble test error over time, using cyclic-learning-rate snapshots and a 300-epoch single-model reference.
- A. Single model and Snapshot Ensemble performance over time: Under almost all settings, complete Snapshot Ensembles outperform single-model baselines.
- A. Single model and Snapshot Ensemble performance over time: Ensembles of just 2 or 3 model snapshots often match the performance of a single model trained with a standard learning rate.
- A. Single model and Snapshot Ensemble performance over time: Snapshot Ensembles consistently outperform their individual members, producing a smooth test-error curve over time.