Source-linked AI summary
Swapout: Learning an ensemble of deep architectures
Saurabh Singh, Derek Hoiem, David Forsyth
TL;DR
Deep networks face parameter-growth and optimization difficulties, motivating methods that improve trainability without relying only on greater depth. Swapout stochastically samples a rich family of architectures, including dropout, stochastic depth, and residual networks, and achieves comparable or improved performance with wider, shallower models.
Problem
Increasing depth can improve accuracy but creates parameter growth and difficulties training long chains of dependent variables.
Method
Swapout is a stochastic training method that averages over a rich set of architectures, including dropout, stochastic depth, and residual architectures as special cases.
Results
A 32 layer wider swapout model matches the performance of a 1001 layer ResNet on CIFAR-10 and CIFAR-100.
Takeaways & Limitations
Relatively shallow swapout networks can achieve comparable performance to extremely deep residual networks while using parameters efficiently.
Takeaways & Limitations
The paper has not systematically searched for the best stochastic training schedule, so further improvements may be possible.
Abstract
from arXiv · showhide
We describe Swapout, a new stochastic training method, that outperforms ResNets of identical network structure yielding impressive results on CIFAR-10 and CIFAR-100. Swapout samples from a rich set of architectures including dropout, stochastic depth and residual architectures as special cases. When viewed as a regularization method swapout not only inhibits co-adaptation of units in a layer, similar to dropout, but also across network layers. We conjecture that swapout achieves strong regularization by implicitly tying the parameters across layers. When viewed as an ensemble training method, it samples a much richer set of architectures than existing methods such as dropout or stochastic depth. We propose a parameterization that reveals connections to exiting architectures and suggests a much richer set of architectures to be explored. We show that our formulation suggests an efficient training method and validate our conclusions on CIFAR-10 and CIFAR-100 matching state of the art accuracy. Remarkably, our 32 layer wider model performs similar to a 1001 layer ResNet model.
1 Introduction
Swapout is a stochastic training method that generalizes several architectural and regularization strategies by sampling a rich set of networks. The paper reports improved accuracy over same-depth residual networks and competitive performance from wider, shallower models.
- Method: Swapout samples architectures that include dropout, stochastic depth, and residual networks as special cases.Its general formulation averages over architectures formed from current and previous layer outputs.
- Method: Swapout produces independently sampled outputs per unit, allowing feedforward behavior, skip connections, or sums of earlier outputs.This creates architectural variation both within and across layers.
- Motivation: Increasing depth improves swapout accuracy, while depth also creates parameter growth and training difficulties in deep networks.The introduction notes that architectural innovations address optimization through shorter paths from input to loss.
- Motivation: 1000-layer depth has thin theoretical support as a requirement for practical problems, despite evidence that very large depths can help experimentally.The passage distinguishes empirical usefulness from theoretical necessity.
- Results: Increasing width substantially improves swapout accuracy, making a wider model competitive with a residual network 1.5 orders of magnitude deeper.The comparison also involves a deeper model with more parameters.
- Results: Swapout improves residual-network performance at the same depth and makes wider, much shallower networks competitive with very deep residual networks.These are the paper's stated contribution-level conclusions.
2 Related Work
The paper places swapout among methods that improve trainability and generalization through stochastic unit- or layer-level behavior. Unlike dropout or stochastic depth alone, it combines multiple architectural choices and can be evaluated as an implicit ensemble.
- Architectural context: Skip connections improve trainability by allowing signals to bypass layers, while residual networks use identity skips to support extremely deep models.The related-work discussion contrasts these architectures with swapout's random unit-level skipping.
- Architectural context: Swapout differs from highway and residual architectures by applying random skipping at the unit level rather than deterministically or through gated layer-level paths.The method exploits the same broad skipping principle at finer granularity.
- Stochastic training: Dropout discourages unit co-adaptation, while stochastic depth randomly skips layers and reliably improves test-time performance through regularization.These methods motivate swapout's stochastic training design.
- Stochastic training: Swapout shares dropout's stability behavior in the cited framework, where dropout reduces a Lipschitz constant.The paper reports that swapout enjoys the same behavior as dropout in this framework.
- Ensembling: Swapout can be viewed as an implicit ensemble because multiple stochastic instances draw from one shared parameter set.This contrasts with explicit ensembles, which train distinct parameter sets.
- Architectural comparison: Figure 1 compares feedforward, residual, skipforward, and swapout blocks by showing what each unit reports from X and F(X).Swapout allows each unit to report 0, X, F(X), or X + F(X).
3 Swapout
Swapout is a stochastic architecture-sampling method that generalizes dropout and stochastic depth by combining randomly selected current and earlier-layer outputs. It represents a tied-parameter family of networks, with inference choices and normalization interactions that affect how its ensemble is used.
- Swapout: Swapout generalizes dropout and stochastic depth by sampling unit-level combinations of current and previously computed layer outputs.Earlier outputs can include the block input, enabling feedforward, skipped, and residual-like unit behavior.
- Swapout: Swapout extends the sampled architecture family to include dropout, stochastic depth, block-level skips, unit-level skips, and residual connections.These behaviors arise from different random choices of the stochastic parameters.
- Swapout: Swapout inherits dropout-like regularization while additionally presenting units with inputs from earlier layers, discouraging co-adaptation across layers.The authors connect this behavior to stability arguments for randomly trained networks and state that swapout makes stability no worse under their bound.
- Swapout: A swapout model represents a tied-parameter family of networks sampled during training, supporting either deterministic inference or averaged stochastic inference.Stochastic inference samples several family members and averages their predictions.
- Inference: Deterministic expectation-based inference is generally inexact for swapout, and stochastic inference produces significantly better results in the reported experiments.The mismatch arises because ReLU does not generally commute with the expectation of sums from multiple stochastic paths.
- Inference: Deterministic inference interacts poorly with batch normalization because training-time variance estimates differ from test-time statistics as stochastic paths are averaged.The resulting scale-estimation errors can accumulate across layers.
4 Experiments
Experiments on CIFAR-10 and CIFAR-100 evaluate Swapout across training schedules, inference schemes, widths, and comparable ResNet baselines. Swapout consistently outperforms comparable models, while wider shallow models approach the performance of much deeper ResNets.
- Baselines: Swapout models consistently outperform comparable 20-layer baselines on CIFAR-10.Table 1 compares models trained with Swapout against fair baselines of similar architecture.
- Training schedule: Different stochastic training schedules significantly affect performance, with less randomness in early layers performing best.The schedules vary Bernoulli parameters across layers and are evaluated with deterministic and stochastic inference.
- Inference: Stochastic inference with 30 forward passes outperforms deterministic inference across all Table 3 experiments.The authors attribute this to more accurate expectations and better interaction with batch normalization, not only to parameter-sharing ensembles.
- Width: Increasing width improves Swapout performance, and Swapout outperforms corresponding ResNets across the investigated model sizes.Table 3 reports deterministic and 30-sample stochastic inference for models with different filter counts.
- Parameter efficiency: 4.76% error with 7.43M parameters compares with 4.91% error and 10.2M parameters for the ResNet version.This example illustrates the reported parameter efficiency of Swapout on CIFAR-10.
- Cross-dataset comparison: 22.72% error with 7.46M parameters is competitive with a 1001-layer ResNet at 22.71% error and 10.2M parameters on CIFAR-100.The shallower Swapout model also improves a 20-layer ResNet Ours model by more than 2%.
5 Discussion and future work
Swapout shows reliable performance and parameter-efficiency gains, including competitive results from shallower networks, while the study leaves schedule optimization systematicity and broader architectural comparisons open.
- Discussion: Swapout achieves comparable performance to extremely deep residual networks with relatively shallow architectures.The paper reports this as a broader conclusion about Swapout’s performance and parameter use.
- Future work: The authors have not systematically searched for the best stochastic training schedule, leaving possible improvements unexplored.They state that different schedules produce different behaviors but do not optimize schedule choice systematically.
- Discussion: A 32-layer wider Swapout model performs competitively with a 1001-layer ResNet on CIFAR-100.Table 5 identifies the 32-layer model as much shallower and reports the comparison against the 1001-layer model.
- Discussion: Figure 2 indicates that relatively few stochastic-inference samples provide a reliable estimate of mean error.The estimate uses 30 repetitions for each sample count, with uncertainty shown as the standard error of the mean.
- Future work: The experiments compare Swapout primarily with residual networks because they are current top performers on CIFAR-10 and CIFAR-100.The authors note that other versions of the method remain interesting to study.