Source-linked AI summary
Maxout Networks
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio
TL;DR
The paper addresses how models can be designed to better exploit dropout’s approximate model averaging. It introduces maxout for improved optimization and averaging with dropout, achieving state-of-the-art classification performance on four benchmark datasets.
Problem
The paper asks whether models designed specifically for dropout can better exploit its approximate model-averaging behavior than arbitrary models.
Method
The authors introduce maxout, a model designed to facilitate optimization with dropout and improve dropout’s approximate model averaging.
Results
Maxout with dropout set the state of the art on all four evaluated benchmark datasets.
Takeaways & Limitations
The findings support maxout as an activation function particularly well suited to training with dropout.
Abstract
from arXiv · showhide
We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. We empirically verify that the model successfully accomplishes both of these tasks. We use maxout and dropout to demonstrate state of the art classification performance on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.
1. Introduction
Dropout provides an inexpensive way to train and approximately average predictions from parameter-sharing model ensembles, improving performance across diverse tasks (Hinton et al., 2012). The paper argues for designing models that enhance dropout’s model-averaging abilities and proposes maxout, which achieves state-of-the-art results on four benchmark datasets.
- Background: Dropout trains a large parameter-sharing ensemble and approximately averages its models’ predictions at low cost.It has improved state-of-the-art performance on tasks including audio classification and large-scale object recognition (Hinton et al., 2012).
- Motivation: The paper challenges treating dropout as an indiscriminate, modest enhancement and instead advocates designing models specifically around its averaging mechanism.This approach aims to enhance dropout’s abilities rather than applying it arbitrarily.
- Motivation: Dropout training differs substantially from ordinary stochastic gradient descent and is most effective when updates take relatively large steps in parameter space.In this regime, each update can make a significant change to the model.
- Contribution: Maxout is proposed as a simple model designed to improve both optimization with dropout and dropout’s model-averaging performance.The authors use maxout with dropout to achieve state-of-the-art results on four benchmark datasets.
2. Review of dropout
Dropout trains an ensemble of masked sub-models sharing parameters, using a different binary mask for each training example. Its exponentially large ensemble motivates inexpensive predictive averaging, including geometric means for suitable model families.
- Dropout trains sub-models formed by randomly masking variables in the input and hidden layers while sharing one parameter set.Each mask μ defines a member distribution p(y | v; θ, μ), and training follows its gradient on each example.
- Because dropout produces exponentially many models, prediction requires a tractable approximation to averaging their outputs.The functional form of the model determines how the ensemble’s predictions can be combined efficiently.
- For softmax models, renormalizing the geometric mean of the masked models’ predictive distributions provides an inexpensive ensemble prediction.
3. Description of maxout
Maxout is a feed-forward architecture whose hidden layers use maxout units, taking maxima over learned affine responses. It supports dropout by masking inputs before weight multiplication and can approximate broad classes of convex activations.
- Maxout units: Maxout replaces conventional activations with units that compute the maximum of k learned affine feature responses.In convolutional networks, feature maps pool across k channels in addition to spatial locations.
- Dropout: During dropout training, masks are applied immediately before multiplication by the weights rather than to inputs of the max operator.
- Activation flexibility: In multiple dimensions, a maxout unit can approximate arbitrary convex functions, including behaviors corresponding to rectified-linear, absolute-value, and approximate quadratic activations.Figure 1 illustrates these activation-function constructions in a two-dimensional diagram using a one-dimensional input.
- Representation properties: Maxout representations are not sparse, but their gradients are highly sparse and dropout artificially sparsifies the effective representation during training.Maxout may saturate on one side or the other, but this is described as a measure-zero event.
4. Maxout is a universal approximator
Maxout networks are universal approximators: with arbitrarily many affine components per unit, just two hidden maxout units can approximate any continuous function arbitrarily well on a compact domain. The proof combines piecewise-linear approximation with representing continuous piecewise-linear functions as differences of convex piecewise-linear functions.
- Proof mechanism: The construction works because any continuous piecewise-linear function can be expressed as the difference of two convex piecewise-linear functions.Each convex component is represented by a maxout unit, with enough affine components to achieve the desired approximation.
- Proof mechanism: By Stone-Weierstrass, every continuous function on a compact domain can first be approximated arbitrarily well by a continuous piecewise-linear function.For any positive ϵ, the approximation g satisfies |f(v) − g(v)| < ϵ throughout the compact domain.
- Universal approximator theorem: Two hidden maxout units can approximate any continuous function arbitrarily well on a compact domain.This is the universal approximator theorem for maxout networks.
- Proof mechanism: The resulting piecewise-linear representation exactly matches a two-hidden-unit maxout network when each unit has sufficiently many affine components.Increasing the number of affine components controls the desired approximation accuracy.
5. Benchmark results
Maxout achieved state-of-the-art classification performance on MNIST, CIFAR-10, CIFAR-100, and SVHN. Reported test errors were 0.45% on MNIST, 11.68% on CIFAR-10, 38.57% on CIFAR-100, and 2.47% on SVHN.
- Overall results: Maxout set the state of the art on all four evaluated benchmark datasets.
- MNIST: 0.45% test error on standard MNIST established a new state of the art using three convolutional maxout layers with spatial max pooling.
- CIFAR-10: 11.68% test error on CIFAR-10 improved the state of the art by over two percentage points with convolutional maxout, dropout, and data augmentation.
- CIFAR-100: 38.57% test error on CIFAR-100 was state of the art despite applying hyperparameters found on CIFAR-10 without extensive cross-validation.
- SVHN: 2.47% test error on SVHN set the state of the art using three convolutional maxout hidden layers and a densely connected maxout layer.
6. Comparison to rectifiers
A large cross-validation experiment found that maxout clearly outperformed rectifiers, while preprocessing and model size also improved rectifier and dropout performance beyond prior state of the art. Rectifiers approached maxout’s generalization only with roughly k times more state and parameters.
- Comparison to rectifiers: Maxout offers a clear improvement over rectifiers in a large cross-validation experiment, while preprocessing and model size also improve rectifier and dropout results beyond prior state of the art.The comparison used four architectures, including matched-parameter and matched-unit rectifier networks, as well as a larger rectifier model.
- Comparison to rectifiers: Rectifier performance correlates with output units, whereas maxout performance correlates with the number of filters.Rectifier units benefit little from cross-channel pooling, while maxout benefits from increasing filters.
- Comparison to rectifiers: Rectifiers need about k times more state and parameters than maxout to approach comparable generalization performance.The strongest rectifier setting uses the same number of filters without cross-channel pooling, but requires k times as many units.
7. Model averaging
This section explains maxout’s compatibility with dropout by showing that dropout averaging is exact for locally linear deep networks and more accurate for maxout than tanh. It also argues that dropout encourages maxout units to form large linear regions around training inputs.
- Model averaging: Dividing weights by 2 achieves the best test error despite sampled sub-model predictions approaching it, supporting dropout as an approximation to averaging many models.The correspondence is clearer for maxout, according to Figure 7.
- Model averaging: Dropout performs exact model averaging in deeper architectures when they are locally linear over inputs visited under different dropout masks.The exactness extends from softmax regression to multiple linear layers, which retain the same representational power but impose a different inductive bias through factorized weights.
- Model averaging: Dropout training encourages maxout units to develop large linear regions around training inputs, helping sub-models maintain similar activations when inputs are dropped.Because each sub-model must predict well, each unit should have roughly the same activation across dropout masks; arbitrary parameters may instead move effective inputs outside the clean-input region.
- Model averaging: The dropout weight-division approximation matches sampled-model averaging more accurately for maxout units than for tanh units.Figure 8 measures this agreement using the KL divergence between the divided-weight prediction and the geometric mean of sampled-model predictions, which decreases as samples increase.
8. Optimization
Maxout improves dropout optimization relative to pooled rectifier units by reducing training error, preserving optimization as depth increases, and improving filter usage and gradient variability. Dropout differs from SGD by benefiting from large learning rates and rapidly exploring fluctuating objectives.
- Optimization experiments: Maxout reduced SVHN training error from 7.3% with rectifier units to 5.1% and degraded gracefully with depth, unlike pooled rectifiers at six and seven layers.The SVHN test used a small two-hidden-layer convolutional model; the depth experiment used MNIST models with 80 units per layer and k=5.
- Activation dynamics: During dropout training, rectifiers transition from positive to zero activation more often than oppositely, whereas maxout moves between positive and negative signs at roughly equal rates.This activation-transition contrast is presented as an optimization-related difference between rectifier and maxout units.
- Dropout optimization: Dropout optimization benefits from large learning rates and fluctuating objectives, rapidly exploring directions and rejecting those that worsen performance, unlike steady small-step SGD.Dropout behaves differently from ordinary stochastic gradient descent because the dropout mask changes the gradient substantially; otherwise training simplifies to SGD.
- Gradient flow: Maxout produced 1.4× greater output-weight gradient variance and 3.4× greater first-layer gradient variance than rectifiers during MNIST dropout training.The measurements tested whether rectifier networks suffer diminished gradient flow to lower layers by monitoring variance across dropout masks.
9. Conclusion
The paper proposes maxout, an activation function suited to dropout and covered by a universal approximation theorem. Experiments show that dropout approximates model averaging well in deep models, with more accurate averaging for maxout than tanh units.
- Maxout is a new activation function designed to be particularly well suited for training with dropout, with a proven universal approximation theorem.
- Dropout empirically achieves a good approximation to model averaging in deep models.
- Model-averaging approximation is more accurate for maxout units than for tanh units, enabling maxout to exploit dropout’s averaging behavior.