Source-linked AI summary
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Tim Salimans, Diederik P. Kingma
TL;DR
Deep-network training depends on objective curvature and parameterization, creating a need for broadly useful conditioning improvements. The paper introduces weight normalization, which separates each weight vector’s scale from its direction and optimizes the reparameterized variables directly. Across supervised recognition, generative modeling, and deep reinforcement learning, it reports consistent advantages, with low overhead and no minibatch dependencies.
Problem
First-order optimization is sensitive to objective curvature and parameterization, while batch normalization introduces minibatch dependencies that limit suitability for some models and applications.
Method
Weight normalization reparameterizes each weight vector using a direction vector and separate scalar scale, then applies stochastic gradient descent to those parameters.
Results
Weight normalization shows a consistent advantage across four models spanning supervised image recognition, generative modeling, and deep reinforcement learning, while providing much of batch normalization’s speed-up.
Takeaways & Limitations
The method is easy to apply, has low computational overhead, and does not introduce dependencies between minibatch examples.
Takeaways & Limitations
The data-dependent initialization method is unavailable for recursive models such as RNNs and LSTMs, which require standard initialization.
Abstract
from arXiv · showhide
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.
1 Introduction
Deep-network optimization is sensitive to objective curvature and parameterization, motivating methods that improve conditioning across common neural-network building blocks. Weight normalization offers a simple, deterministic reparameterization with low overhead.
- First-order optimization can struggle when the objective has pathological curvature, and curvature changes under reparameterization.
- Neural networks widely reuse weighted-sum neurons followed by elementwise nonlinearities, so improving these building blocks can benefit many architectures.
- Existing conditioning approaches include approximate natural gradients based on Fisher-matrix preconditioning, whitening, or related transformations.
- Weight normalization reparameterizes neural-network weights to improve optimizability without minibatch-dependent gradient noise.
- The method adds negligible computation and no additional memory while showing encouraging results across diverse deep-learning applications.
2 Weight Normalization
Weight normalization expresses each weight vector through a direction parameter and a separate scale, then optimizes those parameters directly. This decoupling changes gradient geometry, improving conditioning while retaining low-cost, minibatch-independent computation.
- A neuron computes a weighted input sum plus bias, followed by an elementwise nonlinearity.
- Weight normalization reparameterizes w using a vector v and scalar g, with ||w|| = g independent of v.
- Unlike earlier post-update normalization, the method performs stochastic gradient descent directly in the new v,g parameterization.
- Gradients: The resulting gradient is scaled by g/||v|| and projected away from the current weight vector, bringing gradient covariance closer to identity.
- Optimization dynamics: The norm of v can grow during updates, self-stabilizing the effective gradient scale; this mechanism is qualitative for momentum and separately tuned learning rates.
- Optimization dynamics: Weight normalization is empirically robust to learning-rate choice because ||v|| grows when the learning rate is too large, reducing the effective rate.
- Relation to batch normalization: For a single layer with whitened inputs, weight normalization is equivalent to batch normalization, while deeper-network equivalence does not generally hold.
- Relation to batch normalization: Weight normalization is often cheaper and less noisy than batch normalization and remains independent of minibatch inputs, aiding use in RNNs, LSTMs, and reinforcement learning.
3 Data-Dependent Initialization of Parameters
Weight normalization does not itself stabilize feature scales across layers, so the method relies on data-dependent initialization to set suitable initial activation statistics. This initialization works well experimentally but is limited for recursive models.
- Weight normalization lacks batch normalization’s robustness to parameter initializations with varying feature scales across layers.Proper initialization is therefore important.
- The initialization sets bias b and scale g using pre-activation statistics from one minibatch before training.This matches the initial minibatch statistics without introducing minibatch dependencies during subsequent optimization.
- Initially, features have zero mean and unit variance before the nonlinearity, although later minibatches may have slightly different statistics.The authors report that this initialization works well experimentally.
- For recursive models such as RNNs and LSTMs, this data-dependent initialization must be replaced with standard initialization methods.
4 Mean-only Batch Normalization
Mean-only batch normalization combines weight normalization with minibatch mean subtraction, preserving scale control while centering activations and backpropagated gradients. It uses less computation and introduces gentler noise than full batch normalization, with improved test accuracy reported.
- Mean-only batch normalization subtracts each neuron’s minibatch pre-activation mean without dividing by the minibatch standard deviation.A running average of the mean is used at test time.
- Weight normalization makes neuron activation scales approximately independent of v, while mean-only batch normalization additionally centers activations.
- Mean-only batch normalization centers gradients propagated during backpropagation through minibatch mean subtraction.
- Mean-only batch normalization has lower computational overhead and gentler, lighter-tailed training noise than full batch normalization.The lighter-tailed noise arises because minibatch means are approximately normally distributed, unlike variance estimates with highly kurtotic noise.
- The authors report that mean-only batch normalization leads to improved accuracy at test time.
5 Experiments
Across supervised classification, generative modelling, and reinforcement learning, the experiments evaluate weight normalization in several neural-network applications. The reported results show faster convergence, lower variance or better optima, and improved performance in selected comparisons.
- The experiments validate weight normalization across supervised image recognition, generative modelling, and deep reinforcement learning.The evaluated applications include CIFAR-10 classification, convolutional variational auto-encoders, DRAW, and DQN.
- Supervised Classification: CIFAR-10: 7.31% test error was achieved by mean-only batch normalization combined with weight normalization on CIFAR-10 without data augmentation.Weight normalization, the normal parameterization, and mean-only batch normalization alone had similar test accuracy at approximately 8.5% error, while batch normalization achieved 8.05% error.
- Generative Modelling: Convolutional VAE: Weight normalization produced a lower-variance test-set lower bound and converged to a better optimum for convolutional variational auto-encoders on MNIST and CIFAR-10.The comparison used training curves with standard error bars based on different random parameter initializations, with similar results across hyper-parameter settings.
- Generative Modelling: DRAW: Weight normalization significantly sped convergence for DRAW on MNIST, even with initialization and learning-rate settings tuned for the normal parameterization.The reported 100-epoch training interval was insufficient for convergence, but weight normalization made progress much more quickly.
- Reinforcement Learning: DQN: For DQN, weight normalization progressed more quickly and reached a better final result on Space Invaders, while improving performance on average across four games.The Atari experiment used evaluation scores over training epochs and final scores from four games.
6 Conclusion
Weight normalization is a simple reparameterization that accelerates stochastic-gradient-descent convergence across several deep-learning applications. Its low overhead and minibatch independence support broad adoption in developing new architectures.
- Weight normalization accelerates stochastic gradient descent convergence through a simple reparameterization of neural-network weight vectors.
- The method showed a consistent advantage across four models spanning supervised image recognition, generative modelling, and deep reinforcement learning.
- Weight normalization is easy to apply, has low computational overhead, and avoids dependencies between examples in a minibatch.
A Neural network architecure for CIFAR-10 experiments
The supplied passage identifies Table 1 as describing the neural-network architecture used for CIFAR-10 experiments.
- Table 1 describes the neural-network architecture used for CIFAR-10 experiments.
- No architectural components or configuration values are specified in the supplied table text.