Source-linked AI summary
Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities
Tommi Vatanen, Tapani Raiko, Harri Valpola, Yann LeCun
TL;DR
The paper asks whether transforming hidden nonlinearities can make back-propagation learn more like second-order optimization while remaining computationally simple. It introduces a third scale-normalizing transformation, analyzes Fisher-information changes, and finds that the transformations speed learning, although the third can worsen solutions or cause autoencoder divergence.
Problem
Basic gradient descent is easier to use than second-order methods but can require more iterations, while second-order methods are computationally heavy for large models.
Method
The paper transforms hidden nonlinearities to have zero mean, zero slope, and unit variance, then studies their effects through Fisher-information analysis and update-direction comparisons.
Results
The transformations make simple gradient learning behave closer to second-order optimization and improve learning speed; the third transformation can also produce worse local optima.
Takeaways & Limitations
Transforming the model can reproduce some benefits associated with second-order updates while retaining a simple gradient-learning procedure.
Takeaways & Limitations
Adding the third transformation caused autoencoder experiments to diverge because many decoding-layer neurons became inactive and induced very large γ values.
Abstract
from arXiv · showhide
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.
1 Introduction
The paper studies back-propagation in deep networks and builds on transformations that speed MLP learning. It extends this direction by examining three hidden-unit transformations and their relationship to second-order optimization.
- The work studies back-propagation learning in networks with up to five hidden layers, continuing earlier results.
- Known approaches speed MLP learning by centering inputs, rescaling nonlinearities, or centering gradient factors with shortcut connections.
- The paper adds unit-variance normalization to earlier zero-mean and zero-slope transformations and analyzes their connection to second-order updates.
- Second-order methods reduce iterations but are costly for high-dimensional models, motivating simpler approximations based on gradient descent.
2 Proposed Transformations
The method transforms hidden nonlinearities and uses shortcut connections so nonlinear components do not compete with expected outputs or linear input dependencies. A third transformation additionally normalizes hidden-unit signal and slope scales.
- Model convention: The model assumes inputs include an added constant-one component to avoid separate bias vectors in the formulas.
- Auxiliary variables: Each hidden nonlinearity is augmented with auxiliary scalars α_i, β_i, and γ_i that are updated before gradient evaluation.These variables support learning of the main parameter matrices A, B, and C.
- Motivation for the transformations: The architecture uses separate shortcut connections so nonlinear mappings do not compete with bias terms or expected linear input dependencies.The first two transformations enforce zero mean and zero slope in hidden activations, respectively.
- Third transformation: The third transformation normalizes the geometric mean of hidden-unit signals and slopes because both cannot simultaneously be normalized to unity.This aims to make diagonal Fisher-information elements more similar in scale.
- Compensation: The first two transformations can be compensated exactly by updating the shortcut mapping C.
- Transformation targets: The three transformations target zero mean, zero slope, and unit variance on average for hidden nonlinearities.
3 Theoretical Comparison to a Second-Order Method
The analysis explains how the transformations make basic gradient descent resemble a natural-gradient update. Centering reduces off-diagonal Fisher terms, while scale normalization makes diagonal terms more comparable.
- The first two transformations move nondiagonal Fisher-information elements toward zero, while the third makes diagonal elements more similar in scale.Together, these changes make basic gradient descent behave closer to the natural gradient.
- The Fisher matrix is formed from parameter derivatives involving the weight matrices A, B, and C and expectations over Gaussian output noise.
- The analysis uses the Fisher information because the Hessian depends on output data and contains more terms.The Hessian is closely related but is not the simpler matrix analyzed here.
- When hidden units are uncorrelated, zero-mean activations and slopes make representative nondiagonal Fisher elements exactly zero; otherwise the reduction is approximate.The analysis also identifies zero-mean inputs as beneficial for related Fisher elements.
- The third transformation targets Fisher diagonal terms containing f(·)^2 and f′(·)^2 by keeping their scales similar.
4 Empirical Comparison to a Second-Order Method
The transformed networks produce Hessians with more even eigenvalue and diagonal-element distributions, and their gradients point closer to regularized second-order update directions than regular backpropagation. However, omitting the scale transformation γ can yield closer second-order directions, while its overall learning benefit remains unclear.
- Experimental setup: The experiment approximates Hessians by finite differences and compares regular, two-transformation, and three-transformation networks after 4000 training epochs.The Hessian inversion uses regularization because finite-difference Hessians can contain very small or negative eigenvalues.
- Eigenvalues and update directions: Transformed networks have more even Hessian eigenvalue distributions and fewer negative eigenvalues than the regular MLP, while γ is not necessarily useful for this alignment.The comparison used a 2600×2600 Hessian and measured angles to the second-order direction as regularization varied.
- Hessian diagonals: The transformed networks make Hessian diagonal elements less spread and closer to unimodal, with the smallest log-scale variance reported for LTMLP.The unimodality comparison is qualified by the authors as debatable.
- Hessian diagonals: c = 1.43, indicating that transformations make the regularized second-order rule correct gradient elements more evenly than regular backpropagation.The authors infer that the gradient is therefore closer to the second-order update direction when all transformations are used.
- Learning comparison: There is no clear evidence that adding γ benefits learning beyond α and β, although transforming nonlinearities benefits learning relative to standard backpropagation.This conclusion concerns the comparison of the transformation variants in the experiment.
5 Experiments: MNIST Classification
On MNIST, the proposed transformations substantially improve classification results over regular backpropagation under minimal regularization. Adding γ does not improve final error on this dataset, although it can accelerate convergence, especially for the deeper network.
- Results: 112 errors is the best reported result, achieved by the smaller architecture without γ, while the three-layer architecture with γ reaches 114 errors.The test set contains 10,000 samples, and transformed networks improve significantly over same-architecture regular backpropagation.
- Results: The proposed transformations achieve results comparable to dropout networks using only minimal regularization.Training used Gaussian input noise with σ = 0.3 and no weight decay, pretraining, weight-sharing, or data augmentation.
- Transformation comparison: Adding γ does not seem to affect final results on this dataset, but learning appears to converge faster with γ, especially for the three-layer architecture.The reported comparison covers two- and three-hidden-layer MLP architectures.
6 Experiments: MNIST Autoencoder
Adding the third transformation γ to the MNIST autoencoder caused decoder neurons to become inactive and training to diverge, while fixing γ=1 avoided the instability without accelerating learning over α and β.
- Adding γ caused the MNIST autoencoder to diverge as many decoder neurons became inactive and their γ values grew very large.The affected neurons were concentrated in the fourth and fifth hidden layers, and divergence persisted despite constraining γ_i ≤ 100.
- Fixing γ=1 avoided the instability, but did not speed learning compared with the earlier model using only α and β.
7 Discussion and Conclusions
The transformations significantly improve back-propagation learning in deep MLPs and make its updates resemble second-order optimization, although the usefulness of γ remains insufficiently established.
- Linear transformations in nonlinearities significantly improve back-propagation learning in deep MLP networks.
- Experiments confirmed that transformed networks behave like a second-order update method relative to regular MLP training.
- The practical usefulness of γ remains uncertain because more experiments, especially on slowly converging problems, are needed.
- Transforming the model rather than the gradient could extend the approach to MCMC, variational Bayes, and genetic algorithms.These methods may not compute the basic gradient at all.
Appendix
The appendix describes the experimental training protocols, hyperparameter choices, and learning-rate schedules used to compare transformed and regular networks, including the second-order-method experiment.
- Experiments compared LTMLP, a no-γ variant, and regular networks using full-batch training, weight decay, Gaussian input noise, and separately validated hyperparameters.
- LTMLP obtained the best training and test errors in the second-order comparison, although the difference from training without γ was small.
- MNIST experiments used 28 × 28 digit images, 60,000 training samples, 10,000 test samples, and minibatches of 1,000.
- The appendix notes that the shortcut-learning-rate heuristic is weakly supported by Figure 2 and should be re-examined.
- Table 1 reports the neural-network hyperparameters used in the experiments.
- Transformations permitted larger learning rates: ε0 was 0.05 for regular training, 0.3 for LTMLP with γ, and 0.7 without γ.