Source-linked AI summary
Gradient-based Hyperparameter Optimization through Reversible Learning
Dougal Maclaurin, David Duvenaud, Ryan P. Adams
TL;DR
Hyperparameter selection is difficult because validation performance depends on an inner training procedure, making naive reverse-mode differentiation impractical in memory. The paper exactly reverses stochastic gradient descent with momentum to compute hyperparameter gradients, enabling optimization of thousands of hyperparameters and detailed training choices. The approach nevertheless faces finite-precision and long-horizon gradient difficulties, and the authors do not optimize initial parameters.
Problem
Hyperparameter selection is difficult because computing validation loss requires an inner loop of elementary optimization, making naive reverse-mode differentiation infeasible from a memory perspective.
Method
The paper exactly reverses stochastic gradient descent with momentum, using compact storage of discarded information to compute gradients through the training dynamics.
Results
The method enables optimization of validation loss with respect to thousands of hyperparameters, including detailed schedules, initialization distributions, regularization schemes, and architectures.
Takeaways & Limitations
Hypergradients allow automatic tuning of most details of neural-network training and provide insight into learned training procedures and architectures.
Takeaways & Limitations
Finite numerical precision makes reversal unstable because repeated inversion of momentum decay accumulates errors, while long training horizons compound gradient difficulties.
Abstract
from arXiv · showhide
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.
1. Introduction
Hyperparameter selection is crucial but difficult because conventional gradient-free methods generally scale poorly beyond 10 to 20 hyperparameters. The paper addresses this by backpropagating through training, enabling large-scale hyperparameter optimization and richer model and training choices.
- Hyperparameters control model complexity and learning procedures, making their selection both crucial and difficult.
- Gradient-free model-based optimization typically cannot effectively optimize more than 10 to 20 hyperparameters.
- The paper computes hyperparameter gradients by propagating validation-loss gradients backward through the elementary training iterations.
- The method supports richer hyperparameterizations over model classes, regularization, and training methods, with optimized procedures compared against standard advice.
- Exactly reversing stochastic gradient descent with momentum computes gradients with respect to continuous training parameters while storing only information needed for reversal.
- These gradients enable optimization of thousands of hyperparameters, including fine-grained schedules, initialization distributions, regularization schemes, and preprocessing.
2. Hypergradients
The paper computes hyperparameter gradients by reversing stochastic gradient descent with momentum, avoiding storage of the full training trajectory. Exact reversibility requires addressing information lost through momentum decay, which the method stores parsimoniously.
- The hypergradient problem: Naive reverse-mode differentiation of validation loss is memory-intensive because it must retain the entire many-step training trajectory.Checkpointing reduces storage but remains impractical for large neural networks trained for thousands of minibatches.
- Reversible learning: SGD with momentum is exactly reversible under exact arithmetic, allowing gradients of trained-weight objectives with respect to initial weights and training hyperparameters.The reverse procedure reconstructs prior parameters and velocities while propagating derivatives through the training dynamics.
- Hyperparameter gradients: The reversible procedure targets gradients with respect to broad training choices, including learning-rate and momentum schedules, initialization conditions, and other hyperparameters affecting training gradients.The paper motivates these choices as important but difficult to tune because hyperparameter gradients are generally unavailable.
- Reversible learning: Reverse SGD has O(T) time complexity, matching forward SGD, because its Hessian-vector products are computed with reverse-mode differentiation.The method differentiates the dot product of the gradient with a vector to obtain the required products exactly.
- Finite precision: Finite precision breaks naive reversal because repeated division by momentum decay amplifies discarded bits, while γ < 1 is needed for convergent dynamics.The paper frames momentum decay as dissipative and therefore non-reversible without retaining the information lost during division.
- Finite precision: The proposed information buffer stores discarded remainders while reusing the velocity representation, enabling exact reversal with fine-grained momentum decay ratios.For γ = 0.98, storage is 0.029 bits on average, reducing memory requirements by a factor of one thousand versus storing a 32-bit value each iteration.
3. Experiments
Hypergradients enable experiments that richly parameterize learning schedules, initialization, regularization, data, and architecture. These proof-of-concept studies optimize thousands of hyperparameters and reveal structured training and sharing patterns.
- Scope: Hypergradients make it possible to optimize thousands of hyperparameters rather than the fewer than 20 typical of conventional approaches.The experiments include fine-grained schedules, initialization distributions, per-input regularization, and per-pixel data preprocessing.
- Learning-rate schedules: An 800-dimensional learning-rate schedule separately optimized rates across 100 SGD iterations, four layers, and weights versus biases.The schedule was optimized on training error because learning schedules can implicitly regularize networks.
- Learning-rate schedules: The optimized schedule takes large steps first in the topmost layer, then in the first layer, before reducing step sizes for all layers during the final 10 iterations.The caption reports less structure for bias and momentum schedules.
- Initialization: Initialization-scale optimization used eight hyperparameters, separately covering weights and biases in each layer, and compared the learned values with an activation-based heuristic.The first-layer weight scale matched the cited heuristic.
- Regularization: Per-parameter L2 regularization was applied to logistic regression, producing 7,840 interpretable hyperparameters indexed by input-pixel and output-label pairs.The resulting scheme is described as a generalization of automatic relevance determination.
- Data and architectures: The method also differentiates through training-data transformations and supports learned datasets, while softened architectural constraints make continuous architecture parameterizations feasible.The paper describes optimizing blank-image training examples and learning penalties that share lower-layer weights across alphabets and top-layer weights across rotated alphabets.
4. Limitations
The paper identifies compounded gradient difficulties, overfitting risks, and limits for discrete hyperparameters. It partially stabilizes meta-learning with small learning rates and early stopping, while continuous parameterizations can address some discrete choices.
- Hypergradients inherit and compound the difficulties that can make elementary learning difficult across hundreds of SGD iterations and nested neural-network derivatives.
- Large learning rates can induce chaotic dynamics, making gradients uninformative about the training objective’s medium-term shape.
- The experiments partially address unstable meta-learning by initializing learning rates small and stopping when the meta-gradient magnitude grows.
- Overfitting: Validation-objective overfitting limits how many hyperparameters can be fruitfully optimized, with validation-set size providing a rough guide under i.i.d. examples.
- Discrete parameters: Gradients are not necessarily useful for discrete hyperparameters, although continuous parameterizations can represent some apparently discrete choices.
5. Related work
Related work applies gradient-based hyperparameter tuning to specialized settings, but differs in scalability, tightness, convergence requirements, or available dynamics. This paper emphasizes exact gradients without requiring convergence and memory-efficient reversal through training.
- Domke derived reverse-mode derivatives through momentum gradient descent and L-BFGS, but caching every parameter vector was impractical for large models and long training.
- Earlier work obtained closed-form regularization gradients only after exact convergence, whereas this procedure handles any hyperparameter whether or not learning has converged.
- SVM work differentiated a bound to tune hundreds of hyperparameters, but the bound was not tight because optimizing the SVM objective requires discrete training-point selection.
- Bayesian methods: Bayesian models commonly expose gradients for continuous hyperparameters when marginal likelihoods are closed-form, with variational inference extending this to some Bayesian neural networks.
- Bayesian methods: Hamiltonian Monte Carlo hyperparameter tuning chained gradients through a small number of leapfrog steps, whereas reversible memory tapes could extend this approach to many more iterations.
6. Extensions and future work
The paper suggests extending hypergradient methods to parallel Bayesian optimization, reversible recurrent computations, and other momentum-based optimizers. These directions target broader scalability and memory savings beyond the demonstrated SGD setting.
- Bayesian optimization with gradients: Hypergradients could augment parallel Gaussian-process model-based hyperparameter optimization, where parallel evaluations may offset sequential evaluation costs.
- Reversible elementary computation: Reversible computation may reduce memory for recurrent architectures whose evaluations contain long chains of mostly small parameter updates.
- Exactly reversing other learning methods: The memory-saving reversal trick could presumably apply to other momentum-based SGD variants, including RMSprop and Adam.
7. Conclusion
The paper derives efficient exact hypergradients through stochastic gradient descent with momentum by exploiting approximate reversibility. These gradients enable validation-loss optimization over thousands of hyperparameters and detailed training, regularization, and architecture choices.
- The procedure computes gradients through stochastic gradient descent with momentum while drastically reducing memory for exact back-propagation across hundreds of training iterations.
- Hypergradients make validation-loss optimization over thousands of hyperparameters feasible, including detailed training schedules, regularization schedules, and neural network architectures.
Appendix: Forward vs. reverse-mode differentiation
Forward- and reverse-mode differentiation evaluate the same chain-rule product in opposite orders. Reverse mode is advantageous for scalar objectives with high-dimensional intermediates, but it ordinarily requires storing forward-pass values.
- Forward-mode differentiation multiplies gradients in the same order that functions are evaluated.
- Reverse-mode differentiation multiplies Jacobians in reverse function order, starting from the final result.
- For scalar outputs and D-dimensional intermediate vectors, reverse mode accumulates a D-dimensional vector, whereas forward mode accumulates a D × D matrix.
- When vector-to-vector functions are elementwise operations or reshaped matrix multiplications, sparse Jacobians make multiplication efficient.
- Reverse mode ordinarily requires intermediate values from the forward pass to remain in memory, motivating memory reductions for full learning procedures.