Source-linked AI summary
Learning to learn by gradient descent by gradient descent
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas
TL;DR
Optimization algorithms remain hand-designed despite the success of learned representations, motivating a way to learn update rules for selected problem classes. The paper parameterizes optimizers with recurrent neural networks trained across example problems, and reports favorable performance and transfer to related tasks, including CIFAR datasets from the same distribution.
Problem
Optimization methods are often hand-designed for specific problem classes, while the paper seeks a procedure that performs well on a selected class through example problem instances.
Method
The paper casts optimizer design as a learning problem and models the update rule with a recurrent neural network parameterized by φ.
Results
Learned neural optimizers compare favorably with state-of-the-art deep-learning optimizers and outperform hand-engineered optimizers when transferring across CIFAR datasets from the same data distribution.
Takeaways & Limitations
The learned optimizer transfers across related tasks, including larger neural-art problems, different styles and content images, and different MNIST architectures.
Takeaways & Limitations
The optimizer-training assumption ∂φ = 0 avoids computing second derivatives of f.
Abstract
from arXiv · showhide
The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.
1 Introduction
Optimization methods are typically hand-designed for particular problem classes, but this paper casts optimizer design as learning and uses recurrent networks to learn update rules from example problems.
- 1 Introduction: Gradient descent can underperform because it uses gradients while ignoring second-order information such as curvature.Classical methods address this by rescaling gradient steps using Hessian, generalized Gauss-Newton, or Fisher information matrices.
- 1 Introduction: Optimization methods are often specialized to exploit structure in particular problem classes, potentially performing poorly outside their intended scope.The paper relates this specialization to the No Free Lunch result that no optimization algorithm universally outperforms random search in expectation for combinatorial problems.
- 1 Introduction: The paper replaces hand-designed update rules with a learned optimizer g parameterized by φ.The learned rule produces optimizee updates from gradients and is modeled using a recurrent neural network with its own state.
- 1.1 Transfer learning and generalization: Algorithm design is recast as a learning problem in which example problem instances specify the class of optimization problems of interest.This contrasts with designing learning algorithms from analytically characterized problem properties.
- 1.1 Transfer learning and generalization: In this framework, generalization means transferring knowledge between different optimization problems by reusing their shared structure.The paper frames transfer learning as generalization from a meta-learning perspective.
2 Learning to learn with recurrent neural networks
The paper formulates optimizer design as learning an update rule whose parameters are trained over optimization trajectories. It implements this rule coordinatewise with recurrent networks, using an LSTM architecture designed to scale to many parameters.
- The optimizer m, parameterized by φ, produces update steps through a recurrent network whose state evolves during optimization.The learned rule updates optimizee parameters from their gradients and maintains recurrent state.
- Training uses a trajectory objective that can weight losses at multiple time steps, rather than relying only on the final iterate.Setting intermediate weights positive provides training signal along partial trajectories; the experiments use w_t = 1 for every t.
- The optimizer gradient is computed by backpropagation through the optimization graph while dropping gradients through optimizee gradients, avoiding second derivatives of f.This uses the assumption ∂∇_t/∂φ = 0.
- 2.1 Coordinatewise LSTM optimizer: A coordinatewise architecture applies the same small update network independently to each objective-function parameter, making it invariant to parameter ordering.This design avoids the large hidden state and parameter count required by a fully connected RNN over tens of thousands of parameters.
- 2.1 Coordinatewise LSTM optimizer: The LSTM optimizer takes each coordinate’s gradient and previous hidden state, then outputs the corresponding parameter update.Recurrence lets the learned rule integrate gradient history, similarly to momentum.
- The experiments compare learned optimizers with SGD, RMSprop, ADAM, and Nesterov’s accelerated gradient on optimization tasks.Figure 4 presents learned optimizers with solid lines and hand-crafted optimizers with dashed lines.
3 Experiments
Across quadratic, MNIST, CIFAR-10, and Neural Art tasks, LSTM-based learned optimizers were evaluated against hand-designed baselines and tested on modified problem instances. They generally outperformed baselines and generalized to similar architectures, datasets, resolutions, and styles, with clear failures when activation dynamics changed substantially.
- Experimental setup: Two-layer LSTMs with 20 hidden units per layer were trained using truncated BPTT, ADAM, and early stopping, then evaluated on freshly sampled test problems.Baseline optimizers included SGD, RMSprop, ADAM, and NAG, with learning rates tuned separately for each problem.
- 3.1 Quadratic functions: The learned optimizer substantially outperformed standard baselines on synthetic 10-dimensional quadratic functions sampled from the same distribution as training problems.Functions were optimized for 100 steps, while learned optimizers were unrolled for 20 steps during training.
- 3.2 Training a small neural network on MNIST: On the base MNIST MLP, the LSTM optimizer significantly outperformed NAG, ADAM, and RMSprop and retained this advantage when run for 200 rather than 100 steps.The base network had one sigmoid-activated hidden layer with 20 units, and minibatches contained 128 examples.
- 3.2 Training a small neural network on MNIST: The optimizer generalized well to MNIST networks with 40 rather than 20 hidden units and two rather than one hidden layers, but not to ReLU activations.The authors attribute the ReLU failure to sufficiently different learning dynamics; similar test-set problems produced better generalization than baselines.
- 3.3 Training a convolutional network on CIFAR-10: On CIFAR-10 and label subsets, the LSTM optimizer learned faster than baselines, with especially significant performance boosts on CIFAR-5 and CIFAR-2.An optimizer trained only on disjoint held-out labels also transferred well to the additional dataset.
- 3.4 Neural Art: For Neural Art, the LSTM optimizer outperformed all standard optimizers at the training resolution and style and continued to perform very well with changed resolution and style.Content images came from a test set not used during optimizer training.
4 Conclusion
The learned optimizer is trained as an optimization problem and specializes to function classes, while showing strong transfer across tasks, architectures, datasets, and image-styling settings.
- The optimizer-design problem is cast as learning, enabling optimizers specialized to particular classes of functions.
- The LSTM optimizer trained on 12,288-parameter neural art tasks generalized to 49,152 parameters, different styles, and different content images simultaneously.
- The learned optimizer transferred successfully to different architectures in the MNIST task.
- On CIFAR, LSTM optimizers outperformed handengineered optimizers when transferred to datasets from the same data distribution.
A Gradient preprocessing
Optimizer inputs can have highly unequal gradient magnitudes, so preprocessing is used to prevent small gradients from being disregarded and to improve robustness.
- Gradient coordinates can differ greatly in magnitude, making optimizer training difficult because neural networks concentrate on larger input values.
- The proposed preprocessing avoids the divergence of log(|∇|) as ∇→0 by transforming the optimizer inputs.
- The preprocessing parameter p controls how small gradients are disregarded, with p = 10 used in all experiments.
- Rescaling inputs by a constant also works, but the proposed preprocessing is more robust and slightly better on some problems.
B Visualizations
The visualizations show that the LSTM optimizer uses larger, noisier, momentum-like updates and often maps gradients to step-like updates that can differ from ADAM.
- The visualization examines LSTM optimizer decisions along full optimization trajectories for neural art.
- Histories of updates: Compared with SGD and ADAM, the trained optimizer makes bigger updates and uses noisier momentum-like behavior with a shorter apparent timescale.
- Proposed update as a function of current gradient: The proposed update is visualized as a function of the current gradient at selected time steps.
- Proposed update as a function of current gradient: For these plots, the x-axis is the current gradient and the y-axis is the update proposed by each optimizer, using the LSTM trajectory for all methods.
- Proposed update as a function of current gradient: The LSTM update function is often step-like, and its step can point opposite to ADAM’s, with larger gradients sometimes producing larger updates.
C Neural Art
The neural-art examples show images generated by the LSTM optimizer from paired content and style images.
- Each neural-art example presents a content image, a style image, and the image generated by the LSTM optimizer.
D Information sharing between coordinates
The optimizer can share information across coordinates by allowing different LSTMs to communicate, thereby modeling coordinate correlations beyond diagonal methods.
- Coordinatewise optimizers resemble learned RMSprop or ADAM, but richer optimizers can account for correlations between coordinates.
Global averaging cells
Global averaging cells let coordinatewise LSTMs communicate by averaging selected activations across coordinates at each step.
- Global averaging cells reserve communication units in each LSTM layer whose outgoing activations are averaged across coordinates.
- LSTM+GAC optimizers can implement L2 gradient clipping when each LSTM can compute the squared gradient.
NTM-BFGS optimizer
NTM-BFGS combines an LSTM+GAC controller with shared external memory and learned read/write operations, preserving key structural features of BFGS while discarding its fixed forms.
- NTM-BFGS uses shared external memory to learn algorithms resembling low-memory approximations to Newton’s method, such as L-BFGS.
- The architecture is motivated by BFGS and differs from a Neural Turing Machine through low-rank memory updates and coordinatewise controller operation.
- BFGS represents optimization through read, update, and write operations on an inverse Hessian approximation M_t.
- In standard BFGS, read(M_t, θ_t) = −M_t∇f(θ_t), while writing applies a particular low-rank update to M_t.
- NTM-BFGS retains matrix-vector reads and low-rank writes but learns the vectors used in both operations.
- The controller’s read heads feed memory-derived results back into the controller, while write heads produce vectors that update memory.
- With one read head and 3 write heads, NTM-BFGS can simulate inverse Hessian BFGS under sufficient coordinatewise computation and 2 GACs.
NTM-L-BFGS optimizer
NTM-L-BFGS constrains memory through a low-rank approximation, storing recent write vectors so the optimizer can efficiently perform the required read multiplication.
- When memory is constrained, NTM-L-BFGS can maintain a low-rank approximation of the full inverse-Hessian memory.
- A sliding history of left and right write vectors enables efficient matrix-vector multiplication for the read operation.
- The controller uses replicated coordinatewise LSTMs, while read and write operations remain global across all coordinates.