PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to learn by gradient descent by gradient descent

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Nando de Freitas

arXiv:1606.04474v1cs.NEcs.LG

TL;DR

Optimization methods are often hand-designed to exploit structure in particular problem classes, while vanilla gradient descent ignores second-order information. This paper learns update rules with recurrent neural networks; its learned optimizers compare favorably with hand-engineered methods and transfer to related problems.

  • Problem

    Vanilla gradient descent uses gradients but ignores second-order information, while optimizer design is specialized to problem classes and may perform poorly outside their scope.

    Source: §1 · pp. 1–2

  • Method

    The paper learns optimizer update rules with recurrent neural networks, training them across optimization trajectories and using an approximation that avoids computing second derivatives of the optimizee.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Learned neural optimizers compare favorably with deep-learning methods; on CIFAR image labeling, LSTM optimizers outperform hand-engineered optimizers when transferring to datasets from the same distribution.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The learned optimizers can transfer beyond their training problems, including neural-art tasks with four times as many parameters and MNIST networks with different architectures, though not ReLU activations.

    Source: §3.2 · p. 7, §5 · p. 9

    The authors identify memory usage as a scaling concern; NTM-BFGS outperformed LSTM optimizers on quadratic functions but showed no benefit on other stochastic optimization tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.

1 Introduction

Hand-designed optimizers exploit structure in particular problem classes, but may perform poorly outside their intended scope. This paper proposes learning update rules from example problem instances and frames transfer between optimization problems as generalization.

1 Introduction

  • Specializing optimizers to a subclass of problems is presented as the route to improved performance, since no algorithm beats random search in expectation across combinatorial optimization problems.The paper’s motivation is that optimization methods can exploit structure in their target problems at the expense of performance outside that scope.Source: §1 · p. 2
  • The paper replaces hand-designed update rules with a learned optimizer g, parameterized by φ, to exploit structure in selected classes of functions.The introduction describes modeling g as a recurrent neural network with state that updates dynamically as a function of its iterates.Source: §1 · p. 2

1.1 Transfer learning and generalization

  • In this framework, generalization means transferring knowledge between problem instances, making transfer learning a form of generalization.The examples used to learn the optimizer are themselves optimization problems.Source: §1.1 · p. 2

1.2 Related work and a brief history

  • Earlier recurrent meta-learning work trained higher-level networks as gradient descent procedures, but did not directly address transfer to novel problem instances.The paper distinguishes that focus on online adaptivity from its interest in transferring a learned training procedure.Source: §1.2 · p. 3

2 Learning to learn with recurrent neural networks

The paper trains an optimizer across functions, using an RNN to produce updates and a trajectory-based objective to train its parameters. It develops coordinatewise LSTM updates and extensions that share information across coordinates.

2 Learning to learn with recurrent neural networks

  • The optimizer is trained by minimizing expected loss over functions, with an RNN producing updates from its state and parameterized by φ.The training objective can weight losses across an optimization trajectory; using positive weights at intermediate steps enables training on partial trajectories, and experiments set every weight to 1.Source: §2 · p. 3
  • Gradients for optimizer parameters are estimated by backpropagating through a sampled function's computational graph while stopping gradients through optimizee gradients.This assumes optimizee gradients do not depend on optimizer parameters, avoiding second derivatives of the function.Source: §2 · p. 3

2.1 Coordinatewise LSTM optimizer

  • A coordinatewise two-layer LSTM maps each parameter's gradient and its prior hidden state to that parameter's update, sharing parameters across coordinates.Separate activations allow coordinate-specific behavior, while applying the same rule independently makes the optimizer invariant to parameter order.Source: §2.1 · p. 4, Figure 3 · p. 4
  • The LSTM's recurrence can use gradient history to learn dynamic update rules, including behavior similar to momentum.Because input and output magnitudes vary across objective classes, the paper rescales them with shared constants; an appendix preprocessing method is described as more robust and slightly better-performing.Source: §2.1 · p. 4

2.2 Information sharing between coordinates

  • Global averaging cells communicate averaged activations across coordinates, allowing LSTM+GAC to implement L2 gradient clipping if each LSTM can compute squared gradients.Source: §2.2 · p. 5
  • NTM-BFGS adds shared external memory to LSTM+GAC, designed to allow simulation of low-memory Newton approximations such as (L-)BFGS.Its memory permits only low-rank updates, and its controller, including read/write heads, operates coordinatewise.Source: §2.2 · p. 5

3 Experiments

The experiments test learned LSTM optimizers on quadratic functions, neural-network training, and neural art, including whether they transfer to altered problem settings. Across these settings, the reported results include gains over hand-crafted optimizers and both successful and unsuccessful generalization.

3 Experiments

  • Experiments train two-layer LSTM optimizers with 20 hidden units per layer, use early stopping, and compare against standard optimizers with tuned learning rates.The reported standard-optimizer result uses the learning rate giving the best final error for each problem.Source: §3 · p. 5

3.1 Quadratic functions

  • On synthetic 10-dimensional quadratics, learned optimizers substantially outperform generic counterparts, while LSTM+GAC and NTM-BFGS outperform the coordinatewise LSTM.Functions are sampled from the same distribution for training and testing; the global-information variants incorporate global information at each step.Source: §3.1 · p. 6

3.2 Training a small neural network on MNIST

  • On the base MNIST MLP, the LSTM outperforms ADAM by a large margin; it also beats hand-crafted baselines at 200 steps and transfers to wider or deeper networks, but not ReLU activations.The optimizer was trained for 100 steps; baseline learning rates were retuned for the 200-step comparison and for each architecture comparison.Source: §3.2 · p. 7

3.3 Training a convolutional network on CIFAR-10

  • On CIFAR-10, the LSTM optimizer learns much more quickly than standard optimizers and converges to the same value.The optimizer uses separate LSTMs for fully connected and convolutional parameters, with weights shared within each parameter type.Source: §3.3 · p. 7, Figure 7 · p. 8

3.4 Neural Art

  • For neural art, the LSTM outperforms standard optimizers at its training style and resolution and performs very well with a new style and higher test-time resolution.Training used one fixed style and 64x64 content images; testing included a different style at 128x128.Source: Figure 6 · p. 6, §3.4 · p. 8

4 Visualizations

The visualizations inspect the LSTM optimizer’s decisions along optimization trajectories and as a function of current gradients. They show larger, noisier updates than standard methods and step-like gradient responses that can run opposite to ADAM’s.

Visualizations

  • The authors note that optimizer decisions are difficult to visualize because proposed updates depend on the full optimization trajectory.Their visualizations examine selected coordinates and steps.Source: §4 · pp. 8–9

Histories of updates

  • The LSTM optimizer makes larger updates than SGD and ADAM, uses momentum, and produces noisier updates than ADAM.The authors suggest ADAM’s updates may reflect longer-timescale momentum.Source: §4 · pp. 8–9, Figure 9 · p. 9

Neural Art

  • Figure 8 plots Neural Art optimization curves on a log-scale y-axis for training style and resolution on the left and test style at double training resolution on the right.The content images are from a test set not used to train the LSTM optimizer.Source: Figure 8 · p. 8

Proposed update as a function of current gradient

  • For selected steps, the LSTM optimizer’s update as a function of the current gradient is often step-like, sometimes in the opposite direction from ADAM.In those cases, larger gradients correspond to larger updates.Source: §4 · p. 9

5 Conclusion

The paper presents learned optimizers that compare favorably with state-of-the-art deep-learning methods and transfer across several task variations. It also reports that NTM-BFGS has not improved other stochastic optimization tasks and identifies memory scaling as future work.

Conclusion

  • Learned neural optimizers compare favorably with state-of-the-art deep-learning methods, with transfer across Neural Art parameter counts, styles and content images, and MNIST architectures.The Neural Art optimizer trained on 12,288 parameters generalized to 49,152 parameters while style and content images also changed.Source: §5 · p. 9
  • On CIFAR image labeling, LSTM optimizers outperform hand-engineered optimizers when transferring to datasets drawn from the same data distribution.The conclusion scopes this result to same-distribution transfer.Source: §5 · p. 9
  • NTM-BFGS outperformed LSTM optimizers for quadratic functions but showed no benefit on the other stochastic optimization tasks; improving memory scaling remains future work.The authors identify further investigation of NTM-BFGS and memory usage as future directions.Source: §5 · p. 9

A Gradient preprocessing

Gradient inputs can vary greatly in magnitude across optimizee coordinates, making optimizer training difficult. The appendix preprocesses gradients to avoid divergence near zero and reports slightly better results on some problems than simple rescaling.

Gradient preprocessing

  • Different optimizee coordinates can have gradients with very different magnitudes, which can make optimizer training difficult because neural networks may disregard small input variations.The passage gives weights in different neural-network layers as an example.Source: A Gradient preprocessing · p. 12
  • The proposed preprocessing supplies the optimizer with a transformed gradient representation instead of raw gradients, avoiding the divergence of log(|∇|) as ∇ approaches zero.The passage introduces (log(|∇|), sgn(∇)) as a possible input, then motivates a different formula to handle the near-zero issue.Source: A Gradient preprocessing · p. 12
  • The preprocessing parameter p controls how small gradients are disregarded, and all experiments use p = 10.Source: A Gradient preprocessing · p. 12
  • Rescaling all inputs by a suitable constant also works, while the proposed preprocessing is more robust and gives slightly better results on some problems.Source: A Gradient preprocessing · p. 12

B NTM-BFGS optimizer

NTM-BFGS replaces BFGS’s specific read and write operations with learned operations while retaining their matrix-vector and low-rank-update structure. It uses an LSTM-based controller and memory, with a limited-memory variant for constrained settings.

B NTM-BFGS optimizer

  • BFGS builds an inverse-Hessian approximation from observed gradients, using a read operation to propose updates and a write operation to update the approximation.In BFGS, the read is a matrix-vector multiplication and the write is a low-rank update.Source: B NTM-BFGS optimizer · p. 12
  • NTM-BFGS retains BFGS’s matrix-vector read and low-rank write structure but learns the vectors used in those operations, with the memory corresponding to an NTM memory [Graves et al., 2014].Source: B NTM-BFGS optimizer · p. 12
  • An LSTM+GAC controller produces read and write vectors: read results feed back into the controller, while write-vector outer products accumulate into memory.Figure 11 depicts one head’s read and write operations; Figure 12 depicts their connection to the controller.Source: B NTM-BFGS optimizer · p. 12, Figure 11 · p. 13, Figure 12 · p. 13
  • With one read head, three write heads, arbitrary coordinatewise controller functions, and access to two GACs, NTM-BFGS can simulate inverse-Hessian BFGS.Source: B NTM-BFGS optimizer · p. 12
  • When memory is constrained, NTM-L-BFGS stores a sliding history of write vectors to form the read matrix-vector product efficiently, and is used for MNIST and Neural Art.Source: B NTM-BFGS optimizer · p. 12

C Gradient Visualizations

The section shows proposed updates to the three color channels of a corner pixel from one neural art instance.

C Gradient Visualizations

  • The visualization shows proposed updates for all three color channels of a corner pixel in one neural art instance.The figures are explained in Section 4.Source: C Gradient Visualizations · p. 14

Related papers

Loading 1606.04474v1…