Learning to Optimize Neural Nets

1. Introduction

• Optimization algorithms largely remain static tools whose design is reserved for human experts.The paper frames learning these tools as an untackled opportunity for machine learning.Source: §1 · p. 1
• Li & Malik (2016) pursue task-independent learned optimizers, whereas Andrychowicz et al. (2016) target a particular training task.This paper studies the task-independent paradigm for high-dimensional stochastic optimization.Source: §1 · p. 1
• Learning the update formula φ turns optimizer design into a policy-search problem.Different choices of φ correspond to different optimization algorithms.Source: §1 · pp. 1–2
• The MNIST-trained optimizer outperforms ADAM (Kingma & Ba, 2014), AdaGrad, RMSprop, and Andrychowicz et al. (2016), and generalizes to TFD, CIFAR-10, and CIFAR-100.The learned method extends learning optimization algorithms to high-dimensional stochastic problems.Source: §1 · p. 2

2. Related Work

• Li & Malik (2016) learn task-independent optimization algorithms with reinforcement learning, while Andrychowicz et al. (2016) learn task-dependent algorithms with supervised learning.These were the first proposals for learning general optimization algorithms described here.Source: §2 · p. 2

2.1. Learning What to Learn

• Learning what to learn seeks base-learner parameter values useful across related tasks, enabling faster learning on a new task in that family.The passage connects this line of work with transfer learning and multi-task learning.Source: §2.1 · p. 2

2.2. Learning Which Model to Learn

• Learning which model to learn selects a base-level learner using task-performance correlations, but requires a representation rich enough for varied learners and compact enough for tractable search.Prior proposals include storing learner examples or representing learners as programs; Hochreiter et al. (2001) encode the training procedure in a recurrent network's memory state.Source: §2.2 · p. 2
• Hyperparameter optimization searches a predefined learner space on the same task, so its settings are generally task-specific and must be retuned for new tasks.The passage lists Bayesian optimization, random search, and gradient-based optimization as approaches.Source: §2.2 · p. 2

2.3. Learning How to Learn

• Learning how to learn targets the learning process itself, often reducing to learning an optimization algorithm that generalizes across learners and tasks.Related approaches learn step sizes or damping factors, tune solver operands, or learn rules such as the Hebb-like rule of Bengio et al. (1991), which does not depend on the objective function.Source: §2.3 · pp. 2–3

3.1. Setting

• The framework learns an optimizer across training objectives so it can be evaluated on unseen objectives, rather than learning only for one task.Its meta-loss penalizes undesirable behavior such as slow convergence or excessive oscillations.Source: §3.1 · p. 3

3.2. RL Preliminaries

• Reinforcement learning seeks a policy that minimizes expected cumulative cost over a finite horizon, with states, observations, actions, transitions, and costs defining the problem.The preliminaries describe an undiscounted finite-horizon POMDP with continuous state, observation, and action spaces.Source: §3.2 · p. 3

3.3. Formulation

• Here, the state includes the current iterate and history-dependent features, while the action is the update step; a policy therefore specifies an optimization algorithm.The state transition captures how gradients and objective values change after a step, encoding local objective geometry.Source: §3.3 · p. 4

3.4. Guided Policy Search

• Guided policy search alternates optimization of a time-varying linear policy with training a stationary nonlinear policy to match it.Policy matching is relaxed to equality of mean actions at each time step and solved using Bregman ADMM; the mean-action penalty is a squared Mahalanobis distance.Source: §3.4 · pp. 4–5

3.5. Convolutional GPS

• Convolutional GPS imposes permutation-invariant coordinate groups, decomposing policy-search subproblems so LQG can run more efficiently.The groups can correspond to weight matrices or bias vectors; the paper also describes block-diagonal structures and shared parameters within groups.Source: §3.5 · pp. 5–6

3.6. Features

• State features summarize recent iterates, gradients, and objective values, while observation features are designed for use during both training and testing.The state features include relative objective change, a normalized recent gradient, and a ratio of recent iterate changes; undefined early features are replaced with zeros.Source: §3.6 · p. 6

• The optimizer was meta-trained on projected MNIST using a 48–48–10 two-layer network, then evaluated on TFD, CIFAR-10, and CIFAR-100 against seven hand-engineered algorithms and a supervised learned optimizer.Meta-training used 400 iterations and mini-batches of 64; the supervised baseline used the same training objective and setting.Source: §4 · pp. 6–7
• Predicted Step Descent descends to the optimum fastest across all three datasets; L2LBGDBGD performs well on CIFAR but does not reach the optimum on TFD.Source: §4 · pp. 7–8
• On the enlarged architecture, Predicted Step Descent outperforms other algorithms on every dataset despite not being trained on that architecture.It initially oscillates on TFD and CIFAR-10, then recovers and overtakes the alternatives.Source: Figure 2 · p. 7, §4 · p. 8
• With mini-batches reduced to 10, Predicted Step Descent still outperforms all other algorithms on the original architecture and handles increased stochasticity fairly well.Conjugate gradient and L2LBGDBGD diverge on TFD and progress slowly on CIFAR-10.Source: Figure 3 · p. 7, §4 · p. 8
• On the enlarged architecture with mini-batches of 10, Predicted Step Descent oscillates on TFD and CIFAR-10 but achieves a much better objective value than other algorithms.L2LBGDBGD diverges fairly quickly on both datasets.Source: §4 · p. 8, Figure 4 · p. 8
• Predicted Step Descent behaves reasonably beyond its 400-iteration training horizon when evaluated for 800 iterations.Source: §4 · p. 8, Figure 5 · p. 8

• The proposed method learns optimization algorithms for high-dimensional stochastic problems.Source: §5 · p. 9
• An optimizer learned on MNIST generalizes to neural-net training on TFD, CIFAR-10, and CIFAR-100.Source: §5 · p. 9
• The learned optimizer is robust to changes in gradient stochasticity and neural-net architecture.Source: §5 · p. 9