Source-linked AI summary
ADADELTA: An Adaptive Learning Rate Method
Matthew D. Zeiler
TL;DR
Learning-rate selection in gradient descent is difficult because inappropriate values can cause divergence or slow learning. ADADELTA introduces a first-order, per-dimension adaptive rate using windowed gradient statistics and minimal overhead, showing promising results across MNIST and large-scale speech recognition. Its later-training performance can trail momentum, and its derivation assumes diagonal curvature.
Problem
Learning-rate selection is difficult because rates above the suitable range can cause divergence, whereas lower rates slow optimization.
Method
ADADELTA computes per-dimension dynamic learning rates from first-order information, using windowed accumulation to avoid continual denominator growth.
Results
ADADELTA achieved 2.00% MNIST test error after six epochs, compared with 2.10% for Schaul et al., while continuing to reduce error after ADAGRAD slowed.
Takeaways & Limitations
The method showed promising results across MNIST and a large-scale speech-recognition dataset while requiring trivial overhead and little hyperparameter tuning.
Takeaways & Limitations
Without explicit learning-rate annealing, ADADELTA can underperform properly tuned momentum later in training; its derivation also assumes diagonal curvature.
Abstract
from arXiv · showhide
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
1. INTRODUCTION
Gradient descent updates parameters along negative-gradient directions, but its learning rate must be manually chosen: excessive values can cause divergence, while small values slow learning. ADADELTA addresses this by computing dynamic per-dimension rates from first-order information with minimal extra computation.
- Gradient descent updates parameters by following the negative gradient to optimize an objective function.
- SGD applies these updates using gradients estimated from individual samples or mini-batches, with η controlling step size.
- Learning-rate selection is difficult because rates that are too high can cause divergence, while rates that are too low produce slow learning.
- ADADELTA computes a dynamic learning rate separately for each parameter dimension using only first-order information.
- The proposed approach targets no manual learning-rate setting, hyperparameter insensitivity, low overhead, robustness, and local or distributed applicability.
2. RELATED WORK
Prior methods improve gradient descent through momentum, accumulated gradient statistics, annealing, or curvature approximations, but face sensitivity, computational cost, or continual learning-rate decay. ADADELTA motivates windowed accumulation to retain adaptive per-dimension behavior without indefinite denominator growth.
- Newton’s method uses second-order derivatives and can determine optimal quadratic-problem steps, but Hessian computation is prohibitive for large models.
- Per-dimension rates can compensate for differing parameter sensitivities, including gradient-scale differences across neural-network layers.
- Momentum accelerates dimensions with consistently directed gradients and slows dimensions whose gradient signs change, using decayed past updates.
- ADAGRAD uses a global learning rate with per-dimension rates based on the ℓ2 norm of all previous gradients.
- ADAGRAD can be sensitive to initial gradients and eventually reduce learning rates toward zero because squared gradients accumulate continually.
3. ADADELTA METHOD
ADADELTA replaces ADAGRAD’s continually shrinking learning rates with exponentially decaying, windowed estimates, while combining per-dimension scaling with momentum-like updates and a Hessian-inspired approximation.
- 3.1. Idea 1: Accumulate Over Window: ADAGRAD’s continually accumulated squared gradients shrink learning rates toward zero, motivating a fixed-size window of recent gradients.The windowed denominator remains a local estimate and cannot accumulate without bound.
- 3.1. Idea 1: Accumulate Over Window: The windowed accumulation is implemented as an exponentially decaying average of squared gradients rather than by storing w previous gradients.The decay constant ρ plays a role similar to momentum’s decay parameter.
- 3.2. Idea 2: Correct Units with Hessian Approximation: ADADELTA uses RMS measures of gradients and previous updates to construct a per-dimension update with a small stabilizing constant ϵ.The constant initializes the method when previous updates are zero and helps maintain progress when updates become small.
- 3.2. Idea 2: Correct Units with Hessian Approximation: The method assumes diagonal curvature and approximates the diagonal Hessian using RMS measures of gradients and updates, producing a positive approximation.This preserves movement in the negative-gradient direction.
- 3.2. Idea 2: Correct Units with Hessian Approximation: Because the previous-update RMS lags the gradient denominator, sudden large gradients can reduce the current effective learning rate before the numerator responds.This lag is described as making the system robust to large sudden gradients.
- 3.2. Idea 2: Correct Units with Hessian Approximation: ADADELTA combines SGD’s negative-gradient direction, momentum’s accumulated updates, and ADAGRAD’s per-dimension squared-gradient scaling using only first-order information.The method requires one gradient computation per iteration while leveraging information from past updates.
4. EXPERIMENTS
ADADELTA was evaluated against SGD, Momentum, and ADAGRAD on MNIST and large-scale speech recognition, including varied architectures, hyperparameters, and distributed settings. It converged quickly, remained comparatively insensitive to hyperparameter choice, adapted effective learning rates across layers, and performed well under noisy distributed training.
- MNIST classification: 2.00% MNIST test error versus 2.10% for Schaul et al.’s method after 6 epochs, although training was far from convergence.The comparison used tanh nonlinearities, 500 and 300 hidden units, mini-batches of 100 images, and ϵ = 1e−6, ρ = 0.95.
- MNIST classification: ADADELTA matched ADAGRAD’s fast initial convergence while continuing to reduce test error, approaching Momentum’s best performance over 50 epochs.SGD performed worst, while Momentum ultimately reached a better final solution after many epochs.
- Hyperparameter sensitivity: ADADELTA’s performance changed little across its two hyperparameters, unlike the highly variable results obtained by varying learning rates for SGD, Momentum, and ADAGRAD.The reported comparisons used MNIST test error after 6 epochs with rectified linear units.
- Effective learning rates: ADADELTA used larger effective step sizes in lower network layers, balancing their smaller gradients during tanh-network training.Figure 2 tracked 10 randomly selected dimensions from each of three weight matrices every 60 batches for 25 epochs.
- Effective learning rates: Near the end of training, effective step sizes approached 1 while parameter updates tended toward zero, acting like an implicit annealing schedule.The behavior occurred when gradients and updates were small and the ϵ terms dominated the accumulated quantities.
- Limitations: Momentum eventually outperformed ADADELTA because ADADELTA lacked an explicit annealing schedule and could accumulate oscillations near a minimum.The paper suggests adding an annealing schedule in future work.
- Speech data: On the speech dataset, ADADELTA initially converged faster than ADAGRAD with 100 replicas and quickly matched the other methods with 200 noisy replicas.The 100-replica experiment used logistic nonlinearities, while the 200-replica experiment used rectified linear nonlinearities and the same hyperparameter settings.
5. CONCLUSION
The report introduces ADADELTA as a first-order learning-rate method with minimal overhead, per-dimension adaptation, and promising results across MNIST and large-scale speech recognition.
- ADADELTA uses only first-order information while providing a per-dimension learning rate with trivial computational overhead over SGD.
- Across varied input data, model sizes, nonlinearities, and distributed replicas, ADADELTA required no hyperparameter tuning.