PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Natural Gradient for Deep Networks

Razvan Pascanu, Yoshua Bengio

arXiv:1301.3584v7cs.LGmath.NA

TL;DR

The paper evaluates natural gradient for deep models amid several related optimization methods. It connects these methods, extends natural gradient with second-order information, and examines unlabeled data, training-order robustness, and scalable metric inversion. The reported contributions include improved generalization-error protection with more accurate metrics and good performance with small minibatches under specified sampling and step-size conditions.

  • Problem

    Evidence is limited on how natural gradient relates to other deep-model optimizers and on its behavior with unlabeled data, training-order changes, and second-order information.

  • Method

    The paper analyzes connections to Hessian-Free Optimization, Krylov Subspace Descent, and TONGA; adds unlabeled-data and second-order extensions; and uses truncated Newton to invert the full metric.

  • Results

    Natural gradient can guard against large generalization-error drops when unlabeled data improves metric accuracy and can perform well with small minibatches under separate gradient and metric samples.

  • Takeaways & Limitations

    Natural gradient’s metric-based formulation provides connections to established deep-learning optimizers and supports robustness properties beyond stochastic gradient descent.

  • Takeaways & Limitations

    The method relies on local approximations that can be harmed by large steps, and one conjugate-gradient treatment assumes consecutive metrics are identical despite expected metric changes.

Abstract

from arXiv · show

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods for training deep models: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We describe how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent. Finally we extend natural gradient to incorporate second order information alongside the manifold information and provide a benchmark of the new algorithm using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.

1 Introduction

The paper situates natural gradient among optimization methods for deep models and develops connections, extensions, and scalable implementations for evaluating it.

  • Recent deep-learning optimizers include Hessian-Free Optimization, Krylov Subspace Descent, natural gradient descent, and TONGA.
  • Efficient Jacobian- or Hessian-vector products from automatic differentiation support truncated-Newton approaches used by Hessian-Free Optimization and Krylov Subspace Descent.
  • The paper connects natural gradient descent with Hessian-Free Optimization and Krylov Subspace Descent.
  • It studies unlabeled-data use, training-order robustness, second-order information, and benchmarking with truncated Newton inversion of the full metric matrix.

2 Natural gradient descent

Natural gradient descent chooses parameter updates by optimizing loss subject to a local KL-divergence constraint, using the Fisher Information Matrix to measure functional change.

  • The functional manifold consists of probability density functions indexed by model parameters, with the Fisher Information Matrix as its metric.
  • Natural gradient descent corrects the loss gradient using the local curvature of the KL-divergence surface.
  • The KL constraint yields locally constant-speed movement and robustness to local model reparametrizations.
  • The update minimizes a first-order loss approximation while constraining the second-order KL approximation to remain constant.
  • The method uses damping and learning-rate selection because its local approximations can be harmed by large steps.
  • For neural networks, probabilistic interpretations of outputs allow natural gradient to use conditional densities for regression and classification.

3 Properties of natural gradient descent

Natural gradient descent has properties that support online learning, reparametrization robustness, plateau traversal, and constrained model changes.

  • Natural gradient can operate online while estimating its intrinsic metric from held-out, potentially unlabeled data.
  • Natural gradient is robust to local reparametrization because KL divergence measures changes in probability density independently of parameterization.
  • The Fisher metric can help natural gradient traverse plateaus in the model distribution and corresponding error function.
  • The KL constraint limits both insufficient and excessive model changes, which the paper associates with robustness to overfitting.

4 Natural gradient descent and TONGA

The paper distinguishes TONGA’s covariance-based update from natural gradient’s model-distribution metric, while explaining why the methods can appear similar.

  • TONGA models minibatch-gradient uncertainty with a covariance matrix to adjust steps toward downward generalization-error moves.
  • TONGA’s simplified uncentered covariance can be confused with Amari’s metric, although the derivations use different expectations.
  • Amari’s metric directly measures how the model distribution changes, whereas the paper questions whether TONGA’s uncentered covariance has that interpretation.

5 Natural gradient descent and Hessian-Free Optimization

The paper connects natural gradient descent with Hessian-Free Optimization through the extended Gauss–Newton approximation and interprets related damping and structural-damping heuristics geometrically.

  • Natural gradient and Hessian-Free Optimization: Hessian-Free Optimization uses the extended Gauss–Newton approximation rather than the actual Hessian.The approximation is described as better behaved because it is positive semi-definite by construction.
  • Metric equivalence: For sigmoid with cross-entropy and softmax with negative log-likelihood, the extended Gauss–Newton matrix matches the natural gradient metric.The paper extends the known Fisher–Gauss–Newton identity beyond linear activations with squared error.
  • Damping and trust regions: Levenberg–Marquardt damping adds αI to improve invertibility and address negative curvature, with trust-region adaptation decreasing or increasing α according to prediction quality.Small α increases reliance on the second-order approximation, while large α emphasizes the gradient direction.
  • Damping and trust regions: Under the same near-converged conjugate-gradient assumption, natural-gradient reduction ratios behave identically to Hessian-Free reduction ratios.The comparison uses the natural-gradient direction involving F^-1 and the gradient.
  • Structural damping: Structural damping arises by adding a second KL constraint on hidden-state changes alongside the output-distribution constraint.The theoretically derived regularization term has a fixed coefficient of 1.

6 Natural gradient descent and Krylov Subspace Descent

The paper shows that Krylov Subspace Descent implements natural gradient through the extended Gauss–Newton metric and argues that its warm-restart modification resembles natural conjugate gradient.

  • Connection to natural gradient: Krylov Subspace Descent implements natural gradient because it relies on the extended Gauss–Newton approximation of the Hessian.KSD restricts updates to a lower-dimensional Krylov subspace and solves within it using a second-order method such as BFGS.
  • Connection to natural gradient: KSD differs from Hessian-Free Optimization by adding the previous search direction to its Krylov subspace to mimic a warm restart.This modification gives the previous direction a distinct role in the update subspace.
  • Connection to natural conjugate gradient: The paper hypothesizes that this change makes KSD more similar to natural conjugate gradient than to natural gradient alone.Natural conjugate gradient applies nonlinear conjugate gradient updates on top of natural gradients.

7 Using unlabeled data

The paper uses unlabeled inputs to estimate the natural-gradient metric and finds that separating gradient and metric samples improves test error and reduces minibatch overfitting.

  • Metric estimation: Natural-gradient metric estimation does not require target values because targets are sampled from the model distribution p(t|x).For standard neural networks, the expectation over targets can be evaluated analytically.
  • Using unlabeled data: Figure 1 compares same-minibatch estimation, a different training minibatch, and a randomly sampled unlabeled batch across updates on TFD fold 4.The plot shows training error on a log scale and test error as the percentage of misclassified examples.
  • Separating gradient and metric samples: Using the same samples for gradient and metric estimates can make the estimates dependent and lead to overfitting the current minibatch.The paper presents Figure 1 as empirical evidence supporting this hypothesis.
  • Separating gradient and metric samples: Reusing the same samples for the metric and gradient produces worse global training error and worse test error.The comparison concerns training error over the entire training set.
  • Using unlabeled data: Using unlabeled data produces better test error but worse training error, acting like a regularizer.The experiment uses the Toronto Face Dataset, which has a small labeled training set and a large unlabeled-data pool.

8 Robustness to reorderings of the train set

The paper tests whether natural gradient is less sensitive than SGD to training-example order by measuring output variance across reordered data. Natural gradient shows lower global variance and more consistent functional behavior, with reliability depending on minibatch and step-size choices.

  • Evaluation protocol: The experiment splits 32.7M samples into chunks and segments, replaces segments with new examples, retrains from scratch, and measures variance across five runs.The protocol evaluates the influence of different parts of an online training set on the learned model.
  • Evaluation protocol: Figure 2 compares variance curves for minibatch SGD and natural gradient descent under reordered training examples.The plotted variance reflects model influence from different parts of the online training set.
  • Results: Natural gradient can run reliably with smaller minibatches when metric and gradient samples differ and the learning rate is sufficiently small or damping sufficiently large.These conditions account for noise in the estimated natural direction.
  • Results: Natural gradient has globally lower variance than SGD, producing more consistent functional behavior regardless of input-example order.The authors note that relative variance can remain high early in learning because algorithms choose a basin of attraction, while overall variance is lower.
  • Implications: The authors suggest natural gradient may be less sensitive to early examples and better suited to nonstationary data than SGD.SGD may move early toward different models based on only a few examples.

9 Natural conjugate gradient

The paper develops natural conjugate gradient to combine natural-gradient geometry with conjugate-direction ideas without directly computing or transporting directions on the manifold. Its direction coefficients are optimized jointly, yielding Euclidean conjugacy under a local second-order approximation.

  • Motivation: Natural conjugate gradient adapts nonlinear conjugate gradient to the manifold, where previous directions and current natural gradients belong to different spaces.The local geometry changes with the Fisher metric, creating a direction-transport problem.
  • Limitation: Assuming consecutive Fisher metrics are identical is detrimental because large conjugate-gradient steps are likely to change the metric.The paper identifies this assumption as a limitation of related approaches.
  • Method: Because generic transportation operators are difficult to compute, the method solves for the correction coefficient β and step size α instead.This avoids imposing strict constraints on the model density family.
  • Method: The resulting algorithm uses natural directions and jointly optimizes α_t and β_t with an off-the-shelf solver, rather than standard gradients and separately chosen coefficients.Its form resembles standard nonlinear conjugate gradient while changing the direction and coefficient optimization.
  • Analysis: Under a symmetric-Hessian second-order Taylor approximation in Euclidean space, the new direction is conjugate to the previous direction.The relation is expressed as (α_t∇_N L^T + β_t d_{t−1})^T H d_{t−1} = 0.
  • Analysis: The authors state that their approach is guaranteed to minimize the cost at each step, unlike using Polak–Ribiere while ignoring changes between consecutive metrics.The manifold method remains an approximation because it does not perform multidimensional geodesic line searches or transport gradients in the same space.

10 Benchmark

The benchmark compares natural-gradient and conjugate-gradient variants with SGD on a six-layer autoencoder using the Curves dataset and truncated-Newton computations. Natural gradient is reliable with smaller minibatches under separate-sample and noise-control conditions, while adding second-order information speeds convergence.

  • Setup: The benchmark uses a six-layer deep autoencoder on the Curves dataset, with 20K training examples of 784 dimensions, and applies a truncated-Newton pipeline to all methods except SGD.Experiments run on a GTX 580 using Theano.
  • Setup: The natural-gradient runs include minibatch size 5000 with fixed learning rate and full-batch training with line search, alongside a Polak–Ribiere full-batch variant.SGD uses minibatch 100 and learning rate 0.01.
  • Setup: Figure 3 plots whole-dataset training error on a log scale against iterations and clock time for NGD, NGD-L, NCG-L, NCG-F, and SGD.NGD uses minibatch 1000 and fixed learning rate; NGD-L uses batch line search, while NCG variants differ in direction and line-search procedures.
  • Results: Natural gradient runs reliably with smaller minibatches when metric and gradient samples differ and the learning rate is sufficiently small or damping sufficiently large.These settings compensate for noise in the natural direction.
  • Results: Incorporating second-order information into natural gradient might be beneficial, with the benchmark showing a speedup in convergence.The authors relate this observation to faster convergence previously reported for Krylov Subspace Descent than Hessian-Free Optimization.

11 Discussion and conclusions

The paper connects Hessian-Free Optimization and Krylov Subspace Descent to natural gradient through the extended Gauss–Newton approximation, and proposes a lower-memory manifold conjugate-gradient approximation. It also reports robustness to example ordering and protection against large generalization-error drops when unlabeled data improves the metric.

  • Connections: Using the extended Gauss–Newton Hessian approximation, the paper interprets Hessian-Free Optimization and Krylov Subspace Descent as implementing natural gradient descent.It further views KSD’s inclusion of the previous search direction as akin to nonlinear conjugate gradient on the manifold.
  • Proposed method: The proposed manifold nonlinear-conjugate-gradient approximation uses linear conjugate gradient to invert the metric instead of a Krylov subspace.This distinguishes it from the KSD-style construction.
  • Proposed method: The method reduces memory requirements that can become prohibitive for large models while retaining some second-order cost information and converging faster than vanilla natural gradient.The stated trade-off is lower memory with partial second-order benefits.
  • Additional properties: Compared with SGD, natural gradient is empirically more robust to training-example order and can guard against large generalization-error drops when unlabeled data improves metric accuracy.The ordering robustness may matter for nonstationary data.
  • Additional properties: Natural gradient can perform well with relatively small minibatches when metric and gradient samples differ and step size and damping are bounded against direction noise.This is presented as an empirical observation rather than an unconditional guarantee.

Appendix

The appendix specifies Fisher-metric construction for several output distributions, a truncated-Newton natural-gradient implementation, and experimental training settings. It also explains damping and learning-rate choices used to stabilize optimization, including the decision not to perform line searches routinely.

  • Metric construction: The Fisher Information Matrix is derived for linear, sigmoid, and softmax output models under Gaussian, binomial, and multinoulli assumptions.The sigmoid case recovers the usual cross-entropy error.
  • Implementation: Natural gradient is implemented with metric-vector products and a truncated Newton solver rather than explicitly inverting the metric matrix.The implementation uses automatic computational graphs and MinRes-QLP for singular or ill-conditioned systems.
  • Implementation: Each iteration computes the gradient, approximately solves the metric system, and updates parameters using a learning rate.The pseudocode uses conjugate gradients with a maximum of 20 iterations and relative tolerance 1e-4.
  • Optimization choices: The method generally avoids per-step line searches because they can overfit the current minibatch, favoring a first-order-style learning rate for small batches.The authors report using a fixed learning rate of 0.2 with no line search in one experiment.
  • Optimization choices: Damping accounts for minibatch metric error, ill-conditioning, and the local validity of the Taylor approximation, thereby enforcing a trust region.The Levenberg-Marquardt heuristic adapts damping using the ratio between predicted and observed error reduction.
Loading 1301.3584v7…