Limitations of the Empirical Fisher Approximation for Natural Gradient Descent

Frederik Kunstner; Lukas Balles; Philipp Hennig

Source-linked AI summary

Limitations of the Empirical Fisher Approximation for Natural Gradient Descent

Frederik Kunstner, Lukas Balles, Philipp Hennig

arXiv:1905.12558v3May 29, 2019cs.LGstat.ML

TL;DR

The paper asks whether the empirical Fisher genuinely supplies the second-order information attributed to the Fisher. It critically analyzes the approximation and its theoretical justifications, finding that it is not generally curvature-aware and that its practical success may instead reflect gradient-noise adaptation. These conclusions are supported by convergence conditions, exact-fit behavior, and simple optimization examples.

Problem
The paper addresses the unclear theoretical relationship between the empirical Fisher, the Fisher, and second-order information despite widespread empirical-Fisher use.
Method
The paper reviews natural-gradient and generalized Gauss-Newton connections, critically examines empirical-Fisher arguments, and uses simple deterministic examples.
Results
The empirical Fisher is not guaranteed to capture useful second-order information, and its practical success may instead reflect adaptation to gradient noise.
Takeaways & Limitations
Empirical-Fisher preconditioning should not be treated as generally equivalent to Fisher-based curvature adaptation.
Takeaways & Limitations
Fisher–empirical-Fisher agreement requires a correct model and enough data relative to model capacity, conditions unlikely in typical machine-learning settings.

Abstract

from arXiv · show

Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between approximate second-order methods and heuristics like Adam. We dispute this argument by showing that the empirical Fisher---unlike the Fisher---does not generally capture second-order information. We further argue that the conditions under which the empirical Fisher approaches the Fisher (and the Hessian) are unlikely to be met in practice, and that, even on simple optimization problems, the pathologies of the empirical Fisher can have undesirable effects.

1 Introduction

The paper examines whether the empirical Fisher provides the curvature information attributed to the Fisher and argues that this justification fails in general. It distinguishes the two quantities, illustrates their divergent optimization behavior, and points to gradient-noise adaptation as an alternative explanation for empirical Fisher methods’ practical success.

The paper studies supervised probabilistic models whose losses include least-squares regression and cross-entropy classification.
Natural gradient descent preconditions gradient updates with the Fisher, which provides theoretical grounding as an approximate second-order method.The Fisher coincides with a generalized Gauss-Newton approximation for the problems considered.
Even on simple linear regression, empirical-Fisher preconditioning distorts the gradient field, whereas natural gradient descent adapts to curvature.The paper presents this as a counterexample to arguments treating the empirical Fisher as a curvature approximation.
The empirical Fisher replaces the model-output expectation with a training label, so it is not generally a Monte Carlo estimate of the Fisher.Because training labels are not sampled from the model’s predictive distribution, the relationship between the two matrices depends on model–data agreement.
The paper’s main contribution is a critical discussion arguing that the empirical Fisher is not guaranteed to capture useful second-order information.It also proposes a clarifying amendment to the generalized Gauss-Newton definition and argues that Fisher-convergence conditions are unlikely in practice.
The paper suggests that empirical-Fisher methods may succeed through adaptation to gradient noise rather than adaptation to curvature.This interpretation is presented as an alternative explanation, not as a settled result.

2 Related work

Related work has used the Fisher and empirical Fisher for approximate second-order optimization, stochastic-gradient adaptation, curvature analysis, and diagnostics. The paper positions its contribution as a detailed clarification of their relationships and limitations rather than an invalidation of prior applications.

KFAC uses a computationally efficient structural approximation to the Fisher in deep-learning optimization.
Many papers rely on the empirical Fisher for preconditioning and other purposes, while often giving limited attention to its relationship with the Fisher, Gauss-Newton matrices, and Hessian.
Adam uses a moving average of squared gradients as an approximation to the diagonal of the Fisher, motivating connections between Adam and natural-gradient methods.
The empirical Fisher has been used to study neural-network curvature, explain characteristics of SGD, and diagnose training.
Some work interprets the empirical Fisher as a non-central covariance of stochastic gradients and explicitly adapts updates to gradient stochasticity rather than curvature.
The paper elaborates on previously noted concerns about the empirical Fisher while adding novel results and insights.

3 Generalized Gauss-Newton and natural gradient descent

This section defines natural-gradient and generalized Gauss-Newton curvature matrices and clarifies their relationships with the Fisher and Hessian. Under exponential-family assumptions, the Fisher matches a particular GGN and approximates the Hessian with error controlled by residuals.

3.1 Natural gradient descent: Natural gradient descent measures parameter steps using KL divergence between model distributions, yielding a Fisher-preconditioned steepest-descent direction.
Terminology: The paper distinguishes statistical and machine-learning uses of “Fisher” and “empirical Fisher,” with the latter term reserved here for the training-label-based quantity.
Generalized Gauss-Newton: The generalized Gauss-Newton matrix omits second-order information from the prediction map and can remain positive semidefinite when the loss is convex.
Generalized Gauss-Newton: The GGN depends on the chosen decomposition of the objective, so different splits can recover classical Gauss-Newton or the full Hessian.
Connections between the Fisher, the GGN and the Hessian: For exponential-family likelihoods with natural parameters, the Fisher coincides with the GGN under the canonical split.
Connections between the Fisher, the GGN and the Hessian: The Fisher and GGN provide Hessian approximations whose error can be bounded using generalized residuals.The approximation improves as residuals diminish and is exact when the data is perfectly fit.

4 Critical discussion of the empirical Fisher

The paper challenges two justifications for the empirical Fisher: its generalized Gauss–Newton interpretation and its convergence to the Fisher near well-fitted minima. It argues that both claims require qualifications, while empirical-Fisher preconditioning can distort update scaling and direction away from optima.

4.1 The empirical Fisher as a generalized Gauss-Newton matrix: The empirical Fisher formally matches a generalized Gauss–Newton construction, but this does not provide the approximation guarantee usually associated with the GGN.The paper examines the specific split underlying this correspondence and argues that formal correctness is insufficient for useful second-order information.
4.1 The empirical Fisher as a generalized Gauss-Newton matrix: For least-squares problems, the empirical-Fisher split ignores curvature from exp while retaining curvature from log, contrary to the classical Gauss–Newton rationale based on small residuals.The Fisher approaches the Hessian as residuals become small, whereas the empirical Fisher approaches zero when individual residual gradients vanish.
4.2 The empirical Fisher near a minimum: The empirical Fisher converges to the Fisher at a minimum only under realizability and enough data to recover the true parameters.Under these conditions, both matrices approach the Hessian; the paper emphasizes that proximity to a minimum alone is not sufficient.
4.2 The empirical Fisher near a minimum: Model realizability is unlikely for approximate machine-learning models, and overparameterized models can have near-zero empirical Fisher at a minimum while the Hessian remains nonzero.Flexible models may fit training data nearly perfectly without making the empirical Fisher a reliable representation of curvature.
4.3 Preconditioning with the empirical Fisher far from an optimum: The near-minimum relationship does not generally hold during optimization, making the empirical Fisher a questionable preconditioner far from the optimum.The paper reports that empirical-Fisher updates can point almost opposite to natural-gradient updates and can perform unfavorably even on simple regression and classification tasks.
4.3 Preconditioning with the empirical Fisher far from an optimum: Because the empirical Fisher sums squared gradients, inverse preconditioning can produce update magnitudes nearly inversely proportional to gradient magnitude, requiring step-size adaptation and manual tuning.The selected step size can become dependent on the starting point.

5 Variance adaptation

The empirical Fisher can be interpreted as adapting updates to stochastic gradient noise rather than curvature. This perspective may explain practical success, but computing the relevant full-matrix adaptation is difficult.

As a non-central second moment of stochastic gradients, the empirical Fisher provides an inexpensive but noisy estimate-related quantity.The stochastic gradient is an inexpensive but noisy estimate of the full gradient.
The empirical Fisher contains gradient-noise covariance information that can attenuate harmful stochastic-gradient effects.It may scale back updates in directions with high noise.
Full-matrix variance adaptation extends diagonal signal-to-noise rescaling ideas to preconditioning based on gradient variability.Earlier work connected covariance-based updates and coordinate-wise signal-to-noise adaptation to expected-error reduction.
Empirical-Fisher preconditioning can be viewed as adapting to gradient noise instead of curvature, while its gradient outer-product term counteracts poor scaling.This interpretation distinguishes its role from natural-gradient curvature adaptation.
Computing the full variance-adaptation matrix requires evaluating all gradients, and meaningful mini-batch estimates with provable noise attenuation are not obvious.The authors present variance adaptation as a possible explanation of practical success and a direction for future research.
Using the square-root of the empirical Fisher, as in Adam, avoids inverse-gradient scaling but widens the conceptual gap from natural gradient and effectively cancels gradient magnitude.The resulting behavior has been examined as sign gradient descent.

6 Conclusions

The paper concludes that the empirical Fisher lacks reliable second-order justification, while variance adaptation may better explain its practical usefulness. It also argues that computing the true Fisher need not be more expensive.

The empirical Fisher’s formal generalized-Gauss–Newton status does not ensure that it retains useful second-order information.The paper proposes clarifying the generalized-Gauss–Newton definition to require useful approximation properties.
The empirical Fisher approaches the Fisher only near a minimum under strong assumptions of a correct model and sufficient data relative to model capacity.These assumptions are considered unlikely in practice, especially for overparameterized approximators and approximate minima.
Far from an optimum, empirical-Fisher preconditioning makes update magnitudes inversely proportional to gradient magnitude, complicating step-size tuning and often harming performance even for linear models.
The paper identifies variance adaptation as a possible alternative explanation for the practical success of empirical-Fisher methods.
Existing arguments do not justify the empirical Fisher as a general approximation to the Fisher or Hessian, although particular model classes may still behave reasonably.The authors therefore regard the true Fisher as the safer curvature matrix when such cases are not clearly identified.
An unbiased one-sample estimate of the true Fisher can have the same computational cost as the empirical Fisher, and exact Fisher computation is feasible in many cases.The authors attribute reluctance partly to limited implementation support and caution against choosing the empirical Fisher solely for convenience.

Supplementary Material

The supplementary material develops natural-gradient and generalized-Gauss–Newton foundations, explains when Fisher-related matrices coincide, and connects the generalized Gauss–Newton to local linearization.

Natural gradient defines steepest descent using KL divergence between model distributions rather than Euclidean distance between parameters.This makes the update depend on the information geometry of the statistical model.
The Fisher information arises as the Hessian of the KL divergence and as an expected outer product of score gradients.The derivation uses the chain rule and the identity for the derivative of a log density.
The Fisher, empirical Fisher, and Hessian coincide when the empirical data distribution equals the model distribution under the stated probabilistic setting.
For exponential-family conditional models, the Fisher can be computed through generalized-Gauss–Newton structure involving model-output derivatives and exploitable Hessian structure.The construction uses Jacobians of the model output and Hessians of the log likelihood with respect to model outputs.
Squared loss can be represented with a unit-variance Gaussian, making the Fisher an outer product of gradients for scalar-valued model outputs.The supplementary material separately treats the Hessian of the negative log likelihood with respect to the model output.
The Fisher’s outer products of individual gradients differ from the outer product of gradients of the overall loss.
For cross-entropy classification, the loss corresponds to an exponential-family model using softmax, whose output-space Hessian is independent of the class label.
The generalized Gauss–Newton preserves the outer loss’s second-order information while ignoring the inner function’s curvature, and equals the Hessian when that inner function is linear.It is the Hessian of a local approximation formed by linearizing the inner function.

B Computational aspects

A one-sample Monte Carlo estimate from the model distribution is an unbiased Fisher approximation with the same computational complexity as the empirical Fisher. Alternative biased choices and repeated sampling introduce trade-offs.

A one-sample Monte Carlo Fisher estimate samples an output from the model distribution for each input and computes the corresponding gradient outer product.The estimate is noisy but unbiased and avoids the empirical Fisher’s stated problems.
Using the most likely model output instead of sampling reduces variance but introduces bias, and for least squares can produce an identically zero gradient.
Averaging multiple Monte Carlo outputs improves estimate quality but repeats most computation, including the expensive model Jacobian calculation.The Jacobian does not depend on the sampled output, so repeated sampling duplicates the difficult computation.
Closed-form generalized-Gauss–Newton computation can avoid repeated Monte Carlo Jacobian work, but current autodifferentiation libraries provide limited Jacobian support.Empirical-Fisher implementations also require individual gradients, which is equivalent to accessing the relevant Jacobian.

C.1 Proof of Propositon 1

For the exponential-family problems considered, the proof establishes that the Fisher information coincides with the generalized Gauss–Newton matrix through chain-rule and expected-curvature identities.

C.1 Proof of Propositon 1: For exponential-family distributions with natural parameters f, the Fisher information matrix coincides with the generalized Gauss–Newton matrix.The result applies the stated split in the generalized Gauss–Newton definition.
C.1 Proof of Propositon 1: The chain rule rewrites the model-gradient score using the Jacobian of f with respect to θ.This Jacobian does not depend on y and can therefore be taken outside the expectation.
C.1 Proof of Propositon 1: The equivalence follows by combining the chain-rule rewrite with equality between expected gradient outer products and expected Hessians.For exponential-family log densities, the Hessian with respect to f is independent of y.

C.2 Proof of Proposition 2

The proposition bounds the difference between the Fisher or generalized Gauss–Newton matrix and the Hessian using model residuals and the smoothness of the model mapping.

C.2 Proof of Proposition 2: The Fisher or generalized Gauss–Newton matrix differs from the Hessian by a quantity bounded using residuals and the model f’s smoothness constant.This is the central claim of Proposition 2.
C.2 Proof of Proposition 2: The proof expresses the Hessian in terms of derivatives of the negative log likelihood with respect to the model output components.The componentwise derivative is evaluated at f_n(θ).
C.2 Proof of Proposition 2: Taking absolute values inside the double sum produces the upper bound used in the proposition.The resulting spectral-norm statement follows from this bound.

D Experimental details

The experiments use average-loss scaling, specified synthetic and benchmark datasets, fixed or searched optimization settings, and an initialization favorable to the empirical Fisher in logistic regression.

D Experimental details: The implementation uses the average loss, so the Fisher and empirical Fisher are rescaled by 1/N relative to the main-text sum formulation.Reported step sizes and damping parameters correspond to the average-loss implementation.
D Experimental details: The linear-regression experiment uses N = 1000 samples with yi = 2 + 2xi + ϵi and ϵi ∼N (0, 1).The trajectories use relative rescaling of the gradient, natural gradient, and empirical-Fisher-preconditioned gradient for visibility.
D Experimental details: The optimization experiments run for 100 iterations from θ0 = 0, with step size and damping selected by grid search.This initialization is favorable to the empirical Fisher for the logistic-regression problems.
D Experimental details: At θ = 0, the empirical Fisher and Fisher coincide for logistic regression when predicted probabilities are uniform.The alternative condition σ(f) ∈{0, 1} requires infinite weights.

E Additional plots

Additional plots show that empirical-Fisher behavior depends on model misspecification and initialization: it can approximate the Fisher near favorable points but degrade elsewhere and slow optimization.

E Additional plots: Additional linear-regression examples show that poor empirical-Fisher-preconditioned updates compared with NGD are not isolated to the main-text examples.Figure 5 compares the Fisher and empirical Fisher as preconditioners and tracks cosine similarity along the empirical-Fisher path.
E Additional plots: On the Boston dataset, empirical-Fisher behavior is close to NGD near the minimum but degrades as the initial loss increases.Far from the minimum, large individual gradients produce very small steps and slow convergence.
E Additional plots: A larger initial step size could counteract slow progress far from the minimum but would cause oscillations near the minimum.The suitable step size therefore depends on the starting point and would ideally decrease over time.
E Additional plots: Under model misspecification, the empirical Fisher can be arbitrarily wrong at the minimum even with large N.It is accurate at the minimum when the data-generating model matches the assumed observation model.