Source-linked AI summary
Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, James Stokes
TL;DR
The paper asks how to characterize and control neural-network generalization in a way invariant to parameter transformations. It introduces the information-geometric Fisher-Rao norm, derives structural and comparison results, and studies resulting generalization bounds. The analysis connects derivative identities in rectifier networks to stationary-point margin properties and a capacity measure encompassing existing norm-based approaches.
Problem
Neural-network generalization requires complexity measures that respect functional and parametrization invariances, while existing flatness and norm-based notions do not fully reflect those invariances.
Method
The paper uses information geometry to define the Fisher-Rao norm, derives an analytical characterization and norm-comparison inequalities, and analyzes derivative structure in multilayer rectifier networks.
Results
The Fisher-Rao norm serves as an umbrella for existing norm-based capacity measures and supports upper bounds on generalization error for geometrically distinct subsets of its norm ball.
Takeaways & Limitations
The Fisher-Rao norm provides an invariance-motivated capacity framework linking information geometry, flatness, norm-based complexity, and generalization analysis.
Takeaways & Limitations
The theoretical development assumes positively homogeneous activations and uses zero biases for simplicity, while the expectation underlying the Fisher-Rao norm is specialized by context.
Abstract
from arXiv · showhide
We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.
1 Introduction
The paper frames neural-network generalization as a complexity-control problem and proposes addressing it through geometric invariance. It introduces the Fisher-Rao norm as an information-geometric capacity measure that unifies existing norm-based measures.
- Deep networks’ strong out-of-sample performance raises questions about which complexity notions control generalization and why optimization finds low-complexity parameters.
- ReLU networks admit continuous parameter transformations that preserve predictions, so generalization should depend on functional equivalence classes rather than arbitrary parametrizations.
- Existing flatness definitions are not invariant to nodewise rescaling of ReLU networks or general coordinate transformations, limiting their use as generalization descriptors.
- The Fisher-Rao norm is motivated by information geometry as a geometric complexity notion incorporating distributional data aspects while connecting flatness and norm-based views.
- The paper positions the Fisher-Rao norm as an umbrella for previously studied norm-based capacity measures, addressing the difficulty of comparing competing notions.
2 Geometry of Deep Rectified Networks
The paper establishes structural derivative identities for homogeneous-activation networks and uses them to derive properties of stationary points and geometric complexity. These results support the analysis of Fisher-Rao geometry in deep rectified networks.
- The network class consists of depth-L feedforward architectures with coordinate-wise activations and parameterized layer weight matrices.
- The analysis assumes homogeneous activations satisfying σ(z) = σ'(z)z, including ReLU, leaky ReLU, and linear activations, and sets biases to zero for simplicity.
- Lemma 2.1 establishes identities relating parameter-direction derivatives of multilayer networks to network outputs, revealing equality constraints in over-parameterized gradient spaces.
- A stationary point of the hinge-loss empirical objective that separates binary data must achieve margin at least 1 on every sample.
- For deep linear networks with square loss, stationarity implies an orthogonality condition involving the composed weights and the data matrices, not necessarily global optimality.
- The derivative identity yields a tower of higher-order constraints, including <θ, Hess_θ(f_θ)θ> = L(L + 1)f_θ.
3 Fisher-Rao Norm and Geometry
The paper introduces the Fisher-Rao norm as a geometrically motivated complexity measure designed to respect neural-network reparameterization invariances. An analytical characterization yields invariance results, comparisons with existing norms, and generalization-related consequences.
- 3.1 Motivation and invariance: The Fisher-Rao norm is motivated by Information Geometry as a geometric complexity measure for neural networks.Its construction targets invariance under parameter transformations and connects steepest descent in this geometry to natural gradient descent.
- 3.1 Motivation and invariance: Existing flatness measures can change under nodewise rescalings and general coordinate transformations, motivating an invariant geometric alternative.The Fisher-Rao norm has exact invariance under linear reparameterizations and an infinitesimal invariance property under nonlinear coordinate transformations.
- 3.2 An analytical formula: The Fisher-Rao norm is defined by the quadratic form ||θ||_fr^2 = ⟨θ, I(θ)θ⟩, with I(θ) given by an expected outer product of loss gradients.The expectation distribution is left unspecified so the definition can be specialized to different contexts, and the quantity need not satisfy the triangle inequality.
- 3.2 An analytical formula: For feedforward networks with positively homogeneous activations, the paper derives an analytical Fisher-Rao identity based on the network depth and expected squared output.For absolute loss, the norm becomes proportional to a function-space norm; a related specialization is given for squared loss with Gaussian residuals.
- 3.3 Norms and geometry: The Fisher-Rao norm is invariant across equivalent parameterizations that realize the same function, unlike several standard norm-based measures.The paper compares it with ℓ2, spectral, path, and group norms, which capture distinct geometries and inductive biases.
- 3.3 Norms and geometry: For spectral, matrix-induced, group, and path norms, the paper establishes the comparison 1/(L+1)||θ||_fr ≤ ||θ||.These inequalities support viewing the Fisher-Rao norm as an umbrella for existing norm-based capacity measures and motivate more constrained geometric capacity controls.
4 Capacity Control and Generalization
The paper defines Fisher-Rao capacity through information-geometric invariance and compares its function class with several norm-induced subsets. These comparisons support generalization bounds for deep linear and rectified networks.
- Fisher-Rao capacity: The Fisher-Rao ball is defined as a parameter subset scaled by the number of network layers and a radius.The corresponding set is B_fr(r) = {θ: ||θ||_fr ≤ (L + 1)r}.
- Generalization bounds: For deep linear networks, the Fisher-Rao bound depends on the Fisher-Rao radius and intrinsic input dimension p, without width, depth, or extraneous input-dependent factors.The theorem assumes the input Gram matrix is full rank.
- Generalization bounds: The resulting generalization analysis uses Rademacher complexity of a norm-induced subset plus a deterministic function-space approximation error.For ReLU networks, direct control of the Fisher-Rao ball is replaced by this decomposition.
- Norm comparison: Spectral, group, and path norm balls form distinct geometric subsets of the unit Fisher-Rao ball for suitable data-dependent radii.The construction transfers existing capacity bounds to the Fisher-Rao function class.
- Generalization bounds: The generalization bounds extend to group and path norm-induced classes, while approximation error remains measured in function space rather than parameter space.The cancellation of comparison pre-factors removes additional factors from the resulting bound.
5 Experiments
Experiments on CIFAR-10 examine how capacity measures respond to width, label randomization, and optimization. The empirical Fisher-Rao norm tracks generalization-related behavior and improves normalized margins relative to the spectral norm.
- Over-parameterization with hidden units: The CIFAR-10 width experiments vary fully connected networks with L hidden layers and k neurons per hidden layer, without explicit regularization or data augmentation.The parameter count is varied through hidden-layer width.
- Corruption with random labels: As width increases, the empirical Fisher-Rao norm decreases for original labels alongside generalization error, and this correlation persists under label randomization.The trend is reported as stable across optimization choices.
- Margin story: The Fisher-Rao-normalized margin accounts for the generalization gap between original and randomly labeled CIFAR-10.The comparison uses models trained with vanilla gradient and natural gradient.
- Margin story: The empirical Fisher-Rao norm improves normalized margin relative to the spectral norm, with qualitatively similar results for GD+momentum.The reported table uses natural-gradient training on original and randomly labeled data.
- Width and optimization: Figure 1 compares how different norms depend on hidden-layer width after vanilla versus natural-gradient optimization.The plotted widths concern networks with L = 2 hidden layers.
- Corruption with random labels: Figure 2 compares capacity measures across label randomization while varying width from k = 200 to k = 1000 in increments of 100.Color encodes width, from red for k = 200 to blue for k = 1000.
6 Further Discussion
The paper presents the Fisher-Rao norm as a non-convex capacity measure motivated by invariance, with theoretical and empirical support for its geometric properties.
- Further discussion: The Fisher-Rao norm is proposed to study deep-learning generalization while respecting information-geometric and parametrization-related invariances.The paper reports both theoretical and empirical investigation of its properties.
A Proofs
The appendix supplies proofs for the structural derivative lemma, norm comparisons, and generalization results. It also includes arguments for function-space approximation and related geometric claims.
- Structural lemma: The proof of the structural lemma proceeds by induction on the layer gap s - t and handles hard-coded zero weights.The induction begins with the base case s - t = 0 and uses a chain-rule step.
- Norm comparisons: The norm-comparison proofs use Holder’s inequality and recursive or peeling arguments tailored to spectral, group, path, and induced norms.The appendix also gives a separate strategy for chain-of-induced norms.
- Geometric properties: The appendix constructs convex combinations in function space by placing two networks side-by-side and adding an output layer with mixing weights.The resulting network remains within the appropriately scaled Fisher-Rao ball.
- Generalization: The generalization proof bounds the Fisher-Rao class through a comparison class and an approximation term measured in function space.The relevant function-class bounds are obtained by selecting subsets of the Fisher-Rao ball.
A.1 Invariance of natural gradient
Natural gradient flow is invariant under invertible re-parametrizations and approximately invariant under over-parametrization, with the latter governed by a projection-like matrix.
- Re-parametrization: Natural gradient flow remains equivalent across invertible re-parametrizations when corresponding initial parameter locations represent the same point.This result assumes invertible Fisher Information matrices and an invertible Jacobian between parameterizations.
- Over-parametrization: Under over-parametrization, the infinitesimal change in represented coordinates is transformed by M_t, whose eigenvalues are only 0 or 1.The matrix maps changes between the higher-dimensional and original parameter spaces.
- Over-parametrization: The over-parametrized dynamics are nearly invariant whenever the represented function coordinates satisfy ξ(θ_t) = ξ_t.This claim follows from the relation between the coordinate trajectories and the matrix M_t.
- Special case: When parameter and coordinate dimensions match and the Jacobian has full rank, M_t becomes the identity, reducing the setting to the re-parametrized case.This is the p = q special case of the over-parametrized derivation.
B Experimental details
The experiments use a K-class softmax classifier with ReLU hidden layers and cross-entropy loss, then estimate the Fisher-Rao norm from test-set samples.
- Model and loss: The network has a linear K-dimensional output layer, ReLU activations in intermediate layers, and one-hot cross-entropy loss with softmax outputs.The softmax maps logits to class probabilities used by the cross-entropy objective.
- Fisher-Rao calculation: The loss gradient with respect to the network output is g(f) − e_y, which is substituted into the Fisher-Rao norm expression.Here e_y denotes the one-hot encoded class label and g(f) the softmax output.
- Empirical estimation: The empirical Fisher-Rao norm is estimated by sampling from a test set of size m because the population covariate density is unavailable.The sampling procedure yields the practical formulas used in the experiments.
B.1 Additional experiments and figures
Additional figures examine how capacity measures vary with depth and width, how margin distributions change under rescaling, how Fisher-Rao and path-2 geometries differ, and how Adam compares with K-FAC in conditioning.
- Depth dependence: Figure 3 compares norm dependence on depth L for width k = 500 after vanilla gradient descent and natural gradient descent, with Fisher-Rao norms normalized by L + 1.Red denotes vanilla gradient descent and blue denotes natural gradient descent.
- Width and label randomization: Figure 4 varies label randomization under natural gradient descent while encoding network widths from k = 200 to k = 1000 by color.Widths increase in increments of 100, from red for k = 200 to blue for k = 1000.
- Margin distributions: Figure 5 displays margin distributions for natural versus vanilla gradient across unrescaled, spectral-norm-rescaled, and empirical Fisher-Rao-rescaled models.Optimization method occupies rows, while rescaling condition occupies columns.
- Norm geometry: Figure 6 overlays solid Fisher-Rao and dotted path-2 norm level sets, using color to represent norm values.The plot compares the geometries induced by the two capacity measures.
- Conditioning: Figure 7 reproduces a conditioning experiment after 10^4 iterations, contrasting Adam with dashed curves against K-FAC in red.The experiment concerns the conditioning setting studied in reference [16].