PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Pruning for Neural Architectures using Fisher Information Distances

David S. Berman, Yen-Yu Fu, Edward Hirst, Thelma Chiwete Obirai

arXiv:2609.16129v1cs.AIcs.ITmath.DGmath.IT

TL;DR

Pruning removes parameters by setting them to zero, but magnitude and local-Fisher criteria capture only parts of the resulting model change. This paper uses Fisher geodesic distance and increasingly faithful approximations to rank parameters; across four testbeds, the family outperforms both baselines on two performance measures, with cheaper approximations often matching the most accurate one.

  • Problem

    Magnitude pruning uses a Euclidean displacement measure, while local-Fisher pruning ignores how far a parameter must move to reach zero.

    Source: §1 · p. 3, §2.5.1 · p. 7, §2.5.2 · p. 7

  • Method

    The method ranks parameters by their Fisher geodesic distance to the zero-parameter hypersurface and evaluates progressively more faithful approximations.

    Source: §1 · p. 3, §3.1 · p. 9

  • Results

    The Fisher-distance family outperforms magnitude and local-FIM pruning in all four testbeds on accuracy and Matthews correlation coefficient, while a cheap member matches or exceeds the exact method in every testbed.

    Source: §5 · p. 17

  • Takeaways & Limitations

    The geometric measure provides a pruning criterion, and the tested cheaper approximations retain strong performance without requiring the most refined distance computation.

    Source: §5 · pp. 17–18

    The diagonal Fisher approximation ignores parameter correlations, leaving block-structured Fisher distances as a potential refinement.

    Source: §2.3 · p. 5, §5 · p. 17

Abstract

from arXiv · show

A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete $0$-$100\%$ pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

1 Introduction

The paper treats pruning as a finite model-space displacement and proposes Fisher geodesic distance to measure the effect of setting a parameter to zero. Across four architecture–dataset combinations, Fisher-distance methods outperform magnitude and local-Fisher baselines, while cheaper approximations often match the most accurate one.

1 Introduction

  • Pruning ranks parameters for removal, and that ranking largely determines the quality of the pruned model.Pruning can reduce memory, latency, and energy costs.Source: §1 · p. 3
  • Magnitude pruning measures displacement in a non-invariant Euclidean metric, whereas local-Fisher criteria measure sensitivity at the trained point without accounting for the distance required to zero a parameter.The paper argues that these methods capture only part of the pruning problem.Source: §1 · p. 3
  • The proposed importance measure is a parameter’s Fisher geodesic distance from its trained value to the zero-parameter hypersurface.The Fisher information metric measures distances between models through the length of the shortest path joining them.Source: §1 · p. 3
  • With a diagonal, constant Fisher metric and a coordinate-line path, the distance reduces to the product of parameter magnitude and the square-rooted Fisher criterion.Relaxing the constancy assumption yields successively more faithful approximations by recomputing the metric and evaluating it at points along the path.Source: §1 · p. 3
  • The Fisher-distance family outperforms magnitude and local-FIM pruning in all four architecture–dataset combinations on accuracy and Matthews correlation coefficient.The Fisher criterion alone is worse than magnitude pruning in three of four combinations, while cheaper approximations often perform as well as the most accurate approximation.Source: §1 · p. 3
  • The geometric framework supplies a parameter-importance measure and cheaper approximations intended to combine effective pruning with computational efficiency.The paper’s remaining sections cover background, derivation, numerical results, and a summary.Source: §1 · pp. 3–4

2.1 Neural Architectures

The study uses a fully connected network and a compact vision transformer to span a simple model and an attention-based architecture while keeping exact computations affordable.

2.1 Neural Architectures

  • The networks are treated as conditional probability models, making them points in a space of probability distributions where pruning is finite motion.The parameter vector provides coordinates on that model space.Source: §2.1 · p. 4
  • The two architectures were chosen to span a simple useful model and a contemporary attention-based model while keeping exact computations affordable.The architectures are evaluated on MNIST and CIFAR-10.Source: §2.1 · p. 4
  • SimpleNN flattens images into two 64-unit ReLU hidden layers and a 10-class output, with 55,050 MNIST or 201,482 CIFAR-10 parameters.Its parameter count varies with input image dimensions.Source: §2.1 · p. 4
  • SimpleViT uses 4 × 4 image patches, 128-dimensional embeddings, four four-head transformer blocks, and mean pooling, with about 5.4 × 10^5 parameters.LayerNorm parameters and positional embeddings, about 1.9% of parameters, are excluded from pruning, although pruning ratios use all parameters.Source: §2.1 · p. 4

2.2 Datasets

MNIST and CIFAR-10 provide balanced ten-class image-classification testbeds, paired with both architectures. Their different pruning behavior helps distinguish methods across the pruning range.

2.2 Datasets

  • MNIST and CIFAR-10 each contain 60,000 training and 10,000 test examples for balanced ten-class classification.MNIST images are 28 × 28 grayscale digits; CIFAR-10 images are 32 × 32 color natural images.Source: §2.2 · p. 4
  • Using each dataset with each architecture yields four testbeds.This pairing supports comparisons across both a fully connected network and a vision transformer.Source: §2.2 · p. 4
  • MNIST accuracy remains near its ceiling until many weights are removed, while CIFAR-10 degrades more gradually and allows comparisons across the full pruning range.The contrast highlights a ceiling effect in MNIST, where methods separate mainly at extreme pruning.Source: §2.2 · p. 4

2.3 Fisher Information

The Fisher information matrix measures log-likelihood curvature and induces a Riemannian metric on model parameter space. The study uses the model Fisher and a diagonal approximation to make the metric computationally tractable.

2.3 Fisher Information

  • The Fisher information matrix quantifies log-likelihood curvature and induces a natural Riemannian metric on the parameter manifold.The paper gives two equivalent forms under mild regularity conditions: expected score outer products and the negative expected Hessian.Source: §2.3 · pp. 4–5
  • The model Fisher averages over the model’s predicted output distribution, whereas the empirical Fisher uses observed labels; only the model Fisher carries the geometric meaning used here.The empirical Fisher coincides with the model Fisher at a perfectly fitted optimum, but not generally.Source: §2.3 · p. 5
  • Retaining only the diagonal reduces computation to O(P) but treats parameter-coordinate directions as mutually orthogonal.This approximation omits parameter correlations; block-structured alternatives such as Kronecker-factored metrics could retain some correlations.Source: §2.3 · p. 5

2.4 Information Geometry of Model Space

The Fisher information supplies a natural metric for model space, linking local parameter changes to statistical distinguishability. Integrating this metric gives distances for finite changes such as pruning, and the metric is invariant to reparameterisation.

2.4 Information Geometry of Model Space

  • The Fisher metric is the infinitesimal limit of statistical distinguishability and is unique up to scale among metrics invariant under sufficient statistics.The Kullback–Leibler divergence determines this local quadratic form, although it is not itself a distance.Source: §2.4 · pp. 5–6
  • Unlike Euclidean parameter length, Fisher-based distance is unchanged by reparameterisation, so it reflects the model rather than its arbitrary coordinates.Magnitude-based importance can change when parameters are rescaled, even when the represented function is unchanged.Source: §2.4 · p. 6
  • Integrating the metric along a path extends local distinguishability to finite changes, including moving a trained parameter to zero.The Fisher metric also underlies the natural gradient and analyses of learning dynamics and representation.Source: §2.4 · p. 6

2.5 Existing Pruning Schemes

The section presents magnitude and Fisher-based pruning as complementary baselines: one measures parameter displacement without model geometry, while the other uses local curvature without accounting for the displacement to zero. The Fisher-distance approach combines these factors.

2.5.1 Magnitude Pruning

  • Magnitude pruning ranks parameters by absolute value, is inexpensive, and remains widely used despite difficulty demonstrating consistent improvements over it.Its implicit Euclidean-distance assumption can make rankings sensitive to parameter rescaling and treat stiff and flat directions alike.Source: §2.5.1 · pp. 6–7

2.5.2 Fisher Pruning

  • Fisher pruning uses local curvature to account for model geometry, but ignores how far a parameter must move to reach zero.Its second-order expansion is local, whereas pruning may apply a large perturbation.Source: §2.5.2 · p. 7

2.5 Existing Pruning Schemes

  • The two criteria are complementary fragments: magnitude measures displacement without a metric, while Fisher measures the metric without displacement.The Fisher distance combines both in its leading term, and further terms yield measures reported to outperform either baseline.Source: §2.5.2 · p. 7

2.6 Relation to This Work

The work applies Fisher–Rao distance in a per-coordinate form as a pruning importance measure. Its contribution is a geometric derivation of this criterion and a hierarchy of approximations, rather than the Fisher geometry or magnitude-weighted scores themselves.

2.6 Relation to This Work

  • The per-coordinate Fisher distance asks how far the model must move to zero one parameter, complementing global spectral approaches that assess resolvable directions.The authors state that this local metric use had not previously been made explicit as a pruning criterion, to their knowledge.Source: §2.6 · p. 7
  • Magnitude multiplied by a data-dependent weight is not new; the novelty is deriving this specific Fisher weighting geometrically, with its assumptions explicit and individually relaxable.The leading term √Ikk |θk| is presented as part of a defined geometric quantity rather than as a heuristic.Source: §2.6 · pp. 7–8
  • The contribution combines a computable per-coordinate Fisher–Rao pruning measure with the empirical finding that cheaper approximations remain effective.The Fisher–Rao distance itself is classical.Source: §2.6 · p. 8

3 Deriving the Fisher-Distance Pruning Scheme

The scheme treats pruning as finite motion from a trained parameter to zero and scores that motion using Fisher distance in model space. A sequence of explicit approximations yields estimators that trade fidelity against computational cost.

3 Deriving the Fisher-Distance Pruning Scheme

  • The safest parameter to prune is the one whose removal moves the model the shortest Fisher-metric distance to the zero-coordinate hypersurface.The construction reduces this distance through explicit approximations, whose leading term combines the standard magnitude and Fisher criteria.Source: §3 · p. 8, §3.1 · p. 8

3.1 The Distance Measure

  • The distance approximation assumes a diagonal Fisher matrix and a path that changes only the scored coordinate; holding the metric constant further reduces it to magnitude times square-rooted Fisher information.The diagonal assumption treats parameter directions as orthogonal.Source: §3.1 · pp. 8–9
  • Relaxing metric constancy along the coordinate path allows recomputing the Fisher matrix at the prior pruning step’s endpoint or sampling and summing segment lengths.The implemented estimator samples a uniform grid of K path points and averages Ikk along it.Source: §3.1 · p. 9
  • The implemented path score moves only the coordinate being assessed, holding all other parameters at their current values.Its square root is taken after averaging the metric, not before; the authors report the difference from literal length discretisation as immaterial at experimental precision.Source: §3.1 · pp. 9–10

3.2 The Estimators Evaluated

  • The one-shot estimator fixes the metric at the trained point, while the iterative estimator recomputes it at each pruning step.The iterative version uses one Fisher evaluation per pruning step.Source: §3.2 · p. 10
  • The exact-path estimator recomputes the metric along each surviving coordinate’s path, requiring O(P) Fisher probes per pruning step.It is the reference quantity against which the cheaper estimators are judged.Source: §3.2 · p. 10
  • The global-path surrogate shrinks all coordinates together along a ray and reduces the cost to K Fisher evaluations, independent of parameter count.Unlike the coordinate-line estimator, it has no a priori guarantee of tracking the true distance because it evaluates the metric on a ray through the origin.Source: §3.2 · pp. 10–11

4 Numerical Verification

Across four architecture–dataset testbeds, Fisher-distance pruning outperforms magnitude and local-Fisher baselines, while a cheap iterative approximation matches the exact measure. The experiments also examine path resolution and the substantial cost of exact sweeps.

4.1 Protocol

  • The four testbeds use five seeds and a shared trained checkpoint per seed; global, unstructured pruning sweeps 0–100% without retraining.Performance is recorded as normalized accuracy and MCC; chance-floor correction is also reported for accuracy comparisons across architectures.Source: §4.1 · p. 11

4.2 Comparative Performance

  • Fisher-distance schemes beat magnitude pruning across all four testbeds on accuracy and MCC; a cheap member matches the exact measure in every testbed.Figure 4.1 shows normalized accuracy across pruning ratios; MCC widens some gaps, and the local Fisher criterion alone is worse than magnitude in three testbeds.Source: Figure 4.1 · p. 13, §4.2 · pp. 13–14, §4.3 · p. 14

4.3 The Approximation Hierarchy

  • Recomputing the metric at each pruning step improves on one-shot pruning, while path resolution adds little and some cheap schemes differ by architecture.The iterative measure matches the exact measure in three testbeds within 0.02–0.3 standard deviations, at 37–10,720 times lower cost; the global approximation trails it on both transformer testbeds.Source: §4.3 · p. 14

4.4 Resolution of the Path

  • On SimpleNN/MNIST, increasing path resolution leaves AUC within 0.7 standard deviations of K = 2 while cost rises 5.8-fold.The authors infer that a two-point quadrature captures the coordinate-line integral within the precision resolved by pruning outcomes.Source: Figure 4.3 · p. 16, §4.4 · p. 16

4.5 Computational Cost

  • Exact SimpleViT sweeps take 3.4–4.6 days on 128 cores, and the exact transformer runs total approximately 40 CPU-node-days.A direct forward-mode derivative computes one Fisher diagonal entry, speeding probes by 8.9×–325× versus the back-propagation route.Source: §4.5 · p. 17

5 Summary

The paper treats pruning as a finite displacement in model space and measures it with the Fisher information metric, yielding a hierarchy of pruning measures. Across the evaluated testbeds, the Fisher-distance family outperforms both baselines, while the iterative measure is recommended for practical use.

5 Summary

  • Fisher distance measures the finite model displacement to setting a parameter to zero; under diagonal, coordinate-line, constant-metric assumptions, its leading term is √I_kk |θ_k|.Magnitude and Fisher pruning are complementary factors of this geometric quantity, not competing heuristics.Source: §5 · p. 17
  • Across all four testbeds and both performance measures, Fisher-distance pruning beats both baselines; a cheap family member matches or exceeds the exact measure at up to four orders of magnitude lower cost.The iterative measure is indistinguishable from the exact computation on every examined testbed.Source: §5 · p. 17
  • The recommended iterative measure is the cheapest, most robust across architectures, and indistinguishable from the exact computation on every examined testbed.Further refinements beyond recomputing the metric at each pruning step make no significant difference in these tasks.Source: §5 · p. 17
  • The derivation retains a diagonal Fisher approximation, leaving parameter correlations unmodeled; a Kronecker-factored metric is proposed as a route to block-structured distances at realistic scale.The authors also leave open whether residual weakness on SimpleNN/MNIST reflects a testbed ceiling or the diagonal metric.Source: §5 · p. 17
  • The assumed coordinate-line path is a geodesic only in special cases, and the authors infer—but do not demonstrate—that the true geodesic departs little from it in these models.Computing genuine Fisher geodesics for even a small network is proposed as informative evidence.Source: §5 · p. 17
  • Why the crudest approximation performs comparably to the exact measure remains theoretically unresolved, as does the geometry behind magnitude pruning’s durable competitiveness.The authors suggest trained ensembles, loss landscapes near minima, or overparameterisation as possible sources of the relevant property.Source: §5 · p. 18
  • The framework could extend to structured pruning by measuring distance to a submanifold where a parameter group vanishes, and could be combined with fine-tuning or pruning-aware training.The reported experiments omit retraining to isolate the importance measure.Source: §5 · p. 18
  • The paper frames its broader contribution as a geometrically motivated understanding of effective pruning methods for reducing AI inference costs.The authors connect this direction to the challenge of AI sustainability.Source: §5 · p. 18

A Matthews Correlation Coefficient Results

The Matthews correlation coefficient (MCC) results corroborate the main-text conclusions and show larger Fisher-distance advantages over magnitude pruning than accuracy does. MCC also distinguishes majority-class collapse that can leave accuracy near the class prior.

A Matthews Correlation Coefficient Results

  • MCC is chance-corrected and scores a single-class predictor at zero, whereas accuracy can stay near the class prior when pruning degrades predictions toward the majority class.The widening MCC gaps indicate that losing schemes lose more discriminative information than accuracy alone suggests.Source: A Matthews Correlation Coefficient… · pp. 19–20
  • MCC curves corroborate the main-text conclusions across the four testbeds; scheme rankings match accuracy in three, with only a statistically indistinguishable top-two exchange on SimpleViT/CIFAR-10.Figure A.1 plots normalized MCC against pruning ratio, with five-seed means and ± one standard deviation bands.Source: Figure A.1 · p. 19, A Matthews Correlation Coefficient… · p. 19
  • Fisher-distance advantages over magnitude grow on MCC: 1.02×→1.03×, 1.41×→1.58×, 1.14×→1.15×, and 1.17×→1.21×.These four changes compare the advantage on accuracy with the advantage on MCC.Source: A Matthews Correlation Coefficient… · p. 19
  • Fisher-criterion deficits deepen on MCC: 0.82×→0.80× for SimpleNN/MNIST, 0.74×→0.71× for SimpleViT/MNIST, and 0.71×→0.69× for SimpleViT/CIFAR-10.The criterion already underperforms magnitude pruning in those three testbeds.Source: A Matthews Correlation Coefficient… · pp. 19–20

Related papers

Loading 2609.16129v1…