PaperlayerView exact revision on arXiv

Source-linked AI summary

Functional Degeneracy in Neural Networks: Measurement and Pruning

Maria Matveev, Pascal Esser, Ayush Bharadwaj, Lucius Bushnaq, Gitta Kutyniok

arXiv:2608.30741v1cs.LG

TL;DR

The paper addresses how much a trained model can be compressed while preserving behavior and introduces behavioral recovery rank to measure the required functional degrees of freedom. Across controlled regression and MNIST experiments, pruning retains more parameters than this behavioral benchmark beyond task saturation, while the paper notes that broader validation and stronger compression pipelines remain open.

  • Problem

    The paper asks how many parameter directions preserve a learned function and how much of the resulting redundancy is exploited through pruning for more efficient deployment.

  • Method

    It defines behavioral recovery rank as the number of leading behavioral-Hessian eigendirections needed to recover performance within a specified tolerance.

  • Results

    Structural and magnitude pruning retain more parameters than behavioral recovery rank beyond task saturation, while recovery rank grows sublinearly with width on MNIST.

  • Takeaways & Limitations

    Functional redundancy beyond task saturation is not effectively exposed by the one-shot weight- and neuron-level pruning criteria studied.

  • Takeaways & Limitations

    Whether these scaling regimes persist at scale requires broader architectures and datasets, and the study compares only two one-shot pruning baselines.

Abstract

from arXiv · show

A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.

1 Introduction

The paper asks how many parameter directions preserve a learned function and how much functional redundancy pruning exploits. It introduces behavioral recovery rank and finds that pruning retains more degrees of freedom beyond task saturation.

  • Motivation: Trained networks can contain many parameter directions that change the realized function little, creating functional redundancy relevant to compression.Retaining suitable subsets could reduce deployment memory, compute, and energy while approximately preserving performance.
  • Research questions: The paper asks how many parameter directions are required to preserve the learned function and how much of this redundancy pruning exploits.
  • Contributions: The behavioral recovery rank counts leading behavioral-Hessian eigendirections needed to recover model performance to a specified tolerance.It provides a theoretically grounded measure of approximate functional degeneracy.
  • Contributions: Controlled nonlinear teacher-student experiments use known task complexity to distinguish capacity-limited from task-saturated regimes.
  • Results: Behavioral recovery rank grows with student width and flattens once the student exceeds teacher width, while MNIST growth is sublinear.
  • Results: Structural and magnitude pruning retain parameters that continue growing past teacher width and exceed the behavioral recovery rank under loss-optimal ordering.This suggests redundancy beyond task saturation is not exposed by weight- or neuron-level structure and is only partially accessible to standard pruning.

2 Methodology

The method measures behavioral changes around a trained checkpoint and counts how many behaviorally important Hessian directions recover performance. It compares this basis-dependent rank with parameter counts retained by pruning.

  • Behavioral geometry: Behavioral loss compares model predictions with those of a reference checkpoint on a finite probe set, defining geometry tied to learned behavior.Unlike empirical loss, it does not compare predictions with ground-truth labels and acts as local self-distillation.
  • Behavioral geometry: The behavioral Hessian identifies perturbation directions that induce the largest local changes in model behavior.Its leading eigenspaces provide the behaviorally relevant directions used for reconstruction.
  • Recovery rank: Behavioral recovery rank is the smallest number of leading eigendirections whose projection recovers a chosen performance level within tolerance.
  • Recovery rank: A small behavioral recovery rank means learned behavior can be recovered from a low-dimensional part of the trained solution.If rank grows more slowly than parameter count, additional parameters mainly enlarge degeneracy.
  • Pruning baselines: Magnitude pruning removes globally smallest weights, while structural pruning greedily removes neurons; both report retained parameters needed to meet the same tolerance.The comparison uses coordinate-basis counts for pruning and behavioral-Hessian eigenbasis counts for recovery rank.
  • Experimental settings: The experiments vary student width in nonlinear teacher-student regression and use width-scaled MLPs on MNIST.Teacher width controls task complexity in the regression setting.

3 Results and Discussion

The behavioral recovery rank reveals how many behaviorally relevant directions are needed across capacity regimes and provides a benchmark for pruning-accessible redundancy. Results indicate that pruning retains more degrees of freedom than the behavioral basis, while the basis itself remains impractical for direct compression and broader validation is needed.

  • Q1: The task-saturated and capacity-limited regimes: The teacher-student experiments expose capacity-limited and task-saturated regimes as student width increases relative to fixed teacher complexity.Added parameters improve performance while students are narrower than the teacher; the recovery-rank growth slows after task saturation.
  • Q1: The task-saturated and capacity-limited regimes: On MNIST, accuracy and behavioral recovery rank both increase with width, but the rank grows sublinearly relative to the number of parameters.These models remain capacity-limited, with width adding behaviorally relevant directions more slowly than parameters.
  • Q2: Redundancy accessible to pruning: After task saturation, the behavioral recovery rank slows while structural and magnitude pruning retain increasingly more parameters.The one-shot pruning criteria therefore do not expose the additional redundancy identified by the behavioral benchmark.
  • Q2: Redundancy accessible to pruning: The loss-optimal ordering approximately halves recovery rank, yet both pruning baselines still retain more degrees of freedom at every width.This ordering minimizes quadratic behavioral loss for a fixed direction budget and indicates greater degeneracy than eigenvalue ordering alone.
  • Discussion: The behavioral eigenbasis is low-dimensional but impractical for direct compression because each basis element is as large as the dense model.The authors identify structured transforms and operational scaling as open problems, while output refitting recovers much of the pruning gap.
  • Limitations and future work: The experiments are limited to two small settings, and broader architectures, datasets, probe sets, reference points, and parametrizations are needed to test whether the regimes persist at scale.The study also compares only two one-shot pruning baselines, which may understate compression achievable with retraining or quantization.

C Experimental details

The experimental-details section consolidates the configurations used for synthetic regression and MNIST classification, including optimization and computational analysis procedures.

  • Experimental configuration: The experiments cover synthetic teacher-student regression and MNIST classification, with optimization settings and computational analysis described alongside each setup.The section points to dataset, architecture, training, and implementation details for both experiments.
  • Experimental configuration: The reported configurations include datasets, model architectures, training procedures, and implementation specifics.
  • Experimental configuration: The experimental setups are divided between Section C.1 and the optimization and computational methods in Section C.2.

C.1 Data, models, training

The study uses controlled two-layer ReLU teacher-student regression to vary capacity and width-scaled ReLU MLPs on MNIST classification, with fixed training configurations.

  • Synthetic regression: Synthetic regression uses two-layer ReLU MLPs in a teacher-student setup to systematically control model capacity.
  • Synthetic regression: The main regression models use teacher architecture 10 → 10 → 10 and student architecture 10 → d → 10, with noiseless data.Teacher and student networks include bias terms; alternative choices are ablated in Appendix G.
  • Synthetic regression: For each d ∈ {2, . . . , 20}, regression uses 2048 training examples, 512 test examples, and 4000 full-batch Adam steps at learning rate 10^-3.The task loss is mean-squared error, with no weight decay and θref = 0 for projection experiments.
  • MNIST classification: MNIST experiments train width-scaled two-hidden-layer ReLU MLPs on flattened, normalized 784-dimensional image vectors.Normalization uses the empirical mean and standard deviation of the selected training subset.
  • MNIST classification: MNIST training uses Adam for 2000 steps with learning rate 10^-3, mini-batch size 64, cross-entropy loss, and no weight decay.Projection experiments use θref = θ0 and a probe size of 10000.

C.2 Implementation

Implementation computes behavioral geometry through matrix-free eigensolves and evaluates recovery using ordered behavioral-Hessian directions, with the behavioral loss comparing model outputs on probes.

  • Implementation: The experiments use PyTorch on NVIDIA RTX 2080 Ti, TITAN RTX, or RTX A6000 GPUs.Reported MNIST experiments required approximately 150 GPU-hours, estimated across the heterogeneous environment.
  • Behavioral-Hessian eigensolves: The behavioral Hessian is accessed through a matrix-free dual operator using Jacobian-vector and vector-Jacobian products.Leading eigenspaces are estimated with symmetric Lanczos iteration and full orthogonalization.
  • Behavioral-Hessian eigensolves: Eigenpairs are retained when λj > 10^-6λ1, using a scale-invariant threshold across output scales.
  • Recovery-rank variants: Behavioral recovery directions are ordered by decreasing eigenvalue, while the loss-optimal statistic orders them by λj⟨vj, θ*−θref⟩^2.The loss-optimal ordering minimizes the quadratic behavioral loss for a fixed number of retained directions.
  • Behavioral loss and geometry: The behavioral loss compares probe-set predictions with a reference checkpoint and has the trained checkpoint as a global minimizer.At that checkpoint, the leading local object is the behavioral Hessian.
  • Behavioral loss and geometry: The behavioral Hessian is rank-limited by parameter dimension and probe-output dimension, leaving a nullspace of at least p − |P|m when |P|m < p.

E.2 Connection to the task loss Hessian

The behavioral Hessian is closely connected to the task-loss Hessian through a Jacobian Gram representation. On the training probe set with MSE loss, it becomes the task Hessian’s Gauss–Newton component and shares functional degeneracy structure with the evaluation Jacobian.

  • The task-loss Hessian decomposes into a generalized Gauss–Newton term and a curvature-residual term.
  • On the training probe set with MSE loss, the behavioral Hessian becomes the Gauss–Newton component of the task Hessian.
  • The behavioral Hessian is a Jacobian Gram matrix, Hbeh = J_P^⊤J_P, and therefore has the same kernel as the probe-set Jacobian.
  • Directions in the Jacobian kernel are infinitesimal functional degeneracies because they do not change the represented function to first order.
  • The behavioral Hessian and empirical NTK have identical nonzero eigenvalues and rank, while acting respectively on parameter and output perturbations.

E.5 Connection to Information Geometry

The behavioral Hessian admits an information-geometric interpretation as the Fisher information matrix of a local probe-induced statistical model. This connects behaviorally degenerate parameter directions with statistical nonidentifiability, while refitting experiments distinguish compensation effects from coordinate-basis effects.

  • The behavioral loss can be reinterpreted locally as a negative log-likelihood under a Gaussian observation model on probe-set outputs.
  • At the reference checkpoint, the Fisher information matrix equals the behavioral Hessian for this probe-induced model.
  • Behavioral degeneracy corresponds to statistical nonidentifiability under the probe distribution, and recovery rank counts informationally significant directions needed to recover performance.
  • The behavioral Hessian is a parameter-space metric, whereas the empirical NTK is a function-space metric linked through the evaluation map.
  • The local statistical model is not unique and is intended to capture local output deviations rather than describe a globally realistic data-generating process.
  • Refitting the output layer makes structure-pruning curves approximately flat, but both pruning baselines remain above the loss-optimal recovery rank.

G Further experimental results

This appendix provides additional experimental results omitted from the main paper for brevity.

  • The appendix contains further experimental results omitted from the main paper for brevity.

G.1 Teacher-student Experiments

Additional experiments test whether recovery-rank behavior is robust to reconstruction choices and architectural changes. The observed trends persist across probe sets, reference points, model dimensions, and training-data noise, while the behavioral Hessian retains concentrated spectral structure.

  • Changes of probe set and reference point: Recovery behavior remains qualitatively stable when the probe set and reference point are changed.
  • Changes of probe set and reference point: For θref = θ0, recovery-rank curves flatten less above teacher width but still significantly flatten.
  • Changes of model architecture: Architectural changes to input size, output size, both, and training-data noise are used to test robustness of the teacher-student findings.
  • Changes of probe set and reference point: The task-saturated regime shows greater recovery-rank growth for θref = θ0 than for θref = 0, while still flattening more than the capacity-limited regime.
  • Changes of model architecture: The experiments support robustness of the findings across different architectural parameters, with a slight double descent visible in the noisy setting.
  • Spectral structure: Leading behavioral-Hessian eigenvalues remain separated from a broad tail, while a limited subset of eigenmodes captures most spectral mass as parameter count grows.

G.2 MNIST

Across the MNIST experiments, performance exceeds 95% for all considered widths, while projections onto leading eigendirections affect training behavior and the behavioral spectrum remains concentrated.

  • Final test accuracy exceeds 95% for all considered MLP widths.
  • Figure 13 examines how projecting onto leading eigendirections changes MNIST training loss and accuracy.
  • The behavioral spectrum is concentrated, indicating that a restricted subset of local parameter directions dominates functional recovery.
  • The experiments test robustness by changing input size, output size, both dimensions, and adding training-data noise.
  • Figure 10 compares final train and test MSE with and without noise, while Figure 11 reports spectral quantities for the teacher-student experiments.
Loading 2608.30741v1…