WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

• Large neural networks create deployment costs, motivating compression methods rooted in the Optimal Brain Damage/Surgeon framework.Source: §1 · p. 1
• Existing second-order approximations are computationally coarse, and their quality and scalability for neural networks remain poorly understood.Source: §1 · pp. 1–2
• WoodFisher approximates the Hessian through empirical-Fisher structure and uses the Woodbury identity for iteratively improving, efficient inverse-Hessian-vector products.Source: §1 · p. 2
• WoodFisher achieves state-of-the-art compression on models including RESNET-50 [4] and MOBILENET [27] on ILSVRC [28].Source: §1 · p. 2
• WoodFisher outperforms prior methods for one-shot pruning without retraining and matches or exceeds state-of-the-art approaches under gradual pruning.Source: §1 · p. 2
• Its extensions provide global parameter-importance measures, automatic layer-wise sparsity allocation, limited-data compression, and gradient-aware pruning of unconverged models.Source: §1 · p. 2

• The training objective averages a loss over input-output examples and learns a parameterized function whose predictions approximate observed outputs.Source: §2 · p. 2
• The Hessian captures local loss geometry and supports a quadratic approximation around the current parameters.Source: §2 · p. 2
• The Fisher matrix is defined from the model’s conditional output distribution in the probabilistic formulation.Source: §2 · pp. 2–3
• When model and data conditional distributions match, the Fisher and Hessian are equivalent.Source: §2 · p. 3
• The empirical Fisher replaces the model distribution with the empirical training distribution, yielding a more practical approximation.Source: §2 · p. 3
• The paper distinguishes true Fisher from empirical Fisher and directs readers to prior work for further exposition.Source: §2 · p. 3

3 Efficient Estimates of Inverse-Hessian Vector Products

• Inverse-Hessian-vector products are useful across optimization and machine learning, but direct Hessian storage and computation are prohibitive.Source: §3 · p. 3
• The empirical Fisher trades the true Fisher’s convergence guarantee for substantially lower computational cost, motivating empirical evaluation in neural networks.Source: §3 · p. 3

3.1 The (Empirical) Fisher and the Hessian: A Visual Tour

• Hessian and empirical-Fisher blocks show similar structure across layers, including before full convergence, although their main difference is scale.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 The WoodFisher Approximation

• WoodFisher uses the Woodbury identity to update inverse empirical-Fisher estimates, with damping making the initial matrix positive definite.Source: §3.2 · pp. 4–5
• The WoodFisher local quadratic model accurately predicts actual training loss along three 50%-sparsity pruning directions, with damping controlling conservativeness.Source: §3.2 · p. 5, Figure 2 · p. 5
• Block-wise approximation reduces inversion cost to O(mcd), making computation linear in parameter dimension d under fixed block size c.Source: §3.2 · p. 5

3.3 Context and Alternative Methods

• Diagonal approximations can be inaccurate, whereas WoodFisher provides better IHVP estimates at an additional constant-factor cost; K-FAC also has architectural and approximation limitations.Source: §3.3 · pp. 5–6
• WoodFisher extends earlier empirical-Fisher ideas to modern network sizes and can outperform K-FAC even for fully connected networks.Source: §3.3 · p. 6

4 Model Compression

• Pruning methods are organized into second-order, iterative, dynamic, and variational or regularization-based classes; WoodFisher belongs to the second-order class and can combine with iterative or dynamic methods.Source: §4 · p. 6
• Optimal Brain Damage minimizes the local quadratic loss increase caused by setting selected weights to zero, typically omitting the gradient term at a local optimum.Source: §4 · p. 6

4.1 Removing a single parameter wq

• For one parameter, the constrained perturbation fixes that weight to zero and the pruning statistic ranks candidates by their estimated loss increase.Source: §4.1 · p. 7

4.2 Removing multiple parameters at once

• Exact selection of many parameters becomes exponentially difficult, so practical pruning requires an approximation rather than solving the full constrained problem.Source: §4.2 · p. 8
• Multi-parameter interactions make the dual problem nonseparable whenever relevant off-diagonal inverse-Hessian terms are nonzero.Source: §4.2 · p. 8

4.3 Discussion

• WoodFisher forms a pruning direction by sorting parameters by pruning statistic and summing optimal updates for the selected removals.Source: §4.3 · p. 8
• Diagonal Hessians recover Optimal Brain Damage, while isotropic Hessians recover global magnitude pruning.Source: §4.3 · p. 8
• Global WoodFisher jointly selects parameters across layers, automatically allocating sparsity without layer sensitivity analysis or heuristics such as skipping boundary layers.Source: §4.3 · p. 9

5.1 One-shot pruning

• In one-shot ResNet-20 pruning, WoodFisher-joint beats global magnitude pruning by about 5% and 10% test accuracy at 70% and 80% sparsity, respectively.Both joint and independent variants outperform magnitude and diagonal-Fisher pruning; diagonal-Fisher becomes worse than magnitude pruning above 30% sparsity.Source: §5.1 · p. 9, Figure 3 · p. 9
• Larger WoodFisher chunks improve performance monotonically, while Hessian-inverse recomputation further improves one-shot ResNet-50 pruning without retraining.The chunk-size trend supports more accurate inverse estimates, and recomputation addresses the local quadratic model’s limited trust region.Source: §5.1 · pp. 9–10, Figure 4 · p. 10
• WoodFisher also outperforms L-OBS at every reported sparsity level, with independent and joint variants improving test accuracy by up to approximately 3.5% and 20% at 65% sparsity.The comparison uses Top-5 accuracy for one-shot ResNet-50 pruning on ImageNet.Source: §5.1 · p. 10, Figure 5 · p. 11

5.2 Gradual Pruning

• WoodFisher outperforms all baselines across gradual and dynamic pruning for ResNet-50, with Top-1 gains of approximately 1%, 1.5%, and 3.3% over STR at 90%, 95%, and 98% sparsity.The comparison is reported in Table 1 for the specified sparsity regimes.Source: §5.2 · p. 11, Table 1 · p. 12

ADDITIONAL COMPARISONS

• WoodFisher outperforms state-of-the-art methods across all considered sparsity regimes for both ResNet-50 and MobileNetV1, establishing the reported unstructured-pruning benchmark result.The summary covers the additional comparisons, including statistically significant gains over Global Magnitude on MobileNetV1.Source: Table 4 · p. 13, ADDITIONAL COMPARISONS · p. 14, Table 5 · p. 14

5.3 What goes on during gradual pruning?

• During gradual pruning, WoodFisher produces better pruning directions after almost every step, and global magnitude pruning fails to catch up despite substantial intervening retraining.Magnitude pruning performs worse in the corresponding stepwise comparison.Source: §5.3 · p. 14, Figure 6 · p. 14

5.4 Actual inference speed-up

• At approximately 90% global sparsity, WF-Joint gains 0.95% Top-1 accuracy over STR for a 1.2 ms batch-1 latency difference, while WF-Independent offers a similar accuracy improvement with a 0.5 ms difference.All evaluated methods speed up inference relative to the dense baseline, and the standard WoodFisher variants trade higher accuracy for higher inference time than STR.Source: §5.4 · p. 15, Table 6 · p. 15

6.1 WoodFisher for structured pruning

• Structured pruning extends the pruning statistic from individual parameters to parameter groups, with a correlation-aware formulation intended for blocks or filters.The group indicator is represented by a one-hot vector, and the formulation is obtained by combining multiple removal constraints.Source: §6.1 · pp. 15–16
• The structured-pruning extension remains incompletely evaluated, with a detailed empirical investigation left for future work.Source: §6.1 · p. 16

6.2 WoodTaylor: Pruning at a general point

• WoodTaylor incorporates the first-order gradient term into the pruning analysis, making the method applicable when gradients are not close to zero.This targets pruning during training or models that are far from an optimum.Source: §6.2 · pp. 16–17
• WoodTaylor improves over magnitude and diagonal-Fisher pruning, even exceeding the initial dense model’s accuracy by about 5% up to roughly 90% sparsity.The experiment uses one-shot compression of a partially trained MLPNET on MNIST, with 8000 Fisher samples and dampening 1e-1.Source: §6.2 · p. 17, Figure 7 · p. 17
• On pre-trained RESNET-20 on CIFAR-10, WoodTaylor can also outperform WoodFisher because the gradient is usually not exactly zero.Source: §6.2 · p. 17

6.3 Sampled Fisher: The case of unlabelled data

• Sampled WoodFisher computes gradients using samples from the model distribution rather than labels, enabling unsupervised compression; on RESNET-50, it performs similarly to empirical WoodFisher with slightly higher variance.The comparison concerns one-shot pruning on ImageNet, and the authors conclude that sampled WoodFisher is usable when labels are unavailable.Source: §6.3 · p. 18, Table 7 · p. 18

7 Discussion

• Future applications include structured pruning, compression of Transformer models, and efficient inverse-Hessian-vector products for influence functions.Source: §7 · p. 18
• WoodFisher extends foundational neural-network pruning work to large-scale settings and yields state-of-the-art results.Source: §7 · p. 18

S1.1 Pruning schedule

• The overall procedure runs for 100 epochs with retraining between pruning steps and afterward, using SGD with learning rate 0.005, momentum 0.9, and weight decay 0.0001.Source: S1.1 Pruning schedule · p. 23
• ResNet-50 is pruned every five epochs from epoch 1 through epoch 40, followed by exponential learning-rate decay from epochs 40 to 90.Source: S1.1 Pruning schedule · p. 23
• MobileNetV1 is pruned every four epochs from epoch 4 through epoch 24, followed by exponential learning-rate decay from epoch 30 through epoch 100.Source: S1.1 Pruning schedule · p. 23
• The pruning amount at each step follows a polynomial schedule with initial sparsity set to 0.05.Source: S1.1 Pruning schedule · p. 23
• WoodFisher, Global Magnitude, and Magnitude use the same gradual-pruning schedule, with the pruner replaced inside a procedure originally designed for magnitude pruning.Source: S1.1 Pruning schedule · p. 23

S1.2 WoodFisher Hyperparameters

• Fisher subsample size counts empirical-Fisher outer products, while Fisher mini-batch size controls gradient averaging before each outer product to reduce computation.Source: S1.2 WoodFisher Hyperparameters · p. 23
• The Hessian and inverse-Hessian approximations use diagonal blocks of size 2000 for ResNet-50 and 10,000 for MobileNetV1.Source: S1.2 WoodFisher Hyperparameters · p. 23
• The hyperparameters for the gradual-pruning results are documented in Table S1, with a thorough ablation study reported separately.Source: Table S1 · p. 23, S1.2 WoodFisher Hyperparameters · p. 23
• Batch sizes of 256 or 180 depend on available GPU memory, while each Global Magnitude comparison uses the same batch size across sparsity levels.Source: S1.2 WoodFisher Hyperparameters · p. 23
• WoodFisher uses dampening λ = 1e-5 in all experiments to make the empirical Fisher positive definite.Source: S1.2 WoodFisher Hyperparameters · p. 23

• The additional pruning time is limited to roughly 6–8 steps and is small relative to the 100 training epochs in gradual ImageNet pruning.Source: S1.3 Run time costs for WoodFisher… · p. 24
• Pruning-step time for ResNet-50 ranges from approximately 15 to 47 minutes as Fisher subsample sizes and chunk size increase.The reported configurations span Fisher subsample sizes of 80 to 160, Fisher sample sizes of 400 to 800, and chunk sizes of 1000 to 2000.Source: S1.3 Run time costs for WoodFisher… · p. 24
• After a threshold, larger Fisher and chunk-size hyperparameters provide relatively small gains, enabling an accuracy–runtime trade-off.Source: S1.3 Run time costs for WoodFisher… · p. 24
• Increasing WoodFisher pruning cost is worthwhile because pruning occurs once, while the resulting model runs numerous times with extra test accuracy.Source: S1.3 Run time costs for WoodFisher… · p. 24
• The current sequential inverse-Hessian computation across blocks or chunks could be parallelized to obtain further speedups.Source: S1.3 Run time costs for WoodFisher… · p. 24

S2.1 K-FAC

• K-FAC enables efficient Fisher inversion through Kronecker factors, but its expectation-factorization approximation can be inaccurate and requires further approximations for convolutional, recurrent, and large networks.WoodFisher is presented as an alternative using the empirical Fisher with a Woodbury-based inverse.Source: S2.1 K-FAC · pp. 24–25

S2.2 Other methods

• Other inverse-Hessian approaches trade accuracy or scalability for simplicity: exact Hessians are cubic-time, diagonal estimates are less accurate, Hessian-free solvers may require many iterations, and Neumann methods require spectral scaling.Woodbury-based OBS extensions are more specialized, whereas WoodFisher is described as more general and experimentally better at scale.Source: S2.2 Other methods · pp. 25–26
• At 65% sparsity on ImageNet ResNet-50, independent and joint WoodFisher outperform L-OBS by up to ∼3.5% and 20% top-5 accuracy, respectively.Both WoodFisher variants outperform L-OBS at all reported sparsity levels, with joint WoodFisher also automatically adjusting sparsity distributions.Source: S2.2 Other methods · p. 26, Figure S1 · p. 26

S3.1 All figures for Hessian and empirical Fisher comparison on CIFARNET

• The empirical Fisher shares Hessian structure in both diagonal and off-diagonal CIFARNET blocks, supporting its use as a Hessian approximation.The comparison used a 3072 →16 →64 →10 CIFAR10 network and batches of 100 examples; the study focused on matrix structure rather than test accuracy.Source: S3.1 All figures for Hessian and… · pp. 27–28, Figure S2 · p. 27, Figure S3 · p. 28

S3.2 Across different stages of training

• The empirical Fisher retains Hessian-like structure throughout training, including as early as 0.5 epochs, even when model and data distributions theoretically do not match.MNIST experiments found substantial similarity in 200 × 200 last-layer submatrices using 16 samples, with similar trends for larger sample counts.Source: S3.2 Across different stages of training · pp. 28–29, Figure S4 · p. 29

S4.1 One-shot Pruning

• For one-shot pruning on ImageNet, WoodFisher outperforms Global Magnitude and Magnitude on ResNet-50, including with as few as 8,000 data samples, and also outperforms both on MobileNetV1.Increasing the fisher subsample size from 80 to 240 helps a bit. Recomputing the Hessian inverse estimate during pruning significantly improves performance because the local quadratic model is valid only within a small trust-region.Source: S4.1 One-shot Pruning · p. 30, Figure S5 · p. 30

S5 Gradual pruning scores at best runs

• Across best runs and pruning scenarios, WoodFisher remains at least as strong as Global Magnitude or Magnitude pruning, including fixed layer-wise sparsity targets on ResNet-50.Tables S4 and S5 report these best-run comparisons and the targets used for the ResNet-50 comparison.Source: S5 Gradual pruning scores at best runs · p. 36, Table S4 · p. 36, Table S5 · p. 36

S6 FLOPs and Inference Costs

• FLOPs-aware pruning normalizes each parameter’s pruning statistic by its layer’s FLOP cost, controlling the FLOPs–accuracy trade-off without manually designing sparsity distributions.The parameter β controls FLOP influence, while the remainder of the pruning procedure stays unchanged and costs are measured on active parameters during gradual pruning.Source: S6 FLOPs and Inference Costs · p. 37
• STR produces more uniform layer-wise sparsity and lower theoretical FLOP costs than standard WoodFisher or Global Magnitude, despite worse pruned-model accuracy.The methods can redistribute sparsity differently across layers, while practical sparsity profiles correlate with layer parameter counts.Source: S6 FLOPs and Inference Costs · p. 37
• With FLOPs-aware pruning, WoodFisher achieves higher accuracy and lower FLOP costs than STR for both ResNet-50 and MobileNetV1, including an ≈1% accuracy gain at 101 M FLOPs on MobileNetV1.The comparison uses β = 0.30 for ResNet-50 and β = 0.35 for MobileNetV1.Source: S6 FLOPs and Inference Costs · p. 37
• FLOPs-aware WoodFisher yields a 20% relative inference-time improvement over STR at batch size 64.The result is reported for the FLOPs-aware statistic with β = 0.30 on the CPU inference benchmark.Source: S6 FLOPs and Inference Costs · p. 38, Table S7 · p. 38

S7.1 ResNet-50

• The supplement reports WoodFisher and Global Magnitude’s layer-wise sparsity distributions for ResNet-50 at 80%, 90%, 95%, and 98% sparsity.The reported distributions cover ImageNet sparsification settings for both methods.Source: Table S8 · p. 39

S7.2 MobileNetV1

• A corresponding layer-wise sparsity comparison for MobileNetV1 reports WoodFisher and Global Magnitude at 75% and 89% sparsity.These distributions are reported for ImageNet sparsification settings.Source: Table S9 · p. 40

S8 FLOPs-aware sparsity distributions

• FLOPs-aware sparsity distributions are reported for WoodFisher on ResNet-50 at 90% sparsity and MobileNetV1 at 75.28% sparsity across the listed β settings.The reported β values are 0.00 and 0.30 for ResNet-50, and 0.00, 0.30, and 0.35 for MobileNetV1.Source: Table S10 · p. 41, Table S11 · p. 42

S9.1 Pre-trained model

• On full-matrix ResNet-20 pruning, WoodTaylor with damp=1e−3 outperforms WoodFisher across almost all target sparsity levels.The comparison prunes three convolutional layers in a CIFAR-10 pretrained model without block-wise assumptions.Source: S9.1 Pre-trained model · p. 43, Figure S13 · p. 43, Figure S14 · p. 43
• WoodTaylor requires dampening tuning because its gradient–inverse-Hessian interaction makes performance more sensitive to hyperparameters, whereas WoodFisher is hyperparameter-free.With damp=1e−5, the WoodTaylor weight update has about 10× the original weight norm; these experiments use 50,000 samples.Source: S9.1 Pre-trained model · p. 43