Source-linked AI summary
Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression
Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov
TL;DR
Large language models make full Hessian or Fisher analysis infeasible, especially when existing practical methods discard cross-layer structure. The paper introduces a scalable Kronecker-based Fisher approximation that preserves such interactions, and finds consistent value-projection sensitivity patterns whose estimates correlate with degradation and recovery across compression experiments.
Problem
Full Hessian and Fisher analysis is infeasible for large models, while practical estimators often neglect cross-layer interactions that may be structurally meaningful.
Method
The paper uses a Kronecker-factorized empirical Fisher with truncated terms, an exact diagonal, and matrix-free computation to analyze large language models.
Results
Across four model families, value projection layers show the highest sensitivity and strongest cross-layer correlations, while the approximation correlates with degradation and recovery across compression and fine-tuning experiments.
Takeaways & Limitations
The framework supports identifying fragile layers for mixed-precision compression, targeted fine-tuning, and LoRA adaptation without full Hessian computation.
Takeaways & Limitations
The Fisher approximation can disagree with empirical effects for O-projection pairs, possibly because its local gradients miss finite perturbation behavior involving residual-stream outlier channels.
Abstract
from arXiv · showhide
In this paper, we propose a scalable Kronecker-based approximation that captures cross-layer interactions without storing the entire Fisher matrix, enabling practical Hessian analysis for billion-parameter networks where full computation is infeasible. Our approach reveals consistent vulnerability patterns: value projection layers exhibit the highest sensitivity and strongest cross-layer correlations across multiple model families, while other components exhibit architecture-specific behaviors. Through extensive experiments on quantization, sparsification, inter-layer corruption, and post-corruption fine-tuning, we demonstrate that our approximation strongly correlates with both performance degradation and recovery. Our framework provides a practical, theoretically grounded tool for identifying fragile components in large models, opening new avenues for guided compression and optimization strategies, such as mixed-precision allocation, layer-wise sparsity, and adaptive low-rank decomposition across layers and even individual weight groups.
1 Introduction
The paper addresses the lack of scalable curvature estimators that preserve cross-layer interactions in large models. It proposes a Kronecker-based Fisher approximation and evaluates its usefulness for identifying compression-sensitive structure.
- Motivation: Large-model Hessian and Fisher analysis is difficult, leading practical methods to neglect cross-layer interactions or use diagonal and block-diagonal approximations.Existing full-structure analyses are restricted to small models or datasets, motivating principled scalable approximations.
- Approach: The proposed approximation uses a Kronecker factorization framework to analyze Fisher structure at scale.
- Approach: The method reduces memory complexity from quadratic to linear in model size while retaining richer structure than diagonal and block-diagonal methods.It combines truncated Kronecker terms, an exact diagonal, and a matrix-free eigensolver.
- Evidence: Experiments validate non-diagonal Hessian structure in four LLMs ranging from 350M to 7B parameters.
- Evidence: The approximation correlates with layer sensitivity to quantization and sparsification and captures non-additive joint effects and recovery under fine-tuning.
2 Related Work
Prior scalable curvature methods simplify structure through layer-local, diagonal, or block-diagonal estimates, while theory and empirical evidence indicate meaningful cross-layer curvature. This paper extends Kronecker-based Fisher estimation to capture those interactions at billion-parameter scale and across compression tasks.
- Existing curvature methods: Full Hessian or Fisher matrices are quadratic in parameter count, so scalable methods avoid materializing them or impose diagonal, block-diagonal, or low-rank structure.
- Cross-layer structure: Transformer Hessians exhibit heterogeneous parameter-block structure, and near-block-diagonal behavior does not imply genuinely vanishing cross-layer terms.
- Compression applications: Curvature-guided compression uses second-order sensitivity to guide pruning, quantization, mixed-precision allocation, and low-rank decomposition.
- Paper positioning: Unlike per-layer estimators such as GFWSVD, the proposed method estimates inter-layer curvature and transfers across quantization, sparsification, and fine-tuning recovery.
3 Hessian Kronecker Approximation
The method approximates the Hessian with an empirical Fisher matrix, decomposes it into Kronecker terms, truncates the decomposition, and restores its exact diagonal. Matrix-free products make the construction practical, with linear memory in parameter count for fixed rank.
- Formulation: The Hessian is approximated by the empirical Fisher, defined as the expected outer product of sample gradients when weights lie near a local optimum.The paper notes this assumption is typically satisfied by converged pre-trained models.
- Kronecker decomposition: The Fisher is vectorized and rewritten through Kronecker products after reshaping the gradient into a matrix.
- Kronecker decomposition: An exact Kronecker decomposition is obtained from the singular value decomposition of the expected reshaped-gradient Kronecker product.
- Low-rank approximation: Truncating to the largest r singular values yields the best rank-r approximation in Frobenius norm.
- Matrix-free computation: The large decomposition matrix is never formed; a matrix-free Arnoldi eigensolver obtains leading singular components through vector products and reshaped-gradient multiplications.
- Diagonal correction: The approximation substitutes the exact Fisher diagonal into the low-rank decomposition, improving fidelity over the truncated terms alone.
- Compressed representation: The compressed visualization averages Fisher blocks into an n × n image, while its Kronecker factors require O(r(n^2 + m^2)) memory.
- Complexity: For fixed rank r, the method’s memory footprint is O(rd), linear in parameter count, instead of O(d^2) for explicitly storing the Fisher matrix.
4 Experiments
Experiments validate the Kronecker-Fisher approximation against an exact Hessian and test whether its diagonal and off-diagonal structure predicts compression sensitivity, inter-layer damage, and recovery across four language models.
- 4.1 Validating the Hessian Approximation: On a two-layer perceptron, the rank-16 approximation recovers the structure of the true Hessian.The approximation achieves R2(H, J) of 42.3% with explicit diagonal computation versus 29.9% without it.
- 4.1 Validating the Hessian Approximation: The approximation is evaluated on OPT-350M, Qwen2-0.5B, OLMo2-1B, and Qwen2.5-7B using WikiText2, with visualizations focused on individual Transformer blocks.The full Hessian is too large to display directly, so the experiments use compressed representations over batches and groups of Transformer blocks.
- 4.2 Compression within a Single Layer Type: The V-projection is the most vulnerable component across compression schemes, except that OLMo2 ranks its downscale projection as most vulnerable under quantization.Under 50% sparsification, V-projections are most sensitive in every model; OLMo2 reverses the V/downscale order between quantization and sparsification.
- 4.2 Compression within a Single Layer Type: Layers with high Fisher values consistently correspond to components whose 4-bit quantization or 50% sparsification causes the largest perplexity increases.Architecture-specific sensitive layers include OPT upscale, Qwen K, and OLMo2 V, Q, and downscale projections.
- 4.3 Inter-Layer Quantization and Sparsification: Pairwise corruption produces the largest perplexity increases for V-involving pairs and upscale-downscale pairs, matching the approximation’s highlighted off-diagonal structure.Reported examples are OPT V–FC1, Qwen2.5 V–upscale, and OLMo2 V–downscale.
- 4.3 Inter-Layer Quantization and Sparsification: The interaction difference confirms that prominent layer pairs exhibit excess joint degradation beyond the sum of their individual effects.The largest interaction differences occur for attention–MLP pairs involving the V-projection, rather than reflecting only cumulative independent errors.
- 4.3 Inter-Layer Quantization and Sparsification: The O-projection is an exception: it has low approximated Hessian values but forms notably damaging pairs under both quantization and sparsification.The paper hypothesizes that local Fisher curvature misses finite perturbations affecting high-magnitude residual-stream outlier channels.
- 4.3 Inter-Layer Quantization and Sparsification: Fine-tuning or applying LoRA to the V-projection recovers more performance after FFN corruption than adapting the Q- or K-projections.This supports using high off-diagonal Hessian values relative to corrupted layers to select recovery targets.
5 Conclusion
The Kronecker-based Fisher approximation makes Hessian analysis practical for billion-parameter language models while preserving cross-layer interactions. Across model families, it identifies recurring sensitivity patterns and supports compression and optimization decisions, while motivating further study of outlier-mediated interactions.
- The approximation reduces memory complexity from quadratic to linear while preserving cross-layer interaction information for billion-parameter language models.
- Value projection layers consistently show the highest sensitivity and strongest cross-layer correlations across four model families, whereas other components behave architecture-specifically.
- Experiments across quantization, sparsification, inter-layer corruption, and fine-tuning show strong correlations between approximation values, performance degradation, and recovery.
- The framework supports identifying fragile layers for mixed-precision compression, targeted fine-tuning, and LoRA adaptation without full Hessian computation.
- Future work will test whether outlier channels mediate strong interactions such as V–upscale pairs and explain the O-projection discrepancy.
A Additional Hessian Visualizations
Additional visualizations show that Hessian values and inter-layer correlations are stronger in earlier and middle Transformer blocks than in final blocks. A complete OPT-125M view also exposes repeated block structure and cross-block correlations.
- Earlier and middle Transformer block groups contain more pronounced high-value Hessian regions and inter-layer correlations than the final group.
- The full OPT-125M approximation reveals repeated diagonal block structure and correlations between layers in different Transformer blocks.
- For larger models, block-level and multi-block visualizations preserve distinguishable layer boundaries that full-model plots would obscure.
B Pairwise Quantization with Concrete Values
Annotated pairwise-quantization heatmaps place measured perplexity increases directly in each cell, enabling quantitative comparison of layer pairs alongside the overall interaction pattern.
- Each heatmap cell reports the measured perplexity increase for jointly quantizing a particular layer pair.
- Pairs involving the V-projection and pairs combining upscale and downscale layers produce some of the largest perplexity increases.
C.1 Zero-Shot Results on Downstream Tasks
The downstream evaluation tests whether WikiText-2 sensitivity patterns transfer to zero-shot task accuracy under layer-type-wise 90% sparsification. The principal rankings largely persist, but the most sensitive component varies by model architecture.
- Evaluation setup: 90% sparsification is applied to one layer type at a time in middle Transformer blocks and evaluated without task-specific fine-tuning on five zero-shot tasks.
- Results: The downstream results recover the principal layer-sensitivity patterns previously observed through perplexity.
- Results: FC1 is most damaging in OPT-350M, V-projection in Qwen2-0.5B, upscale and downscale projections in OLMo2-1B, and gate projection in Qwen2.5-7B.
- Results: Sensitivity magnitudes vary by task, with HellaSwag and ARC-Easy often showing the largest accuracy drops, while rankings remain consistent across criteria.
- Visual context: OPT-350M visualizations show higher Hessian values in earlier blocks than later blocks, while the annotated quantization heatmap supports pairwise comparison of concrete perplexity increases.
- Table interpretation: Table 3 reports accuracy decreases relative to dense baselines in percentage points, with larger positive values indicating worse performance and bold averages marking the largest decrease.