PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Approximation: Fisher-Weighted SVD Compression for ViTs

Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann

arXiv:2609.07155v1cs.CVcs.AIcs.LG

TL;DR

Fisher approximation fidelity is a poor predictor of post-compression accuracy for ViTs. The paper proposes FACTS, which preserves token-local aggregation and within-token activation-gradient dependence, alongside CoRS for layer-wise rank allocation under a FLOP constraint. Across ViTs and hybrid architectures, the methods improve accuracy-efficiency trade-offs, with up to +5.8 p.p. Top-1 on Swin-B over the strongest SVD baseline.

  • Problem

    For ViT Fisher-weighted SVD, higher fidelity to the empirical FIM does not guarantee higher post-compression accuracy, despite the need for approximations because the full FIM is prohibitively expensive.

    Source: §1 · p. 2

  • Method

    FACTS uses token-local aggregation while preserving within-token activation-gradient dependence, and CoRS searches layer-wise rank allocations with a MILP under a FLOP budget.

    Source: §1 · p. 2

  • Results

    FACTS improves accuracy over strong SVD baselines across ViTs and hybrid architectures, with up to +5.8 p.p. Top-1 on Swin-B and +9.0 p.p. with CoRS.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Together, FACTS and CoRS achieve strong accuracy retention with practical inference speedups under fixed FLOP budgets across the evaluated architectures.

    Source: §6 · p. 9

    Theoretical understanding remains incomplete of when and why cross-token mixed moments misalign with compression-induced perturbations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.

1. Introduction

The paper targets loss-aware SVD compression for ViTs, where better empirical-FIM approximation does not reliably predict post-compression accuracy. It proposes FACTS and CoRS, and reports improved accuracy over strong SVD baselines.

1. Introduction

  • The paper focuses on parameter reduction amid deployment demands from growing models, distinguishing this focus from quantization and architecture search.It describes SVD compression as reducing parameters while retaining dense matrix multiplications and avoiding complex architectural changes.Source: §1 · p. 1
  • Fisher-weighted SVD incorporates the FIM to make compression sensitive to how parameter perturbations affect model predictions.Direct FIM computation is prohibitively expensive, so Fisher-weighted SVD relies on approximations.Source: §1 · pp. 1–2
  • FACTS uses token-local aggregation while preserving activation-gradient dependence to improve post-compression accuracy in Fisher-weighted SVD.The paper finds that higher cosine similarity to the empirical FIM is not aligned with post-compression accuracy.Source: §1 · p. 2
  • CoRS uses a MILP-based search to find proxy-optimal layer-wise rank allocations, outperforming heuristic or iterative methods.The search is framed as global layer-wise budget allocation based on interpolated sensitivity measurements.Source: §1 · p. 2
  • Up to +5.8 p.p. Top-1 on Swin-B over the strongest SVD baseline, rising to +9.0 p.p. with CoRS, with gains extending across ViTs and hybrid architectures.The evaluations report improvements across downstream tasks and model sizes as well.Source: §1 · p. 2

2. Related Work

Related work spans weight reconstruction, feature reconstruction, and loss-aware SVD, while rank-allocation strategies range from learned searches to heuristics. The paper positions FACTS and CoRS as responses to limitations in Fisher approximation and rank selection.

2. Related Work

  • Loss-aware SVD weights the objective by loss sensitivity, but improving empirical-FIM approximation fidelity alone does not ensure strong post-compression accuracy.GFWSVD fits Kronecker factors to the empirical FIM, while the paper's analysis identifies a mismatch between fidelity and compression accuracy.Source: §2 · p. 2
  • Existing rank allocation includes learned rank choices, NAS, greedy sensitivity pruning, and equal-error heuristics, but these strategies can be costly, hyperparameter-sensitive, or lack global error guarantees.FLORA's supernet approach is described as computationally expensive and sensitive to hyperparameters; equal-error heuristics lack guarantees of globally minimizing reconstruction error.Source: §2 · pp. 2–3
  • CoRS formulates rank selection as a MILP that globally minimizes total proxy error under a hard FLOP budget.FACTS is presented as a factorization tailored to ViT compression to retain higher post-compression accuracy.Source: §2 · pp. 2–3

3. Preliminaries

The preliminaries establish Fisher-weighted SVD and the Kronecker structure of the Fisher for weight-sharing layers. They provide the basis for understanding how compression objectives and token structure shape the approximations used by the paper.

3.1. Generalized Fisher-Weighted SVD Compression

  • Fisher-weighted SVD minimizes an expected second-order loss approximation by whitening weights with Kronecker factors, truncating the SVD, and projecting the result back.The rank-k approximation is split into two consecutive linear layers, reducing parameter count when the rank is sufficiently low.Source: §3.1 · p. 3

3.2. Kronecker Structure of the Fisher for Weight-Sharing Layers

  • Weight sharing makes the layer gradient a sum of token contributions, so its FIM contains token-pair Kronecker terms.Estimators differ in whether they aggregate cross-token moments and whether they preserve activation-gradient coupling.Source: §3.2 · p. 3
  • A single Kronecker product approximates the full FIM, and existing estimators differ in the structural choices used to obtain its factors.These choices include token aggregation and assumptions about activation-gradient dependence.Source: §3.2 · p. 3

4. Methodology

The methodology tests how Kronecker Fisher structure relates to Fisher-weighted SVD accuracy, then builds FACTS around token-local, coupled statistics and allocates ranks under a FLOP budget.

4.1. Kronecker Fisher Approximations for SVD Compression

  • Shampoo2 and KFAC-reduce have higher empirical-FIM cosine similarity than KFAC-expand (0.17/0.13 vs. 0.12), but KFAC-reduce accuracy is 67.0% vs. 75.9% for KFAC-expand.Only Shampoo2 improves accuracy despite its higher similarity.Source: §4.1 · p. 4
  • Cross-token mixed moments reflect stable spatial relationships among vision tokens and can raise Fisher similarity without selecting low-rank directions that best preserve compression accuracy.KFAC-reduce represents these moments, whereas KFAC-expand discards them; KFAC-reduce has lower post-compression accuracy in the reported comparison.Source: §4.1 · p. 4
  • Shampoo2 outperforms KFAC-expand (77.2% vs. 75.9%), consistent with retaining within-token activation-gradient dependence that KFAC-style factorization decouples.The investigated layer shows substantial activation-gradient dependence.Source: §4.1 · p. 4

4.2. FACTS

  • FACTS removes cross-token mixed moments while retaining within-token activation-gradient coupling, using the Zero Cross-Moment prior as a compression-oriented approximation rather than token independence.Its token-wise structured approximation uses token-weighted activation and gradient statistics.Source: §4.2 · pp. 4–5

4.3. Constrained Rank Search (CoRS)

  • CoRS profiles each layer’s reconstruction error and FLOP cost, then densifies sparse profiles with rolling-window cubic interpolation.The reconstruction error is measured as KL divergence between the original model output and the output with that layer compressed.Source: §4.3 · p. 6
  • CoRS formulates rank allocation as a MILP that minimizes total proxy error under a hard FLOP budget, with exactly one rank choice active per layer.The resulting solution is proxy-optimal under the stated formulation.Source: §4.3 · p. 6

5. Evaluation

The evaluation tests FACTS and CoRS across vision models, compression methods, rank-search strategies, hardware, and downstream tasks. FACTS improves post-compression accuracy over SVD baselines, while CoRS and practical throughput measurements further characterize the efficiency trade-offs.

5.1. Experimental Setup

  • The evaluation uses ImageNet-1k validation Top-1 accuracy and calibrates decomposition methods on 16,384 ImageNet training samples.The compared architectures include DeiT, Swin, ConvNeXt, and MambaVision; compression is applied to linear layers, excluding the head.Source: §5.1 · p. 6

5.2. FACTS Evaluation

  • FACTS improves Top-1 accuracy over the strongest SVD baselines, including by 5.8 p.p. on Swin-B; CoRS raises the reported gains to 3.6–9.0 p.p.The comparison covers models at uniform compression, with FACTS plus CoRS reported separately.Source: Table 1 · p. 7, §5.2 · p. 7

5.3. CoRS Evaluation

  • CoRS achieves the highest accuracy among the evaluated rank-allocation methods, beating the best baselines by 0.6 p.p. on DeiT and 0.5 p.p. on Swin.The comparison includes gradient-based, greedy, and equal-error strategies; the authors report CoRS provides the best accuracy-efficiency trade-off.Source: §5.3 · p. 7, Table 3 · p. 8

5.2. FACTS Evaluation

  • FACTS achieves the best Top-1 accuracy on DeiT-B and Swin-B despite not having the highest cosine similarity to the empirical FIM.On Swin-B, Shampoo2 has the highest cosine similarity, while FACTS is best in accuracy; the reported gains over the next-best method are 0.3 p.p. on DeiT-B and 0.7 p.p. on Swin-B.Source: §5.2 · p. 7, Table 2 · p. 8

5.4. Ablations

  • Across tested small and large model variants, FACTS improves over the second-best baseline by up to 13.3 p.p. and 2.1 p.p., respectively, without search.With CoRS, the reported maximum gains are 24.4 p.p. for small models and 6.2 p.p. for large models.Source: §5.4 · p. 8, Table 5 · p. 9
  • FACTS outperforms the structured-pruning baselines in both finetuning scenarios, reaching within 0.4 p.p. of baseline accuracy after one epoch.Without finetuning, it matches isomorphic pruning’s speed while achieving 54.0 p.p. higher accuracy; with one epoch, it exceeds the second-best method by 1.0 p.p. using a 4x smaller compute budget.Source: §5.4 · p. 8, Table 4 · p. 9
  • On COCO detection and instance segmentation with a Swin-B backbone, FACTS significantly outperforms SVD-LLM and ties fully retrained PELA without multistage retraining.One finetuning epoch closes the remaining gap to the uncompressed baseline and surpasses converged PELA; DeiT-based UPerNet results on ADE20k follow similar trends.Source: Table 6 · p. 9, §5.4 · p. 9
  • 1.6x DeiT-B and 1.4x Swin-B throughput speedups are observed for 50%-SVD-compressed models across the evaluated GPUs and CPU.The platforms include Nvidia V100, A100, and H100 GPUs, and an Intel Xeon E5-2698 v4 CPU.Source: §5.4 · p. 8, Figure 2 · p. 9

6. Conclusion and Limitations

FACTS combines token-local aggregation with preserved activation-gradient dependence, and CoRS allocates ranks globally. Together they achieve accuracy-efficiency trade-offs across evaluated architectures, while the role of cross-token moments remains theoretically unresolved.

6. Conclusion and Limitations

  • FACTS combines token-local aggregation with preserved activation-gradient dependence for Fisher-weighted SVD compression of ViTs.The conclusion describes this curvature model as more effective for SVD compression than standard Kronecker choices.Source: §6 · p. 9
  • Together, FACTS and CoRS achieve the best accuracy-efficiency trade-offs across evaluated architectures under fixed FLOP budgets.The paper reports strong accuracy retention with practical inference speedups.Source: §6 · p. 9
  • When and why cross-token mixed moments misalign with compression-induced perturbations remains a direction for future theoretical work.The paper also proposes extending structured curvature to quantization and reducing calibration cost with cheaper block- or layer-wise FIM approximations.Source: §6 · p. 9

A. Extended Experimental Setup

The extended setup describes controlled Fisher-approximation comparisons, baseline and search implementations, and a short-schedule finetuning protocol.

A.1. Comparison between Kronecker Fisher Approximations

  • The comparison estimates Kronecker factors from 16k class-balanced ImageNet validation images and uses a disjoint 2k-image set for FIM cosine similarity and diagnostics.Methods share batch size 64, the SVD pipeline, compression budget, and whitening procedure.Source: §A.1 · p. 14
  • Operator cosine similarity is estimated with shared random probe matrices and Fisher-vector products, without materializing the empirical FIM.The cross-token diagnostic uses normalized Frobenius magnitudes of activation-gradient Gram products; off-diagonal values indicate coupling, but the heatmap is qualitative and not itself a FIM approximation.Source: §A.1 · p. 14
  • Within-token activation-gradient coupling is measured by distance correlation between activations and output gradients per token on the 2k-image sample.The statistic is computed with the hyppo Dcorr implementation.Source: §A.1 · p. 14

A.2. Decomposition

  • Vision baselines were reimplemented or adapted, with GFWSVD using 64-sample average gradients because its iterative method requires layer gradient samples in memory.Other Kronecker FIM methods use gradient regularization and Tikhonov shrinkage for computational stability.Source: §A.2 · p. 14

A.3. Search

  • Search baselines use method-specific settings, including non-squared singular values for MemViT and an automated fixed-FLOP routine for FLAR-SVD.ComCat converged within one epoch and reports no Swin values; ASVD’s sensitivity range was broadened and its metric changed to KL-Divergence.Source: §A.3 · p. 15

A.4. Finetuning

  • Finetuning comparisons use DeiT and a one-epoch budget, with learning rates of 8.7 × 10−6 for the proposed method and 1.0 × 10−4 for structured-pruning baselines.Both use the DeiT distillation soft loss instead of standard Cross-Entropy for short-schedule stability and performance.Source: §A.4 · p. 15

A.5. Downstream Tasks

The downstream-task setup evaluates compressed backbones on object detection and semantic segmentation using zero-shot and PELA-comparison protocols.

A.5. Downstream Tasks

  • Transferability is evaluated on COCO 2017 object detection with Mask-RCNN and Swin-B, and ADE20k semantic segmentation with UPerNet and Swin-B or DeiT-B.The section uses mmdetection and mmsegmentation frameworks.Source: §A.5 · p. 15
  • Compression is applied only to the backbone; in the zero-shot protocol, compressed downstream checkpoints are evaluated without further optimization.Rank configurations come from the search procedure; sensitivity calibration uses 10 images and task-specific metrics.Source: §A.5 · p. 15
  • The PELA comparison follows a four-stage procedure: compress the ImageNet-pretrained backbone, retrain it on ImageNet, then train the full downstream model from scratch.The retrained backbone initializes the downstream model before its full training.Source: §A.5 · p. 16

A.6. LLM Compression and Evaluation

The appendix specifies the LLM compression and evaluation protocol and extends visual diagnostics of token interactions and activation-gradient coupling across models.

A.6. LLM Compression and Evaluation

  • The LLM ablation compresses Qwen3-1.7B to 0.7 remaining parameters per layer using WikiText2 calibration and evaluates WikiText2 test perplexity.Calibration uses 256 training sequences of length 2048.Source: §A.6 · p. 16
  • The LLM evaluation also reports average zero-shot accuracy across PIQA, OpenBookQA, HellaSwag, ARC, and WinoGrande benchmarks.Evaluation uses the lm-eval library.Source: §A.6 · p. 16

B.1. Extended Analysis on Model Properties

  • The extended model-property analysis expands the token-interaction and activation-gradient visualizations; Figure 3 compares token and activation-gradient correlations across selected Swin, ConvNeXt, and Mamba layers.The figure’s top panel is a cross-token coupling diagnostic and its bottom panel shows activation-output-gradient distance correlations.Source: §B.1 · p. 16, Figure 3 · p. 16

B.1.1. GENERALIZATION ACROSS ARCHITECTURES.

Cross-token structure and activation–gradient coupling appear across the examined vision architectures and DeiT layer types, though their strength and spectral signatures vary. The extended analysis also compares the approach with alternative compression methods.

B.1.1. GENERALIZATION ACROSS ARCHITECTURES.

  • Swin-B, ConvNeXt-B, and MambaVision-B exhibit cross-token patterns and activation–gradient coupling, supporting the prevalence of both phenomena beyond DeiT.The passages note that these patterns empirically challenge standard approximations such as KFAC, which assume independence between activations and gradients.Source: §B.1.1 · p. 16

B.1.2. GENERALIZATION ACROSS LAYER TYPES.

  • QKV, Projection, and FC1 layers all show significant off-diagonal structural importance, despite differing spectral signatures.Figure 4 compares token correlations and activation–gradient correlations across these layer types.Source: §B.1.2 · p. 17, Figure 4 · p. 17

B.1.3. SUMMARY.

  • Both cross-token interactions and activation–gradient coupling occur across the examined vision models, with strength varying by layer and architecture.The section identifies these patterns in vision models beyond ViTs as well.Source: §B.1.3 · p. 17

B.2. Extended Throughput Analysis

  • The extended throughput analysis compares the approach with alternative compression methods beyond the main DeiT and Swin acceleration results.The next subsection benchmarks semi-structured sparsity and a MemViT SVD variant.Source: §B.2 · p. 17

B.2.1. BENCHMARKING SEMI-STRUCTURED SPARSITY.

The throughput comparisons find that 2:4 semi-structured sparsity is slower than dense inference on the tested GPUs, while MemViT's low-rank extension reduces standard SVD's speedup. These results motivate the use of standard SVD in the work.

B.2.1. BENCHMARKING SEMI-STRUCTURED SPARSITY.

  • 2:4 sparsity reaches only 0.8× baseline throughput on A100 and 0.5× on H100, while Figure 5 reports no latency improvement over baseline.The passage attributes the result to implementation effectiveness at layer sizes common in ViTs; V100 lacks hardware support for 2:4 sparsity.Source: §B.2.1 · p. 17, Figure 5 · p. 18

B.2.2. COMPARING STANDARD SVD TO MEMVIT.

  • At 50% compression, MemViT's low-rank extension yields ≈1.3× acceleration over baseline, versus ≈1.6× for standard SVD.The extension adds 5% of the original layer parameters; both variants still improve throughput over baseline.Source: §B.2.2 · p. 18

B.3. Semantic Segmentation Results

The expanded evaluations cover semantic segmentation, additional compression ratios, calibration-set sizes, CoRS search costs, and an LLM benchmark. FACTS and CoRS show competitive or improved results across these settings, with outcomes varying by task and configuration.

B.3. Semantic Segmentation Results

  • 43.5 mIoU without retraining beats SVD-LLM (38.1 mIoU) and retrained PELA (42.4 mIoU) for DeiT-B; Swin-B FACTS reaches 46.2 mIoU versus SVD-LLM's 37.7 mIoU.For DeiT-B, finetuning raises FACTS to 44.0 mIoU against the uncompressed baseline's 44.9 mIoU; for Swin-B, 10k finetuning iterations (< 10% of standard training) reach 48.1 mIoU versus PELA's 48.2 mIoU.Source: §B.3 · p. 18, Table 7 · p. 19

B.4. Additional Compression Ratios

  • At 40% and 60% remaining linear-layer FLOPs, FACTS shows the same improvements over prior works as at 50%, with larger gains when paired with CoRS.Tables 8 and 9 report the additional compression ratios.Source: §B.4 · pp. 18–19, Table 8 · p. 19, Table 9 · p. 20

B.5. Impact of Calibration Set Size

  • Across calibration sets of 1,024–8,192 images, FACTS remains competitive with less data, and adding CoRS recovers most performance across set sizes.The passage reports that most SVD methods benefit from additional calibration data, with MambaVision exceptions for GFWSVD and FLAR-SVD.Source: §B.5 · p. 19

B.6. Individual Contribution to CoRS Performance

  • ILP-based CoRS improves accuracy by +3.8 p.p. on DeiT-B and +4.4 p.p. on Swin-B; five-point search plus interpolation matches or improves on high-resolution search.Five-point search reduces total runtime to 14.8 min for DeiT-B and 27.3 min for Swin-B, roughly 50% less, with interpolation recovering its small accuracy drop.Source: §B.6 · p. 19, Table 12 · p. 21

B.7. What about LLMs?

  • On Qwen3-1.7B at identical compression, FACTS achieves the best reported results: 40.8 perplexity and 36.5% zero-shot accuracy.The passage suggests the combination of token-local aggregation and within-token activation–gradient coupling may remain beneficial even when cross-token structure is less spatially regular.Source: §B.7 · pp. 20–21, Table 11 · p. 21

C. Extended Methodology for FACTS

The extended methodology explains the Fisher-weighted SVD objective, FACTS's Kronecker-factor computation, and CoRS's rank search. It also contrasts Fisher approximations and describes interpolation for the search cost–error profiles.

C.1. Extended Preliminary Explanations

  • KFAC-expand discards cross-token mixed moments and assumes within-token activation–gradient independence; KFAC-reduce includes cross-token moments but remains decoupled.The alternative optimal rank-1 Kronecker fit targets the globally dominant energy of the Fisher matrix, which can implicitly reflect both structures.Source: §C.1 · pp. 21–22

C.2. Decomposition (The Fisher-Weighted SVD Objective)

  • Under the stated block-diagonal Hessian, Hessian-equals-Fisher, and Kronecker-factorization assumptions, loss-aware rank-k compression reduces to truncated SVD of a whitened weight matrix, followed by un-whitening.The objective is equivalent to minimizing the Frobenius norm of the whitened weight error.Source: §C.2 · p. 22

C.2.1. ALGORITHMIC DESCRIPTION OF FACTS

  • FACTS computes layer-specific row and column Fisher factors from activations and output gradients, regularizes them, whitens each weight matrix, and applies truncated SVD.The pseudocode also describes un-whitening the truncated result before replacing the layer weights.Source: §C.2.1 · p. 23, §C.3 · p. 24

C.3. Search

  • CoRS interpolates sampled discrete cost–error profiles into denser profiles so its ILP solver can select fine-grained ranks.The profiles are generated by sampling sensitivity at fixed intervals.Source: §C.3 · p. 23
  • Sliding-window interpolation fits local cubic splines and retains monotone fits, avoiding artifacts such as artificial minima from global cubic splines.The method uses overlapping three-point windows; the reported performance gains over simple linear interpolation are marginal, making linear interpolation a practical option.Source: §C.3 · p. 23, Figure 7 · p. 24
Loading 2609.07155v1…