GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression

• Deep transformer stacks contain cross-layer weight redundancies that independent layer-wise factorization overlooks.Compression is motivated by the memory and computational costs of transformer deployment.Source: §1 · p. 1
• Layer-specific activation distributions create incompatible whitening spaces, while covariance averaging and fixed-adjacency sharing can distort geometry and miss non-local alignments.Prior methods aggregate layer covariances into global whitening transforms and restrict sharing to adjacent layers.Source: §1 · p. 1
• GeoPair uses a generalized Sylvester equation for exact shared-dictionary updates and optimal maximum-weight layer matching via Edmonds’ Blossom algorithm [12].Its shape-agnostic column-space alignment metric identifies compatible projections under distinct layer-specific whitening transforms.Source: §1 · p. 2
• HTP with a conjugate-gradient solver adds structured, layer-specific coefficient sparsity without heuristic budget allocation or dynamic scheduling.The framework reports convergence guarantees and outperforming dense low-rank baselines at high compression ratios.Source: §1 · p. 2
• Experiments across architectures, scales, and modalities report state-of-the-art results over independent factorization, heuristic merging, and existing dictionary-learning approaches.The paper presents the unified pipeline as a reproducible, optimization-driven alternative to heuristic compression strategies.Source: §1 · p. 2

Data-Aware Matrix Factorization

• SVD gives the optimal rank-r Frobenius approximation and is equivalent to PCA, but isotropic activation assumptions hurt accuracy at high compression; whitening-based methods [42] [38] [45] preserve performance while treating layers independently.Source: §2 · p. 2

Dictionary Learning and Sparse Decomposition

• k-SVD, Method of Optimal Directions, CoSpaDI [18], ROCKET [3], and COMPOT [19] adapt dictionary learning to transformer compression, with structured sparsity improving trade-offs but no explicit cross-layer sharing.These methods optimize dictionaries and coefficients per layer or rely on heuristic update schedules.Source: §2 · p. 2

Cross-Layer Grouping and Shared Basis Learning

• Basis Sharing [37] pairs adjacent layers and beats independent baselines, but fixed adjacency, global covariance aggregation, and model-specific constraints limit its universality.The approach restricts sharing between certain projection types to maintain numerical stability.Source: §2 · pp. 2–3
• Matrix PCA [45] uses EVD on stacked weights, yet drift-based grouping and independent refinement require manual budgeting and ignore distinct whitening geometries.Source: §2 · p. 3

Positioning of Our Approach

• GeoPair jointly optimizes layer pairing and shared dictionaries, preserving distinct whitening geometries while enforcing structured coefficient sparsity without manual budgets or schedules.It uses generalized Sylvester updates, global maximum-weight matching based on shape-agnostic column-space alignment, and HTP with conjugate gradient.Source: §2 · p. 3

3.1 Overview and Problem Setup

• Compression is formulated as activation reconstruction on a small calibration set, with a Tikhonov-regularized Gram matrix ensuring a well-posed whitening transform even when calibration data are rank-deficient.The regularizer is equivalent to adding a weight-space ℓ2 penalty; the objective becomes reconstruction in the calibration-induced whitened space.Source: §3.1 · p. 3

3.2 Cross-Layer Shared Dictionary Optimization

• For each compatible pair, GeoPair learns a shared dictionary with layer-specific coefficients to reduce calibration-weighted reconstruction error across both projections.Alternating coefficient and dictionary updates use closed-form steps and monotonically descend to a block-stationary point under the stated BSUM conditions.Source: §3.2 · p. 4

3.3 Sparse Matrix Coefficients via Hard Thresholding Pursuit

• HTP replaces the dense coefficient update with an ℓ0-constrained update that retains a target number of nonzeros per column within the alternating minimization.When sparsity is disabled, the coefficient procedure reduces to the standard closed-form Cholesky update.Source: §3.3 · p. 5

3.4 Optimal Cross-Layer Grouping via Graph Matching

• GeoPair pairs projections globally by minimal whitened-space structural distance, including attention and feed-forward weights with a shared input dimension, rather than defaulting to adjacent layers.Exact maximum-cardinality matching minimizes total structural distance and is reported to yield lower activation-weighted reconstruction error and improved downstream perplexity than fixed adjacent pairing.Source: §3.4 · pp. 5–6

3.5 Algorithmic Summary

• The alternating procedure monotonically decreases the calibration-weighted objective and stops when relative improvement falls below τ or the maximum iteration count Tmax is reached.Source: §3.5 · p. 6

4 Experiments

• On Llama-3 1B and 8B, Sylv improves on GW, OG adds zero-shot gains, and Sylv + OG + SP recovers over 90% of baseline accuracy.Removing OG from the sparsified pipeline degrades performance, underscoring the importance of optimal grouping for coefficient recovery.Source: §4 · p. 6, Table 1 · p. 7
• Across compression ratios on Llama3 8B, GeoPair achieves the highest accuracy among the evaluated dense low-rank and sparse dictionary factorization methods.The reported comparison also evaluates WikiText log-perplexity.Source: Figure 2 · p. 7, §4 · p. 7
• On Llama2 7B at 20% compression, GeoPair achieves the highest average accuracy across benchmarks without post-compression fine-tuning.Competing methods rely on healing or recovery training, which adds computational overhead and requires large datasets.Source: Table 2 · p. 7, §4 · p. 7
• Across model families and scales from 1B to 32B, GeoPair is evaluated at increasing compression ratios against Basis Sharing at 0.2, 0.3, and 0.4.At small compression ratios, minor improvements align with findings that small rank truncation can denoise while preserving salient features [29].Source: §4 · pp. 7–8, Table 3 · p. 8

4.1 Generalization across tasks

• On Wan2.2 5B, GeoPair preserves video generation alignment without fine-tuning: the average CLIP score drops by 1 × 10−4 at 20% compression and 2.5% at 40%.The evaluation uses X-CLIP with 16-frame sampling over 50 prompts; generated video frames provide qualitative evidence of preserved fidelity.Source: §4.1 · p. 8, Figure 3 · p. 8

Ablation Studies

• Appendix D benchmarks sparse algorithms for target sparsity and examines GeoPair’s convergence and sensitivity to calibration data.Source: §5 · p. 8

Grouping algorithm

• Frobenius norm-based grouping achieves the best trade-off among the evaluated layer-grouping strategies.The ablation compares two distance metrics with the greedy approach used in Basis Sharing [37], at CR=0.4 and KS=2.0 on Llama3.2 1B.Source: Table 4 · p. 9, §5 · p. 9

Ablation on the KS ratio

• KS=2.5 leads to the best results and is the default for experiments that sparsify matrix coefficients.The KS ratio is the number of dictionary atoms divided by the non-zero elements in each coefficient-matrix column.Source: Table 5 · p. 9, §5 · p. 9

6 Conclusion & Limitations

• Across diverse transformer architectures, parameter scales, and modalities, GeoPair recovers > 90% of baseline accuracy at high compression ratios without fine-tuning.The paper describes this performance as state of the art.Source: §6 · p. 9
• GeoPair combines global weighted matching for layer pairs, closed-form generalized Sylvester dictionary updates, and convergent Hard Thresholding Pursuit to preserve layer-specific activation geometries while exploiting cross-layer redundancy.The framework sequentially optimizes cross-layer grouping and shared-dictionary factorization.Source: §6 · p. 9
• The framework is restricted to pairwise layer grouping because the exact closed-form solver depends on simultaneous diagonalization of two matrix pencils, which does not analytically generalize to m-term systems.Larger shared groups may require iterative numerical approximations or higher-order tensor factorizations, with added computational overhead and potential numerical instability.Source: §6 · p. 9

A.1 Background and recap

• The classical Sylvester equation AX + XB = C has a unique solution when the spectra of A and −B are disjoint, and Bartels-Stewart solves it stably in O(m^3 + n^3) time [4].Direct Kronecker-product inversion scales as O(m^3n^3) and is described as numerically unstable for large dimensions.Source: §A.1 · p. 14
• For the two-term generalized Sylvester update, simultaneous diagonalization yields a closed-form solution when the matrix-pencil conditions hold; Tikhonov regularization ensures well-posed decomposition and the solution costs O(d^3).With the specified positive-definite and positive-semidefinite matrices, the scalar denominators are nonzero, yielding a unique, numerically stable dictionary update at each alternating-minimization step.Source: §A.1 · p. 14
• HTP improves on pure thresholding by combining gradient-based support identification with least-squares fitting on the selected subspace, and converges linearly under mild restricted isometry-type conditions.In GeoPair, HTP solves the sparsity-constrained coefficient subproblem for a fixed dictionary and layer-specific calibration geometry.Source: §A.1 · pp. 14–15
• GeoPair's HTP selects the top-k_i coefficients column-wise, then refines active coefficients by solving a masked linear system with batched Conjugate Gradient rather than explicitly inverting the restricted Gram submatrix.The procedure repeats for a fixed number of inner iterations before updating the dictionary.Source: §A.1 · p. 15

Optimization Objective

• With coefficients fixed, the shared dictionary minimizes joint calibration-weighted reconstruction error by solving a two-term generalized Sylvester equation.The optimality condition matches the dictionary update rule in the main text.Source: §A.2 · pp. 15–16

Generalized Sylvester Equation

• Regularized generalized eigenvalue decompositions yield a unique, well-conditioned closed-form dictionary update, with denominator 1 + λ_jσ_k ≥ 1.The coefficient Gram matrices receive Tikhonov regularization ε = 10^-6, including under aggressive sparsification.Source: §A.2 · p. 16

Convergence Guarantee

• Under standard regularity conditions, alternating dictionary and coefficient updates converge to a block-wise stationary point of the calibration-weighted reconstruction objective.The dictionary subproblem is solved exactly; HTP coefficient updates provide sufficient decrease. The guarantee relies on smoothness, the KL property, and bounded iterates.Source: §A.2 · p. 16

Practical Convergence Properties

• The objective decreases monotonically, and the iterate sequence converges to a critical point with finite length and an empirically sublinear rate.The stationary configuration has no feasible descent direction for the dictionary or coefficient blocks.Source: §A.2 · p. 17

Caveats and Engineering Considerations

• Because the ℓ0-constrained subproblem is NP-hard, convergence is to a local stationary point, not a global optimum.The paper recommends SVD-based initialization and multiple random restarts for high-sparsity regimes; it reports T_HTP = 2 with conjugate-gradient preconditioning as a reliable runtime–descent trade-off.Source: §A.2 · p. 17

B Implementation Details

• Zero-shot evaluation uses PIQA, HellaSwag, LAMBADA, ARC, SciQ, RACE, and MMLU; Table 2 follows pruning-paper benchmarks, while Table 7 evaluates Qwen3-8B.The evaluation protocols follow prior work for fair comparison.Source: B Implementation Details · p. 17
• The grouping distance is computed without materializing shifted submatrices, using a constant first term, prefix sums for sliding-window column norms, and row-wise cross-correlation.The prefix sums take O(d_j) time; the cross-correlation outputs are summed across rows.Source: B Implementation Details · p. 17
• The distance computation takes O(d · d_i · (d_j − d_i)) time with O(1) auxiliary memory, simplifying to O(d · d_i^2) when d_i ≈ d_j.The vectorized sliding-window computation uses prefix sums and 1D convolutions and is optimized on modern GPUs.Source: B Implementation Details · p. 18
• For Blossom matching, GeoPair assigns symmetric edge weights from one distance calculation and uses the smaller normalized Frobenius norm for equal-dimensional matrices.This enforces an undirected graph and a conservative, symmetric error estimate.Source: B Implementation Details · p. 18
• Masked batched Conjugate Gradient updates stop at relative residual 10^-5 or 10 iterations, while Blossom matching takes under 2 seconds for 560 candidate weights.CG starts from the previous iterate masked to current support; matching uses Edmonds’ Blossom algorithm [12] via NetworkX, with worst-case O(N^3) complexity.Source: B Implementation Details · p. 18

C.1 Statistical Significance

• 45.85% ± 0.19% average accuracy and 20.74 ± 0.76 average Lambada perplexity show stable Llama-1B performance across five seeds at 40% compression.The seeds used varied data sampling; values are mean ± standard deviation.Source: §C.1 · p. 18

C.2 Comparison with Basis sharing and Cospadi

• GeoPair consistently achieves higher average accuracy and lower perplexity than Basis Sharing and CoSpaDi across compression ratios 0.2, 0.3, and 0.4.The comparison uses consecutive grouping, omits sharing for down- and out-projection layers, and reports both dense GeoPair* and the full GeoPair pipeline.Source: §C.2 · p. 18, Table 6 · p. 19

C.3 Evaluations on another set of benchmarks

• GeoPair maintains competitive performance across advanced Qwen3-8B benchmarks at higher compression ratios, with graceful degradation favoring reasoning-intensive MuSR over surface-level accuracy.The suite includes IFEval, BBH, GPQA, MuSR, and MMLU-Pro.Source: §C.3 · p. 18, Table 7 · p. 19

C.4 Evaluations on Other Modalities

• The evaluation targets VibeVoice 1.5B [24], chosen as a challenging test case because smaller models are typically more sensitive to compression.The authors frame the audio evaluation as an extension of demonstrations spanning text to video.Source: §C.4 · p. 19
• Calibration used the first 256 samples from the dataset introduced in [25], while evaluation used 100 held-out samples from a disjoint partition of that dataset.Source: §C.4 · p. 19
• Linguistic accuracy was measured by transcript WER against reference text using Whisper-3 Large, and perceptual quality by UTMOS.Transcripts underwent standard text normalization before WER was calculated.Source: §C.4 · p. 19
• At 20% compression, VibeVoice 1.5B maintains strong performance with modestly higher WER and slightly lower UTMOS, preserving core functionality without fine-tuning or post-hoc recovery.Table 8 reports WER and UTMOS for the original and compressed model; the supplied passages do not state the scores.Source: §C.4 · p. 19, Table 8 · p. 19
• Basis Sharing failed on VibeVoice, producing a compressed model with WER over 100% and UTMOS around 1.4.The authors anticipate that the larger VibeVoice 9B may be more compressible because of greater parameter-space redundancy.Source: §C.4 · p. 19
• The authors present the audio findings as a proof of concept for applying the method across model scales, architectural designs, and generative modalities.Source: §C.4 · p. 19

D.1 Convergence ablation

• At CR=0.4 on Llama 3.2 1B, 2 HTP and 10 CG iterations achieve the selected efficiency-accuracy balance at 850s runtime.The Sylvester solver terminates when the fifth-digit residual stabilizes; Table 9 identifies this setting as the default based on runtime and performance.Source: §D.1 · p. 20, Table 9 · p. 20
• The Sylvester solver uses adaptive convergence, terminating when the residual stabilizes at the fifth digit.The table describes the stopping condition as fifth floating-point error stabilization.Source: §D.1 · p. 20, Table 9 · p. 20
• The selected 2 HTP and 10 CG iteration configuration is used throughout the sparse compression experiments.Source: §D.1 · p. 20

D.2 Ablation on the Sparsification algorithm

• HTP achieves better reconstruction fidelity, accuracy, and perplexity than IHT using two rather than ten inner iterations and approximately four times less computational time.The comparison is at KS=2.5 on Llama3.2 1B; lower perplexity and higher accuracy indicate better capability preservation.Source: §D.2 · p. 20, Table 10 · p. 20

D.3 Ablation on Calibration Data Selection

• The study compares WikiText, Alpaca instructions, and fineweb [23] using 256 calibration samples at a 40% compression ratio.Source: §D.3 · p. 20
• FineWeb is adopted for all main experiments because it has the lowest Lambada perplexity and aligns with compression methods that use large web corpora.The passage names CoSpaDi and ROCKET as examples.Source: §D.3 · p. 20, Table 11 · p. 21
• FineWeb achieves 21.69 Lambada perplexity versus 38.86–39.02 for the alternatives, while Alpaca reaches 45.9% average accuracy versus FineWeb’s 45.6%.Lower perplexity indicates better language-modeling capability preservation.Source: §D.3 · p. 20, Table 11 · p. 21

D.4 Ablation on Calibration Sequence Count

• With FineWeb at 40% compression, performance generally improves as calibration samples increase from 32 to 1024, with notable gains up to 256.Source: §D.4 · p. 21, Table 12 · p. 21
• Increasing calibration samples from 256 to 1024 yields only a +0.9% average-accuracy gain and a modest Lambada-perplexity reduction, while requiring 4x calibration cost.Source: §D.4 · p. 21
• At 256 samples, GeoPair reaches 45.6% average accuracy and 21.69 Lambada perplexity; this is selected as the default efficiency–capability trade-off.The choice is consistent with CoSpaDi and ROCKET, which also adopt 256 calibration samples.Source: Table 12 · p. 21, §D.4 · p. 21

D.5 Ablation Study: Dictionary Initialization Strategies

• Original-space dictionary initialization yields marginally better average accuracy and more stable convergence, especially for layers with divergent activation statistics.The comparison holds other pipeline components constant and evaluates Llama-3 1B at CR=0.4; the passage reports qualitative results without giving numerical values.Source: Table 13 · p. 22, §D.5 · p. 22

D.6 Running Time and Environmental Impact

• Runtime scales approximately linearly with parameter count; Qwen-32B uses 0.099 kWh/B versus 0.147 kWh/B for Llama-1B, and total emissions are ≤1.4 kg CO2eq.The reported metrics cover compression-only runs at CR=0.4 with 256 calibration samples.Source: Table 14 · p. 22, §D.6 · p. 22

NeurIPS Paper Checklist

• The authors identify grouping more than two layers as a limitation because it requires numerical approximations and is outside the work’s scope.Source: NeurIPS Paper Checklist · p. 23
• The checklist reports that experiments require no training, use public benchmarks, and are reproducible through public data and supplementary implementation details.It also states that error bars were added to the appendix.Source: NeurIPS Paper Checklist · pp. 24–25
• The checklist gives “does not involve crowdsourcing” as the justification for its response in this passage.The supplied text includes an IRB question but does not provide a distinct IRB approval or risk assessment.Source: NeurIPS Paper Checklist · p. 28
• The checklist marks LLM usage [N/A], stating that an LLM was used only to polish the paper and correct grammar.It says LLMs were not an important, original, or non-standard component of the core method.Source: NeurIPS Paper Checklist · p. 29