PaperlayerView exact revision on arXiv

Source-linked AI summary

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Baher Mohammad, Stamatios Lefkimmiatis

arXiv:2602.15200v1cs.LG

TL;DR

Transformer compression must reduce deployment costs despite restrictive shared-subspace assumptions and expensive iterative sparse coding. COMPOT uses calibration-guided orthogonal dictionary learning with closed-form updates and one-shot global allocation, achieving a better quality–compression trade-off across tasks and integrating with quantization. Its reliability depends on representative calibration data and practical whitening assumptions, while fixed sparsity patterns remain an open limitation.

  • Problem

    Existing SVD compression can be restrictive because it enforces one shared subspace, while sparse dictionary learning often relies on iterative dictionary and coefficient updates.

  • Method

    COMPOT is a training-free, calibration-guided framework using complete or undercomplete orthogonal dictionaries, closed-form Procrustes and analytical sparse-coding updates, and one-shot global allocation.

  • Results

    Across language, vision-language, and speech models, COMPOT improves the quality–compression trade-off over strong SVD and sparse dictionary baselines and composes with quantization under matched memory budgets.

  • Takeaways & Limitations

    COMPOT provides a deterministic post-training compression approach that combines flexible sparse representations with efficient orthogonal updates and quantization compatibility.

  • Takeaways & Limitations

    Results can depend on calibration-data representativeness, whitening conditioning, and the fixed sparsity structure used during factorization.

Abstract

from arXiv · show

Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available $\href{https://github.com/mts-ai/COMPOT}{here}$.

1. Introduction

Transformer compression is motivated by growing deployment costs and heterogeneous redundancy that makes a single shared low-rank subspace restrictive. COMPOT addresses this with orthogonal sparse dictionary learning, closed-form updates, one-shot allocation, and quantization compatibility.

  • Transformer scaling increases memory, bandwidth, and compute constraints, motivating post-training compression methods.
  • SVD-based methods use calibration and closed-form updates but still impose one shared subspace per weight matrix.
  • COMPOT introduces training-free orthogonal dictionary factorization with closed-form Procrustes dictionary updates and analytical sparse coefficient updates.
  • COMPOT allocates compression in one pass by pooling singular values globally while constraining per-matrix compression and sensitive-layer over-compression.
  • 4-bit GPTQ applied on top of COMPOT yields superior results to quantization alone under equal memory budgets.
  • COMPOT outperforms strong SVD baselines across architectures and downstream tasks while integrating with post-training quantization.

2. Related Work

Related work spans pruning, quantization, low-rank factorization, and sparse dictionary learning. COMPOT builds on evidence that heterogeneous redundancy requires adaptive allocation and that orthogonal constraints can simplify sparse factorization.

  • Post-training compression includes pruning, quantization, distillation, and factorization to reduce Transformer inference cost and memory.
  • Low-rank factorization: Low-rank methods increasingly use calibration, activation information, loss modeling, and non-uniform allocation to guide truncation.
  • Low-rank factorization: SVD represents each matrix with one shared subspace, which can be restrictive when columns are better explained by different local subspaces.
  • Dictionary learning and sparse coding: Sparse dictionary learning enables columns to select different atom subsets, inducing a union-of-subspaces structure, but K-SVD-style updates are computationally heavy at billion-model scale.
  • Dictionary learning and sparse coding: Orthogonal dictionaries reduce dictionary learning to closed-form orthogonal Procrustes updates, while sparse coding admits an analytical solution.
  • Dynamic allocation: Heterogeneous redundancy makes uniform compression suboptimal, motivating COMPOT’s pooled-singular-value allocation with safeguards against negative and excessive per-matrix compression.

3. Method

COMPOT compresses Transformer projections by replacing each weight matrix with a calibration-guided structured factorization. It combines union-of-subspaces modeling, whitening, closed-form alternating updates, and one-shot global allocation under compression constraints.

  • The compression goal is to reduce projection storage while preserving layer behavior on a small calibration set without back-propagation.
  • Data-aware whitening: The calibration objective minimizes output mismatch, with whitening converting functional error into reconstruction error in whitened coordinates.
  • Subspace and union-of-subspaces modeling: Unlike low-rank factorization’s single shared subspace, sparse dictionary learning lets columns select different atom subsets and form a union of subspaces.
  • COMPOT factorization: COMPOT uses an orthogonal complete or undercomplete dictionary and column-sparse codes to retain union-of-subspaces flexibility in a whitened, data-aware space.
  • Alternating minimization: Orthogonality yields analytical hard-thresholding for sparse coding and a closed-form SVD-based Procrustes dictionary update, enabling alternating minimization without iterative pursuits.
  • Dynamic compression allocation: One-shot allocation pools singular values from normalized matrices, globally truncates the smallest values, and applies minimum and maximum compression guards.
  • Compression ratio: The compression ratio accounts for storing both 16-bit dictionary values and sparse-code nonzeros with a binary position mask, parameterized by a k/s-ratio.

4. Experiments

COMPOT is evaluated across model families, domains, compression levels, and strong low-rank, dictionary-learning, pruning, and quantization baselines. Results and ablations indicate that its initialization, global pooling, dynamic allocation, and factorization improve the quality–compression trade-off while remaining training-free.

  • Experimental setup: Evaluations cover Llama, OPT, and Qwen models from 0.6B to 30B across language, vision-language, and audio tasks at compression ratios 0.2–0.6.The study compares SVD-LLM variants, Dobi-SVD, CoSpaDi, structured pruning, and GPTQ.
  • Ablations: SVD initialization consistently improves static and dynamic compression over random-column initialization at a fixed iteration budget.The comparison uses Llama3.2-1B at 0.2 compression with 20 alternating minimization iterations.
  • Ablations: SVD initialization saturates at approximately 100 alternating updates, while random initialization improves up to approximately 300; the main experiments use 20 iterations.The chosen setting is described as a pragmatic accuracy–time compromise.
  • Ablations: Global singular-value pooling yields the best overall accuracy and perplexity among individual, partial, and fully grouped allocation strategies.The grouping study uses Llama3.2-1B at 0.2 compression with 20 alternating minimization iterations.
  • Main results: Across compression ratios, static COMPOT achieves higher average accuracy and lower perplexity than SVD-LLM and CoSpaDi under fixed per-layer budgets.The comparison covers Llama3-8B and Qwen3-8B, with smaller-model results reported in the appendix.
  • Main results: Under matched global compression, training-free COMPOT consistently achieves stronger accuracy and perplexity than Dobi-SVD and SVD-LLM V2.The SVD-LLM V2 comparison includes reproduced and published results, while some baseline runs failed under the released pipeline.
  • Main results: At comparable effective compression, COMPOT preserves substantially higher accuracy and lower perplexity than structured pruning methods.The comparison is performed on Llama3-8B against ReplaceMe and LLM-Pruner.
  • Main results: COMPOT retains strong multimodal performance and achieves the best average accuracy across benchmarks on Qwen3-VL-8B-Instruct at compression ratio 0.2.The reported benchmarks are MMMU, OCRBench, RealWorldQA, and MMStar; SVD-LLM degrades severely in this comparison.

5. Limitations and Conclusion

The paper concludes that COMPOT improves the quality–compression trade-off across language, vision-language, and speech models while composing with post-training quantization. It also identifies calibration dependence, whitening assumptions, and fixed sparsity patterns as limitations or open directions.

  • Conclusion: COMPOT combines orthogonal dictionary learning, closed-form Procrustes updates, analytical sparse coding, and deterministic one-shot global allocation in a training-free framework.These design choices target Transformer projection compression.
  • Conclusion: Across language, vision-language, and speech models, COMPOT improves the quality–compression trade-off over strong SVD and sparse dictionary baselines.It also composes effectively with post-training quantization under matched memory budgets.
  • Limitations: Results may be less reliable when calibration data lack diversity or differ from the deployment distribution.The limitation follows from dependence on representative calibration statistics.
  • Limitations: Whitening is most convenient when the Gram or covariance matrix is well-conditioned and admits a Cholesky factorization.SVD- or eigendecomposition-based alternatives with mild regularization can address violations of this assumption.
  • Limitations: The current implementation uses a fixed sparsity structure, leaving pattern learning and fixed-pattern healing as open extensions.These extensions may improve accuracy or adaptivity at the same storage budget, but may increase cost.

Impact Statement

COMPOT targets lower memory, hardware, and energy costs for deploying large language, vision, and audio models. More accessible deployment may also broaden exposure to existing risks associated with powerful models.

  • Benefits: By reducing hardware and energy costs, COMPOT may make state-of-the-art language, vision, and audio models more accessible to resource-limited researchers and practitioners.The paper also notes potential mitigation of the environmental impact of large-scale inference.
  • Risks: Lower deployment barriers may broaden applications involving documented risks such as bias, privacy, misinformation, and misuse.The paper recommends content filtering, monitoring, safeguards, and fairness and robustness evaluation.

A. Supplementary Material

Supplementary material provides additional results and descriptions to document COMPOT’s compression performance.

  • Supplementary material: Additional results and descriptions are included to provide further evidence on COMPOT’s compression performance.The paper attributes this material to limitations in the main-text space.

A.1. COMPOT Optimization Details

The appendix presents the derivation of COMPOT’s analytical sparse-coding solution and details the optimization procedure for equation (7), including closed-form updates and alternating minimization.

  • The appendix derives COMPOT’s analytical sparse-coding solution and details optimization procedure (7).It covers the closed-form sparse-coding update under orthogonality and the full alternating minimization algorithm.

A.1.1. DERIVATION OF SPARSE CODING ANALYTICAL SOLUTION

Under orthogonality, COMPOT decouples the sparse-coding updates and reduces them to hard-thresholding projections, yielding an exact global minimizer without iterative sparse pursuits.

  • Under D^⊤ODO = Ik, each column update decouples.
  • The sparse-coding problem becomes projection onto the set of s-sparse vectors.
  • Hard thresholding keeps the s largest-magnitude entries and zeros the rest.The operator Hs(·) performs this projection.
  • The resulting update is an exact global minimizer under orthogonality and avoids iterative sparse pursuits.

A.1.2. OBJECTIVE IN THE WHITENED SPACE

COMPOT transforms the calibration objective into whitened coordinates using the Gram matrix’s Cholesky factorization, where functional-error minimization becomes reconstruction-error minimization.

  • Assuming G ≻ 0, COMPOT factors the calibration Gram matrix G = X⊤X as G = LL⊤.
  • With fW ≜ L⊤W and fcW ≜ L⊤cW, original-space functional-error minimization is equivalent to whitened-space reconstruction-error minimization.

A.1.3. COMPOT FACTORIZATION PROBLEM

COMPOT formulates orthogonal sparse factorization in whitened coordinates, then maps the optimized solution back to the original parameter space. The dewhitening factor is computed offline, so inference uses only the resulting factors.

  • COMPOT solves orthogonal sparse factorization in whitened coordinates.
  • The orthogonal dictionary DO has dimensions m×k with k ≤ m, while SO has dimensions k×n.The dictionary may be complete or undercomplete.
  • After optimization, COMPOT maps the solution back to the original parameter space.
  • The dewhitening factor A is computed offline, and inference uses only (A, SO).

A.1.4. CLOSED-FORM SPARSE CODING UNDER ORTHOGONALITY

COMPOT combines exact sparse coding under orthogonality with one-shot global allocation and reports favorable accuracy, compression, and optimization-time results across models and tasks.

  • A.1.4. CLOSED-FORM SPARSE CODING UNDER ORTHOGONALITY: Under orthogonality, sparse coding is solved exactly by hard thresholding rather than iterative pursuit methods.The update retains the s largest-magnitude entries column-wise.
  • A.1.4. CLOSED-FORM SPARSE CODING UNDER ORTHOGONALITY: COMPOT alternates closed-form sparse coding with an orthogonal Procrustes dictionary update to obtain compressed factors for each projection matrix.The dictionary update uses a thin SVD, while the factorization maps W approximately to ASO.
  • A.2. Dynamic Compression Ratio Allocation: One-shot global allocation redistributes compression across heterogeneous Transformer layers while enforcing per-matrix guards and avoiding factorization when it is not beneficial.The procedure pools singular values and truncates globally under a model-wide budget.
  • A.2. Dynamic Compression Ratio Allocation: The allocation procedure normalizes weight matrices before pooling singular values, equalizing scale across layers and projection types.Its importance criterion truncates smaller singular values first, while allocation itself uses the original non-whitened space.
  • A.3. Comparison with Low-rank and Sparse Dictionary Learning Methods on Small Models: On Qwen3-0.6B and Llama3.2-1B, COMPOT outperforms SVD-based and dictionary-learning methods by a wide margin across tested compression ratios.The comparison includes both static and dynamic compression-ratio settings.
  • A.5. Wall-clock Time for Optimization: COMPOT’s thin-SVD Procrustes update is approximately 2× faster per iteration than CoSpaDi’s K-SVD updates with power iterations.SVD-LLM remains substantially faster overall because it performs a single truncated SVD per matrix.
  • A.7. Acceleration Strategies for Alternating Minimization Process: Stricter early-stopping tolerance increases optimization iterations while correlating with higher final model accuracy under a fixed 100-iteration budget.The experiment uses randomly initialized dictionaries.
  • A.8. Optimal Dictionary-to-sparsity ratio: A dictionary-to-sparsity ratio of k/s = 2 provides the best reported balance between representation capacity and sparse regularization.This setting achieves the highest accuracy across the evaluated benchmarks with N = 100 iterations.
Loading 2602.15200v1…