PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

Denis Makhov, Dmitriy Shopkhoev, Magauiya Zhussip, Ammar Ali, Stamatios Lefkimmiatis

arXiv:2509.22075v6cs.CLcs.AI

TL;DR

Post-training low-rank compression constrains each projection to one shared subspace, which can be restrictive for heterogeneous transformer projections. CoSpaDi uses calibration-guided sparse dictionary learning and improves reported quality–compression trade-offs over activation-aware SVD baselines across Llama and Qwen models, while remaining competitive with structured pruning.

  • Problem

    Low-rank post-training compression represents each projection in one shared low-dimensional subspace, a constraint that can be restrictive for heterogeneous transformer projections.

    Source: §1 · p. 1

  • Method

    CoSpaDi uses calibration activations and Gram orthonormalization to transform functional output reconstruction into a standard dictionary-learning problem with column-sparse coefficients.

    Source: §1 · p. 2

  • Results

    Across Llama and Qwen models, CoSpaDi improves quality–compression trade-offs over strong activation-aware SVD baselines and is competitive with structured pruning methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Its sparse coefficients support sparse–dense computation and post-training quantization, and hardware-aware deployment choices show measurable speedups over the dense baseline.

    Source: §1 · p. 2, §5 · p. 9

    The main experiments fix ρ and use mostly uniform budgets, leaving adaptive layer- or type-wise allocation as a natural extension.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across Llama and Qwen models, CoSpaDi improves accuracy--compression and perplexity--compression trade-offs over SVD-based and structured pruning baselines at 20--40\% compression ratios, while naturally supporting sparse--dense computation and post-training quantization of sparse coefficients.

1 Introduction

Post-training compression makes LLMs easier to store and run, but common low-rank methods constrain each projection to one shared subspace. CoSpaDi replaces that constraint with calibration-guided sparse dictionary learning and improves the reported quality–compression trade-off across Llama and Qwen models.

1 Introduction

  • LLMs are costly to store and run, creating a practical deployment barrier on memory- and compute-constrained hardware.Source: §1 · p. 1
  • Truncated SVD and its activation-aware variants remain constrained to a shared low-dimensional subspace, which may be restrictive for heterogeneous transformer projections.Source: §1 · p. 1
  • A learned dictionary with column-sparse coefficients lets different weight columns use different atom subsets, replacing one shared low-dimensional subspace with a union of subspaces.The paper motivates this representation as more flexible than a single shared subspace at the same parameter budget.Source: §1 · p. 1
  • CoSpaDi uses calibration activations and Gram orthonormalization to turn functional output reconstruction into a standard dictionary-learning problem, without gradient-based fine-tuning.The resulting sparse coefficients can also be quantized post-training, and the framework supports cross-layer dictionary sharing.Source: §1 · p. 2
  • CoSpaDi improves the quality–compression trade-off over strong activation-aware SVD baselines across Llama and Qwen models, and is competitive with recent structured pruning methods.The comparison covers per-layer and grouped scenarios.Source: §1 · p. 2

2 Related Work

Prior post-training compression work spans low-rank factorization, dictionary learning, pruning, and quantization. CoSpaDi applies calibration-guided sparse dictionary learning to transformer weights, replacing the shared-subspace model of SVD with a union-of-subspaces representation.

2 Related Work

  • SVD-based methods use a shared low-dimensional subspace per matrix, while CoSpaDi instead uses a union-of-subspaces representation.Activation-aware and cross-layer SVD variants retain the shared-subspace assumption (Wang et al., 2025a; Wang et al., 2025d,c,b).Source: §2 · p. 2
  • Dictionary learning represents each vector with a subset of learned atoms, unlike low-rank models’ single shared basis.Source: §2 · p. 2
  • Figure 1 contrasts two dense factors for a single r-dimensional subspace with DS, where each column of S selects at most s of k dictionary atoms.The caption distinguishes undercomplete, complete, and overcomplete dictionaries by whether k is below, equal to, or above d1.Source: Figure 1 · p. 3
  • Pruning and quantization are complementary post-training compression axes; CoSpaDi’s sparse coefficients may be quantized post hoc, while runtime benefits depend on implementation factors.The cited work notes dependencies on metadata overhead, memory access patterns, sparsity structure, and kernel or hardware support.Source: §2 · pp. 2–3

3 Method

CoSpaDi minimizes calibration-aware output reconstruction by transforming weights with an activation-derived Gram factor, then fitting a sparse dictionary. Its optimization alternates sparse coding and dictionary updates, with optional sharing across related projections and storage and inference costs shaped by sparsity.

3.1 Activation-space reconstruction

  • For calibration-guided compression, CoSpaDi targets layer-output reconstruction error rather than weight-space error and uses an activation-derived Gram factor to convert the objective to transformed-weight Frobenius reconstruction.The transformed problem removes explicit activations from the optimization.Source: §3.1 · p. 3

3.2 From SVD to sparse dictionary learning

  • SVD reconstructs every column in one shared subspace, whereas sparse dictionary learning uses a generally non-orthogonal dictionary and column-sparse codes so columns can select different atom subsets.Each column’s support selects its active atoms, yielding a union of at most s-dimensional subspaces.Source: §3.2 · p. 4

3.3 Optimization and grouped sharing

  • The non-convex factorization is optimized by alternating independent column-wise sparse coding with dictionary updates; grouped sharing concatenates related weights and calibration inputs before learning one dictionary.Sparse coding is approximated with batched OMP, and learned coefficients are sliced back into layer-specific blocks after the grouped update.Source: §3.3 · p. 4

3.4 Storage and inference cost

  • CoSpaDi storage includes dictionary values and sparse coefficients, while inference cost depends on active atoms and nonzeros as well as mask handling, memory traffic, and sparse–dense kernel efficiency.Figure 2 plots accuracy and WikiText perplexity against ρ; Figure 3 plots relative metrics against sparse-coefficient bitwidth.Source: §3.4 · p. 4, Figure 2 · p. 5, Figure 3 · p. 5

4 Experiments

The experiments compare CoSpaDi with low-rank and structured-pruning baselines across per-layer and grouped compression, alongside ablations and deployment tests. They report gains in accuracy and perplexity, while examining solver speed, coefficient storage, and throughput trade-offs.

4.1 Experimental Setup

  • Evaluation covers per-layer LLaMA and Qwen models and grouped LLaMA-2-7B compression, using 256 RefinedWeb calibration sequences and accuracy and perplexity metrics.The comparisons include SVD-LLM, truncated SVD, and structured-pruning baselines.Source: §4.1 · pp. 5–6

4.3.1 Per-Layer Compression

  • Across LLaMA3-8B and Qwen3-8B at matched compression ratios, CoSpaDi improves average benchmark accuracy and perplexity over SVD-LLM.Table 3 covers compression ratios 0.2–0.4 and multiple benchmarks.Source: Table 3 · p. 6, §4.3.1 · p. 7

4.2 Ablation Studies

  • ρ=2 provides the best observed accuracy/perplexity trade-off across tested compression ratios, and dictionary methods outperform corresponding SVD baselines in both data-free and activation-aware regimes.The data-free comparison is truncated SVD versus CoSpaDi†; the activation-aware comparison is SVD-LLM versus CoSpaDi.Source: §4.2 · p. 6
  • Power-iteration K-SVD provides the best overall accuracy, perplexity, and compression-time balance among solver variants; sequential dictionary updates dominate runtime.The solver comparison fixes factorization shape, hardware, batching, and stopping criteria.Source: §4.2 · pp. 6–7

4.2.1 Coefficient Quantization

  • Truncating 2 mantissa bits from sparse coefficients causes almost negligible degradation across tested compression ratios and offsets packed-mask overhead.The default setting uses ρ=k/s=2, bf16 dictionaries, and 14-bit sparse coefficients.Source: §4.2.1 · p. 7

4.3.2 Cross-Layer Dictionary Sharing

  • At all budgets, CoSpaDi substantially outperforms Basis Sharing and per-layer baselines, suggesting cross-layer sharing is most effective when paired with sparse codes letting each column and layer select its own atoms.Source: §4.3.2 · p. 7

4.3.3 End-to-end Throughput

  • At matched compression ratios, CoSpaDi beats SVD-LLM on quality but runs slower; asynchronous variants beat dense at most concurrency levels, with CoSpaDi retaining stronger quality.The synchronous comparison reports raw tokens per second; asynchronous results use prompt length 1.Source: §4.3.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8

5 Conclusions

CoSpaDi establishes activation-aware sparse dictionary learning as a post-training alternative to single-subspace low-rank compression. Its dense-times-sparse computation depends on sparse-runtime support, while the fixed allocation and solver leave room for complementary improvements.

5 Conclusions

  • CoSpaDi establishes activation-aware sparse dictionary learning as a post-training compression paradigm beyond single-subspace low-rank approximation.The current implementation uses fixed ρ, mostly uniform budgets, and a standard alternating solver.Source: §5 · p. 8
  • Dynamic allocation, improved calibration objectives, cross-layer sharing, and quantization-aware storage remain applicable without changing the union-of-subspaces factorization.Source: §5 · p. 8
  • The dense-times-sparse computation depends on sparse runtimes and kernel support; deployment shows measurable speedups over the dense baseline under hardware-aware choices.The authors characterize those deployment results as unlikely to represent the factorization’s final systems limit.Source: §5 · pp. 8–9

Limitations

The limitations concern the studied solver, budget allocation, and grouping choices; the authors characterize them as properties of this direct instantiation, not fundamental restrictions of sparse dictionary factorization.

Solver efficiency and scalability

  • CoSpaDi’s alternating sparse coding and per-atom dictionary refinement is more expensive than closed-form SVD, with sequential dictionary updates the main runtime bottleneck.The authors describe this as an algorithmic limitation and report that power-iteration updates offer a favorable accuracy–runtime trade-off.Source: Limitations · p. 9

Budget allocation and hyperparameter selection

  • Uniform compression budgets and a fixed ρ = k/s isolate the factorization effect, but layer-wise or type-wise allocation and adaptive ρ are possible extensions.Prior low-rank work shows that sensitivity-based allocation can improve quality at a fixed global budget.Source: Limitations · p. 9

Grouped sharing design

  • Grouped CoSpaDi uses a simple sharing scheme, and its performance can depend on which layers are clustered and which projections share a dictionary.The authors suggest grouping based on activation or weight similarity, projection type, or learned sharing patterns as possible improvements.Source: Limitations · p. 9

Limitations

  • These limitations reflect the direct instantiation studied, rather than fundamental restrictions of sparse dictionary factorization.Source: Limitations · p. 9

A.1 Data-Aware Low-Rank Weight Approximation

This section formulates calibration-aware low-rank weight approximation by transforming the activation objective into a standard matrix approximation problem. An invertible transform orthogonalizes the calibration inputs, allowing the transformed weights to be approximated before mapping back.

A.1 Data-Aware Low-Rank Weight Approximation

  • Activation-aware low-rank approximation targets weight outputs on calibration inputs because direct weight-space low-rank approximation can perform poorly on LLM tasks.The calibration matrix X contains the input vectors used to approximate XW.Source: §A.1 · p. 12
  • When X has full rank and N ≥ d, a nonsingular transform L yields Y = XL^-1 with orthonormal columns, so X = YL.The passage notes QR/SVD of X or Cholesky/eigenvalue decomposition of X^T X as ways to compute L.Source: §A.1 · p. 12
  • The transformed problem approximates Γ = LW by a rank-r matrix, then maps the solution back through L^-1 to obtain the optimal approximation of W.The rank-r solution is the orthogonal projection of Γ onto rank-r matrices.Source: §A.1 · p. 13
  • For this activation-aware solution, B = L^-1U_r is not an orthonormal basis because its columns are no longer orthonormal.Source: §A.1 · p. 13
  • If the empirical Gram matrix G = X^T X is not positive definite, diagonal shifting stabilizes it before Cholesky factorization.The transform is exact when G is positive definite; otherwise it corresponds to a numerically stabilized activation-space objective.Source: §A.1 · p. 13

A.2 Derivation of the Optimal Pair of Basis and Coefficient Matrices

The derivation minimizes reconstruction error for the columns of W using an orthonormal basis and coefficient matrix. It obtains the optimal coefficients for a fixed basis, then selects the basis from the leading singular vectors of W.

A.2 Derivation of the Optimal Pair of Basis and Coefficient Matrices

  • Each column w_j of W is approximated in a lower-dimensional subspace by minimizing total approximation error over the basis and coefficient matrix.The basis has dimensions d1 × r, and the coefficient matrix has dimensions r × d2.Source: §A.2 · p. 13
  • For a fixed orthonormal basis B, the optimal coefficient matrix is C = B^T W.This follows by setting the objective’s gradient with respect to C to zero.Source: §A.2 · p. 13
  • The optimal basis B is formed from the r eigenvectors of WW^T with largest eigenvalues, equivalently the top-r left singular vectors of W.The reduced right-singular-vector matrix V_r and top singular values in Σ_r specify the corresponding rank-r coefficient solution.Source: §A.2 · p. 14

A.3 Pseudo Algorithm of the Proposed Method

The proposed procedure uses calibration activations to whiten the reconstruction objective, then learns a dictionary and sparse codes for the whitened weights. It de-whitens the dictionary for storage and applies the resulting factorization with sparse–dense kernels.

A.3 Pseudo Algorithm of the Proposed Method

  • Given W, calibration activations X, and a target compression ratio, the procedure seeks an activation-aware factorization W ≈ D S.Source: §A.3 · p. 14
  • Whitening chooses L so Y = XL^-1 has orthonormal columns and converts the activation-aware loss into a standard Frobenius dictionary-learning objective on LW.Source: §A.3 · p. 14
  • The whitened dictionary is initialized, and the target ratio sets dictionary size k and exactly s nonzeros per coefficient column, optionally using fixed ρ = k/s.Source: §A.3 · p. 14
  • Alternating minimization uses OMP for sparse coding and sequential atom updates over each atom’s support, stopping at T iterations or when relative improvement falls below tolerance.Source: §A.3 · p. 14
  • After de-whitening, storage keeps the dictionary and nonzero coefficients in bf16 with a packed binary mask.Source: §A.3 · p. 14
  • Inference applies the factorization with sparse–dense kernels, reuses dictionary inner products across columns, and derives practical speedup from the number of active atoms.Source: §A.3 · p. 14

A.4 Grouped CoSpaDi details

Grouped CoSpaDi shares a dictionary across layers by transforming their weights into a common metric, then solves a sparse dictionary-learning problem. The shared-metric approximation makes this tractable for LLM-scale projections and uses the per-layer alternating optimization procedure.

A.4 Grouped CoSpaDi details

  • The exact shared-dictionary objective is expensive for LLM-scale projections, motivating a shared-metric approximation.Different layers induce different Gram metrics, and the exact dictionary update depends on them.Source: §A.4 · pp. 14–15
  • The approximation transforms and concatenates layer weights, yielding a grouped objective with at most s nonzeros per column.The resulting objective has the same form as the per-layer CoSpaDi objective.Source: §A.4 · p. 16
  • The grouped objective is solved with the same alternating OMP/K-SVD procedure used for per-layer CoSpaDi.The algorithm alternates sparse coding and sequential dictionary updates.Source: §A.4 · pp. 15–16
  • After optimization, the shared activation-aware dictionary is recovered and each layer is reconstructed as its dictionary times that layer’s sparse codes.The algorithm takes calibration inputs and outputs the activation-aware dictionary, sparse coefficients, and compressed weight matrix.Source: §A.4 · pp. 15–16

A.5 Derivation of the CoSpaDi Compression Ratio

This appendix details CoSpaDi’s storage accounting, inference costs, implementation and runtime trade-offs, and comparisons with SVD and pruning baselines, alongside convergence settings and a selective up/gate variant. It reports that the variant maintains model quality at matched global compression ratios and is compatible with synchronous and asynchronous serving, with throughput competitive with a low-rank counterpart and potential for latency improvement through sparse-kernel and runtime advances.

A.5 Derivation of the CoSpaDi Compression Ratio

  • With ρ = 2, truncating two coefficient mantissa bits exactly compensates the packed mask’s storage cost; without truncation or when ρ ≠ 2, compensation is not exact.The detailed breakdown says the two storage conventions leave total footprint essentially unchanged up to rounding, while reallocating storage between dictionary capacity and coefficient precision.Source: §A.5 · p. 16, §A.11 · p. 20, Table 10 · p. 21

A.6 Low-rank and CoSpaDi Inference Complexity

  • Under matched effective storage and the bound Kactive ≤ k, CoSpaDi’s nominal leading multiplication count matches low-rank compression, but this is not a latency prediction.Actual wall-clock performance depends on support overlap, indexing overhead, memory traffic, batching, and sparse-kernel efficiency.Source: §A.6 · p. 17

A.7 Relationship to Recent SVD-Based Extensions

  • SVD-LLM V2 (Wang et al., 2025c) and Dobi-SVD address axes orthogonal to CoSpaDi’s factorization change; CoSpaDi instead tests sparse-dictionary versus single-subspace representation.The paper primarily compares against SVDLLM, a strong activation-aware SVD baseline under a comparable post-training setting, to isolate factorization family.Source: §A.7 · pp. 17–18

A.9 Number of Alternating Minimization Steps and Power Iteration Analysis

  • With ρ = 2, performance on Llama3.2-1B stabilizes around 50 K-SVD iterations and five power iterations, motivating final settings of 60 and 8, respectively.The implementation uses batched OMP with mini-batches of 8192, 256 calibration sequences of length 1024, and bfloat16 dictionaries with truncated 14-bit coefficients unless specified otherwise.Source: §A.8 · p. 18, §A.9 · p. 19, Table 8 · p. 19

A.10 Runtime Breakdown and Scalability Discussion

  • Dictionary updates dominate CoSpaDi’s compression runtime, while sparse coding contributes a smaller but substantial share; the appendix identifies this cost as primarily solver-implementation dependent.Suggested acceleration directions include online or stochastic learning, Batch-OMP, and approximate rank-1 updates; compression-time efficiency remains future work.Source: §A.10 · p. 19, Table 9 · p. 19

A.15 From Theory to Practice: Deployment Considerations

  • Figure 5 plots average accuracy and WikiText perplexity against compression ratio for LLaMA-3.2-1B and Qwen-3 0.6B; the deployment variant selectively compresses up and gate projections.The selective setting concentrates compression on wider MLP projections and uses pretrained compressed checkpoints to replace only those modules in the dense model.Source: §A.12 · p. 20, Figure 5 · p. 21, §A.15 · p. 21

A.14 Comparison with semi-structured pruning baseline

  • On Llama3 8B at compression ratio 0.4, CoSpaDi has the strongest average accuracy, outperforming SVD-LLM and Wanda 2:4, with substantially better LAMBADA perplexity than both and better WikiText perplexity than SVD-LLM.Source: §A.14 · pp. 20–21, Table 12 · p. 22

A.13 More Benchmarks

  • On Qwen3-8B and Qwen3-14B reasoning benchmarks, CoSpaDi matches or improves on SVD-LLM across most benchmarks and settings.Table 13 reports results across several compression ratios; the supplied passages do not specify individual metric values.Source: §A.13 · p. 20, Table 13 · p. 22

A.15 From Theory to Practice: Deployment Considerations

  • On LLaMA-3-8B, selective CoSpaDi preserves uniformly compressed CoSpaDi’s quality and outperforms SVD-LLMUG at matched global compression ratios.This supports the benefit of the union-of-subspaces representation when compression is restricted to deployment-favorable projections.Source: §A.15 · p. 22, Table 14 · p. 23
  • In synchronous generation, SVD-LLMUG is faster in raw tokens/s, but CoSpaDiUG remains close while delivering substantially stronger quality at matched global compression ratios.Throughput was measured on an RTX 3090 across prompt lengths from 1 to 256 tokens.Source: §A.15 · p. 22, Table 15 · p. 23
  • In asynchronous serving, both compressed variants usually outperform the dense baseline, and CoSpaDiUG achieves the highest throughput at the largest tested concurrency.The comparison uses prompt length 1, UG CR = 0.60, and ρ = 2.Source: §A.15 · p. 22, Table 16 · p. 23
  • CoSpaDiUG’s asynchronous throughput remains stable across prompt lengths and scales with concurrent users, supporting compatibility with multi-request serving.The sweep uses UG CR = 0.60 and ρ = 2; CoSpaDi evaluates projections as (XDa)S, so sparse-kernel, coefficient-layout, and runtime improvements can directly improve latency.Source: §A.15 · p. 22, Table 17 · p. 23

Related papers

Loading 2509.22075v6…