PaperlayerView exact revision on arXiv

Source-linked AI summary

HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs

Mehdi Makni, Kayhan Behdin, Zheng Xu, Natalia Ponomareva, Rahul Mazumder

arXiv:2502.00899v1stat.MLcs.LG

TL;DR

Large foundation models are costly to serve, motivating compression of their dense weights. HASSLE-free directly optimizes a layer-wise sparse-plus-low-rank reconstruction objective with scalable alternating minimization. It improves reported LLM benchmark performance, including perplexity and zero-shot task results, over existing methods.

  • Problem

    Serving large foundation models requires substantial storage and computation, and sparse-plus-low-rank compression methods have not directly optimized the local layer-wise reconstruction objective introduced here.

    Source: §1 · pp. 1–2

  • Method

    HASSLE-free uses scalable alternating minimization to decompose pretrained weights into sparse and low-rank components while directly minimizing local layer-wise reconstruction error.

    Source: §1 · p. 2

  • Results

    12% lower WikiText-2 test perplexity and a 15% smaller dense-model gap on the average of eight zero-shot tasks for Llama3-8B at 2:4 sparsity plus rank 64, compared with existing methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HASSLE-free outperforms existing sparse-plus-low-rank LLM decomposition methods across a wide range of evaluation benchmarks, especially on perplexity.

    Source: §6 · p. 9

    For the diagonal-approximation setting, the stated assumption requires the input activation matrix X to be full-rank, with no identically zero columns.

    Source: §4.2.2 · p. 6

Abstract

from arXiv · show

The impressive capabilities of large foundation models come at a cost of substantial computing resources to serve them. Compressing these pre-trained models is of practical interest as it can democratize deploying them to the machine learning community at large by lowering the costs associated with inference. A promising compression scheme is to decompose foundation models' dense weights into a sum of sparse plus low-rank matrices. In this paper, we design a unified framework coined HASSLE-free for (semi-structured) sparse plus low-rank matrix decomposition of foundation models. Our framework introduces the local layer-wise reconstruction error objective for this decomposition, we demonstrate that prior work solves a relaxation of this optimization problem; and we provide efficient and scalable methods to minimize the exact introduced optimization problem. HASSLE-free substantially outperforms state-of-the-art methods in terms of the introduced objective and a wide range of LLM evaluation benchmarks. For the Llama3-8B model with a 2:4 sparsity component plus a 64-rank component decomposition, a compression scheme for which recent work shows important inference acceleration on GPUs, HASSLE-free reduces the test perplexity by 12% for the WikiText-2 dataset and reduces the gap (compared to the dense model) of the average of eight popular zero-shot tasks by 15% compared to existing methods.

1. Introduction

HASSLE-free formulates one-shot foundation-model compression as sparse-plus-low-rank decomposition that directly minimizes local layer-wise reconstruction error. It combines efficient alternating minimization with GPU-compatible sparsity and improves reported LLM evaluation results over existing methods.

1. Introduction

  • HASSLE-free targets improved algorithms for one-shot sparse-plus-low-rank decomposition, whereas Mozaffari et al. [34] focus on accelerating LLM pretraining with specialized CUDA kernels.Those kernels can also be exploited for faster acceleration and reduced inference memory footprint in this setting.Source: §1 · p. 2
  • Unlike prior work that minimizes an approximation of the objective, HASSLE-free minimizes the introduced reconstruction objective directly and supports unstructured, semi-structured, or structured sparsity.The framework mostly focuses on N:M sparsity, which can be accelerated on GPUs.Source: §1 · p. 2
  • HASSLE-free directly minimizes local layer-wise reconstruction error for a sparse-plus-low-rank decomposition of pretrained dense weights, using alternating minimization for the sparse and low-rank components.It uses a pruning method as a plug-in for sparse minimization and gradient-descent-type methods for the low-rank subproblem.Source: §1 · p. 2
  • 12% lower WikiText-2 test perplexity and a 15% smaller dense-model gap on the average of eight zero-shot tasks for Llama3-8B at 2:4 sparsity plus rank 64, compared with existing methods.The decomposition combines a 2:4 sparse component with a rank-64 component.Source: §1 · p. 3
  • The framework connects approximations of its optimization subproblems to OATS, Wanda, and MP [20] [33], while OATS [33] is inspired by Wanda [20].The introduction notes that Wanda can fail in high-sparsity regimes [18] and that the utility-compression difference is especially pronounced at higher compression.Source: §1 · pp. 2–3

2. Related Work

Prior work spans pruning structures, sparse-plus-low-rank decomposition, and one-shot LLM compression. HASSLE-free builds on OATS by incorporating more calibration-data information into the decomposition.

2. Related Work

  • Pruning yields unstructured, semi-structured, or structured sparsity; these structures differ in flexibility, hardware-friendliness, and potential performance degradation.The paper mostly considers N:M sparsity, while supporting other sparsity structures.Source: §2 · p. 3
  • Sparse-plus-low-rank decomposition has been studied in robust PCA and deep learning, then applied to LLM fine-tuning and sparse-plus-low-rank model compression [26] [30] [32] [33].Li et al. [32] studied utility of fine-tuning, while Zhang and Papyan [33] studied model compression.Source: §2 · p. 3
  • Prior one-shot LLM decomposition includes data-free or data-aware quantized-plus-low-rank methods [36] [37] and OATS [33], which uses Wanda-inspired outlier information from calibration data.This paper builds on OATS [33] to incorporate more calibration-dataset information in the decomposition.Source: §2 · p. 3

3. Problem Formulation

The formulation seeks sparse and low-rank weights whose layer outputs reconstruct those of the dense layer on calibration activations. The objective is subject to sparsity and rank constraints.

3. Problem Formulation

  • For a matrix Z, C_r(Z) denotes the rank-r truncation formed from its top singular vectors and values, and the Eckart–Young theorem gives the best rank-at-most-r approximation in the stated norm.The notation section also defines rk(Z) as matrix rank.Source: §3 · p. 3
  • The trace of a square matrix is the sum of its diagonal entries, while diag(Z) retains those entries on a diagonal matrix and sets off-diagonal entries to zero.The notation also defines the all-ones and all-zeros vectors.Source: §3 · p. 4
  • The decomposition minimizes the reconstruction error between dense- and compressed-layer outputs over calibration inputs, with a target sparsity constraint on the sparse weights and a rank constraint on the low-rank component.The activation matrix X contains inputs to the layer, corresponding to outputs of the preceding layer.Source: §3 · p. 4

4. Algorithm Design

The algorithm alternates sparse and low-rank updates to minimize local layer-wise reconstruction error, whose scale makes computationally efficient methods necessary. It considers data-free, diagonal-Hessian, and full-Hessian variants, then combines pruning with low-rank optimization.

4. Algorithm Design

  • Alternating minimization separates the decomposition into a sparse-component subproblem and a low-rank-component subproblem, addressed iteratively.The sparse update solves (P1), followed by the rank-constrained low-rank update (P2).Source: §4 · p. 4

4.1. Minimizing Subproblem (P1)

  • With an identity Hessian, hard-thresholding gives an optimal data-free sparse update; with a diagonal Hessian, hard-thresholding D W̃ yields Wanda’s approximation.The diagonal-Hessian approximation is also used in OATS’s sparse update.Source: §4.1.1 · p. 5, §4.1.2 · p. 5

4.1.3. Full Hessian: H = X⊤X + λI

  • Full-Hessian pruning targets the layer-wise objective directly, and HASSLE-free uses a pruning algorithm as a plug-in to solve its sparse subproblem.SparseGPT is the pruning method used in the numerical results.Source: §4.1.3 · p. 5

4.2. Minimizing Subproblem (P2)

  • For the low-rank subproblem, the data-free version has a truncated-SVD solution, while the diagonal-Hessian version has a closed-form minimizer when the activation matrix has no zero columns.OATS uses the same low-rank operation and minimizes the objective with a diagonal Hessian approximation.Source: §4.2.1 · p. 5, §4.2.2 · p. 6

4.2.3. Full Hessian: H = X⊤X + λI

  • For the full-Hessian low-rank update, reparameterizing M as UV⊤ enables first-order optimization; diagonal scaling addresses ill-conditioning and can speed convergence.The equivalent scaling sets the new Hessian’s diagonal to 1 and permits a shared learning rate across layers and models.Source: §4.2.3 · p. 6

4.3. Our Proposed Approach

  • The proposed procedure alternates SparseGPT-based sparse minimization with rescaled Adam optimization of the low-rank factors, reusing the Hessian and warm-starting updates for efficiency.The Hessian and its rescaled version remain invariant across iterations, so construction and inversion costs are incurred once and amortized.Source: §4.3 · p. 7

5. Experimental Results

The experiments evaluate one-shot compression on Llama-3 and Llama-3.2 models, comparing methods across perplexity and zero-shot tasks. The supplied passages describe the evaluation settings and Table 1’s scope, but not its outcome values.

5.1. Experiment Setup

  • Evaluations cover Llama-3 and Llama-3.2 models from 1 to 8 billion parameters, using shared calibration data and no retraining.The Hessian is constructed from 128 randomly sampled 2048-token segments of C4 training data.Source: §5.1 · pp. 7–8
  • Perplexity is measured on WikiText-2, PTB, and C4 validation data, while eight zero-shot tasks are evaluated and averaged.The tasks are PIQA, ARC-E, ARC-C, HellaSwag, Winogrande, RTE, OpenbookQA, and BoolQ.Source: §5.1 · p. 8

5.2. Results

  • The results settings compare HASSLE-free with OATS at 80 alternating-minimization steps and include a Llama3-8B comparison at 50% compression.The fixed-rank setting uses rank 64; the fixed-compression-ratio setting also compares against SparseGPT, Wanda, and DSNoT.Source: §5.2 · p. 8, Table 1

C4 WT2 PTB PIQA ARC-E ARC-C

The reconstruction-error analysis compares methods on layers of Llama-3-8B’s first transformer block under a fixed 2:4 sparse plus rank-64 decomposition. The supplied text identifies the metric and comparison conditions but does not report which method has the lowest error.

5.3. Reconstruction error on a single Transformer block

  • The analysis compares local layer-wise reconstruction error for OATS, scaled HASSLE-free, and unscaled HASSLE-free after 80 iterations.Lower error is preferred.Source: §5.3 · p. 8, Figure 1 · p. 9
  • Table 2 describes one-shot N:M sparse plus rank-64 evaluations on Llama3 and Llama3.2, with perplexity and zero-shot metrics.The caption states that lower perplexity and higher zero-shot scores are preferred.Source: Table 2 · p. 9
  • The comparison fixes the decomposition to 2:4 sparsity plus a rank-64 low-rank component on Llama-3-8B’s first transformer block.All methods use the same 80 alternating-minimization steps.Source: §5.3 · p. 8, Figure 1 · p. 9

6. Conclusion

HASSLE-free is a one-shot sparse plus low-rank decomposition framework that minimizes local layer-wise reconstruction error without approximating the objective. The experiments report that it outperforms existing decomposition methods across LLM benchmarks, especially perplexity.

6. Conclusion

  • HASSLE-free uses alternating minimization to optimize the local layer-wise reconstruction objective without objective-level approximations.The framework exploits Hessian invariance and diagonal rescaling for efficiency.Source: §6 · p. 9
  • Experiments report that HASSLE-free outperforms existing sparse plus low-rank LLM decomposition methods across a wide range of benchmarks, especially perplexity.Source: §6 · p. 9
  • Future work could extend the sparse subproblem to quantized-sparse compression to study optimization-based decomposition of dense weights into a compressed component plus a low-rank component.Source: §6 · p. 9

A.1. Experimental Setup

The experiments minimize the layer-wise objective sequentially using activations from previously compressed layers, then compare HASSLE-free with SparseGPT and OATS implementations and use a shared evaluation strategy for benchmarks.

A.1. Experimental Setup

  • For each layer, Equation (1) is minimized sequentially using activations produced by the previously compressed layers across N calibration samples.Source: §A.1 · p. 15
  • SparseGPT’s author implementation is used as a pruning plug-in to minimize (P1), with the Hessian constructed as H = X⊤X.Source: §A.1 · p. 15
  • OATS is evaluated using its author implementation and default hyperparameters for benchmarks and layer-wise reconstruction error.The reported LLM benchmarks in Table 1 follow ALPS’s evaluation strategy and code; Table 3 reports all zero-shot task results for OATS and HASSLE-free.Source: §A.1 · p. 15

A.2. Hyperparameter Choice

The setup uses specified defaults for regularization, optimization, pruning, and scaling, with some values proposed by the authors and others inherited from SparseGPT or OATS.

A.2. Hyperparameter Choice

  • λ is set to 0.01 Tr(H), the default value in SparseGPT.Source: §A.2 · p. 15
  • TAM is set to 80, the OATS default, while TLR is set to 50 as the authors’ proposed default for all experiments.Source: §A.2 · p. 15
  • η is set to 1e−2 as the proposed default, and this works well only with the scaling introduced in Section 4.2.3.Source: §A.2 · p. 15
  • SparseGPT is the proposed default pruning plug-in, Adam the proposed optimizer, and is_scaled is set to True by default for faster convergence and to skip learning-rate tuning.Source: §A.2 · p. 15

A.3. Additional Experimental Resuls

The additional experiments cover N:M sparsity with fixed compression ratio and unstructured sparsity with fixed rank ratio; the supplied passages describe table scope and metric directions but not comparative cell values.

A.3. Additional Experimental Resuls

  • Table 3 reports one-shot N:M sparse plus low-rank decomposition for Llama3 and Llama3.2 at fixed compression ratio ρ = 0.5.The accompanying text says this setting extends earlier results to eight zero-shot tasks and Llama3.2.Source: §A.3 · p. 15, Table 3 · p. 16
  • For both tables, lower perplexity and higher zero-shot task scores are preferred.Source: Table 3 · p. 16, Table 4 · p. 16
  • Table 4 reports one-shot unstructured sparsity plus low-rank decomposition for Llama3 and Llama3.2-3B at fixed rank ratio κ = 0.3.The setting uses uniform unstructured sparsity throughout layers and does not apply OWL ratios.Source: §A.3 · p. 15, Table 4 · p. 16

Related papers

Loading 2502.00899v1…