PaperlayerView exact revision on arXiv

Source-linked AI summary

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman

arXiv:2401.15024v2cs.LGcs.CL

TL;DR

Large language models are expensive to deploy, while existing sparsification methods can require extra structures or provide limited hardware speedups. SliceGPT uses computationally invariant transformations to delete rows and columns and reduce embedding dimensions. Experiments report substantial compression, faster inference, and preserved task performance.

  • Problem

    Large language models impose high compute and memory costs, while existing pruning methods can require recovery fine-tuning or offer limited practical speedups.

  • Method

    SliceGPT applies a prediction-preserving transformation, projects inter-block signals onto principal components, and removes rows or columns from weight matrices.

  • Results

    SliceGPT maintains competitive perplexity and downstream performance while reducing inference cost to 64% on 24GB RTX6000 GPUs and 66% on 40GB A100 GPUs for LLAMA-2 70B.

  • Takeaways & Limitations

    Sliced models require fewer GPUs and run faster without additional code optimization or recovery fine-tuning in the reported setting.

  • Takeaways & Limitations

    The evaluation omits continuous batching and model sharding, so dense-model inference time could improve more than the sliced model in GPUms.

Abstract

from arXiv · show

Large language models have become the cornerstone of natural language processing, but their use comes with substantial costs in terms of compute and memory resources. Sparsification provides a solution to alleviate these resource constraints, and recent works have shown that trained models can be sparsified post-hoc. Existing sparsification techniques face challenges as they need additional data structures and offer constrained speedup with current hardware. In this paper we present SliceGPT, a new post-training sparsification scheme which replaces each weight matrix with a smaller (dense) matrix, reducing the embedding dimension of the network. Through extensive experimentation, we show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models. Code is available at: https://github.com/microsoft/TransformerCompression

1 INTRODUCTION

SliceGPT is a post-training pruning method that removes entire rows or columns after an invariant network transformation, reducing embedding dimensions while retaining competitive performance. Experiments show compression of large language models without recovery fine-tuning.

  • LLM deployment is costly because large models require substantial compute, memory, multiple GPUs, and repeated autoregressive forward passes.
  • Recovery fine-tuning is not required: SliceGPT compresses large models on a single GPU in a few hours while retaining competitive generation and downstream performance.
  • SliceGPT transforms the network before pruning so predictions remain invariant while entire weight-matrix rows or columns can be removed.The method reduces the embedding dimension and the sizes of signals passed between transformer blocks.
  • SliceGPT applies orthogonal-matrix transformations to transformer weight matrices without changing the model, introducing computational invariance.
  • SliceGPT projects inter-block signals onto principal components, then removes rows or columns of transformed matrices to reduce model size.The transformation and weight removal together are called SliceGPT.
  • 30% slicing compresses OPT and LLAMA-2 models while achieving superior perplexity to the 2:4 scheme and maintaining >90% of dense downstream performance across tested models.On WikiText-2, 25% slicing outperforms SparseGPT 2:4 for all tested LLAMA-2 models, while 30% slicing does so for nearly all tested OPT models.

2 BACKGROUND

Transformers pass signal matrices through attention, normalization, and feed-forward blocks, whose linear operations define the main computational structure. Background work covers transformer components and compression methods including pruning, quantization, and low-rank approximation.

  • 2.1 TRANSFORMER NETWORKS: Transformer layers alternate multi-head self-attention and feed-forward blocks, with normalization and residual connections between them.
  • 2.1 TRANSFORMER NETWORKS: The initial signal X has shape N × D, where D is the embedding dimension and N is the sequence length.Token and position IDs index the embedding matrices to produce X.
  • 2.1 TRANSFORMER NETWORKS: Attention projects inputs into key, query, and value matrices, applies multi-head attention, and uses input and output linear operations denoted Win and Wout.
  • 2.1 TRANSFORMER NETWORKS: FFN blocks apply a linear transformation, an element-wise nonlinearity, and a second linear transformation, including gated variants represented with Win and Wout.
  • 2.1 TRANSFORMER NETWORKS: The language-model head computes logits from the final transformer-block output using XWhead + bhead.
  • 2.1 TRANSFORMER NETWORKS: The forward pass repeatedly applies block nonlinearities, normalization, and residual connections to signal matrices before the model head returns predictions.
  • 2.2 RELATED WORK: Related compression methods include magnitude pruning, Hessian-informed pruning, quantization, semi-structured sparsity, low-rank approximation, and structured pruning with fine-tuning.Unstructured pruning can be difficult to accelerate end-to-end, while low-rank methods replace matrices with products of smaller matrices.

3 SLICEGPT

SliceGPT exploits computational invariance in RMSNorm-connected transformers to transform signals and weight matrices without changing the network, then remove dimensions using PCA-based slicing. LayerNorm models are first converted to RMSNorm, and block-specific transformations enable smaller matrices despite residual connections.

  • 3.1 COMPUTATIONAL INVARIANCE IN TRANSFORMER NETWORKS: RMSNorm commutes with orthogonal transformations, so inserting Q before normalization and Q⊤ after it preserves the network computation.This property follows because orthogonal matrices preserve vector norms.
  • 3.1 COMPUTATIONAL INVARIANCE IN TRANSFORMER NETWORKS: Computational invariance allows orthogonal transformations of transformer weights without changing the model's output.The transformation can be absorbed into neighboring linear layers, while residual connections require corresponding transformations across adjacent layers.
  • 3.2 LAYERNORM TRANSFORMERS CAN BE CONVERTED TO RMSNORM: LayerNorm transformers are converted to RMSNorm by absorbing scale and mean-subtraction operations into adjacent weight matrices without affecting network output.Output matrices receive mean subtraction, input matrices absorb preceding scales, and embeddings and the head are adjusted accordingly.
  • 3.3 A TRANSFORMATION PER BLOCK: Different orthogonal matrices Qℓ are applied per block because signals across blocks are not aligned.Residual connections receive Qℓ−1⊤Qℓ, adding a small D × D overhead that is needed for slicing and still permits overall speedup.
  • 3.3 A TRANSFORMATION PER BLOCK: PCA derives each block's orthogonal matrix from calibration signals, ordering eigenvectors by decreasing eigenvalues.The transformed signal is projected onto principal components before dimensions are removed.
  • 3.4 SLICING: Slicing applies a deletion matrix to preceding and succeeding operations, removing rows of Win and columns of Wout and Wembd.The deletion matrix provides a lower-dimensional representation with an L2-optimal linear reconstruction.

4 EXPERIMENTAL VALIDATION

SliceGPT is evaluated across language generation, zero-shot tasks, throughput, inference time, and compression cost using OPT, LLAMA-2, and Phi-2 models. Results show competitive accuracy and perplexity alongside reduced GPU requirements and faster dense-kernel inference, with calibration and recovery fine-tuning affecting outcomes.

  • Experimental setup: OPT and LLAMA-2 experiments use WikiText-2, while zero-shot evaluation additionally includes Phi-2 across five tasks.Calibration uses WikiText-2 or Alpaca; zero-shot tasks are PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c.
  • Zero-shot tasks: 30% slicing removes only a few percentage points from the largest OPT and LLAMA-2 models on mean zero-shot accuracy.OPT models are more compressible than LLAMA-2 models, while accuracy loss is less pronounced for larger models.
  • Recovery fine-tuning: 74.3% average accuracy is achieved by LLAMA-2 70B sliced 30% with Alpaca RFT, versus 76.6% for the dense model.The sliced model has approximately 51.6B parameters; Alpaca RFT performs better than WikiText-2 RFT.
  • Recovery fine-tuning: 65.2% average accuracy is achieved by Phi-2 with 25% slicing and RFT, versus 72.2% for the dense model, retaining 90.3% of accuracy.The sliced model has approximately 2.2B parameters, and Alpaca recovers several percentage points more than WikiText-2.
  • Throughput: 25% slicing achieves up to 1.55× throughput improvement, while 50% slicing lets the largest models use one GPU instead of two.At fixed GPU counts, the reported throughput reaches 6.26× and 3.75× that of dense models for the largest models.
  • Inference time: 25% slicing speeds single-token inference by 16–17% on RTX6000 GPUs and 11–13% on A100 GPUs.For LLAMA-2 70B on RTX6000 GPUs, compute falls from 1764 GPUms to 1075 GPUms, or 64% of dense compute.
  • Compute cost: SliceGPT compression and recovery fine-tuning take 1 to 5 hours total, with slicing alone taking 1 to 3 hours on one GPU.The compute-cost setup uses 30% slicing and Alpaca recovery fine-tuning.

5 CONCLUSION AND FUTURE WORK

SliceGPT enables structured pruning that reduces inference cost and GPU requirements while retaining substantial model performance. The authors identify computational invariance as a basis for future efficiency research and suggest complementary methods for further gains.

  • 5 CONCLUSION AND FUTURE WORK: 66% inference cost on 40GB A100 GPUs for LLAMA-2 70B, reducing GPU requirements from 4 to 3 without additional code optimization.On 24GB RTX6000 GPUs, inference cost falls to 64%, requiring 5 rather than 7 GPUs.
  • 5 CONCLUSION AND FUTURE WORK: At 25% slicing, OPT 66B, LLAMA-2 70B, and Phi-2 retain 99%, 96%, and 87% of dense zero-shot performance, respectively.Recovery fine-tuning raises LLAMA-2 70B and Phi-2 to 99% and 90%, respectively.
  • 5 CONCLUSION AND FUTURE WORK: Further gains could combine SliceGPT with quantization, structural pruning, or improved methods for computing Q.The authors also note that smaller dense models can outperform similarly sized pruned models in some settings.
  • 5 CONCLUSION AND FUTURE WORK: Computational invariance may support future research on improving deep-learning efficiency and inspire new theoretical insights.The authors frame this as a prospective opportunity rather than a demonstrated additional result.

A.1 PROOF OF EQUATION 2

The proof uses orthogonal transformations and norm preservation to show that RMSNorm commutes with the transformation, preserving the normalized representation.

  • A.1 PROOF OF EQUATION 2: An orthogonal matrix Q satisfies Q^⊤Q = QQ^⊤ = I, so multiplying a vector by Q preserves its norm.The norm is defined as the square root of the sum of squared vector elements.
  • A.1 PROOF OF EQUATION 2: For a row x of X, the corresponding row of XQ is Q^⊤x, whose norm equals the norm of x.Orthogonal transformations therefore change orientation without changing row magnitude.
  • A.1 PROOF OF EQUATION 2: RMSNorm divides each row by its norm, and multiplying afterward by Q^⊤ restores the original normalized row.The proof uses QQ^⊤ = I to obtain 1/∥x∥x.

A.2 SINGLE PRECISION EIGENVALUE CALCULATION

The appendix examines numerical precision in the PCA step used by SliceGPT. FP32 PCA can introduce numerical errors that affect the accuracy of larger models, motivating double-precision calculation.

  • A.2 SINGLE PRECISION EIGENVALUE CALCULATION: Double precision is used for PCA to mitigate potential numerical errors in computing SliceGPT’s orthogonal matrix.The impact of lower-precision PCA on ultimate accuracy remains an open question.
  • A.2 SINGLE PRECISION EIGENVALUE CALCULATION: FP32 PCA can affect the accuracy of larger OPT and LLAMA-2 models through numerical errors during eigenvector and eigenvalue calculation.The reported results use PyTorch torch.linalg for these calculations.
  • A.2 SINGLE PRECISION EIGENVALUE CALCULATION: Table 4 reports WikiText2 perplexity for OPT and LLAMA-2 using FP32 PCA with calibration size 128 and sequence length 2048.The table provides the appendix’s empirical setting for assessing single-precision PCA.

A.3 SENSITIVITY TO THE CALIBRATION SET SIZE AND SEQUENCE LENGTH

Calibration-set size and sequence length both affect SliceGPT’s WikiText2 perplexity. The experiments support using sufficiently large calibration sets and longer sequences, with the main experiments using size 1024 and length 2048.

  • A.3 SENSITIVITY TO THE CALIBRATION SET SIZE AND SEQUENCE LENGTH: At least 128 calibration samples provide sensible choices for the calibration set in the tested 25%-sparsity generation experiments.The ablation uses OPT 6.7B and LLAMA-2 7B.
  • A.3 SENSITIVITY TO THE CALIBRATION SET SIZE AND SEQUENCE LENGTH: Longer calibration sequences from 128 to 4096 can result in better perplexity.With B samples and sequence length N, the PCA input matrix contains NB embedding vectors, exposing a tradeoff between sample count and sequence length.
  • A.3 SENSITIVITY TO THE CALIBRATION SET SIZE AND SEQUENCE LENGTH: The main experiments use calibration-set size 1024 and sequence length 2048.This setting follows the calibration ablation findings.
  • A.3 SENSITIVITY TO THE CALIBRATION SET SIZE AND SEQUENCE LENGTH: Reducing calibration-set size degrades WikiText2 perplexity across all evaluated models and sizes.Table 5 evaluates OPT and LLAMA-2 with calibration size 128 and sequence length 2048.

A.4 SPECTRUM ANALYSIS OF LLAMA-2 AND OPT MODELS

Spectrum analysis finds that LLAMA-2 has a more tightly compressed embeddings spectrum than OPT, while eigenvalue decay varies across layers and motivates layer-specific slicing. Varying slicing levels by layer improves OPT perplexity but worsens it for LLAMA-2.

  • Spectrum comparison: LLAMA-2 has a more tightly compressed embeddings spectrum than OPT despite comparable parameter counts.The spectrum lacks dominant principal components, making pruning more challenging.
  • Layerwise decay: Except for LLAMA-2’s first layer, both models show faster eigenvalue decay in early than later layers.This suggests that early layers may tolerate more slicing after orthogonal transformation.
  • Layerwise slicing: Layer-specific slicing sets rows and columns to remove by discarding a target fraction of variance during each PCA calculation.Three target-variance experiments were used to approach a 25% total network reduction.
  • Results: Varying the slicing level by layer improves WikiText-2 perplexity for OPT but worsens it for LLAMA-2.The comparison is reported in Table 6 using a calibration set of 128 and each model’s maximum sequence length.

A.5 DETAILED ZERO-SHOT RESULTS

The zero-shot evaluation covers OPT, LLAMA-2, and Phi-2 across slicing and recovery fine-tuning conditions using WikiText2 and Alpaca calibration data. The supplied passages identify the relevant result tables and their evaluation settings.

  • WikiText2 slicing: Zero-shot results are presented for OPT, LLAMA-2, and Phi-2 models sliced using the WikiText2 dataset.These results are reported in Table 7.
  • Alpaca slicing: A separate evaluation reports zero-shot performance after slicing with the Alpaca dataset.The compared models are OPT, LLAMA-2, and Phi-2 in Table 8.
  • Recovery fine-tuning: Recovery fine-tuning after slicing is evaluated on LLAMA-2 and Phi-2 using WikiText2 and Alpaca datasets.The corresponding results appear in Tables 9 and 10.

A.6 BENCHMARKING THROUGHPUT EXPERIMENT

The throughput experiment benchmarks SliceGPT against dense and SparseGPT 2:4 matrix multiplications on GPU hardware. Reported tables compare architecture-specific timing, speedup, and perplexity trade-offs across model sizes and slicing levels.

  • Experimental setup: Throughput benchmarking on 80GB H100 GPUs uses sequence length 128 and doubles batch size until memory exhaustion or throughput decline.The maximum throughput is recorded under this procedure.
  • Experimental setup: Matrix multiplication runtimes are measured on an 80GB A100 GPU over 103 attempts using sequence length 2048.Dense PyTorch, smaller dense SliceGPT, and CuSparseLT sparse multiplications are compared by median runtime.
  • LLAMA-2 benchmark: LLAMA-2 timing includes gated-FFN up, down, and gated projections plus differently sized query versus key-value attention multiplications.The architecture-specific table reports per-operation and total times with relative speedup.
  • LLAMA-2 results: For larger LLAMA-2 models, SliceGPT at 25% matches SparseGPT 2:4 speedup with better WikiText-2 perplexity.For smaller models, SparseGPT 2:4 is faster but has worse perplexity; 50% slicing increases speedups while trading off perplexity.
  • OPT benchmark: OPT timing uses equal-sized Key, Value, Query, and Out multiplications with two MLP multiplications, FC1 and FC2.This architecture is benchmarked using the same general matrix-multiplication approach.
  • OPT results: For larger OPT models, SliceGPT at 25% provides slightly better speedup than SparseGPT 2:4 with better WikiText-2 perplexity.For smaller models, SparseGPT 2:4 is faster but has worse perplexity; 50% slicing increases speedups while trading off perplexity.
Loading 2401.15024v2…