PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Elias Frantar, Dan Alistarh

arXiv:2301.00774v3cs.LG

TL;DR

Massive GPT models are difficult to deploy because of their size and computational cost, while accurate pruning has not scaled effectively to them. SparseGPT formulates pruning as sparse regression and solves it efficiently in one shot. On OPT-175B and BLOOM-176B, it reaches 50-60% sparsity with low accuracy loss, dropping more than 100B weights, while also supporting semi-structured sparsity and quantization.

  • Problem

    Accurate pruning methods had not scaled effectively to billion-parameter GPT models because retraining and existing one-shot approaches were computationally expensive.

  • Method

    SparseGPT reduces layer-wise pruning to large-scale sparse regression and solves it with an efficient approximate solver without retraining.

  • Results

    50-60% sparsity is reached on OPT-175B and BLOOM-176B with low accuracy loss, dropping more than 100B weights.

  • Takeaways & Limitations

    SparseGPT identifies accurate sparse models near dense GPT models and shows that larger models are more compressible at fixed sparsity.

  • Takeaways & Limitations

    Evaluation uses 128 randomly selected 2048-token C4 segments as generic calibration data, without task-specific data during pruning.

Abstract

from arXiv · show

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.

1. Introduction

SparseGPT addresses the lack of accurate, scalable one-shot pruning for billion-parameter GPT models. It reaches high sparsity with minor accuracy loss, including challenging semi-structured patterns and combined quantization.

  • Motivation: 175 billion-parameter GPT models require at least 320GB in FP16 and at least five 80GB A100 GPUs for inference.These deployment costs motivate model compression.
  • Motivation: Accurate existing pruning methods require expensive retraining, while prior one-shot methods become impractical for models with billions of parameters.The paper identifies essentially no accurate pruning work at billion-parameter scale before SparseGPT.
  • Approach: SparseGPT reduces pruning to large-scale sparse regression and solves it with an approximate solver efficient enough for 175B models on a single GPU.The method is designed for one-shot pruning without fine-tuning.
  • Results: 50-60% sparsity is induced in OPT-175B and BLOOM-176B in one shot, with minor loss measured by perplexity or zero-shot accuracy.The result applies to the largest publicly available generative language models.
  • Results: 60% uniform layer-wise sparsity is reached on OPT-175B with minor accuracy loss, whereas magnitude pruning preserves accuracy only to 10% and collapses beyond 30%.SparseGPT also handles 2:4 and 4:8 semi-structured patterns, though smaller models incur accuracy loss relative to dense baselines.
  • Results: 50% sparsity can be combined with 4-bit weight quantization on OPT-175B with negligible perplexity increase.SparseGPT is therefore compatible with quantization-based compression.

2. Background

Post-training pruning seeks compressed models from optimized weights using calibration data, but scaling accurate methods to massive models is difficult. SparseGPT uses layer-wise sparse regression approximations to make this computation tractable while preserving accuracy.

  • Post-Training Pruning: Post-training pruning compresses a well-optimized model using calibration data, producing sparse and/or quantized weights.This setting is presented as a practical compression scenario.
  • Layer-Wise Pruning: Layer-wise pruning minimizes each compressed layer's output error relative to the uncompressed layer for given inputs, then stitches the compressed layers together.A sparsity mask and reconstructed weights define each layer's compressed solution.
  • Mask Selection & Weight Reconstruction: The joint optimization of pruning masks and remaining weights is NP-hard, so practical methods rely on approximations.Fixing the mask turns reconstruction into a linear squared-error problem.
  • Difficulty of Scaling: Prior post-training techniques target models up to a few hundred million parameters, whereas this work aims to sparsify models up to 1000× larger.Scaling existing methods to 175B Transformers can require several hundred hours or a few weeks.
  • SparseGPT: SparseGPT uses a layer-wise solver based on careful approximations to closed-form equations, scaling to giant models in runtime and accuracy.Its reconstruction procedure can iteratively prune weights using optimal updates derived from the quadratic row-wise loss.

3. The SparseGPT Algorithm

SparseGPT makes one-shot layer-wise pruning practical for massive GPT models by approximating Hessian-based reconstruction while preserving error compensation. It reuses synchronized inverse Hessians, supports semi-structured sparsity, and jointly performs sparsification and quantization.

  • Iterative Reconstruction: SparseGPT reframes fixed-mask layer-wise pruning as sparse reconstruction and uses iterative OBS updates to optimally compensate for each removed weight under a quadratic loss.After pruning a weight, the remaining weights are adjusted to compensate for its removal; repeated OBS updates produce the reconstruction incrementally.
  • Motivation: The original reconstruction problem is computationally prohibitive because different row masks require separate O(d_col × d_col) Hessian inversions, yielding O(d_row · d_col^3) complexity and fourth-power scaling in the hidden dimension.The inverse of a masked Hessian is not generally the masked full inverse, while shared masks would harm accuracy because large-structure pruning is harder.
  • Fast Approximate Reconstruction: The resulting algorithm combines column-wise pruning with rightward error-compensating updates and achieves a practical complexity sufficient for extremely large models.Its design is interpreted as an exact column-wise greedy scheme, and the reconstruction strategy reaches a complexity described as d_hidden-factor more efficient than exact reconstruction.
  • Partial Updates: The method restricts compensation to subsets of unpruned weights, preserving optimality for the restricted quadratic problem while reducing update cost.Using the Hessian for the selected subset U updates only those weights; the restriction can reduce compensation effectiveness because fewer weights are available.
  • Hessian Synchronization: SparseGPT synchronizes Hessian information by recursively removing feature indices from nested subsets and reusing each inverse Hessian across rows with different masks.The sequence starts with all indices, removes one index at a time, and updates inverse Hessians efficiently through Gaussian elimination.
  • Extensions: SparseGPT extends to n:m semi-structured sparsity and performs pruning and quantization jointly in one pass, allowing later decisions to reflect earlier rounding.For n:m patterns, the mask selector enforces exactly n zeros in each consecutive block of m weights.

4. Experiments

Experiments evaluate SparseGPT on OPT and BLOOM models using perplexity and ZeroShot accuracy, showing that large models can sustain substantial sparsity with limited degradation. SparseGPT also supports semi-structured sparsity and joint sparsity–quantization compression.

  • Experimental setup: SparseGPT fully sparsifies 175-billion-parameter models in approximately 4 hours on a single NVIDIA A100 GPU.The experiments use sequential Transformer-layer sparsification and calibration data from C4.
  • Pruning versus model size: At 66B parameters, SparseGPT produces essentially zero perplexity loss, while the largest models can even show a dataset-specific slight improvement over the dense baseline.The largest-scale improvement is explicitly qualified as dataset specific.
  • Pruning versus model size: At 50% unstructured sparsity, larger OPT models generally lose less accuracy, while 4:8 and 2:4 patterns incur perplexity increases of 0.11 and 0.39 at the largest scale.The study evaluates 50% unstructured, full 4:8, and full 2:4 sparsity across the OPT family.
  • Sparsity scaling: SparseGPT enables up to 60% sparsity on OPT-175B, whereas magnitude pruning reaches at most 10% before significant accuracy loss.On BLOOM-176B, SparseGPT delivers 50% sparsity, a 1.66× improvement over magnitude pruning at similar perplexity degradation.
  • ZeroShot evaluation: Across ZeroShot tasks, magnitude-pruned models collapse toward random performance, while SparseGPT models remain close to the original accuracy.The ZeroShot results are described as noisier than perplexity evaluations, with effects averaging out across many tasks.
  • Joint sparsification and quantization: 50% sparsity plus 4-bit quantization is more accurate than size-equivalent 3-bit quantization on OPT models with at least 2.7B parameters.For OPT-175B, the reported perplexities are 8.29 for 50% + 4-bit and 8.68 for 3-bit.

5. Related Work

Related work established pruning methods for smaller models and post-training compression, but accurate GPT-scale pruning remained constrained by retraining and computational cost. SparseGPT is positioned as a post-training pruning method without finetuning for GPT-scale models.

  • Pruning methods: Existing high-performing pruning methods generally require extensive retraining to recover accuracy, which is costly for GPT-scale models.The paper identifies this cost as a reason accurate pruning of massive GPT models had received little attention.
  • Post-training compression: Prior post-training pruning methods had been investigated mainly for classic CNN or BERT-scale models, whereas SparseGPT targets GPT-scale models without finetuning.The paper contrasts this setting with post-training quantization work on billion-parameter GPT models.

6. Discussion

SparseGPT enables massive GPT-family models to reach high sparsity through one-shot weight pruning without retraining and with low accuracy loss. The results also indicate that larger models are easier to sparsify, with 50% sparsity causing practically no accuracy decrease in the largest models.

  • SparseGPT compresses large GPT-family models to high sparsity through one-shot weight pruning without retraining and with low loss of accuracy.Accuracy is measured using perplexity and zero-shot performance.
  • 50–60% sparsity on OPT-175B and BLOOM-176B removes more than 100B weights with low accuracy fluctuations.
  • Larger models are easier to sparsify: at fixed sparsity, their relative accuracy drop narrows as model size increases.At 50% sparsity, the largest models show practically no accuracy decrease.

A. Ablation Studies

Ablations show that SparseGPT is generally robust to calibration-data choices and Hessian dampening, while calibration quantity and mask-selection blocksize have clearer operating trade-offs.

  • Amount of Calibration Data: More calibration samples improve accuracy initially, but gains flatten quickly after a certain point.Even a few 2048-token segments produce decent results.
  • Hessian Dampening: Hessian dampening values from 0.001 to 0.1 perform similarly, while very high dampening significantly decreases solution quality.The method uses 0.01 dampening as a conservative choice for inverse calculations.
  • Mask Selection Blocksize: Reasonable mask-selection blocking outperforms column-wise blocksize 1 and near-full blocksizes 4096 and 8192.Blocksizes around a few hundred are slightly more accurate, motivating the choice of blocksize 128.
  • Sensitivity to Random Seeds: Five 50% pruning runs with different calibration-data seeds yield 13.52 ± 0.075 perplexity, indicating robustness to sampling choices.

A.1. Approximation Quality

SparseGPT’s partial-update approximation is evaluated against exact reconstruction at 50% sparsity on OPT-2.7B, with layer-wise reconstruction errors compared under the same mask and Hessian.

  • SparseGPT reconstruction error is compared with exact reconstruction error layer by layer for the first half of OPT-2.7B.Both reconstructions use the same mask and Hessian at 50% sparsity.

B. Evaluation Details

Evaluation details define perplexity using concatenated tokenized datasets split into 2048-token segments, while the supplied passages identify the datasets used and a reconstruction-error figure.

  • Figure 11 reports SparseGPT reconstruction error relative to exact reconstruction for the first half of OPT-2.7B at 50% sparsity.
  • Perplexity: Perplexity is computed by exponentiating the average language-modeling loss over non-overlapping 2048-token segments.Samples are concatenated and encoded with each model’s matching tokenizer before segmentation.
  • Datasets: The evaluation uses WikiText2, PTB, and a C4 subset with dataset-specific concatenation and sampling procedures.

C. Additional Results

Additional results show that SparseGPT maintains strong perplexity under alternative datasets and joint sparsification–quantization settings. The method also outperforms GPTQ in one reported 2.5-bit comparison.

  • Additional datasets: 50% sparsity on PTB and C4 follows similar trends to the main results, except the largest models show no slight perplexity decrease relative to the dense baseline.The decrease at 50% sparsity is identified as dataset-specific.
  • Joint sparsification and quantization: 8.60 PPL on raw-WikiText2 at 50% sparse plus 3-bit quantization outperforms GPTQ’s 8.94 state-of-the-art 2.5-bit result.This setting corresponds to 2.5-bit storage.
  • Joint sparsification and quantization: 8.93 PPL is achieved for the 4:8 sparse plus 3-bit configuration.The paper presents this as an initial investigation of combining sparsity and quantization.

D. Partial 2:4 Results

SparseGPT supports partial 2:4 sparsity, allowing only selected consecutive layer segments to be sparsified while the remaining layers stay dense. This creates a binary trade-off between inference speedup and accuracy.

  • D. Partial 2:4 Results: Partial 2:4 sparsity sparsifies an initial fraction of layers while keeping the remaining layers dense.The experiments cover OPT-175B and BLOOM-176B.
  • D. Partial 2:4 Results: Speedup and accuracy can be traded off through binary compression choices such as n:m pruning.The fraction of sparsified layers controls this trade-off.

E. Sparsity Acceleration

The paper preliminarily evaluates practical CPU and GPU acceleration for sparse language models using off-the-shelf tools. CPU results are close to the theoretical optimum, while further model-specific optimization remains future work.

  • E. Sparsity Acceleration: Preliminary CPU and GPU studies evaluate how well sparse language models can already be accelerated with off-the-shelf tools.The authors identify more model-specific optimization as an important topic for future work.
  • CPU Speedups: OPT-2.7B CPU speedups are measured end-to-end with DeepSparse for a single batch of 400 tokens on 18 CPU cores.Support for larger model variants was still under development in this experiment.
  • CPU Speedups: CPU speedups are close to the theoretical optimum, suggesting that unstructured sparsity acceleration for LLM inference on CPUs is already practical.The comparison uses sparse and dense models in the same DeepSparse environment.
  • GPU Speedups: 2:4 sparsity theoretically offers 2× matrix-multiplication acceleration on NVIDIA Ampere and newer GPUs.The paper evaluates practical speedups using CUTLASS and compares them with optimized dense cuBLAS results.
Loading 2301.00774v3…