PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple and Effective Pruning Approach for Large Language Models

Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter

arXiv:2306.11695v3cs.CLcs.AIcs.LG

TL;DR

Large language models are difficult to prune efficiently because existing methods may require costly retraining or weight reconstruction, while magnitude pruning can fail at low sparsity. Wanda prunes pretrained LLMs using weight magnitudes combined with input-activation norms on a per-output basis, without retraining or weight updates, and performs strongly across LLaMA and LLaMA-2 benchmarks.

  • Problem

    Existing LLM pruning methods may require retraining or computationally expensive weight reconstruction, while magnitude pruning can fail even at relatively low sparsity.

  • Method

    Wanda removes weights with the smallest products of weight magnitudes and corresponding input-activation norms, comparing weights within each output.

  • Results

    Wanda outperforms magnitude pruning by a large margin and competes favorably with SparseGPT across LLaMA and LLaMA-2 evaluations.

  • Takeaways & Limitations

    Wanda identifies effective sparse networks from pretrained LLMs without retraining or weight updates, while requiring lower computational cost than the prior best pruning method.

  • Takeaways & Limitations

    The benefit of comparing weights per output may be specific to LLMs, since similar experiments on image classifiers did not show the same trend.

Abstract

from arXiv · show

As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

1 INTRODUCTION

LLM pruning is difficult because existing approaches often require costly retraining or weight updates, while magnitude pruning can fail even at low sparsity. Wanda addresses this by combining weight magnitudes with input activations and pruning locally, achieving strong results without modifying remaining weights.

  • Existing pruning methods commonly require retraining, random initialization, or extensive iterative weight updates, limiting their practicality for billion-parameter LLMs.
  • Magnitude pruning fails dramatically on LLMs even at relatively low sparsity, unlike its prior success on smaller networks.
  • Wanda scores each weight using its magnitude multiplied by the norm of the corresponding input activations, estimated from calibration data.
  • Wanda compares scores locally within each output of a linear layer and removes lower-priority weights without modifying the remaining weights.
  • Wanda is computationally efficient, requiring a single forward pass and minimal memory overhead.
  • Across LLaMA and LLaMA-2, Wanda outperforms magnitude pruning by a large margin and competes favorably with SparseGPT at lower computational cost.

2 PRELIMINARIES

Magnitude pruning induces sparsity by removing weights with the smallest magnitudes. LLMs also exhibit emergent, sparse hidden-state features with unusually large magnitudes once they reach sufficient scale.

  • Magnitude pruning removes individual weights whose magnitudes fall below a threshold determined locally within layers or globally across the network.
  • Once LLMs reach around 6B parameters, a small set of hidden-state features emerges with magnitudes far larger than typical values.
  • These outlier features are about 100 times larger than typical hidden-state values and are usually sparse in specific feature dimensions.

3 WANDA: PRUNING BY WEIGHTS AND ACTIVATIONS

Wanda prunes pretrained LLMs using weight magnitudes scaled by input activation norms, comparing weights separately for each output. It requires no weight updates and supports efficient unstructured or structured sparsity.

  • Pruning metric: Wanda scores each weight by multiplying its magnitude by the corresponding input feature's ℓ2 norm.The feature norm aggregates activations across batch and sequence tokens.
  • Pruning metric: The activation-aware metric preserves low-magnitude weights connected to unusually large-magnitude input features.This addresses a limitation of magnitude pruning in LLMs, where input feature scales can differ substantially.
  • Comparison group: Wanda compares importance scores within each output neuron and removes a predefined sparsity ratio from every output.This localized grouping is stricter than layer-wise pruning but consistently performs better for LLMs.
  • Empirical scope: Balanced pruning across output features is important for LLMs, whereas the advantage of per-output over per-layer pruning was not observed in image classifiers.This suggests the comparison-group finding may be specific to LLMs rather than general across model families.
  • Procedure: The method can be implemented in a single forward pass using calibration activations, with pruning decisions made layer by layer without iterative weight updates.Subsequent layers receive activations produced by the already-pruned preceding layers.
  • Structured N:M sparsity: Wanda extends naturally to structured N:M sparsity by comparing weights within every group of M consecutive weights connected to an output.This structure can leverage NVIDIA sparse tensor cores for practical matrix-multiplication acceleration.
  • Efficiency: Wanda's simplified metric avoids matrix inverses and has lower computational complexity than SparseGPT.The paper reports that computing Wanda's pruning metric can be 300 times faster than SparseGPT in practice.

4 EXPERIMENTS

The experiments evaluate Wanda on LLaMA and LLaMA-2 using zero-shot accuracy, language-modeling perplexity, pruning speed, and inference speed. Across these evaluations, Wanda outperforms magnitude pruning, competes with SparseGPT, and provides practical computational benefits without weight updates.

  • Models and Evaluation: Wanda is evaluated on LLaMA and LLaMA-2 model families across seven zero-shot tasks and language modeling.The evaluated families include LLaMA 7B/13B/30B/65B and LLaMA-2 7B/13B/70B.
  • Experimental Setup: The experiments compare Wanda with magnitude pruning and SparseGPT under unstructured, 4:8, and 2:4 sparsity settings.All methods prune linear layers, which account for around 99% of total LLM parameters, using uniform sparsity across those layers.
  • Zero-Shot Tasks: Wanda outperforms magnitude pruning by a large margin and rivals SparseGPT across unstructured and structured sparsities in mean zero-shot accuracy.Unstructured 50% sparse LLaMA-65B and LLaMA-2-70B match the zero-shot accuracies of their dense counterparts.
  • Language Modeling: Wanda finds sparse networks without weight updates, achieving LLaMA-7B perplexity of 7.26 versus 17.29 for magnitude pruning.At unstructured 50% sparsity, Wanda performs on par with SparseGPT, while structured-sparsity comparisons vary with model size.
  • Inference Speed: Structured 2:4 sparsity yields around 1.6× linear-layer inference speedup and 1.24× end-to-end speedup on LLaMA-7B.The end-to-end comparison is 251ms versus 312ms; the reported inference benefit comes from sparsity itself rather than uniquely from Wanda.
  • Speed and Robustness: Wanda's pruning-metric computation can be much faster than SparseGPT, while its calibration-based results are often more stable across random calibration sets.The robustness analysis evaluates five random seeds; the supplied experiment description reports lower variance for Wanda in many cases.

5 ANALYSIS

The analysis examines calibration robustness, fine-tuning, pruning configuration, and weight updates for Wanda. Wanda’s default configuration performs strongly, while fine-tuning can narrow—but does not necessarily eliminate—the gap to dense models.

  • Fine-tuning: Full-parameter fine-tuning improved unstructured 50% Wanda pruning from 54.21% to 58.15% zero-shot accuracy, close to the dense model’s 59.99%.The fine-tuning used a limited budget of four GPUs and three days.
  • Pruning Configuration: The default Wanda pruning configuration achieved the best pruned-model perplexity of 7.26 in the configuration ablation.For magnitude pruning, the (input, 1) comparison group achieved perplexity 8.86 and outperformed other grouping options.
  • Robustness to Calibration Samples: Wanda remained robust with few calibration samples, achieving perplexity 7.66 even with a single sample.The study varied calibration-set sizes from 1 to 256 samples.
  • Weight Update: Weight updates substantially improved magnitude pruning but provided no improvement to Wanda at unstructured 50% and 4:8 sparsities.At 2:4 sparsity, Wanda improved only marginally from 11.53 to 10.89; at 70% sparsity, updating improved Wanda from 84.50 to 29.65, still far behind dense perplexity 5.68.

6 RELATED WORK

Related work situates Wanda within network pruning, limited-data pruning, and research on emergent large-magnitude features in LLMs. Prior approaches often involve retraining or computationally expensive reconstruction and second-order updates.

  • Network Pruning and Sparsity: Network pruning compresses neural networks by eliminating weights and is broadly divided into structured and unstructured approaches.Structured methods remove organized network components, while unstructured methods operate at the individual-weight level.
  • Network Pruning and Sparsity: Existing pruning methods may require training modifications, retraining, or computationally intensive procedures to recover accuracy.These requirements have limited the straightforward application of pruning to large language models.
  • Pruning with Limited Data: Limited-data pruning avoids modifying the original training procedure and full-dataset retraining by using a small calibration set.Layer-wise reconstruction methods minimize output changes on calibration data, but existing solvers rely on heavy second-order Hessian inverses.
  • Pruning with Limited Data: SparseGPT addresses LLM pruning with an efficient weight-update procedure based on synchronized second-order Hessian updates.This places SparseGPT among limited-data methods that preserve performance through weight reconstruction or update.
  • Emergent Properties of LLMs: Large-magnitude outlier features emerge in LLMs beyond a scale such as 6B parameters and strongly affect all layers.These features motivate pruning approaches that account for activation behavior rather than weight magnitude alone.

7 CONCLUSION

The paper concludes that Wanda is a simple LLM-pruning method that identifies effective sparse networks without retraining or weight updates. It also points to sparse training as a setting where pruning efficiency may matter repeatedly.

  • Conclusion: Wanda removes weights with the smallest products of weight magnitudes and corresponding input activation norms on a per-output basis.The method is motivated by emergent large-magnitude features in LLMs.
  • Conclusion: Wanda identifies effective sparse networks within pretrained LLMs without retraining or weight-update procedures.The conclusion frames this as a contribution toward understanding sparsity in LLMs.
  • Conclusion: Wanda’s fast pruning makes sparse training a potential future application because pruning may need to be conducted repeatedly.The paper presents this as an open direction rather than an evaluated result.

A IMAGE CLASSIFIERS

The image-classifier study evaluates Wanda on pretrained ConvNeXt-B and DeiT-B models using one-shot pruning without retraining. Layer-wise pruning slightly outperforms per-output pruning, and Wanda’s metric improves over magnitude pruning, especially at high sparsity.

  • Experimental Setup: ConvNeXt-B and DeiT-B start at 83.8% and 81.8% top-1 accuracy, respectively, before pruning.The experiments prune only linear layers, including equivalent 1×1 convolution layers for ConvNeXt.
  • Experimental Setup: 4096 ImageNet-1K training images provide stable calibration results, with only marginal effects from additional samples.Accuracy is reported for one-shot pruned models without subsequent retraining.
  • Comparison Groups: Layer-wise pruning is slightly better than per-output pruning for both ConvNeXt-B and DeiT-B under both compared metrics.The comparison is shown in Figure 3.
  • Pruning Metrics: Wanda’s pruning metric outperforms magnitude pruning under layer-wise pruning, especially at 70% and 80% sparsities.The comparison is shown in Figure 4.

B WANDA ON PREVIOUS LLMS

Wanda is evaluated across OPT, BLOOM, and Pythia, where it remains effective across model families and pruning configurations. It also supports the finding that pruning per output is preferable to pruning per layer for OPT and BLOOM.

  • Cross-family evaluation: Wanda is evaluated on OPT, BLOOM, and Pythia models, including varying sparsity levels and model sizes.The experiments cover OPT and Pythia at 10%–50% sparsity and additional OPT and BLOOM model sizes.
  • Baseline comparison: Magnitude pruning fails catastrophically on OPT-13B and Pythia-12B even at 20% sparsity.This extends the observed limitations of magnitude pruning beyond the LLaMA family.
  • Baseline comparison: Wanda successfully prunes OPT, BLOOM, and Pythia to much higher sparsities without updating the retained weights.The result supports the existence of effective sparse subnetworks across these LLM families.
  • Pruning granularity: For OPT and BLOOM, experiments compare pruning per layer with pruning per output using the metric |W_ij| · ∥X_j∥.The comparison tests whether the proposed per-output pruning observation generalizes beyond LLaMA.

C ADDITIONAL BASELINES

The paper compares Wanda with activation-pruning methods and prior BERT-pruning approaches under settings designed to preserve pretrained language-model ability. These comparisons include both one-shot pruning and limited fine-tuning.

  • Activation pruning: Prior activation-pruning methods remove entire neurons using output statistics such as mean, standard deviation, correlation, or mean squared norm.These statistics are computed on a calibration set before removing neurons with smaller values.
  • Prior pruning methods: BERT-pruning methods commonly interleave pruning with fine-tuning and target downstream-task performance rather than general pretrained-model performance.The paper identifies this as a key distinction from its own focus.
  • Evaluation settings: The adopted prior methods are evaluated on LLMs using pre-training autoregressive loss to compute pruning metrics.The goal is to preserve language-modeling ability rather than optimize only a downstream task.
  • Evaluation settings: The comparison evaluates both one-shot pruning and one-shot pruning followed by fine-tuning within a limited computational budget.Fine-tuning is limited to one day.

D.1 NUMBER OF CALIBRATION SAMPLES

The paper examines how calibration-set size, random calibration seeds, and higher sparsity affect pruned-model performance. Calibration beyond 128 samples yields slight gains, while extreme sparsity creates a large gap from dense models.

  • Calibration-set size: 128 calibration samples is the default, and using more than 128 produces a slight performance improvement at 50% sparsity.This analysis is conducted on LLaMA-7B and LLaMA-2-7B.
  • Random-seed robustness: Wanda and SparseGPT are evaluated across five random calibration sets using perplexity means and standard deviations.The analysis tests robustness because both methods estimate input statistics from calibration data.
  • Higher sparsity: At 60% sparsity, Wanda remains competitive with SparseGPT.The 60% evaluation extends the main 50% unstructured-sparsity setting.
  • Higher sparsity: At 80% sparsity, SparseGPT outperforms Wanda, while the best 80% sparse model reaches perplexity 25.86 versus 5.68 for the smallest dense LLaMA-7B model.The reported gap suggests that a small dense model trained to convergence may be preferable at extreme sparsity.

D.4 FEW-SHOT RESULTS ON MMLU

Additional experiments assess Wanda in few-shot MMLU and task-wise settings alongside zero-shot results. At 50% unstructured sparsity, Wanda is competitive with SparseGPT, and larger sparse models can surpass smaller dense counterparts.

  • Few-shot MMLU: The few-shot evaluation uses 5-shot MMLU and reports mean accuracies for dense and 50% unstructured sparse models.MMLU is selected to evaluate in-context learning in addition to the main zero-shot experiments.
  • Few-shot MMLU: In the few-shot setting, Wanda performs competitively with SparseGPT.This comparison is reported for the MMLU evaluation.
  • Few-shot MMLU: Sparse LLaMA-13B and LLaMA-2-13B surpass dense LLaMA-7B and LLaMA-2-7B on the reported few-shot comparison.The passage states that this trend is not observed with standard magnitude pruning.
  • Task-wise results: Fine-tuning recovers most of the pruning-related performance drop on individual zero-shot tasks.The reported fine-tuning budgets are 12 hours for LoRA and 3 days for full-parameter fine-tuning.
  • Zero-shot task coverage: The zero-shot evaluation covers seven tasks, including BoolQ, RTE, HellaSwag, WinoGrande, ARC, and OpenbookQA.The task-wise results are reported for LLaMA and LLaMA-2 under several sparsity structures.
Loading 2306.11695v3…