PaperlayerView exact revision on arXiv

Source-linked AI summary

Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, Shiwei Liu

arXiv:2310.05175v4cs.LG

TL;DR

LLM pruning must determine whether uniform layerwise sparsity overlooks important differences among layers, particularly the non-uniform distribution of activation outliers. The paper studies this relationship and introduces OWL, which aligns each layer’s sparsity with its outlier ratio. Across LLaMA-V1 and OPT experiments, OWL outperforms prior pruning methods at high sparsity and accelerates CPU inference.

  • Problem

    Existing LLM pruning methods commonly use uniform layerwise sparsity, despite non-uniform outlier distributions and stronger non-uniform sparsity results in vision models.

  • Method

    OWL assigns non-uniform layerwise sparsity ratios according to each layer’s outlier ratio to preserve outliers.

  • Results

    Across LLaMA-V1 and OPT, OWL consistently outperforms existing pruning methods, surpassing Wanda and SparseGPT by 61.22 and 6.80 perplexity points at 70% sparsity.

  • Takeaways & Limitations

    Layerwise sparsity ratios are a crucial and previously overlooked factor for pushing one-shot LLM pruning to 70% sparsity and improving practical deployment.

Abstract

from arXiv · show

Large Language Models (LLMs), renowned for their remarkable performance across diverse domains, present a challenge when it comes to practical deployment due to their colossal model size. In response to this challenge, efforts have been directed toward the application of traditional network pruning techniques to LLMs, uncovering a massive number of parameters that can be pruned in one-shot without hurting performance. Prevailing LLM pruning strategies have consistently adhered to the practice of uniformly pruning all layers at equivalent sparsity, resulting in robust performance. However, this observation stands in contrast to the prevailing trends observed in the field of vision models, where non-uniform layerwise sparsity typically yields stronger results. To understand the underlying reasons for this disparity, we conduct a comprehensive study and discover a strong correlation with the emergence of activation outliers in LLMs. Inspired by this finding, we introduce a novel LLM pruning methodology that incorporates a tailored set of non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise sparsity (OWL). The sparsity ratio of OWL is proportional to the outlier ratio observed within each layer, facilitating a more effective alignment between layerwise weight sparsity and outlier ratios. Our empirical evaluation, conducted across the LLaMA-V1 family and OPT, spanning various benchmarks, demonstrates the distinct advantages offered by OWL over previous methods. For instance, OWL exhibits a remarkable performance gain, surpassing the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high sparsity level of 70%, respectively, while delivering 2.6x end-to-end inference speed-up in the DeepSparse inference engine. Codes are available at https://github.com/luuyin/OWL.

1. Introduction

LLM pruning offers a way to reduce the size and computational demands of billion-parameter models, but existing methods commonly apply uniform sparsity across layers. OWL instead uses non-uniform layerwise sparsity informed by outlier distributions, improving pruning results at high sparsity.

  • LLMs’ exceptional capabilities come with substantial model size, computational demands, financial costs, and environmental concerns.
  • Traditional network pruning often requires fine-tuning or retraining, which is especially challenging for LLMs.
  • SparseGPT and Wanda use uniform layerwise sparsity, pruning every layer at the same ratio despite differences among Transformer components.
  • Non-uniform sparsity is common in vision models, while LLMs exhibit emergent behaviors and outlier features that may require layer-specific treatment.
  • OWL assigns greater emphasis to layers with more outliers, aligning layerwise weight sparsity with the observed outlier distribution.
  • 2.6×–3.9× end-to-end CPU speedup is achieved with OWL at 70%–90% sparsity in the DeepSparse inference engine.

2. Related Work

Prior LLM pruning typically uses uniform layerwise sparsity, whereas vision-model pruning has extensively explored non-uniform alternatives. Research on LLM outliers identifies a sparsely distributed feature pattern important to prediction but scarcely explored for pruning.

  • Traditional LLM pruning commonly uses uniform layerwise sparsity, while vision-model research has developed several non-uniform alternatives.
  • Global pruning methods can be extremely expensive and ineffective for LLMs.
  • LLM outliers are features with significantly larger magnitudes than others and constitute only a very small portion of feature dimensions.
  • Although few, outliers play an imperative role in models’ predictive performance and have motivated quantization methods designed to preserve them.
  • Outliers have been scarcely explored in LLM pruning, apart from limited prior work.

3. Outlier Weighed Layerwise Sparsity

OWL addresses the mismatch between highly non-uniform layerwise outlier distributions and uniform LLM pruning by assigning constrained, outlier-aware sparsity ratios. Empirical studies connect outlier retention and pruning granularity to performance, while experiments show OWL improves perplexity across methods and model families.

  • Outlier measurement: OWL computes weight outlier scores as Aij = ∥Xj∥2 · |Wij| and derives layerwise outlier ratios from weights exceeding M times each layer’s mean score.The method uses M = 5 or M = 7 to sketch the weight-outlier distribution.
  • Empirical Study I: Dense LLMs exhibit highly non-uniform layerwise outlier distributions, motivating differential treatment of layers during pruning.The distribution loosely follows a U shape, with notable proportions at both ends and a descending central region.
  • Empirical Study II: Pruning performance closely correlates with outlier retention: Wanda and SparseGPT increase LOD, whereas magnitude pruning changes LOD by -0.110 at 70% sparsity.SparseGPT achieves the highest LOD and performance despite not being explicitly designed for outlier preservation.
  • Empirical Study III: Finer pruning granularity generally improves perplexity compared with global or coarser layerwise sparsity.The observed trend motivates constraining layerwise sparsity differences around the target sparsity.
  • OWL design: OWL assigns lower sparsity to layers with higher outlier proportions, while constraining each layer to [S − λ, S + λ] and preserving average sparsity S.This design aligns layerwise sparsity with outlier distribution while limiting aggressive differences between layers.

4. Experiments

Experiments evaluate OWL across model families, pruning methods, sparsity levels, downstream tasks, fine-tuning settings, and deployment scenarios. OWL generally improves pruning performance, especially at high sparsity, while adding negligible pruning-time overhead and enabling substantial inference speedups.

  • Language modelling: OWL reduces WikiText perplexity across pruning methods, architectures, and model sizes, including a 6.81 reduction for LLaMA-7B when combined with SparseGPT.The reported gains extend to a second-order pruning method rather than only Wanda.
  • Language modelling: 61.22 perplexity points separate OWL from Wanda for LLaMA-7B, compared with a 6.62 gain for LLaMA-65B.The LLaMA-7B result reaches 24.55 perplexity, and the reported benefit increases as model size decreases.
  • Zero-shot tasks: At 70% sparsity, OWL improves accuracy across nearly all zero-shot settings and averages gains of 4.72 over Wanda and 2.19 over SparseGPT across seven tasks and four model sizes.RTE is noted as an exception among the evaluated datasets.
  • Efficiency: OWL incurs at most 2 seconds of pruning-metric overhead relative to prior approaches and achieves 2.6× end-to-end decode speedup at 70% sparsity in DeepSparse.The reported speedup is around 4× at 90% sparsity on an Intel Xeon Platinum 8360Y CPU with 36 cores.
  • Practical applications: OWL-SVD assigns less compression to layers with higher LoD scores and surpasses uniform SVD compression across evaluated rank-reduction levels without fine-tuning.This extends OWL’s layer-importance principle beyond unstructured pruning.

5. Analysis

The analysis compares OWL with alternative layerwise sparsity rules and examines when its outlier-based allocation helps. OWL is strongest at higher sparsity, while reversing its outlier ratio substantially degrades performance.

  • Layerwise sparsity comparisons: Global, Uniform, Erdős-Rényi, ER-plus, and OWL-inverse provide comparison schemes for layerwise sparsity allocation.OWL-inverse uses an outlier ratio of 1−LOD, while ER-based rules scale sparsity according to layer connectivity.
  • Layerwise sparsity comparisons: Below 40% sparsity, most methods perform similarly on LLaMA-7B with Wanda, but differences become evident above 50%.The Global method is the exception in the low-sparsity region.
  • Layerwise sparsity comparisons: OWL consistently outperforms Uniform across the tested sparsity levels, whereas the Erdős-Rényi family is less suitable for LLM pruning.Inverting OWL’s outlier ratio causes a significant performance decline, supporting LOD’s role in identifying critical layers.
  • Vision models: OWL’s improvement is less pronounced in vision models, where outliers are not particularly evident.This contrast is consistent with the paper’s emphasis on outlier structure in LLMs.

6. Conclusion

OWL introduces non-uniform layerwise sparsity for LLM pruning by aligning each layer’s sparsity ratio with its outlier ratio. At 70% sparsity, it substantially surpasses Wanda and SparseGPT in perplexity.

  • OWL aligns each layer’s sparsity ratio with its outlier ratio to preserve outliers during LLM pruning.
  • 61.22 perplexity points separate OWL from Wanda at 70% sparsity.
  • 6.80 perplexity points separate OWL from SparseGPT at 70% sparsity.
  • OWL opens avenues for specialized sparse algorithms that further optimize practical LLM deployment.

7. Impact Statements

Effective layerwise sparsity can reduce SOTA LLM parameters by up to 70% while retaining essential functionality. The resulting sparsity supports deployment on constrained devices and faster inference across commodity hardware.

  • Up to 70% of SOTA LLM parameters can be reduced while retaining essential functionality.
  • Effective layerwise sparsity facilitates deployment on resource-constrained devices and expedites inference.
  • Real speed-up extends the benefits of sparsity beyond GPUs to commodity hardware such as CPUs and FPGAs.

A. Vision Model Pruning

The appendix evaluates OWL with Wanda on ConvNeXt-Base and DeiT-Base using one-shot ImageNet-1K pruning without fine-tuning. OWL improves DeiT accuracy but shows less pronounced benefits for vision models overall.

  • OWL is evaluated with Wanda on ConvNeXt-Base and DeiT-Base using one-shot pruning without fine-tuning.
  • The evaluation uses ImageNet-1K to compare OWL with uniform layerwise sparsity.
  • OWL enhances sparse DeiT accuracy compared with Wanda, whereas ConvNeXt accuracy neither increases nor degrades.
  • OWL’s performance improvement is less pronounced on vision models than on LLMs, possibly because vision-model outliers are less evident.

B.1. N:M Sparsity

For mixed N:8 sparsity, OWL assigns distinct per-layer N values while maintaining the overall parameter count. It consistently improves over uniform N:M sparsity, especially at high sparsity.

  • OWL assigns individual layers distinct N values in a mixed N:8 configuration while keeping the overall parameter count constant.
  • OWL consistently enhances performance compared with uniform N:M sparsity.
  • 2× perplexity reductions occur with OWL at 3:8 sparsity over the uniform baseline.
  • 8× perplexity reductions occur with OWL at 2:8 sparsity over the uniform baseline.

B.2. Structured Pruning

The paper applies structured pruning by removing entire groups of weights to support hardware acceleration, and extends OWL to mixed-precision quantization across layers.

  • Structured pruning removes groups such as weight blocks, neurons, filters/channels, or attention heads rather than individual weights.The implementation follows LLM Pruner by removing entire neurons and attention heads.
  • Removing entire neurons and attention heads facilitates direct acceleration of pruned LLMs on GPUs or TPUs.
  • Perplexity is evaluated for structured pruning with LLaMA-7B on WikiText and PTB.
  • OWL assigns higher precision to layers exhibiting more outliers in a mixed-precision quantization strategy.The baselines select layers randomly or according to the L1 norm of weights.
  • Perplexity is evaluated for mixed-precision quantization with LLaMA-7B on WikiText.

C. Per-Block Vs. Per-Layer

The paper compares assigning pruning ratios per Transformer block with assigning them per layer. At 70% sparsity, the reported perplexities favor the per-block configuration over the per-layer configuration.

  • Per-Block Vs. Per-Layer: The method assigns a distinct pruning ratio for each Transformer block rather than for each layer.
  • Per-Block Vs. Per-Layer: The comparison reports layerwise sparsity for seven fully connected layers across Transformer layers 1, 2, 15, 30, and 31.The seven layers are q proj, k proj, v proj, o proj, gate proj, down proj, and up proj.
  • Per-Block Vs. Per-Layer: At 70% sparsity, per-layer OWL reports perplexity 86.285 for LLaMA-7B.
  • Per-Block Vs. Per-Layer: At 70% sparsity, per-block OWL reports perplexity 24.55 for LLaMA-7B.
  • Per-Block Vs. Per-Layer: The experiments use hyperparameters listed in Table 17.
Loading 2310.05175v4…