PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Pruner: On the Structural Pruning of Large Language Models

Xinyin Ma, Gongfan Fang, Xinchao Wang

arXiv:2305.11627v3cs.CL

TL;DR

Large language models are costly to deploy and compress because their size and training corpora are enormous, while task-specific methods can sacrifice general-purpose capabilities. LLM-Pruner uses dependency-aware structural pruning with gradient-based importance estimation and LoRA recovery, retaining zero-shot abilities across several models with limited data and rapid compression. High pruning rates still cause substantial degradation and incoherent generation.

  • Problem

    LLMs’ substantial size creates deployment, inference, and training challenges, while task-specific compression can compromise their general-purpose task-solving ability.

  • Method

    LLM-Pruner detects dependent structures, estimates group importance using gradient information, prunes coupled structures, and uses LoRA for fast recovery with limited data.

  • Results

    LLM-Pruner retains zero-shot capabilities after pruning across LLaMA, Vicuna, and ChatGLM, including 94.97% of original accuracy for LLaMA after post-training.

  • Takeaways & Limitations

    The results support task-agnostic LLM compression with reduced dependence on the original training corpus and rapid recovery using limited data.

  • Takeaways & Limitations

    Removing 50% of LLaMA’s parameters causes substantial performance degradation and can produce incoherent sentences.

Abstract

from arXiv · show

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner

1 Introduction

LLM-Pruner targets task-agnostic compression that preserves an LLM’s diverse task-solving capabilities while reducing dependence on its enormous training corpus. It uses dependency-aware structural pruning and rapid recovery to compress models efficiently.

  • LLMs’ scale creates substantial computational demands, long latency, and deployment and inference challenges.
  • Task-specific compression preserves performance on designated tasks but compromises an LLM’s versatility as a general-purpose solver.
  • Task-agnostic compression seeks to reduce model size while preserving diverse capabilities as a general-purpose task solver.
  • LLM compression faces enormous training corpora, reaching 1 trillion tokens or more, and unacceptably long post-training durations.
  • LLM-Pruner detects dependent structures, estimates their importance, selects groups for pruning, and rapidly recovers the pruned model with limited data.
  • LLM-Pruner provides task-agnostic structured pruning using 50k public samples and completes compression within three hours across LLaMA, Vicuna, and ChatGLM.

2 Related Work

Prior language-model compression methods include pruning, distillation, quantization, early exit, and dynamic token reduction. Efficient and low-resource approaches reduce model size or training-data dependence, but some data-free distillation methods remain time-consuming.

  • Language-model compression methods include network pruning, knowledge distillation, quantization, early exit, and dynamic token reduction.
  • Structural pruning removes entire filters from neural networks and is the pruning approach emphasized in this work.
  • Prior efficient-compression methods accelerate post-training through reconstruction-error optimization or layer-wise optimal brain surgeon techniques.
  • Data-free pruning measures neuron similarity, while other methods distill models without relying on the original training corpus.
  • Data-free distillation methods can remain time-consuming because they synthesize samples by backpropagating through pretrained language models.

3 Methods

LLM-Pruner compresses LLMs by discovering coupled structures, estimating their group importance, pruning low-importance groups, and recovering performance with LoRA-based post-training.

  • Pipeline: LLM-Pruner follows discovery, estimation, and recovery stages to identify coupled structures, rank their importance, prune selected groups, and restore performance.The recovery stage uses fast post-training to alleviate pruning-related performance degradation.
  • Dependency Discovery: Dependency-based pruning removes coupled structures together, preventing partial pruning from increasing parameters or misaligning intermediate representations.A dependency graph identifies structures that must be pruned in unison when one structure depends solely on another.
  • Dependency Discovery: Dependency detection iteratively propagates activation from an initial trigger neuron until no new dependent neurons are found, forming a pruning group.The resulting groups can be automatically identified across different LLM structures.
  • Importance Estimation: Group importance is estimated from loss changes using public or manually created data, with gradient information favored because Hessian computation is impractical for LLMs.The method uses a small dataset D and evaluates the impact of coupled structures on next-token prediction loss.
  • Importance Estimation: Importance can be aggregated by summation, maximum, or last-only criteria, after which groups are ranked and lower-importance groups are pruned under a predefined ratio.The last-only criterion uses the importance of the last executing structure because deleting it erases the group’s computed results.
  • Recovery: LoRA recovery trains only low-rank matrices P and Q, reducing optimization complexity while allowing their updates to be reparameterized into the final weight matrix.The forward computation is f(x) = (W + ∆W)X + b = (WX + b) + (PQ)X.

4 Experiments

Experiments evaluate LLM-Pruner on LLaMA, Vicuna, and ChatGLM using zero-shot classification and perplexity, then analyze pruning strategies, importance estimation, recovery, and comparisons with alternative compression approaches. Across models, moderate pruning preserves substantial performance, while dependency-aware block pruning and efficient post-training are important for maintaining quality.

  • Experimental settings: The evaluation covers LLaMA-7B, Vicuna-7B, and ChatGLM-6B on seven zero-shot classification datasets plus WikiText2 and PTB perplexity.Models are assessed without task-specific fine-tuning, using zero-shot classification and perplexity analysis.
  • Zero-shot performance: 20% parameter pruning retains 89.8% of the unpruned LLaMA performance without post-training and reaches 60.07% classification accuracy, or 94.97% of the original, after post-training.The reported LLaMA results support compression with limited data-free degradation and recovery through tuning.
  • Zero-shot performance: 20% pruning on Vicuna-7B maintains 92.03% of the original model’s performance, while 10% pruning on ChatGLM-7B causes a 0.89% decrease that post-training can recover.The ChatGLM comparison is qualified by its limited English pre-training exposure and additional English data during post-training.
  • Ablations: Block pruning outperforms Channel pruning because transformer-layer importance is uneven, making uniform channel pruning damage the first and last layers disproportionately.The layer-sensitivity analysis motivates preserving sensitive boundary layers during structured pruning.
  • Pruning rates: LLM-Pruner tolerates pruning rates near 60% on LLaMA with perplexity comparable to the L2 baseline, whereas L2 collapses near 20%; Vicuna shows a similar advantage.For Vicuna-7B, removing 10% of parameters produces a decline equivalent to LLM-Pruner pruning 60%.
  • Ablations: Dependency-aware structural pruning is essential: removing dependencies nearly eliminates zero-shot generation and classification performance, even after tuning.The experiments select the Sum importance strategy as a trade-off between generalization quality and classification performance.

5 Conclusion

LLM-Pruner provides structured, task-agnostic compression across LLaMA, Vicuna, and ChatGLM, while retaining zero-shot capabilities. The paper also identifies substantial degradation at high pruning rates and illustrates dependency rules for coupled structures.

  • Conclusion: LLM-Pruner uses structured pruning and evaluates compressed LLaMA, Vicuna, and ChatGLM models on zero-shot datasets.The approach aims to reduce computational burden while preserving zero-shot capabilities.
  • Conclusion: Removing 50% of LLaMA’s parameters causes substantial performance degradation and can produce incoherent sentences.The paper identifies higher pruning rates as an unresolved challenge.
  • Dependency rules: The dependency rules require coupled nodes to be pruned together when their directional relationships satisfy the specified conditions.The rules apply in both directions, so pruning a later-layer node can also require pruning an earlier-layer node.
  • Dependency rules: Nodes that do not satisfy the dependency requirement can remain unaffected when a related node is pruned.In the third illustrated case, pruning Node J does not affect Node K because the degree condition is not met.

B Implementation Details

The evaluation compares LLM-Pruner with magnitude-based and random pruning over dependency groups. Implementation choices cover block and channel pruning, with LoRA recovery performed using a short, resource-conscious training setup.

  • Baselines: Because no prior baseline exists for task-agnostic, low-resource structural pruning of LLMs, the study evaluates L2, Random, and data-free pruning methods.All methods operate on dependent groups identified by the framework.
  • Block group: Block pruning leaves the first three and final layers unchanged because modifying them significantly impacts the model.Within MLP and multi-head attention modules, groups are pruned according to a preset ratio.
  • Channel group: Channel pruning removes selected dimensions, with dependency established automatically across query, key, value, and output projections.For query, key, and value projections, only input dimensions are pruned; for output projections, only output dimensions are pruned.
  • Recovery: LoRA recovery uses rank d = 8, learning rate 1e-4, 100 warming steps, and two training epochs on a single 24GB GPU.The reported runtime is approximately 2.5 hours on an RTX4090, and longer training can hurt performance.

C.1 More Data for Recovery

The recovery study compares 50k samples with 2.59 million samples and examines combining pruning with quantization. Additional data may improve recovery, but at higher computational cost.

  • More data for recovery: The paper compares model recovery using 50k samples versus a dataset containing 2.59 million samples.The authors state that additional data could substantially enhance recovery while increasing computational cost considerably.
  • Pruning and quantization: LLM.int8() outperforms LLM-Pruner, while LLM-Pruner reduces latency and parameter size.The two techniques can be combined to reduce memory consumption and accelerate inference with little performance degradation.

C.3 Global Pruning vs. Local Pruning

Global pruning ranks all groups together, whereas local pruning ranks groups within each module. In the LLaMA-7B comparison at 20% pruning, local pruning has a slight observed advantage.

  • Pruning strategies: At a 20% pruning ratio on LLaMA-7B, global pruning ranks all groups together, while local pruning ranks groups within each module.Global pruning produces varying widths across layers and modules; local pruning maintains uniformity across layers.
  • Results: Local pruning shows a slight advantage over global pruning in the reported experiments.The authors attribute this to incomparable importance-score magnitudes across different layers or modules.

C.4 Overfitting Phenomena in Post-Training

Recovery quality initially improves but later declines slightly, indicating that post-training completes quickly and can overfit its domain-constrained corpus. At a 50% pruning rate, performance drops substantially before recovery improves it by approximately 7.39%.

  • Overfitting during recovery: Accuracy or generation quality initially improves across training steps, then slightly declines, indicating overfitting during recovery.The recovery corpus is domain-constrained, so additional epochs may compromise the model’s original capabilities.
  • Large pruning rates: At 50% pruning, LLaMA-7B and Vicuna-7B show a significant performance decrease compared with their base models.The authors characterize pruning at this rate as challenging.
  • Large pruning rates: Recovery improves performance by approximately 7.39% after 50% parameter pruning on LLaMA-7B and Vicuna-7B.This improvement follows the substantial degradation observed immediately after pruning.

D Generations From Compressed Model

The compressed models are illustrated through generated examples from LLaMA, Vicuna, and bilingual ChatGLM, including outputs before and after post-training. These examples cover factual, instructional, technical, promotional, and personal-language prompts, while the unpost-trained outputs help expose retained information.

  • Example coverage: Generated examples are provided for compressed LLaMA-5.4B, Vicuna-5.4B, and ChatGLM-5.6B in English and Chinese.The examples are organized in Tables 18–21.
  • Post-training comparison: Examples with and without post-training are presented so the information retained in the pruned model can be examined.ChatGLM-6B results are shown in two languages because it is bilingual.
  • Example outputs: Other examples include neural-network descriptions, logo-marketing copy, romantic prose, and personal reflections about partnership.The collection spans technical, promotional, creative, and personal-language generation.
  • Example outputs: The examples include factual answers about McDonald’s and instructional guidance on adopting a cat.These outputs illustrate different generation prompts rather than a single specialized task.
Loading 2305.11627v3…