PaperlayerView exact revision on arXiv

Source-linked AI summary

Fluctuation-based Adaptive Structured Pruning for Large Language Models

Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang

arXiv:2312.11983v1cs.CLcs.AI

TL;DR

Retraining-free LLM pruning has largely focused on unstructured methods, limiting hardware-friendly structured compression. FLAP introduces fluctuation-based adaptive structured pruning with bias compensation and significantly outperforms prior structured pruning methods across language benchmarks without retraining.

  • Problem

    Most retraining-free LLM pruning methods use unstructured pruning, while structured pruning still requires effective importance metrics and adaptive global structure search.

  • Method

    FLAP uses fluctuation-based structured importance scores, adaptive global compression, and baseline bias compensation to recover pruned output feature maps without retraining.

  • Results

    Across language benchmarks on LLaMA, FLAP significantly outperforms prior structured pruning methods without retraining, achieving 31.80 perplexity for LLaMA-7B at 50% pruning.

  • Takeaways & Limitations

    FLAP maintains perplexity and zero-shot performance without retraining, with bias compensation becoming more important at higher pruning ratios.

Abstract

from arXiv · show

Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.

Introduction

FLAP addresses LLM deployment constraints with retraining-free structured pruning that combines structured importance estimation, adaptive global structure search, and bias compensation. It avoids retraining and significantly outperforms LLM-Pruner and structured Wanda extensions across language benchmarks.

  • Motivation: Structured pruning removes entire weight rows or columns, reducing parameters and inference time without requiring specialized hardware.The paper contrasts this with unstructured pruning, which requires specific hardware support for acceleration.
  • Design criteria: Effective structured pruning requires a structured importance metric, adaptive global compression search, and compensation to limit performance degradation.These criteria correspond to discovering redundancy, selecting the global compressed structure, and recovering performance.
  • FLAP framework: FLAP estimates fluctuation-based structured importance, standardizes scores across layers and modules, and adaptively determines the global compressed model structure.The method identifies structured sample stability in hidden-state channels and uses it to guide pruning.
  • Compensation: FLAP adds bias terms from baseline values to recover pruned layers’ output feature maps without retraining.Pruning and bias compensation require only a single forward pass, maintaining low memory overhead.
  • Evaluation: Without retraining, FLAP significantly outperforms LLM-Pruner and structured Wanda extensions across language benchmarks on the LLaMA model family.The framework is evaluated on a variety of language benchmarks.

Related Works

Related work frames pruning as a model-compression strategy for reducing neural-network redundancy, while emphasizing that conventional structured pruning depends on costly retraining. LLM-specific properties and billion-scale resource requirements make retraining-free structured pruning especially important.

  • Network Pruning: Pruning removes redundant network structures or parameters using importance metrics and performance-recovery methods, with unstructured and structured pruning as its two categories.Unstructured pruning operates at the individual-weight level, according to the passage.
  • Structured Pruning: Conventional structured pruning typically relies on retraining, creating memory and computational scalability challenges for billion-scale LLMs.The passage therefore identifies retraining-free structured pruning for LLMs as critical.
  • LLM Compression: LLMs’ billions of parameters make gradient backpropagation and training highly resource-intensive, rendering many conventional compression techniques infeasible.Knowledge distillation is cited as an approach facing implementation challenges because of high training costs.
  • LLM Properties: LLMs exhibit channels with abnormally large hidden-state magnitudes beyond certain parameter scales, such as 6B, motivating specialized compression techniques.Dettmers et al. suggest these channels explain why existing quantization methods fail on LLMs.

Preliminaries

The preliminaries motivate layer-wise pruning as a practical decomposition of the globally difficult LLM pruning problem and distinguish low-damage from easy-recoverability pruning metrics. They also define channel masks, grouped self-attention pruning, compensation-adjusted weights, and the ℓ2-error notation.

  • Layer-wise pruning: Global LLM pruning is computationally challenging, motivating layer-wise decomposition as a practical solution for both unstructured and structured pruning.SparseGPT demonstrated this decomposition for unstructured pruning, and the principle extends to structured LLM pruning.
  • Structured pruning formulation: The input-channel mask M_ℓ∈R^C_in indicates whether channels are pruned, with self-attention channels typically removed in groups of 128.The notation cW_ℓ denotes potentially updated pruned-layer weights.
  • Structured pruning formulation: The notation ||·||_2 represents the ℓ2-error.
  • Pruning metrics: Existing pruning methods follow either low-damage or easy-recoverability principles, corresponding to OBD and OBS, respectively.Wanda uses a localized low-damage metric, whereas SparseGPT employs an easy-recoverability metric.

Methodology

FLAP is a retraining-free structured-pruning framework built around baseline bias compensation, a structured fluctuation metric, and adaptive structure search. It estimates recoverability from feature variation, compensates removed components with baseline-derived biases, and standardizes scores across layers and modules for global pruning decisions.

  • Method overview: FLAP comprises three components: Baseline Bias Compensation, Structured Fluctuation Metric, and Adaptive Structure Search.The method overview is presented in Figure 1.
  • Baseline Bias Compensation: Baseline-derived bias terms compensate output-feature reconstruction errors caused by removing structured-pruned components.The baseline value is the average of the corresponding channel across calibration samples and is translated into the linear layer’s bias after establishing the pruning mask.
  • Structured Fluctuation Metric: Low channel fluctuation indicates that replacing a pruned input channel with its baseline value can effectively counterbalance the resulting output-feature change.This observation motivates using feature stability to identify easily recoverable structured pruning candidates.
  • Structured Fluctuation Metric: The fluctuation metric approximates structured recoverability by weighting each input feature’s sample variance with the squared norm of its corresponding weight column.The variance uses N calibration samples and applies the Bessel correction with denominator N−1.
  • Adaptive Structure Search: Because metric magnitudes vary greatly across layers and modules, FLAP standardizes each layer’s metric distribution to a common mean and standard deviation before comparing pruning candidates.The standardized metric represents relative output-feature variation when input features are replaced by baseline values, enabling consistent global structure search.

Experiments

Experiments on LLaMA models evaluate FLAP’s language-modeling and zero-shot performance against structured-pruning baselines, alongside ablations of its pruning components and efficiency analyses. FLAP consistently outperforms competing methods without retraining while preserving performance more robustly as pruning increases.

  • Overall Evaluation: Experiments evaluate LLaMA-7B/13B/30B/65B on WikiText2 perplexity and seven zero-shot common-sense benchmarks against Wanda-sp and LLM-Pruner.Wanda-sp is Wanda generalized to structured pruning.
  • Language Modeling: FLAP remains stable as pruning increases, whereas Wanda-sp degrades sharply and LLM-Pruner requires LoRA fine-tuning for acceptable performance at high ratios.The comparison is reported on LLaMA-7B WikiText2 results across varying pruning ratios.
  • Zero-Shot Evaluation: Across seven downstream zero-shot tasks and varying pruning ratios, FLAP outperforms LLM-Pruner with LoRA fine-tuning without retraining.At 20% pruning, Wanda-sp can surpass the original unpruned model, suggesting structured redundancy in LLMs.
  • Ablation Studies: At 50% pruning on LLaMA-7B, FLAP’s default configuration achieves WikiText2 perplexity 31.80, while AL-AM outperforms other global compression structures across pruning metrics.The ablation studies also examine pruning metrics, bias compensation, and calibration-sample robustness.
  • Efficiency and Robustness: Using 1024 calibration samples, FLAP requires one forward propagation and prunes LLaMA-7B in 3 to 5 minutes on a single GPU.Performance improves as the calibration dataset grows.

Conclusion

The paper presents FLAP, a retraining-free structured pruning framework for LLMs that combines a novel pruning metric, adaptive global compression, and compensation mechanisms to mitigate performance loss.

  • Conclusion: FLAP is a retraining-free structured pruning framework explicitly designed for Large Language Models.The framework is introduced to address challenges in structured pruning without retraining.
  • Conclusion: The method introduces a novel structured pruning metric and adaptive global model compression strategies.These components address the challenges posed by structured pruning.
  • Conclusion: FLAP implements robust compensation mechanisms designed to mitigate potential performance losses.The conclusion states that empirical results affirm the effectiveness of the structured compression model.

A Detailed Experimental Settings · Models.

The experiments evaluate FLAP on the LLaMA family and Vicuna-7B, covering widely adopted open-source models with important roles in applications.

  • Models.: FLAP is evaluated on the LLaMA model family and Vicuna-7B.These models are introduced as the experimental evaluation targets.
  • Models.: LLaMA is a Transformer-based large language model family open-sourced by Meta.The passage characterizes LLaMA as a set of open-source Transformer-based models.
  • Models.: The LLaMA family mainly includes 7B, 13B, 30B, and 65B models.These are the model sizes explicitly listed in the passage.
  • Models.: Vicuna is an instruction fine-tuned model based on the LLaMA framework.Its training uses user-shared conversations, according to the passage.
  • Models.: Vicuna leverages user-shared conversations for training.This describes the data source identified for Vicuna’s training.
  • Models.: The selected models have widespread adoption in the open-source community.The passage gives this adoption as a reason for evaluating them.
  • Models.: The selected models serve as foundational models in numerous applications.Their foundational role is presented as another reason their compression performance matters.
  • Models.: Their compression performance is therefore used as an evaluation focus.The supplied passage states that their adoption and application roles make compression performance relevant.

Evaluation. · Baselines.

The evaluation measures pruned-model language modeling on WikiText2 perplexity and zero-shot performance across seven pivot tasks using the EleutherAI LM Harness. FLAP is compared with Wanda-sp, a structured Wanda variant, while structured SparseGPT modification produces unreasonable results.

  • Evaluation.: Evaluation uses WikiText2 validation perplexity to assess the pruned model’s language modeling capabilities.Perplexity gauges predictive accuracy for the sample set.
  • Evaluation.: The study additionally employs the EleutherAI LM Harness for a broader evaluation of zero-shot performance.The benchmark covers seven pivot tasks.
  • Baselines.: FLAP is compared against two previous pruning methods.The baselines section introduces the comparison methods before describing them.
  • Baselines.: Wanda assigns importance using weight magnitude multiplied by the corresponding input activation’s ℓ2-norm and prunes locally within each output feature.This describes Wanda’s original unstructured pruning formulation for LLMs.
  • Baselines.: Wanda-sp generalizes Wanda to structured pruning by using each linear-layer weight group’s ℓ2-norm as its importance score.The method counts the ℓ2-norm of each group of weights and names the resulting variant Wanda-sp.
  • Baselines.: A structured-pruning modification of SparseGPT was attempted but failed to produce reasonable results.The passage reports unsuccessful adaptation rather than a quantitative comparison.

B Implementation Details · B.1 Wanda for structured pruning · B.2 Pseudo Code

The implementation runs FLAP with CPU-based score processing and GPU-based pruning and recovery, while its structured Wanda variant adapts importance scoring to grouped pruning. Algorithm 1 combines layer-wise fluctuation scoring, standardized global structure selection, baseline-bias compensation, streaming statistics, and module-aware head normalization.

  • B Implementation Details: Experiments run on an NVIDIA A100 GPU with 40 GB memory, loading LLaMA models in 16-bit floating-point format.Importance standardization and threshold filtering run on the CPU, while pruning and recovery run on GPUs.
  • B.1 Wanda for structured pruning: The structured Wanda metric modifies the original Wanda approach to match structured pruning characteristics.Its score uses the absolute weight value and the ℓ2-norm of aggregated token features, combined by summation of their products.
  • B.2 Pseudo Code: Algorithm 1 takes the original model F, calibration samples D_t, original weights W_ℓ, and pruning ratio p as inputs.It outputs the structured pruning mask M_ℓ, baseline bias B_ℓ, and pruned model F⋆.
  • B.2 Pseudo Code: FLAP decomposes LLM pruning into layer-wise subproblems that compute fluctuation-based importance scores from calibration features.It standardizes module-level scores to identify a global compression structure.
  • B.2 Pseudo Code: Welford’s method updates feature means and variances in a streaming manner, avoiding repeated recalculation and reducing storage and computational time.The method immediately updates variance as new samples arrive.
  • B.2 Pseudo Code: Transformer self-attention is pruned at head granularity because its weights cannot be directly pruned by rows or columns.FLAP computes fluctuation metrics separately for each layer and module before standardizing them across modules.
  • B.2 Pseudo Code: A normalization factor such as 512 / 3 makes importance scores comparable when pruning an attention head and an MLP neuron removes different numbers of parameters.This adjustment supports uniform search across layers and modules.

C Additional Experiments · C.1 Zero-shot performance in larger scale · C.2 Pruning on Vicuna-7B

The additional experiments evaluate FLAP on larger-scale LLaMA-13B zero-shot tasks and on Vicuna-7B, reporting stronger performance than LLM-Pruner in both settings. Tables 5 and 6 provide the corresponding compressed-model and perplexity evaluations.

  • C.1 Zero-shot performance in larger scale: FLAP outperforms LLM-Pruner on zero-shot downstream tasks with the compressed LLaMA-13B model.This result is reported in Table 5.
  • C.1 Zero-shot performance in larger scale: Table 5 reports zero-shot performance for various downstream tasks after applying FLAP to LLaMA-13B.The table compares compressed-model performance across pruning ratios.
  • C.1 Zero-shot performance in larger scale: Table 5 highlights the best result for each pruning ratio in bold.Underscored results indicate the second-best performance for each pruning ratio.
  • C.2 Pruning on Vicuna-7B: FLAP demonstrates enhanced performance relative to LLM-Pruner on the Vicuna-7B model.Table 6 presents their direct comparison.
  • C.2 Pruning on Vicuna-7B: The Vicuna-7B experiment extends the comparison beyond the LLaMA-13B zero-shot evaluation.It directly compares FLAP and LLM-Pruner on a different model.
  • C.2 Pruning on Vicuna-7B: Table 6 evaluates pruning methods for Vicuna-7B using WikiText2 validation perplexity.The table is used for the direct FLAP-versus-LLM-Pruner comparison.

C.3 Different calibration data selection · C.4 Generations From Compressed Model

C.3 finds that calibration-data choices affect task-specific generalization but change average zero-shot accuracy by only about ±1%. C.4 reports that FLAP’s 5.1B- and 4.5B-parameter pruned LLaMA models retain general knowledge in generation examples.

  • C.3 Different calibration data selection: Calibration-data selection affects the method’s generalization across downstream tasks.Different calibration datasets suit different downstream tasks.
  • C.3 Different calibration data selection: About ±1% is the average-accuracy fluctuation across zero-shot tasks when using C4 versus WikiText2 calibration data.The overall average-accuracy differences are not significant.
  • C.3 Different calibration data selection: The same calibration-data phenomenon appears in replication experiments with other calibration-dependent methods, including SparseGP.This observation extends beyond FLAP’s calibration-data experiments.
  • C.3 Different calibration data selection: Table 7 evaluates how different calibration datasets affect generalization ability.The table compares calibration-data choices in relation to downstream-task performance.
  • C.4 Generations From Compressed Model: Tables 8 and 9 provide additional generation examples from models pruned by FLAP.The examples compare outputs from dense and pruned models.
  • C.4 Generations From Compressed Model: The pruned LLaMA models with 5.1B and 4.5B parameters effectively retain general knowledge.This conclusion is supported by the reported generation results for the compressed models.
Loading 2312.11983v1…