PaperlayerView exact revision on arXiv

Source-linked AI summary

SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks

Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, Jae-Joon Kim

arXiv:2402.09025v6cs.CLcs.LG

TL;DR

LLMs' memory and computational demands constrain deployment, while existing pruning approaches face speedup challenges. SLEB removes redundant transformer blocks; experiments report pruning up to 20% while maintaining linguistic performance across language-modeling and zero-shot tasks, with substantial end-to-end inference speed improvements.

  • Problem

    LLM deployment is constrained by memory and compute demands, while unstructured and 2:4 pruning face constraints in achieving speedups.

    Source: §1 · p. 1, Challenge 1) Limitation in Achieving… · pp. 2–3

  • Method

    SLEB identifies and removes redundant transformer blocks, assessing block output similarity and the impact of block removal on token predictions.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    SLEB prunes up to 20% of blocks while maintaining linguistic performance across language-modeling and zero-shot tasks, with substantial end-to-end inference speed improvements.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Whole-block removal delivers end-to-end speed improvements across implementation scenarios, making SLEB a practical option for real-world LLM serving.

    Source: §5 · p. 9

    Unstructured pruning typically requires more than 90% sparsity for speedup on NVIDIA GPUs, while LLM pruning above 50% is often difficult without substantially compromising linguistic prowess.

    Source: Challenge 1) Limitation in Achieving… · p. 2

Abstract

from arXiv · show

Large language models (LLMs) have proven to be highly effective across various natural language processing tasks. However, their large number of parameters poses significant challenges for practical deployment. Pruning, a technique aimed at reducing the size and complexity of LLMs, offers a potential solution by removing redundant components from the network. Despite the promise of pruning, existing methods often struggle to achieve substantial end-to-end LLM inference speedup. In this paper, we introduce SLEB, a novel approach designed to streamline LLMs by eliminating redundant transformer blocks. We choose the transformer block as the fundamental unit for pruning, because LLMs exhibit block-level redundancy with high similarity between the outputs of neighboring blocks. This choice allows us to effectively enhance the processing speed of LLMs. Our experimental results demonstrate that SLEB outperforms previous LLM pruning methods in accelerating LLM inference while also maintaining superior perplexity and accuracy, making SLEB as a promising technique for enhancing the efficiency of LLMs. The code is available at: https://github.com/jiwonsong-dev/SLEB.

1. Introduction

LLMs offer strong NLP capabilities, but memory and computational demands challenge deployment, and conventional pruning faces sparse-matrix difficulties on GPUs optimized for dense operations. SLEB targets redundancy between neighboring transformer blocks by removing redundant blocks.

1. Introduction

  • LLM deployment is constrained by memory and computational demands, motivating more compact and efficient models that preserve linguistic capability.Source: §1 · p. 1
  • Pruning LLMs is challenging because sparse-matrix management is complex on modern GPUs optimized for dense operations.Source: §1 · p. 1
  • Neighboring transformer blocks produce similar outputs as their contributions accumulate along the residual path, resulting in redundancy.The blocks contribute outputs from attention and feed-forward layers to the residual path.Source: §1 · pp. 1–2
  • SLEB removes redundant transformer blocks to streamline LLMs, aligning speedup with pruning ratio while aiming to preserve text-generation capabilities.The paper presents block removal as a way to avoid challenges associated with traditional network pruning.Source: §1 · p. 2

2. Motivation

Pruning can compact LLMs, but weight sparsity often fails to translate into practical inference speedup, while dynamic block skipping brings quality, batching, memory, and training constraints. SLEB instead statically removes verified redundant transformer blocks, targeting speedups with lower memory use and no intensive training.

2.1. Pruning

  • Transformer blocks contain attention and feed-forward layers whose matrix multiplications can be reduced by pruning weights, while removing blocks reduces the total number of computations.These are the two compactness and speed strategies described for sequentially stacked, identically structured blocks.Source: §2.1 · p. 2

Challenge 1) Limitation in Achieving LLM Inference

  • Unstructured pruning creates sparse matrices that can hinder acceleration and often needs over 90% sparsity on NVIDIA GPUs, while LLM pruning above 50% is often difficult without compromising linguistic prowess.Structured pruning creates hardware-friendly patterns, but its expected speedup proportional to pruning ratio often fails in practice.Source: Challenge 1) Limitation in Achieving… · p. 2
  • 2:4 pruning produces no speedup at batch sizes 1–128 and reaches about 1.5× dense-multiplication speed only at larger batches, with gains dependent on hardware and model size.Channel-wise pruning keeps matrices dense, but preserving linguistic capabilities can require extensive fine-tuning, and faster matrix multiplication does not guarantee faster end-to-end inference.Source: Figure 3 · p. 2, Challenge 1) Limitation in Achieving… · p. 3

2.2. Early Exit

  • Early exit and removing consecutive blocks both raise WikiText-2 perplexity as more blocks are removed, even when testing the best removal positions.Early exit removes the final blocks; “Chunk Best” tests all possible removable positions.Source: Figure 4 · p. 3, §2.2 · p. 4
  • Dynamic early exit complicates multi-batch processing, retains all model parameters, and requires extensive training; even an ideal predictor is estimated to leave about 90% of blocks to process.Blocks matching the final token predictions are typically near the model’s end, and training includes both the skipping predictor and adaptation of the LLM.Source: §2.2 · pp. 3–4, Figure 5 · p. 3

Solution to Challenge 2) Acceleration in Multi-batch

  • SLEB statically eliminates blocks after redundancy verification, aiming for speedup proportional to blocks removed while supporting multi-batch acceleration, reducing memory use, and avoiding intensive training.Its redundancy analysis is training-free, which the paper presents as suitable for large-scale models with prohibitive training demands.Source: §2.2 · p. 4, Solution to Challenge 2) Acceleration… · p. 4

3. Proposed SLEB

SLEB builds on the high similarity between neighboring transformer-block outputs and iteratively removes blocks assessed as redundant. Its Metric3-based algorithm uses calibration data and can be integrated into the model’s forward pass without additional training.

3.1. Output Similarity across Transformer Blocks

  • Neighboring transformer blocks consistently have high output similarity, indicating potential redundancy despite varying similarity between more distant blocks.The paper measures cosine similarity between block outputs and reports results for OPT-6.7B and LLaMA-2-7B.Source: §3.1 · pp. 4–5

3.2. Redundancy Verification of Transformer Blocks

  • Metric2 improves perplexity over Metric1, but removing consecutive blocks can degrade token predictions as block significance changes.For OPT-6.7B, Metric2 removes blocks 3, 4, 5, 6, 7, 8, and 10.Source: §3.2 · p. 5
  • Metric3 removal of seven OPT-6.7B blocks maintains perplexity remarkably well across target models, as compared on WikiText-2.The removed block indices are 6, 7, 3, 24, 18, 30, and 11.Source: Figure 7 · p. 5, §3.2 · p. 6

3.3. Proposed SLEB Algorithm

  • SLEB uses calibration data and Metric3 to iteratively identify and remove the currently most redundant block, without additional training.At each step, it recalculates redundancy in the already-streamlined model.Source: §3.2 · p. 5, §3.3 · p. 6

4. Experiments

SLEB is evaluated across model families, pruning levels, language modeling, zero-shot accuracy, inference speed, and quantization compatibility. It preserves linguistic performance while removing transformer blocks and provides speedups across serving scenarios.

4.1. Experimental Setup

  • Experiments use OPT and LLaMA-2 models at 10% and 20% target sparsity, with the removal count rounded up when the target product is non-integer.Calibration uses 128 randomly selected WikiText-2 training samples; pruning is inference-only, and LLaMA-2-70B can be compressed in about 1.5 hours.Source: §4.1 · p. 6

4.2. Elimination of Transformer Blocks using SLEB

  • Removed-block locations vary by model: OPT-13B more often loses earlier blocks, while LLaMA-2-13B loses middle and later blocks.The authors say model-specific sensitivity makes Metric3 important for selecting blocks to remove.Source: §4.2 · p. 6, Figure 8 · p. 6

4.3. Language Modeling

  • SLEB preserves perplexity scores across evaluated cases, outperforming previous pruning methods in most cases despite using a coarser pruning unit.Wanda and DSnoT completely fail on OPT-66B.Source: §4.3 · p. 7, Table 1 · p. 7

4.4. Dependency on Calibration Dataset

  • SLEB shows the least dependence on calibration-dataset choice, while SliceGPT struggles when calibration and evaluation datasets differ.SLEB assesses block redundancy at the network level using Metric3; SparseGPT and Wanda measure weight significance in each linear layer.Source: §4.4 · p. 7, Figure 9 · p. 7

4.5. Zero-shot Tasks

  • At 10% sparsity, SLEB achieves superior zero-shot accuracies to previous pruning methods; at 20%, it still outperforms them in most cases.The reported averages cover PIQA, WinoGrande, HellaSwag, ARC-easy, and ARC-challenge.Source: §4.5 · p. 8, Table 2 · p. 8

4.6. Speedup

  • SLEB improves latency and throughput in proportion to pruning ratio and provides speedups across serving scenarios, unlike prior methods whose gains vary by scenario.The evaluation separates prompt processing from token generation, which have different inference characteristics.Source: §4.6 · p. 8

4.7. Compatibility with Post-Training Quantization

  • Applying AWQ 4-bit quantization to SLEB-pruned models has no discernible impact on perplexity.The comparison uses SLEB at 20% target sparsity.Source: §4.7 · p. 8, Figure 10 · p. 8

5. Conclusion

SLEB removes redundant transformer blocks without resource-intensive training. The paper reports pruning up to 20% of blocks while maintaining linguistic performance and improving end-to-end inference speed.

5. Conclusion

  • SLEB removes redundant transformer blocks to streamline LLMs while preserving their language capabilities without resource-intensive training.The paper reports pruning up to 20% of blocks while maintaining performance across language modeling and zero-shot tasks.Source: §5 · p. 9
  • Pruning entire transformer blocks produces end-to-end inference speed improvements across the implementation scenarios discussed.The conclusion characterizes SLEB as a practical solution for real-world LLM serving scenarios.Source: §5 · p. 9
  • The reported performance scope includes language modeling and zero-shot tasks after pruning up to 20% of blocks.Source: §5 · p. 9

A.1. Runtime

SLEB’s pruning runtime is generally within the range of other methods, and its selected block indices vary by model and target sparsity.

A.1. Runtime

  • SLEB’s pruning runtime is generally within the range observed for other pruning methods.Runtime is the time to identify and remove elements, and each algorithm is applied to each model once.Source: §A.1 · p. 12, Table 3 · p. 12

A.2. Selected Transformer Blocks

  • For 20% sparsity, SLEB’s selected blocks include those chosen at 10% sparsity plus additional blocks to reach the higher target.Table 4 lists the number and indices of removed blocks; the indices are ordered by removal sequence.Source: Table 4 · p. 12, §A.2 · p. 12

B.1. Limitations of Early Exit

Ideal early exits can speed up single-batch generation, but their benefit diminishes as batch size increases. WikiText-2 perplexity results for SliceGPT reflect the match between its calibration and evaluation datasets.

B.1. Limitations of Early Exit

  • Ideal early exit reaches up to 1.14× speedup in single-batch generation, but its effectiveness diminishes with larger batches because tokens can require different exit points.A batch follows the longest path required by any token.Source: §B.1 · p. 13, Table 6 · p. 13, Table 5 · p. 13

B.2. Language Modeling

  • SliceGPT shows good WikiText-2 perplexity when calibration data comes from WikiText-2, but poor perplexity on C4 when calibration and evaluation datasets differ.The authors attribute this behavior to SliceGPT relying on activation-matrix information to select channels for pruning.Source: §B.2 · p. 14, Table 7 · p. 14

B.3. Dependency on Calibration Dataset

This section tests how calibration data affects perplexity in pruned LLaMA-2 models. SLEB, SparseGPT, and Wanda show more stable results across datasets than SliceGPT.

B.3. Dependency on Calibration Dataset

  • The comparison uses 128-sequence calibration sets sampled from WikiText-2 or C4 training data and evaluates four pruning methods.It compares SparseGPT and Wanda at 2:4 pruning, SliceGPT at 25% sparsity, and SLEB at 20% sparsity.Source: §B.3 · p. 15, Table 8 · p. 15
  • SLEB, SparseGPT, and Wanda show less dataset dependence than SliceGPT, whose perplexity is more dependent on input patterns.The section attributes this difference to the former methods using pretrained weight-matrix information, while SliceGPT uses input activations for PCA-based neuron selection.Source: §B.3 · p. 15
  • Table 8 presents perplexity results for different pruning methods under different calibration datasets.Source: Table 8 · p. 15

B.4. Zero-shot tasks

The section reports zero-shot task accuracies and measures LLaMA-2 inference speed. SLEB consistently improves latency and throughput across serving scenarios, unlike weight-pruning methods whose impact varies by scenario.

B.4. Zero-shot tasks

B.5. Speedup

  • Prompt-processing latency averages 50 measurements after one dummy test, while token-generation throughput averages 10 measurements after one dummy test.Measurements cover NVIDIA A100 and A6000 GPUs.Source: §B.5 · p. 17
  • SLEB consistently improves LLM inference latency and throughput across serving scenarios, while weight-pruning methods vary with the scenario.The cited discussion attributes scenario dependence to matrix-size effects and to non-matrix-multiplication operations contributing to end-to-end inference speed.Source: §B.5 · p. 17, Table 10 · p. 17, Table 11 · p. 17

B.6. Compatibility with Post-Training Quantization

SLEB-pruned models can be further compressed with AWQ, a 4-bit weight quantization, with negligible perplexity impact on C4 and WikiText-2. The reported impact is less pronounced for models above 13 billion parameters.

B.6. Compatibility with Post-Training Quantization

  • AWQ quantization has negligible impact on perplexity for SLEB-pruned models on C4 and WikiText-2.Figures 11 and 12 compare perplexity before and after AWQ compression on the respective evaluation datasets.Source: §B.6 · p. 18, Figure 11 · p. 18, Figure 12 · p. 18
  • The perplexity impact of quantization is less pronounced for models with more than 13 billion parameters.Source: §B.6 · p. 18
  • AWQ is used as a post-training quantization step to further compress models already pruned with SLEB.Source: §B.6 · p. 18

B.7. Perplexity and Accuracy Results of SLEB across Various Sparsity Ratios

The section evaluates SLEB across sparsity ratios and reports that it continues to outperform prior pruning methods in perplexity and zero-shot accuracy after retraining. Fine-tuning uses one Alpaca epoch with rank-8 LoRA modules.

B.7. Perplexity and Accuracy Results of SLEB across Various Sparsity Ratios

  • SLEB is evaluated at sparsity ratios from 10% to 50% using C4 perplexity and zero-shot task accuracy.Table 12 reports OPT and LLaMA-2 results.Source: §B.7 · p. 19, Table 12 · p. 19

B.8. Fine-tuning

  • Fine-tuning uses one Alpaca training epoch and rank-8 LoRA modules for pruned LLaMA-2-7B models.The comparison includes SparseGPT with 2:4 pruning, SliceGPT with channel-wise pruning, and SLEB.Source: §B.8 · p. 20
  • Even after retraining, SLEB outperforms previous pruning methods in perplexity and zero-shot accuracy.Table 13 covers perplexity and accuracy before and after fine-tuning pruned LLaMA-2-7B.Source: §B.8 · p. 20, Table 13 · p. 20

Related papers

Loading 2402.09025v6…