Source-linked AI summary
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen
TL;DR
Large language models demand substantial hardware resources, while evidence suggests that many layers are redundant. The paper measures layer importance with Block Influence and removes low-scoring layers in ShortGPT, achieving strong pruning results while preserving much of model performance.
Problem
LLMs with billions to trillions of parameters require substantial hardware resources, motivating investigation of layer redundancy for model compression.
Method
ShortGPT computes Block Influence from hidden-state transformations, ranks layers by BI, and removes those with lower scores using calibration data.
Results
ShortGPT maintains approximately 90% of performance while reducing approximately 25% of parameters and outperforms previous state-of-the-art pruning methods.
Takeaways & Limitations
Layer removal can substantially reduce model size and computation, and the approach can be combined with quantization methods.
Takeaways & Limitations
Removing 25% of layers can reduce generative-task performance in Llama2-7B and Baichuan2-7B to nearly zero, while the reasons remain unexplored.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) continue to advance in performance, their size has escalated significantly, with current LLMs containing billions or even trillions of parameters. However, in this study, we discovered that many layers of LLMs exhibit high similarity, and some layers play a negligible role in network functionality. Based on this observation, we define a metric called Block Influence (BI) to gauge the significance of each layer in LLMs. We then propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers in LLMs based on their BI scores. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning. Moreover, ShortGPT is orthogonal to quantization-like methods, enabling further reduction in parameters and computation. The ability to achieve better results through simple layer removal, as opposed to more complex pruning techniques, suggests a high degree of redundancy in the model architecture.
1 Introduction
LLMs require substantial deployment resources, motivating compression. ShortGPT targets layer redundancy with Block Influence and removes low-influence layers while retaining most performance.
- Billions-to-trillions-parameter LLMs require significant hardware resources, creating barriers to practical deployment.
- Quantization reduces parameter precision, whereas pruning removes redundant parameters to reduce model size and computation.
- Block Influence quantifies how much the hidden state changes after each layer, providing a direct measure of layer importance.
- ShortGPT removes layers with lower BI scores, significantly reducing model size without sacrificing much performance.
- Removing 10 of 40 LLaMA 2-13B layers reduced MMLU performance from 55.0 to 52.2.
- ShortGPT maintains approximately 90% performance while reducing approximately 25% of parameters and can combine with quantization methods.
2 Motivation
Pre-normalization is associated with high input-output similarity in transformer layers, suggesting layer redundancy. Experiments indicate that redundancy is especially concentrated in middle-to-later layers.
- 2.1 Background: Pre-normalized transformer layers tend to have higher similarity between their inputs and outputs.
- 2.1 Background: High input-output similarity indicates that some layers make minimal hidden-state changes and may contribute little to overall model function.
- 2.1 Background: Figure 2 tracks cosine similarity against training tokens and shows post-normalized model divergence after approximately ∼26B tokens.
- 2.2 Layer redundancy: Removing individual layers from Llama2-7B-Base and Baichuan2-7B-Base confirms that some layers cause little performance degradation when omitted.
- 2.2 Layer redundancy: The observed redundancy is primarily concentrated in the network’s middle-to-later layers.
3 Methodology
ShortGPT ranks layers by Block Influence, a hidden-state transformation measure, and removes the lowest-scoring layers using calibration data to control the speed-performance trade-off.
- 3.1 Layer importance: The methodology assesses layer importance through Block Influence, a metric measuring each layer’s degree of hidden-state transformation.
- 3.1 Layer importance: Lower BI scores indicate higher cosine similarity between consecutive hidden states and smaller layer transformations.
- 3.1 Layer importance: BI scores positively correlate with layer importance, as measured by perplexity after layer removal.
- 3.2 Layer Removal: ShortGPT collects hidden states on unlabelled calibration text, computes BI scores, sorts layers ascending by BI, and deletes the lowest-scoring layers.
- 3.2 Layer Removal: The number of deleted layers can vary to trade off inference speed and performance.
4 Experiments
Experiments evaluate ShortGPT across multiple models, benchmarks, pruning metrics, ratios, and comparisons with structured pruning baselines. The results report stronger performance from layer removal, evidence of depth redundancy across architectures, and compatibility with quantization.
- Experimental Setup: Experiments evaluate Llama2 and Baichuan2 models on reasoning, language, knowledge, examination, and understanding benchmarks.The benchmark suite includes CMNLI, HellaSwag, PIQA, CHID, WSC, CommonSenseQA, BoolQ, MMLU, CMMLU, Race, XSum, C3, and PG19.
- Main Results: ShortGPT significantly surpasses baseline pruning methods while maintaining most of the model’s capabilities.The comparison uses approximately 1/4 parameter reduction and reports relative performance retention in the final table column.
- Main Results: Removing entire layers often outperforms reducing embedding dimensions or pruning finer-grained structures.The comparison suggests greater redundancy in model depth than width, while the authors attribute fine-grained difficulty to robustness under individual deep-layer removal.
- Varying metric and pruning ratio: BI outperforms alternative importance metrics, while Relative Magnitude remains highly competitive across the evaluated measures.Hidden-state norm performs well on MMLU but relatively poorly on perplexity; perplexity is computed after single-layer removal, whereas other metrics use layer hidden states.
- Varying metric and pruning ratio: Model performance generally declines as pruning ratio increases, with a sharp MMLU drop at a specific layer indicating critical layers.These experiments evaluate perplexity and MMLU on Llama2 and Baichuan2.
- Redundancy on non-transformer LLM: ShortGPT applies to RWKV and Mamba, indicating layer redundancy beyond Transformer models, although RWKV appears less redundant.The method is also reported as compatible with GPTQ quantization, and pruning remains orthogonal to quantization.
5 Limitation
Layer removal has a stronger negative effect on generative tasks than on multiple-choice tasks, especially in smaller models, and the underlying reasons remain unresolved.
- Removing 25% of layers from Llama2-7B or Baichuan2-7B reduced XSum and C3 performance to nearly zero.The decline was less severe for the larger 13B model.
- The authors speculate that generative tasks accumulate errors and that larger models are more robust, but leave these explanations for future work.Post-training techniques may mitigate the issue and warrant further exploration.
6 Related works
Related work covers model pruning, quantization, and studies of redundancy in neural architectures. ShortGPT is positioned as a simple structured pruning method focused on removing redundant layers.
- Model compression research mainly includes pruning and quantization, which reduce model size and inference costs through different mechanisms.Quantization changes parameter precision, while pruning removes redundant parameters or structures.
- Unstructured pruning removes parameters without considering model structure, whereas structured pruning removes larger organized components but can preserve a more practical model composition.Examples include SparseGPT and LoRAPrune for unstructured pruning, and LLM-Pruner, ShearedLLaMA, and LaCo for structured approaches.
- ShortGPT differs from prior structured pruning methods by using a simple layer-removal strategy, and its pruning can be combined with quantization.The paper describes the two approaches as orthogonal.
- Quantization converts floating-point parameters into integers or other discrete forms and can substantially reduce storage and computational costs.Methods such as LUT-GEMM and SPQR use specialized low-precision representations or selectively preserve abnormal weights.
- Prior redundancy studies have examined nonlinear models and Transformer components, including attention-head redundancy and redundancy in pretrained models.These studies provide context for investigating redundancy beyond individual parameters or attention heads.
7 Conclusion
The paper identifies substantial layer-wise redundancy in LLMs and uses Block Influence to remove minimally contributing layers. The resulting method preserves much of model performance while reducing parameters and computation.
- Certain LLM layers contribute minimally to network functionality and can be removed without substantially compromising performance.This finding motivates layer-removal pruning.
- Block Influence quantifies the importance of each layer and guides the ShortGPT layer-removal method.The method directly removes layers with lower influence.
- Approximately 90% of LLM performance can be maintained while reducing parameter amount and computational requirements by approximately 25%.The paper presents this as an experimental result across its evaluated models and settings.
- The pruning method is orthogonal to quantization and can be further improved through continual training.The authors also suggest reducing inherent redundancy as a direction for improving training efficiency.
A Mathematical explanation for why pre-norm brings high similarity
The mathematical analysis explains why pre-norm architectures can produce high similarity between a layer’s input and output. As depth increases, the residual transformation becomes relatively small, making some layers removable with limited impact.
- Motivation: Pre-norm architectures tend to produce high similarity between transformer-layer inputs and outputs, motivating analysis of layer redundancy.The paper uses RMSNorm as a representative pre-norm configuration and notes its use in models such as Llama and Mamba.
- Mathematical explanation: The analysis states that the hidden-state norm continuously increases with depth in a pre-norm model.This conclusion follows from the cited initialization lemma and the paper’s assumptions about hidden-state components.
- Mathematical explanation: The residual update xL+1 = xL + fL(xL, θL) separates the layer output into its incoming hidden state and a layer transformation.The transformation fL may represent operations such as attention or an MLP, with θL denoting learnable parameters.
- Implication: As layer depth L increases, the similarity between a layer’s input and output becomes high, indicating that fL may make relatively small changes.The paper connects this small change to the possibility that removing the layer has limited effect on model behavior.
- Empirical observation: The high input-output similarity is observed in both the authors’ trained models and existing models, despite the theoretical analysis targeting randomly initialized models.The empirical evidence is described as appearing in Figures 2 and 4.
E Setup for training post-norm model and pre-norm model
The training setup for pre-norm and post-norm models is documented in Table 11.
- Table 11 lists the specific training settings for pre-norm and post-norm models.
- The passage identifies these settings as part of the experimental configuration.
- The relevant setup is organized under the heading “Training Parameters.”
F post-training settings
The post-training procedure replaces removed layers with lightweight gated MLP layers and documents the resulting settings in Table 12.
- Removed layers are replaced with lightweight gated MLP layers with hidden size = 2048.
- Table 12 reports the post-training settings used after layer replacement.
- The post-training configuration is organized under the heading “Post-training Parameters.”
G Evaluation Benchmarks
The evaluation examines pruned language models across broad reasoning, knowledge, language, examination, and understanding benchmarks, including English and Chinese assessments.
- Evaluations compare model abilities before and after pruning using MMLU, CMMLU, and a wider dataset following LaCo.
- MMLU measures pretrained knowledge and problem-solving across 57 subjects, using zero-shot and few-shot settings.
- CMMLU evaluates advanced knowledge and reasoning in Chinese language and cultural contexts across 67 topics.
- Additional benchmarks cover Chinese language understanding, commonsense reasoning, daily physical scenarios, and other evaluation tasks.
- The experiments use a GPU heterogeneous platform, with hardware details reported in Table 13.
- Table 13 provides the setup of removed layers for benchmark models.