Source-linked AI summary
A Survey on Model Compression for Large Language Models
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, Weiping Wang
TL;DR
LLMs’ scale creates substantial memory and computational demands, motivating compression for practical deployment. This survey synthesizes LLM compression methods, metrics, benchmarks, challenges, and future directions, concluding that the field remains early and compressed models retain a significant performance gap from uncompressed counterparts.
Problem
LLMs’ extensive size and computational requirements create challenges for practical use, especially in resource-limited settings.
Method
The paper conducts a survey of LLM compression methods, evaluation metrics, benchmarks, challenges, and future research directions.
Results
The survey covers representative compression methods, metrics, benchmark datasets, and the current challenges and opportunities of LLM compression.
Takeaways & Limitations
The survey provides a reference on the current LLM compression landscape and promotes continued exploration of the topic.
Takeaways & Limitations
Compressed LLMs continue to exhibit a significant performance gap compared with their uncompressed counterparts.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.
1 Introduction
LLMs deliver strong performance but require substantial memory and computation, motivating model compression. The survey reviews compression methods, evaluation resources, challenges, and future directions specifically for LLMs.
- Motivation: LLMs contain billions or more parameters and are trained on massive text data, creating substantial deployment demands.GPT-175B requires at least 350GB of FP16 memory and at least five 80GB A100 GPUs for inference.
- Motivation: Model compression transforms resource-intensive LLMs into compact versions for resource-constrained devices while improving inference speed and resource efficiency.
- Survey scope: The survey covers quantization, pruning, knowledge distillation, low-rank factorization, metrics, benchmarks, challenges, and future research directions.
- Survey scope: The paper presents itself as the inaugural survey dedicated solely to model compression for LLMs.
2 Metrics and Benchmarks
Compressed LLMs are evaluated through complementary efficiency, size, accuracy, and capability measures using diverse benchmarks and standardized testing frameworks. The survey distinguishes task coverage and task dependence when organizing compression methods and evaluation resources.
- Metrics: Compression evaluation combines accuracy and zero-shot ability with metrics capturing computational, latency, size, and deployment efficiency.
- Metrics: FLOPs measures operations required per instance, whereas MFU measures the ratio of actual to maximum theoretical device FLOPS.
- Metrics: Inference time measures response-generation latency, while speedup ratio compares uncompressed-model inference time with compressed-model inference time.
- Metrics: Compression ratio is original size divided by compressed size, and model size is typically measured by total parameter count.
- Benchmarks and datasets: Benchmarks compare compressed with uncompressed LLMs across diverse NLP tasks, including perplexity, zero-shot ability, and reasoning.
- Compression taxonomy: The taxonomy includes quantization, pruning, knowledge distillation, and low-rank factorization, while QAT and KD are task-based and other methods are task-agnostic.
- Benchmarks and datasets: BIG-Bench covers over 200 NLP tasks, while unseen-instruction datasets test compact LLMs on unfamiliar tasks and user-oriented instructions.
- Evaluation frameworks: The EleutherAI LM Harness supports over 60 standard academic benchmarks and hundreds of subtasks and variants for reproducible, comparable evaluation.
3 Quantization
Quantization reduces parameter precision to compress LLMs, primarily through QAT or PTQ, with methods targeting weights, activations, or KV caches. The surveyed techniques trade retraining cost, inference efficiency, memory use, and accuracy preservation.
- Quantization reduces the number of parameter bits while aiming to minimize inference-performance loss.
- QAT retrains quantized models to counteract accuracy degradation, whereas PTQ converts full-precision models without retraining.
- Quantization-Aware Training: QAT retraining is expensive for LLMs with tens or hundreds of billions of parameters, motivating PEFT-based approaches that remain typically task-dependent.
- Weight-Only Quantization: Weight-only quantization is widespread, with GPTQ reaching 3/4-bit precision and QuIP reaching 2 bits with minimal performance loss.
- Weight-Only Quantization: Sensitive-weight methods preserve selected parameters in high precision to reduce quantization errors, as demonstrated by AWQ’s top 1% selection.
- Weight-activation quantization targets both weights and activations, while KV-cache quantization addresses memory growth from longer input sequences.
4 Pruning
Pruning compresses LLMs by removing redundant components through unstructured, structured, or semi-structured sparsity. These choices balance performance preservation, hardware acceleration, retraining needs, and task-specific adaptation.
- Pruning reduces model size or complexity by removing redundant components and is categorized as unstructured, structured, or semi-structured.
- Unstructured Pruning: Unstructured pruning can preserve performance without retraining, but its irregular sparsity requires specialized inference handling or software optimization.
- Structured Pruning: Structured pruning is hardware-agnostic and can accelerate inference on traditional hardware, but removing larger components may degrade performance and require fine-tuning.
- Structured Pruning: Structured pruning methods assess component importance using loss, gradient, magnitude, fluctuation, PCA, or regularization-based criteria.
- Structured Pruning: Knowledge distillation can be combined with structured pruning to help smaller models maintain performance while reducing size.
- Semi-Structured Pruning: Semi-structured N:M sparsity retains N non-zero elements among every M contiguous elements, with methods such as E-Sparse, SparseGPT, and Wanda exploring this pattern.
- Dynamic Pruning: Dynamic pruning can remove different model parts according to current task requirements, balancing task-specific performance and efficiency.
- Downstream-task performance can vary significantly with the selected calibration data, making calibration-data quality and preparation important for compressed models.
5 Knowledge Distillation
Knowledge distillation transfers capabilities from large teacher LLMs to smaller student models through black-box or white-box supervision. The surveyed work covers reasoning, in-context learning, and instruction-following transfer while identifying generalization and teacher-quality constraints.
- Knowledge distillation transfers knowledge from a large teacher model to a smaller student model through black-box or white-box supervision.
- Black-Box Distillation: Black-box distillation uses teacher outputs to generate datasets for fine-tuning smaller language models, including instruction-oriented examples.
- Reasoning Distillation: Chain-of-thought distillation transfers reasoning capability to smaller models, with a trade-off between model size and dataset size in reasoning ability.
- In-Context Learning Distillation: In-context-learning distillation combines in-context objectives with language-modeling objectives, while Multitask-ICT is more effective but computationally more demanding than Meta-ICT.
- Black-Box Distillation: Black-box teacher outputs may not cover all input scenarios, leaving student generalization on unknown data and data diversity as open issues.
- White-Box Distillation: White-box distillation provides access to teacher parameters or output distributions and can yield higher-level performance improvements.
- White-Box Distillation: White-box distillation is limited because open-source LLMs currently perform worse than closed-source models, constraining student-model improvement.
6 Low-Rank Factorization
Low-rank factorization compresses LLM weight matrices by approximating them with smaller matrices, reducing space and computational effort. Recent work combines randomized low-rank and low-precision factorization for LLM compression.
- Low-rank factorization decomposes a large matrix W into smaller matrices U and V, with rank k much smaller than the original dimensions.W ≈ UV, where U is m × k and V is k × n.
- LPLR compresses LLM weight matrices through randomized low-rank and low-precision factorization.It uses random sketching to approximate column spaces, quantizes the columns, and projects the result.
7 Challenges and Future Directions
LLM compression research remains early, with compressed models still showing a significant performance gap and many methods relying on manual design. Future directions include extending established techniques, improving deployment evaluation, automating compression, and addressing explainability and scaling-law trade-offs.
- 7.1 More Advanced Methods: Compressed LLMs still exhibit a significant performance gap compared with uncompressed counterparts.
- 7.2 Scaling up Model Compression Methods from Other Models: Classic methods such as lottery tickets and parameter sharing remain underexplored opportunities for further LLM compression.
- 7.3 The Efficiency of Compressed LLMs: Deployment studies should evaluate arithmetic intensity, memory size, and throughput, using tools such as the Roofline Model for hardware-specific analysis.
- 7.4 The Effect of Scaling Law: The scaling law creates a trade-off between compressed-model size and performance, motivating further study of its underlying mechanisms and theories.
- 7.5 AutoML for Model Compression: AutoML could reduce manual effort by selecting compression hyperparameters, architectures, scales, and task-specific strategies.
- 7.6 Explainability of Model Compression: Explainability remains challenging because the mechanism by which compression methods transfer capabilities, such as chain-of-thought ability, is unclear.
8 Conclusion
The survey examines LLM compression methods, metrics, and benchmark datasets while highlighting the field’s challenges and opportunities. It aims to provide a reference that supports continued exploration of LLM compression.
- The survey covers LLM compression techniques, evaluation metrics, and benchmark datasets.
- It highlights challenges and opportunities in LLM compression and aims to provide a valuable reference for ongoing research.