PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Large Language Models: A Survey

Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, Mi Zhang

arXiv:2312.03863v4cs.CLcs.AI

TL;DR

LLMs require substantial resources for training and inference, motivating more efficient techniques. This survey systematically reviews efficient LLM research through a taxonomy spanning model-centric, data-centric, and framework-centric perspectives, providing a structured resource for researchers and practitioners.

  • Problem

    LLMs’ substantial training and inference resource demands motivate research into techniques that improve their efficiency.

    Source: §1 · p. 2

  • Method

    The survey organizes efficient LLM research into model-centric, data-centric, and framework-centric categories covering algorithmic, system, and framework techniques.

    Source: §1 · p. 2, §5 · p. 36

  • Results

    The survey provides a systematic and comprehensive review of efficient LLM research across its proposed taxonomy.

    Source: §1 · p. 2, §5 · p. 36

  • Takeaways & Limitations

    The taxonomy and review provide researchers and practitioners with a structured way to understand efficient LLM research and begin work in the field.

    Source: §5 · p. 36

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.

1 Introduction

Large language models achieve strong performance across diverse tasks through massive model and data scales, but their training and inference costs increase substantially with scale. This survey organizes efficient LLM research into model-centric, data-centric, and framework-centric categories, focusing on models with more than billions of parameters.

Motivation

  • Scaling LLMs improves performance but increases training GPU hours and inference latency, creating substantial resource demands that motivate efficiency techniques.Larger models require exponentially more training GPU hours and deliver lower inference throughput, while Mistral-7B achieves comparable performance and higher throughput than LLaMA-1-33B using grouped-query and sliding-window attention.Source: §1 · p. 2, Figure 1 · p. 3, Figure 2 · p. 3

Survey scope

  • The survey presents a taxonomy covering efficient LLM research from model-centric, data-centric, and framework-centric perspectives.The taxonomy is intended as a systematic and comprehensive review of distinct yet interconnected efficiency topics.Source: §1 · p. 2, Figure 3 · p. 4

Model-Centric Methods

  • Model-centric methods address model compression, efficient pre-training, fine-tuning, inference, and architecture design.These methods include algorithm-level and system-level techniques focused on the model itself.Source: §1 · p. 2

Data-Centric Methods

  • Data-centric methods improve LLM efficiency through data selection and prompt engineering, emphasizing data quality and structure.The survey treats data as crucial to efficiency alongside the model.Source: §1 · p. 2

LLM Frameworks

  • LLM frameworks specialize training, fine-tuning, inference, and serving by providing optimizations and features not built into mainstream frameworks such as TensorFlow and PyTorch.The survey covers framework features, underlying libraries, and specializations.Source: §1 · p. 2

Positioning and resources

  • The survey focuses on efficiency techniques for LLMs exceeding billions of parameters and maintains a GitHub repository of its featured papers for ongoing updates.The repository is intended to help researchers and practitioners navigate the literature and support further research.Source: §1 · pp. 3–4

2 Model-Centric Methods · 2.1 Model Compression

Model compression improves LLM efficiency by reducing model size and arithmetic operations, primarily through post-training techniques that avoid costly retraining. The survey organizes these methods into four orthogonal categories: quantization, parameter pruning, low-rank approximation, and knowledge distillation.

2.1 Model Compression

  • Compression methods reduce LLM size and computation, with four orthogonal categories covering quantization, parameter pruning, low-rank approximation, and knowledge distillation.Most LLM compression approaches use post-training settings to avoid resource-intensive retraining.Source: §2.1 · p. 5

2.1.1 Quantization

  • Quantization converts high-precision weights or activations to low-precision types and includes PTQ and QAT, offering superior compression-accuracy trade-offs over pruning and low-rank approximation (Li et al., 2024c).PTQ uses a small calibration dataset and comprises weight-only and weight-activation co-quantization; QAT learns quantization-friendly representations during training.Source: §2.1.1 · p. 5
  • PTQ methods preserve accuracy through increasingly low-bit weight quantization and outlier handling, reaching 2 bits per weight with QuIP and 3–4 bits with GPTQ, SpQR, and related methods.GPTQ quantizes 175-billion-parameter GPT models in roughly four GPU hours with minimal accuracy loss, while AWQ, OWQ, SpQR, and FineQuant preserve vulnerable or salient weights at higher precision.Source: §2.1.1 · p. 6
  • QAT is more expensive than PTQ because it requires the complete training dataset, but it supports strong low-bit compression, including 14.4× and 13.4× compression for QuantGPT and 1-bit BitNet.LLM-QAT also quantizes the key-value cache and performs better than training-free methods in low-bit settings, while BitNet reduces memory footprint and energy consumption versus FP16 Transformers.Source: §2.1.1 · p. 7

2.1.2 Parameter Pruning

  • Parameter pruning removes redundant weights through structured or unstructured sparsity, with structured methods targeting deployable groups and unstructured methods offering greater flexibility but harder hardware deployment.Structured approaches such as Sheared LLaMA prune LLaMA2-7B to 1.3B parameters, whereas SparseGPT reaches about 60% unstructured sparsity on OPT-135B with only a slight performance drop.Source: §2.1.2 · pp. 7–8

2.1.3 Low-Rank Approximation

  • Low-rank approximation replaces weight matrices with smaller factors, reducing parameters through methods such as TensorGPT, LoSparse, FWSVD, ASVD, and SVD-LLM.FWSVD uses task-specific Fisher information, ASVD is training-free and activation-aware, and SVD-LLM links singular-value truncation to compression loss while outperforming FWSVD and ASVD in compression performance and speed.Source: §2.1.3 · p. 8

2.1.4 Knowledge Distillation

  • Knowledge distillation transfers knowledge from a large teacher to a smaller student, but its distillation process is more resource-demanding than other compression methods and is divided into white-box and black-box approaches.White-box methods use teacher parameters or logits, whereas black-box methods use only teacher-generated outputs.Source: §2.1.4 · p. 9
  • Black-box distillation transfers capabilities through generated instructions, explanations, counterfactual data, or chain-of-thought rationales, enabling smaller models to improve reasoning with less data or computation.Multitask-ICT transfers few-shot capabilities, explanation distillation improves multiple scenarios, and SOCRATIC CoT enables GPT-2 large to outperform a 10× larger GPT-3 6B model.Source: §2.1.4 · pp. 9–10

2.2 Efficient Pre-Training

Efficient pre-training reduces compute, time, memory, and energy costs through complementary techniques spanning precision, model scaling, initialization, optimizers, and distributed systems. These methods improve numerical stability, convergence, resource utilization, or training speed under different hardware and model-growth conditions.

Mixed Precision Training

  • Mixed precision lowers arithmetic precision during propagation while preserving higher-precision updates; BFLOAT16 addresses FP16 accuracy loss through its greater numerical range (Kalamkar et al., 2019).AMP stores weights, activations, and gradients in FP16 while maintaining FP32 update precision; its improved version removes the FP32 weight copy but AdamW remains internally FP32.Source: §2.2 · p. 11

Scaling Models

  • Scaling smaller models into larger ones accelerates pre-training convergence and reduces training costs; LiGO outperforms bert2BERT, which saves about 30% computational costs.Such methods reuse smaller-model computation while progressively increasing model depth, breadth, or input length.Source: §2.2 · p. 12

Initialization Techniques

  • Initialization methods accelerate convergence by preserving signal identity, balancing variances, or stabilizing deep Transformer optimization, as in Fixup, ZerO, ReZero, and DeepNet (Wang et al., 2024a).DeepNet’s Post-LN-init stabilizes layer-normalization inputs and mitigates gradient vanishing.Source: §2.2 · p. 12

Training Optimizers

  • Lion is more memory-efficient than Adam by tracking only momentum, while Sophia doubles pre-training speed through clipped gradient-to-Hessian updates (Chen et al., 2023g; Liu et al., 2024a).Sophia uses moving-average gradients and an estimated Hessian to moderate update sizes, improving memory utilization and efficiency.Source: §2.2 · pp. 12–13

System-Level Pre-Training Efficiency Optimization

  • Distributed optimizations partition or offload training states and tune hardware usage: ZeRO-3 reduces runtime memory with about 50% more communication, while Perseus cuts energy consumption by up to 30%.ZeRO-Offload and ZeRO-Infinity move states to CPU memory or NVMe, though slow CPU–GPU data movement can impair performance; Zeus and Perseus optimize GPU configurations for energy efficiency.Source: §2.2 · p. 13

2.3 Efficient Fine-Tuning

Efficient fine-tuning reduces LLM fine-tuning costs through parameter-efficient and memory-efficient techniques. PEFT updates only a small set of added parameters, whereas MEFT targets memory savings during fine-tuning.

2.3 Efficient Fine-Tuning

  • Efficient fine-tuning comprises parameter-efficient fine-tuning and memory-efficient fine-tuning, addressing computational cost through complementary strategies.The taxonomy is summarized in Figure 8.Source: §2.3 · p. 13

2.3.1 Parameter-Efficient Fine-Tuning

  • PEFT freezes the LLM backbone and updates few added parameters through low-rank adaptation, adapters, prefix tuning, or prompt tuning.These four categories provide the section’s main taxonomy for parameter-efficient adaptation.Source: §2.3.1 · p. 14
  • LoRA updates low-rank matrices A and B while freezing W, and extensions support adaptive ranks, long contexts, mixture-of-experts routing, compressed models, and weight tying.LoRA uses W ← W + A·B; LoRA-FA achieves comparable accuracy to full-parameter fine-tuning and LoRA while avoiding full-rank input-activation storage.Source: §2.3.1 · p. 14
  • Adapters insert bottleneck modules, while prompt-based methods add trainable vectors at each layer or input layer; several variants improve few-shot or multitask adaptation efficiency.Adapters may be series or parallel, and prompt methods include task-specific, modular, and multitask initialization or routing strategies.Source: §2.3.1 · pp. 15–16

2.3.2 Memory-Efficient Fine-Tuning

  • MEFT reduces fine-tuning memory through quantization, activation or optimizer-state savings, and zeroth-order optimization while retaining or approaching full-model performance.QLoRA uses 4-bit NormalFloat quantization with LoRA; Selective Fine-Tuning uses up to one-third of full-model GPU memory, LOMO reduces gradient-tensor requirements to O(1), and MeZO can train a 30-billion-parameter model on one A100 80GB GPU.Source: §2.3.2 · p. 16

2.4 Efficient Inference

Efficient inference reduces LLM inference costs through algorithm-level techniques, including speculative decoding and KV-cache optimization, and system-level techniques tailored to hardware architectures. These methods accelerate decoding, reduce memory use, improve throughput, and address user-perceived quality of experience.

Algorithm-Level Inference Efficiency Optimization

  • Algorithm-level inference efficiency uses speculative decoding and KV-cache optimization to accelerate generation and reduce the memory required for past-token key-value pairs.Speculative decoding uses smaller or auxiliary models, references, or parallel prediction structures, while KV-cache methods compress or evict cached values.Source: §2.4 · pp. 16–18

Speculative Decoding

  • 1.36x improvement over standard speculative decoding is reported by Staged Speculative (Spector & Re, 2023), while LLMA (Yang et al., 2023b) exceeds two-times speedup with unchanged greedy-decoding results.Staged Speculative uses tree-structured speculative batches and an additional initial-model phase; LLMA copies tokens from related references and evaluates them concurrently.Source: §2.4 · p. 17

KV-Cache Optimization

  • 2.6x less peak inference memory is achieved by KIVI (Zirui Liu et al., 2023), while KV-cache methods also quantize caches or evict less important and outdated tokens.KIVI uses tuning-free 2-bit quantization; H2O, Scissorhands, and StreamingLLM retain selected or recent keys and values, with StreamingLLM providing constant memory and decoding speed after cache filling.Source: §2.4 · pp. 17–18

System-Level Inference Efficiency Optimization

  • 4.86x and 2.18x acceleration on Nvidia and AMD GPUs, respectively, is achieved by FlashDecoding++ (Hong et al., 2023) over HuggingFace implementations.System-level methods coordinate hardware resources, scheduling, memory, batching, attention computation, and quality-of-experience objectives; PagedAttention improves throughput by 2-4× versus FasterTransformer and Orca at the same latency.Source: §2.4 · pp. 18–19

2.5 Efficient Architecture Design

Efficient architecture design optimizes LLM architectures and computational processes to improve performance and scalability while reducing resource consumption. Efficient attention addresses the quadratic cost of attention through several lightweighting strategies.

2.5 Efficient Architecture Design

  • Efficient architecture design targets improved LLM performance and scalability while minimizing resource consumption.Figure 12 summarizes existing efforts in designing efficient LLM architectures.Source: §2.5 · p. 19

2.5.1 Efficient Attention

  • Attention’s quadratic time and space complexity slows LLM pre-training, inference, and fine-tuning, motivating lightweight attention techniques.These techniques include sharing-based attention, kernelization or low-rank methods, fixed and learnable patterns, and hardware-assisted attention.Source: §2.5.1 · p. 19
  • Sharing-based attention accelerates inference by sharing key-value heads, as illustrated by LLaMA-2’s use of MQA and GQA.Traditional multi-head attention assigns distinct linear components to each head, whereas MQA and GQA optimize autoregressive decoding through KV-head sharing.Source: §2.5.1 · p. 19

W1 W2 x

The section surveys model-centric efficiency methods spanning attention optimization, sparse Mixture-of-Experts, long-context adaptation, and Transformer-alternate architectures. These approaches reduce computation, memory, or communication through sparsity, efficient representations, recurrence, retrieval, or near-linear sequence processing.

W1 W2 x

  • Attention efficiency methods use shared heads, low-rank or kernelized representations, fixed or learned sparsity patterns, and hardware-aware execution to reduce attention costs.MQA accelerates inference but may reduce quality, GQA improves that tradeoff, FlashAttention reduces HBM–SRAM communication, and HyperAttention speeds long-context inference and training with minimal degradation versus FlashAttention.Source: W1 W2 x · pp. 20–21

2.5.2 Mixture of Experts (MoE)

  • MoE models improve efficiency by activating only selected experts, with Mixtral 8x7B outperforming LLaMA-2 70B on most listed benchmarks using 12.9B parameters per token.System optimizations further report 1.37×–17.87× speedups for FasterMoE, up to 9× cheaper inference for DeepSpeed-MoE, and 40% shorter training for MegaBlocks.Source: §2.5.2 · pp. 22–23

2.5.3 Long Context LLMs

  • Long-context LLM methods adapt beyond training lengths through positional scaling, recurrence, segmentation or sliding windows, and memory-retrieval augmentation.Examples include RAPTOR’s recursive document trees, which improve multi-step reasoning, and retrieval augmentation that can match a fine-tuned 16K-context model with a 4K-context LLM using less computation.Source: §2.5.3 · pp. 23–25

2.5.4 Transformer-Alternate Architectures

  • Transformer-alternate architectures replace quadratic attention with SSMs or other sequential operators, offering near-linear or sub-quadratic processing for long sequences.GSS is reported as 2–3 times faster than DSS while maintaining perplexity, Mamba achieves 5x throughput than [the truncated comparison], and SeqBoat targets a state-of-the-art quality-efficiency trade-off.Source: §2.5.4 · pp. 25–26

3 Data-Centric Methods

Data-centric methods improve LLM efficiency by selecting informative training data and engineering prompts for pretraining, fine-tuning, and inference. The survey covers data selection, few-shot prompting, prompt compression, and prompt generation, including methods that reduce data, context, or computation while preserving or improving performance.

3.1 Data Selection

  • Data selection improves efficiency by using informative, diverse, or curated subsets for LLM pretraining and fine-tuning.The survey organizes data selection around pretraining efficiency and fine-tuning efficiency.Source: §3.1 · p. 26

3.1.1 Data Selection for Efficient Pre-Training

  • DSIR matches expert-curated performance across eight target distributions and outperforms random selection and heuristic filtering by 2–2.5% on GLUE.DSIR uses importance resampling based on feature-based data importance, while DoReMi addresses distribution shift between pretraining and downstream tasks.Source: §3.1.1 · pp. 26–27

3.1.2 Data Selection for Efficient Fine-Tuning

  • Fine-tuning data selection yields 2–23% relative improvement, while AlpaGasus uses 9K filtered samples, outperforms full-data training, and reduces fine-tuning time by 5.7x.Other methods show that less than 0.5% of the original dataset can train task-specific models without compromising performance, and LIMA matches or exceeds GPT-4 in 43% of considered cases.Source: §3.1.2 · p. 27

3.2 Prompt Engineering

  • Prompt engineering improves inference efficiency by tailoring inputs, and its techniques include few-shot prompting, prompt compression, and prompt generation.For simple tasks such as semantic classification, prompt engineering can substitute fine-tuning while achieving high accuracy.Source: §3.2 · p. 27

3.2.1 Few-Shot Prompting

  • Few-shot prompting guides LLMs with a small set of demonstrations, enabling task adaptation without fine-tuning and improving efficiency through concise prompts.Demonstrations are commonly organized through selection and ordering, while templates use instruction generation and multi-step reasoning.Source: §3.2.1 · pp. 27–29
  • Automatic instructions improve prompting, with Self-Instruct achieving a 33% improvement over vanilla GPT3 and optimized prompts improving performance by up to 8% on GSM8K and 50% on Big-Bench Hard.Multi-step reasoning trades lower efficiency than fine-tuning for often weaker accuracy improvement; Least-to-Most Prompting reaches 99.7% on GPT-3 code-davinci-002 with 14 exemplars, while GoT raises sorting quality by 62% and cuts cost by over 31%.Source: §3.2.1 · pp. 29–30

3.2.2 Prompt Compression

  • Prompt compression reduces input cost through compact representations, achieving up to 26x compression with up to 40% lower FLOPs and 17.1% better performance with 4x fewer tokens.Other methods report 24.6% fewer tokens, 4x context compression, and 20x compression with 98% BLEU reconstruction.Source: §3.2.2 · pp. 30–31

3.2.3 Prompt Generation

  • Prompt generation automatically creates task-specific prompts, with TempLM reducing BART’s unfaithfulness rate from 83% to 0% and PromptGen outperforming AutoPrompt on LAMA.AutoPrompt uses gradient-guided search and outperforms manually created prompts on LAMA for factual knowledge elicitation.Source: §3.2.3 · p. 31

4 LLM Frameworks

LLM frameworks are grouped by support for training, fine-tuning, and inference, with training-oriented systems targeting scalable computation, memory, communication, and reliability, and inference systems targeting throughput, memory, latency, and deployment. The section surveys representative frameworks and their parallelism, memory-management, quantization, batching, compilation, and monitoring features.

Framework taxonomy

  • Frameworks divide into training/fine-tuning systems and inference systems, targeting respectively scalable efficiency and reliability or improved throughput, memory usage, latency, and deployment.Table 2 summarizes the frameworks and their key features.Source: §4 · p. 31, Table 2 · p. 32

Training and serving frameworks

  • DeepSpeed integrates training and serving, using ZeRO-Inference to distribute model states across GPUs and CPUs and ZeRO-Offload (Ren et al., 2021) to train across CPU and GPU memory.DeepSpeed-Inference also provides operation fusion, Dynamic SplitFuse, continuous batching, quantization, 3D parallelism, and RLHF support.Source: §4 · pp. 31, 33, 34
  • Megatron (Shoeybi et al., 2020) improves distributed training through tensor decomposition, 3D and sequence parallelism, selective activation recomputation, and FasterTransformer-based inference across FP16 and INT8 modes.Colossal-AI unifies multiple parallelism methods, heterogeneous memory management, quantization, RLHF, and mixed-precision training, while Nanotron emphasizes minimal-overhead training optimizations including ZeRO-1 and DoReMi (Xie et al., 2023a).Source: §4 · p. 34

Training frameworks

  • MegaBlocks (Gale et al., 2023) trains Mixture-of-Experts models with dropless-MoE block-sparse operations, avoiding token dropping and the capacity-factor hyper-parameter.FairScale uses FSDP and AdaScale, Pax provides JAX-based distributed training, and Composer combines acceleration methods, FSDP, elastic checkpointing, streaming datasets, and trainer APIs.Source: §4 · p. 34

Inference and serving frameworks

  • vLLM (Kwon et al., 2023) improves inference memory management with PagedAttention, which flexibly shares uniformly sized KV-cache blocks, supports on-demand allocation, and swaps or recomputes blocks to prevent overflow.It also supports continuous batching, Multi-LoRA, quantization, and efficient Nvidia and AMD GPU kernels; related serving systems add tensor parallelism, batching, quantization, compilation, autoscaling, and monitoring.Source: §4 · pp. 35–36

5 Concluding Remarks

The survey systematically reviews efficient LLMs research through a taxonomy spanning model-centric, data-centric, and framework-centric perspectives, aiming to support broader access and future progress.

5 Concluding Remarks

  • The survey covers algorithm- and system-level efficiency techniques alongside optimized LLM frameworks, and aims to help researchers and practitioners enter the field and inspire further research.It frames efficiency as increasingly important for LLMs and LLM-oriented systems and as a means of democratizing LLMs.Source: §5 · p. 36

Related papers

Loading 2312.03863v4…