PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Large Language Models: A Survey

Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, Mi Zhang

arXiv:2312.03863v4cs.CLcs.AI

TL;DR

LLMs offer broad capabilities but demand substantial computational resources, motivating efficient techniques. This survey systematically reviews efficient LLM research through model-centric, data-centric, and framework-centric categories, including techniques and specialized frameworks. The surveyed literature includes sparse MoE systems with up to 4.5x faster and 9x cheaper inference than quality-equivalent dense models.

  • Problem

    LLMs’ capabilities require substantial resources, creating a need for techniques that address training, inference, and operational efficiency.

  • Method

    The survey organizes efficient LLM research into model-centric, data-centric, and framework-centric categories and reviews representative techniques, prompting methods, and frameworks.

  • Results

    Sparse MoE systems provide up to 4.5x faster and 9x cheaper inference than quality-equivalent dense models, while DeepSpeed-MoE provides up to 7.3x reduction in inference latency and cost versus existing MoE solutions.

  • Takeaways & Limitations

    The survey provides a systematic resource for researchers and practitioners to navigate efficient LLM research and encourages further work in the field.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.

1 Introduction

LLMs achieve strong performance across diverse tasks, but their scale creates substantial training and inference resource demands. This survey organizes efficiency research into model-centric, data-centric, and framework-centric categories.

  • LLMs support natural language understanding, generation, complex reasoning, and domain-specific tasks, with capabilities associated with billions or trillions of parameters and massive training datasets.
  • Larger LLaMA models achieve better performance while requiring exponentially more GPU hours as model size increases.Inference also contributes significantly to operational cost as models scale.
  • Mistral-7B achieves comparable performance and much higher throughput than LLaMA-1-33B using grouped-query attention and sliding window attention.
  • The survey classifies efficient LLM research into model-centric, data-centric, and framework-centric perspectives.
  • Model-centric coverage includes compression, pre-training, fine-tuning, inference, and architecture design, while data-centric coverage includes data selection and prompt engineering.
  • The survey also reviews specialized LLM frameworks and maintains a GitHub repository of featured papers for incorporating emerging research.

2.1 Model Compression

Model compression improves LLM efficiency by reducing model size and arithmetic operations, typically without resource-intensive retraining. The survey covers quantization, pruning, low-rank approximation, and knowledge distillation as distinct compression approaches.

  • Model compression reduces LLM sizes and arithmetic operations, with most approaches designed for post-training use to avoid expensive retraining.
  • Quantization: Quantization converts high-precision weights or activations into lower-precision types and includes post-training quantization and quantization-aware training.Post-training quantization uses a small calibration dataset, whereas quantization-aware training occurs during training and is more expensive.
  • Quantization: GPTQ compresses GPT weights to 3 or 4 bits and quantizes a 175-billion-parameter GPT model in roughly four GPU hours with minimal accuracy degradation.
  • Parameter pruning: Parameter pruning removes redundant or less important weights through structured or unstructured approaches.Unstructured pruning offers greater flexibility and lower accuracy drop but generally produces irregular sparsity that is difficult to deploy on hardware.
  • Low-rank approximation: Low-rank approximation replaces a weight matrix W with smaller matrices U and V such that W ≈ UVᵀ, reducing parameters and improving efficiency.
  • Knowledge distillation: Knowledge distillation transfers knowledge from a large teacher LLM to a smaller student LLM and is categorized into white-box and black-box methods.Distillation can be effective but requires a resource-demanding distillation process.

2.2 Efficient Pre-Training

Efficient pre-training reduces compute, time, memory, and energy costs through complementary techniques spanning numerical precision, model scaling, initialization, optimization, and distributed systems.

  • Mixed Precision Training: Mixed precision training uses low-precision computation while maintaining high-precision weights for updates.Automatic Mixed Precision stores weights, activations, and gradients in FP16 while retaining a master FP32 weight copy.
  • Scaling Models: Scaling models transfers knowledge from smaller models to larger ones to accelerate convergence and reduce training costs.Progressive stacking transfers knowledge from a simpler model to a more complex one.
  • Initialization Techniques: Initialization techniques improve pre-training efficiency by accelerating model convergence.Examples include variance-balancing initialization and methods that set the backbone to zero while preserving signal identity.
  • Training Optimizers: New training optimizers target the memory and computational costs of Adam and AdamW during LLM pre-training.The discussed approach moderates update sizes and addresses non-convexity and rapidly changing Hessians through clipping.
  • System-Level Pre-Training Efficiency Optimization: System-level pre-training optimizations partition training states across devices for large-scale distributed training.ZeRO provides three optimization stages that partition various training states across different devices.

2.3 Efficient Fine-Tuning

Efficient fine-tuning reduces LLM adaptation costs through parameter-efficient and memory-efficient approaches that modify or represent training states more economically.

  • Efficient Fine-Tuning: Efficient fine-tuning methods are grouped into parameter-efficient fine-tuning and memory-efficient fine-tuning.PEFT focuses on reducing updated parameters, whereas MEFT focuses on memory savings during fine-tuning.
  • Parameter-Efficient Fine-Tuning: PEFT freezes the LLM backbone and updates only a small set of added parameters.Its categories include low-rank adaptation, adapter-based tuning, prefix tuning, and prompt tuning.
  • Low-Rank Adaptation: LoRA adapts frozen weight matrices with two trainable low-rank matrices, A and B.It updates W through W ← W + ΔW = W + A·B while leaving the original large weight matrix frozen.
  • Adapter-based Tuning: Adapter-based tuning inserts bottleneck modules that down-project, transform, and up-project feature vectors.Adapters may be arranged in series after attention and feed-forward modules or in parallel alongside them.
  • Prefix and Prompt Tuning: Prefix tuning adds task-specific trainable prefix tokens to each LLM layer, whereas prompt tuning adds trainable tokens only at the input layer.Prompt tuning keeps the entire pre-trained model fixed while adding extra trainable tokens to downstream-task inputs.
  • Memory-Efficient Fine-Tuning: QLoRA reduces fine-tuning memory usage by quantizing the model to 4-bit NormalFloat and training added LoRA weights.The passage reports no performance degradation compared with standard full-model fine-tuning.

2.4 Efficient Inference

Efficient inference reduces LLM inference costs through algorithm-level decoding and cache methods alongside system-level hardware-aware execution and scheduling optimizations.

  • Efficient Inference: Efficient inference techniques are grouped into algorithm-level and system-level optimizations.The survey organizes inference methods according to whether they optimize algorithms or execution systems.
  • Speculative Decoding: Speculative decoding uses a smaller draft model to generate candidate prefixes in parallel for evaluation by a larger target model.This decoding strategy speeds autoregressive sampling by separating preliminary token generation from target-model evaluation.
  • Speculative Decoding: 1.36x over standard speculative decoding is reported for a distributed-serving approach that runs a faster model K times before evaluating preliminary output.The approach generates larger speculative batches and adds an initial-model speculative-decoding phase.
  • KV-Cache Optimization: KV-cache optimization reduces memory consumption and inference latency by shrinking or evicting cached Key-Value pairs.KV-cache size grows substantially with generated token length; KVQuant quantizes the LLaMA cache to 3-bit.
  • System-Level Inference Efficiency Optimization: System-level inference methods coordinate hardware resources, quantization, model partitioning, and scheduling under specific architectures.FlexGen combines GPU, CPU, and disk memory and computation, while Orca uses iteration-level scheduling for variable-length batched sequences.
  • System-Level Inference Efficiency Optimization: Up to 4.86x acceleration on Nvidia GPUs and 2.18x on AMD GPUs is reported for FlashDecoding++ versus HuggingFace implementations.The system uses asynchronous softmax, double buffering for flat GEMM optimization, and heuristic dataflow.

2.5 Efficient Architecture Design

Efficient architecture design optimizes model architecture and computation to improve scalability while minimizing resource consumption, especially by addressing attention's quadratic complexity.

  • Efficient Architecture Design: Efficient architecture design targets performance and scalability while minimizing computational resource consumption.The survey summarizes existing efforts on designing efficient LLM architectures.
  • Attention Efficiency: Attention's quadratic time and space complexity slows LLM pre-training, inference, and fine-tuning.The survey motivates lightweight attention techniques as a response to this computational burden.
  • Attention Efficiency: Lightweight attention techniques include sharing-based, kernelization or low-rank, fixed-pattern, learnable-pattern, and hardware-assisted strategies.Sharing-based attention accelerates inference by sharing Key-Value heads.

W1 W2 x

Efficient LLM research spans attention optimization, sparse expert architectures, long-context processing, and alternative architectures, with methods targeting both computational and memory costs.

  • Attention Optimization: Attention optimizations reduce efficiency costs through shared key-value heads, low-rank representations, fixed or learned sparsity patterns, and hardware-aware memory access.MQA and GQA reduce inference overhead through key-value sharing; other methods sparsify or approximate attention, while FlashAttention reduces GPU memory transfers.
  • Attention Optimization: HyperAttention improves inference and training speeds on long contexts with minimal performance degradation compared with FlashAttention.It combines large-entry removal, sortLSH, and row-norm sampling for fast matrix multiplication.
  • Mixture of Experts: Sparse MoE models scale to trillion-parameter regimes while achieving strong language-modeling and downstream-task performance relative to dense models.Reported examples include GShard with up to one trillion parameters and Artetxe et al.’s 1.1T-parameter sparse language models.
  • Mixture of Experts: System-level MoE optimizations address load imbalance, scheduling, communication congestion, and inference cost across distributed hardware.FasterMoE reports 1.37×–17.87× speedups, while other systems report up to 3×, 4.5×, 9×, and 7.3× improvements under stated comparisons.
  • Long Context LLMs: Long-context methods use hierarchical summaries or memory retrieval to process extensive documents and improve multi-step reasoning or long-text modeling.MemWalker and RAPTOR construct recursive structures, while memory-retrieval methods fetch or cache relevant past context during inference.
  • Transformer-Alternate Architectures: Sparse Modular Activation unifies MoE, adaptive computation, dynamic routing, and sparse attention to achieve a quality-efficiency trade-off in SeqBoat.The mechanism extends state-space models and is used to develop the SeqBoat architecture.

3 Data-Centric Methods

Data-centric methods improve LLM efficiency by selecting more useful training examples and designing prompts that reduce inference demands while preserving task performance.

  • Data Selection: Data selection improves pre-training and fine-tuning efficiency by retaining informative, diverse, or curated subsets of examples.The survey covers selection for both pre-training and fine-tuning, including importance resampling and curated instruction data.
  • Data Selection: DSIR uses importance resampling and achieves performance similar to expert curation across eight target distributions.For general-domain pre-training, it also outperforms random selection and heuristic filtering baselines.
  • Data Selection: Fine-tuning with selected data yields an 2-23% relative improvement, while Instruction Mining-curated datasets outperform generic datasets in 42.5% of cases.The cited results concern retrieval of similar labeled examples and quality-based instruction-data selection, respectively.
  • Few-Shot Prompting: Few-shot prompting guides LLMs with a limited set of similar demonstrations selected from a training corpus.The demonstrations provide task-relevant examples without fine-tuning the model.
  • Few-Shot Prompting: Demonstration organization affects inference efficiency through two stages: selecting useful examples and ordering them appropriately.Poor organization can add unnecessary information, while suitable ordering can reach good generation quality with fewer samples.
  • Template Formatting: Template formatting makes prompts concise through instruction generation and multi-step reasoning.Automatic instruction generation can improve vanilla GPT-3 by 33%, whereas multi-step reasoning is more efficient than fine-tuning but often less accurate.
  • Prompt Compression: Prompt compression reduces input-processing cost by condensing prompts or learning compact representations, including up to 26x compression and 40% fewer floating-point operations.Other surveyed methods report 24.6% lower token counts and 4x context compression.

4 LLM Frameworks

LLM frameworks provide specialized infrastructure for efficient training, fine-tuning, inference, and serving by combining parallelism, memory, batching, quantization, and deployment optimizations.

  • Framework Scope: Training and fine-tuning frameworks target computation, memory, communication, and reliability, while inference frameworks target throughput, memory footprint, and latency.The survey groups frameworks according to whether they support training, fine-tuning, inference, or combinations of these tasks.
  • Training and Serving Frameworks: DeepSpeed integrates training and serving through ZeRO-based optimizations, parallelism, quantization, attention, batching, and Dynamic SplitFuse.Its framework components include DeepSpeed-Inference, ZeRO-Inference, ZeRO-Offload, and related serving optimizations.
  • Framework Comparison: The framework comparison covers support for training, fine-tuning, and inference alongside each system’s distinctive optimization features.Examples include data and tensor parallelism, continuous batching, quantization, compiler acceleration, and monitoring metrics.
  • Training Frameworks: Megatron distributes tensor operations across GPUs and combines 3D, sequence, and expert parallelism with attention and activation-recomputation optimizations.These design choices target processing speed and memory utilization for large-model training and serving.
  • Training Frameworks: Colossal-AI unifies multiple parallelism techniques, ZeRO optimization, auto-parallelism, expert parallelism, and heterogeneous memory management through a streamlined API.Its comparison also lists PagedAttention, FlashAttention-2, quantization, and RLHF support.
  • Specialized Frameworks: Nanotron emphasizes minimal overhead, MegaBlocks targets dropless Mixture-of-Experts training, and FairScale provides large-scale PyTorch training with FSDP.MegaBlocks avoids token dropping through block-sparse operations, while FairScale uses AdaScale as its distributed optimizer.
  • Inference Frameworks: vLLM improves inference and serving by storing KV-cache blocks flexibly with PagedAttention, enabling block-level memory sharing.Its design is compared with contiguous KV-cache storage and resembles virtual memory management.

5 Concluding Remarks

The survey reviews efficient LLM research systematically through a taxonomy spanning model-centric, data-centric, and framework-centric techniques, with the broader aim of democratizing LLMs and supporting future research.

  • Concluding Remarks: The survey provides a systematic review of efficient LLMs, an area aimed at democratizing access to large language models.It covers algorithm-level and system-level techniques and specialized LLM frameworks.
  • Concluding Remarks: Its taxonomy organizes research from model-centric and data-centric perspectives and additionally reviews frameworks with specialized efficiency features.The authors position the survey as a resource for researchers and practitioners entering the field.
  • Concluding Remarks: The authors expect efficiency to become increasingly important in LLMs and LLM-oriented systems and hope the survey catalyzes new research.The stated intended audience includes both researchers and practitioners.
Loading 2312.03863v4…