PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding LLMs: A Comprehensive Overview from Training to Inference

Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, Bao Ge

arXiv:2401.02038v2cs.CL

TL;DR

LLM use has expanded rapidly, creating a need for cost-efficient training and deployment. This paper surveys the evolution and technical foundations of LLM training, inference, and application, concluding with future directions involving improved efficiency, architectures, and broader use.

  • Problem

    The growing use of LLMs creates a need to understand training and inference techniques that support cost-efficient development and deployment.

  • Method

    The paper provides a comprehensive survey of LLM training, inference and deployment technologies, utilization, and future directions.

  • Results

    The review synthesizes training and inference processes and identifies improved model architectures, training efficiency, and broader industrial applications as future directions.

  • Takeaways & Limitations

    Effective LLM development requires understanding large-scale data handling, distributed parallel training, and the engineering capabilities needed for training and deployment.

  • Takeaways & Limitations

    LLMs remain exposed to privacy leakage and adversarial vulnerabilities, including jailbreaks that can produce toxic content.

Abstract

from arXiv · show

The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There's an increasing focus on cost-efficient training and deployment within this context. Low-cost training and deployment of LLMs represent the future development trend. This paper reviews the evolution of large language model training techniques and inference deployment technologies aligned with this emerging trend. The discussion on training includes various aspects, including data preprocessing, training architecture, pre-training tasks, parallel training, and relevant content related to model fine-tuning. On the inference side, the paper covers topics such as model compression, parallel computation, memory scheduling, and structural optimization. It also explores LLMs' utilization and provides insights into their future development.

1. Introduction

The evolution from statistical and neural language models to large pre-trained models has enabled strong general-purpose language capabilities and broad downstream use. This review surveys LLM training and inference techniques with a focus on cost-efficient development and deployment.

  • Model evolution: Language models evolved from statistical models to neural and then pre-trained architectures, with Transformers shaping modern LLM development.The Transformer architecture introduced parallel self-attention and changed pre-training and fine-tuning practices.
  • Scaling and capabilities: Scaling model size and training data can significantly improve performance, with larger models exhibiting emergence, strong reasoning, and few-shot in-context learning.These capabilities support applications across diverse downstream tasks and domains.
  • Large language models: LLMs typically contain more than 6–10 billion parameters and are prominently represented by the GPT series.ChatGPT’s release in November 2022 marked a pivotal moment in LLM and artificial-intelligence development.
  • Review scope: The paper comprehensively reviews LLM training and inference techniques to support researchers developing, deploying, and applying these models.Its organization covers foundational background, training, inference and deployment, utilization, and future directions.

2. Background Knowledge

This section introduces Transformer-based language modeling, focusing on self-attention, multi-head attention, positional encoding, and prompt learning as foundations for modern NLP systems. It also describes how prompt learning adapts pretrained models to downstream tasks without retraining the entire model.

  • Transformer: Transformer is an attention-based sequence model with encoder and decoder modules that supports parallel computation and captures complex language dependencies.Its encoder and decoder are built from repeated attention and feed-forward layers.
  • Self-Attention: Self-attention selects and weights relevant information among sequence elements to model internal correlations and long-range dependencies.It computes query, key, and value vectors, then derives attention weights from query-key similarity.
  • Self-Attention: Multi-head attention performs attention calculations in parallel across multiple heads, concatenating and linearly transforming their outputs.Different heads can capture different dependencies and patterns in the input sequence.
  • Positional Embedding: Positional encoding supplies token-order information that Transformer architectures otherwise lack because they avoid recurrent connections.Absolute encoding uses sine and cosine functions, while relative encoding represents distances between tokens and can better capture relationships in long sequences.
  • Positional Embedding: Transformer positional methods include RoPE, which represents relative positions through absolute encoding, and ALiBi, which adds distance-based biases to attention scores.RoPE is used in PaLM, LLaMA, and GLM-130B, while ALiBi is used in BLOOM.
  • Prompt Learning: Prompt learning guides pretrained language models toward task-specific outputs through designed prompts, adapting them to tasks without retraining the entire model.It can align downstream task formats with pretrained objectives and supports tasks such as question answering, text generation, and semantic understanding.

3. Training of Large Language Models

LLM training proceeds from data collection and preprocessing through pre-training and fine-tuning or alignment. The review emphasizes diverse corpora, data quality controls, and multiple dataset sources as foundations for model performance and security.

  • Training pipeline: LLM training comprises data collection and processing, pre-training with architectural and parallel-training choices, and fine-tuning or alignment.These stages organize the paper’s overview of model-training techniques.
  • Training data: Training corpora include books, web crawls, Reddit links, Wikipedia, and code from sources such as GitHub and Stack Overflow.The review describes these as distinct sources used to provide varied textual and domain content.
  • Training data: CommonCrawl contains over 250 billion web pages and adds 3–5 billion pages monthly, but its low-quality content requires preprocessing.The review also notes that 82% of GPT-3’s raw training tokens came from CommonCrawl.
  • Training data: LLMs are typically trained on multiple datasets, with dataset usage organized for comparison in Table 2.The review presents Table 2 as a reference for datasets used by distinct LLMs.
  • Data preprocessing: Preprocessing filters low-quality, toxic, and biased text, removes duplicates, and scrubs sensitive information to address performance, security, contamination, and privacy concerns.The described methods include heuristic or classifier-based filtering, deduplication, anonymization, redaction, and tokenization.

3.2. Architecture

LLMs use Transformer-based encoder-decoder or decoder-only designs, distinguished largely by which tokens each position can attend to. Causal decoding is unidirectional, whereas prefix decoding combines bidirectional prefix attention with autoregressive generation.

  • Architecture overview: Current LLMs are built on Transformer architectures that scale to several tens of billions or even a trillion parameters.The review focuses on encoder-decoder and decoder-only designs because encoder-only architectures are no longer used in the latest LLMs.
  • Encoder-decoder: Encoder-decoder models encode the input sequence and use decoder cross-attention over encoder representations to generate the target sequence autoregressively.The architecture contains separate encoder and decoder components.
  • Decoder-only: Decoder-only models generate tokens sequentially while attending to preceding tokens in the sequence.This design uses only the decoder component of the traditional Transformer.
  • Causal Decoder: Causal decoders restrict each token to attending to itself and preceding tokens through a unidirectional mask.GPT models represent this architecture, which is also used in BLOOM, OPT, Gopher, and LLaMA.
  • Prefix Decoder: Prefix decoders allow bidirectional attention within a prefix while preserving unidirectional attention for subsequent autoregressive generation.PaLM and GLM are representative models using this mask configuration.

3.3. Pre-training Tasks

Pre-training primarily uses language modeling, in which models predict tokens from context to learn linguistic representations. Some models instead recover randomly replaced text intervals through autoregressive prediction.

  • Language modeling: Language modeling is a common self-supervised pre-training task that predicts the next word from its preceding context.The task exposes models to vocabulary, grammar, semantics, and text structure.
  • Language modeling: Next-token prediction gradually encodes patterns and linguistic knowledge into the model’s parameters.The model performs the prediction task across large amounts of textual data and at each position in the text.
  • Alternative pre-training tasks: Some pre-training approaches randomly replace portions of text and train models to autoregressively recover the replaced intervals.This provides an alternative pre-training task alongside standard language modeling.

3.4. Model Training

The paper reviews parallel training methods that distribute data, model parameters, or layers across GPUs while reducing memory and communication costs. It also covers mixed precision, ZeRO optimization, asynchronous memory operations, checkpointing, and instruction-tuning datasets.

  • Parallel training: Parallel training uses collective communications including broadcast, reduce, all-reduce, reduce-scatter, and all-gather to coordinate GPUs.These operations respectively distribute, aggregate, or synchronize data across GPUs.
  • Data parallelism: Data parallelism partitions each batch across GPUs, which reduces intermediate-result memory but still requires each GPU to store the full model state.This can remain infeasible when parameters, gradients, and optimizers exceed a single GPU’s memory.
  • Model parallelism: Model parallelism partitions parameter matrices across GPUs, requiring identical inputs and all-gather communication to combine sub-results.The method distributes portions of a linear layer’s matrix so each GPU computes a sub-result.
  • ZeRO: ZeRO reduces redundancy in data-parallel training by partitioning gradients and parameters, then using reduce-scatter and all-gather operations for updates and synchronization.ZeRO3 stores only the parameters, gradients, and optimizer state associated with each GPU’s assigned parameter portion.
  • Memory and computation optimization: Pipeline parallelism assigns different Transformer layers to different GPUs, while mixed precision uses FP16 for faster computation and FP32 for optimizer updates.The paper also describes overlapping parameter retrieval with computation and checkpointing selected intermediate states to reduce memory use.
  • Training resources: The training discussion includes instruction-tuning datasets alongside parallelism, precision, memory scheduling, and checkpoint mechanisms.Table 3 is identified as covering commonly used instruction-tuning datasets.

3.5. Fine-Tuning

The paper divides fine-tuning into supervised fine-tuning, alignment tuning, and parameter-efficient tuning. It presents these methods as ways to adapt pretrained models, address harmful or unhelpful behavior, and reduce the cost of updating large models.

  • Fine-tuning categories: Fine-tuning methods are categorized into supervised fine-tuning, alignment tuning, and parameter-efficient tuning.This classification follows the paper’s three-stage account of LLM training.
  • Supervised fine-tuning: Supervised fine-tuning adapts a pretrained model to target tasks using labeled input-output data, including instruction-output pairs.Instruction tuning is described as a specific form of supervised fine-tuning.
  • Alignment tuning: Alignment tuning addresses outputs that may be false, biased, misleading, or otherwise divergent from human intent.The paper frames helpfulness, honesty, and harmlessness as alignment criteria.
  • Alignment tuning: RLHF trains a reward model from human feedback and uses reinforcement learning, such as PPO, to fine-tune the language model.The language model is treated as the policy and the reward model supplies the reinforcement-learning objective.
  • Parameter-efficient tuning: Parameter-efficient tuning reduces computational and memory overhead by updating only a small or additional parameter subset while keeping most pretrained parameters fixed.The paper identifies LoRA as a widely used parameter-efficient method for LLMs.
  • Safety tuning: Safety fine-tuning techniques include supervised safety demonstrations, safety RLHF, and safety context distillation.These techniques use adversarial prompts, safety reward models, or safety-prefixed demonstrations to improve responses to safety risks.

3.6. Evaluation

The paper presents LLM evaluation as covering capabilities, safety, security, bias, privacy, adversarial robustness, and evaluation methodology. It emphasizes combining automated metrics with manual assessment because each captures different aspects of model behavior.

  • Evaluation scope: LLM evaluation has become more complex because large models have broad capabilities alongside potential problems and risks.The paper motivates evaluation across multiple aspects rather than a single capability measure.
  • Evaluation datasets: Common evaluation datasets include GLUE, SuperGLUE, MMLU, CMMLU, ImageNet, and Open Images, depending on model modality and language.The paper also identifies SQuAD among datasets used for open-domain question answering.
  • Security evaluation: Security assessment should examine vulnerabilities, biases, inaccuracies, malicious use, and areas for safety improvement through techniques such as red teaming.The paper specifically highlights domain-specific red teaming as a way to identify model weaknesses.
  • Security evaluation: Security evaluation considers potential bias in training data, privacy leakage from user data, and susceptibility to adversarial attacks such as jailbreaks.The paper discusses gender or race bias, extraction of training data, and manipulation through role-playing or special suffixes.
  • Evaluation methods: Automated metrics such as BLEU, ROUGE, and BERTScore enable rapid large-scale comparison, whereas manual evaluation can reveal subtle issues in open-ended generation.Manual assessment is more time-consuming and subjective, so the paper recommends combining both approaches.

3.7. LLM Framework

The paper reviews frameworks that use distributed training to address the memory and efficiency challenges of models with billions or trillions of parameters. It covers general Transformer libraries, optimization toolkits, and systems supporting data, model, mixed-precision, or memory-partitioning techniques.

  • Framework overview: Distributed training helps fit models with billions to trillions of parameters into limited device memory while improving computation, communication, and development efficiency.The reviewed systems may use GPU, CPU, and NVMe memory.
  • General-purpose libraries: Hugging Face Transformers provides a user-friendly API for building and customizing Transformer-based models, supported by an active development community.The library continuously updates its models and algorithms.
  • Optimization frameworks: DeepSpeed supports ZeRO optimizer, gradient, and parameter partitioning, mixed-precision training, optimized CUDA operations, and CPU or NVMe offloading.It is described as a PyTorch-compatible library used to train large language models.
  • Optimization frameworks: BMTrain trains models with tens of billions of parameters using distributed methods while keeping code close to standalone training.It can be enabled in PyTorch pipelines with few code changes and without model refactoring.
  • Parallel-training frameworks: Megatron-LM supports model and data parallelism, mixed precision, and FlashAttention for training very large Transformer models.Its model-parallel implementation uses Transformer structure and synchronization primitives.
  • Framework ecosystem: Other frameworks include Colossal-AI and FastMoE, while general parallel-computing support is available in systems such as PyTorch, TensorFlow, JAX, and others.The paper presents framework choice as broader than a single implementation ecosystem.

4. Inference with Large Language Models

LLM inference seeks to reduce computational and storage costs through compression, memory management, parallelization, and computation-aware optimizations. These techniques address hardware constraints while supporting efficient deployment.

  • Cost-efficient inference: Inference cost reduction targets both computation and storage, motivated by rapidly increasing model scale, computational consumption, and carbon emissions.The review frames efficient inference as a response to the growing resource demands of large models.
  • Model compression: Model compression reduces deployment requirements through knowledge distillation, pruning, quantization, weight sharing, and low-rank decomposition.These methods respectively transfer knowledge, remove redundancy, reduce numerical precision, reuse parameters, or lower matrix rank.
  • Memory management: Memory scheduling organizes memory access to help deploy parameter-intensive LLMs on hardware with limited video memory.This is especially relevant to single consumer-grade GPUs.
  • Parallel computation: Inference parallelization uses data, tensor, and pipeline parallelism to distribute computation across GPU devices or model components.Data parallelism primarily increases system throughput, while tensor and pipeline parallelism partition model computation.
  • Transformer-specific optimization: FlashAttention and PagedAttention improve inference speed by using chunked computation to reduce storage overhead and memory accesses.The methods target the mismatch between fast forward computation and comparatively slower memory access.
  • Deployment frameworks: Inference frameworks integrate parallel computing, model compression, memory scheduling, and Transformer-specific optimizations for deployment.Framework selection depends on project requirements, hardware support, and user preference.

5. Utilization of LLMs

LLMs support broad task usage through prompting, ranging from zero-shot instructions to few-shot in-context learning. Users can access proprietary APIs, deploy open-source models locally, or fine-tune them for specialized domains.

  • Prompt-based utilization: LLMs can be applied across specialized domains by designing suitable prompts after pre-training and fine-tuning.Simple prompts can support zero-shot task completion, while more complex tasks can use few-shot in-context learning.
  • Prompt-based utilization: Few-shot in-context learning extends prompting by providing examples for tasks that simple instructions cannot accomplish.Figure 6 identifies in-context learning and chain of thought as utilization approaches.
  • Deployment approaches: Inference frameworks are part of the practical deployment ecosystem for running LLMs across application scenarios.The review separately lists inference frameworks in Table 4.
  • Deployment approaches: LLMs can be used through proprietary model APIs, local deployment of open-source models, or domain-specific fine-tuning followed by local deployment.These approaches differ in whether customization and infrastructure are controlled by the user.

6. Future Directions and Implications

Future LLM development is expected to combine continued scaling with multimodal capabilities, interdisciplinary collaboration, and stronger attention to ethics and public awareness. These directions span technical progress, research practice, and societal governance.

  • Technology trends: Future LLMs are expected to continue expanding in model scale while incorporating modalities such as images, video, and speech.The paper links larger scale with learning capabilities and performance, while identifying multimodal processing as a potential trajectory.
  • Research directions: AI development increasingly requires collaboration between researchers and professionals from diverse industries and disciplines.The paper emphasizes combining expertise to address cross-domain challenges.
  • Societal implications: Growing LLM use raises ethical and societal concerns involving model bias and misuse risks.The paper discusses standards and ethical frameworks involving decision-making, legal, and sociological expertise.
  • Societal implications: Public awareness training is proposed before large-scale deployment to improve understanding of LLM capabilities and limitations.The paper highlights education and journalism as especially relevant application areas.

7. Conclusion

The paper surveys LLM training and inference technologies in response to the growing importance of cost-effective development and deployment. It concludes that future progress depends on improved efficiency, engineering expertise, collaboration, and broader applications.

  • Scope and contribution: The review covers the evolution of LLM training techniques and inference deployment technologies toward low-cost development.Its scope includes training, inference, utilization, and future directions.
  • Motivation: ChatGPT accelerated LLM use for downstream tasks, while reliance on OpenAI infrastructure underscores the need for alternative and domain-specific models.The conclusion connects this need with advances in training and deployment processes.
  • Implementation challenges: LLM development requires expertise in large-scale data handling and distributed parallel training, reinforcing collaboration between researchers and engineers.The review presents this expertise as important for navigating training and inference complexity.
  • Future outlook: Future directions include improved model architectures, more efficient training, and broader applications across industries.The conclusion presents these as promising directions for continued LLM evolution.
Loading 2401.02038v2…