PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Large Language Models

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, Ji-Rong Wen

arXiv:2303.18223v19cs.CLcs.AI

TL;DR

Enabling machines to read, write, and communicate like humans remains a longstanding challenge. This survey reviews recent LLM advances across pre-training, adaptation, utilization, and evaluation. It synthesizes key techniques and findings central to LLM success, alongside development resources and implementation guidelines.

  • Problem

    Enabling machines to read, write, and communicate like humans remains a longstanding research challenge.

    Source: §1 · p. 1

  • Method

    The survey reviews recent LLM advances across pre-training, adaptation, utilization, and evaluation.

    Source: §10 · p. 96

  • Results

    The survey identifies techniques and findings central to LLM success and summarizes development resources and implementation guidelines for reproducing LLMs.

    Source: §10 · p. 96

  • Takeaways & Limitations

    The survey provides researchers and engineers with a reference for recent LLM literature.

    Source: §10 · p. 96

    LLM benchmark results are sensitive to question prompts, zero-shot versus few-shot settings, and answer parsing methods.

    Source: §7.3.2 · p. 65

Abstract

from arXiv · show

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

1 INTRODUCTION

This survey traces language models’ evolution toward general-purpose task solving and reviews LLMs through pre-training, adaptation, utilization, and capability evaluation. It highlights scaling-related abilities alongside unresolved questions about their origins, training accessibility, and alignment.

1 INTRODUCTION

  • Scaling pre-trained models can improve downstream performance and yield emergent abilities, but neither greater capability nor emergence is guaranteed.For example, GPT-3 can solve few-shot tasks through in-context learning, whereas GPT-2 performs poorly.Source: §1 · pp. 2–3
  • Language models evolved from statistical prediction through neural and contextual representation learning to scaled, general-purpose task solving.Statistical models use Markov assumptions and smoothing to address sparse high-order estimates; neural models learn distributed word representations useful beyond prediction. Pre-trained models learn context-aware representations from large unlabeled corpora and typically require downstream fine-tuning.Source: §1 · pp. 1–3
  • Beyond emergent abilities, LLMs differ from smaller pre-trained models through prompting-based access and development that closely combines research with engineering.Users must format tasks so LLMs can follow them, while developers need practical expertise in large-scale data processing and distributed parallel training.Source: §1 · p. 3
  • LLMs are changing NLP task solving, information retrieval, and multimodal dialogue while renewing discussion of artificial general intelligence.Applications include chatbot-based information seeking, office automation, and specialized functions through plugins. ChatGPT’s release was also followed by sharply increased LLM research attention.Source: §1 · p. 3
  • Unexplained abilities, costly and opaque training, and difficult alignment remain major obstacles to understanding and developing capable LLMs.Computational demands hinder repeated ablations, while industry developers often withhold training details. The origins and contributing factors of LLM abilities remain unclear, and models can produce toxic, fictitious, or harmful content.Source: §1 · p. 3
  • The survey organizes LLM development and use around four aspects: pre-training, adaptation, utilization, and capability evaluation.It synthesizes key findings, techniques, and methods, and provides a GitHub collection of supporting resources.Source: §1 · p. 4

2 OVERVIEW

This overview focuses on Transformer language models larger than 10B parameters, while acknowledging the absence of a universal size threshold, and introduces scaling laws, emergent abilities, and enabling techniques. It traces GPT development through scaling, code training, human alignment, dialogue, multimodal inputs, and tool use, while noting hallucinations and potentially risky responses as remaining limitations.

2.1 Background for LLMs

  • Scaling laws connect capacity to model size, data, and compute; KM prioritizes model growth, whereas Chinchilla recommends proportional model-and-data growth.The survey mainly covers Transformer models above 10B parameters without asserting a universal cutoff; the fitted laws assume other factors do not bottleneck the factor being analyzed. Data quality remains important, and Chinchilla outperforms the larger Gopher at the same compute budget by increasing training data.Source: §2.1 · pp. 4–6
  • Predictable scaling supports training forecasts, but lower language-modeling loss does not guarantee better downstream performance and can accompany inverse scaling.Small-model experiments can guide data mixtures and detect abnormal training; semantic quality may improve near irreducible loss [58], although limited text supplies motivate data-constrained scaling research [61]. Task-level predictability depends on task characteristics, while apparent emergence may partly reflect discontinuous evaluation metrics rather than abrupt capability changes [71].Source: §2.1 · pp. 5–6
  • Emergent abilities appear only at larger scales; GPT-3 demonstrated in-context learning from instructions or examples without parameter updates.The 175B GPT-3 exhibited strong general in-context learning, unlike GPT-1 and GPT-2, but success depended on the task: 13B supported some arithmetic tasks while 175B struggled with Persian QA. Critical emergence thresholds remain uncertain because they vary across models and tasks, and studies test relatively few model sizes.Source: §2.1 · p. 5
  • Instruction tuning improves generalization to unseen instructions, while chain-of-thought prompting elicits multistep reasoning through intermediate steps.Instruction-tuned LaMDA-PT significantly outperformed its untuned counterpart at 68B, but not at 8B or below; PaLM needed at least 62B for broad performance across four evaluated benchmarks. Chain-of-thought improved arithmetic reasoning in PaLM and LaMDA variants above 60B, with clearer advantages above 100B and task-dependent gains.Source: §2.1 · pp. 5–6
  • Training LLMs requires distributed parallelism and optimization techniques that support stability and performance.DeepSpeed and Megatron-LM facilitate parallel training, while restarting after loss spikes and mixed precision are practical optimization techniques. GPT-4 emphasizes infrastructure and optimization methods that reliably predict large-model performance using much smaller models.Source: §2.1 · p. 6
  • Alignment tuning incorporates human feedback, while external tools address numerical-computation and up-to-date-information limitations.InstructGPT uses reinforcement learning with human feedback to improve instruction following, and ChatGPT uses a similar approach to produce helpful, harmless responses. Calculators support accurate computation, search engines retrieve unknown information, and ChatGPT plugins broaden accessible capabilities.Source: §2.1 · pp. 6–7

2.2 Technical Evolution of GPT-series Models

  • GPT-1 established the series’ decoder-only Transformer architecture and next-word prediction foundation through unsupervised pre-training followed by supervised fine-tuning.Released in 2018, GPT-1 introduced the name Generative Pre-Training. The overview identifies accurate next-word prediction and increasing model scale as the two central principles underlying subsequent GPT development.Source: §2.2 · p. 7
  • 1.5B-parameter GPT-2 framed multitask solving as conditional text prediction, pursuing unsupervised task performance without task-specific supervised fine-tuning.Trained on WebText, GPT-2 represented inputs, outputs, and task descriptions uniformly as language. Despite this ambition, its overall performance remained inferior to supervised fine-tuning state-of-the-art methods.Source: §2.2 · p. 7
  • 175B parameters characterize GPT-3, which formalized in-context learning and demonstrated substantial capacity gains from scaling.In-context learning uses natural-language task descriptions and demonstrations for few-shot or zero-shot prediction. This unifies pre-training and utilization as context-conditioned text prediction.Source: §2.2 · p. 9
  • Code training improved GPT-model reasoning, with Codex fine-tuned on GitHub code and GPT-3.5 developed from code-davinci-002.Codex improved mathematical problem solving, while contrastive text–code embeddings improved classification and search. Code training’s proposed benefit for chain-of-thought prompting remains speculative.Source: §2.2 · pp. 9–10
  • Human alignment culminated in InstructGPT’s three-stage RLHF procedure, improving instruction following and mitigating harmful outputs as GPT-3 evolved into GPT-3.5.The procedure begins with supervised fine-tuning on human demonstrations. PPO [128] became the foundational reinforcement-learning algorithm for learning from human preferences.Source: §2.2 · p. 10
  • ChatGPT adapted InstructGPT-style training for dialogue, demonstrating mathematical reasoning, multi-turn context tracking, and alignment with human values.Training combined human-generated user–AI conversations with dialogue-formatted InstructGPT data. Plugins subsequently extended its capabilities through existing tools and applications.Source: §2.2 · p. 10
  • GPT-4 extended inputs to multimodal signals, improved complex-task performance over GPT-3.5, and responded more safely to malicious queries.Six-month iterative alignment incorporated an additional safety reward, while red teaming targeted harmful outputs. Predictable scaling estimated final performance using a small proportion of training compute.Source: §2.2 · p. 10
  • 128k tokens of context accompany GPT-4 Turbo’s improved capability over GPT-4 and expanded functionality, following GPT-4V’s vision-focused release.GPT-4V emphasized vision-input risk assessment and mitigation; Turbo updated knowledge through April 2023, lowered pricing, and added function calls and reproducible outputs. The Assistants API, vision, image generation, speech synthesis, and audio features expanded tool-enabled and multimodal applications.Source: §2.2 · pp. 10–11
  • Hallucinations, factual errors, and context-dependent risky responses persist in advanced LLMs, motivating safer models and iterative deployment.OpenAI’s iterative strategy follows a five-stage development and deployment life cycle intended to reduce potential usage risks.Source: §2.2 · p. 11

3 RESOURCES OF LLMS

This section surveys reusable resources for LLM development, including public checkpoints, APIs, datasets, and libraries. These resources support pre-training, instruction and alignment tuning, distributed training, hardware-accelerated computation, inference and serving, and RLHF workflows.

3.1 Publicly Available Model Checkpoints or APIs

  • Public checkpoints reduce pre-training costs, while APIs enable inference without running models locally; model selection depends on resource budgets and application needs.The survey compares parameter sizes, training data and computation, employed technologies, and downstream evaluations.Source: §3.1 · p. 11
  • 405B-parameter LLaMA-3.1 achieves competitive performance against GPT-4, GPT-4o, and Claude 3.5 Sonnet on benchmarks including MMLU, GSM8k, and HumanEval.LLaMA-3.1 uses 15T pre-training tokens and a 128K context window; training its 405B version involves more than 16,000 H100 GPUs.Source: §3.1 · p. 11
  • LLaMA supports relatively low-cost customization through instruction tuning, while its derivatives also serve as language backbones for multimodal models.Non-English adaptation often requires vocabulary expansion or target-language tuning. Alpaca uses synthetic demonstrations, Vicuna uses ShareGPT conversations, and Vicuna is particularly favored in multimodal models.Source: §3.1 · p. 12

3.2 Commonly Used Corpora for Pre-training

  • Large-scale pre-training requires diverse corpora; CommonCrawl supplies web data requiring cleaning, while RedPajama-Data provides filtered multilingual data and quality labels.RedPajama-Data contains approximately 30T tokens across five languages and offers over 40 quality labels for filtering or reweighting.Source: §3.2 · p. 13
  • Books, Wikipedia, licensed code, and mixed datasets complement web corpora with literary, encyclopedic, programming, and multisource content.Project Gutenberg provides public-domain books, whereas Books1 and Books2 remain unreleased. Mixed resources include The Pile, ROOTS, and Dolma [177].Source: §3.2 · pp. 13–14

3.3.1 Instruction Tuning Datasets

  • Instruction-tuning datasets support abilities such as instruction following through three construction types: formatted NLP tasks, daily conversations, and synthetic examples.NLP collections add natural-language task descriptions, chat collections capture user conversations, and synthetic collections use LLM-guided generation, including self-instruct.Source: §3.3.1 · pp. 14–15

3.3.2 Alignment Datasets

  • Alignment datasets encode response preferences through human judgments, separate helpfulness and harmlessness annotations, or feedback generated by other LLMs.SHP contains naturally occurring human-written responses; PKU-SafeRLHF [195] supplies safety labels and separate preference annotations; Sandbox Alignment Data uses simulated interactions and model feedback.Source: §3.3.2 · p. 15

3.4 Library Resource

  • Transformers simplifies pretrained-model use and customization, while DeepSpeed supplies distributed-training optimizations.DeepSpeed supports ZeRO memory optimization, gradient checkpointing, and pipeline parallelism.Source: §3.4 · p. 16
  • Megatron-LM and Colossal-AI support large-scale model training through distributed-training optimizations and parallel training strategies.Megatron-LM provides model and data parallelism, mixed-precision training, and FlashAttention to improve training efficiency. PyTorch-based Colossal-AI also optimizes heterogeneous memory management using methods from PatrickStar.Source: §3.4 · p. 16
  • JAX supports hardware-accelerated array computation for machine learning, with automatic differentiation and just-in-time compilation.Google’s Python library enables computation across devices such as GPUs and TPUs.Source: §3.4 · p. 16
  • BMTrain emphasizes simple, low-resource distributed training, while FastMoE specializes in efficient, user-friendly mixture-of-experts training.BMTrain offers directly usable models through ModelCenter, including Flan-T5 and GLM. PyTorch-based FastMoE simplifies Transformer-to-MoE conversion and supports both data and model parallelism.Source: §3.4 · p. 16
  • vLLM enables fast, memory-efficient LLM inference and serving through PagedAttention, continuous batching, and optimized CUDA kernels.It supports multiple decoding algorithms, tensor parallelism, and streaming outputs. HuggingFace model compatibility and OpenAI-compatible API servers facilitate integration.Source: §3.4 · p. 16
  • Over 13,000 models across three architectures are supported by DeepSpeed-MII, which targets high-throughput, low-latency, cost-effective LLM inference.Its acceleration mechanisms are blocked KV caching, continuous batching, dynamic SplitFuse, and high-performance CUDA kernels.Source: §3.4 · p. 16
  • DeepSpeed-Chat [210] provides a complete RLHF pipeline and a unified hybrid engine for switching between training and inference.It replicates the training mode of InstructGPT, covering SFT, reward model fine-tuning, and RLHF. Simple scripts support multiple training or inference steps, while the hybrid engine incorporates DeepSpeed Inference optimizations.Source: §3.4 · p. 16

4 PRE-TRAINING

Pre-training on large-scale corpora establishes LLMs’ language understanding and generation abilities, which depend critically on corpus scale and quality. This section reviews data preparation and scheduling, model architectures and training objectives, decoding strategies, and techniques for stable, efficient, and scalable training.

4.1.1 Data Source

  • General corpora include webpages for broad linguistic knowledge, conversations for dialogue skills, and books for long-range dependencies and coherent narrative generation.Web data requires filtering because high-quality material coexists with spam. Reply-linked conversation trees can yield training sub-conversations, but excessive dialogue data may cause instructions to be misinterpreted as conversation openings.Source: §4.1.1 · p. 17
  • Specialized multilingual, scientific, and code corpora strengthen multilingual performance, scientific reasoning, and program synthesis.BLOOM, PaLM, and FLM [102] achieve comparable or superior multilingual-task performance to models fine-tuned on target-language corpora. Scientific formats require specialized preprocessing; code-trained LLMs can pass expert-designed unit tests or solve competitive programming questions, while code-induced reasoning remains speculative.Source: §4.1.1 · pp. 17–18

4.1.2 Data Preprocessing

  • Data selection uses quality classifiers, heuristic rules, or LLM-based scoring, with trade-offs involving corpus diversity and computational cost.Quality classifiers may unintentionally discard dialectal, colloquial, and sociolectal texts, introducing bias and reducing diversity. LLM-based selection uses perplexity [232] or direct prompting [233], but becomes computationally intensive at large scale.Source: §4.1.2 · p. 18
  • Subword tokenization commonly uses BPE, WordPiece, or Unigram, while corpus-specific tokenizers can better accommodate diverse languages, domains, and formats.BPE merges frequent adjacent tokens, WordPiece selects merges maximizing training-data likelihood, and Unigram prunes a large vocabulary using a unigram model. Normalization can degrade tokenization, and an English-focused vocabulary can increase Chinese generation latency when extending a model.Source: §4.1.2 · pp. 19–20
  • Cleaning pre-training corpora improves model performance, while noisy, toxic, and duplicate data harms capabilities.Sentence-, document-, and dataset-level deduplication should be combined, including removing training–evaluation overlap to prevent contamination. Removing personally identifiable information and duplicate personal data also reduces privacy risks.Source: §4.1.2 · pp. 18–20

4.1.3 Data Scheduling

  • Diverse data sources improve downstream performance [227] [248] [249], whereas excessive concentration on one domain can degrade generalization elsewhere.Source-removal experiments show that dropping highly heterogeneous sources, such as webpages, damages capabilities more than dropping less heterogeneous academic corpora [227].Source: §4.1.3 · p. 20
  • Data-mixture optimization can use downstream-task relevance or small-model experiments, but transferring small-model preferences to larger models relies on an uncertain assumption.DoReMi [59] trains a small reference model and a proxy model, upweights domains with greater likelihood discrepancies, and transfers the learned domain weights to a larger LLM.Source: §4.1.3 · p. 20
  • Data curricula order training examples or adapt source proportions, sometimes making basic-to-target skill sequences more effective than target-only training [253].Increasing mathematical, code, or book data can strengthen corresponding skills; general-to-specialized stages provide a practical curriculum [227]. Long-context continual training can combine longer training sequences with position-embedding modifications or external memory, but general-purpose curriculum reporting remains limited.Source: §4.1.3 · pp. 20–21

4.1.4 Summary of Data Preparation

  • Data preparation combines diverse, skill-aligned collection, deduplication, quality and privacy filtering, tokenization, and deliberate mixture scheduling.Cleaning removes low-quality, toxic, and privacy-sensitive content before tokenizer training on the filtered, deduplicated corpus. Candidate mixtures and ordering plans can be tested with small models [59], then adjusted during pre-training using checkpoint evaluations.Source: §4.1.4 · p. 21

4.2.1 Typical Architectures

  • Transformer LLMs use encoder-decoder, causal-decoder, or prefix-decoder architectures, distinguished by their attention patterns; causal decoders are widely adopted.Encoder-decoder models encode inputs and generate outputs through cross-attention, whereas causal decoders restrict attention to preceding tokens and themselves. Prefix decoders share encoding and decoding parameters, combining bidirectional prefix attention with causal generation; conversion from causal decoders can accelerate convergence.Source: §4.2.1 · p. 22
  • Mixture-of-experts sparsely activates model weights to expand parameter capacity at constant computational cost, but routing can destabilize training.Increasing expert counts or total parameters can improve performance. Higher-precision routing tensors and smaller initialization ranges can improve training stability.Source: §4.2.1 · p. 22
  • State-space architectures target Transformers’ quadratic sequence costs through recurrent decoding and parallel sequence encoding, but their performance still trails Transformers.Mamba makes state-update parameters input-dependent to selectively retain or filter information. Hyena replaces attention with gated long convolutions, but unlike recurrent state-space models, it must explicitly access all previous states during inference.Source: §4.2.1 · pp. 22–23

4.2.2 Detailed Configuration

  • RoPE rotates queries and keys to encode relative positions, while ALiBi applies fixed distance penalties to improve length extrapolation.ALiBi outperformed sinusoidal PE, RoPE, and T5 bias on longer-than-training sequences, and improved training stability in BLOOM. Extending RoPE, xPos adds exponential decay to improve translation invariance and length extrapolation.Source: §4.2.2 · p. 25
  • Attention optimizations address long-sequence computation, inference overhead, and KV-cache memory waste through sparsity, shared projections, IO-aware kernels, and paging.Sparse attention restricts attended tokens; multi-query attention [300] shares key/value projections across heads with minor quality loss, while grouped-query attention [301] shares them within groups. FlashAttention uses SRAM-aware blocking and recomputation, whereas PagedAttention allocates KV caches in non-contiguous blocks to reduce fragmentation and enable memory sharing.Source: §4.2.2 · p. 25
  • Pre-RMSNorm and SwiGLU or GeGLU are recommended for generalization and training stability, although normalization placement and activation choices introduce trade-offs.RMSNorm uses RMS-only rescaling; pre-LN generally improves stability at a performance cost but can fail in very large GLM models. Sandwich-LN can also permit training collapse, while gated activations often outperform GeLU but require additional feed-forward parameters.Source: §4.2.2 · pp. 24, 26

4.2.3 Pre-training Tasks

  • Language modeling predicts subsequent tokens, while denoising autoencoding reconstructs corrupted spans; decoder-only LLMs predominantly use language modeling.Prefix language modeling excludes prefix tokens from the loss and performs slightly worse with equal training-token exposure. Denoising autoencoding is less widely used because of implementation complexity; Mixture-of-Denoisers, or UL2 loss, introduces a unified pre-training objective.Source: §4.2.3 · p. 26
  • Mixture-of-denoisers combines language modeling with short-span, low-corruption and long-span or high-corruption denoising objectives, selected through special input tokens.The S-denoiser resembles conventional language modeling, while R- and X-denoisers differ in corrupted span lengths and corruption ratios. MoD has been applied in PaLM 2 [120].Source: §4.2.3 · p. 26

4.2.4 Decoding Strategy

  • Greedy search suits input-dependent generation but can repeat in open-ended tasks; beam search and penalties address decoding weaknesses.Beam search retains high-probability candidate sequences, although larger beams can worsen performance; length normalization counters its preference for short outputs. Repetition penalties discourage previously generated tokens or n-grams, while diverse beam search produces varied outputs.Source: §4.2.4 · pp. 26–27
  • Sampling increases generation diversity, while temperature adjustment, top-k, and top-p [306] sampling control the selection of low-probability tokens.Lower temperatures favor high-probability tokens and approach greedy search as temperature approaches zero. Top-k restricts sampling to a fixed number of highest-probability tokens, whereas top-p [306] selects the smallest set reaching a cumulative probability threshold, accommodating different contextual distributions.Source: §4.2.4 · pp. 26–27
  • Contrastive decoding emphasizes important tokens through large–small model likelihood differences, while DoLa contrasts logits across layers of one model.η-sampling introduces a distribution-dependent threshold to improve top-p sampling. Contrastive search and typical sampling target generation coherence.Source: §4.2.4 · p. 27

4.2.5 Summary and Discussion

  • Causal decoders show strong zero-shot and few-shot generalization and scaling gains, but their architectural advantage lacks theoretical justification.Instruction and alignment tuning further enhance their capabilities. More investigation is needed into encoder-decoder scaling, architecture–objective interactions, architectural diversity, and detailed model configurations.Source: §4.2.5 · p. 28

4.3.1 Optimization Setting

  • LLM optimization combines large or growing batches, learning-rate warm-up and decay, adaptive optimizers, and safeguards against training instability.GPT-3 and PaLM increase batch sizes during training; linear learning-rate warm-up precedes cosine decay, with Adam, AdamW, or memory-saving Adafactor used for optimization. Weight decay and gradient clipping combat instability; PaLM and OPT restart before loss spikes and skip suspect data, while GLM shrinks embedding gradients.Source: §4.3.1 · pp. 28–29

4.3.2 Scalable Training Techniques

  • 3D parallelism combines data, pipeline, and tensor parallelism to increase training throughput and distribute large models across GPUs.Data parallelism replicates models over data shards, pipeline parallelism partitions layers, and tensor parallelism partitions parameter matrices; GPipe and PipeDream mitigate pipeline inefficiencies. ZeRO, FSDP, and activation recomputation reduce memory redundancy, while GPT-4’s predictable scaling forecasts large-model performance using smaller models.Source: §4.3.2 · pp. 29–30
  • Mixed-precision training reduces memory and communication costs, while BF16 generally offers better pre-training representation accuracy than FP16.FP16 can impair computational accuracy and final performance; BF16 allocates more exponent bits and fewer significant bits. Using BF16 to improve efficiency and memory usage requires suitable hardware support, such as A100 GPUs.Source: §4.3.2 · pp. 29–30

5 POST-TRAINING OF LLMS

This section reviews post-training methods for enhancing LLM capabilities and aligning their behavior with human values and preferences through instruction tuning and alignment tuning. It covers instruction-data construction and evaluation, supervised and feedback-based tuning strategies, and parameter-efficient adaptation.

5.1 Instruction Tuning

  • Instruction tuning fine-tunes pretrained LLMs on natural-language-formatted instances, using supervised learning to improve generalization to unseen tasks.The survey emphasizes instance-collection and tuning guidelines for meeting users’ real needs, as in InstructGPT and GPT-4.Source: §5.1 · p. 30

5.1.1 Formatted Instance Construction

  • Instruction instances pair task descriptions with outputs and optional inputs and demonstrations, drawing on NLP datasets, daily chat, or synthetic generation.InstructGPT pairs real-user queries with human-written answers, while GPT-4 includes supervised refusal training for high-risk instructions. Synthetic generation expands a filtered task pool; WizardLM increases complexity and diversity, while Self-Align [336] filters using human-aligned principles.Source: §5.1.1 · pp. 30–31
  • Instruction diversity and quality matter alongside dataset scale, whose generalization gains eventually saturate.Small high-quality datasets can be effective [338], while prompt-based refinement improves instruction complexity and diversity. Selection uses perplexity and heuristics, LLM judges, diversity-aware sampling, or target-task gradient similarity through LESS.Source: §5.1.1 · pp. 31–32
  • Task descriptions are crucial for generalization; demonstrations improve performance, while mixed chain-of-thought and non-chain-of-thought training supports varied reasoning requirements.Demonstrations also reduce sensitivity to instruction engineering. Additional instruction components, such as reasons or suggestions, can have negligible or adverse effects.Source: §5.1.1 · pp. 30, 32

5.1.2 Instruction Tuning Strategies

  • Tuning balances task mixtures, can incorporate pretraining data as regularization, and can progress from task-formatted instructions to chat.Dataset caps prevent large sources from dominating, high-quality collections can be upweighted, and retaining task instructions during later chat tuning helps avoid forgetting. Vicuna reduces redundant computation by processing whole conversations and computing loss only on chatbot responses.Source: §5.1.2 · pp. 32–33

5.1.3 The Effect of Instruction Tuning

  • More than 50% improvement in multilingual sentence completion over BLOOM demonstrates BLOOMZ-P3’s transfer from English-only instruction tuning.Instruction tuning improves performance across scales and architectures; smaller tuned models can outperform larger untuned models. It also supports unseen tasks, reduces repetitive or incomplete task execution, and costs less than pre-training.Source: §5.1.3 · pp. 33–34
  • Instruction tuning adapts general LLMs into domain specialists, including medicine, recommendation, law, finance, and arithmetic.Med-PaLM [354], developed by tuning Flan-PaLM on medical datasets, achieves performance comparable to expert clinicians. Instruction-tuned FLAN-T5 also shows strong performance across recommendation tasks.Source: §5.1.3 · p. 34

5.1.4 Empirical Analysis for Instruction Tuning

  • 7B and 13B LLaMA experiments compare instruction datasets and synthetic-data improvements in complexity, diversity, difficulty, and scale.Synthetic instructions can lack topic diversity or have uneven difficulty; the study subsamples FLAN-T5 to 80,000 instructions for comparison with ShareGPT and Self-Instruct-52K. Diversity enhancement follows YuLan-Chat to cover 293 topics, while computational limits exclude larger LLaMA variants.Source: §5.1.4 · p. 34, Table 10 · p. 35
  • 43.69 versus 38.58 MMLU accuracy favors LLaMA (7B) tuned on mixed FLAN-T5, ShareGPT, and Self-Instruct-52K instructions over FLAN-T5 alone.The mixture performs well in both chat and QA; FLAN-T5 generally favors QA, whereas ShareGPT better supports daily user queries. Chat evaluation uses ChatGPT-judged win rates against Self-Instruct-52K baselines, while QA evaluation uses MMLU and BBH accuracy.Source: §5.1.4 · pp. 35–36
  • Greater instruction complexity and diversity improve chat and QA performance, while LLaMA (13B) generally follows instructions better than LLaMA (7B).Increasing instruction complexity produces larger improvements on QA tasks, whose difficult questions benefit from learning complex instructions during fine-tuning.Source: §5.1.4 · p. 36
  • 29.81 to 26.63 is the BBH performance decrease for LLaMA (7B) when synthesized instruction counts increase without quality control.Keeping moderate-difficulty instructions performs well in chat but slightly reduces QA performance. Filtering out complex, high-perplexity instructions may explain the QA decline.Source: §5.1.4 · p. 36

5.2.1 Background and Criteria for Alignment

  • Alignment targets helpfulness, honesty, and harmlessness, but can reduce general capabilities through an alignment tax.Helpfulness assists users, honesty includes calibrated uncertainty, and harmlessness avoids discriminatory outputs and refuses dangerous requests. These criteria are difficult to formalize; red teaming adversarially probes harmful outputs before updating models to prevent them.Source: §5.2.1 · pp. 36–37

5.2.2 Collecting Human Feedback

  • Human feedback collection combines labeler screening with preference rankings, targeted questions, and rule-violation assessments.InstructGPT screens labelers for agreement with researchers; preference rankings, document-usefulness questions in WebGPT, and Sparrow’s rule checks provide complementary signals. GPT-4 also uses its own zero-shot classifiers as rule-based reward models.Source: §5.2.2 · pp. 37–38

5.2.3 Reinforcement Learning from Human Feedback

  • RLHF combines optional supervised fine-tuning, human-preference reward modeling, and policy optimization, commonly PPO [128], with penalties limiting departure from the initial model.The language model acts as the policy, generated tokens are actions, and the reward model supplies rewards; reward-model training and policy optimization can be iterated. InstructGPT uses a KL-divergence penalty against its initial model.Source: §5.2.3 · p. 38
  • Alternatives use AI preference feedback, replace unstable RL with supervised updates, or supervise individual generation steps instead of whole outputs.RLAIF uses preset alignment principles to reduce human annotation and address harmless-but-unhelpful evasion; supervised alternatives reuse highly rewarded samples. Process supervision supplies finer-grained corrections, although obtaining these signals is harder than obtaining outcome rewards.Source: §5.2.3 · pp. 38–39
  • Practical RLHF strategies strengthen reward models, stabilize policy training through best-of-N warm starts and iterative refinement, and reduce inference overhead.Larger reward models can judge outputs better; LLaMA 2 [99] uses pretrained chat checkpoints and criterion-specific rewards, while preferred-response language-model losses regularize reward learning [372] [373]. Best-of-N fine-tuning precedes RL, retaining earlier samples limits iterative regression [99], and separate reward servers plus beam-search candidate generation improve efficiency.Source: §5.2.3 · p. 39
  • Process reward models provide step-level RLHF feedback and support search-guided policy improvement, answer reranking, and reasoning-step selection.Annotations can come from humans, powerful LLMs, or Monte Carlo tree search; expert iteration alternates PRM-guided search with supervised distillation. RLMEC supplies token-level rewards through constrained rewriting, while outcome supervision can support process-supervised RLHF when fine-grained labels are difficult to collect.Source: §5.2.3 · p. 40

5.2.4 Alignment without RLHF

  • Non-RL alignment replaces reinforcement learning with supervised tuning on reward-selected, principle-guided, or interaction-generated alignment data.RAFT selects high-reward responses, Self-Align [336] generates principle-guided responses, and Stable Alignment collects feedback and revisions from interacting agents. Reward-model training requires costly human labels, and reused reward models may miss another LLM’s misaligned behaviors.Source: §5.2.4 · pp. 40–41
  • DPO reparameterizes rewards through the policy model, eliminating explicit reward modeling while optimizing an objective equivalent to reward optimization.DPO can struggle to balance positive and negative examples, and a weak reference model can impair alignment. Other supervised approaches supplement cross-entropy with response-ranking losses, preference calibration, or instruction-response contrastive learning.Source: §5.2.4 · p. 41

5.2.5 Remarks on SFT and RLHF

  • SFT clones demonstration tokens, whereas RLHF optimizes response-level preferences; demonstrations beyond model capabilities can promote hallucination.SFT mainly unlocks existing abilities rather than injecting new ones, and demonstration quality matters more than quantity [99]. These comparisons partly reflect the authors’ opinions and experience.Source: §5.2.5 · p. 42
  • RLHF improves helpfulness and harmlessness in LLaMA 2 [99] and reduces hallucination in GPT-4, but introduces substantial training difficulties.Preference judgments reduce annotator discrepancies and permit exploration beyond human demonstrations, but RLHF suffers from sample inefficiency, instability, dependence on strong SFT initialization, and sensitivity to annotation and optimization choices. The survey positions SFT immediately after pretraining and RLHF afterward, while calling for more reliable annotation and simpler optimization.Source: §5.2.5 · pp. 42–43

5.3.1 Parameter-Efficient Fine-Tuning Methods

  • Parameter-efficient adaptation trains inserted adapters, layerwise prefixes, input prompts, or low-rank weight updates instead of all model parameters.Adapters use bottleneck modules; prefix tuning can stabilize training through an auxiliary mapping network, while input-prompt performance depends strongly on underlying model capacity. LoRA freezes original weights and stores task-specific low-rank updates, reducing memory and storage requirements.Source: §5.3.1 · p. 43, §5.3 · p. 43

5.3.2 Parameter-Efficient Fine-Tuning on LLMs

  • Efficient-tuning methods matched GPT-3.5 on simple mathematical tasks but underperformed on difficult ones; LoRA performed relatively well with significantly fewer trainable parameters.The study evaluated GPT-J (6B), BLOOM (7.1B), and LLaMA (7B) on six mathematical reasoning datasets. LoRA has been applied to open-source LLMs, but thorough comparisons across large-model settings and tasks remain lacking.Source: §5.3.2 · p. 44

6 UTILIZATION

This section surveys LLM utilization through prompt creation and optimization, demonstration-based in-context learning, chain-of-thought prompting, and planning for complex tasks. It covers demonstration design, intermediate reasoning, and plan refinement using environmental feedback, backtracking, and long-term memory.

6.1.1 Prompt Creation

  • Prompt creation combines task description, input data, contextual information, and prompt style to elicit task-relevant abilities.Recommended principles include clear objectives and response constraints, decomposition into easier subtasks, and high-quality few-shot demonstrations. Model-friendly formatting and role-playing provide additional guidance.Source: §6.1.1 · pp. 44–46
  • 53.20 to 66.75 on Colored Objects demonstrates ChatGPT’s improvement with carefully designed prompts that clarify complex task requirements.Detailed descriptions clarify task goals and output formats. In another comparison, a more detailed WikiFact task description increases performance from 29.25 to 31.21.Source: §6.1.1 · p. 46
  • 78.47 to 79.30 on GSM8k results from code-formatted demonstrations that turn mathematical reasoning into code generation supported by an external program executor.This approach leverages ChatGPT’s code synthesis ability while delegating arithmetic to the executor for more precise results.Source: §6.1.1 · p. 46
  • 31.21 (ChatGPT) versus 34.20 (supervised baseline) on WikiFact exemplifies remaining gaps, despite comparable or superior prompted performance on some knowledge and reasoning tasks.Task-specific supervised models still outperform ChatGPT on tasks including ARC and WikiFact.Source: §6.1.1 · p. 46
  • Suitable prompts enable general and conversational recommendation, but ChatGPT remains far below reference performance because these tasks require domain knowledge and adaptation.These non-traditional NLP tasks become accessible when their requirements can be expressed in natural language.Source: §6.1.1 · p. 46

6.1.2 Prompt Optimization

  • Prompt optimization searches discrete text through gradients, reinforcement learning, editing, or LLM generation, while continuous prompts use gradient-optimized embeddings and transfer learning.LLM-based optimization incorporates textual feedback, prompt-score histories, and strategic search using Monte Carlo tree search. Continuous prompting has received limited LLM attention; SPoT transfers source-task prompts to data-scarce targets.Source: §6.1.2 · pp. 46–49

6.2.1 ICL Formulation

  • In-context learning performs new tasks without gradient updates using natural-language task descriptions and/or demonstrations, with the test input appended for prediction.Demonstration construction involves example selection, formatting, and ordering. Unlike instruction tuning, ICL requires no fine-tuning, although instruction tuning can improve ICL, particularly in zero-shot settings.Source: §6.2.1 · pp. 49–50

6.2.2 Demonstration Design

  • Demonstration design strongly affects ICL performance; heuristic selection balances query relevance and example-set diversity rather than selecting each example independently.k-NN retrieval selects semantically relevant examples, diversity-based methods seek representative sets [483, 484], and another approach combines relevance with diversity. Formatting and ordering are additional design dimensions.Source: §6.2.2 · p. 50
  • Effective ICL demonstration design selects task-informative, query-relevant examples, formats them as prompts, and orders them to address recency bias.Templates can incorporate task descriptions or reasoning; Auto-CoT generates reasoning steps, while least-to-most prompting decomposes problems and solves subproblems sequentially. Selection can use LLM-estimated gains, EPR’s retrieval, reinforcement-learning feedback, or generated examples; ordering can use similarity or entropy with model-sampled validation data.Source: §6.2.2 · pp. 50–51

6.2.3 Underlying Mechanism

  • 350M-parameter models can recognize tasks, whereas demonstration-driven task learning emerged only at 66B parameters or above in the reported experiments.Recognition uses pretrained knowledge, whereas task learning is interpreted as attention-mediated implicit gradient descent or algorithm learning during forward computation. Training-task design matters, theories connect ICL to corpus coherence and compositional structure, and related-task prompts or Symbol Tuning target task learning.Source: §6.2.3 · pp. 51–52

6.3.1 Basic CoT Prompting Approach

  • CoT prompting augments input-output demonstrations with intermediate reasoning steps, enabling LLMs to generate reasoning chains before answering new inputs.Simple instructions can elicit reasoning chains without human-annotated demonstrations. Chains usually use natural language, but code offers more structured reasoning, and dynamic format selection combines text and code.Source: §6.3.1 · p. 52

6.3.2 Improved CoT Prompting Strategies

  • Improved CoT strategies strengthen prompts, sample reasoning paths, verify correctness, and introduce tree- or graph-structured search to address unreliable reasoning.Diverse CoTs and complex examples improve prompts, self-consistency majority-votes over sampled answers, and verification checks reasoning steps to mitigate accumulating errors. Tree of Thoughts supports lookahead and backtracking; Graph of Thoughts reuses thoughts across paths but requires many LLM interactions, motivating XoT’s search-guidance networks [521].Source: §6.3.2 · pp. 52–53

6.3.3 Further Discussion on CoT Prompting

  • 10B or more parameters typically characterize models benefiting from CoT prompting, which mainly helps stepwise reasoning but can hurt tasks without complex reasoning.Code-training explanations lack public ablation evidence; non-CoT instruction tuning does not improve held-out CoT benchmarks, and substantial gains may require poor standard-prompting performance. Patterns and text are essential and mutually supportive, although symbol/pattern correctness appears less critical; preliminary extensions address multimodal and multilingual tasks.Source: §6.3.3 · pp. 53–54

6.4.1 The Overall Framework

  • Prompt-based planning separates an LLM task planner, a plan executor, and an environment, using execution feedback to iteratively refine plans for complex tasks.Plans can be natural-language action sequences [432] or executable programs, with memory supporting long-horizon planning. Executors can be LLMs or code interpreters, while environments return textual or multimodal feedback.Source: §6.4.1 · p. 54, §6.4 · p. 54

6.4.2 Plan Generation

  • Text-based planning generates natural-language action sequences, whereas code-based planning produces executable programs for deterministic solvers to address unfaithful execution.Plan-and-Solve uses zero-shot instructions, while Self-planning and DECOMP use demonstrations; ToolFormer learns API calls, and HuggingGPT [437] coordinates models. Faithful CoT and PAL separate plan generation from execution, while PROGPROMPT and LLM+P extend code-based plans to embodied settings.Source: §6.4.2 · pp. 54–55

6.4.3 Feedback Acquisition

  • Planning feedback comes internally from LLM evaluations or externally from tools, models, virtual environments, and interacting agents.RAP scores success likelihood, Tree of Thoughts compares plans, and Reflexion turns sparse outcomes into textual feedback stored in long-term memory. External feedback includes interpreter errors, visual perception, virtual-world interaction [530], and agent communication in Generative Agents.Source: §6.4.3 · p. 55

6.4.4 Plan Refinement

  • Explicit reasoning extracts critical information from environmental feedback, while ChatCoT enables flexible switching between reasoning and planning through multi-turn, tool-augmented conversations.Feedback can contain irrelevant information or take non-language forms; React uses demonstrations to generate reasoning traces over it. Earlier approaches typically fix the order of reasoning and planning.Source: §6.4.4 · p. 55
  • Tree of Thoughts uses backtracking with breadth-first or depth-first search for global planning, addressing locally optimal plans produced by forward-only approaches.It revisits earlier states and chooses unexplored actions. Entire-plan revision approaches include DEPS, which selects better plans using feedback, and TIP, which incorporates feedback into prompts to revise each step.Source: §6.4.4 · p. 55
  • Long-term memory supports plan refinement for long-horizon tasks by retaining feedback and successful plans beyond the short-term memory provided through in-context learning.Reflexion retrieves stored self-reflection feedback; Generative Agents uses memory streams for planning and reflection; skill libraries [438] [530] reuse and combine successful plans for novel tasks. Vector databases enable large-scale storage and retrieval, while MemoryBank introduces forgetting and strengthening following the Ebbinghaus Forgetting Curve.Source: §6.4.4 · p. 55

7 CAPACITY AND EVALUATION

This section reviews LLM capabilities and evaluation across language generation, knowledge utilization, numerical and complex reasoning, human alignment, environmental interaction, and tool manipulation. It examines benchmark-, human-, and model-based evaluation approaches and empirical comparisons of open- and closed-source models, highlighting evaluation limitations, capability gaps, and the influence of instruction tuning and model size.

7.1.1 Language Generation

  • Language generation evaluation spans language modeling, conditional text generation, and code synthesis, with substantial reported capabilities across these tasks.Language modeling commonly uses zero-shot perplexity and improves with scaling; conditional generation uses automatic metrics and human ratings, while code synthesis evaluates functional correctness through pass@k. Code training [86], sampling multiple solutions, and planning-guided decoding improve code synthesis.Source: §7.1.1 · p. 56
  • Generation evaluation remains unreliable: reference-based metrics can contradict human judgments, annotators disagree, and LLM evaluators introduce their own biases.On OpenDialKG, ChatGPT trails fine-tuned GPT-2 on BLEU and ROUGE-L but receives better human judgments. Para-Ref expands reference paraphrases, while reference-free evaluators assess individual predictions or compare candidates, but can exhibit order bias and favor LLM-generated text.Source: §7.1.1 · p. 57
  • Specialized generation remains difficult because injecting domain knowledge can compromise existing abilities, while complex structured-data tasks often favor task-specific models.Specialization can cause catastrophic forgetting, and human alignment can impose an “alignment tax”, including potential loss of in-context learning ability. Sequence-modeling limitations also hinder structured-data understanding and generation, including knowledge-base question answering and semantic parsing.Source: §7.1.1 · p. 58

7.1.2 Knowledge Utilization

  • Knowledge utilization evaluates pretrained factual knowledge, externally retrieved evidence, and knowledge completion through closed-book QA, open-book QA, and completion tasks.Closed-book QA benefits from model and data scaling but struggles with fine-grained knowledge; open-book retrieval improves answer accuracy. Knowledge completion favors frequent over rare relations, and InstructGPT outperforms GPT-3 across all WikiFact subsets under the same evaluation settings.Source: §7.1.2 · pp. 58–59
  • Hallucination occurs when generated information contradicts its source or cannot be verified from it, and even strong LLMs struggle to recognize hallucinated content.Mitigations include alignment tuning, credible external tools, and uncertainty estimation; SelfCheckGPT detects inconsistencies across sampled outputs. TruthfulQA tests mimicked human falsehoods, while HaluEval evaluates hallucination recognition in task-specific and general scenarios.Source: §7.1.2 · p. 59
  • Knowledge recency remains unresolved: updating parametric knowledge is costly and risks catastrophic forgetting, while external retrieval offers a practical but limited workaround.Retrieved information can supply new facts through context, but directly updating intrinsic knowledge remains difficult, with several editing methods performing poorly on LLMs. EasyEdit provides a framework for researching knowledge editing.Source: §7.1.2 · pp. 59–60

7.1.3 Complex Reasoning

  • Complex reasoning covers knowledge, symbolic, and mathematical tasks, supported by chain-of-thought prompting, executable code, mathematical pretraining, and theorem-prover assistance.Knowledge reasoning still trails human commonsense performance, while symbolic reasoning struggles with longer operation sequences, motivating scratchpads, tutoring, and executable representations. Mathematical evaluation spans numerical problem solving and theorem-proving success rates; mathematical pretraining improves reasoning, while informal-to-formal conversion and proof sketches address limited formal corpora and proof-search difficulty.Source: §7.1.3 · pp. 60–61
  • LLMs can produce correct answers through invalid reasoning or wrong answers through correct reasoning, and small task-description changes can alter results.Mitigations include process-based supervision, executable code, and ensembles of reasoning paths through self-consistency. Tree of Thoughts (ToT) explores and evaluates alternative paths, while Self-Refine iteratively improves solutions using self-generated feedback.Source: §7.1.3 · p. 61
  • LLMs struggle with numerical computation, especially rarely encountered symbols; arithmetic tuning, intermediate calculation traces, external tools, and digit-level tokenization offer remedies.Synthetic arithmetic training [359] and tracing intermediate calculations [359] target calculation difficulties, while tool-use examples or instructions teach calculator manipulation. Individual digit tokens avoid inconsistent numerical splits produced by subword tokenizers, potentially improving inherent arithmetic ability [359].Source: §7.1.3 · p. 61

7.2.1 Human Alignment

  • Human alignment evaluates conformity to human values and needs through helpfulness, honesty, and safety criteria.Assessment combines falsehood-detection and harmlessness benchmarks with direct human evaluation; expert-interaction alignment tuning substantially reduces GPT-4’s incorrect behavior on sensitive or disallowed prompts. Annotation instructions and rules support evaluation, while high-quality pre-training data can reduce alignment effort.Source: §7.2.1 · p. 62

7.2.2 Interaction with External Environment

  • LLMs can interact with external environments and generate accurate action plans, evaluated through plan executability, correctness, and real-world success rates.Evaluation spans household simulators and open-world environments, including Minecraft agents that learn from environmental feedback or use task decomposition, planning, and interfaces; code-like prompts and real-world grounding improve interaction. Multi-agent simulations examine planning, memory, and emergent social behavior through human interviews and descriptive measurements.Source: §7.2.2 · p. 62

7.2.3 Tool Manipulation

  • External tools extend LLM capabilities in calculation and timely question answering, and LLMs can autonomously create task-specific tools.Tool manipulation is evaluated through mathematical reasoning and knowledge question answering, taught through contextual exemplars or simulated training data; retrieving relevant tool usage or embedding tool information addresses context-length constraints. Discovering and evaluating further emerging abilities, including self-improvement, remains an open direction.Source: §7.2.3 · pp. 62–63

7.3.1 Comprehensive Evaluation Benchmarks

  • 86.4% in the 5shot setting places GPT-4 ahead of previous state-of-the-art models on MMLU, which evaluates knowledge across multiple domains.Scaled LLMs exceed average humans on 65% of BIG-bench tasks under few-shot evaluation, while CoT prompting can exceed average human performance on BBH. HELM covers 16 scenarios and 7 metric categories, finding benefits from instruction tuning for accuracy, robustness, and fairness and from code pre-training for reasoning; human-exam benchmarks favor API models, with GPT-4 exceeding average humans on AGIEval but remaining below top human performance.Source: §7.3.1 · p. 63

7.3.2 Evaluation Approaches

  • Benchmark-based evaluation automates basic-ability testing, but prompts, shot settings, answer parsing, and training-data contamination can affect results.The evaluation taxonomy distinguishes base, generally fine-tuned, and specialized models by methodology and tested abilities; few-shot evaluation often suits base models, with CoT prompts needed for some complex reasoning tasks. Specialized models require domain-specific and general benchmarks together, exemplified by evaluating Med-PaLM with MultiMedQA [354] and MMLU.Source: §7.3.2 · pp. 63–65
  • Human evaluation directly assesses real-world performance but is subjective, costly, and difficult to reproduce; model judges improve scalability but exhibit biases and reasoning limitations.Human evaluators use pairwise comparisons or single-answer grading, while capable LLM judges can achieve high agreement with humans; closed-source judges introduce access and leakage risks, motivating open-source evaluator tuning. Model judges can explain scores but favor particular answer positions, verbosity, and their own generations; prompting and fine-tuning can partially mitigate these problems.Source: §7.3.2 · pp. 65–66

7.4.1 Experimental Settings

  • The empirical setup compares open-source base, instruction-tuned, and aligned models with closed-source API models; computational constraints limit evaluation of larger models.Open-source candidates include LLaMA (7B), LLaMA 2 (7B) and LLaMA 2-Chat (7B) [99], Pythia (7B and 12B) [96], Falcon (7B), Vicuna (7B and 13B), Alpaca (7B), and ChatGLM (6B). API-accessed candidates include text-davinci-002/003, ChatGPT, Claude, and Claude 2.Source: §7.4.1 · p. 66

7.4.2 Results Analysis and Findings

  • 78.47 versus 49.96 on GSM8k illustrates ChatGPT’s advantage over Davinci002; ChatGPT wins the most tasks among evaluated closed-source models.On HumanEval, ChatGPT scores 79.88 versus Claude’s 51.22.Source: Table 16 · p. 67, §7.4.2 · p. 69

7.4.1 Experimental Settings

  • Evaluation covers eight abilities, primarily using zero-shot prompting, with task-specific few-shot settings for harder reasoning, environment interaction, and search-tool use.Symbolic and mathematical reasoning use three-shot exemplars, with chain-of-thought prompting for mathematics following; ReAct settings use one shot for WebShop, two for ALFWorld, and three for HotpotQA, while Gorilla is zero-shot. Models share prompts and result parsing; closed-source evaluation uses official APIs, while open-source inference uses eight A800-80G GPUs, with experimental code and data released.Source: §7.4.1 · pp. 66, 67, 69, Table 16 · p. 67

7.4.2 Results Analysis and Findings

  • Claude 2, ChatGPT, and Davinci003 outperform other models on environment interaction and tool manipulation, potentially reflecting targeted optimization for these advanced capabilities.Support for external plugins is offered as a possible explanation, rather than an established cause.Source: §7.4.2 · p. 69
  • Closed-source models struggle with difficult reasoning and translation, while evaluated small open-source models perform poorly on mathematics, environment interaction, and tool manipulation.MATH and HotpotQA remain difficult even for ChatGPT; weak WMT performance may reflect insufficient pre-training coverage of minor languages. Open-source weaknesses may reflect instruction data not designed for these tasks and limited model capacity.Source: §7.4.2 · p. 69
  • Instruction-tuned open-source models generally outperform base models, but Vicuna (7B) and Alpaca (7B) lose performance on the LAMBADA language-modeling task.The authors suggest that instruction-following data does not necessarily benefit general language generation.Source: §7.4.2 · p. 69
  • Human-alignment rankings vary by task: LLaMA 2-Chat (7B) leads on TruthfulQA, whereas Vicuna (13B) leads on CrowS-Pairs among evaluated open-source models.Different alignment tasks assess different aspects, and even variants of the same model can exhibit inconsistent rankings.Source: §7.4.2 · p. 69
  • 43.95 versus 29.80 on C-Objects favors LLaMA 2 (7B) over Falcon (7B), illustrating its generally strong complex reasoning among evaluated base models.LLaMA 2 (7B) also performs comparably to the best base models on language generation and knowledge utilization. The authors attribute its performance mainly to about 2 trillion pre-training tokens, alongside more robust data cleaning.Source: §7.4.2 · pp. 69–70
  • Larger evaluated open-source models generally outperform smaller variants, with especially large advantages on symbolic and mathematical reasoning.The open-source findings remain limited to the model sizes evaluated.Source: §7.4.2 · p. 70

8 APPLICATIONS

This section surveys LLM applications in classic NLP, information retrieval, recommendation, multimodal modeling, automated evaluation, and specialized tasks, as well as how knowledge graphs can strengthen LLMs. It also examines limitations involving task performance, computational efficiency, context handling, data quality, hallucinations, evaluation biases, safety, and privacy.

8.1.1 LLM for Classic NLP Tasks

  • LLMs offer few-shot NLP capabilities, but fully supervised small models often outperform them on classic tasks, while tagging and complex extraction remain challenging.Ambiguous or rare tagging categories can be misunderstood from instructions and limited examples. Information extraction can benefit from LLM–small-model collaboration [765], while a two-stage workflow achieves competitive zero-shot performance [766].Source: §8.1.1 · p. 70
  • LLMs provide flexible text generation, but low-resource languages and domains remain difficult because training data are unbalanced [772].Prompting supports generation [768] [769], specialized requirements such as document-level translation [770], and interactive refinement [771]; model selection should weigh flexibility, labeled-data availability, training compute, and efficiency. Future directions include injecting task/domain knowledge through tuning or prompting, investigating complex semantic relations, combining LLMs with fine-tuned small models [775], and human–machine collaboration [771].Source: §8.1.1 · pp. 70–71

8.1.2 LLM for Information Retrieval

  • LLMs support retrieval through prompted reranking, synthetic supervision, and query/document rewriting, but inference costs, context limitations, and ranking adaptation constrain their use.Training-free rerankers use pointwise [779], pairwise [778], or listwise [780] instructions; LLMs also generate retrieval outputs [784], relevance labels [791], and document-conditioned queries [793], while dedicated retrieval or reranking fine-tuning is expensive. Remaining directions include better system adaptation, redesigned information-seeking architectures [799], and multimedia retrieval using multimodal LLMs.Source: §8.1.2 · pp. 71–72

8.1.3 LLM for Recommender Systems

  • LLM agents simulate recommendation interactions using profiles, memory, and self-reflection; AgentCF additionally models both users and items as agents.Most simulators are user-oriented, whereas AgentCF uses collaborative reflections to capture two-sided user–item relationships.Source: §8.1.3 · p. 73
  • LLMs can directly recommend items or enhance traditional recommenders, but zero/few-shot recommendations tend to underperform traditional ID-based systems.Direct approaches use prompting, instruction tuning, or semantic identifiers [813]; enhancement approaches infer intentions, encode side information, or transfer capabilities into small models deployed without LLM inference [819]. Instruction tuning only partly bridges the collaborative-semantic gap, while tuning costs, latency, memory demands, and long interaction histories remain challenges.Source: §8.1.3 · pp. 72–73

8.1.4 Multimodal Large Language Model

  • Multimodal LLMs connect image encoders to language models, typically using vision-language alignment pre-training followed by visual instruction tuning.A connection module maps image-patch embeddings into representations combined with text embeddings for autoregressive responses. Alignment strategies depend on data availability: connector-only updates are suggested below 1M image-text pairs, high-quality data can justify LLM tuning, and roughly 1B pairs may support vision-encoder tuning, although its benefit requires verification.Source: §8.1.4 · p. 73
  • Multimodal evaluation distinguishes perception from cognition, with studies exposing persistent object hallucinations and weaknesses on challenging visual questions.Exact-match evaluation can underestimate open-ended responses, whereas human or LLM assessment improves evaluation accuracy at higher cost. Benchmarks aggregate existing tasks through LVLM-eHub and Reform-Eval, or introduce perception/cognition, consistency, temporal-understanding, and integrated-capability assessments through MME, MMBench, SEED-Bench, and MM-Vet.Source: §8.1.4 · p. 74
  • Improving multimodal LLMs requires high-quality visual instructions and training that balances multimodal adaptation against preservation of pretrained language capabilities.Instruction quantity and quality matter; distillation, task reformulation, diversity, fine-grained annotations, and complex reasoning examples expand supervision, but synthesized instructions require verification against visual misinterpretations. Connector-only tuning preserves language capabilities but may adapt weakly, whereas tuning both connector and LLM improves adaptation while risking capability loss; reliance on language-model capabilities also limits tasks such as spatial positioning.Source: §8.1.4 · pp. 74–75
  • Even advanced multimodal models such as GPT-4V can produce factual inaccuracies, baseless image inferences, and harmful content.Mitigations include specialized visual instructions against hallucination, post-hoc revision models [862], and RLHF to improve factuality [863]. Existing multimodal alignment techniques mainly address specific problems rather than comprehensively covering alignment criteria.Source: §8.1.4 · p. 75

8.1.5 KG-Enhanced LLM

  • Knowledge graphs address hallucinations and domain-knowledge gaps through retrieved factual context or planned, multi-step interaction.Retrieval generally inserts subgraphs into prompts without parameter updates, but serialization loses structure and graph-to-text conversion requires sufficient training pairs and model capability; synergy decomposes tasks into subgoals, with StructGPT’s specialized interfaces reducing costly candidate enumeration. Open directions include unified access to heterogeneous sources, cost-effective knowledge updates, and factual alignment using knowledge graphs.Source: §8.1.5 · pp. 75–76

8.1.6 LLM for Evaluation

  • GPT-4 closely matches human preferences in MT-Bench’s no-tie comparisons, but advanced LLMs fall below human-level evaluation on LLMBar’s deceptive examples.Evaluators provide scores or language feedback; improvement methods include aggregating varied prompts, multi-model consensus, and specialized fine-tuning. Order, self-preference, and length biases persist, while difficulty detecting their own reasoning or factual errors without external feedback limits evaluation of capable models and complex tasks.Source: §8.1.6 · pp. 76–77

8.2 LLM for Specific Domains

  • Med-PaLM models achieve expert-level USMLE performance and greater physician approval when answering consumer medical questions [354].Healthcare applications include information extraction, consultation, mental-health analysis, and report simplification, typically guided by task-specific prompts. Medical misinformation and uploading patient information to commercial servers [765] remain important risks.Source: §8.2 · p. 77
  • LLMs achieve student-level standardized-test performance and assist educational writing and reading, but raise plagiarism, bias, overreliance, and access concerns.ChatGPT produces logically consistent cross-disciplinary answers, and students using its answers outperform average students in some computer-security courses. Proposed classroom uses include teacher–student collaboration, personalized learning, and automated assessment.Source: §8.2 · p. 77
  • Top 10% is GPT-4’s score in a simulated bar exam relative to human test-takers.Legal applications include document analysis, judgment prediction, and drafting; specialized prompting supports long-document comprehension and complex legal reasoning. Copyright, personal-information leakage, and bias or discrimination remain concerns.Source: §8.2 · p. 77
  • General-purpose LLMs underperform million-parameter domain-specific PLMs on financial tasks, motivating continued pre-training on financial corpora.BloombergGPT performs strongly across financial tasks while retaining competitive general-purpose performance [358]. Inaccurate or harmful financial content could adversely affect markets, requiring stricter review and monitoring [358].Source: §8.2 · p. 77
  • LLMs assist scientific research through literature surveys, hypothesis generation, data analysis, writing, and automated paper review.Training on scientific corpora supports knowledge-intensive scientific tasks, but scientific-content quality and harmful hallucinations still need improvement. Additional applications include psychological studies of human-like characteristics and software code suggestion, summarization, and repair, with accuracy, bias, privacy, and data security remaining cross-domain concerns.Source: §8.2 · pp. 77–78

9 ADVANCED TOPICS · 9.1 Long Context Modeling · 9.2 LLM-empowered Agent

Long-context modeling extends LLMs through position-embedding changes, context-window adaptation, and curated long-text training. LLM-based agents integrate memory, planning, and execution, but face computational, alignment, tool-use, and reliability challenges.

9 ADVANCED TOPICS · 9.1 Long Context Modeling · 9.2 LLM-empowered Agent

  • Long-context modeling and LLM-empowered agents are advanced topics involving technical challenges that limit LLM capabilities.Document parsing, extraction, and summarization motivate scaling position embeddings or adapting context windows. LLMs offer a core computation unit for agents beyond traditional heuristic or environment-specific approaches.Source: §9.1 · p. 78, §9 · p. 78, §9.2 · p. 80

9.1.1 Scaling Position Embeddings

  • RoPE’s limited extrapolation motivates long-text fine-tuning, position interpolation, and position truncation with ReRoPE and LeakyReRoPE.Direct fine-tuning is slow and sensitive to data quality; interpolation scales position indices by L/L′ (L < L′), extending context efficiently but potentially harming original-window performance. Position truncation preserves local attention while truncating or interpolating distant positions, improving extrapolation but requiring two attention-matrix computations.Source: §9.1.1 · p. 78
  • Decreasing RoPE’s base improves extrapolation, whereas increasing it performs better within the training length.Increasing the base reduces rotation frequencies to accommodate longer distances; decreasing it brings wavelengths below training length but often requires continual pre-training. Basis truncation modifies low-frequency dimensions to avoid unseen rotation angles, but performs poorly on long-context tasks.Source: §9.1.1 · p. 79

9.1.2 Adapting Context Window

  • Context-window adaptation handles overlength sequences using parallel segments, Λ-shaped attention, token-level retrieval, or block-level retrieval.Parallel segments share position embeddings but lose intersegment order; Λ-shaped masks retain initial and recent tokens for fixed-memory generation, sacrificing access to discarded information and long-range dependencies. Token retrieval uses external keys and k-NN, typically at one decoder layer, whereas block retrieval selects relevant, neighboring, and initial blocks using head-specific retrieval.Source: §9.1.2 · p. 79

9.1.3 Long Text Data

  • Over 100K tokens of context with effective utilization are achievable by LLaMA-2-7B or LLaMA-2-13B after training on several billion tokens.Continual pre-training needs comparatively little curated long-text data but can impair short-text performance. LongWanjuan supports removing chaotic texts and up-sampling cohesive texts; data mixtures should avoid substantial drift from original pre-training distributions.Source: §9.1.3 · p. 79

9.2.1 Overall Framework.

  • LLM-based agents integrate memory, planning, and execution under LLM control, repeatedly adjusting their behavior through environmental feedback.Short-term memory uses the context window; long-term storage supports retrieval, reflection, and role profiles, while planning selects candidate instructions or programs and incorporates execution feedback [530]. The workflow receives information, updates memory, retrieves relevant knowledge, generates a plan, and executes it internally or with external tools.Source: §9.2.1 · p. 80

9.2.2 Applications

  • Single-agent applications autonomously solve user requests, while multi-agent applications coordinate specialized roles through cooperation or competition.AutoGPT combines memory, tools, planning, and feedback-driven refinement to resolve requests. Cooperation uses free-form dialogue, structured documents, or embeddings for applications including software engineering and society simulation, whereas competitive debate supports mathematical reasoning and evaluation.Source: §9.2.2 · pp. 80–81

9.2.3 Discussion

  • Agent development faces computational burdens, human-social alignment gaps, tool-capability limitations, and robustness and trustworthiness requirements.Increasing agent counts compounds computation and communication costs, while rare-role imitation and inadequate cognitive-psychology modeling hinder human alignment. Tool availability, dynamic environments, compatibility, and extensibility constrain capabilities; adversarial training, augmentation, and detection strengthen security, while constrained decoding and external knowledge only partly mitigate hallucinations.Source: §9.2.3 · p. 81

9.3 Analysis and Optimization for Model Training

Training larger LLMs with limited computational resources requires controlling GPU memory consumption while maintaining training efficiency. The section estimates memory costs and describes checkpointing, state partitioning, offloading, attention optimization, and sequence parallelism.

9.3.1 Estimation of Training Memory Consumption

  • Mixed-precision training with Adam or AdamW requires 16P bytes for model states of a P-parameter model; LLaMA-7B, with approximately 6.7 billion parameters, needs around 100GB for these states alone.Parameters and gradients require 2P bytes each, while optimizer states require 12P bytes.Source: §9.3.1 · p. 81
  • Approximately 1GB for PyTorch, 1GB to 4GB for distributed frameworks, and 0.5GB to 1GB for fragmentation add to training memory requirements.Distributed-framework memory usage depends on optimization and hyperparameter settings. Transformers-library softmax also stores two additional 32-bit input copies, increasing peak memory consumption.Source: §9.3.1 · p. 82

9.3.2 Memory Optimization Methods

  • Gradient checkpointing saves activation memory by retaining only selected activations during the forward pass and recomputing them during backpropagation.Implementations typically retain each Transformer layer’s input and recompute its activations during the backward pass, trading additional computation for reduced memory usage.Source: §9.3.2 · p. 82
  • About 50% additional communication accompanies ZeRO-3, which saves memory proportional to GPU count; ZeRO-1 and ZeRO-2 do not increase communication overhead.ZeRO counters replicated model states in data parallelism through optimizer-state partitioning, gradient partitioning, and parameter partitioning at its three respective stages. PyTorch provides a similar technique, fully sharded data parallel (FSDP).Source: §9.3.2 · p. 82
  • DeepSpeed offload moves gradients and optimizer states to CPU memory while retaining model parameters and forward/backward computation on GPUs.Parameter updates run on the CPU to reduce GPU memory requirements. Infinity extends this approach to high-speed disk storage, such as NVMe, allowing training beyond GPU memory limits.Source: §9.3.2 · p. 82

9.3.3 Efficiency Optimization Methods

  • One-tenth of standard computation time is required by FlashAttention [303] for LLaMA2 (7B) with sequence length 2,048 and batch size 8.Matrix partitioning and operator fusion keep intermediate results in cache, reducing memory reads and writes. FlashAttention also reduces peak memory usage and activation memory consumption during training and inference.Source: §9.3.3 · pp. 82–83
  • Sequence parallelism distributes long sequences across devices, adding a fourth pre-training parallelism dimension while making attention communication a central challenge.DeepSpeed-Ulysses partitions along the hidden dimension for parallel attention-head computation, whereas Ring Attention partitions sequence length and computes local queries against other devices’ keys and values. Ring Attention is compatible with FlashAttention and can be considered its distributed extension.Source: §9.3.3 · p. 83

9.4 Analysis and Optimization for Model Inference

Decoding’s memory-transfer bottleneck is the main source of inference inefficiency in the analyzed configuration. System-level optimizations reduce data transfers or enlarge batches, while algorithm-level methods change token generation and computational effort.

9.4.1 Analysis of Inference Efficiency

  • Arithmetic intensity compares computation with data transfer; operations are memory-bound below the GPU’s compute-to-bandwidth ratio and compute-bound above it.GPU capability is measured in FLOP/s and bandwidth in byte/s; model operations use FLOPs and transferred bytes, yielding arithmetic intensity in FLOP/byte.Source: §9.4.1 · p. 83
  • Below 8, decoding linear-transformation and multi-head-attention arithmetic intensities indicate a memory-bound bottleneck, whereas prefill is compute-bound for the analyzed LLaMA (7B) configuration.The example uses batch size 8 and sequence length 1024, with an A100 comparison threshold of Imax = 142.51. Prefill arithmetic intensities are approximately 2730.67 for linear transformations and 114.67 for multi-head attention.Source: §9.4.1 · p. 83, Table 20 · p. 84, Table 19 · p. 84

9.4.2 System-level Optimization

  • Flash-Decoding parallelizes attention over KV-cache chunks, while PagedAttention reduces cache-management transfers and parallelizes attention.FlashAttention reduces prefill transfers but does not directly apply to decoding; Flash-Decoding targets decoding, particularly for long sequences. PagedAttention preallocates memory blocks to reduce repeated allocation and copying, and uses operator fusion for parallel attention computation.Source: §9.4.2 · p. 84
  • Continuous batching in vLLM increases effective decoding batch size, while DeepSpeed-MII’s Dynamic SplitFuse combines prefill and decoding to improve throughput.Continuous batching schedules requests at prefill or single-step decoding granularity to increase arithmetic intensity. Dynamic SplitFuse splits prefill into multiple iterations, enabling larger batches containing both prefill and decoding work.Source: §9.4.2 · pp. 84–85

9.4.3 Algorithm-level Optimization

  • 2× to 3× speedup without compromising generation quality is reported for speculative decoding, which uses smaller-model token drafting followed by larger-model verification.Variants combine several small models through learning or first accelerate the small drafting model with an even smaller model.Source: §9.4.3 · p. 85
  • FrugalGPT processes requests through models ordered from higher to lower efficiency, stopping when a trained reliability classifier accepts an answer.The classifier threshold trades inference efficiency against generation quality across open-source models and commercial APIs. For reasoning, another approach invokes a larger model only when smaller-model answers have low self-consistency.Source: §9.4.3 · p. 85
  • Non-autoregressive decoding generates tokens simultaneously but sacrifices quality; semi-autoregressive decoding and Medusa instead generate token groups.Mainstream next-token pretraining prevents direct non- or semi-autoregressive generation; Medusa adds two Vicuna prediction heads to generate three tokens simultaneously. Quality limitations motivate combining these methods with speculative verification, including verification by the original Vicuna model for Medusa.Source: §9.4.3 · p. 85
  • Early exit generates tokens from intermediate layers, whereas mixture-of-depths selectively skips individual layers using learned routing scores.Early-exit conditions can use prediction confidence or next-token distribution entropy. Mixture-of-depths computes a layer only when its routing score exceeds a threshold, rather than skipping every subsequent layer.Source: §9.4.3 · p. 85

9.5 Model Compression

This section focuses on reducing LLM inference memory through quantization, while briefly considering pruning and distillation. It favors post-training quantization for computational efficiency and identifies activation outliers as a quantization challenge.

9.5 Model Compression

  • Model compression targets LLMs’ costly inference memory footprint [999], primarily through quantization, with pruning and distillation also discussed.Source: §9.5 · p. 85

9.5.1 Quantization Methods

  • Quantization-aware training requires additional full-model retraining, whereas post-training quantization requires no model retraining.Source: §9.5.1 · p. 85
  • LLMs’ huge parameter counts favor cheaper post-training quantization, while large activation outliers complicate quantization, especially of hidden activations.Source: §9.5.1 · pp. 85–86

Background for Quantization.

Quantization maps floating-point weights and activations to integers [1000], using scaling, rounding, and zero-point offsets; clipping ranges require calibration offline or at runtime. This section reviews LLM quantization methods and precision trade-offs, alongside distillation, pruning, and deployment libraries.

Background for Quantization.

  • Post-training methods reduce quantization error through mixed-precision outlier separation, finer-grained quantization, and migration of activation quantization difficulty to weights.LLM.int8() separates outlier feature dimensions during matrix multiplication; ZeroQuant uses dynamically calibrated token-wise activations and group-wise weights, commonly with group size 128. SmoothQuant uses an equivalent scaling transformation controlled by migration strength; efficient deployment also requires hardware or system support.Source: Background for Quantization · p. 86
  • Protecting quantization-sensitive weights and minimizing layerwise reconstruction error provide complementary approaches to accurate low-bit weight quantization.Static salient weights can be identified before deployment using magnitude, structural patterns, or Hessian-based sensitivity,,,. GPTQ [1009] enables 3- or 4-bit quantization of models such as OPT-175B through fixed ordering, lazy batch updates, and Cholesky reformulation; AWQ instead searches activation-aware weight scales on calibration data.Source: Background for Quantization · p. 86
  • 4-bit quantized models can match full 16-bit fine-tuning performance using QLoRA [1011], which adds small, trainable 16-bit adapters.Efficient fine-tuning can compensate for quantization degradation through high-precision adapter updates [1015] or suitable low-rank initialization. Updating only small adapters also supports lightweight instruction- or chat-oriented customization [1011].Source: Background for Quantization · p. 87
  • INT8 weight quantization usually avoids performance degradation, whereas INT4 and INT3 depend on specialized strategies [1009].LLMs appear less sensitive to low-bit weight quantization than smaller language models; at equal memory cost, a 4-bit 60B model outperforms an 8-bit 30B model. In-context learning, step-by-step reasoning, and instruction following are reportedly seldom affected by 4-bit weight quantization [1015].Source: Background for Quantization · p. 87
  • Activations are harder to quantize than weights, with large activation outliers appearing at model sizes of 6.7B parameters or above.High-quality INT8 activation quantization remains difficult, and lower precision remains unresolved. Data-free distillation-based QAT on LLaMA shows promising 4-bit weight and key-value-cache quantization, but not 4-bit activation quantization.Source: Background for Quantization · p. 87
  • 8-bit and 4-bit weight quantization yield performance close to 16-bit fine-tuned LLaMA models while substantially reducing memory consumption.Experiments cover 7B and 13B models fine-tuned on FLAN-v2, Alpaca-52K, and ShareGPT, evaluated on AlpacaFarm, MMLU, and BBH. The authors recommend testing 4-bit weight quantization first when deployment memory is critical.Source: Background for Quantization · p. 87, Table 21 · p. 88

9.5.2 Other Model Compression Methods

  • 60% unstructured sparsity is achieved by SparseGPT on OPT-175B while retaining relatively low perplexity, illustrating parameter reduction through pruning.Structured pruning removes components rather than individual connections: LLM-pruner removes 20% of LLaMA-7B parameters while retaining 93.6% of original performance. Distillation instead transfers teacher capabilities to students through output or intermediate-state alignment when weights are accessible, or textual responses in black-box settings.Source: §9.5.2 · pp. 87–88

9.5.3 Open-source Libraries

  • Deployment libraries support low-bit quantization, adapter integration, device-specific inference, and structural pruning.Bitsandbytes supports 8-/4-bit matrix multiplication and 8-bit optimizers,; AutoGPTQ provides INT4 quantization based on GPTQ [1009] with PEFT/LoRA integration, while llama.cpp supports INT4/INT5/INT8 inference on MacBooks. Torch-Pruning uses dependency graphs for structural pruning, whereas LLM-Pruner provides gradient-based LLM pruning with limited training samples and time.Source: §9.5.3 · p. 88

9.6 Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) [1024] [1025] supplements LLM prompts with external knowledge to reduce factual errors involving real-time or specialized information. Its workflow combines retrieval, prompt construction, and generation, with improvements targeting retrieval quality, context refinement, iterative retrieval, and training.

Basic Workflow

  • RAG [1024] [1025] retrieves external context to reduce factual errors when an LLM’s internal knowledge is insufficient for real-time information or specialized domains.The standard workflow comprises context retrieval, prompt construction, and response generation. Retrieval commonly uses sparse lexical matching with inverted indexes or dense embedding similarity with approximate nearest neighbor indexes.Source: §9.6 · pp. 88–89

Prompt Construction

  • Prompt construction combines retrieved documents with task instructions, but long concatenated contexts can suffer from biased attention, including lost-in-the-middle effects.Reranking selects the most relevant documents [1027], while extraction or compression retains highly relevant information and shortens the input context [1028].Source: §9.6 · p. 89

Response Generation

  • During response generation, irrelevant or contradictory retrieval results can impair outputs, motivating self-checking and renewed retrieval or confidence-based retrieval decisions.Confidence assessment can determine whether the task requires retrieval or whether retrieved content should be used.Source: §9.6 · p. 89

Retrieval method improvement

  • Retrieval optimization balances relevance against latency using propositions [1031] and improves queries through expansion [796], decomposition, rewriting, and distillation.Coarse retrieval admits more irrelevant content, whereas fine-grained retrieval increases latency; propositions offer semantically complete units, extracted by a smaller model trained using GPT-4-synthesized instruction data [1031]. Expansion adds related information [796] but can distort complex queries, motivating separate expansion of decomposed subqueries; prompted LLMs can rewrite queries [1034], with distillation reducing inference overhead.Source: §9.6 · p. 89

Retrieval results refinement

  • Retrieval-result refinement filters or reorders documents by relevance [1036] and extracts concise content or removes unimportant tokens to improve context utilization.Generation and reranking can be jointly optimized [1027], while prompted LLMs can directly rerank documents [777]. Information extraction and summarization retain concise, query-relevant content.Source: §9.6 · pp. 89–90

Iterative retrieval enhancement

  • Iterative retrieval refines queries using generated results, while adaptive retrieval gives LLMs greater autonomy over retrieval use [1038].Accumulated evidence can become redundant or conflicting, motivating confidence-based stopping; intermediate chain-of-thought results can also guide subsequent retrieval and incorporate its findings. Both approaches require deciding when to retrieve and using preset prompts for query generation and retrieval-result processing [1039].Source: §9.6 · p. 90

RAG-enhanced training

  • RAG-enhanced training uses instruction tuning [1040] and retrieval-oriented pre-training to improve how LLMs use retrieved context.Instruction data can vary relevant-document positions to counter positional bias and include irrelevant information to improve interference resistance [1041]. A common pre-training task uses document portions as queries and reconstructs the remaining content from retrieval results.Source: §9.6 · p. 90

9.7 Hallucination

Hallucination—generating content inconsistent with factual information—significantly affects LLM task performance [1044]. This section classifies hallucinations, examines their sources, and reviews detection and mitigation methods.

9.7.1 Definition of Hallucination

  • Factual hallucinations include incorrect entities, incorrect relationships, and incomplete answers to questions requiring aggregated facts [1044].Earlier definitions distinguish intrinsic contradictions of the input from extrinsic claims unverifiable against it; open-domain factual hallucinations instead concern consistency with world knowledge [1044] [1046]. Incompleteness particularly affects lengthy or list-based responses.Source: §9.7.1 · p. 90
  • Other factual hallucinations involve outdated information, claims exceeding factual knowledge, or information unverifiable against existing sources [1044].Outdatedness commonly arises because training corpora cover limited periods; unverifiability makes accuracy difficult to assess.Source: §9.7.1 · p. 90

9.7.2 Source of Hallucination

  • Training-data errors, biases, temporal limits, missing domain knowledge, and sparse long-tail knowledge can contribute to hallucinations.Repeated inaccuracies can be memorized and reproduced as “imitative falsehoods”, while biased content can distort learned world knowledge. Knowledge becomes outdated, and infrequently represented facts are more likely to elicit inaccurate content [1044].Source: §9.7.2 · p. 91
  • Pre-training and post-training can introduce hallucinations through attention decay, exposure bias, contaminated instruction data, and reward-driven agreement with humans.Autoregressive attention decay can impair long-range dependencies, while teacher forcing creates a mismatch between training on correct preceding tokens and inference on generated tokens. Distilled instruction data may contain hallucinations, and alignment rewards may encourage answers that cater to humans rather than facts.Source: §9.7.2 · p. 91
  • Poorly formulated prompts and diversity-oriented decoding can increase hallucination risk.Unclear, complex, non-standard, or abstract instructions can cause misunderstanding; readability, format, and concreteness affect outputs [1044]. Higher sampling temperature flattens token probabilities, increasing the chance of selecting low-frequency, irrelevant words.Source: §9.7.2 · p. 91

9.7.3 Hallucination Detection

  • Hallucination detection uses model-based factual assessment, uncertainty estimation, and external verification tools.Model-based methods can extract and assess individual claims [1044], with HaluEval finding that CoT prompting improves detection; uncertainty methods examine token probabilities, repeated-answer consistency through SelfCheckGPT, or reconstructed-question consistency. Tool-based methods verify fine-grained claims using knowledge sources through FActScore, calculators and code interpreters through FacTool, or tool-assisted agents through HaluAgent.Source: §9.7.3 · pp. 91–92

9.7.4 Hallucination Mitigation

  • Retrieval-augmented generation and improved decoding mitigate hallucinations using reliable external context, internal model representations, or knowledge-guided token selection.RARR and Verify-and-Edit revise generated answers using retrieved evidence, LLM-Augmenter adds local databases and agent feedback, and IRCoT interleaves retrieval with reasoning. DoLa contrasts lower- and final-layer logits, ITI shifts attention-head activations along truth-correlated directions, CAD contrasts contextual and context-free probabilities, and KCTS reweights tokens using a knowledge-faithfulness classifier.Source: §9.7.4 · p. 92
  • Human alignment can mitigate hallucinations, but preference optimization may encourage sycophancy rather than truthfulness.HaluEval 2.0 [1044] trains a reward model on hallucinated and non-hallucinated responses, then uses its feedback for reinforcement learning. Proposed safeguards aggregate multiple human preferences or fine-tune on prompts whose factual truth is independent of user opinion.Source: §9.7.4 · p. 92

9.8 Complex Reasoning

Long CoT reasoning supports trial-and-error search in natural language, developed through automatically constructed reasoning data and supervised or reinforcement learning. Within broader test-time scaling, scaling RL offers sustained improvements, while supervised tuning can encounter performance ceilings and unnecessary extended reasoning.

9.8.1 Overview and Analysis

  • Long CoT searches natural-language solution spaces through trial and error, incorporating planning, evaluation, reflection, exploration, and opportunities to verify or revise attempts.Analyses of o1 identify systematic analysis, method reuse, divide-and-conquer, self-refinement, context identification, and constraint emphasis, with patterns varying across tasks [1065]. With appropriate training, textual forward exploration and backward revisits can emulate tree-of-thought and graph-of-thought search rather than requiring strictly linear reasoning.Source: §9.8.1 · p. 93

9.8.2 Construction of Long CoT Data

  • Teacher-model distillation produces long CoT training data, with STILL-2 [1067] identifying response-length distribution as a critical quality factor.Costly expert annotation motivates automatic synthesis from curated prompts, particularly mathematical problems requiring extended solutions. Small curated datasets can activate slow thinking, and DeepSeek-R1 distillation improves multiple Qwen and Llama models.Source: §9.8.2 · pp. 93–94
  • Search-based synthesis scores multi-step reasoning trajectories, while multi-agent collaboration [1072] generates long CoT through iterative reflection and debate.MCTS samples one-step candidates and uses rollout-derived Q-values to favor steps leading to correct solutions, retaining successful and trial-and-error steps in trajectories. Agents propose hypotheses, challenge assumptions, assess logical consistency, and refine conclusions through alternative perspectives and counterarguments.Source: §9.8.2 · p. 94

9.8.3 Training Methods

  • 3.9K distilled long CoT examples let STILL-2 [1067] fine-tune Qwen2.5 (32B) to mathematical performance comparable to o1preview and QwQ.Supervised tuning teaches thought-then-solution formatting and activates existing reasoning abilities; mathematics-only training also improves physics and chemistry [1067]. This reasoning mode can extend to multimodal LLMs built on language-model backbones [1073].Source: §9.8.3 · p. 94
  • Supervised long CoT tuning risks imitation-learning ceilings and unnecessary extended reasoning on simple problems, motivating integrated long-reasoning and standard-response modes.Rejection sampling and directional preference optimization can help when warm-up data are limited [1067]. Their benefits diminish with sufficient instruction data, especially when self-generated samples are no better than demonstrations [1067].Source: §9.8.3 · p. 95
  • Scaling RL develops reasoning through policy exploration, verifiable rewards, and efficient algorithms such as GRPO [1078] and RLOO.Strong policies usually receive supervised warm-up, although DeepSeek-R1-Zero skips this step using format rewards; answer- or test-based rewards support verifiable tasks, while open-ended tasks require trainable reward models. GRPO [1078] and RLOO avoid PPO’s costly value model; effective training requires tracking response length alongside performance, selecting challenging-but-solvable problems, updating reference models, and encouraging exploration.Source: §9.8.3 · p. 95

9.8.4 Extended Discussion

  • Test-time scaling trades additional inference computation for performance; long CoT searches within one response, with potential specialized-domain applications and reasoning-specific safety needs.Other approaches include Self-Consistency’s multiple-response majority voting and planning or agentic methods using repeated prompts, tools, or memory. Potential expert-level applications could support scientific research, while improved inference methods and hardware could reduce costs; specialized alignment remains necessary.Source: §9.8.4 · p. 96
  • More than 8,000 training steps still yield DeepSeek-R1-Zero improvements; scaling RL also offers higher token efficiency than existing test-time scaling methods.Token efficiency measures performance improvement per token cost, while performance ceiling measures maximum attainable performance. Heuristic methods and supervised fine-tuning often reach ceilings, whereas scaling RL can continue improving reasoning performance with additional training [1067] [1080].Source: §9.8.4 · p. 96

10 CONCLUSION AND FUTURE DIRECTIONS

The survey reviews models larger than 10B parameters across pre-training, adaptation, utilization, and evaluation, alongside development resources and reproduction guidelines. Future directions address foundational understanding, architectures, training, utilization, safety, and application ecosystems.

10 CONCLUSION AND FUTURE DIRECTIONS

  • The survey focuses on models larger than 10B parameters, covering pre-training, adaptation, utilization, evaluation, development resources, and implementation guidelines.It excludes early pretrained models such as BERT and GPT-2, which existing literature already covers, and targets both researchers and engineers.Source: §10 · p. 96

Basics and Principles

  • Explaining how unsupervised language modeling produces broad capabilities requires understanding data-dependent capacity learning, scaling, generalization, and reliable evaluation.The survey calls for analyzing which large-model behaviors can be predicted from smaller models, given scaling’s importance, and whether generalization extends beyond pre-training knowledge. Data contamination makes appropriate evaluation protocols essential for assessing capacity fairly [740].Source: §10 · p. 96

Model Architecture

  • Transformer architectures need lower training and inference costs, better use of long contexts, and broader exploration beyond decoder-only designs.System- and hardware-level optimization, including FlashAttention [303], merits further exploration. Despite long-context modeling efforts, models still struggle to use information within their windows, motivating architecture adaptations or specialized algorithms.Source: §10 · pp. 96–97

Model Training

  • LLM training needs reproducible, data-centric infrastructure, economical optimization, and effective adaptation of existing checkpoints rather than repeated training from scratch.Priorities include systematic collection, cleaning, mixture, curriculum, flexible cluster-resource scheduling, shared training recipes, and economical approaches such as predictable scaling and proxy model training [59]. Continued pre-training and fine-tuning must address catastrophic forgetting and task specialization, while knowledge injection or editing can correct outdated facts.Source: §10 · p. 97

Model Utilization

  • LLM utilization requires better explanations and selection of effective prompts, stronger reasoning methods, and lower inference costs for large-scale deployment.ICL combines task descriptions and demonstrations, sometimes outperforming full-data fine-tuning; CoT supplies intermediate reasoning steps, while planning iteratively invokes LLMs using tools. Retrieval augmentation extends knowledge and improves question answering [454], but remains constrained by ineffective long-context utilization.Source: §10 · p. 97

Safety and Alignment

  • Safety priorities include reducing hallucinations and malicious outputs, while making human-feedback alignment less costly and difficult.Proposed directions include model-assisted annotation with quality guarantees and simplified alignment optimization [388] to reduce training difficulty and instability. Red teaming uses adversarial prompts to improve safety, while federated learning can support privacy-restricted fine-tuning.Source: §10 · p. 97

Application and Ecosystem

  • LLMs could reshape information seeking and enable ecosystems of agent-like applications, while exploration toward AGI must prioritize benefits and safety for humanity.The survey anticipates effects on search engines and recommender systems, alongside autonomous agents for complex tasks. It identifies OpenAI’s Assistants API and GPT Store as examples supporting application development and an emerging ecosystem.Source: §10 · pp. 97–98

CODA

The survey is a repeatedly revised reference whose model coverage and experiments remain constrained by space and computing resources. The authors document updates and invite reader feedback to improve coverage and accuracy.

Survey Writing

  • The survey originated as a readable overview for the research team and underwent extensive revisions after its March 13, 2023 first draft.Space limits restrict model coverage in Figure 3 and Table 1; the regularly maintained GitHub page uses more relaxed model-selection criteria.Source: CODA · p. 98

Seeking for Advice

  • The authors acknowledge possible missing references, omitted topics, and non-rigorous discussion, inviting reader feedback to guide continuing revisions.Readers can submit suggestions through GitHub or email; constructive contributions will be acknowledged.Source: CODA · p. 98

Update log

  • The update log records expanded architecture and advanced-topic coverage in 2024, long chain-of-thought reasoning in 2025, and new LLMs in 2026.Updates include SSM-based architectures, architecture parallelism and complexity comparisons, and topics such as retrieval-augmented generation and hallucination. Long chain-of-thought additions cover reasoning patterns, data construction, instruction tuning, and reinforcement learning.Source: CODA · p. 99

Clarifications on Experiments

  • Computational constraints limit the survey’s experiments to smaller models or few comparisons, leaving results incomplete.Experiments cover instruction-tuning (Table 10), overall ability evaluation (Table 16), and prompt engineering (Table 17). The authors plan larger-model results and additional comparisons and request computing support.Source: CODA · p. 99

Related papers

Loading 2303.18223v19…