Large Language Models: A Survey

Motivation

• The rapidly changing LLM field makes it difficult for researchers and practitioners to identify effective recipes for building LLM-powered systems.The paper presents a timely survey intended as an accessible resource for students, researchers, and developers.Source: §I · p. 1

LLM foundations

• Language modeling evolved from statistical n-gram models through neural and pre-trained models to large language models.N-gram models estimate word probabilities from local context but struggle with data sparsity and the diversity of natural language; neural models address sparsity using continuous embeddings.Source: §I · p. 1
• LLMs are transformer-based neural language models with tens to hundreds of billions of parameters, pretrained on massive text datasets.Compared with pre-trained language models, they exhibit stronger language understanding and generation, along with emergent abilities such as in-context learning.Source: §I · p. 2

LLM applications

• LLMs can support general-purpose AI agents, but agents must augment static language models to obtain updated information and interact with dynamic environments.The survey discusses these LLM-based agents in Section IV.Source: §I · pp. 1–2

Survey scope

• The survey reviews GPT, LLaMA, and PaLM alongside other representative models, then covers how LLMs are built, used, and augmented.It also reviews datasets, benchmarks, evaluation results, challenges, and future research directions.Source: §I · p. 2, §II · p. 2

LLM foundations

• Transformers use self-attention to compute word interactions in parallel, enabling efficient pretraining of very large language models on GPUs.Transformer-based pre-trained models are organized into encoder-only, decoder-only, and encoder-decoder architectures.Source: A. Early Pre-trained Neural Language… · p. 2

Encoder-Decoder PLMs

• T5 casts all NLP tasks as text-to-text generation, while mT5 extends this framework to 101 languages.mT5 is pretrained on a Common Crawl-based multilingual dataset.Source: 3) Encoder-Decoder PLMs · p. 4
• MASS masks a contiguous sentence fragment for the encoder and trains the decoder to predict that fragment.This jointly trains the encoder and decoder through masked sequence reconstruction.Source: 3) Encoder-Decoder PLMs · p. 4
• BART uses a sequence-to-sequence architecture trained to reconstruct original text from arbitrarily corrupted inputs.Its pretraining objective learns denoising reconstruction.Source: 3) Encoder-Decoder PLMs · p. 5

Large Language Models

• LLMs are transformer-based pretrained language models with tens to hundreds of billions of parameters, stronger language abilities, and emergent capabilities than smaller PLMs.The survey reviews GPT, LLaMA, and PaLM as representative LLM families.Source: B. Large Language Model Families · p. 5

GPT Family

• GPT models use decoder-only Transformers; GPT-3 demonstrated in-context learning, while GPT-4 accepts image and text inputs and produces text outputs.GPT-3 has 175 billion parameters, and GPT-4 was pretrained for next-token prediction before RLHF fine-tuning.Source: 1) The GPT Family · pp. 5–7
• ChatGPT uses instruction-following models derived from GPT-3.5 and later GPT-4 to support tasks including question answering, information seeking, and summarization.The models are trained to follow an instruction in a prompt and provide a detailed response.Source: 1) The GPT Family · p. 7

• LLaMA releases open model weights under a noncommercial license, enabling research groups to develop competing or task-specific models.Source: 2) The LLaMA Family · p. 7
• The initial LLaMA models range from 7B to 65B parameters and use GPT-3’s Transformer architecture with several architectural modifications.These include SwiGLU activation, rotary positional embeddings, and root-mean-squared layer normalization.Source: 2) The LLaMA Family · p. 7
• LLaMA-2 Chat combines pretraining, supervised fine-tuning, and iterative refinement with RLHF, rejection sampling, and proximal policy optimization.Source: 2) The LLaMA Family · p. 7
• Alpaca performs similarly to GPT-3.5 on the self-instruct evaluation set despite being much smaller, using 52K generated instruction-following demonstrations.Source: 2) The LLaMA Family · p. 7
• Vicuna-13B achieves more than 90% of ChatGPT and Bard’s quality in preliminary GPT-4 evaluation, while its training cost is $300.Source: 2) The LLaMA Family · p. 7
• Mistral-7B outperforms LLaMA-2-13B across evaluated benchmarks and exceeds LLaMA-34B in reasoning, mathematics, and code generation.It uses grouped-query attention and sliding-window attention to reduce inference cost and handle arbitrary-length sequences.Source: 2) The LLaMA Family · p. 7

• FLAN improves zero-shot performance on unseen tasks by instruction-tuning a 137B-parameter model on more than 60 verbalized NLP datasets.Source: C. Other Representative LLMs · p. 9
• Gopher achieves state-of-the-art performance across the majority of 152 diverse tasks at scales ranging to 280B parameters.Source: C. Other Representative LLMs · p. 9
• T0 maps natural-language tasks into human-readable prompted forms and trains an encoder-decoder model on a multitask mixture of NLP datasets.Source: C. Other Representative LLMs · pp. 9–10
• RETRO conditions autoregressive modeling on retrieved document chunks and matches GPT-3 and Jurassic-1 performance on the Pile with 25% fewer parameters.Source: C. Other Representative LLMs · p. 10
• GLaM uses sparsely activated mixture-of-experts layers; its 1.2T-parameter model uses one-third of GPT-3’s training energy and half its inference computation while achieving better performance across 29 NLP tasks.Source: C. Other Representative LLMs · p. 10
• Chinchilla finds that compute-optimal training scales model size and training tokens equally, then tests the finding with a 70B-parameter model.Source: C. Other Representative LLMs · p. 11
• Galactica outperforms Chinchilla on mathematical MMLU, 41.3% to 35.7%, and PaLM 540B on MATH, 20.4% versus 8.8%.Source: C. Other Representative LLMs · p. 11
• StarCoderBase is trained on one trillion permissively licensed GitHub tokens and outperforms every open multilingual Code LLM evaluated while matching or exceeding code-cushman-001.Source: C. Other Representative LLMs · p. 12

• LLM training comprises data preparation, tokenization, self-supervised pretraining, instruction tuning, and alignment.Source: §III · p. 12
• The main LLM architectures are encoder-only, decoder-only, and encoder-decoder, most commonly built from Transformer components.Source: A. Dominant LLM Architectures · p. 12
• Transformer self-attention captures long-term contextual information while enabling more parallelization than recurrence and convolution on GPUs.Source: 1) Transformer · p. 12
• Decoder-only models restrict attention to preceding words and are best suited to text generation, whereas encoder models access the full input sequence for understanding tasks.Source: 3) Decoder-Only · p. 13, 2) Encoder-Only · p. 13
• Data cleaning, filtering, and deduplication substantially affect model performance; Falcon40B used filtered web data to obtain five trillion tokens from CommonCrawl.Source: B. Data Cleaning · pp. 13, 15

• Data filtering aims to improve training-data quality and the effectiveness of trained LLMs.Source: 1) Data Filtering · p. 15
• Noise removal eliminates irrelevant or false information using classifier-based or heuristic-based quality filters.Source: 1) Data Filtering · p. 15
• Filtering also addresses outliers, class imbalance, unnecessary text elements, and ambiguous or contradictory data.Source: 1) Data Filtering · p. 15
• Deduplication removes repeated data that can bias training, reduce diversity, and promote overfitting to particular examples.Source: 1) Data Filtering · p. 15
• Document-level deduplication commonly detects duplicates through overlap ratios of high-level features such as n-grams.Source: 1) Data Filtering · p. 15
• Subword tokenizers address out-of-vocabulary problems by representing words through smaller units that can be recombined.Source: C. Tokenizations · p. 15, 1) BytePairEncoding · p. 15
• SentencePieceEncoding addresses the assumption that words are always separated by whitespace, which can fail in noisy or nonstandard text.Source: 3) SentencePieceEncoding · p. 16

Pre-training objectives

• Pre-training uses massive usually unlabeled text, with autoregressive modeling predicting subsequent tokens and masked modeling predicting corrupted tokens from context.Decoder-only models are naturally suited to autoregressive tasks.Source: E. Model Pre-training · p. 16

Architectural techniques

• Mixture of Experts replaces dense FFN layers with sparse expert networks selected by a learned router, enabling larger models or datasets under the same compute budget.Tokens may be routed to one or more experts.Source: E. Model Pre-training · p. 16, Figure 29 · p. 17

Fine-tuning

• Fine-tuning adapts foundation models to specific or multiple tasks, can reduce prompt-engineering complexity, and can expose models to new or proprietary data.Task-specific labeled data is used for supervised fine-tuning.Source: F. Fine-tuning and Instruction Tuning · p. 17

Instruction tuning

• Instruction tuning aligns responses with human expectations, and instruction-tuned models generally outperform their original foundation models on benchmarks.InstructGPT outperforms GPT-3 on most benchmarks, while Alpaca outperforms LLaMA in the cited comparisons.Source: F. Fine-tuning and Instruction Tuning · p. 17

Alignment

• Alignment methods address unintended toxic, harmful, misleading, or biased behavior through approaches including RLHF, DPO, and KTO.DPO avoids reward-model fitting and reinforcement learning, while KTO uses desirable-or-undesirable labels without paired preferences.Source: G. Alignment · p. 18
• Further alignment beyond supervised fine-tuning mainly improves models with at least 7B parameters, whereas supervised fine-tuning is sufficient for smaller models.This scope boundary is attributed to recent research cited by the survey.Source: G. Alignment · p. 18

Greedy Search

• Greedy search selects the most probable token at every step, making it fast but potentially sacrificing temporal consistency and better overall sequences.It discards alternative token choices at each step.Source: 1) Greedy Search · pp. 18–19

Beam Search

• Beam search tracks the N most likely tokens until termination, then outputs the beam with the highest overall score.For beam size 2 and maximum length 5, the cited example requires tracking 2^5 = 32 possibilities.Source: 2) Beam Search · p. 19

Top-k Sampling

• Top-k sampling randomly selects among the k most likely tokens, retaining high-probability choices while introducing randomness.With k=2, probabilities are renormalized over the two retained tokens.Source: 3) Top-k Sampling · p. 19
• Temperature modifies softmax probabilities by dividing input logits by T, controlling the influence of likely tokens and the creativity of generated text.The cited discussion describes T as ranging from 0 to 1.Source: 3) Top-k Sampling · p. 19

Top-p Sampling

• Top-p sampling chooses a probability-mass cutoff and samples from the resulting nucleus, often producing more variable, diverse, and creative outputs.Tokens are added in descending probability order until the cumulative probability exceeds p.Source: 4) Top-p Sampling · p. 19

Efficient training

• ZeRO removes memory redundancies in parallel training, improving training speed and enabling larger efficiently trained models with low communication volume.The approach scales model size proportional to the number of devices while sustaining high efficiency.Source: 1) Optimized Training · p. 19

Efficient architectures

• RWKV combines Transformer-style parallelizable training with RNN-style inference, maintaining constant computational and memory complexity during inference.It uses linear attention and can formulate the model as either a Transformer or an RNN.Source: 1) Optimized Training · p. 19, Figure 33 · p. 20

Low-Rank Adaptation (LoRA)

• LoRA freezes pretrained weights and learns two smaller matrices whose low-rank product represents the update, reducing trainable parameters and memory requirements.For W0 ∈ R^d×k, the update is represented as W0 + BA with rank r ≪ min(d,k).Source: 2) Low-Rank Adaption (LoRA) · p. 20

• LLMs lack persistent state or memory, have stale information, and cannot access data absent from their training sets without external augmentation.Source: A. LLM limitations · p. 21, §IV · p. 21
• LLMs are stochastic and may produce different responses to the same prompt, although temperature can limit variability.Source: A. LLM limitations · p. 21
• LLMs can produce plausible but untruthful answers because they lack a notion of truth and are trained on mixed-quality content.Source: A. LLM limitations · p. 21
• Hallucinations are evaluated with statistical, model-based, and human methods, while mitigation combines product design, data management, prompting, and model configuration.Source: A. LLM limitations · p. 21, B) Augmenting LLMs through external… · p. 22

B. Using LLMs: Prompt Design and Engineering

• Prompts combine instructions, questions, input data, and optional examples to guide a generative model toward a desired output.Source: B. Using LLMs: Prompt Design and… · p. 23
• Prompt engineering requires domain knowledge, model understanding, contextual tailoring, and iterative practices analogous to evaluation or hyperparameter tuning.Source: B. Using LLMs: Prompt Design and… · p. 23

1) Chain of Thought (CoT):

• Chain of Thought makes implicit reasoning explicit by directing an LLM through reasoning steps, using zero-shot instructions or manual examples.Source: 1) Chain of Thought (CoT) · p. 23
• Manual CoT is more effective than zero-shot CoT, but depends on diverse examples whose hand construction is difficult and error prone.Source: 1) Chain of Thought (CoT) · p. 23

2) Tree of Thought (ToT):

• Tree of Thought explores multiple reasoning branches, evaluates them for validity and relevance, and selects a coherent outcome for complex problems.Source: 2) Tree of Thought (ToT) · p. 23

3) Self-Consistency:

• Self-Consistency generates multiple responses and treats agreement across them as an indicator of accuracy, especially when factual precision matters.Source: 3) Self-Consistency · pp. 23–24

4) Reflection:

• Reflection prompts an LLM to evaluate and revise its own output, enabling self-editing that can improve response quality and reliability.Source: 4) Reflection · p. 24

5) Expert Prompting:

• Expert Prompting asks an LLM to simulate expert responses and can synthesize multiple expert perspectives into a comprehensive answer.Source: 5) Expert Prompting · p. 24

6) Chains:

• Chains link sequential components so each stage performs a specific function and passes its output to the next stage.Source: 6) Chains · p. 24

7) Rails:

• Rails constrain LLM outputs through predefined rules or templates that target relevance, safety, accuracy, topicality, or factuality.Source: 7) Rails · p. 24

8) Automatic Prompt Engineering (APE):

• Automatic Prompt Engineering generates, scores, refines, and iterates candidate prompts using LLMs themselves.Source: 8) Automatic Prompt Engineering (APE) · p. 24

C. Augmenting LLMs through external knowledge - RAG

• RAG retrieves relevant external information from sources such as search engines or knowledge graphs and adds it to the prompt to address stale or private knowledge.Source: C. Augmenting LLMs through external… · p. 24, Figure 37 · p. 25, Figure 38 · p. 25
• FLARE iteratively uses predicted upcoming content as retrieval queries, keeping generation dynamically informed throughout the response.Source: C. Augmenting LLMs through external… · p. 25

D. Using External Tools

• Tools generalize retrieval augmentation by allowing LLMs to call external functions or services, including databases, APIs, search engines, and calculators.Source: D. Using External Tools · p. 25
• ART combines automated chain-of-thought prompting with external tool use and task-library examples for tasks requiring reasoning and data interaction.Source: D. Using External Tools · pp. 25–26

• An LLM-based agent is an augmented LLM system that interacts with users or environments, accesses tools, and makes decisions toward an intended goal.Source: E. LLM Agents · p. 26
• Generic agents provide tool access and utilization plus decision-making based on inputs, context, available tools, and reasoning processes.Source: E. LLM Agents · p. 26
• Agents can use APIs to retrieve information or act externally, such as answering weather questions or completing purchases.Source: E. LLM Agents · p. 26
• Conversational agents may combine working memory, planning policy, action execution, and utility modules to manage dialogue and external evidence.Source: E. LLM Agents · p. 26, Figure 40 · p. 27
• ReWOO separates planning from immediate observation, while ReAct interleaves verbal reasoning with actions and DERA assigns dialogue roles across multiple agents.Source: E. LLM Agents · p. 26
• Evaluation resources span question answering and knowledge, including Natural Questions, TriviaQA, and MMLU, as well as code generation through MBPP, HumanEval, APPS, and WikiSQL.Source: §V · pp. 26–27, A. Datasets for Basic Tasks: language… · pp. 27–28

B. Datasets for Emergent: ICL, reasoning (CoT), instruction following

• Emergent-ability evaluation includes instruction following, in-context learning, chain-of-thought reasoning, and related benchmark tasks.Source: B. Datasets for Emergent: ICL,… · p. 29
• GSM8K contains 8.5K grade-school math problems requiring 2 to 8 solution steps, with 7.5K training and 1K test problems.Source: B. Datasets for Emergent: ICL,… · pp. 29–30
• MATH contains 12,500 high-school competition problems spanning seven subjects, with step-by-step solutions and difficulty ratings from 1 to 5.Source: B. Datasets for Emergent: ICL,… · p. 30
• Commonsense benchmarks include HellaSwag, ARC, PIQA, and SIQA, covering activity continuation, science questions, physical situations, and social scenarios.Source: B. Datasets for Emergent: ICL,… · p. 30
• OpenBookQA contains around 6,000 multiple-choice questions linking each question to a core fact and additional facts, requiring multi-hop reasoning.Source: B. Datasets for Emergent: ICL,… · p. 30
• TruthfulQA evaluates truthfulness with 817 questions across 38 categories, including health, law, finance, and politics.Source: B. Datasets for Emergent: ICL,… · p. 30
• OPT-IML Bench covers 2,000 NLP tasks from eight benchmarks, with 17.9M training examples, 145K development samples, and 321K test samples.Source: B. Datasets for Emergent: ICL,… · p. 30

C. Datasets for Augmented: using external knowledge/tools

• Augmented-ability datasets evaluate external knowledge, multi-hop retrieval, tool use, and tool-oriented instruction following through HotpotQA, ToolQA, and GPT4Tools.Source: C. Datasets for Augmented: using… · p. 30

A. Popular Metrics for Evaluating LLMs

• Classification tasks such as sentiment analysis can use accuracy, precision, recall, and F1.Source: A. Popular Metrics for Evaluating LLMs · p. 31
• Open-ended text generation requires evaluation methods different from categorization metrics.Source: A. Popular Metrics for Evaluating LLMs · p. 31
• Pass@k evaluates code generation by testing multiple generated solutions and measuring the probability that a correct solution appears among them.Source: A. Popular Metrics for Evaluating LLMs · p. 31
• Exact match counts a prediction as correct when it matches a reference token by token; EM equals M/N, where M is correct answers and N is questions.Source: A. Popular Metrics for Evaluating LLMs · p. 31
• Rouge and BLEU compare generated text with reference text using largely n-gram-based similarity, while BERT-Score uses another model and can be erroneous.Source: A. Popular Metrics for Evaluating LLMs · p. 32
• Purely generated content remains difficult to evaluate because available metrics capture simplistic features or depend on models with uncertain accuracy.Source: A. Popular Metrics for Evaluating LLMs · p. 32
• Benchmark comparisons are organized by task category and model size because performance depends on evaluation aspects and parameter scale.Source: A. Popular Metrics for Evaluating LLMs · p. 32

B. LLMs’ Performance on Different Tasks

• GPT-4 achieves the best reported HellaSwag result, while Davinci-003 is best on the reported OBQA results, though OBQA coverage is incomplete.Source: B. LLMs’ Performance on Different Tasks · pp. 32–33

A. Smaller and more efficient Language Models

• Larger models improve benchmark accuracy but are costly and inefficient, motivating smaller language models and parameter-efficient training methods.Source: A. Smaller and more efficient Language… · p. 35

B. New Post-attention Architectural Paradigms

• Transformers remain dominant, while State Space Models and other post-attention architectures are explored as alternatives.Source: B. New Post-attention Architectural… · p. 35
• Post-attention models address the lack of larger context windows found in traditional attention-based architectures.Source: B. New Post-attention Architectural… · p. 35

C. Mixture of Experts

• Mixture-of-Experts models train extremely large networks while activating only selected experts during inference; GLaM has 1.2T parameters and uses 2 of 64 experts.Source: B. New Post-attention Architectural… · p. 35

D. Multi-modality

• Future LLMs are expected to unify text, images, video, and audio for applications including question answering, creative work, healthcare, and robotics.Source: C. Multi-modal Models · p. 36

E. Augmentation and tool use

• Prompt engineering, tools, and augmentation techniques are expected to remain important approaches for addressing shortcomings such as hallucination.Source: D. Improved LLM Usage and Augmentation… · p. 36

F. Applications

• LLM systems are being applied to personalization and recommendations by using user preferences and past interactions as context.Source: D. Improved LLM Usage and Augmentation… · p. 36

G. LLM-based agents and multi-agent systems

• LLM-based agents and multi-agent systems combine language models with external tools and decision-making capabilities, although their development is challenging.Source: D. Improved LLM Usage and Augmentation… · p. 36

Training frameworks

• DeepSpeed, Megatron-LM, BMTrain, GPT-NeoX, and ColossalAI support distributed, parallel, or large-scale LLM training.Source: A. LLM Training/Inference Frameworks · pp. 42–43

Training and model libraries

• Transformers provides pretrained models across text, vision, and audio, reducing the resources required to train models from scratch.Source: A. LLM Training/Inference Frameworks · pp. 42–43

Parameter-efficient adaptation

• LoRA freezes original weights and learns low-rank matrices, reducing trainable parameters and storage while enabling efficient task switching.Source: A. LLM Training/Inference Frameworks · p. 43

Deployment tools

• FastChat, SkyPilot, vLLM, text-generation-inference, and OpenLLM provide tools for serving, hosting, inference, and deployment across models and clouds.Source: B. Deployment Tools · p. 43

Application frameworks

• LangChain supports context-aware and reasoning applications, while AutoGen enables multiple customizable agents to converse with one another and involve humans or tools.Source: B. Deployment Tools · p. 43

Retrieval-augmented generation

• Embedchain manages unstructured data for RAG by chunking data, generating embeddings, and storing them in a vector database.Source: B. Deployment Tools · p. 43

Prompting and evaluation tools

• Guidance constrains generation with regex or CFGs and interleaves control flow with generation, while PromptTools, PromptBench, and Promptfoo support evaluation.Source: C. Prompting Libraries · p. 43

Vector search and databases

• Faiss, Milvus, Qdrant, Weaviate, LlamaIndex, and Pinecone provide vector search, storage, or retrieval infrastructure for AI applications.Source: D. VectorDB · p. 44