PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Overview of Large Language Models

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, Ajmal Mian

arXiv:2307.06435v10cs.CL

TL;DR

LLM research spans diverse topics, making a concise, comprehensive overview useful to researchers and practitioners. This article surveys that work, synthesizing findings and design aspects across architectures, datasets, and training pipelines.

  • Problem

    LLM research spans diverse topics, creating a need for a concise, comprehensive overview for researchers and practitioners.

    Source: §1 · pp. 2–3

  • Method

    The article surveys LLM developments, summarizing pre-trained models, major findings, and design aspects across a broad range of concepts.

    Source: §1 · pp. 2–3, §8 · p. 35

  • Results

    The review synthesizes significant findings and analyzes architectures, datasets, and training pipelines, comparing zero- and few-shot performance, fine-tuning, and model families.

    Source: §8 · p. 35

  • Takeaways & Limitations

    The summaries and design discussions are intended to help practitioners effectively leverage LLMs.

    Source: §1 · p. 2

    LLMs’ black-box nature and limited insight into their operation limit their effectiveness and trustworthiness.

    Source: §7 · pp. 34–35

Abstract

from arXiv · show

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the existing literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to not only provide a systematic survey but also a quick comprehensive reference for the researchers and practitioners to draw insights from extensive informative summaries of the existing works to advance the LLM research.

1. Introduction

LLMs have grown from pretrained language models through scaling and instruction tuning into systems that generalize across tasks, while their computational demands have driven research into more efficient use. This article offers a concise, self-contained overview of LLM research, from model architectures and training to applications, evaluation, and challenges.

Introduction

  • Scaling pretrained models to tens or hundreds of billions of parameters and vast datasets helped establish LLMs, which can generalize across tasks and exhibit emergent abilities such as reasoning and in-context learning.Earlier models such as T5 and mT5 used transfer learning, while GPT-3 demonstrated zero-shot transfer without fine-tuning.Source: §1 · p. 2
  • Pretrained LLMs may fail to follow user intent and perform worse zero-shot than few-shot, while instruction tuning and human-preference alignment improve generalization and reduce misaligned behavior.The passage cites instruction-data fine-tuning [16] [17] [19] and preference alignment [21] as approaches.Source: §1 · p. 2
  • LLMs’ broad capabilities come with slow training and inference, extensive hardware needs, and high running costs, motivating research into efficient architectures, training strategies, and utilization methods.Approaches include parameter-efficient tuning, pruning, quantization, distillation, and context-length interpolation.Source: §1 · p. 2
  • This survey aims to provide a concise, comprehensive account of LLM research, including pretrained-model architecture and training details, fine-tuning, multimodal and augmented LLMs, agents, datasets, evaluation, applications, and challenges.It follows existing terminology and treats pretrained LLMs with 10B parameters or more, directing readers interested in smaller models to other surveys.Source: §1 · pp. 2–3

2. Background

Section 2 outlines LLM fundamentals, training and adaptation stages, and prompting, summarizing core concepts and directing readers to original works for further detail.

2.1. Tokenization

  • Tokenization parses text into non-decomposing units called tokens, which may be characters, subwords, symbols, or words; common schemes include wordpiece, BPE, and unigramLM.Tokenization is an essential preprocessing step in LLM training.Source: §2.1 · p. 4

2.2. Encoding Positions; 2.3. Attention in LLMs

  • Positional encodings add position information because transformer attention does not capture it, while attention weights values using query-key scores.Alibi subtracts a distance-increasing bias to favor recent tokens, and RoPE [66] rotates queries and keys according to token position. Sparse attention [67] uses sliding windows to reduce the cost of self-attention, while Flash Attention uses input tiling to reduce GPU memory transfers.Source: §2.2 · p. 4, §2.3 · p. 4

2.6. Distributed LLM Training; 2.7. Libraries

  • Distributed training partitions data, tensor computations, or model layers across devices, with combinations including model and 3D parallelism; optimizer parallelism partitions optimizer states, gradients, and parameters to reduce memory use.Libraries covered include Transformers, DeepSpeed, Megatron-LM [80], JAX, Colossal-AI [84], BMTrain, FastMoE [85], MindSpore, PyTorch, TensorFlow, and MXNet [89].Source: §2.6 · p. 4, §2.7 · p. 5

2.8. Data PreProcessing; 2.11. LLMs Scaling Laws

  • LLM training preprocessing filters data for quality, deduplicates it to limit performance effects and memorization, and removes private information such as names, addresses, and phone numbers.Scaling laws relate loss to model size, dataset size, and compute; one study recommends scaling model size and training tokens equally [96], while another suggests larger models matter more than big data [95].Source: §2.8 · p. 5, §2.11 · p. 6

2.9. Architectures

  • LLM architectures include encoder-decoder, causal decoder, prefix decoder, and Mixture-of-Experts designs, which activate only a few experts to scale model size without increasing computational cost [92].Encoder-decoder models use cross-attention, causal decoders predict from prior tokens, and prefix decoders allow bidirectional attention; MoE uses a router to direct tokens among feed-forward experts.Source: §2.9 · p. 5, Figure 4 · p. 5

2.10. Pre-Training Objectives

  • Pre-training objectives include autoregressive full language modeling, prefix language modeling, masked language modeling, and unified language modeling.Full language modeling predicts future tokens from preceding tokens; prefix training predicts only target tokens after a randomly selected prefix, while masked training predicts randomly masked tokens or spans from past and future context. Unified language modeling combines causal, non-causal, and masked objectives.Source: §2.10 · pp. 5–6, Figure 5 · p. 5

2.12.2. Fine-Tuning

  • Instruction-tuning adapts pretrained models to natural-language instructions and improves zero-shot generalization, while alignment-tuning uses human feedback to make models helpful, honest, and harmless.In RLHF [100], a reward model ranks responses using human preferences, then reinforcement learning with PPO updates the model iteratively. Transfer learning fine-tunes pretrained models on task-specific data [11].Source: §2.12.2 · pp. 6–7

2.12.3. Prompting /Utilization

  • Prompting adapts LLMs with zero-shot queries or in-context examples, while reasoning prompts include Chain-of-Thought, Self-Consistency, and Tree-of-Thought.Single-turn prompts include relevant context at once; multi-turn use passes feedback or tool outputs into later interactions, a pattern used in autonomous agents.Source: §2.12.3 · p. 7

3. Large Language Models

This section reviews large language models through their architectures, training objectives, pipelines, datasets, and fine-tuning details.

3. Large Language Models

  • The section briefly describes LLM architectures and training objectives.Source: §3 · p. 7
  • It reviews LLM training pipelines and datasets.Source: §3 · p. 7
  • It also covers fine-tuning details for LLMs.Source: §3 · p. 7

3.1. Pre-Trained LLMs

This section surveys pretrained LLMs across general-purpose and specialized domains, summarizing their architectures, training approaches, and reported capabilities. It presents them as advances that improve language-task performance and are widely fine-tuned for downstream tasks.

3.1.1. General Purpose

  • General-purpose models use varied designs, including T5’s unified text-to-text span masking, mT5’s [11] multilingual sampling, and GLM-130B’s [33] bilingual autoregressive mask infilling.CPM-2 uses knowledge inheritance to build bilingual MoE models, while DeepSeek-v2 compresses the KV cache into a latent vector through multi-head latent attention.Source: §3.1.1 · pp. 8, 10
  • GLaM (64B/64E) outperformed GPT-3 while using one-third its training energy, Chinchilla [96] outperformed Gopher [116] after fine-tuning, and Gopher’s 280B model beat rivals on 81% of evaluated tasks.Chinchilla used the same compute budget as Gopher 280B but trained on four times more data.Source: §3.1.1 · pp. 8–9
  • Gemini-1 [135] achieved human-level MMLU capability; Gemini-1.5 [136] supports a 2M context window and reasoning up to 10M tokens; DeepSeek-v2 delivers 5.76 times faster inference than DeepSeek.LLaMA-3/3.1 [130] was trained on seven times more data and double the context length of LLaMA-2, and outperformed its earlier variants and other models.Source: §3.1.1 · p. 10

3.1.2. Coding

  • Codex [141], trained on a subset of public Python Github repositories to generate code from docstrings, generated 100 versions per description by repetitive sampling, yielding working solutions that passed unit tests for 77.5% of problems.AlphaCode ranked at the top 54.3% among over 5,000 competitors, with its Codeforces rating within the top 28% of recently participated users.Source: §3.1.2 · p. 11

3.1.3. Scientific Knowledge

  • Galactica [148] is trained on a curated scientific corpus spanning papers, textbooks, compounds, proteins, and other scientific material, and uses < work > tokens to provide step-by-step reasoning context.The passage reports that this reasoning context improves performance on reasoning tasks.Source: §3.1.3 · p. 11

3.1.4. Dialog

  • LaMDA [150] is a decoder-only model pretrained primarily on English public-dialog and web data, with discriminative and generative fine-tuning intended to promote response quality, safety, and groundedness.More than 90% of its pretraining data is English; the passage describes it as usable as a general language model for various tasks.Source: §3.1.4 · p. 11

3.1.5. Finance

  • Xuan Yuan 2.0 [152] combines pretraining and fine-tuning to avoid catastrophic forgetting in a Chinese financial chat model trained on general-purpose, financial, and instruction datasets.BloombergGPT [151] is a 50B-parameter non-causal decoder trained on financial and general-purpose data.Source: §3.1.5 · p. 11

3.2. Fine-Tuned LLMs

Fine-tuning adapts pretrained LLMs to follow instructions, generate safer responses, and generalize across tasks, while the literature explores human- and model-generated instruction data, preference alignment, and continued pretraining.

3.2. Fine-Tuned LLMs

  • Fine-tuning addresses pretrained LLMs’ limited ability to follow intent and tendency toward harmful or inaccurate responses, improving task generalization with little additional compute.For PaLM 540B, instruction tuning used 0.2% of total pretraining compute; instruction tuning also improves usability and reasoning, especially with chain-of-thought data.Source: §3.2 · p. 11

3.2.1. Instruction-Tuning with Manually Created Datasets

  • Expanding task collections and prompt setups improves zero-shot and few-shot generalization, while Tk-Instruct outperformed Instruct-GPT despite using 11B rather than 175B parameters.T0 and multilingual mT0 use templates to convert existing datasets into prompts; performance also depends on dataset, instruction diversity, templates, model size, and training objectives.Source: §3.2.1 · p. 16

3.2.2. Instruction-Tuning with LLMs Generated Datasets

  • Self-instruct outperformed SUPER-NATURALINSTRUCTIONS-trained models by 33%, and complex instruction evolution yielded WizardLM results above Vicuna and Alpaca.Self-instruct iteratively generated 52k instructions and 82k input-output pairs from 175 seed tasks; WizardCoder also outperformed Claude-Plus and Bard.Source: §3.2.2 · p. 16

3.2.3. Aligning with Human Preferences

  • Preference alignment combines supervised demonstrations, reward modeling, and reinforcement learning, while direct preference tuning and synthetic feedback offer alternatives to complex RLHF pipelines.InstructGPT uses PPO on reward-model outputs; DPO [168] learns from preferred versus unpreferred responses, and Constitutional AI [173] uses AI feedback. Prompt-based self-correction can also reduce harmful outputs.Source: §3.2.3 · pp. 16–17

3.2.4. Continue Pre-Training

  • Concatenating randomly selected pretraining examples with fine-tuning data can prevent catastrophic forgetting, while smaller, selected datasets can retain or improve performance.Using 25% of downstream data was sufficient for state-of-the-art results; a 0.5% coreset improved performance by 2%, and LIMA achieved performance comparable to GPT-4 with 1,000 demonstrations.Source: §3.2.4 · p. 17, §3.2.5 · p. 17

3.3. Increasing Context Window

Limited training context windows can prevent generalization to unseen lengths, while longer contexts support longer documents, more in-context samples, and bigger reasoning processes. The section surveys positional interpolation, efficient attention, and extrapolation without training.

3.3. Increasing Context Window

  • Limited-length training often fails to generalize to unseen lengths; ALiBi extrapolates zero-shot but has lower expressivity and benchmark performance, while RoPE generally cannot [186] [49] [66] [46].Longer context windows can improve document understanding, in-context learning, and reasoning, but expanding them through fine-tuning is slow and computationally expensive [49].Source: §3.3 · p. 17
  • Position interpolation within pretrained limits works better than extrapolation; 1,000 fine-tuning steps extend windows without degrading original-context performance, while Giraffe [46] uses power scaling and YaRN [47] uses NTK-aware interpolation [49].Source: §3.3 · p. 17
  • Efficient attention reduces dense-attention costs through LongT5’s TGlobal [48], COLT5’s two-branch routing [187], LongNet’s dilated attention [188], and LongLoRA’s shift-short attention [189].LongT5 pre-trains on 4,098 tokens and fine-tunes up to 16k; LongLoRA uses dense attention at inference and achieves performance similar to full-attention fine-tuning [48] [189].Source: §3.3 · p. 18
  • LM-Infinite [186] and parallel context windows enable length extrapolation without training: the former uses Λ-shaped attention within original limits, and the latter chunks inputs to pretrained lengths with reused positional encodings.Source: §3.3 · p. 18

3.4. Augmented LLMs

Augmented LLMs access external storage programmatically to extend model context, address limitations such as outdated information, and support task completion through retrieval or tool use. The section surveys retrieval and tool augmentation, including zero-shot and trained approaches.

3.4. Augmented LLMs

  • Augmented LLMs access external memory programmatically, complementing in-context examples with long- or short-term, symbolic or non-symbolic memory in documents, vectors, or databases.Memory may retain intermediate representations across iterations or extract important dataset information for recall; read and write operations may involve LLM cooperation or feedback.Source: §3.4 · p. 18

3.4.1. Retrieval Augmented LLMs

  • Retrieval augmentation can make an 11B model competitive with 540B PaLM [25] and a 7.5B model competitive with 280B Gopher.RALM uses a retriever to supply context to a language model; inaccurate retrieval can steer the model toward false responses.Source: §3.4.1 · p. 18, Figure 12 · p. 18
  • RETRO shows pre-training smaller LLMs with RAG can outperform larger LLMs trained without RAG, while RAG fine-tuning alone lacks those pre-training gains [205].RETRO retrieves relevant database chunks for query subsets and encodes them with intermediate representations for token generation, using cross-chunked attention autoregressively.Source: §3.4.1 · pp. 18–19
  • RALM approaches train retrievers alone [206] [207] [208] [209] or jointly with LLMs [25] [210], and use encoded contexts or web search to extend retrieval [210] [25] [213] [214] [166].Retriever training includes contrastive learning and supervision from frozen LLM outputs; RPT [210] predicts document chunks, while Fusion-in-Decoder fuses encoded context through cross-attention.Source: §3.4.1 · p. 19

3.4.2. Tool Augmented LLMs

  • Zero-shot tool augmentation uses LLM reasoning to plan subtasks and select tools; ART retrieves task demonstrations [217], while tool documentation alone can teach tool use [218].RestGPT [219] decomposes tasks into planning and API selection, and ToolkenGPT [220] represents tools as tokens that trigger execution and restart generation with the output.Source: §3.4.2 · p. 19
  • Training with tool augmentation improves planning beyond zero-shot approaches, using API-documentation instruction tuning [221], self-play fine-tuning [27], and generated tool-use datasets [223].ToolLLM [223] draws on 16k RapidAPI APIs and uses a depth-first search decision tree to generate diverse reasoning and planning ground truths.Source: §3.4.2 · p. 19
  • Multimodal tool augmentation plans a sequence of tool selection, execution, inspection, and response generation, using modality-rich tools for multimodal queries [215] [216] [224] [226].Systems may use multimodal LLMs [31] [225] [224] [216] or single-modality LLMs to plan the use of tools across modalities.Source: §3.4.2 · pp. 19–20, Figure 13 · p. 19

3.5. LLMs-Powered Agents

LLM-powered agents use LLMs as cognitive controllers to plan, decide, act, and adapt to dynamic scenarios, enabling applications from web and coding to embodied and conversational agents with minimal or no fine-tuning [224] [216].

Planning and Reasoning

  • Prompting with chain-of-thoughts, tree-of-thoughts, and self-consistency supports agent reasoning; RAP [233] models future outcomes, while Retroformer [234] adds retrospective planning cues.Given a task description and action sequence, LLMs can also generate plan actions without fine-tuning.Source: §3.5 · p. 20

Feedback

  • Closed-loop agents use environmental responses to reassess and update plans [236] [195], while LLM reward functions can train RL policies instead of humans [238].Source: §3.5 · p. 20

Memory

  • Reflexion [195] uses episodic memory of previous responses to improve decisions, while Retroformer [234] prompts with recent responses and summaries of failed attempts.Retroformer’s short-term memory holds recent responses and its long-term memory stores summarized failed attempts as reflection.Source: §3.5 · p. 20

LLMs in Physical Environment

  • SayCan combines an LLM-generated high-level plan with a learned affordance function trained by RL, while PaLM-E grounds tasks using multimodal sensor inputs.LLMs need adaptation for physically grounded tasks because limited real-world knowledge can produce illogical responses in specific physical situations [26].Source: §3.5 · p. 20

Multi-Agents Systems

  • Multi-agent systems assign LLMs unique user-defined roles so domain-expert agents can simulate human behavior and collaborate on complex tasks [229] [239].Source: §3.5 · p. 20

Manipulation

  • For manipulation, LLMs use visual and spatial information to improve robot dexterity and adaptability in object recognition, grasping, and collaboration [236] [241].Source: §3.5 · p. 20

Navigation

  • For navigation, LLMs generate feasible robot paths and trajectories while accounting for environmental details, supporting precise, adaptable movement across varied settings [242] [243] [244] [246].The passage names warehouses, transport, healthcare facilities, and residences as example environments.Source: §3.5 · p. 20

3.6. Efficient LLMs

Because deploying LLMs is expensive, this section surveys methods to reduce their running costs while preserving performance. It covers parameter-efficient fine-tuning, quantization, and pruning.

3.6.1. Parameter E

  • PEFT reduces fine-tuning costs, performing better than full fine-tuning in low-resource setups, comparably with medium resources, and worse when high resources are available [40] [247] [41] [38] [39] [248].Approaches include adapters and LoRA, prompt and prefix tuning, and bias-only BitFit; LoRA fuses learned weights for inference, while prompt tuning trains only 0.001%-3% additional parameters [250] [247] [40] [254].Source: §3.6.1 · pp. 20–21

3.6.2. Quantization

  • Post-training quantization methods address LLM outliers and performance loss through outlier-aware precision, activation smoothing, or layerwise weight updates [255] [256] [259].Other methods use token-level distillation and module-specific scaling [45], or shift and scale per-channel activations [257]. Quantization-aware training fine-tunes quantized models; Alpha Tuning and PEQA train scaling parameters, while LLM-QAT uses generated data and distillation [260] [261] [262] [264].Source: §3.6.2 · pp. 21–22

3.6.3. Pruning

  • Pruning compresses models to reduce deployment costs; structured methods remove parameter groups for faster inference, while unstructured methods remove weights without preserving structure [265] [42] [266].Wanda prunes using weight and input-norm importance without fine-tuning [265]; OWL varies layerwise pruning ratios [267], and LLM-Pruner combines group selection with LoRA fine-tuning [42].Source: §3.6.3 · p. 22

3.7. Multimodal LLMs

Multimodal LLMs extend text-only models to process multiple modalities, and the section organizes their construction into pre-training, fine-tuning, and prompting. It also describes their use in visual reasoning.

Pre-training

  • Pre-training integrates vision and language in unified models, using cross-attention, staged alignment, or a trained projection layer.Flamingo uses gated cross-attention; BLIP-2 [270] uses a two-stage Q-Former strategy; MiniGPT-4 [277] trains only a linear projection layer over frozen components.Source: §3.7 · p. 22

Fine-tuning

  • Fine-tuning multimodal instructions extends pretrained LLMs into chatbots and task solvers, but depends on instruction data and ways to bridge modalities.Data strategies include benchmark adaptation [279] [281] [282], self-instruction [19] [31] [283], and hybrid composition [284] [280]; learnable adapters include LLaMA-Adapter [285] and LaVIN [284], while VideoChat-Text [272] uses Whisper to convert video speech into captions.Source: §3.7 · p. 22

Prompting

  • Prompting supports multimodal tasks without updating model parameters, including Chain of Thought reasoning and tool-assisted task decomposition.Multimodal-CoT [287] separates rationale generation from answer inference, while CoT-PT [288] uses prompt tuning and visual bias for implicit reasoning; multimodal descriptions and tools can divide complex tasks into subtasks [289] [290].Source: §3.7 · p. 23

Visual Reasoning Application

  • LLMs advance visual reasoning through stronger generalization, emergent ability, and interactivity, serving as controllers, decision makers, or semantic refiners.PointClip V2 [292] generates 3D-specific prompts for recognition, and GPT4Tools [31] fine-tunes LLMs with LoRA [250] on tool-related instructions.Source: §3.7 · p. 23

3.8. Summary and Discussion

The discussion surveys architectural and training choices for LLMs while emphasizing tradeoffs and unresolved comparisons. It also contrasts supervised and generalized models, zero-shot and few-shot performance, and model architectures.

3.8.1. Architecture

  • Architectural choices affect performance and stability, but positional-encoding findings conflict and sparse modules require further verification.Pre-norm is common for stability, though GLM-130B [33] reports fine-tuning benefits from deep-norm; BLOOM [13] and GLM-130B [33] reach opposing conclusions on ALiBi versus rotary encodings. Parallel attention reduces training time by 15%, multi-query attention saves memory and speeds sampling, and MoE can scale capacity efficiently [15] [92].Source: §3.8.1 · p. 23

3.8.2. Training Strategies

  • Training strategies address stability, efficiency, and controllability through precision choices, recovery methods, initialization, learning-rate schedules, parallelism, and task-related tokens.BF16 offers better stability than FP16 but uses more memory and has limited hardware support; divergence can be addressed by restarting, skipping batches, or reshuffling [15] [33]. Small initialization, warmup and decay, and 3D or 5D parallelism support training; mode-switching tokens aid downstream performance, while in-context learning, adversarial ranking, and soft prompts support controlled generation [124] [122] [35].Source: §3.8.2 · pp. 23–24

3.8.3. Supervised Models vs Generalized Models

  • Supervised models still outperform generalized models by a large margin on various NLP tasks.The passage reports this comparison as the current state of the art [15].Source: §3.8.3 · p. 24

3.8.5. Encoder vs Decoder vs Encoder-Decoder

  • LLMs commonly use decoder-only or encoder-decoder architectures, with evidence supporting strengths for both and no settled winner.Decoder-only models achieve gains on NLU and NLG, while T5 and UL2 report encoder-decoder models outperforming them; larger decoder-only models can narrow the gap [15]. Mode-switching tokens and task-dependent objectives also enable flexible encoder-decoder configurations [125, 122, 34].Source: §3.8.5 · p. 24

4. Model Configurations

This section summarizes pre-trained and instruction-tuned LLM configurations, including model statistics, architecture, and optimization settings. Instruction-tuned models share their pre-trained baselines’ architectural details, while some optimization details are omitted when papers do not provide them.

4. Model Configurations

  • Architectural details for pre-trained LLMs appear in Table 5; instruction-tuned models use the same architecture as their pre-trained baselines.The section therefore does not separately provide architectural details for instruction-tuned models.Source: §4 · p. 24
  • Optimization settings for LLMs appear in Tables 6 and 7, but Table 7 omits precision, warmup, and weight decay because these details are considered less important and papers do not provide them.The section also reports publication venue, license type, model creators, training steps, and parallelism among its configuration statistics.Source: §4 · p. 24
  • Tables 3 and 4 summarize pre-trained and instruction-tuned models above 10B parameters, with training-data entries defined as tokens or data size and, for instruction-tuned models, sample counts marked “S-”.Table 3 also defines data-cleaning, training-parallelism, cost, and commercial-use conventions; its cost estimates may differ from actual costs.Source: Table 3 · p. 25, Table 4 · p. 25

5. Datasets and Evaluation

This section surveys widely used training datasets and evaluation benchmarks, emphasizing the importance of training-data quality, size, and diversity and organizing evaluation around language understanding and generation. It describes benchmarks spanning task-specific abilities such as reasoning, reading comprehension, cross-lingual understanding, truthfulness, and bias.

5.1. Training Datasets

  • Training-data quality, size, and diversity shape LLM performance, while preparing high-quality datasets at scale is laborious.The survey summarizes widely used pre-training and fine-tuning datasets in Table 8.Source: §5.1 · p. 25

5.2. Evaluation Datasets and Tasks

  • LLM evaluation covers natural language understanding and generation, with tasks that are softly categorized and often used interchangeably.The survey describes datasets for tasks including question answering, reasoning, summarization, translation, and dialogue, and lists evaluation datasets and benchmarks for pre-trained and fine-tuned models.Source: §5.2 · p. 26

5.2.1. Multi-task

  • Multi-task benchmarks assess language knowledge and capabilities across varied tasks, from subject-based problem solving to reasoning, creativity, and language understanding.MMLU [307] covers 57 subjects in zero-shot and few-shot settings; SuperGLUE tests tasks including question answering and inference, while BIG-bench [308] spans reasoning, creativity, ethics, and domain understanding, and GLUE [309] evaluates language understanding.Source: §5.2.1 · pp. 26–27

5.2.2. Language Understanding

  • Language and contextual-understanding datasets probe abilities such as conversational question answering, word-sense disambiguation, semantic matching, and long-form text modeling.CoQA uses conversation history and free-form answers across seven domains; WiC [317] tests contextual word meaning, LCQMC evaluates Chinese question matching, and PG19 [319] supports long-form language modeling.Source: §5.2.2 · pp. 27–28

5.2.3. Story Cloze and Sentence Completion

  • Story and sentence-completion benchmarks test coherent story understanding and generation as well as contextual next-word prediction.StoryCloze [334] evaluates story understanding, generation, and script learning; LAMBADA [335] asks models to predict a passage’s last word using context beyond its final sentence.Source: §5.2.3 · p. 28

5.2.4. Physical Knowledge and World Understanding; 5.2.5. Contextual Language Understanding

  • Physical-knowledge and contextual-comprehension benchmarks test real-world knowledge, science question answering, long-passage comprehension, and information-seeking dialogue.ARC [342] includes easy and challenging grade-school science questions, RACE [347] uses long English exam passages, and QuAC [348] simulates student–teacher information-seeking dialogue over hidden Wikipedia text.Source: §5.2.4 · p. 28, §5.2.5 · p. 29

5.2.6. Commonsense Reasoning; 5.2.7. Reading Comprehension

  • Commonsense and reading-comprehension datasets assess causal reasoning, context completion, answerability, and understanding of information in passages.HellaSwag [355] uses adversarially filtered context-ending choices, COPA [401] tests cause-and-effect choices, SQuADv2 [364] includes over 50,000 unanswerable questions, and DROP [365] evaluates reading phenomena.Source: §5.2.6 · p. 29, §5.2.7 · p. 31

5.2.8. Mathematical Reasoning; 5.2.9. Problem Solving and Logical Reasoning

  • Mathematical and logical-reasoning benchmarks range from arithmetic and multi-step problem solving to adversarial inference and reasoning over multiple evidence pieces.Math23k contains 23,000 Chinese arithmetic word problems; GSM8K [384] tests multi-step grade-school math, ANLI [393] uses iterative adversarial examples, and StrategyQA requires reasoning over multiple pieces of evidence.Source: §5.2.8 · p. 31, §5.2.9 · p. 31

6. Applications

LLMs are being applied as general-purpose tools and across fields including medicine, education, science, mathematics, law, finance, and robotics, with uses shaped by each domain’s needs and by the data on which they are trained.

General Purpose

  • LLMs support general-purpose tasks from translation and question-answering to programming help, personal assistance, customer service, content generation, and text-data analysis [434] [435].Their outputs can adapt to the style and tone of the text they process, but LLMs are only as good as their training data.Source: §6 · p. 32

Medicine

  • In medicine, LLMs assist with clinical recommendations, patient interactions, literature review, medical education, and public-health initiatives [441] [444].Their use raises concerns about privacy, explainability, and bias.Source: §6 · p. 32

Education

  • In education, LLMs can personalize student materials, support teachers with planning and grading, aid language practice, and improve accessibility for students with disabilities.Accessibility support includes transcription, reading assistance, and simplifying complex texts.Source: §6 · p. 32

Science

  • For science, LLMs summarize literature, help researchers generate hypotheses and questions, and assist with drafting and improving scientific writing [457] [458].Accessible summaries can help researchers follow findings beyond their own fields.Source: §6 · pp. 32–33

Maths

  • In mathematics, LLMs explain problem-solving steps, help detect and correct reasoning or computation errors, preliminarily check proofs, and translate concepts for non-specialists [461] [462] [463] [464].They can support proof verification but are not a substitute for mathematicians’ meticulous work.Source: §6 · p. 33

Law

  • In law, LLMs analyze and classify legal texts, generate more accurate explanations when supplied with relevant case law, and support legal reasoning and question-answering [466] [467] [468] [469].The reported analysis of theft court opinions involved collaboration between legal experts and LLMs.Source: §6 · p. 33

Finance

  • In finance, BloombergGPT [151] uses proprietary financial datasets, while open-source FinGPT [470] supports accessible customization; LLMs can also break complex financial tasks into actionable plans.The overview associates BloombergGPT with custom datasets and FinGPT with a data-centric approach and low-rank adaptation.Source: §6 · p. 33

Robotics

  • In robotics, LLMs support human-robot interaction, planning, navigation, manipulation, and personalization, while helping robots integrate information, learn skills, adapt, and refine paths [28] [472] [246] [236] [476] [224] [233] [234].They can help robots understand their environment and generate collaborative task plans [26].Source: §6 · p. 33

7. Challenges and Future Directions

LLMs face technical, social, and foundational challenges as their capabilities and use expand. These include costs and reliability limits, as well as unresolved issues in adaptation, privacy, real-time use, and responsible governance.

Computational Cost; Economic and Research Inequality

  • Training costs and energy use are high, with diminishing performance gains as compute increases at fixed model and dataset sizes [477]; expense may concentrate development in well-funded organizations [481].The latter could worsen economic and research inequalities in AI.Source: §7 · p. 34

Bias and Fairness; Overfitting

  • Training-data bias can raise fairness concerns, while overfitting noisy patterns can produce illogical responses; models must balance memorization with generalization to unseen inputs [480].Excessive memorization can make models inflexible on new inputs.Source: §7 · p. 34

Reasoning and Planning; Hallucinations

  • LLMs struggle with common-sense reasoning and planning [482], while hallucinations [483] may conflict with user input, earlier context, or established world knowledge.The passage distinguishes input-conflicting, context-conflicting, and fact-conflicting hallucinations.Source: §7 · p. 34

Prompt Engineering; Limited Knowledge; Safety and Controllability; Security and Privacy

  • Prompt variations can substantially change outputs [484]; stale pretraining knowledge makes updating costly, and RAG requires adapting pipelines not trained for retrieval [198] [21] [25].Safe use also requires defenses against backdoors, jailbreaking, prompt injection, and data poisoning [485] [486].Source: §7 · p. 34

Multi-Modality; Catastrophic Forgetting; Adversarial Robustness

  • Multimodal LLMs face data-alignment, fusion, and compute challenges; domain fine-tuning risks catastrophic forgetting, and adversarial robustness measures can compromise generalization.The text calls for adversarial evaluation tools, particularly for safety-critical uses [488] [489].Source: §7 · p. 34

Interpretability and Explainability; Privacy Concerns; Real-Time Processing; Long-Term Dependencies; Hardware Acceleration; Regulatory and Ethical Frameworks

  • Black-box behavior limits trust and explainability [490] [491], while memorization, hardware limits, and regulatory gaps raise privacy, latency, and accountability concerns [493] [494] [497].Models can expose private training data; their depth and size impede edge-device inference, and auditing and proactive ethical frameworks are proposed for responsible deployment.Source: §7 · p. 35

8. Conclusion

The article surveys LLM developments, covering design and training alongside comparative analyses and frontier topics. It aims to serve as a resource for understanding advances and developing better LLMs.

8. Conclusion

  • The review summarizes LLM architectures, datasets, training pipelines, and strategies used by different models.It presents these design aspects as summaries and discussions throughout the article.Source: §8 · p. 35
  • It compares zero-shot and few-shot performance, fine-tuning impacts, supervised and generalized models, and encoder, decoder, and encoder-decoder architectures.Source: §8 · p. 35
  • The review also covers multimodal and retrieval-augmented LLMs, agents, efficiency, datasets, evaluation, applications, and challenges.It is intended as a resource offering insights into recent advances and fundamental concepts for developing better LLMs.Source: §8 · p. 35

Related papers

Loading 2307.06435v10…