PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundations of Large Language Models

Tong Xiao, Jingbo Zhu

arXiv:2501.09223v4cs.CLcs.AIcs.LG

TL;DR

Reasoning systems face credit-assignment and exposure-bias problems, while long-context reasoning requires maintaining logical continuity. The text describes masked-token training, preference-based policy optimization, and inference-memory techniques; it reports that scaled pre-training can improve BERT without architecture changes.

  • Problem

    Reasoning training faces credit-assignment and exposure-bias problems, while long-context reasoning depends on preserving logical continuity.

    Source: §6.4.2 · p. 272, §6.4.3 · p. 276, §6.5 · p. 280

  • Method

    The text describes masked-token pre-training, preference-based policy optimization, and paged KV-cache management for inference.

    Source: §1.2.2.1 · p. 19, §4.4.2 · p. 202, §5.2.2.3 · p. 238

  • Results

    Scaling pre-training can improve BERT without architecture changes, and removing NSP does not degrade downstream performance when training is scaled up.

    Source: §1.3.2 · p. 36

  • Takeaways & Limitations

    Larger base models and longer context windows can raise the reasoning-performance limit, while algorithmic improvements mainly use the available compute budget more effectively.

    Source: §6.4.4.1 · p. 279

    Extensive SFT can overfit, reducing generalization or causing catastrophic forgetting; regularization, early stopping, or smaller learning rates can mitigate this risk.

    Source: §4.2.1 · p. 170

Abstract

from arXiv · show

This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technologies. The book is structured into six main chapters, each exploring a key area: pre-training, generative models, prompting, alignment, inference, and reasoning. It is intended for college students, professionals, and practitioners in natural language processing and related fields, and can serve as a reference for anyone interested in large language models.

Notation

This section introduces notation and foundational concepts for pre-training and adapting large language models, covering model architectures, training objectives, and approaches such as fine-tuning and prompting. It also surveys BERT and decoder-only models, scaling and multilingual transfer, downstream tasks, and alignment.

Notation

  • max f(a) is the maximum value of f(a), arg max_a f(a) is the maximizing a, and Pr(a|b), Pr(·|b), and E_a∼Pr(a)[f(a)] denote conditional probabilities, distributions, and expectations.Source: Notation · p. 5
  • h_t and H denote a sequential model’s hidden state and its time-wide state matrix, while Q, K, V, Softmax(A), and D denote attention matrices, normalization, and the training or fine-tuning dataset.Source: Notation · p. 5
  • ∂θ denotes the gradient of loss L with respect to parameters θ, and DKL(p || q) denotes the KL divergence between distributions p and q.Source: Notation · p. 5

1.1.1 Unsupervised, Supervised and Self-supervised Pre-training

  • Pre-training optimizes a model before downstream adaptation, using tasks that support generalization across tasks and can reduce reliance on task-specific labeled data.The chapter traces pre-training from unsupervised and supervised approaches to self-supervised learning, which derives training signals directly from unlabeled data, such as predicting masked tokens, and is the focus of its discussion.Source: §1.1.1 · p. 11, §1.1 · p. 12, §1.2 · p. 16

1.1.2 Adapting Pre-trained Models

  • Sequence encoding models produce vector representations, whereas sequence generation models produce tokens from context; pre-training outputs may be vocabulary distributions or sequence representations.The chapter focuses on Transformer-based self-supervised pre-training, including decoder-only, encoder-only, and encoder-decoder architectures.Source: §1.1 · p. 11, §1.1.2 · p. 12, §1.2 · p. 16

1.1.2.1 Fine-tuning of Pre-trained Models

  • Fine-tuning adapts pre-trained encoders by adding a task-specific classifier and optimizing its parameters with labeled data, either alongside the encoder or with the encoder frozen.Because fine-tuning generally uses much less labeled data than pre-training, it is less computationally expensive and efficient in practice.Source: §1.1.2.1 · p. 13, §1.1 · p. 14

1.1.2.2 Prompting of Pre-trained Models

  • Large language models trained to predict the next token can adapt to tasks through text prompts, including instructions without additional training and demonstrations for in-context learning.The chapter notes that tuning may still be needed for instruction following and alignment with human values.Source: §1.1.2.2 · p. 14, prompting · p. 15

1.2.1 Decoder-only Pre-training

  • A decoder-only language model predicts the next-token distribution from preceding context and is trained by minimizing sequence-level cross-entropy, mathematically equivalent to maximum likelihood estimation.The target at each position is a one-hot representation of the correct next token, and sequence loss sums position-wise losses.Source: §1.2.1 · p. 16

1.2.2 Encoder-only Pre-training

  • An encoder maps an input sequence to vectors, and a Softmax output layer supplies self-supervision; unlike autoregressive language modeling, it observes the entire sequence.For downstream application, the pre-training Softmax layer is removed and the encoder is paired with a task-specific prediction network, generally fine-tuned on labeled data.Source: §1.2.2 · p. 17, Figure 1.2 · p. 18

1.2.2.1 Masked Language Modeling

  • Masked language modeling predicts masked tokens from unmasked context and maximizes reconstruction probabilities only at masked positions.Because the encoder sees unmasked tokens unchanged, they need not be predicted; the pretrained encoder can then be fine-tuned or applied to downstream tasks.Source: §1.2.2.1 · pp. 18–19

1.2.2.2 Permuted Language Modeling

  • Permuted language modeling predicts tokens in a chosen order, enabling broader contexts while preserving original positions through self-attention masks.It addresses the training–inference discrepancy from [MASK] tokens and the independence of masked-token predictions in masked language modeling.Source: §1.2.2.2 · pp. 19–20

1.2.2.3 Pre-training Encoders as Classifiers

  • Encoder pre-training can use sentence-level NSP or token-level ELECTRA classification, with ELECTRA’s generator proposing replacements for a discriminator to label.NSP distinguishes consecutive sentence pairs from randomly paired sentences; ELECTRA combines generator maximum-likelihood and discriminator classification losses, then discards the generator.Source: §1.2.2.3 · p. 22

1.2.3 Encoder-Decoder Pre-training

  • Encoder-decoder models support a single text-to-text system: self-supervised pre-training builds general language knowledge, followed by fine-tuning on targeted task data.Tasks such as sentiment description can treat text as both the input and output.Source: §1.2.3 · p. 23

1.2.3.1 Masked Encoder-Decoder Pre-training

  • In T5 [Raffel et al., 2020], a training sample separates a source containing the task instruction and input from a target containing the response.The source–target separator is →.Source: §1.2.3.1 · p. 24
  • Task instructions and inputs are provided as text, allowing one model to perform many tasks simultaneously.The text-based instruction framework can acquire instruction-following capabilities during pre-training and facilitate zero-shot generalization to problems with unseen task-specific instructions.Source: §1.2.3.1 · pp. 24–25
  • Encoder-decoder models can use prefix language modeling or masked-token reconstruction, learning to encode context and generate continuations or restore corrupted sequences.In prefix language modeling, the encoder processes the prefix and the decoder predicts subsequent tokens; denoising training instead reconstructs the original sequence from corrupted input.Source: §1.2.3.1 · pp. 25–26
  • Sentinel tokens such as [X] and [Y] can mark masked spans for reconstruction, shortening training sequences and allowing masking rates and span lengths to vary.Raffel et al. [2020]’s notation represents consecutive masked tokens as spans; the model fills the placeholder slots using surrounding context.Source: §1.2.3.1 · p. 26

1.2.3.2 Denoising Training

  • Denoising corruption can mask or delete tokens, mask spans, reorder sentences, or rotate documents; Lewis et al. [2020] propose these options in BART.Span masking can include length-0 spans, requiring the model to predict how many tokens a span generates; corruption choices require careful experimentation.Source: §1.2.3.2 · pp. 28–29

1.2.4 Comparison of Pre-training Tasks

  • Pre-training tasks are best compared by objective: autoregressive next-token prediction, masked-token prediction, permuted-factorization prediction, classifier-based discrimination, or corrupted-sequence reconstruction.The chapter notes that objectives can apply across architectures, so architecture alone is insufficient for categorizing pre-training tasks.Source: §1.2.4 · p. 29

1.3 Example: BERT

  • BERT is a widely used pre-trained sequence-encoding model.The standard BERT model, introduced by Devlin et al. [2019], is a bidirectional Transformer encoder trained with masked language modeling and next sentence prediction; its loss sums the two task losses.Source: §1.3 · p. 31, §1.3.1 · p. 31

1.3.1 The Standard Model

  • BERT training minimizes the combined loss over text-corpus mini-batches using gradient descent or its variants until a stopping criterion is met.Stopping may occur when training loss converges or when a predefined maximum number of steps is reached.Source: §1.3.1 · p. 31

1.3.1.1 Loss Functions

  • BERT uses masked language modeling and next sentence prediction: 15% of tokens are selected, then masked 80% of the time, randomly replaced 10%, or left unchanged 10%.Masked-token prediction learns from surrounding context, while leaving some tokens unchanged helps reduce the mismatch between pre-training with [MASK] and fine-tuning without it; NSP classifies sentence pairs as IsNext or NotNext.Source: §1.3.1.1 · pp. 31–32

1.3.1.2 Model Setup

  • BERT sums token, positional, and segment embeddings before a post-norm Transformer encoder, with common BERTbase and BERTlarge configurations using 110M and 340M parameters, respectively.Training can begin on shorter sequences for many steps and continue on full-length sequences for the remaining steps.Source: §1.3.1.2 · pp. 32–35

1.3.2 More Training and Larger Models

  • RoBERTa shows that scaling training data and compute can improve BERT without architectural changes, and that removing NSP need not hurt downstream performance when training is scaled up [Liu et al., 2019].These findings motivate continuing pre-training on simple objectives at larger scale.Source: §1.3.2 · pp. 35–36

1.3.3 More Efficient Models

  • Efficient BERT variants use knowledge distillation, pruning or quantization, adaptive layer or token processing, and parameter sharing; removing some attention heads can speed inference without significantly degrading performance [Michel et al., 2019].Distillation can match teacher and student hidden-layer outputs, while shared layer parameters reduce model size and test-time memory use [Dehghani et al., 2018].Source: §1.3.3 · pp. 36–37

1.3.4 Multilingual Models

  • Multilingual BERT uses a shared, larger vocabulary to represent tokens from 104 languages in a common space, while XLM’s bilingual masked-token training uses aligned sentence pairs to learn cross-lingual correspondences [Lample and Conneau, 2019].Predicting a masked token can draw on its parallel-language context, supporting cross-lingual transfer; multilingual models can also handle code-switching.Source: §1.3.4 · pp. 37–38, Figure 1.7 · p. 38
  • Low-resource languages benefit from cross-lingual transfer, but fixed-capacity multilingual models can degrade on high-resource languages as language coverage grows.Multilingual models treat tokens from different languages as entries in a shared vocabulary; larger language coverage calls for larger models and vocabularies, creating a practical trade-off with model capacity.Source: §1.3.4 · pp. 38–39

1.4 Applying BERT Models

  • BERT adapts pre-trained parameters to downstream tasks by adding a prediction network and fine-tuning; applications include single-text or text-pair classification and regression.Single-text classification typically uses the [CLS] output vector; pair classification concatenates the texts, while regression changes the output to a real-valued score.Source: §1.4 · pp. 39–41
  • BERT also supports sequence labeling, span prediction, and encoder-decoder tasks by supplying contextual representations to token-level, span-boundary, or source-text prediction components.Examples include named entity recognition and reading comprehension; BERT can initialize the encoder for sequence-to-sequence tasks such as machine translation, summarization, question answering, and dialogue generation.Source: §1.4 · pp. 41–43
  • Task-specific BERT fine-tuning can overfit and cause catastrophic forgetting, harming generalization or earlier-task performance; mixing old data or specialized methods can mitigate it.The passage identifies experience replay and elastic weight consolidation as specialized methods.Source: §1.4 · p. 43

Generative Models

  • A well-trained LLM can perform a wide range of tasks and generalize to new tasks with minimal adaptation, supporting its role as a foundation model [Bubeck et al., 2023].The chapter uses LLM to refer here to generative models such as GPT.Source: Generative Models · p. 46

2.1 A Brief Introduction to LLMs

  • Language modeling estimates sequence probability by predicting each token from its preceding context, enabling autoregressive, left-to-right text generation.Neural language models output a probability distribution over the vocabulary given the preceding tokens, then append a predicted token to the context for the next prediction.Source: §2.1 · pp. 46–47

2.1.1 Decoder-only Transformers

  • Decoder-only Transformers represent tokens with token and positional embeddings, then process them through stacked self-attention and feed-forward sub-layers, using post-norm or pre-norm architecture.The input and output rows can be interpreted as contextual representations of the sequence’s tokens.Source: §2.1.1 · pp. 47–48
  • Decoder-only Transformers use masked self-attention to predict each token from prior context, with stacked blocks and a Softmax output supporting autoregressive generation.The architecture uses L stacked Transformer blocks, each with self-attention and an FFN sub-layer.Source: §2.1.1 · p. 48, Figure 2.1 · p. 49

2.1.2 Training LLMs

  • LLMs are trained by maximum likelihood, while scaling introduces distributed-systems, compute-cost, and optimization-stability challenges that can require architectural changes.Training models with tens of billions of parameters from scratch generally requires hundreds or thousands of GPUs.Source: §2.1.2 · pp. 49–50

2.1.3 Fine-tuning and Applying LLMs

  • Instruction fine-tuning with a relatively small amount of labeled data can activate pre-trained abilities and support zero-shot handling of tasks not explicitly trained for.The section distinguishes this from earlier models such as BERT, which are primarily fine-tuned for specific tasks.Source: §2.1.3 · pp. 52, 54

2.1.4 Aligning LLMs with the World

  • Alignment guides LLMs toward human values and responsible behavior through supervised fine-tuning and human feedback, including reinforcement learning from human feedback.Human feedback evaluates model responses against preferences and safety guidelines; human values and expectations can be diverse and shifting.Source: §2.1.4 · pp. 55–56

Pre-training &

This section surveys core techniques and challenges across large language model training and use, from pre-training at scale and resource planning to in-context prompting, reasoning demonstrations, and preference-based policy optimization.

Pre-training &

  • RLHF builds an initial policy, gathers human rankings of its outputs, trains a reward model, and fine-tunes the policy with reward-model supervision.The overview presents these as four steps, with reward-model learning and policy learning as distinct phases.Source: Figure 2.2 · p. 57, Pre-training & · p. 58
  • Human rankings train a smaller, target-architecture reward model with pairwise ranking loss; its scores provide continuous supervision for policy fine-tuning.The model scores the concatenated input and output, and the ranking objective penalizes disagreement with human-labeled preferences.Source: Pre-training & · p. 59
  • RLHF suits settings where desired behavior is hard to demonstrate but easier to recognize, and reinforcement learning can explore beyond annotated samples.Preference annotations are presented as easier than specifying values or producing aligned outputs directly.Source: Pre-training & · p. 60

2.1.5 Prompting LLMs

  • Prompts mean the full model input and can customize tasks without tuning, including role-based answers, code debugging, and dialogue that uses conversation history.Users can include instructions and conversational context in prompts, making prompt design a means of adapting a developed LLM efficiently.Source: §2.1.5 · pp. 60–61
  • In-context learning adds worked examples to prompts, but the illustrated arithmetic problem remains wrong (12 instead of 7), showing demonstrations alone may not suffice.The text links such errors to the multi-step logical inference required by arithmetic and commonsense reasoning.Source: §2.1.5 · p. 62
  • Chain-of-thought prompting [Wei et al., 2022c] demonstrates intermediate steps to mimic multi-step inference, and the example reaches 17; it is useful on challenging tasks such as GSM8K.The worked example structures the marble problem into successive additions and subtraction before producing the answer.Source: §2.1.5 · pp. 62–63
  • Basic prompting ranges from zero-shot instructions to one- and few-shot demonstrations, while zero-shot CoT uses trigger phrases such as “Let’s think step by step”.The examples cover arithmetic reasoning, sentiment classification, and phrase translation; the section omits advanced prompting techniques and refers readers to Chapter 3 for more discussion.Source: §2.1.5 · pp. 63–65

2.2.1 Data Preparation

  • Training data must be filtered and cleaned, balanced for demographic and cultural representation, and protected against privacy leaks, though removing all sensitive information is difficult.Web-scraped corpora can contain errors, toxic content, and fabricated facts; one reported heuristic pipeline discarded 90% of its scraped data.Source: §2.2.1 · pp. 66–67
  • Data diversity supports broader downstream capabilities: code can aid programming and complex reasoning, while multilingual performance depends on language-specific data quality and volume and may be poor for low-resource languages.Source: §2.2.1 · p. 66

2.2.2 Model Modifications

  • Pre-norm layer normalization is widely used to stabilize deep Transformer training, while RMS normalization rescales without re-centering and modern FFNs use alternatives such as GeLU and gated activations.Large FFN hidden sizes make activation choice more consequential; PaLM and Llama use SwiGLU, while the Llama series uses RMS layer normalization.Source: §2.2.2.1 · p. 67, §2.2.2.2 · pp. 68–69

2.2.2.3 Removing Bias Terms

  • Removing bias terms from layer normalization, QKV projections, and FFN sub-layers has been reported to improve training stability and is used in Llama and Gemma.Source: §2.2.2.3 · p. 69

2.2.2.4 Other Issues

  • Stable, efficient large-scale training also depends on carefully designed learning-rate schedules, optimizers, parallel strategies, and mixed-precision methods, often requiring multiple training runs.Increasing batch size during training has also been useful for some LLMs.Source: §2.2.2.4 · p. 69

2.2.3 Distributed Training

  • Data parallelism divides minibatches across workers and can approach N-fold speedup under ideal coordination and low communication overhead, whereas naive model parallelism assigns layer groups sequentially and leaves devices idle.Model parallelism is often combined with other methods to improve device utilization.Source: §2.2.3 · pp. 70–71
  • Tensor parallelism splits large matrix operations across devices, while pipeline parallelism overlaps micro-batch computation; small micro-batches can reduce GPU utilization and system throughput.Source: §2.2.3 · pp. 71–72
  • Distributed training aims for linear efficiency gains, but communication, synchronization, stragglers, failures, and system design can erode speedups; asynchronous training risks stale gradients and uncertain convergence.Practical scaling also depends on load balancing, memory bandwidth, and overlapping data transfer with computation.Source: §2.2.3 · p. 72
  • Mixed-precision training can improve efficiency, but gradient accumulation across nodes may cause numerical differences, convergence effects, overflow, underflow, and hardware inconsistencies.The passage describes using low precision for gradient computation and higher precision for model updates.Source: §2.2.3 · p. 72

2.2.4 Scaling Laws

  • Scaling laws relate LLM performance to model size, computation, and training data; learning curves can progress from slow improvement to a power-law gain and then slow again, with irreducible error.Hestness et al. [2017] illustrate this pattern for test error as training data increases.Source: §2.2.4 · pp. 72–73, Figure 2.3 · p. 73
  • At larger scales, more data can still strengthen models, and some abilities emerge beyond certain model sizes; scaling-law estimates may differ across downstream tasks.Fine-tuning and prompting can influence downstream performance, so lower test loss does not guarantee better results on every task.Source: §2.2.4 · pp. 73, 75
  • There is no universal scaling function: empirical power laws model loss against model size or data, often with irreducible error, but monotonic forms miss curves such as double descent.The text identifies more sophisticated fits as a response to learning curves with inflection points.Source: §2.2.4 · pp. 73–75
  • Scaling laws guide researchers on whether more training resources are promising and help estimate an LLM’s eventual performance given available compute.The text notes that scaling up can remain effective even after trillions of tokens have been used.Source: §2.2.4 · pp. 73, 75

2.3 Long Sequence Modeling

This section surveys methods for enabling long-context LLMs, including computational and architectural optimizations, memory and KV-cache strategies, positional mechanisms, and adaptation to task-specific needs. It also covers trade-offs in context coverage, memory and computation, continuous processing, and the challenges of evaluating long-context ability.

2.3.1 Hardware and Parallel Computing Optimizations

  • Reduced-precision formats and hardware-aware attention implementations improve efficiency, while sequence parallelism shards keys and values across compute nodes and combines partial results collectively [Kim et al., 2023; Dao et al., 2022; Kwon et al., 2023; Li et al., 2023b; Korthikanti et al., 2023].Examples include FP16/BF16 or INT8 instead of FP32, IO-aware attention, and collective reductions such as all-reduce.Source: §2.3.1 · pp. 76–77

2.3.2 Efficient Architectures

  • Standard self-attention has quadratic time and memory costs, while autoregressive KV caches grow with context length, making very long-context deployment impractical.The section therefore considers efficient Transformer variants and alternatives, as well as auxiliary memory that compresses context.Source: §2.3.2 · p. 78, §2.3.3 · p. 80
  • Sparse attention computes only selected weights, motivated by skewed attention distributions, but still requires retaining the full KV cache during inference.A common heuristic selects a window of nearby tokens; this reduces computation but does not remove the memory burden of very long caches.Source: §2.3.2 · pp. 78–79
  • Linear attention replaces Softmax normalization with a linear formulation and recurrent history summaries, avoiding explicit storage of past keys and values and enabling constant-cost steps.Only the latest summary representations are used at each step.Source: §2.3.2 · p. 79
  • Recurrent models update a fixed-size state as tokens arrive, and modern state space models such as Mamba have renewed interest in recurrent alternatives to Transformers [Gu and Dao, 2023].Their fixed-size state supports streaming inputs and real-time predictions.Source: §2.3.2 · pp. 79–80

2.3.3 Cache and Memory

  • An auxiliary memory can encode and compress context as an alternative to retaining the entire KV cache, whose footprint grows linearly during inference.Fixed-size memory makes the attention cost independent of sequence position.Source: §2.3.3 · p. 80, §2.3.3.1 · p. 80

2.3.3.1 Fixed-size KV Cache

  • Fixed-size key-value memory may retain a window of previous keys and values as local attention, summarize prior vectors with moving averages, or use a neural network updated from prior memory and current states.Source: §2.3.3.1 · pp. 80–82
  • The compressive Transformer uses separate fixed-size memories for local context and compressed long-term history, compressing ejected key-value pairs rather than discarding them.This design assumes local context needs relatively explicit representation, while long-range context can be more compressed.Source: §2.3.3.1 · p. 83
  • Global tokens can stabilize attention on very long sequences by acting as an attention sink, but fixed-size global memory may lose information and enlarging the KV cache raises computational cost.The approach commonly combines the first few sequence tokens as global tokens with other sparse-attention methods [Gupta and Berant, 2020]; the attention-sink effect is described in.Source: §2.3.3.1 · pp. 83–84

2.3.3.2 Memory-based Models

  • Memory-based models retrieve similar context-state key-value pairs from a datastore and combine them with local attention, or use retrieved hidden states and next tokens to inform output predictions.In k-NN language modeling, retrieved next-token references are combined with the LLM’s parametric distribution using a tunable interpolation weight.Source: §2.3.3.2 · pp. 84, 86
  • Retrieval-augmented generation retrieves relevant document chunks through information retrieval and adds them to the input, augmenting an LLM without changing its architecture.An indexed datastore can make retrieval efficient when the context spans sequences in a training dataset, though retrieval over many sequences can otherwise be computationally costly.Source: §2.3.3.2 · pp. 86–87

2.3.3.3 Memory Capacity

  • Longer contexts make salient information harder for low-capacity memories to retain, while high-capacity or external memories can be costly and unsuitable for dynamic streaming.Fixed-size memories may lose information; scaling capacity raises costs. Retrieval-based external memory can avoid Transformer context-length limits, but datastore construction and maintenance add overhead and are typically designed for static contexts.Source: §2.3.3.3 · p. 87

2.3.4 Sharing across Heads and Layers

  • Multi-query attention shares keys and values across heads, while grouped-query attention shares them within groups, reducing KV-cache size and allowing a trade-off between efficiency and expressiveness.Grouped-query attention ranges from standard multi-head attention when ng = τ to multi-query attention when ng = 1; sharing KV activations or attention weights across layers can further reduce computation and memory footprints [Brandon et al., 2024].Source: §2.3.4 · pp. 89–91

2.3.5 Position Extrapolation and Interpolation

  • Learned positional embeddings lack representations for positions beyond the training limit; extrapolation extends a learned position function, while interpolation maps longer ranges into the trained range.Sinusoidal encodings extend to arbitrary lengths but do not perform well when sequences are significantly extended.Source: §2.3.5 · pp. 91–92

2.3.5.1 Attention with Learnable Biases

  • T5 bias groups query-key offsets into buckets, sharing learned biases across distances; exact buckets cover short offsets, wider buckets cover larger ones, and the final bucket handles arbitrarily large distances.This sharing is presented as supporting generalization to long sequences and helping control positional-embedding overfitting.Source: §2.3.5.1 · pp. 93–95, Figure 2.10 · p. 94

2.3.5.2 Attention with Fixed Biases

  • ALiBi [Press et al., 2022] uses a fixed negative offset-scaled bias as a linear distance penalty, with geometrically decreasing head-specific scales reported to work across tasks without tuning.Because its biases are predefined rather than learned, the method can be applied to sequences once those biases are set.Source: §2.3.5.2 · p. 95

2.3.5.3 Rotary Positional Embedding

  • Rotary positional embeddings rotate token vectors by position while preserving magnitude, making query-key dot products depend on relative offsets and implicitly modeling relative positional context.The rotation-based formulation extends from two dimensions to multi-dimensional embeddings.Source: §2.3.5.3 · pp. 96, 97, 99

2.3.5.4 Position Interpolation

  • Position interpolation compresses positions [0, m] into the trained range [0, ml] through linear RoPE scaling [Chen et al., 2023c], while NTK-aware base scaling adjusts periods non-uniformly and helps extend context [Ding et al., 2024].Source: §2.3.5.4 · pp. 100–101

2.3.6.1 Need for Long Context

  • Long-context research seeks continuous processing of incoming tokens, using fixed-size recurrent memory or continuous-space attention, and asks whether compact models can represent infinite context and whether all context tokens help predict the next token.Continuous-space attention is described as removing dependence on context length.Source: §2.3.6.1 · p. 101
  • Long-context needs vary by task: summarization distills key aspects, retrieval requires retaining context; research examines in-context compression [Deletang et al., 2024] and multi-step token prediction [Pal et al., 2023; Wu et al., 2024].Source: §2.3.6.1 · p. 102

2.3.6.2 Pre-training or Adapting LLMs?

  • Long-sequence training is computationally prohibitive at scale; fine-tuning longer sequences may improve length extrapolation, while architecture changes can require training from scratch.Pretrained models can still initialize fine-tuning for external-memory or sparse-attention architectures; RAG models are also fine-tuned to use retrieval-augmented inputs.Source: §2.3.6.2 · p. 102

2.3.6.3 Evaluating Long-context LLMs

  • Perplexity tends to reflect local rather than global context; synthetic needle/passkey retrieval and copy tasks test specific long-context capabilities.Needle-in-a-haystack and passkey retrieval tasks [Mohtashami and Jaggi, 2024; Chen et al., 2023c] test retrieval of relevant details; copy tasks were proposed for recurrent models and adopted for LLM evaluation [Gu and Dao, 2023].Source: §2.3.6.3 · p. 103
  • Long-document or multi-document summarization, question answering, and code completion evaluate LLMs on very long NLP inputs and can align evaluations with user expectations.Source: §2.3.6.3 · p. 103
  • There is no general evaluation method [Liu et al., 2024c]: tests may miss whole-context understanding, use preliminary small-scale data, and vary with prompts and experimental setups.Models also remain limited by restricted context length and high latency.Source: §2.3.6.3 · p. 103

2.4 Summary

This chapter gives a foundational, non-comprehensive introduction to LLMs, including pre-training scale and approaches for handling long inputs. The prompting section introduces prompt design, refinements, and automation.

2.4 Summary

  • The chapter surveys large-scale pre-training and methods for adapting LLMs to long inputs, including optimized attention, compressed KV caches, memory models, and positional embeddings.It presents these topics as foundations for later, more advanced discussions.Source: §2.4 · p. 103
  • LLMs learn through next-token prediction, and scaling training, model size, and context size is associated with general-purpose language understanding, generation, and reasoning.Simply increasing model size has not proved sufficient for AGI; inference-time compute has also improved complex reasoning tasks.Source: §2.4 · p. 104
  • The chapter is not a comprehensive literature review because LLM research is evolving rapidly.The text points readers to general reviews [Zhao et al., 2023; Minaee et al., 2024] and focused discussions.Source: §2.4 · p. 104

Prompting

  • Prompting means giving an LLM an input or cue to elicit a desired output or task, and the section covers prompt designs, refinements, and automated design.Its example frames English-to-Chinese translation as a prompt.Source: Prompting · pp. 105–106
  • Prompt design can guide LLM responses toward users’ needs, and iterative refinement of prompts is the focus of the active field of prompt engineering.The text describes prompting as influencing response accuracy, relevance, and contextual appropriateness.Source: Prompting · p. 105
  • In-context learning uses information such as demonstrations in the prompt to adapt an LLM to a task without training or tuning.The chapter illustrates this with sentiment classification and describes the approach as enabling users to adapt context directly.Source: Prompting · p. 105

3.1 General Prompt Design

This section introduces general prompt-design principles and formatting strategies for guiding LLMs in NLP tasks, using illustrative examples rather than prescribing standard prompts. It also shows how few-shot demonstrations can teach question-answering approaches and output formats, while emphasizing the empirical nature of prompting and the role of pretrained abilities.

3.1.1 Basics

  • Templates use text with fillable variables, while clear descriptions, task attributes, roles, and context can specify what the LLM should do.The section defines a prompt as input text x conditioning output y through Pr(y|x), and describes system prompts as context or constraints.Source: §3.1.1 · pp. 106–108

3.1.2 In-context Learning

  • In-context learning uses prompt demonstrations to help LLMs solve new problems without updating model parameters, activating pretraining knowledge for rapid task adaptation.Zero-shot prompting uses no demonstrations; one-shot and few-shot prompts add one or more examples for the model to follow.Source: §3.1.2 · pp. 108–109
  • In-context learning depends on prompt quality and pretrained abilities; if an LLM lacks relevant knowledge, such as Inuktitut, more prompting may not help and continued training may be needed.The text notes that stronger LLMs make better use of in-context learning, while prompt engineering can require substantial trial and error.Source: §3.1.2 · p. 110

3.1 Prompt Engineering Strategies

  • Prompt design is empirical: many prompts can address the same task, so trial and error is often needed, and prompts vary across LLMs and tasks.The section therefore offers guiding principles rather than a comprehensive prompt list for specific models and tasks.Source: §3.1 · p. 106, §3.1.3 · p. 110

3.1.3 Prompt Engineering Strategies

  • Clear, specific instructions help align outputs with expectations, including the requested subject, audience, style, and length.The climate-change examples contrast a broad request with a detailed version tailored to a 10-year-old and capped at 500 words.Source: §3.1.3 · pp. 110–111
  • Prompts can activate LLM reasoning by requesting stepwise work; Kojima et al. [2022] found that adding “Let’s think step by step” improved performance on several reasoning tasks.Another approach uses multiple turns: request an initial solution, then ask the model to evaluate it and solve again if needed.Source: §3.1.3 · pp. 111–112
  • Providing relevant reference text, including retrieved context in RAG, can constrain answers to useful information and support better outputs.When context is highly reliable, prompts can explicitly require answers using only that information.Source: §3.1.3 · p. 113
  • LLM performance is sensitive to prompt format, and even a small prompt modification can substantially change the output.Source: §3.1.3 · p. 113
  • Clear prompt formatting can use separate fields, code-style structure, control characters or XML tags, and explicit instructions for structuring input and output.Delimited sections, such as quoted input text, can help represent complex data and reduce ambiguity.Source: §3.1.3 · p. 114

3.1.4 More Examples

  • The prompt examples are simple illustrations of how LLMs can address NLP problems, not standard prompts for those tasks.Source: §3.1.4 · p. 114

3.1.4.1 Text Classification

  • Classification prompts may elicit descriptive text that requires label mapping, or constrain outputs through cloze completion and instructions to choose from predefined labels.For cloze prompts, the model can select the highest-probability word from the label set; detailed task descriptions or examples may help with unfamiliar classification tasks, while many-category problems remain challenging.Source: §3.1.4.1 · pp. 115–116

3.1.4.2 Information Extraction

  • Information-extraction examples cover named-entity and relation extraction, while a general prompt template uses a task description that can be made more detailed for greater accuracy.The text also lists keyword extraction, key-event extraction, and coreference and anaphoric-link detection as possible task instructions.Source: §3.1.4.2 · pp. 117–118

3.1.4.3 Text Generation

  • Text generation includes continuing an initial text and producing text to meet specified requirements for style, tone, length, or content, as well as generating code from natural-language instructions.The code-generation example assumes a model pre-trained or fine-tuned on both natural language and programming languages.Source: §3.1.4.3 · pp. 119–120

3.1.4.4 Text Transformation

  • Text transformation examples illustrate prompts for machine translation, summarization, and text style transfer.The passage notes that these prompts are not significantly different from the previously presented ones.Source: §3.1.4.4 · p. 120

3.1.4.5 Question Answering

  • Question-answering prompts can address general questions and benchmark tasks, including MMLU multiple-choice questions and GSM8K problems solved in natural language.The cited benchmarks are MMLU [Hendrycks et al., 2021a] and GSM8K [Cobbe et al., 2021].Source: §3.1.4.5 · pp. 121–122
  • In the apple example, Boris ends with 90 apples and Beck with 87, leaving Beck with 3 fewer apples.Source: §3.1.4.5 · p. 122
  • Few-shot problem-solution demonstrations teach the LLM both problem solving and output formatting.Source: §3.1.4.5 · p. 123
  • The format places the final calculation result after #### and uses ≪· · · ≫ to annotate calculation steps.Source: §3.1.4.5 · p. 123

3.2 Advanced Prompting Methods

This section surveys advanced prompting methods for improving LLM problem solving, including explicit reasoning, task decomposition, flexible reasoning paths, feedback and deliberation, and combining prompts or outputs. It also covers retrieval-augmented generation and tool use for external information or computation, along with considerations such as prompt diversity, retrieval quality, and output evaluation.

3.2.1 Chain of Thought

  • Chain-of-thought prompting elicits step-by-step reasoning, helping LLMs break complex problems into smaller steps and expose intermediate reasoning for multi-step tasks.Detailed reasoning demonstrations can encourage intermediate steps that lead to an answer.Source: §3.2.1 · pp. 123–124
  • Few-shot CoT supplies demonstrations of detailed reasoning, while zero-shot CoT uses instructions such as “Let’s think step by step”; a second prompt can extract an answer if needed.The follow-up prompt can combine the original input with the first output so the model continues its reasoning and generates a conclusion.Source: §3.2.1 · pp. 124–125
  • Few-shot CoT can be difficult because detailed demonstrations are hard to obtain, no standard method defines reasoning steps, and intermediate errors can affect final accuracy..Source: §3.2.1 · p. 126

3.2.2 Problem Decomposition

  • Problem decomposition breaks complex tasks into sub-problems; for long documents, segments can be processed in parallel before a prompt combines their relevance into a final result.This can reduce computational cost compared with processing the entire document at once.Source: §3.2.2 · pp. 127–128
  • For reasoning tasks, decomposition is distinct from direct prediction and single-run chain-of-thought: it addresses sub-problems in separate LLM runs or with other systems.The section focuses on this third approach, while noting that the alternatives are also ways to perform multi-step reasoning.Source: §3.2.2 · p. 128
  • A general decomposition framework generates sub-problems and solves them; a fixed two-step outline-and-fill approach is simple but assumes compositional tasks such as writing or code generation.Sub-problems may be solved sequentially or in another order, with their results combined into the final output.Source: §3.2.2 · p. 128
  • Because complex reasoning paths can depend on intermediate results, least-to-most prompting generates a progressive sequence of sub-problems and solves them using prior question-answer pairs as context.The method prompts an LLM to generate sub-problems, then answers them sequentially before solving the original problem with those answers.Source: §3.2.2 · pp. 128–130
  • Problem decomposition can incorporate CoT without changing its framework, or generate and solve sub-problems step by step using prior reasoning history, allowing models to adapt their paths.The dynamic approach does not fix the reasoning path in advance.Source: §3.2.2 · pp. 130–131
  • Sub-problem solvers can use systems such as IR or calculators, recursively decompose complex sub-problems into hierarchies, or be selected by an agent modeled as a reinforcement-learning controller.The chapter leaves the precise reinforcement-learning task undescribed.Source: §3.2.2 · pp. 131–132
  • Problem decomposition connects to multi-hop question answering and compositionality, including work on generating simpler questions and evaluating compositional generalization [Talmor and Berant, 2018].Source: §3.2.2 · p. 132
  • Tool use decomposes tasks between LLMs and external tools, which can retrieve current data through APIs or provide other capabilities.The passage points to Section 3.2.5 for further discussion.Source: §3.2.2 · p. 132

3.2.3 Self-refinement

  • Self-refinement uses an LLM to predict an output, collect feedback, and refine it; repeated feedback and refinement can be done without extra training or enhanced with supervised learning [Ganguli et al., 2023].Feedback may come from manual review, a feedback model, or a reward model; the passage also frames self-refinement improvement as an alignment issue.Source: §3.2.3 · pp. 134, 136
  • In the response example, feedback asks for specific issues, pollutants, and affected regions, and refinement adds concrete examples such as rising temperatures, urban pollution, Amazon deforestation, and coral bleaching.Source: §3.2.3 · pp. 135–136
  • DTG prompts the model to identify translation error types before revising in one run [Li et al., 2023a]; random incorrect translations can instead prompt contrastive deliberation using “negative evidence”, but iterative refinement risks error propagation and difficult stopping decisions.DTG assumes the model can identify errors accurately, while the random-translation variant avoids relying on feedback; iterative approaches can improve progressively through repeated adjustments.Source: §3.2.3 · pp. 136–138

3.2.4 Ensembling

  • Prompt ensembling runs one LLM with multiple prompts for the same task and combines their predictions to produce a final output.Source: §3.2.4 · p. 138
  • Prompt ensembling combines predictions from multiple prompts, often using diverse prompts to capture a broader range of responses and reduce prompt-specific biases.Combination can use voting, overlap, or token-level probability scores; Bayesian marginalization over prompts expresses uncertainty but may require Monte Carlo approximation because direct computation can be infeasible.Source: §3.2.4 · pp. 138–139
  • Prompt diversity can come from manually or automatically creating demonstrations, reordering examples, generating similar prompts, or transforming prompts such as by translation.These approaches can be combined; benefits may be modest for stronger, more robust LLMs whose outputs vary little across similar prompts, but diversity matters for novel or complex problems.Source: §3.2.4 · p. 139
  • Output ensembling samples multiple candidates from a fixed prompt, including completed beam-search hypotheses, then scores or re-ranks candidates by criteria such as agreement.Self-consistency is an output- rather than prompt-ensembling method; other scoring options include rescoring with a stronger model.Source: §3.2.4 · pp. 140–141
  • In self-consistency, the most frequent answer across sampled chain-of-thought paths is selected; in the coin example, consensus chooses 37.5% over an incorrect 33.3%.Self-consistency selects for agreement rather than necessarily choosing the highest-probability prediction.Source: §3.2.4 · pp. 140–141
  • Ensembling, self-refinement, and related methods fit a predict-then-verify framework; learned verifiers can score entire reasoning paths or individual steps.The text distinguishes outcome-based scoring of a full path [Cobbe et al., 2021] from process-based scoring of each step [Uesato et al., 2022].Source: §3.2.4 · pp. 141–142

3.2.5 RAG and Tool Use

  • RAG retrieves relevant texts from an external collection and supplies them with the query to an LLM, supporting factual and up-to-date answers.The book describes retrieval systems as available off the shelf and focuses on prompting the LLM with retrieved context.Source: §3.2.5 · pp. 142–143
  • RAG can fail when retrieved texts are irrelevant or incorrect, so prompts should support faithful answers or abstention when context is insufficient.Retrieval accuracy can be improved, and fine-tuning on human-labeled refusal data is another option; use cases differ in how strictly responses should depend on retrieved context.Source: §3.2.5 · pp. 144–145
  • With tool use, an LLM can call external APIs during inference; returned results re-enter its context, unlike RAG texts supplied before prediction.Examples include web search and a calculator, with an external system executing generated tool commands.Source: §3.2.5 · pp. 145–146
  • Because LLMs lack tool-call markers, tool use requires annotated examples replacing relevant outputs with commands, followed by fine-tuning to generate executable calls.At inference, commands in the model output are executed to obtain assistance from external tools.Source: §3.2.5 · p. 146

3.3 Learning to Prompt

This section examines automated prompt optimization and learned prompt representations as ways to ease prompt design and adapt prompts to tasks. It also covers methods for simplifying prompts and reducing their length, including limitations of soft prompts.

3.3 Learning to Prompt

  • Manual prompts can be difficult and costly to design, may not generalize across models, and can be verbose enough to increase LLM computation.Human expertise can also limit the diversity of approaches and overlook prompts that are not immediately obvious.Source: §3.3 · pp. 146–147

3.3.1 Prompt Optimization

  • Prompt optimization treats prompt design as discrete search, with a search space, performance estimation, and a strategy for exploring candidates.This framing is related to automated machine learning and neural architecture search.Source: §3.3.1 · p. 147
  • LLM-based optimization can iteratively evaluate candidate prompts, retain promising ones, and generate new candidates until a stopping criterion is met.Initial prompts can be written by people or generated by LLMs from task descriptions or input-output examples.Source: §3.3.1 · pp. 148–149
  • Prompt expansion can use paraphrasing [Jiang et al., 2020], token edits, feedback and revision, or evolutionary computation.Evaluation and pruning can filter low-quality candidates during search.Source: §3.3.1 · p. 149
  • When off-the-shelf LLMs generate errors during search, reinforcement learning can train a prompt generator; Deng et al. [2022] update an FFN adaptor while keeping the rest of the LLM fixed.Generated prompts are tested using another LLM to obtain the training reward.Source: §3.3.1 · pp. 149–150

3.3.2 Soft Prompts

  • Soft prompts are learnable continuous hidden-space vectors rather than human-readable text, and can provide dense task guidance at lower computational cost than long hard prompts.They are useful when the same prompt is repeatedly used during inference.Source: §3.3.2 · pp. 150–151

3.3.2.1 Adapting LLMs to Simplified Prompts

  • Fine-tuning can enable simpler instructions, while context distillation trains a student to follow simplified context from a teacher [Snell et al., 2022]; overly simplified instructions can harm generalization.The risk is greater when labeled data cannot cover many instruction variations.Source: §3.3.2.1 · pp. 152–153, Figure 3.4 · p. 153

3.3.2.2 Learning Soft Prompts for Parameter-efficient Fine-Tuning

  • Prefix fine-tuning prepends trainable vectors to each Transformer layer, adapting task behavior by updating prefixes rather than the full model.This is a parameter-efficient alternative to full fine-tuning.Source: §3.3.2.2 · p. 154

3.3.2.2 Learning Soft Prompts for Parameter-efficient Fine-tuning

  • Prefix fine-tuning optimizes trainable vectors while freezing the Transformer, enabling task behavior without explicit hard prompts and using far fewer parameters than the LLM.The optimized prefixes are prepended at inference; their parameter count is L × n × d.Source: §3.3.2.2 · p. 155
  • Prompt tuning adds trainable pseudo-embeddings only at the input embedding layer, preserving the pretrained LLM parameters while conditioning its task behavior.These embeddings need not correspond to natural-language tokens and are trained on task-specific data to interact with token embeddings.Source: §3.3.2.2 · p. 155
  • Soft prompts can be combined with hard prompts in different embedding-sequence patterns, while adaptor layers and task-fine-tuned model parts can also be viewed as soft prompts.A sequence model such as a Transformer can encode a soft-prompt sequence; Liu et al. [2023b] discuss combining soft and hard prompts, and Lialin et al. [2023] frame parameter-efficient fine-tuning broadly as soft-prompt learning.Source: §3.3.2.2 · pp. 156–157

3.3.2.3 Learning Soft Prompts with Compression

  • Soft prompts can compress context by distilling a full-context model’s predictions into real-valued prompt embeddings, using maximum likelihood or KL divergence objectives.The teacher and student may differ in architecture or settings; typically, the teacher is stronger and the student smaller and more efficient. Mu et al. [2024] describe distilling prompting knowledge into pseudo tokens.Source: §3.3.2.2 · p. 156, §3.3.2.3 · p. 158
  • For long sequences, Chevalier et al. process segments sequentially, updating a fixed-size soft-prompt memory with summary-token representations of prior context and each segment.The memory behaves recurrently, represents the full context after the final segment, and requires fine-tuning the Transformer for this representation task.Source: §3.3.2.3 · p. 159

3.3.3 Prompt Length Reduction

  • Soft prompts are generally not directly interpretable, and changing them often requires further fine-tuning, limiting their usefulness when prompts change frequently.Simplified textual prompts remain interpretable and can be edited directly by users.Source: §3.3.3 · p. 160
  • Prompt shortening can use heuristics to remove low-value words, supervised sequence-to-sequence simplification, or LLMs prompted to simplify text within length constraints.The section frames prompt simplification as a standard NLP text-simplification task, with methods applicable beyond prompts.Source: §3.3.3 · pp. 160–161

3.4 Summary

The chapter surveys prompt design and automation, from earlier hand-crafted NLP features to techniques for guiding LLMs. It then introduces alignment as a distinct step for addressing instruction-following and safety concerns that pre-training alone cannot resolve.

3.4 Summary

  • Prompting covers designing and refining prompts for effective problem-solving, automating prompt design and representation, and techniques such as CoT and prompt learning.Source: §3.4 · p. 161
  • Prompting extends from earlier hand-crafted NLP features and templates to adding words or sentences that let pre-trained models handle tasks without changing architecture or pre-training.Source: §3.4 · p. 161
  • Prompt engineering includes hand-crafted and automatically generated prompts, with few-shot, zero-shot, and CoT approaches used across varied scenarios.Source: §3.4 · p. 161
  • Prompting outcomes depend heavily on LLM quality and size: simple prompts may suffice for stronger models, while smaller models may need careful prompt design or fine-tuning.Source: §3.4 · pp. 161–162

Alignment

  • LLM alignment targets outputs that meet human expectations, including following user instructions and avoiding harmful or biased content.Source: Alignment · p. 163
  • Pre-training alone is insufficient for alignment because available data cannot encompass all tasks or adequately represent human preferences.Source: Alignment · p. 163
  • Instruction alignment uses supervised fine-tuning, while human preference alignment typically uses reinforcement learning from human feedback; the approaches are commonly combined.Source: Alignment · p. 163

4.1 Overview of LLM Alignment

This section surveys methods for adapting pretrained LLMs to user instructions, intents, and preferences, including instruction and reward-based fine-tuning, inference-time methods, and fine-tuning data strategies. It also covers approaches that combine small or specialized models or defer uncertain inputs to larger models, along with limits such as dataset coverage and weak-supervisor errors.

4.1 Overview of LLM Alignment

  • Alignment typically proceeds from SFT to RLHF and then inference-time prompting, while prompting or output rescoring can also align models without further training.SFT uses labeled input-output examples; reward models trained on human preferences guide reinforcement learning when human values are difficult to specify directly.Source: §4.1 · pp. 163–164

4.2 Instruction Alignment

  • Instruction alignment fine-tunes pretrained models to follow user instructions because next-token training alone may lead them to continue text rather than perform the requested task.The section identifies instruction following, instruction-data construction, and generalization as topics for the discussion.Source: §4.2 · p. 165

4.2.1 Supervised Fine-tuning

  • SFT maximizes the likelihood of desired outputs conditioned on inputs, computing loss on response tokens while masking input tokens, including in multi-turn conversations.Conversational SFT can process the whole dialogue in one sequence and compute loss only for the model’s responses given conversational history.Source: §4.2.1 · pp. 165, 167, 168, Figure 4.2 · p. 167, Figure 4.3 · p. 169
  • SFT requires carefully curated labeled data and substantial compute, while extensive fine-tuning can cause overfitting or catastrophic forgetting.Regularization, early stopping, smaller learning rates, and diverse training data can mitigate forgetting; parameter-efficient methods can reduce resource demands.Source: §4.2.1 · pp. 169–170

4.2.2 Fine-tuning Data Construction

  • Fine-tuning datasets can be constructed manually or automatically, with manual approaches using human-created prompts and examples, existing task resources, or crowdsourced user queries.Task-derived prompts require annotation guidelines and quality control; existing QA websites can provide large, diverse examples, while crowdsourcing can capture real user behavior and new problem types.Source: §4.2.2 · p. 170, §4.2.2.1 · pp. 170–172

4.2.2.1 Manually Generated Data

  • Manual data construction can use prompt templates filled with paired examples, and greater diversity in prompts and tasks may improve robustness and generalization.The text notes that human-generated data quality and diversity depend on annotator experience and creativity.Source: §4.2.2.1 · pp. 170–172

4.2.2.2 Automatically Generated Data

  • Human-generated fine-tuning data can vary in quality and diversity, so automatic generation offers an alternative for producing large-scale samples.A simple approach collects questions through crowdsourcing and uses a tuned LLM to generate answers for fine-tuning.Source: §4.2.2.2 · p. 172
  • Self-instruct grows a pool of instruction-sample pairs by prompting an LLM with selected pool instructions, then filtering generated samples before adding them [Wang et al., 2023d; Honovich et al., 2023].The pool begins with hand-crafted examples, and the process can be repeated; relying on human-crafted or collected inputs can limit coverage of user requests beyond existing datasets.Source: §4.2.2.2 · pp. 172, 174, Figure 4.4 · p. 173

4.2.3 Fine-tuning with Less Data

  • A carefully crafted instruction-following dataset of 1,000 samples made LLaMa 65B competitive with or better than models trained with much more fine-tuning effort [Zhou et al., 2023a].Other approaches score, select, or filter examples to preserve performance with fewer samples; one explanation is that pre-training already establishes core abilities that fine-tuning activates for instruction following.Source: §4.2.3 · pp. 175–176

4.2.4 Instruction Generalization

  • Instruction generalization must handle variation in both instructions and inputs, while supervised instruction-response data may miss subtle preferences and is bounded by dataset diversity and quality.The text presents preference models as an additional fine-tuning step and notes that diverse data can help adaptation to specific purposes.Source: §4.2.4 · pp. 177–178

4.2.5 Weak-to-Strong Generalization

  • Weak-to-strong fine-tuning trains a strong model on predictions from a weaker one, effectively distilling its knowledge but risking imitation of the weaker model’s errors [Burns et al., 2023a].The section motivates this setup when strong or human supervision is difficult to obtain and reports preliminary positive results across NLP tasks.Source: §4.2.5 · pp. 178–179
  • Performance gap recovered reached around 0.8 on 22 NLP classification tasks, although substantial weak-to-strong generalization remains challenging [Burns et al., 2023a; Aschenbrenner, 2024].PGR measures how much of the gap between weak-model performance and the strong model’s ground-truth ceiling is recovered.Source: §4.2.5 · p. 179
  • Small models can also guide large-model training through distillation losses or data filtering, while other approaches select or combine small and large models [fig. 4.5].A small model’s likelihood or cross-entropy can rank training sequences; an auxiliary loss can be applied in pre-training or fine-tuning, with its influence adjusted over training.Source: §4.2.5 · pp. 179, 181, Figure 4.5 · p. 180
  • Ensemble learning combines predictions from multiple small or specialized models using majority voting, weighted averaging, or stacking.The final prediction is derived from the aggregated results.Source: §4.2.5 · p. 181
  • Model cascading sends inputs to a larger, more accurate model only when a fast small model’s preliminary results fail predefined criteria, reducing computational costs and latency.Preliminary results that meet the criteria can be used directly.Source: §4.2.5 · p. 181

4.3 Human Preference Alignment: RLHF

This section explains how RLHF aligns LLMs with human preferences beyond supervised instruction-following, using human preference pairs to train a reward model and PPO to optimize an LLM policy with reward, value, and reference models.

4.3 Human Preference Alignment: RLHF

  • RLHF addresses preferences and ethical nuances that are difficult to encode in fine-tuning data by using human comparisons to train a reward model that guides LLM updates.The reward model scores generated outputs, and reinforcement learning uses those scores to update the LLM.Source: §4.3 · pp. 181–182, Figure 4.6 · p. 182

4.3.1 Basics of Reinforcement Learning

  • In the RL formulation, an LLM's context is the state, a candidate next token is the action, and its token distribution is the policy trained to maximize cumulative reward.Policy-gradient methods optimize expected returns from sampled trajectories, and the reward need not be differentiable; the discussion assumes sparse rewards available after a complete prediction.Source: §4.3.1 · pp. 183–185
  • High-variance policy-gradient estimates motivate reward baselines and return-to-go; A2C uses a critic's value estimate to calculate advantages that guide the actor's policy updates.The advantage compares an action's return with the expected return from its state; the temporal-difference error is one way to compute it.Source: §4.3.1 · pp. 186–188

4.3.2 Training Reward Models

  • A reward model scores a complete input-output sequence, commonly using a pre-trained Transformer decoder's final-position representation mapped to a scalar.This score represents how well the output aligns with desired behavior given the input.Source: §4.3.2 · pp. 188–189, Figure 4.8 · p. 189
  • Reward-model training collects judgments on multiple generated candidates through ratings or rankings, with pairwise comparisons highlighted as a simple, common approach.Experts choose between randomly drawn output pairs using criteria such as clarity, relevance, and accuracy; preferences can be encoded as binary labels.Source: §4.3.2 · pp. 189–190
  • The reward model learns from human-annotated preference pairs using a Bradley-Terry loss, then scores input-output pairs independently to guide policy optimization.The framework can use different ranking losses while applying the resulting reward models in the same way.Source: §4.3.2 · pp. 190–191

4.3.3 Training LLMs

  • For LLMs, A2C estimates an action's advantage with a TD error using the value function trained from reward-model feedback.RLHF generally has an input-only dataset, so outputs are sampled from the current policy rather than taken from human-annotated input-output pairs.Source: §4.3.3 · pp. 191–192
  • Clipped policy optimization bounds the current-to-reference policy likelihood ratio to reduce high-variance gradient estimates and prevent large updates.The surrogate objective uses importance sampling to reweight rewards according to the current policy's likelihood relative to the reference policy.Source: §4.3.3 · pp. 192–193
  • PPO adds a KL-style penalty to keep the current policy close to the reference and limit destabilizing updates within a trust region.The penalty can be approximated using policy probabilities while ignoring dynamics.Source: §4.3.3 · p. 193
  • RLHF uses four Transformer-based models: reward, value, reference, and target policy; the reference remains fixed while the target policy is trained with reward and value supervision.The reward model predicts scores, the value model estimates expected future rewards, and the reference model provides a baseline for policy training.Source: §4.3.3 · p. 194
  • After initialization and reward-model training, the value model and policy are trained simultaneously, updating value by MSE and policy by PPO loss.The RLHF workflow is illustrated in Figure 4.9.Source: §4.3.3 · pp. 194–195, Figure 4.9 · p. 195

4.4 Improved Human Preference Alignment

This section surveys refinements to reward modeling and alternative methods for human-preference alignment, including ranking supervision, reward sparsity and granularity, combining reward models, DPO, and AI-generated preferences. It also considers alignment of intermediate reasoning steps and inference-time selection of outputs.

4.4.1.1 Supervision Signals

  • Listwise supervision extends pairwise ranking by ordering multiple outputs; Plackett-Luce models the full ranking, while other learning-to-rank methods also apply to preference modeling.A simple listwise loss sums pairwise comparison losses across outputs.Source: §4.4.1.1 · pp. 196–197
  • Pointwise reward models regress against individual human scores, but inconsistent ratings and limited data can impair performance; they may suit abundant, reliable annotations.Because pointwise methods fit absolute scores, relative-preference methods can better generalize when feedback is inconsistent or training data is limited.Source: §4.4.1.1 · pp. 197–198
  • Eisenstein et al. add a squared-sum-of-rewards regularizer to pairwise loss, which can mitigate reward-model underdetermination.The regularization term is weighted by λ.Source: §4.4.1.1 · p. 198

4.4.1.2 Sparse Rewards vs. Dense Rewards

  • In RLHF, terminal rewards yield dense supervision when accumulated rewards are transferred back across time steps; advantage is equivalent to PPO’s one-step temporal-difference error and acts as shaped reward.For contextual linguistic tasks, terminal human feedback can remain useful because it is informative and accurate despite being sparse.Source: §4.4.1.2 · p. 199

4.4.1.3 Fine-grained Rewards

  • Fine-grained reward models score sequence segments and sum their rewards; segment ratings can use differences between cumulative scores, while segmentation can follow linguistic cues or score changes.Segment-level preference data is difficult to obtain, motivating rating-based or classification-based supervision.Source: §4.4.1.3 · pp. 199–201

4.4.1.4 Combination of Reward Models

  • Combining reward models can reduce mismatch from a single imperfect proxy; off-the-shelf LLM reward models have shown strong alignment performance, including state-of-the-art results on popular tasks [Lambert et al., 2024].Models can evaluate complementary aspects such as factual accuracy and completeness, and their outputs can be combined as an ensemble.Source: §4.4.1.4 · pp. 201–202

4.4.2 Direct Preference Optimization

  • DPO directly optimizes the policy from human preferences in a supervised-learning-like fashion, eliminating explicit reward-model training used in standard RLHF.Its derivation assumes the reward and reference models are fixed, then expresses the objective as minimizing KL divergence to a distribution derived from them.Source: §4.4.2 · pp. 202, 204, Figure 4.10 · p. 203
  • DPO directly optimizes preference feedback without a separate reward model, using the target policy and reference model to calculate preference probabilities.Its fixed-dataset training avoids PPO’s computationally expensive sampling process and is generally more sample-efficient; unlike online methods, it does not explore new states.Source: §4.4.2 · p. 205

4.4.3 Automatic Preference Data Generation

  • AI feedback scales preference-data generation beyond costly human annotation: an LLM generates response pairs and labels the preferred response, though data accuracy and diversity remain important.Demonstrations or CoT prompting can improve labeling; label-token probabilities can also provide pointwise reward-model supervision [Lee et al., 2023].Source: §4.4.3 · pp. 205–206

4.4.4 Step-by-step Alignment

  • Process-based alignment supervises intermediate reasoning steps, catching errors that outcome-based methods overlook when they reward a correct final answer.A reward model can classify each step as correct or incorrect, optionally neutral, and provide step-level feedback; obtaining annotations along long reasoning paths is a key challenge.Source: §4.4.4 · pp. 207–208, Figure 4.11 · p. 208

4.4.5 Inference-time Alignment

  • Best-of-N inference-time alignment uses a reward model to select the best of N generated outputs, while candidate diversity affects reranking quality.Selected candidates can also be used in rejection sampling to fine-tune the LLM, a simpler way to incorporate preferences than RLHF.Source: §4.4.5 · pp. 209–210

4.5 Summary

The chapter reviews instruction fine-tuning and RLHF while outlining broader challenges in aligning AI with human values. The inference chapter addresses efficient probability computation and search, including the use of additional inference-time compute.

4.5 Summary

  • Fine-tuning is less computationally expensive than pre-training and is suited to human-value alignment; the chapter focuses on instruction fine-tuning and RLHF.The book notes that it cannot comprehensively cover all recent alignment techniques.Source: §4.5 · pp. 210–211
  • An oracle gap between a stronger model’s best selection and an older model’s top output can indicate whether errors come from the model or its search.A significant gain suggests more model errors; a small gain may indicate search errors because the best candidates were not found.Source: §4.5 · p. 210
  • Diverse, context-dependent human values, changing real-world goals, and unintended ways to satisfy surrogate objectives complicate AI alignment.These challenges motivate work on more aligned systems, and the growing capabilities of LLMs have made alignment more urgent.Source: §4.5 · p. 211

Inference

  • Inference requires both efficient computation of Pr(y|x) and efficient approximate search for the highest-probability output.In most settings, both probability computation and the arg max operation pose computational challenges.Source: Inference · p. 212
  • Inference methods combine established algorithms and model-specific optimizations while trading off accuracy against efficiency.The chapter covers the prefill-decode framework, decoding algorithms, performance metrics, efficiency methods, and model compression.Source: Inference · p. 212
  • Additional prompts and context, including long chain-of-thought prompts, can improve performance, motivating inference-time scaling through increased compute.The passage cites OpenAI [2024]’s o1 and DeepSeek [2025]’s R1 as demonstrating performance on complex reasoning and programming tasks.Source: Inference · p. 212

5.1 Prefilling and Decoding

This section explains the prefill–decode framework for LLM inference, including model computation and output search. It surveys decoding strategies, stopping criteria, and evaluation metrics for assessing inference quality and deployment efficiency.

5.1.1 Preliminaries

  • Inference seeks to maximize Pr(y|x), which a decoder-only model computes through token probabilities conditioned on the input and preceding output tokens.The task separates into model computation of Pr(yi|x, y<i) and search for an optimal or near-optimal output sequence.Source: §5.1.1 · pp. 213–214
  • Standard Transformer self-attention has O(L×d×len^2) time complexity for a sequence of length len, while a KV cache reuses previously generated key-value pairs.The quadratic sequence-length cost makes inference slow for long sequences.Source: §5.1.1 · pp. 215–216

5.1.2 A Two-phase Framework

  • Prefilling processes the input in parallel to build a KV cache, while decoding generates tokens one at a time using that cache and is memory-bound.Prefilling uses causal masking and is generally compute-bound; decoding updates the cache at each step and stops when a criterion such as the end symbol is met.Source: §5.1.2 · pp. 218–220

5.1.3 Decoding Algorithms

  • Exhaustive search can find the globally optimal output but is impractical because possible sequences grow exponentially, so decoding algorithms prune the search space.Common heuristic approaches, including greedy and sampling-based search, trade search quality against computational efficiency.Source: §5.1.3 · pp. 220, 222

5.1.3.1 Greedy Decoding

  • Greedy decoding selects the highest-probability next token at each step, making it simple and efficient but potentially pruning high-quality sequences early.Each step adds the current token’s prediction log-probability to the accumulated sequence log-probability.Source: §5.1.3.1 · pp. 222–223

5.1.3.2 Beam Decoding

  • Beam search keeps the top K candidate sequences by accumulated probability, balancing search breadth and efficiency; K = 2 to K = 5 is usually sufficient in LLM inference.Sampling-based methods are now more commonly used for open-ended LLM generation because they produce more diverse and flexible outputs.Source: §5.1.3.2 · p. 223

5.1.3.3 Sampling-based Decoding

  • Top-k sampling draws from the k most likely tokens, while top-p sampling draws from the smallest probability-ranked set whose cumulative probability reaches p.Higher temperature makes the distribution more uniform and outputs more diverse; low temperature sharpens it, while poorly chosen k or p can produce overly deterministic or degenerate outputs.Source: §5.1.3.3 · pp. 225–226

5.1.3.4 Decoding with Penalty Terms

  • Penalty terms extend the likelihood objective to discourage repetition, control length, promote diversity, or enforce content and style constraints.The penalty function is flexible and can be incorporated into greedy or sampling-based search objectives.Source: §5.1.3.4 · p. 227

5.1.3.5 Speculative Decoding

  • Speculative decoding uses a fast draft model to propose tokens that the full model verifies in parallel, accepting valid tokens and generating a replacement after rejection.A smaller draft model can reduce prediction cost, but lower accuracy may mean fewer accepted tokens and a weaker speed benefit.Source: §5.1.3.5 · pp. 228, 230

5.1.3.6 Stopping Criteria

  • Decoding can stop at an end-of-sequence token or a maximum output length, under computational limits, or when probability or repetition criteria are met.For beam search, generation may continue until a specified number of complete sequences have been produced.Source: §5.1.3.6 · p. 230

5.1.4 Evaluation Metrics for LLM Inference

  • Inference evaluation can assess output quality, robustness, usability, and fairness, including standard metrics such as perplexity and F1 score and human judgments of fluency or relevance.Robustness tests may include ambiguous, adversarial, perturbed, and out-of-distribution inputs; fairness metrics can assess bias and harmful content.Source: §5.1.4 · p. 230
  • Efficiency evaluation includes request latency, throughput, time to first token (TTFT), inter-token latency (ITL), and CPU/GPU and memory utilization.TTFT mainly reflects prefilling and first-token generation when transmission time is small; ITL measures the time for each subsequent token.Source: §5.1.4 · p. 231
  • Energy efficiency measures electrical power consumed during inference, while cost efficiency covers the total expenses of deploying and maintaining an LLM.Source: §5.1.4 · p. 231
  • A comprehensive evaluation framework should combine quality-focused and efficiency metrics to assess an LLM’s performance and practicality in real-world applications.The appropriate metrics depend on the specific task and application.Source: §5.1.4 · p. 231

5.2 Efficient Inference Techniques

This section surveys techniques for improving LLM inference efficiency in deployment and serving, including caching, batching and scheduling, KV-cache management, chunked prefilling, and inference parallelization. It frames these techniques in terms of trade-offs among speed, accuracy, memory, throughput, specialization, and energy.

5.2.1 Prefix Caching

  • Prefix caching stores KV states for dataset prefixes and reuses a matching prefix for new inputs, computing only the remaining tokens.A basic sequence cache bypasses computation only for exact query matches; LRU or similar policies can limit prefix-cache memory use.Source: §5.2.1 · p. 232

5.2.2 Batching

  • Batching parallelizes sequences to raise throughput, but larger batches increase latency; length bucketing reduces padding, while disaggregated prefill/decode packing trades throughput for KV-transfer overhead [Wu et al., 2023a; Patel et al., 2024; Zhong et al., 2024].Disaggregated prefill and decoding can run on separate GPUs, and optimal performance typically requires a high-bandwidth, low-latency network.Source: §5.2.2 · p. 233, Figure 5.8 · p. 233, inference · p. 234

5.2.2.1 Scheduling

  • Iteration-level scheduling adjusts a batch at each token-prediction step, unlike request-level scheduling, which cannot interrupt a batch once processing begins.Dynamic adjustment can admit latency-sensitive requests earlier during decoding.Source: §5.2.2.1 · pp. 234–235

5.2.2.2 Continuous Batching

  • Continuous batching adjusts batches between iterations, adding requests when capacity allows and removing completed sequences while overlapping prefilling and decoding to keep the inference engine utilized.Prefilling prioritization can increase throughput but also latency; frequent batch reorganization adds memory and compute overhead and can cause fragmentation or further latency.Source: §5.2.2.2 · pp. 235, 238

5.2.2.3 PagedAttention

  • PagedAttention divides KV caches into fixed-size, non-contiguous memory blocks, improving memory utilization under fragmentation and supporting dynamic sequence growth without costly reallocation or copying.The blocks can also support parallel cache reads and writes; a well-designed paging strategy can keep added memory-access overhead minimal.Source: §5.2.2.3 · pp. 238–240, Figure 5.12 · p. 239

5.2.2.4 Chunked Prefilling

  • Chunked prefilling divides input sequences into chunks so prefilling can overlap decoding, reducing decoding idle time and helping balance computation across sequences.Each chunk requires a separate forward pass and KV-cache maintenance, increasing memory overhead and scheduling complexity; choosing chunk size balances throughput and latency.Source: §5.2.2.4 · p. 240
  • Chunked prefilling divides prefill into steps so decoding can proceed during prefilling, while scheduling priorities can be tailored to latency, throughput, deadlines, or user importance.For example, decoding step D22 can run during sequence 1’s prefilling step P12 rather than waiting for the entire prefill to finish.Source: Figure 5.13 · p. 241, §5.2.2.4 · p. 241

5.2.3 Parallelization

  • Parallelization methods transfer from pre-training, but real-time variable-length requests, heterogeneous devices, load imbalance, and communication overhead complicate utilization and scheduling; serving systems are beyond this discussion.The section points readers to systems such as vLLM, TensorRT-LLM, and TGI for serving concepts and techniques.Source: §5.2.3 · pp. 241–242

5.2.4 Remarks

  • Inference speed and accuracy trade off: quantization, pruning, and distillation can reduce computation and latency while slightly degrading performance, whereas higher precision or larger models can improve accuracy at higher cost.Source: §5.2.4 · p. 242
  • KV caches reduce repeated attention computation but consume growing memory; chunked or windowed attention can reduce cache size through recomputation, while lower precision can save memory and bandwidth but risk instability or accuracy loss.These choices form a memory-compute-accuracy trade-off.Source: §5.2.4 · pp. 242–243
  • Other system-level trade-offs include throughput versus latency under batching, general-purpose versus specialized models with different complexity and storage needs, and energy use versus performance.Batching can raise throughput while increasing per-request waiting time; model compression may save energy while reducing output quality or increasing latency.Source: §5.2.4 · p. 243

5.3 Inference-time Scaling

This section describes ways to scale computation at inference time, including context expansion, search, ensembling, and longer reasoning paths, and notes their costs and constraints. It also distinguishes inference-time approaches from training-based scaling and points to further detail on extending chain-of-thought prompting.

5.3.1 Context Scaling

  • Context scaling adds examples, reasoning steps, or retrieved external knowledge to the input, but long contexts can raise costs and obscure relevant information.Few-shot and chain-of-thought prompting extend context, while RAG retrieves relevant snippets to ground responses in timely or specialized knowledge; effective scaling therefore requires selecting and structuring pertinent information.Source: §5.3.1 · pp. 244–245

5.3.2 Search Scaling

  • Search scaling increases generated output length or expands the candidate space, with beam width, reasoning trees, and Monte Carlo tree search among the approaches described.Long thinking paths have been used for reasoning in OpenAI [2024]’s o1 and DeepSeek [2025]’s R1; wider searches can improve candidate quality but increase memory use and inference time, with diminishing returns.Source: §5.3.2 · pp. 245–246

5.3.3 Output Ensembling

  • Output ensembling combines predictions through probability averaging, majority voting, reranking, or a meta-learner to reduce individual errors and improve reliability.It adds latency and system complexity, and gains may diminish as members are added; diversity among ensemble members is desirable because they make different errors.Source: §5.3.3 · p. 246

5.3.4 Thinking-Path Generation and Verification

  • Reasoning scaling goes beyond simple chain-of-thought because complex problems require dynamically generated steps, verification, backtracking, and self-correction.The section distinguishes training-free methods, which operate at inference through prompting and search, from training-based methods that update parameters, such as fine-tuning on step-by-step reasoning examples.Source: §5.3.4 · p. 247

5.3.4.1 Solution-level Search with Verifiers

  • Solution-level search generates multiple candidate answers and uses a verifier—a model or function that scores or judges candidates—to select the best, with parallel scaling independently sampling K candidates.Source: §5.3.4.1 · pp. 247, 249

5.3.4.2 Step-level Search with Verifiers

  • Step-level search verifies partial reasoning paths, pruning unpromising steps or extending promising ones to guide exploration before a complete solution is formed.A verifier can assess a step in the context of the preceding path and problem, enabling backtracking or focused resource allocation; unlike standard decoding, explicit verifiers can use LLMs, symbolic checks, value-like predictions, or human judgments.Source: §5.3.4.2 · pp. 250–251
  • Step-level search expands multiple candidate reasoning steps, pruning them by likelihood or verifier scores until it finds a complete chain or reaches a search limit.Process reward models score each intermediate step, offering finer-grained supervision than outcome reward models, but require labor-intensive human annotations; frequent verification also adds cost and latency, and inaccurate verifiers can discard good paths or preserve flawed ones.Source: §5.3.4.2 · pp. 251–252
  • Teacher-generated candidate evaluations can supply step-level verification data, while trained PRMs can be generalized from tasks where verification is easier to other tasks.The teacher first generates a reasoning path, then proposes and evaluates alternative next steps; those evaluations become annotations.Source: §5.3.4.2 · p. 252

5.3.4.3 Encouraging Long Thinking

  • Longer thinking can be encouraged through explicit deliberation prompts, decoding changes such as token limits or short-output penalties, and multi-stage generation.Source: §5.3.4.3 · p. 253

5.3.4.4 Training-based Scaling

  • Training-based scaling includes reasoning-data fine-tuning, reward-guided reinforcement learning, teacher-to-student distillation, and iterative refinement using verified solutions.RL uses a reward model—an outcome, process, or rule-based model—to update the LLM toward higher-reward reasoning; iterative refinement adds correct verified paths to training data for further fine-tuning.Source: §5.3.4.4 · pp. 253–254
  • Training-based scaling can improve base reasoning quality and reduce the search or sampling needed for correct answers, while supporting generalization to novel problems.Its tradeoffs include expensive, labor-intensive dataset creation, computationally intensive fine-tuning and engineering, and possible overfitting that limits out-of-distribution performance.Source: §5.3.4.4 · p. 254

6.2 Reasoning via Prompting

  • A reasoning path is a token sequence divided into steps representing individual deductions or calculations; this example uses sentences as steps, and the answer is extracted from the last step.The text notes that reasoning paths can be divided in different ways, such as by mathematical operations.Source: §6.2 · p. 260
  • Self-consistency samples multiple reasoning paths and uses a majority vote to identify the most reliable answer [Wang et al., 2023b].The text also mentions problem decomposition [Zhou et al., 2023b], self-reflection, and deliberate exploration of branching paths.Source: §6.2 · pp. 260–261

6.3 Inference-time Reasoning

This section examines ways to improve LLM reasoning through additional inference-time compute, including sampling, search, and verification of reasoning paths. It also covers methods for training models to internalize reasoning and scaling that training through reinforcement learning, synthetic data, and longer context, including practical requirements and costs.

6.3.1 Inference-Time Compute Scaling

  • Inference-time compute lets models explore reasoning steps on demand, with flexible allocation that can help a smaller model outperform a larger model forced to answer immediately.The section frames this as search scaling: more compute expands the space of possible logical steps, though it cannot supply reasoning abilities the model lacks.Source: §6.3.1 · p. 261

6.3.2 Sampling Multiple Reasoning Paths

  • Multiple sampled paths increase the chance of finding correct logic; temperature with top-k or top-p sampling broadens exploration while limiting low-probability token choices.Higher temperatures can increase diversity but degrade average path quality, motivating an explicit diversity–quality trade-off in search or training.Source: §6.3.2 · pp. 262–263, §6.3.2.3 · p. 264

6.3.2.1 Best-of-N (BoN)

  • Best-of-N selects the sampled reasoning path with the highest verifier score, but a weak verifier may reward flawed logic that happens to reach the correct answer.The final answer is extracted from the selected path; verifier quality is therefore central to avoiding reward hacking.Source: alignment · p. 263

6.3.2.2 Self-Consistency

  • Self-consistency extracts each sampled path’s final answer and chooses the most frequent, relying on correct reasoning paths converging on an answer while errors vary.Unlike Best-of-N, it votes over answers rather than scoring the reasoning process.Source: §6.3.2.2 · pp. 263–264

6.3.2.3 Balancing Diversity and Quality in Sampling

  • Diversity-promoting bonuses or penalties and entropy-regularized RL encourage broader exploration, while excessive exploration can reduce reasoning quality or stability.Inference-time search explores outputs under a fixed model; RL changes the policy, with KL regularization constraining drift from the reference policy.Source: §6.3.2.3 · pp. 264–265

6.3.3 Search Strategies for Reasoning

  • Reasoning can be framed as search over partial paths, using tree search or Monte Carlo tree search, graph structures, or learned search dynamics [Yao et al., 2024; Hao et al., 2023; Zhou et al., 2024; Zhang et al., 2025; Lehnert et al., 2025].Search gains plateau when the model lacks the capability to solve a problem, and increased effort can reduce path diversity by revisiting redundant ideas.Source: §6.3.3 · pp. 265–266

6.3.4.1 Process Reward Models

  • Process reward models assess individual reasoning steps, addressing outcome models’ false positives; they need step-level supervision, obtained through expert labels or rollouts [Uesato et al., 2022; Lightman et al., 2024; Wang et al., 2024c].A path score can aggregate step scores by taking the minimum, so its weakest step determines whether the path is considered sound.Source: §6.3.4.1 · pp. 266–267

6.3.4.2 Generative Verifiers

  • Generative verification prompts an LLM to critique reasoning steps in natural language, avoiding the architecture changes and training required for scalar-scoring verifiers.The verifier receives the query and intermediate steps and generates textual feedback on the current step’s logic.Source: §6.3.4.2 · p. 268
  • Generative verifiers produce natural-language critiques that support interpretation, self-correction, and reuse of the same LLM as solver and verifier without training.The critique can identify a specific error and explain how to fix it; an existing strong LLM can provide verification directly.Source: §6.3.4.2 · pp. 268–269

6.3.4.3 Formal Symbolic Systems as Verifiers

  • Formal verifiers execute generated code or formal statements under fixed rules, avoiding hallucination-related false positives but requiring formalization skill and suiting math or code rather than open-ended tasks.For example, an equivalency checker can verify whether optimized LLVM-IR preserves the original code’s semantics.Source: §6.3.4.3 · p. 270

6.4.1 Supervised Reasoning Data

  • Supervised reasoning training uses reasoning paths, rather than only final conclusions, as outputs so models learn to generate intermediate steps and answers.Reasoning can be separated from the user-facing answer with <think> tags, step labels, or role-based message separation.Source: §6.4.1 · pp. 271–272

6.4.2 Process Supervision

  • Process supervision addresses the credit assignment problem by providing feedback for individual reasoning steps instead of penalizing only the final outcome.In reinforcement learning, a PRM checks generated paths and step-level verification turns sparse outcome rewards into dense signals.Source: §6.4.2 · pp. 272–273

6.4.2.1 Training with Binary Verification Scores

  • With binary verification scores, a path receives positive sequence reward only if every step is correct, or training can penalize the first incorrect step and stop generation.A strict checker returns 1 for a correct step and 0 for an error.Source: §6.4.2.1 · p. 273

6.4.2.2 Training with Real-valued Verification Scores

  • For real-valued verifier scores, reward shaping assigns each step the change from the previous score, yielding dense feedback while preserving the final score as the sequence reward.The intermediate scores cancel in the telescoping sum; the step-level feedback encourages steps with positive future returns and penalizes those with negative returns.Source: §6.4.2.2 · pp. 273–274

6.4.2.3 Incorporating Verification into the Advantage Function

  • Verification scores can inform advantages through temporal-difference errors or GRPO’s standardized group-relative step scores [Shao et al., 2024], with baselines reducing variance.GRPO uses the group mean as a baseline, avoiding a separate value network; relative feedback helps prevent overemphasis on queries that are too easy or too hard.Source: §6.4.2.3 · pp. 275–276

6.4.3 On-Policy Distillation for Reasoning

  • On-policy distillation is presented as a post-training method for providing a richer, denser training signal than scalar step rewards.It is introduced as an extension to process supervision.Source: §6.4.3 · p. 276
  • On-policy distillation trains a student on its own trajectories to address exposure bias, with teacher probability distributions providing dense feedback that can help it recover from deviations.Unlike fixed teacher-generated paths, this supervision covers student-generated states, including deviations; running the teacher alongside the student makes training more expensive than off-policy distillation.Source: §6.4.3 · pp. 276–277
  • On-policy distillation uses student rollouts and teacher guidance, approximating an intractable step-level KL objective with sampled token-level KL divergence.A reasoning step may contain multiple tokens, making its full probability distribution exponentially large; the practical approach samples one or a small batch of student paths and computes token-level KL on them.Source: §6.4.3 · pp. 276–277

6.4.4.1 Large-scale Reinforcement Learning

  • The “aha moment” observed in large-scale reinforcement-learning training includes models pausing, recognizing mistakes, and backtracking without explicit human annotation.As training compute scales, models can allocate longer chain-of-thought traces to harder problems and learn to think more deeply.Source: §6.4.4.1 · pp. 278–279
  • Reinforcement-learning compute and reasoning performance follow an S-shaped scaling curve, with base-model capabilities and context length raising the performance limit while algorithmic changes mainly improve compute use.Scaling RL also carries high computational costs and requires careful tuning; its sensitivity to hyperparameters and data makes stable large-scale training an engineering challenge.Source: §6.4.4.1 · p. 279

6.4.4.2 More Synthetic Data for Reasoning

  • Because high-quality human annotations for complex multi-step reasoning are scarce, researchers use strong LLMs to generate diverse trajectories, then filter and curate them for training [Mukherjee et al., 2023].Common methods sample multiple paths for a seed question, retain paths verified as logically sound and correct, and generate problem variations to broaden training data [Wang et al., 2023b].Source: §6.4.4.2 · p. 279

6.4.4.3 Scaling Context Length

  • Reasoning models need longer context for extensive prompts, reasoning paths that can span thousands or tens of thousands of tokens, and multi-round agent interactions.Prompts may contain detailed constraints, complex environment rules, or multiple long few-shot chain-of-thought examples.Source: §6.4.4.3 · p. 279

6.5 Evaluating Reasoning LLMs

This section examines how to evaluate reasoning LLMs beyond final-answer accuracy, including logical continuity, process assessment, benchmark reliability, and distinctions among textual reasoning, internal computation, memorization, and functional reasoning. It also considers reasoning in agentic settings involving cycles of thinking, acting, observing, and tool use, including challenges with staying on track, choosing tools, and recovering from failures.

6.5 Evaluating Reasoning LLMs

  • Reasoning over long contexts requires reliable retrieval and logical consistency, because losing an assumption or intermediate result can collapse an interdependent derivation.Unlike summarization, reasoning cannot safely discard minor details.Source: §6.5 · p. 280

6.5.1 Outcome-based vs. Process-based Evaluation

  • Outcome-based evaluation checks only the final answer and can reward lucky answers, while process-based evaluation checks each step but is costly and difficult to standardize.Local next-step tests, PRMs, and error-identification tasks offer narrower ways to assess reasoning decisions; process assessment remains expensive and often insufficiently robust.Source: §6.5.1 · pp. 280–281

6.5.2 Evaluation Metrics

  • Exact Match and Accuracy compare extracted answers with ground truth, while Pass@k measures whether at least one of k sampled paths is correct.String matching may require answer normalization; Pass@k is popular in code generation and complex math and requires an oracle to verify sampled answers.Source: §6.5.2 · p. 281
  • Cons@k selects the most frequent answer among k paths to measure consistency, while LLM-as-a-Judge scores open-ended reasoning or compares models using win rates.Unlike Pass@k, consensus evaluation does not require verifying each sample’s correctness.Source: §6.5.2 · p. 281, §6.5 · p. 282

6.5.3 Open Issues

  • Reasoning benchmarks face contamination, prompt sensitivity, unfaithful explanations, and saturation, motivating private or updated tests, harder novel problems, and process supervision.Final-answer-only rewards can encourage flawed reasoning; autonomous agents that use tools and recover from errors are another proposed evaluation setting.Source: §6.5.3 · pp. 282–283

6.6.1 External Reasoning vs. Internal Reasoning

  • Generated chain-of-thought text may not reflect internal computation, prompting research into interpretability and latent reasoning methods that keep thoughts hidden.Approaches include pause tokens, distillation into hidden states [Deng et al., 2023], continuous thoughts, and hidden thought tokens.Source: §6.6.1 · p. 283, §6.6 · p. 284

6.6.2 Memorization vs. Reasoning

  • Memorization replays learned patterns, whereas reasoning applies general rules to new problems; grokking describes a shift from rote fitting to generalization, while models trade parameter capacity for inference computation.In grokking, continued training can trigger a rapid test-accuracy increase as parameters develop a general representation [Power et al., 2022]. Common facts may be retrieved from memory, while rare multi-step tasks favor System 2 thinking; RAG offloads external knowledge retrieval.Source: §6.6.2 · p. 285, §6.6.2.1 · p. 285, §6.6 · p. 286

6.6.3 Can LLMs Truly Reason?

  • LLMs can solve problems through valid logical chains, but text-pattern dependence, weak grounding, and potentially deceptive explanations complicate claims that they reason like humans.The chapter also frames reasoning as a driver of autonomous behavior and identifies long thinking and efficient allocation of inference-time computation as promising directions.Source: §6.6.3 · pp. 286–287, §6.7 · p. 287

6.7 Summary

  • Agentic reasoning uses a continuous loop of thinking, acting, and observing, but models may repeat mistakes, make poor plans, or fail to recover.Keeping a model on track over an extended period remains difficult.Source: §6.7 · p. 287
  • Tool-augmented reasoning lets LLMs write code [Gao et al., 2023b] or call APIs, but they must learn when and which tool to use and handle faulty results.Tools can address limits in precise math and access to real-time facts; models must detect tool errors or unusual data and try again rather than trust bad results.Source: §6.7 · pp. 287–288

Index

The index collects named concepts and methods spanning language modeling, prompting, alignment, inference, and reasoning.

Pre-training and language modeling

  • Pre-training and language-modeling entries include masked language modeling, causal language modeling, prefix language modeling, and scaling law.The index also lists BERT, T5, XLM, and multilingual BERT.Source: Index · pp. 314–315

Prompting

  • Prompting entries cover chain-of-thought, least-to-most prompting, self-consistency, prompt optimization, and automatic prompt design.The index includes zero-shot and few-shot CoT prompting as well as hard and soft prompts.Source: Index · pp. 314–315

Alignment

  • Alignment and reinforcement-learning entries include instruction fine-tuning, RLHF, direct preference optimization, and proximal policy optimization.Related entries include reward models, reward hacking, and weak-to-strong generalization.Source: Index · pp. 314–315

Inference and reasoning

  • Inference and reasoning entries include inference-time scaling, key-value cache, speculative decoding, process reward models, and reasoning traces.The index also names problem decomposition, latent reasoning, and continuous thought.Source: Index · pp. 314–315

Related papers

Loading 2501.09223v4…