PaperlayerView exact revision on arXiv

Source-linked AI summary

The Probabilistic Structure of Large Language Models

Adnan Aboulalaâ

arXiv:2609.25134v1cs.LGcs.AImath.PRstat.ML

TL;DR

Because probabilistic accounts of LLMs leave important objects and sampling steps unclear, this paper assembles a measure-based account of training and generation and uses diffusion as a complementary illustration. It clarifies that token calibration concerns next-token events rather than truth, and that sequential sampling can exactly realize the autoregressive joint law.

  • Problem

    Probabilistic treatments leave key language-model objects and the steps involved in choosing a generated word insufficiently assembled in a single account.

    Source: §1 · p. 2

  • Method

    The paper organizes LLMs around probability measures, estimation, and sequential generation, and develops diffusion models as a complementary probabilistic illustration.

    Source: §1 · p. 3, §8.1 · p. 19, §8.3.2 · p. 22

  • Results

    Token-level calibration does not establish truth, while sequential sampling exactly produces a sample from the autoregressive joint law.

    Source: §5.5 · p. 14, §7 · p. 17

  • Takeaways & Limitations

    Choosing a word is a separate decision over the model's predicted measure, not a model-supplied best word.

    Source: §5 · p. 8

    Closeness to the empirical corpus under the estimation criterion does not establish closeness to the unknown data-generating law; that is a separate generalization question.

    Source: §5 · p. 8

Abstract

from arXiv · show

This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.

1 Introduction

The paper assembles a probabilistic account of language models, separating the measures involved in modeling and training from the process of generating text. It also uses diffusion models as a complementary illustration of probabilistic generation.

1 Introduction

  • The paper treats language models as measures on token sequences specified by autoregressive conditionals, with training fitting a parametric measure to corpus data.The underlying data-generating distribution is generally unknown.Source: §1 · pp. 1–2
  • The account distinguishes statistical plausibility from truth: sequence likelihood reflects learned regularities, while the objective contains no term for whether a statement is correct.The probabilistic formulation is used to discuss both training and inference.Source: §1 · p. 2
  • Probabilistic accounts leave key objects and the direction of the training criterion insufficiently explicit, although that direction governs tail behavior and relates to hallucination propensity.The paper aims to bring partial probabilistic treatments together in one account.Source: §1 · p. 2
  • The paper follows the model through its probabilistic object, estimation, and generation, explaining the transformation of a measure and the subsequent draw of a word.It gives generation greater depth because its probabilistic content is often treated elliptically.Source: §1 · pp. 2–3
  • Diffusion models provide a complementary account, covering score functions, denoising, and reverse-time simulation in discrete and continuous time.The paper emphasizes depth and completeness for diffusion, while disentangling probabilistic objects is central for language models.Source: §1 · pp. 2–3

2 A fundamental goal: construction of a probability measure on sequences of words

The section defines a language model as a probability measure on sequences, represented through predictive conditional distributions. It connects this object to corpus data, model fitting, and the practical generation of finite strings.

2 A fundamental goal: construction of a probability measure on sequences of words

  • An LLM is a probability measure on finite word sequences, whose conditional distributions specify the next-word law given the preceding context.The sequence law and its predictive kernels determine one another through the chain rule.Source: §2 · pp. 3–4
  • The model seeks a distribution from corpus evidence, while the data-generating law is generally unknown and may differ from the empirical corpus distribution.The section describes high probability as typicality, not grammaticality or correctness.Source: §2 · p. 3
  • The construction begins with a text corpus, fits a machine-learning system to sequence probabilities, and uses the resulting system for tasks such as generation and translation.The paper notes that transformer-based systems have proved most effective.Source: §2 · p. 3
  • A probability measure on finite strings requires the stopping time for the end-of-sequence token to be finite almost surely; models satisfying this condition are called tight.Without termination, probability mass can leak to infinite sequences.Source: §2 · p. 3

3 The statistical model

The statistical model uses a parametric family of sequence measures to approximate the unknown language distribution. Its network produces context-dependent logits, which are transformed into predictive probabilities.

3 The statistical model

  • When the model family is misspecified, fitting the unknown language law targets a pseudo-true parameter minimizing its KL divergence to the family.This parameter is the information projection of the true law onto the model family.Source: §3 · p. 4
  • The model's practical output is a conditional probability distribution for the next token given its context.The network parameterizes these predictive probabilities through logits.Source: §3 · p. 4
  • Logits assign an unconstrained real score to each candidate token, with softmax restoring positivity and normalization for the probability distribution.Adding the same constant to every logit leaves the measure unchanged.Source: §3 · p. 4
  • The network scores each candidate by combining a context summary with a token representation, allowing observations from similar contexts to transfer to unseen contexts.The context summary and token vectors occupy the same-dimensional space.Source: §3 · pp. 4–5

4 The learning phase

The learning phase separates the unknown target law, observed empirical law, and parameterized model, then fits the model by minimizing a divergence from the empirical distribution. Because corpus-wide optimization is impractical and the empirical law cannot generalize to unseen sequences, training uses stochastic updates to obtain a fitted probability model.

4.1 Setting

  • The training corpus is idealized as N i.i.d. realizations from P⋆, while P⋆ is unknown, the empirical law is observed, and Pθ is the computable model family.The i.i.d. assumption is rough: fixed-length blocks from the same article can be dependent, and their cut points are arbitrary.Source: §4.1 · p. 5

4.2 The estimation criterion

  • Minimizing KL divergence from the empirical law to Pθ is maximum likelihood and fits predictive kernels together, weighted by context frequency.The criterion corresponds to logarithmic loss, a local strictly proper scoring rule that decomposes by the chain rule.Source: §4.2 · pp. 5–6

Why not simply using bPN?

  • The empirical law cannot generate unseen content and suffers the curse of dimensionality, whereas Pθ shares statistical strength across contexts to generalize beyond observed sequences.The number of possible sequences grows exponentially with sequence length; similar contexts can transfer observations to unseen contexts.Source: Why not simply using bPN? · p. 6
  • Computing each full-corpus gradient is prohibitively expensive, so training estimates it from randomly selected mini-batches and updates parameters repeatedly.The mini-batch gradient fluctuates around the true gradient but is cheaper to compute; the procedure is a stochastic approximation scheme.Source: Why not simply using bPN? · pp. 6–7
  • Backpropagation computes the many parameter derivatives efficiently, while large neural networks commonly use adaptive SGD variants such as Adam and AdamW.Adaptive methods address parameters whose gradients have different magnitudes and behaviors.Source: Why not simply using bPN? · p. 7
  • Training does not recover a unique true parameter vector or guarantee a global minimizer; θ̂ is one configuration providing a satisfactory fitted probability model.Different parameter vectors can define the same model because of neural-network symmetries.Source: Why not simply using bPN? · p. 8

5 The inference phase or how to generate texts and next words from the probability measure obtained

This section explains how inference turns a model’s probability measure over next tokens into text through separate decoding decisions and sequential draws, including why decoding may avoid modes. It also describes how forward-KL training encourages broad, permissive support that enables sampling valid, novel token sequences.

5.1 A visual analogy of the decoding process

  • Decoding first transforms the model’s next-token probability measure and then realizes a draw; a uniform dart analogy illustrates why the most probable token can be sampled often without being chosen every time.Greedy decoding takes the largest piece, while truncation and temperature change piece sizes before sampling.Source: §5 · pp. 8–9, §5.1 · p. 9

5.2 Stage (a): transforming the measure

  • Temperature adjusts randomness but leaves the tail intact; fixed top-k is rigid across contexts, while top-p adapts support size and typical sampling retains entropy-near tokens rather than the mode.Top-p selects a minimal set carrying threshold mass; typical sampling uses tokens whose negative log-probability is near the distribution’s entropy.Source: §5.2 · pp. 10–11

5.3 The mode, and why it is avoided ?

  • The mode is rarely reached by random draws over long sequences and is undesirable as text because its predictability minimizes surprise, producing flat, repetitive output—the likelihood trap [24].The passage contrasts the mode with typical outputs, which lie where probability per token is close to language entropy.Source: §5.3 · pp. 11–12

5.4 Stage (b): performing the draw or how the next word Is finally chosen

  • Inverse-transform sampling selects a token using cumulative probabilities and a uniform draw; Gumbel-max is an equivalent parallel alternative that adds independent noise to logits and takes the maximum.Gumbel-max avoids sorting, cumulative sums, and normalization, and can incorporate truncation by setting excluded logits to negative infinity.Source: §5.4 · pp. 12–13

5.5 Why truncated sampling does not collapse to greedy decoding

  • Ancestral sampling draws each token from its conditional measure and, by the chain rule, yields an exact sample from the fitted joint process; greedy stepwise choices do not maximize the sequence probability.This exactness applies to sampling from the unmodified fitted process, not to decoding rules that transform the measure.Source: §5.2 · p. 10, §5.5 · p. 14

5.6 Practice in deployed systems

  • As of 2026, the passage describes ancestral sampling with temperature and nucleus truncation as the norm in conversational systems, while some frontier models remove user-facing sampling controls.It says OpenAI, Anthropic, and Google default to temperature 1.0 and recommend adjusting either temperature or top-p, not both, without a theoretical basis for that recommendation.Source: §5.6 · p. 14

5.7 The mass-covering property of the forward KL divergence

  • Forward-KL training penalizes assigning near-zero probability to observed data, so the model covers the empirical support without forbidding probability beyond it, and smooth parametrization spreads mass to unseen sequences.The passage characterizes this as forced smoothing: covering training points also assigns probability to surrounding, newly generated sequences.Source: §5.7 · p. 15
  • Forward KL forces non-zero probability across empirical data, maintaining open support that lets generative models sample valid, novel token sequences during inference.This open support is the probabilistic mechanism described for generating those sequences.Source: §5.7 · p. 15

6 Summary of the decoder transformer procedure

The decoder transformer maps a token sequence to logits, using representations built from positional embeddings and repeated layer operations. Its causal mask preserves autoregressive prediction; training processes all positions in parallel, while generation proceeds one token at a time.

6 Summary of the decoder transformer procedure

  • The decoder transformer maps token sequences to logits, using position-wise representations formed from token embeddings and positional encodings.Representations summarize the prefix and are refined layer by layer.Source: §6 · p. 15
  • Each layer combines normalization, multi-head attention as the only position-mixing operator, and a position-wise feed-forward operation.Normalization rescales and shifts standardized coordinates; the feed-forward operation applies a learned nonlinear transformation.Source: §6 · pp. 15–16
  • The causal mask restricts each position to its current and preceding tokens, encoding the chain rule and preventing prediction from seeing the token being predicted.Without the mask, the model would not compute genuinely predictive kernels.Source: §6 · p. 16
  • Training uses true corpus prefixes and computes all token conditionals in one parallel forward pass and one backward pass, which enables transformer scaling.This is teacher forcing; generation instead needs one forward pass per token because each new token enters the next context.Source: §6 · p. 16

7 On the probabilistic nature of LLMs

The section treats LLMs as probabilistic models whose measures are estimated from corpus data and sampled during generation. It connects that account to the distinction between token plausibility and truth, and to the different roles of model architecture, decoding, and post-training.

7 On the probabilistic nature of LLMs

  • The model supplies a probability measure over continuations, while stochastic generation procedures further shape the output; language models represent multiple plausible continuations rather than selecting one deterministically.The next-token kernels define the measure, and the resulting sequence probabilities are specified through autoregressive factorization.Source: §7 · p. 16
  • Maximum-likelihood estimation fits the corpus empirical law to a parametric family by stochastic approximation, with perplexity measuring empirical cross-entropy as an effective branching factor.A perplexity of 10 corresponds to uncertainty comparable to choosing uniformly among ten tokens.Source: §7 · p. 17
  • Generation samples a Markov-chain trajectory; temperature tilts and top-p conditions its transition kernel, while forward-KL mass coverage links plausible false strings to hallucination.Kalai and Vempala [27] prove a statistical lower bound on hallucination rate of the order of the fraction of facts appearing exactly once in the training corpus.Source: §7 · p. 17
  • Token-frequency calibration does not establish truth, and reward optimization can degrade token-level calibration by concentrating probability on preferred responses without a corresponding accuracy gain.The calibrated event is that a token comes next, not that an assertion is true.Source: §7 · p. 17
  • Generative use depends on the full measure, including its tail, and appended outputs make generation a stochastic trajectory requiring simulation rather than only estimation.RNNs and transformers differ in context summarization, not in the probabilistic nature of their outputs.Source: §7 · p. 18
  • A measure over the unbounded space of token sequences cannot be exhaustively listed, so autoregressive factorization specifies it through next-token conditional laws.With a vocabulary of about 5 × 10^4 tokens, the section estimates roughly 10^470 possible texts of a hundred tokens.Source: §7 · p. 18

8 The Diffusion Models Approach : From Gaussian Transitions to Stochastic Differential Equations

This section presents diffusion models as a probabilistic approach to generating samples from complex, high-dimensional data distributions. It contrasts their gradual noise-to-data process with autoregressive token generation and outlines discrete-time and continuous-time formulations.

8 The Diffusion Models Approach : From Gaussian Transitions to Stochastic Differential Equations

  • Unlike autoregressive models that sample tokens successively, diffusion models progressively add noise to data and train a network to reverse the process, generating from fresh noise.The approach is described as successful for image generation and extended to audio, video, three-dimensional data, and scientific and medical imaging.Source: §8 · p. 18

8.1 Object and Motivations

  • Generative modeling seeks to approximate an unknown high-dimensional data distribution and draw new samples, while direct parameterization and sampling are intractable for complex multimodal distributions.Source: §8.1 · p. 19
  • Diffusion models gradually transform a standard Gaussian toward the data distribution through small steps, with a discrete construction and a continuous-time SDE formulation.The section follows the discrete-time formulation and Gaussian transitions before taking the continuous-time limit; the cited development includes Sohl-Dickstein et al., Ho et al., and Song et al..Source: §8.1 · p. 19

8.2 The Intuitive Idea

The intuitive account builds diffusion generation by first corrupting data with Gaussian noise and then learning to reverse that corruption. At generation time, the learned denoising steps transform fresh noise into a new sample.

8.2 The Intuitive Idea

  • The forward process progressively adds Gaussian noise to data without learning, and a network learns the reverse denoising steps.Generation starts with a fresh Gaussian sample and applies the learned reverse process.Source: §8.2 · p. 19
  • For cat images, the network learns to remove noise step by step and can generate a new image from fresh noise; conditioning at each denoising step on text supports text-to-image generation.Source: §8.2 · p. 19
  • Image vectors can contain several million pixel intensities, motivating the sequential Gaussian-based approach over direct density modeling.A 1024 × 1024 RGB image has about 3.1 million pixel values.Source: §8.2 · p. 19

8.3 Discrete-Time Formulation: Gaussian Transition Probabilities

The discrete-time diffusion formulation uses Gaussian transitions to add noise to data and supports tractable reverse transitions. Its score-based interpretation connects noise prediction to the reverse process and training objective.

8.3.1 The Forward Process

  • Each forward step draws x_t from a Gaussian centered on a scaled x_{t−1}, with isotropic noise controlled by β_t.The reparameterization uses coefficients √(1−β_t) and √β_t to preserve unit variance across steps.Source: §8.3.1 · p. 20
  • Gaussian transitions are motivated by their tractability: they compose into closed-form marginals and yield a Gaussian reverse kernel.The passage also notes that cumulative small independent perturbations tend toward a Gaussian by the central limit theorem.Source: §8.3.1 · p. 21
  • Composing Gaussian transitions gives a closed-form marginal for x_t given x_0, allowing direct sampling without simulating every intermediate step.As t approaches T and ᾱ_t approaches zero, this marginal converges to N(0, I).Source: §8.3.1 · p. 21
  • The reverse transition is also Gaussian when forward transitions are Gaussian and β_t is small, making training simpler and sampling well-defined.The text attributes this reverse-kernel result to a standard computation using Bayes’ rule.Source: §8.3.1 · p. 21
  • For Gaussian perturbations, the score is linear in x_t, so denoising score matching reduces to predicting the added noise.The conditional Gaussian score equals the noise up to a scalar factor, and noise prediction therefore estimates the score.Source: §8.3.1 · pp. 21–22
  • The score does not depend on a density’s normalizing constant and can support sample generation through Langevin dynamics.The gradient moves samples toward high-density regions while injected noise prevents collapse onto a single mode.Source: §8.3.1 · pp. 21–22

8.3.2 The Reverse Process and the Training Objective

  • The reverse kernel is approximated by a learned Gaussian, and maximizing a variational lower bound reduces to a mean-squared error on the noise.The network predicts the noise used to form x_t, equivalently estimating the score up to a scaling factor.Source: §8.3.2 · p. 22

8.4 From Unconditional Generation to Text-to-Image

Unconditional diffusion generation repeatedly denoises an initial Gaussian sample, while text conditioning guides each denoising step toward a prompt-dependent distribution. Classifier-free guidance sharpens prompt alignment at a cost in diversity.

8.4.1 Unconditional generation

  • Unconditional generation starts with Gaussian noise and repeatedly predicts the added noise to produce a sample approximating the data distribution.Without conditioning, the process does not let users specify a desired image such as a particular cat or attribute.Source: §8.4.1 · pp. 22–23

8.4.2 Conditioning on text

  • For text-to-image generation, a text encoder converts the prompt to an embedding used alongside the noisy image to predict noise during training and generation.Training uses image-caption pairs; the resulting output is a sample from p(x | c).Source: §8.4.2 · p. 23

8.4.3 Why this works: the score interpretation

  • Noise prediction estimates the text-conditioned score, which guides each denoising step toward images more likely under the prompt.Small score-guided changes accumulate over the reverse process.Source: §8.4.3 · p. 23

8.4.4 Classifier-free guidance

  • Classifier-free guidance combines predictions made with and without the prompt, sharpening prompt alignment at the cost of some diversity.Without guidance, text-to-image models tend to produce images only loosely related to the prompt.Source: §8.4.4 · p. 23

8.4.5 Latent diffusion

  • Stable Diffusion runs diffusion in a lower-dimensional latent representation and decodes the final latent to pixel space, making training and sampling more efficient while preserving perceptual quality.Source: §8.4.5 · p. 24

8.4.6 Summary

  • The same reverse-diffusion machinery supports unconditional image generation, text-to-image, image editing, super-resolution, and other tasks.Text-to-image generation guides the noise-to-data transformation at each step with a text embedding.Source: §8.4.6 · p. 24

8.5 Continuous-Time Limit: Stochastic Differential Equations

Taking the discrete diffusion limit yields a continuous-time stochastic differential equation description with a reverse-time process. A deterministic probability flow ODE shares the SDE’s time-specific marginals while following different paths.

8.5 Continuous-Time Limit: Stochastic Differential Equations

  • Taking T to infinity while β_t tends to zero gives a continuous-time description of diffusion in terms of stochastic differential equations.The passage attributes this formulation to Song et al..Source: §8.5 · p. 24
  • The forward SDE is specified by a drift, a scalar diffusion coefficient, and Brownian motion.Source: §8.5 · p. 24
  • Anderson’s theorem gives a reverse-time SDE whose drift depends on the marginal density’s score.Source: §8.5 · p. 24
  • The probability flow ODE has the same marginal law as the SDE at every time, but the two processes do not share paths.Solved backward from Gaussian noise, the ODE produces data samples and yields exact log-likelihoods through change of variables.Source: §8.5 · p. 24

8.6 Current Usage and comments

Diffusion models are used across image, audio, video, medical, and scientific applications. The paper links their appeal to training stability, mode coverage, and a probabilistic framework connecting discrete and continuous formulations.

8.6 Current Usage and comments

  • Diffusion models have achieved high-quality results in image generation, editing, super-resolution, medical imaging, audio and speech synthesis, video, reinforcement learning, and computational biology.Source: §8.6 · p. 24
  • Their appeal is associated with training stability, mode coverage, and links to score matching, Langevin dynamics, and continuous normalizing flows.Source: §8.6 · p. 24
  • Gaussian transitions provide tractable closed-form marginals and a Gaussian reverse kernel that a neural network can learn.The paper presents discrete DDPM as a practical realization and the continuous-time SDE as its mathematically natural limit.Source: §8.6 · p. 25
  • Text-to-image systems add a text encoder and classifier-free guidance to the same diffusion machinery to generate images conditioned on prompts.Source: §8.6 · p. 25

Related papers

Loading 2609.25134v1…