Source-linked AI summary
A Latent Space Theory for Emergent Abilities in Large Language Models
Hui Jiang
TL;DR
The paper addresses how LLMs can infer meaning despite unknown latent intentions and unavailable true language distributions. It develops a sparse latent-space account, arguing that LLMs’ language-marginal estimates support Bayesian inference underlying emergent abilities, while its theory is asymptotic and may not apply to small models.
Problem
The latent intention space and the true distributions over intentions and messages are unavailable, limiting direct inference about language meaning.
Source: §2 · p. 3Method
The paper models messages as generated from latent intentions, classifies languages by ambiguity, and uses LLM approximation of language marginals to analyze inference over sparse joint distributions.
Source: §1 · pp. 1–2, §2 · p. 3, §3 · p. 4Results
Takeaways & Limitations
The account links language understanding and few-shot in-context learning to inference over latent intentions, with more relevant examples reducing the error bound for ambiguous languages.
Source: §4 · p. 5, §5 · p. 6The results rely primarily on an asymptotic setting; for small models, the distribution gap can be arbitrarily large, making the account inapplicable.
Source: §9 · p. 10
Abstract
from arXiv · showhide
Languages are not created randomly but rather to communicate information. There is a strong association between languages and their underlying meanings, resulting in a sparse joint distribution that is heavily peaked according to their correlations. Moreover, these peak values happen to match with the marginal distribution of languages due to the sparsity. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages, providing a convenient means of exploring the sparse structures in the joint distribution for effective inferences. In this paper, we categorize languages as either unambiguous or {\epsilon}-ambiguous and present quantitative results to demonstrate that the emergent abilities of LLMs, such as language understanding, in-context learning, chain-of-thought prompting, and effective instruction fine-tuning, can all be attributed to Bayesian inference on the sparse joint distribution of languages.
1 Introduction
The introduction frames LLM emergent capabilities as new abilities arising with scale, whose development is difficult to explain given next-token pretraining. It proposes a latent-space account in which Bayesian inference over sparse language–meaning distributions explains these abilities.
1 Introduction
- LLMs exhibit emergent capabilities—including semantic understanding, in-context learning, chain-of-thought prompting, and instruction fine-tuning—as model size and training data increase.The introduction distinguishes these abilities from simply extending existing skills to a new data distribution.Source: §1 · p. 1
- The paper proposes a latent-space theory that uses LLMs’ approximation of language marginals to explore sparse joint language–meaning distributions.The authors contrast their focus on general data distributions with Xie et al.’s HMM-specific setting [29].Source: §1 · p. 1
- The authors categorize languages as unambiguous or ε-ambiguous and attribute emergent abilities to Bayesian inference on sparse joint distributions.They also report synthetic-language simulations validating their theoretical findings.Source: §1 · p. 2
2 A Latent Space Model for Language Generation
The section models language as messages generated to express latent intentions, then distinguishes languages by how reliably a message identifies its intention. Its account relies on sparse joint distributions and frames LLM inference as a way to address unknown intentions and unavailable true distributions.
2 A Latent Space Model for Language Generation
- A message is a sequence of symbols intended to express one intention θ, sampled from a prior before the message is generated conditionally on that intention.The intention space Θ is assumed discrete, complete, and composed of distinct intentions.Source: §2 · p. 2
- An unambiguous message identifies its intention with certainty, while an ε-ambiguous message identifies it with high confidence bounded by ε.The posterior condition is expressed as Pr(θ0|x) ≥ 1 − ε(x), with the corresponding intention-to-residue ratio bounded below by 1/ε(x).Source: §2 · pp. 2–3
- The model treats practically useful languages as unambiguous or dominant, because messages that do not support confident inference fail to communicate; natural-language misunderstanding is described as bounded.The authors note that additional aligned messages can improve the likelihood of adequate communication.Source: §2 · pp. 2–3
- For a composite segment of messages, the latent model sums over their intentions, and the term for the actual intentions dominates for unambiguous or ε-ambiguous languages.The section also states that the joint distribution is sparse for either language category.Source: §2 · p. 3
- Applying the latent-space model is difficult because the complete intention space and the true distributions over intentions and messages are unavailable.Earlier statistical models could only roughly estimate those distributions under computational constraints.Source: §2 · p. 3
- The paper aims to explain how LLM practices improve inference without explicit knowledge of Θ, an improvement associated with emergent abilities as model and data scales increase.The authors connect this aim to inference performance rather than requiring an explicitly specified latent space.Source: §2 · p. 3
3 LLMs are a universal density approximator of marginal distribution q(x)
The section argues that LLMs can approximate the marginal distribution of language by modeling conditional probabilities over token histories. With a universal density approximator trained by maximum likelihood, the model’s distribution converges toward the true marginal as data grows.
3 LLMs are a universal density approximator of marginal distribution q(x)
- Factoring q(x) into conditional probabilities over successive symbols shows that modeling each history-conditioned probability can recover the marginal for messages up to length T.The model notation p_Λ(x|h) denotes the conditional statistical model, with h a history of at most T−1 symbols.Source: §3 · p. 4
- A universal density approximator can represent any distribution q(x) to arbitrary precision, and a sufficiently large transformer is a universal function approximator.The section combines universal function approximators with softmax to model the conditional distribution q(x|h).Source: §3 · p. 4
- A universal density approximator trained by maximum likelihood on n i.i.d. samples from q(x) converges toward q(x) under Theorem 1’s conditions.The section relates this result to consistency of maximum likelihood estimation.Source: §3 · p. 4
- Under the stated idealized LLM setup, the model can accurately assess the language marginal q(x) for messages up to T symbols.The setup assumes a GPT-structured universal density approximator and maximum-likelihood parameters learned from infinitely many messages generated by the latent-space process.Source: §3 · p. 4
4 Language Understanding with LLMs
The section explains prompt-conditioned generation by treating next-token training as estimation of language marginals and deriving reply behavior from a hidden intention. For unambiguous languages, the sampled reply process matches the underlying one; with ε-ambiguity, deviation is bounded by prompt ambiguity.
4 Language Understanding with LLMs
- The section frames next-token training as maximum-likelihood estimation of the language marginal, despite lacking an explicit link to semantic meaning.The authors use this setup to explain how LLMs understand prompts and generate coherent responses.Source: §4 · p. 5
- In unambiguous languages, LLM replies follow the same generation process conditioned on the prompt and its hidden intention.The derivation assumes an unambiguous generation process and relies on its sparsity property.Source: §4 · p. 5
- For ε-ambiguous languages, the reply-distribution difference from the ideal language process is bounded by prompt ambiguity ε(x).The passage links well-crafted prompts to a smaller error margin and more proficient responses [12] [21].Source: §4 · p. 5
5 In-Context Learning in LLMs
The paper analyzes few-shot learning under a shared-intention setting, distinguishing unambiguous from ε-ambiguous languages. It shows exact agreement with the ideal conditional distribution in the unambiguous case, while ambiguity bounds the deviation and more examples reduce the error.
5 In-Context Learning in LLMs
- Under the shared-intention setup, the instruction and independent input-output examples are generated under the same task intention, with examples reflecting the task correctly.The paper compares unambiguous and ε-ambiguous languages under this setting.Source: §5 · p. 6
- For unambiguous languages, the LLM’s conditional distribution given the prompt, examples, and new input equals the ideal distribution under the task intention.The paper notes that additional examples help little in this case.Source: §5 · p. 6
- For ε-ambiguous languages, the LLM’s deviation from the ideal distribution is bounded by the ambiguities of the prompt, examples, and new input.The bound’s ε0 is the maximum ambiguity among the prompt and examples; the error bound decreases exponentially with the number of examples.Source: §5 · p. 6
6 Chain-of-Thought Prompting and Fine-tuning in LLMs
The paper models chain-of-thought prompting as conditioning on intermediate intentions in a coherent reasoning sequence. This can make the final transition more likely than a direct jump and can support generalization across arguments sharing that sequence.
6 Chain-of-Thought Prompting and Fine-tuning in LLMs
- The account treats a coherent chain of thought as a sequence of messages generated under successive intentions.The paper represents the sequence as θ0 → θ1 → ··· → θm.Source: §6 · p. 6
- A direct transition from the initial intention to a distant conclusion may be rare in the training corpus, whereas conditioning on intermediate reasoning steps makes the final transition more likely.The paper attributes this to the greater frequency of the immediate transition when more of the reasoning sequence is conditioned on.Source: §6 · pp. 6–7
- Arguments with different surface messages can share the same intention sequence, and increasing that sequence’s probability can also increase the probability of generating those arguments.The paper links this shared latent sequence to generalization across unseen contexts that follow the same chain of thought.Source: §6 · p. 7
7 Aligning LLMs to Follow Instruction
The paper models instruction responses as a mixture over intentions related to the prompt intention, then describes fine-tuning as reducing probabilities of bad responses. It identifies selective adjustment of intention transitions as a desirable strategy, but says unknown intentions make that adjustment unclear in practice.
7 Aligning LLMs to Follow Instruction
- An instruction prompt leads to responses through a mixture over intentions, with response intentions weighted by their transition probabilities from the prompt intention.With sufficient training data, transitions tend to be higher for intentions strongly related to the prompt, whether through causation or association.Source: §7 · p. 7
- Human-labeled good and bad responses are used in reinforcement learning to reduce the probabilities of bad responses, and changing one intention transition can affect many prompts sharing that intention.The paper presents this as an explanation for the effectiveness of instruction fine-tuning.Source: §7 · p. 8
- Because intentions are unknown, it remains unclear how to adjust only intention-transition probabilities without affecting intention-conditioned language distributions.The paper identifies this selective adjustment as a future research direction.Source: §7 · p. 8
8 Simulation Experiments
Synthetic-language simulations test the theory’s predictions about language understanding and in-context learning, alongside transformer convergence toward the generated distribution. The reported patterns vary with language ambiguity and, for ambiguous languages, with the number of examples.
8 Simulation Experiments
- The simulations generate unambiguous and ε-ambiguous languages with a doubly-embedded Markov chain, varying ambiguity by adding noise to sparse transition matrices.The model uses a six-state circular intention chain and intention-specific message chains over 18 alphabetic states.Source: §8 · p. 8
8.1 Convergence of LLMs
- The GPT-like transformers’ distribution gap on an independent validation set shrinks quickly with more training data, with larger block sizes modeling the true distribution slightly better.The character-level model has three layers and about 21 million parameters.Source: §8.1 · p. 8, Figure 1 · p. 9
8.2 Language Understanding with LLMs
- For language understanding, the average KL divergence is small for unambiguous languages and increases slightly as language ambiguity rises.Figure 2 presents the language-understanding comparison on the left.Source: §8.2 · p. 9, Figure 2 · p. 9
8.3 In-Context Learning in LLMs
- For ε-ambiguous languages, the average KL divergence in in-context learning can be reduced by adding more messages, while it remains small and constant for unambiguous languages.The reported divergence increases for ε-ambiguous languages before adding more messages.Source: §8.3 · p. 9, Figure 2 · p. 9
9 Final Remarks
The paper’s latent-space theory attributes LLM emergent abilities to Bayesian inference over sparse language–intention distributions, with synthetic simulations reported as confirmation. Its appendices also describe tying two messages to a common intention and define a residual term for unambiguous languages.
9 Final Remarks
- The theory attributes LLM emergent abilities to exploring sparse joint distributions as a universal approximator of the marginal distribution, with synthetic simulations confirming findings on ambiguity.The paper classifies languages as unambiguous or ε-ambiguous.Source: §9 · p. 9
A Proof of Proposition 1
- Treating two independent messages as one composite message with one inferred intention restricts analysis to a latent subspace where their intentions are tied.Source: A Proof of Proposition 1 · p. 13
B The sparsity property of the joint distribution of languages
- The proof defines q(Θ\θ0, x) as the total residue after removing the contribution of intention θ0.Source: B The sparsity property of the joint… · p. 13
C Proof of Theorem 1
The theorem proof uses universal density approximation and maximum-likelihood consistency to connect sufficiently large trained models to the target distribution. The proposition proofs apply sparsity to bound ambiguity and analyze in-context learning under a shared-intention assumption.
C Proof of Theorem 1
- Universal density approximation makes the target distribution achievable, while maximum-likelihood consistency supports convergence of objective values for sufficiently large models under over-parameterization.The proof notes that neural networks may fail the identification requirement and uses a canonical model space where it holds.Source: C Proof of Theorem 1 · p. 14
D Proof of Proposition 2
- For ε-ambiguous languages, the proof bounds δ between 0 and ε(x).Source: D Proof of Proposition 2 · p. 15
E Proof of Proposition 3
- The in-context-learning proof assumes the prompt and independent examples share one intention, while the reply depends on all of them; sparsity removes contributions from other intentions.Source: E Proof of Proposition 3 · p. 15
F Proof of Proposition 4
The Proposition 4 proof treats the prompt and in-context examples as sharing a common intention, then extends its derivation to ε-ambiguous languages. Figure 3 depicts the six-state circular Markov chain used to simulate the intention prior.
F Proof of Proposition 4
- The in-context-learning setup ties the prompt and examples to a common intention, and the proof derives a result for ε-ambiguous languages.The appendix identifies ε0 as the maximum ambiguity among the prompt and examples.Source: F Proof of Proposition 4 · pp. 15–16
- Figure 3 uses a six-state circular Markov chain to simulate the prior intention distribution q(θ).Source: Figure 3 · p. 17
G.1 Data Generation
The synthetic-language experiments generate messages from intention-conditioned Markov chains, varying transition noise to control ambiguity. The study trains a character-level GPT-like model on these generated languages.
G.1 Data Generation
- Six states represent distinct intentions, and a circular Markov chain selects the prior intention for synthetic-language generation.Source: §G.1 · p. 17
- Transition noise controls ambiguity: messages use intention-specific letter groups without noise, while added noise allows letters to jump beyond their assigned groups.Generated messages have 20 letters followed by a newline.Source: §G.1 · p. 17
G.2 Model Training
- The experiments train a character-level GPT-like model on the generated languages using ADAM and an automatically decayed learning rate.Table 1 is identified as containing the GPT structure and optimizer hyperparameters.Source: §G.2 · p. 18, Table 1 · p. 19