PaperlayerView exact revision on arXiv

Source-linked AI summary

Language models suffer from a curse of ambiguity

Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

arXiv:2608.15448v1cs.CLcs.LGcs.NE

TL;DR

Sampling-dependent language-model improvement makes accurate learned distributions important, but the paper asks why some next-token distributions are harder to learn than others. Through theoretical analysis and validation on controlled synthetic tasks and real language models, it finds that greater ambiguity consistently makes distributions harder to learn.

  • Problem

    Sampling-based language-model improvement requires faithful and diverse learned distributions, yet the effect of next-token ambiguity on learnability remains insufficiently characterized.

  • Method

    The paper models autoregressive prediction as context-to-token classification, analyzes ambiguity’s architectural and optimization roots theoretically, and validates them empirically.

  • Results

    Theoretical predictions are confirmed in controlled synthetic settings, with the same ambiguity-related signatures appearing in language models trained on real data.

  • Takeaways & Limitations

    The curse of ambiguity offers a mechanistic perspective on learning dynamics and highlights why distributional diversity matters for sampling-dependent model improvement.

  • Takeaways & Limitations

    In natural data, structural similarities between contexts can enable shared compressed mechanisms, making the empirical picture less clean than in synthetic settings.

Abstract

from arXiv · show

Large language models increasingly rely on sampling as a driver of their own improvement, making the fidelity of their learned distributions more critical than ever. Yet, not all distributions are equally easy to learn. In this work, we identify a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately. Through an extensive theoretical analysis, we trace this curse to architectural and learning roots. More ambiguous distributions require more capacity to be stored, larger embeddings to be represented, more steps to be fitted, and amplify token-sampling noise. We validate these findings on synthetic tasks with controlled ground truth and observe the same signatures in language models trained on real data. Our results provide a new perspective on the statistical capabilities of large language models and a practical framework for when to trust their output distribution.

1 Introduction

The paper identifies a curse of ambiguity: more ambiguous next-token distributions are harder for neural networks and large language models to learn accurately. It traces this difficulty to architectural and learning factors and validates the resulting predictions in synthetic tasks and real-data language models.

  • Motivation: Autoregressive language models sample conditional next-token distributions, but greedy decoding can undermine methods that depend on diverse continuations.Inference-time scaling succeeds when probability is spread across diverse continuations rather than concentrated on top tokens.
  • Core phenomenon: The paper formalizes the curse of ambiguity: the more ambiguous the next-token distribution, the harder it is to learn.The phenomenon is presented as applying to large language models and, more broadly, neural networks producing discrete probability distributions.
  • Mechanisms: More ambiguous distributions require greater storage capacity, larger embeddings, more optimization steps, and amplify noise from token sampling.These are identified as architectural roots—capacity and embeddings—and learning roots—optimization and sampling noise.
  • Empirical validation: Controlled synthetic experiments confirm the theoretical predictions, while language models trained on real data exhibit the same signatures.The authors argue that these empirical patterns extend to larger models.

2 Autoregressive language modeling as context classification

The paper frames autoregressive next-token prediction as context-to-token classification: a Transformer maps each context to an embedding, and a linear softmax head predicts the next-token distribution. It analyzes ambiguity through the support size of uniform target distributions, focusing theoretically on the head with fixed context representations.

  • From sequences to classification: Autoregressive language modeling factorizes sequence probabilities into conditional next-token distributions given preceding contexts.This reframes sequence modeling as a collection of context-dependent classification problems.
  • From sequences to classification: A Transformer backbone maps each context to an h-dimensional embedding, while a linear head and softmax produce probabilities over a vocabulary of size d.The head performs multi-class logistic regression on the backbone representation.
  • From sequences to classification: Training jointly fits the backbone and head by minimizing average cross-entropy between ground-truth and predicted distributions.The linear head also modulates gradients flowing back into the backbone.
  • Focusing on the linear head for analytical tractability: For analytical tractability, Sections 3 and 4 study the linear head in isolation while treating context embeddings as given.The simplified setup uses n context embeddings paired with target distributions and examines how the head handles varying ambiguity.
  • Modeling ambiguity through support size: Ambiguity is modeled by making each target distribution uniform over a random support of size k, where k = 1 is unambiguous and k > 1 represents equally likely continuations.Larger support spreads probability mass across more possible next tokens and makes prediction intrinsically harder in the model.

3 Architectural roots of the curse of ambiguity

Ambiguous next-token distributions are harder to represent because they require more storage capacity and larger embedding dimensions. Linear-model analyses trace these architectural costs to interference among supported tokens and the limited vocabulary directions available in the embedding space.

  • Capacity: Embedding interference increases with support size k, raising prediction noise and limiting how many context-dependent distributions the model can store.The noise variance grows with the number of patterns n and support size k because each target affects k output coordinates.
  • Capacity: More ambiguous distributions require more capacity to store accurately in linear-softmax models.A size-k distribution requires k times more capacity than a deterministic one, with the k scaling tight up to logarithmic factors.
  • Embedding dimension: Compressed-sensing and convex-polytope analyses establish embedding dimension as a representational root of the curse of ambiguity.The analysis quantifies the threshold using k-neighborliness, including Radon-type upper bounds and cyclic-polytope constructions.

4 Learning-based roots of the curse of ambiguity

Ambiguity creates a learning-based curse: higher-ambiguity distributions take longer to optimize and are intrinsically noisier. Gradient-flow analysis identifies slower early learning and ambiguity-dependent late-training loss, while label sampling adds a noise floor that causes stochastic-gradient plateaus.

  • 4.1 Learning dynamics: Higher ambiguity slows initial learning because each output token is observed less frequently.In the analyzed linear model, gradient flow provides a proxy for training dynamics and shows this slowdown directly.
  • 4.1 Learning dynamics: During early learning, the logit gap grows at rate 1/k, making the initial learning rate decrease with support size k.The dynamics reduce to a scalar logit-gap trajectory for a uniform target over support size k.
  • 4.1 Learning dynamics: Late-training convergence follows 1/t independently of k, but its prefactor scales approximately linearly with support size k for k ≪ d.Thus, at any fixed training time, more ambiguous distributions remain at higher loss.
  • 4.2 Output sampling noise: Label-sampling noise leaves the mean logit trajectory unchanged at leading order, scattering logits around the gradient-flow path instead of slowing convergence.Finite-step stochastic gradient descent can nevertheless exhibit rate slowdowns, as noted in the paper’s limitation.
  • 4.2 Output sampling noise: The sampling-induced noise floor is zero for deterministic targets and approaches 1/4 as support size k tends to infinity.Early training is dominated by the 1/k rate; later, stochastic gradient descent plateaus at the noise floor while gradient flow continues decreasing.

5 The curse of ambiguity in large language models

The paper establishes that transformers, including large language models on natural data, suffer from a curse of ambiguity: at comparable context frequency, more ambiguous next-token distributions are harder to model. Synthetic experiments with known ground truth and targeted ablations reproduce the theoretical signatures across capacity, training duration, embedding dimension, and sampling noise.

  • Synthetic language modeling: A synthetic word-level 3-gram task uses exactly known conditional distributions to measure prediction errors against ground truth.The default model is a 4-layer transformer with 4M parameters trained on 100M tokens.
  • Synthetic language modeling: Small transformers suffer from the curse of ambiguity when context complexity is characterized by effective support size and context frequency.The previous two tokens determine the relevant context, while context frequency summarizes how often that context appears in training data.
  • Synthetic ablations: Capacity also improves performance uniformly, whereas larger embeddings and removing sampling noise change the ambiguity-dependent learning behavior.Capacity saturates for the largest models when training duration is fixed; the ablations compare sampled cross-entropy with full KL minimization.
  • Synthetic ablations: Increasing training steps shifts iso-loss contours uniformly, consistent with a late-phase convergence rate of 1/t independent of effective support size.The contours are evaluated at DKL = 0.5 in effective support size and context frequency.
  • Natural-data language models: On natural data, OLMo-2 1B is harder to model on more ambiguous contexts at fixed semantic context frequency, using OLMo-2 13B as a ground-truth proxy.Both effective support size and semantic context frequency are estimated from the larger model because natural-data ground truth and exact context frequencies are unobservable.

6 Discussion … A.5 Matching upper bound via sign-pattern counting

The paper argues that ambiguity limits how accurately discrete probability distributions can be stored and learned, while controlled toy models expose mechanisms that remain visible, though less cleanly, in large language models and broader neural networks. Its theoretical results characterize top-k capacity through matching lower and upper bounds, with capacity degrading as support ambiguity increases.

  • 6 Discussion: Toy models expose controlled complexity axes that identify useful setups, observables, and ablations for studying learning phenomena at scale.The discussion places this approach within broader work on the physics or mechanics of learning.
  • 6 Discussion: Natural-data structure makes the curse of ambiguity less clean in large models because similarities between contexts can accelerate learning beyond context frequency.The analysis implicitly assumes that contexts are learned proportionally to their frequency.
  • 6 Discussion / Appendix: The theoretical arguments apply to any neural network that produces discrete probability distributions, not only to language models.The discussion suggests ambiguity may matter even more in smaller-scale settings.
  • Notation / A.2 Setup, Hebbian model and signal-noise decomposition of the model’s logits: The setup represents each context with an h-dimensional embedding and a target distribution uniform on a support of size k, with ambiguity level ρ = k/d.The model predicts ˆpi = softmax(Wei), while cross-context noise has per-component variance σ2 = (n −1)ρ(1 −ρ)/h.
  • A.1 Definitions of top-k and ε capacity / A.4 Formal statement and proof for ε-capacity: ε-capacity is more demanding than top-k capacity because it requires both correct support identification and nearly equal probabilities within the support.For k = 1, correct support identification can drive the KL divergence to zero as β →∞; the nontrivial regime is k ≥2.
  • A.3 Top-k capacity of the Hebbian model: Top-k capacity of the Hebbian construction scales as h/[ρ(1 −ρ)] up to logarithmic factors, and storing sparse support of size k costs approximately k times more capacity than deterministic association.At maximal ambiguity, k = d/2 gives n ≲ h/log(hd/δ), while k = 1 gives n ≲ hd/log(hd/δ).
  • A.4 Formal statement and proof for ε-capacity: The ε-capacity analysis controls KL error through on-support logit variation and the margin separating support from nonsupport.The proof combines noise bounds with this deterministic KL-control relation to guarantee accurate modeling for every context under the theorem’s conditions.
  • A Theoretical results regarding capacity / A.5 Matching upper bound via sign-pattern counting: A sign-pattern counting argument gives a matching upper bound for any linear-softmax model: with high probability, no W ∈Rd×h stores substantially more than roughly h/[ρ(1 −ρ)] patterns.The argument bounds realizable labelings by sign patterns of linear comparisons in W and compares them with uniformly random target labelings.

A.6 Validation of theory

Experiments validate the theory’s predicted dependence of learned linear-head capacity on ambiguity, while showing that the Hebbian construction provides the correct scaling but only a lower bound. Capacity also grows with parameter count as predicted, under controlled top-k accuracy experiments.

  • Capacity scaling: Learned linear heads achieve higher capacity than the Hebbian construction, but differ by a roughly constant factor of 7–15 and scale similarly with k.This empirically supports the construction’s ambiguity dependence while preserving Theorem 1’s lower-bound limitation.
  • Capacity scaling: Capacity grows proportionally to the number of parameters hd, with the 1/k dependence predicted by Theorem 1.Figure 7 sweeps h = d ∈ {32, 64, 128, 256} for k ∈ {1, 4, 16}.
  • Experimental setup: The validation trains a linear head with h = d = 128 on exact KL divergence to uniform size-k targets and evaluates top-k accuracy.Top-k accuracy is the fraction of patterns whose top-k predictions exactly match the target distribution’s support.

B Theoretical results regarding embedding dimension · B.1 A primer on convex polytopes

The appendix characterizes the embedding dimension required for linear-softmax models to represent uniform distributions and motivates the result through convex-polytope neighborliness. Exact representation requires h ≥ 2k, while robustness to embedding perturbations costs a logarithmic factor in d/k.

  • B Theoretical results regarding embedding dimension: The analysis treats output weights and context embeddings as free parameters, with representation defined by predictions converging to targets as β →∞.The weights and embeddings must remain independent of β; only temperature varies to sharpen the softmax.
  • B Theoretical results regarding embedding dimension: h ≥2k is necessary and sufficient for representing every family of uniform distributions on size-k supports of [d], assuming d ≥2k + 2 and d ≥h + 1.This refines the general softmax-bottleneck lower bound for the specific family of uniform support distributions.
  • B Theoretical results regarding embedding dimension: Embedding robustness is recoverable with random weights and an embedding dimension exceeding 2k by a factor of order log(d/k).The resulting shared-per-support embeddings tolerate bounded adversarial context-dependent perturbations while preserving predictions within a small KL-divergence radius.
  • B.1 A primer on convex polytopes: A convex polytope is the convex hull of finitely many vertices, and a face is the subset exposed by a supporting hyperplane with a common maximal alignment.The exposing vector is called the witness vector of the face.
  • B.1 A primer on convex polytopes: An m-neighborly polytope has every subset of at most m vertices as a face, so neighborliness measures how many vertex subsets form faces.For example, every pair forms an edge in a 2-neighborly polytope, whereas the cube is only 1-neighborly.
  • B.1 A primer on convex polytopes: Any convex polytope in R^h with more than h + 1 vertices is at most ⌊h/2⌋-neighborly.This classical threshold follows from Radon’s theorem, which forces intersecting convex hulls among sufficiently many points.
  • B.1 A primer on convex polytopes: For every h and d ≥h + 1, cyclic polytopes attain the upper bound ⌊h/2⌋-neighborliness using vertices on the moment curve ϕ(t) = (t, t^2, . . . , t^h).Their face structure can be analyzed through degree-at-most-h polynomials that vanish on selected vertices and remain positive elsewhere.

B.2 Reduction to neighborliness and proof of Theorem 7

The section reduces exact representation of all uniform size-k support distributions by a linear-softmax model to k-neighborliness of a d-vertex polytope. Combining this equivalence with cyclic-polytope bounds proves representation is possible exactly when h ≥ 2k.

  • Required structure of the logits: As β → ∞, softmax logits yield the uniform distribution on maximizers, requiring equal logits on each support and strictly smaller logits off support.For support S_i, w_j^⊤e_i must equal a_i for j ∈ S_i and be less than a_i otherwise.
  • Reduction to a k-neighborliness problem: These logit conditions are exactly the statement that every size-k subset of rows of W forms a face, so representation is equivalent to a k-neighborly polytope with d vertices.The rows define the polytope, while each embedding is a face witness.
  • Reduction to a k-neighborliness problem: Conversely, a k-neighborly d-vertex polytope supplies row weights and face-witness embeddings whose sharpened softmax outputs converge to the uniform distribution on every size-k support.The common face value occurs on the support, with strictly smaller values elsewhere.
  • Proof of Theorem 7: h ≥ 2k is sufficient via the ⌊h/2⌋-neighborliness of the cyclic polytope, while h < 2k is impossible because no d-vertex polytope in R^h is k-neighborly.The arguments use d ≥ h + 1 for the cyclic-polytope construction and d ≥ 2k + 2 for the impossibility result.

B.3 A robust version of the representation result

The robust version replaces exact representation with bounded embedding perturbations and a KL-error slack under unit-norm centers and bounded output weights. It uses geometric margins, on-support logit variation, and robust null-space properties to obtain perturbation-tolerant representations with random weights, at an additional log(d/k) embedding-dimension cost.

  • Robust representation: Robust representation requires every perturbed unit-norm support embedding to remain within KL slack ε under perturbations of norm at most τ.Output-weight rows have norm at most one, making τ a scale-free precision parameter.
  • Limitation of the exact construction: The exact construction fails robustly because its support-complement margin shrinks rapidly with support size as the witnessing polynomial develops double roots at on-support vertices.Infinite inverse temperature hides this issue only in the exact setting, where any positive margin can be amplified.
  • Proof strategy: The analysis reduces robust representation to the weight polytope’s support-separation margin and on-support logit variation, then connects these quantities to robust null-space properties.The regularized witness controls both separation and center norm, while the deterministic lemma transfers null-space guarantees to uniform geometric bounds.
  • Random weights: Random subgaussian weights satisfy the required robust null-space property with high probability once the embedding dimension scales as h ≳ k log(d/k).The argument uses restricted-isometry guarantees for a normalized subgaussian measurement matrix.
  • Robust representation: Finite inverse temperature is necessary because combining β →∞ with nonzero embedding perturbations makes the KL divergence diverge.Unit-norm centers and bounded output weights prevent rescaling from making τ artificially negligible.
  • Exact versus robust regime: An additional log(d/k) embedding-dimension factor lets Theorem 8 withstand bounded adversarial perturbations and KL slack ε, unlike Theorem 7’s exact representation requiring h ≥ 2k.The exact witness tolerates only perturbations comparable to a margin that rapidly shrinks with support size.

C Theoretical results regarding optimization · C.1 One-dimensional differential equation for softmax gradient flow · C.2 Validation of theory

The theory reduces softmax gradient flow for a uniform-over-k target to one scalar describing the on- versus off-support logit gap, revealing slower early learning and k-scaled late loss. Experiments with shared linear-head parameters and mixed ambiguity qualitatively validate these predictions below capacity.

  • C.1 One-dimensional differential equation for softmax gradient flow: Gradient flow for a single logistic-regression problem can be reduced exactly to a one-dimensional differential equation far below the capacity threshold.The subsequent asymptotic expansions assume k ≪ d.
  • C.1 One-dimensional differential equation for softmax gradient flow: For a fixed context, the analysis initializes zero logits and uses a target distribution uniform over k of d classes.Logit gradient flow is equivalent to weight gradient flow for a unit-norm embedding.
  • C.1 One-dimensional differential equation for softmax gradient flow: Equal on-support and off-support logits remain equal within their groups, so the probability dynamics reduce to the gap between those two logit values.The overall sum of logits does not affect the predicted distribution.
  • C.1 One-dimensional differential equation for softmax gradient flow: Early in optimization, learning a distribution with k possible targets takes k times longer than learning the deterministic case.This follows from the near-zero logit-gap approximation and the resulting early-time dynamics.
  • C.1 One-dimensional differential equation for softmax gradient flow: At late times, the convergence rate is 1/t for every k, while the loss grows linearly with k when k ≪ d.The late-stage approximation is obtained when exp(∆ℓ) dominates.
  • C.2 Validation of theory: The simplified fixed-target theory is tested in a linear head sharing parameters across many contexts with varying ambiguity, where it qualitatively captures learning dynamics below capacity.The validation also examines where the theory breaks down.
  • C.2 Validation of theory: The validation trains with stochastic gradient descent and mixes six ambiguity levels, k ∈ {1, 2, 4, 8, 16, 32}, equally within every batch.Each ambiguity level is assigned to a group of n contexts, producing 6n context-distribution pairs per run.
  • C.2 Validation of theory: Below the capacity threshold, per-k KL curves stack in increasing order of k and decay at similar rates, matching the theoretical prediction that ambiguity makes learning harder.The curves are plotted against normalized step step/n for different numbers of contexts.

D Theoretical results regarding sampling noise … E Connection to classical sampling results

The analysis shows that sampled labels add zero-mean, ambiguity-dependent fluctuations around gradient-flow learning, producing a noise floor while leaving leading-order convergence rates unchanged. Experiments and classical sampling results extend this perspective beyond the direct-logit theory.

  • D Theoretical results regarding sampling noise: Sampled-label updates have gradient-flow drift plus zero-mean fluctuations, analyzed in the small-learning-rate limit where the noise floor scales with η.Time is measured as t = η×(number of steps), and the diffusion term carries √η.
  • D.1 The within-on-support variance controls the excess loss: At leading order, excess KL loss is controlled by the within-on-support logit variance V, reducing the sampling-noise analysis to V’s dynamics.The derivation expands the KL divergence quadratically and uses the fluctuation structure along the gradient-flow trajectory.
  • D.2 Continuous-time limit of the stochastic update: In the continuous-time approximation, drift is η-independent while diffusion carries √η, so learning rate controls wandering around gradient flow rather than travel speed.The approximation applies in the small-learning-rate limit.
  • D.3 Fluctuations live on the within-on-support contrast subspace: All gradient noise lies in the (k−1)-dimensional within-on-support contrast subspace, where damping is curvature-controlled and injected noise is proportional to η.The symmetric on-support direction and off-support coordinates receive no sampling noise.
  • D.4 Dynamics of the within-support variance and noise floor: The asymptotic noise floor is zero for k = 1, increases monotonically with k, and approaches η/4 as k →∞.Its ambiguity-dependent factor is (k−1)/(4k), while η sets the overall scale.
  • D.5 Remark: convergence rate is untouched at leading order: At leading order in η, sampling preserves the gradient-flow convergence rates and adds zero-mean fluctuations whose variance saturates at the noise floor.Finite-step stochastic gradient descent can instead exhibit Jensen-style drift corrections.
  • D.6 Validation of theory: Validation trains a shared linear head with sampled targets across contexts of varying ambiguity, using n ∈{32, 256}, 1.6·10^6 steps, and 32 seeds.The setup compares stochastic cross-entropy training with a matched distributional-target reference.
  • E Connection to classical sampling results: Classical coupon-collector and categorical maximum-likelihood results show that the curse of ambiguity also appears when sampling a single discrete distribution.These non-parametric results complement the learning-based analyses of Appendices C–D.

E.1 The coupon collector problem

Accurately modeling a distribution requires observing every token in its support, creating a coupon-collector lower bound on learning sample complexity. For uniform distributions, this imposes a k log k cost independent of the model, optimizer, or loss, while non-uniform distributions retain a logarithmic overhead tied to the effective support.

  • The coupon collector problem: A neural network must observe every token in the support S at least once to model p accurately, so collection time lower-bounds learning sample complexity.Unobserved support tokens cannot be meaningfully distinguished from off-support tokens.
  • The coupon collector problem: For uniform p over k tokens, the expected time to collect the full support follows the coupon collector problem.Non-uniform generalizations depend on the full distribution rather than one summary statistic.
  • The coupon collector problem: The non-uniform lower bound satisfies E[T] ≥1/pmin, while the logarithmic overhead in the effective number of tokens persists.The qualitative picture remains unchanged beyond the uniform case.
  • The coupon collector problem: The k log k floor means one sample per support element is insufficient, and this sampling cost applies regardless of the model, optimizer, or loss.Even seeing each token a constant number of times requires an additional logarithmic factor in k.

E.2 Maximum likelihood estimation of a categorical distribution

For categorical distributions, maximum likelihood estimation becomes statistically harder as support ambiguity grows. Full-distribution estimation requires coupon collection for finite KL risk and scales linearly with support size, whereas scalar functionals can be estimated faster.

  • Maximum likelihood estimation: The maximum likelihood estimator estimates token probabilities after all k support tokens have been observed.It is the standard non-parametric estimator for categorical probability estimation.
  • Total variation: Reaching total-variation error ε requires m ≍ k/ε^2 samples, matching the minimax lower bound over distributions on k symbols.The stated m ≥ k condition excludes the degenerate regime m ≪ k, where ℓ1 error saturates.
  • KL divergence: The unconditional expected KL divergence of the frequency estimator is infinite at every finite m until all k support tokens have been observed.Any unobserved token gives an estimated probability of zero and infinite D_KL(p ∥ p̂).
  • KL divergence: Add-constant smoothing avoids the infinite-KL issue and achieves the corresponding rate unconditionally for every m ≥ k.Its numerator scales linearly in k, matching the late-time learning loss D_KL ≈ k/t.
  • Estimation of functionals: Scalar functionals can be estimated faster: polynomial-approximation estimators reach m ≍ k/log k, versus Θ(k) for full-distribution estimation.Plug-in estimators incur an O(k/m) bias, while language models target the full next-token distribution.

F Synthetic n-gram task · G Large language models experiments

The synthetic n-gram task constructs ground-truth next-token distributions from TinyStories and evaluates decoder-only transformers under controlled architectural, training, and loss configurations. Its analyses examine ambiguity measurement, per-context losses, and iso-loss summaries, while the supplied passages provide no substantive evidence for the large-language-model experiments section.

  • F Synthetic n-gram task: The dataset uses the 2,000 most frequent TinyStories words to form approximately 1.1 million ground-truth 3-gram contexts.All selected vocabulary words occur more than 10,000 times, and out-of-vocabulary 3-grams are dropped.
  • F Synthetic n-gram task: Raw support size reproduces the same qualitative ambiguity trends as effective support size.Figure 12 replaces effective support size with raw support size on the ambiguity axis while otherwise reproducing Figure 3.
  • F Synthetic n-gram task: Training sequences are sampled autoregressively from the empirical 3-gram distribution, beginning with two beginning-of-sequence tokens.Contexts containing beginning-of-sequence tokens use unigram or bigram marginals; unseen contexts fall back to a uniform vocabulary distribution.
  • F Synthetic n-gram task: The default decoder-only transformer has 4 layers, 4 attention heads, dmodel = 256, and 4M parameters.Training uses sequences of 256 tokens, batches of 128, 3,000 steps, 98M tokens, AdamW, weight decay 0.01, and a cosine schedule.
  • F Synthetic n-gram task: The head width dembed can be smaller than the backbone width dmodel, but embeddings cannot span more than min(dembed, dmodel) dimensions.Therefore, dembed values of 512 and 1024 at dmodel = 256 do not add representational room beyond 256 dimensions.
  • F Synthetic n-gram task: Per-context evaluation samples 262,144 fresh sequences from the same 3-gram process and averages metrics over positions where each context occurs.Each Figure 3 dot represents one context, with frequency taken from the training corpus.
  • F Synthetic n-gram task: The iso-loss pipeline fits log DKL(x) ≈ a log k(x) + b log f(x) + c and reports the contour where fitted DKL equals 0.5 nats.Contexts are weighted by inverse bin counts on a 50 × 50 grid in log k and log f.
  • F Synthetic n-gram task: The ablations vary model capacity, embedding dimension, training duration, and loss while holding other settings at their defaults.Figures 14–17 expose the per-context data underlying the single iso-loss contours summarized in Figure 4.

G.1 Experiment details · G.2 In-context predictability explains the low-loss low-context-frequency band · G.3 Quantifying the curse of ambiguity at fixed semantic context frequency

The experiments compare OLMo-2 1B with a 13B reference across 5.12 × 10^7 context-target pairs, using embedding-based semantic context frequency and distributional support as predictors. Low-loss, low-frequency contexts are primarily explained by in-context repetition, while loss still rises with ambiguity at fixed semantic frequency.

  • G.1 Experiment details: The study evaluates OLMo-2 1B against an OLMo-2 13B reference on 100,000 sequences of length 512, yielding 5.12 × 10^7 context-target pairs.Both checkpoints are taken before post-training, and each model processes the sampled Dolma v1.7 sequences in bfloat16.
  • G.1 Experiment details: Semantic context frequency is estimated by kernel density over PCA-reduced 13B hidden-state embeddings, using λ = 10 and 2,000 nearest neighbors.The procedure extracts 5120-dimensional embeddings, reduces them to dimension 512, and queries nearest neighbors with FAISS.
  • G.1 Experiment details: The density proxy ranks frequent contexts reliably but is noisy on the rare tail, with correlations dropping to 0.30 and 0.24 at lower frequencies.The passages describe this as consistent with the caveat about the low-nest region.
  • G.1 Experiment details: On the synthetic 3-gram task, the same density pipeline validates semantic context frequency against known true context frequency.This provides validation of the proxy’s information content, although natural data lacks direct access to true context frequency.
  • G.3 Quantifying the curse of ambiguity at fixed semantic context frequency: Replacing effective support size with raw support size leaves the qualitative curse-of-ambiguity pattern unchanged, with R2 higher partly because the relationship is more linear.The raw-support-size ablation and its fixed-frequency visualization are reported in Figures 21 and 24.
  • G.2 In-context predictability explains the low-loss low-context-frequency band: Contexts with semantic frequency nest < 10 can still produce tightly agreeing predictions because preceding one- or two-token patterns repeat earlier in the same sequence.A generic in-context copying mechanism, such as an induction head, could support these predictions.
  • G.2 In-context predictability explains the low-loss low-context-frequency band: The low-loss band primarily reflects context-dependent predictions enabled by short-range repetition, allowing generalization on contexts barely seen or unseen during training.The paper notes that measuring the curse in practice would require controlling for this data structure.
  • G.3 Quantifying the curse of ambiguity at fixed semantic context frequency: At fixed semantic context frequency, per-context loss is positively correlated with ambiguity in log-log space for both effective and raw support-size proxies.Figures 23 and 24 quantify this relationship using squared Pearson correlations between log loss and log support measures.
Loading 2608.15448v1…