PaperlayerView exact revision on arXiv

Source-linked AI summary

The Geometry of Language Model Logits: Generic Structure and Learned Departures

Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri

arXiv:2610.18520v1cs.CLcs.AI

TL;DR

The structure of vocabulary-wide logits remains comparatively unexplored, though it determines next-token probabilities and is relevant to decoding procedures that use the vocabulary head. This paper analyzes ordered logits, compares their geometry with controlled order-statistic baselines, and evaluates spectral reconstruction. It finds a strong log-linear fit and links the spectral slope to predictive state.

  • Problem

    The structure of vocabulary-wide logits is comparatively unexplored despite determining next-token probabilities and informing top-k and top-p decoding.

    Source: §1 · p. 1

  • Method

    The paper analyzes logits in rank order, compares their spectra with controlled order-statistic baselines, and reconstructs the tail from fitted spectral parameters and an exact head.

    Source: §1 · p. 1, §3.3 · p. 4

  • Results

    Ordered logits are approximately log-linear across contexts, architectures, and scales, with R2 > 0.99 across a broad range of K; α also correlates with predictive-state measures.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Takeaways & Limitations

    A small exact head and two fitted parameters can approximately reconstruct the ranked probability spectrum, while α provides a compact summary of variation in next-token predictions.

    Source: §4 · p. 4

    The analysis focuses on pretrained decoder-only models, and reconstruction captures the ranked probability spectrum rather than tail token identities.

    Source: §4 · p. 4

Abstract

from arXiv · show

Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation \[ z_{(r)} \approx \mu - \alpha \log r. \] across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $\alpha$: trained language models exhibit $\alpha \approx 1.4$--$1.7$, sitting well above the initialization baseline ($\alpha \approx 0.2$) . Moreover, $\alpha$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.

1 Introduction

The introduction frames vocabulary-wide logit structure as comparatively unexplored despite its role in determining next-token probabilities and informing top-k and top-p decoding. It presents ordered logits as approximately log-linear and proposes controlled order-statistic baselines and predictive-state associations to distinguish generic structure from learned features.

1 Introduction

  • Vocabulary-wide logits determine how probability mass is distributed, but their structure remains comparatively unexplored despite extensive study of probabilities, generated text, and hidden representations.Top-k and top-p decoding operate on the highest-ranked logits, motivating analysis of the vocabulary head.Source: §1 · p. 1
  • Ordered vocabulary-head logits follow an approximately log-linear relation across prediction contexts, architectures, and model scales, emerging during pretraining.The introduction asks how much of this geometry comes from language models versus generic order statistics.Source: §1 · p. 1
  • Controlled distributional baselines distinguish generic order-statistic geometry from learned structure, while α is linked to uncertainty and post-hoc prediction quality.The reported predictive measures include entropy, NLL, ground-truth token probability, and ground-truth token rank.Source: §1 · pp. 1–2
  • The paper also reports that a two-parameter spectral law can reconstruct the full next-token distribution from a small exact vocabulary head.This is presented as a contribution alongside characterizing the geometry and distinguishing generic from learned structure.Source: §1 · pp. 1–2

2 Preliminaries

The preliminaries describe how a decoder-only Transformer produces vocabulary logits and how the analysis represents them in rank space. They distinguish this pre-softmax structural analysis from Zipfian decoding approaches such as Mirostat.

2 Preliminaries

  • A decoder-only Transformer maps preceding context to a hidden state and computes vocabulary logits as z_k = W_u h_k + b.Inference-time sampling may rescale logits by temperature before softmax.Source: §2 · p. 2
  • The analysis sorts logits in descending order and studies the vocabulary head in rank space, setting aside token identities.For a chosen head size K, it analyzes the ordered spectrum.Source: §2 · p. 2
  • Unlike Mirostat's use of a power-law next-token distribution for adaptive decoding, this work studies pre-softmax ordered logits and their generic or learned structure.The two-parameter law is also used to approximately reconstruct the full next-token distribution from a small exact vocabulary head.Source: §2 · p. 2

3 Results

The results characterize ordered vocabulary-head logits as approximately log-linear, compare their slope with generic order-statistic baselines, and track how the spectral regime develops during pretraining. They also test reconstruction of next-token distributions and associations between the slope and predictive state.

3.1 Log-Linear Logit Geometry

  • Across studied models and contexts, ordered vocabulary-head logits fit z(r) ≈ μ − α log r with R2 > 0.99 for a broad range of K.Figure 1(a) shows a representative spectrum and fit; normalized spectra across contexts collapse strongly in Figure 1(b).Source: §3.1 · pp. 2–3, Figure 1 · p. 2

3.2 Generic vs. Learned Structure

  • Across a broad range of K, model slopes are stable while order-statistic baselines depend on distribution; Gaussian and Exponential samples also fit well, unlike Uniform samples.Uniform samples average R2 ≈ 0.75 with α ≈ 0.001–0.020; Gaussian samples reach R2 ≈ 0.99, Exponential R2 ≈ 0.97, and Lognormal samples yield α ≈ 4.67–10.6.Source: §3.2 · p. 3, Figure 2 · p. 3
  • At the population optimum, the log-linear logit law is equivalent to an approximately Zipfian conditional next-token distribution, but this does not explain why language-model distributions should be Zipfian.The equivalence holds exactly only at the unconstrained population optimum.Source: §3.2 · p. 3
  • During Pythia pretraining, α rises from approximately 0.2 at initialization, and the normalized spectra approach the final-model shape, with the regime clearly established by step 512.Figure 3 shows mean normalized spectra across checkpoints for Pythia-410M and Pythia-1B.Source: §3.2 · p. 3, Figure 3 · p. 4

3.3 Spectral Reconstruction of Next-Token Distributions

  • At K = 500 (≤1% of the vocabulary), spectral reconstruction achieves mean Jensen–Shannon divergence 0.008 across four model families.The method retains the exact top-K logits, fits μ and α, and reconstructs the remaining ranks; Figure 1(c) depicts mean divergence against K.Source: Figure 1 · p. 2, §3.3 · p. 4
  • Across 40 model-book pairs, larger α is associated with lower entropy, NLL, and ground-truth log-rank, and with higher ground-truth token probability.The reported correlations are r = −0.332 for entropy, −0.206 for NLL, −0.209 for ground-truth log-rank, and +0.111 for ptrue.Source: §3.3 · p. 4

4 Discussion

The discussion presents ordered logit spectra as both generic in their approximate log-linear geometry and shaped by training through α. It identifies scope boundaries and unresolved questions about the learned regime and tail reconstruction.

4 Discussion

  • A small exact vocabulary head plus two fitted parameters can approximately reconstruct the ranked probability spectrum, while α summarizes variation in next-token predictions.The discussion presents this compact rank-space structure as a possible basis for approximating vocabulary-sized predictive distributions.Source: §4 · p. 4
  • The analysis covers pretrained decoder-only models, gives sufficient rather than necessary theoretical conditions, and does not reconstruct tail token identities.The origin of the learned spectral regime and why optimization drives its observed parameter range remain open questions.Source: §4 · p. 4

A Related Work

Prior work studies corpus-level token-frequency laws, generated-text statistics, decoding assumptions, and output-layer geometry. This paper instead examines the ordered vocabulary logits within each prediction context, including their development during pretraining and relation to conditional Zipf structure.

Statistical laws of language and generated text

  • The paper studies ordered logits within individual prediction contexts, unlike corpus-level token frequencies or statistical laws in generated text.It treats the log-linear law as a pre-hoc property of conditional predictions, before sampling.Source: A Related Work · p. 6

Structure of the conditional next-token distribution

  • Unlike Mirostat’s use of a Zipfian probability assumption for adaptive decoding, this work asks how such structure arises by analyzing logits before softmax.Mirostat estimates the exponent online to adapt the sampling cutoff and control perplexity.Source: A Related Work · p. 6
  • The paper separates sorting-induced generic structure from learned structure, studies slope α across predictive states and pretraining, and uses a small exact head plus fitted tail to reconstruct distributions.At the cross-entropy optimum, its logit law is equivalent to a Zipfian conditional distribution.Source: A Related Work · p. 6

Geometry of the output layer

  • Prior output-geometry work focuses mainly on representations or structure across contexts, whereas this paper identifies within-context vocabulary-logit geometry with a two-parameter descriptor.The cited complementary lines include softmax bottleneck, stolen probability, representation degeneration, anisotropy, and low-rank structure across prompts.Source: A Related Work · p. 6

Training dynamics and theoretical characterization

  • The paper adds pretraining checkpoints and theoretical characterizations linking ordered logits to conditional Zipf structure, distance-based reparameterization, tail counts, and softmax distributions.Its training analysis uses intermediate Pythia checkpoints [Biderman et al., 2023].Source: A Related Work · p. 6

B Experimental Setup

The experiments span multiple decoder-only model families and book corpora, fit a rank-logit law, assess an alternative functional form, and test normalized-spectrum collapse on other corpora. The log-linear fit is selected for parsimony and softmax tractability, not claimed as uniquely correct.

B.1 Models

  • Four decoder-only families span varied scales, vocabularies, and training procedures, with intermediate Pythia checkpoints used to track training emergence.The evaluated families are GPT-2, Pythia, Llama 3.2, and Qwen 2.5.Source: §B.1 · p. 6

B.2 Corpora

  • Ten diverse Project Gutenberg books provide roughly 20,000 tokens each, evaluated with native tokenizers, context length 128, stride 1, and top-2000 logits.The setup yields approximately 2 × 10^4 prediction contexts per book.Source: §B.2 · p. 7

B.3 Fitting Procedure

  • Per-context ordinary least squares fits z(r)=µ−α log r; the study reports slope, intercept, R2, and RMSE, with per-book aggregation and cross-book variability.Normalized spectra are standardized before averaging across contexts.Source: §B.3 · p. 7

C.1 Functional Form Selection

  • The two-parameter log-linear form reaches R2 > 0.99 and is preferred over a three-parameter power law for parsimony and a tractable Zipf-law softmax, not uniqueness.The power law’s small residual fit gain is attributed to its extra parameter.Source: §C.1 · p. 7, Table 1 · p. 7

C.2 Ablations

  • Normalized GPT-2 ordered-logit curves collapse across contexts on both WikiText-2 and French Wikipedia, extending the pattern beyond the primary model and corpus.Figure 4 shows individual context curves and each corpus mean.Source: §C.2 · p. 7, Figure 4 · p. 8

C.3 Additional Results

Across diverse model families, ordered logits remain well fit by a log-linear law with slopes in a narrow range. Increasing the rank cutoff also improves fit for order-statistic baselines, so fit improvement alone does not identify learned structure.

Consistency Across Model Families

  • R2 > 0.99 across evaluated models, whose fitted slopes remain between 1.41 and 1.70 despite differences in architecture, tokenizer, training corpus, and scale.Table 2 summarizes fitted parameters across the evaluated model families.Source: §C.3 · p. 8, Table 2 · p. 8

Dependence of goodness of fit on K

  • As rank cutoff K increases, language-model and Gaussian/Exponential baseline fits improve; baselines reach similarly high or higher R2, so the trend alone is not evidence of learned structure.The comparison is shown as R2 versus K, with error bars across books or synthetic samples.Source: §C.3 · p. 8, Figure 5 · p. 9

D Additional Training Dynamics

Pythia checkpoints show the ordered-logit regime changing during pretraining from a shallow initialization spectrum through a transient non-log-linear phase toward a larger, stable exponent. The paper also evaluates using a fitted spectral tail to reconstruct next-token distributions from a small exact vocabulary head.

D Additional Training Dynamics

  • At K=500, Pythia’s exponent rises from about 0.20 at initialization, near the Gaussian baseline’s 0.3, to final values of about 1.57–1.62.Table 3 reports exponent and fit at training stages for Pythia-410M and Pythia-1B.Source: D Additional Training Dynamics · p. 9, Table 3 · p. 9
  • Training passes through a transient phase: at step 64, R2 is 0.68 and 0.81; by step 512, α reaches 1.13 and 1.16 with R2 ≈0.99 for the two models.At step 8000, α is 1.48–1.50 and R2 ≈0.987; final α values are 1.57 and 1.62.Source: D Additional Training Dynamics · p. 9
  • The evolving regime combines stronger log-linearity with a larger, increasingly stable slope, distinguishing the trained spectra from generic or untrained ordered spectra.The analysis uses intermediate Pythia checkpoints to quantify changes during pretraining.Source: D Additional Training Dynamics · p. 9

E Spectral Approximation of Next-Token Distributions

  • For reconstruction, the first K ordered logits are retained exactly and the remaining ranks are modeled by the fitted two-parameter log-linear tail before softmax.The approximate and exact distributions are compared using KL divergence, JS divergence, and total variation over 39,520 contexts, with K in {100, 200, 500}.Source: E Spectral Approximation of Next-Token… · p. 10, §E.1 · p. 10

E.2 Model-wise Results

Across models, spectral reconstruction improves as more exact logits are retained; aggregate divergences decrease with K, and model-wise results report the remaining divergences at K = 500.

E.2 Model-wise Results

  • At K = 500, model-wise mean JS divergence ranges from 0.0058 to 0.0094 and mean TV ranges from 0.0235 to 0.0369.Source: Table 4 · p. 10, §E.2 · p. 10

E.3 Aggregate Results

  • Mean KL divergence falls from 0.0731 at K = 100 to 0.0479 at K = 200 and 0.0266 at K = 500 across four models.Source: §E.3 · p. 10, Table 5 · p. 11
  • Mean JS divergence falls from 0.0215 to 0.0145 to 0.0083, while mean TV falls from 0.0715 to 0.0518 to 0.0325 at K = 100, 200, and 500.Source: §E.3 · p. 10, Table 5 · p. 11

F Predictive-Behavior Analysis

The analysis fits ranked logits with a log-linear law and tests how its slope relates to predictive statistics across model–book pairs. It also formalizes consequences of the law while acknowledging that its origin in language models is only partially explained.

F Predictive-Behavior Analysis

G Theoretical Characterization

G.1 Relation to a Zipfian conditional distribution

The section relates the ordered-logit law to rank-distance structure and characterizes its consequences for tail counts, conditional probabilities, temperature scaling, and normalized spectra. The normalized collapse is algebraic and does not by itself establish cross-context universality.

G.1 Relation to a Zipfian conditional distribution

  • At the cross-entropy population optimum, the logit law is related to the conditional distribution, while the rank-distance bound expresses the exponential-volume condition in rank form.A larger α_h corresponds to a more selective context and a sharper candidate set under nucleus truncation.Source: §G.1 · pp. 13, 15

G.2 Tail-count form of the rank law

  • For distinct top-K logits, the log-linear law on the vocabulary head is equivalent to an approximately exponential empirical upper tail, deterministically for each context.This characterization does not require treating logits as independent random variables.Source: §G.2 · pp. 15–16

G.3 Softmax consequence: a conditional Zipf law

  • On the vocabulary head, the log-linear logit law implies an approximate Zipf distribution for context-dependent next-token probabilities.Lower temperatures sharpen the head distribution, while higher temperatures flatten it.Source: §G.3 · p. 17

G.4 Normalized spectrum collapse

  • Exactly log-linear spectra collapse to the same standardized curve regardless of μ and α, so collapse alone does not establish cross-context universality.The empirical claim is that individual contexts fit the affine form well with a stable slope.Source: §G.4 · p. 17

Related papers

This paper cites

Loading 2610.18520v1…