PaperlayerView exact revision on arXiv

Source-linked AI summary

Token embeddings violate the manifold hypothesis

Michael Robinson, Sourya Dey, Tony Chiang

arXiv:2504.01002v3cs.CLcs.AI

TL;DR

The paper asks whether LLM token embeddings satisfy the regular geometric assumptions often used to understand and model them. It develops manifold and fiber-bundle hypothesis tests and finds evidence against these structures across several open-source LLMs. The findings connect token-level irregularity with potential instability in prompts and outputs, while the tests cannot identify the specific type of singularity.

  • Problem

    LLM understanding depends on the structure of the input token space, yet regular manifold structure is often assumed without testing and may not describe token embeddings.

    Source: §1 · pp. 1–2

  • Method

    The paper develops statistical tests for manifold and smooth fiber-bundle regularity using local volume-growth and slope behavior around token embeddings.

    Source: §3.1 · p. 5, §5.2 · p. 8

  • Results

    Across several open-source LLMs, the tests frequently reject the regularity hypotheses, with evidence of non-constant dimensions, undefined intrinsic dimensions, and non-manifold token subspaces.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    Irregular token neighborhoods may help explain prompt sensitivity and make semantically equivalent prompts less stable when they contain implicated tokens.

    Source: §6 · p. 9

    The tests establish rejection of smooth regularity but do not determine the type of singularity, which must be identified post hoc if possible.

    Source: §7 · p. 10

Abstract

from arXiv · show

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed. We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token $\psi$ implies an irregularity in the token subspace in a $\psi$-neighborhood, $B(\psi)$. The structure assumed in the null is a generalization of a manifold with boundary called a \emph{smooth fiber bundle} (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.

1 Introduction

The paper challenges the widespread, often unchecked assumption that token embeddings form regular low-dimensional manifolds and introduces tests to evaluate that assumption. Across prior work and four open-source LLMs, it motivates evidence that token subspaces contain heterogeneous and irregular structure.

1 Introduction

  • The manifold hypothesis is widely assumed in AI research, although violations can undermine methods such as UMAP and should be tested statistically.The paper supplies an analogous regularity test for token embeddings rather than relying only on visual inspection.Source: §1 · pp. 1–2
  • Token embeddings form the LLM’s initial token subspace, where subtle wording differences may correspond to dramatically different topological neighborhoods.These irregularities may relate linguistically to polysemy or homonymy.Source: §1 · p. 2

1.1 Contributions

  • The paper contributes manifold and fiber-bundle hypothesis tests, evidence against both structures in several LLMs, and identification of previously unreported embedding structure.Its primary contribution is methodology for hypothesis testing on embedded points.Source: §1.1 · p. 2
  • The tests target local dimensions and slope behavior under a positive-reach assumption, distinguishing regularity from singular or high-curvature structure.The hypotheses estimate dimension within balls centered at tokens with radius r < τ.Source: §1.1 · p. 2, Table 1 · p. 2

1.2 Background and related work

  • Across GPT2, Llemma7B, Mistral7B, and Pythia6.9B, token embeddings exhibit non-constant local dimensions and tokens without intrinsic dimension, rejecting a global manifold interpretation.The multimodal dimension distribution alone rules out a connected manifold unless the space is highly disconnected.Source: §1.1 · p. 2, §1.2 · p. 4
  • The paper addresses limitations in earlier manifold testing by providing practical tests for manifolds and fiber bundles with specified embedded reach.Earlier approaches either lacked practical constants or focused on low-dimensional datasets.Source: §1.2 · p. 3

2 Mathematical background

The mathematical background defines manifolds, singularities, fiber bundles, and embedded reach as the geometric concepts underlying the paper’s tests. Volume growth around points connects local dimension and curvature to observable embedding structure.

2 Mathematical background

  • A connected manifold has neighborhoods of one common Euclidean dimension, while a point lacking such a neighborhood is a singularity.Boundaries are a standard singularity type, and manifolds with boundary allow only boundary singularities.Source: §2 · p. 4
  • For a Riemannian manifold, the volume of a radius-r ball scales with r^d, with correction terms reflecting curvature.Large correction terms indicate high curvature.Source: §2 · p. 4
  • A fiber bundle generalizes a manifold by representing each neighborhood as a product B × F, with a manifold base and a manifold-with-boundary fiber.This structure accommodates controlled boundary singularities.Source: §2 · p. 4
  • Embedded reach is the largest distance from which points have a unique closest point on the embedded space, serving as a curvature-related regularity scale.Cusp-like spaces have reach 0, while small reach corresponds to high curvature.Source: §2 · p. 4

3 Theoretical insights

The theoretical framework characterizes fiber bundles through piecewise linear volume-growth laws and uses slope changes to test regularity. It also shows that, under stated conditions, singularities can persist into transformer outputs despite context.

3.1 Volume versus radius for fiber bundles

  • For a fiber bundle, log volume versus log radius is piecewise linear, with slopes representing dimensions that cannot increase as radius grows.This yields a testable distinction from arbitrary slope changes in manifold testing.Source: §3 · p. 4, §3.1 · p. 5
  • Theorem 1 establishes the volume-growth form for smoothly embedded compact fiber bundles with reach τ, using asymptotic behavior for radii below the relevant scale.The induced volume is evaluated in the ambient Euclidean space.Source: §3.1 · p. 5
  • Fiber boundaries reduce local volume growth when a radius-r ball extends outside the space, producing a smaller slope rather than an arbitrary increase.Token counts within radius are used as Monte Carlo volume estimates.Source: §3.1 · p. 5
  • The manifold test detects any abrupt slope change, whereas the fiber-bundle test detects only slope increases in the log-log volume curve.Figure 2 contrasts these rejection criteria.Source: Figure 2 · p. 5, §3.1 · p. 5

3.2 Context does not persistently resolve singularities

  • Under Theorem 2’s sampling and context conditions, generic transformers produce a topological embedding of the context-product token space in their outputs.A sufficient regime always exists when 2d < ℓ.Source: §3.2 · pp. 5–6
  • When the tests identify token-space singularities and Theorem 2 applies, those singularities persist into LLM outputs even with large context windows.The result concerns generic transformers under the theorem’s assumptions.Source: §3.2 · p. 6

4 Methods

The method deterministically estimates local dimensions from token-to-token distances and tests for slope changes across neighboring volume scales. Multiple-testing correction is applied to both manifold and fiber-bundle hypotheses.

4 Methods

  • For each token, Algorithm 1 sorts all pairwise distances and treats neighborhood token counts as volume indices with corresponding radii.The fixed coordinates make the procedure deterministic, so repeated runs are unnecessary.Source: §4 · p. 6
  • Local dimension is estimated from slopes in the log-log progression of neighborhood indices versus sorted distances.Three-point centered differences provide the slope estimates.Source: §4 · p. 6
  • Adjacent sliding windows are compared with two-sample t-tests to test constant slopes for manifolds and decreasing slopes for fiber bundles.The procedure stores p-values and slope estimates for each token.Source: §4 · p. 6
  • Holm-Bonferroni correction is applied separately to the manifold and fiber-bundle p-value sets.This controls the multiple testing across token-level decisions.Source: §4 · p. 6

5 Results

The tests behave as expected on synthetic spaces, detecting singularities while distinguishing manifold and fiber-bundle structure. Applied to four LLMs, they find widespread violations of manifold and fiber-bundle hypotheses and conditions under which singularities may persist into model outputs.

5.1 Synthetic datasets

  • The synthetic tests correctly distinguish a manifold, a fiber bundle, and a space that is neither, while detecting their respective singularities.The sphere produces no rejections; the cusp test detects the singularity but not the boundary, while the lollipop tests identify both its boundary and dimension-transition irregularities.Source: §5.1 · pp. 6–7
  • p < 5 × 10^-8 detects the cusp singularity, but increasing vmax trades greater boundary sensitivity for lower specificity in identifying singularities.The fiber bundle hypothesis is not rejected for the cusp surface, consistent with its fiber-bundle structure.Source: §5.1 · p. 7
  • p < 10^-5 identifies neighborhoods near both lollipop singularities, and two transition points also reject the fiber bundle hypothesis.Although the exact singular points are absent because of random sampling, the tests correctly classify the lollipop as neither a manifold nor a fiber bundle.Source: §5.1 · p. 7

5.2 LLM token embeddings

  • All four LLM token spaces significantly reject the manifold hypothesis, while GPT2, Llemma7B, and Mistral7B also reject the fiber bundle hypothesis.The models show highly variable local dimensions, and fiber-bundle rejections occur more often at larger than smaller radii, consistent with the polysemy interpretation of Jakubowski et al. [2020].Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • The tested LLMs exhibit different irregular token structures: GPT2 mainly has word-initial tokens, Pythia6.9B mostly has meaningless fragments, and Llemma7B and Mistral7B combine both patterns.Despite identical token sets, Llemma7B and Mistral7B differ substantially in irregularity structure, implying different responses to the same prompt even before transformer-stage differences.Source: §5.2 · p. 8
  • Mistral7B and Pythia6.9B satisfy Theorem 2 at small-radius dimensions, whereas GPT2 and Llemma7B do not; all four satisfy it using larger-radius estimates.These results suggest that singularities may persist into outputs for Mistral7B and Pythia6.9B, and potentially for GPT2 or Llemma7B if their true bounding dimensions are smaller.Source: §5.2 · pp. 8–9, Table 3 · p. 9

6 Discussion

Across four LLM token spaces, local-dimension variation and other irregularities reject globally smooth manifold structure, while homonymy, tokenization, and training differences offer plausible sources and practical consequences.

Empirical findings

  • All four studied token subspaces reject globally smooth manifold structure; three also reject fiber-bundle structure with large reach.The evidence includes non-constant local dimension and neighborhoods without a well-defined intrinsic dimension.Source: §6 · p. 9

Consequences

  • Tokens at irregularities can destabilize de-embedding projections and geodesic distances, while dimension-0 neighborhoods are theoretically isolated from other tokens.The paper cautions that geodesic distance may not correlate with semantic distance.Source: §6 · pp. 9–10

Possible sources

  • Homonyms and polysemous word forms are among the tokens where the fiber-bundle hypothesis is rejected, linking irregularity to multiple meanings or grammatical roles.Examples include “affect,” “monitor,” “launder,” and “wins.”Source: §6 · p. 10

Model dependence

  • Different rejection patterns across LLMs suggest that training methodology leaves model-specific fingerprints, limiting direct prompt portability even for models with identical token sets.The discussion highlights possible ramifications for interacting LLMs in multi-agentic frameworks.Source: §6 · p. 10

Token properties

  • Word-initial and word-fragment tokens often lie at irregularities, while some models show unusually low fiber dimension for tokens containing whitespace or non-printing characters.The authors propose future experiments on how irregular tokens affect response variability.Source: §6 · p. 10

Open questions

  • Vocabulary changes may reduce but not eliminate singularities because tokenization joins distinct meanings, as illustrated by the asterisk symbol across footnotes, multiplication, and Markdown.The authors suggest that analyzing these irregularities may clarify limits of attention and transformer architectures.Source: §6 · p. 10

Broader implication

  • The findings extend concerns about singularities from physical modeling to natural language, whose richness and complexity do not appear to form a smooth representation.Source: §6 · p. 10

7 Limitations

The statistical tests can reject the smooth structure assumed by the null, but they do not identify which alternative singularity type explains the rejection.

Scope of inference

  • Rejections correspond only to slope changes at radii below embedded reach, and rejecting regularity does not reveal the type of singularity present.The singularity must be determined post hoc if possible.Source: §7 · p. 10

A.1 Additional discussion of the literature

Prior work often studies token embeddings under manifold assumptions, while this paper compares those assumptions with fiber-bundle structure and singular-token propagation.

A.1 Additional discussion of the literature

  • Most tokens may fail to reveal the manifold violation: the paper reports that typical vocabularies mostly do not reject the hypothesis, despite important local differences.Table 2 nevertheless shows significant topological differences between models’ local representations.Source: §A.1 · p. 16
  • Lee et al. [2025] report low intrinsic dimensions consistent with the paper’s large-radius results, but do not examine smaller-radius fiber-bundle structure.Their GPT2 dimension estimates fall within the paper’s confidence interval.Source: §A.1 · p. 16
  • Related work suggests that token subspaces may be inferred from model behavior, including responses to single-token prompts.Zhao et al. [2025b] concerns implicit activation spaces, while Robinson et al. [2024] establishes inference up to topological equivalence.Source: §A.1 · p. 16
  • The paper estimates that about 0.1% of tokens may be singular and leaves their rate of occurrence during ordinary use as an open question.The authors suspect users may encounter singular tokens regularly, but explicitly identify occurrence-rate estimation as unresolved.Source: §A.1 · p. 16

A.2 LLM Background

An LLM maps a finite token sequence into a stochastic response through token embeddings, transformer processing, probability distributions, and iterative window updates.

A.2 LLM Background

  • An LLM transforms a query in T^w into a response in T^m, and stochastic sampling means this transformation is generally not a function.The model operates through interacting stages rather than a single deterministic mapping.Source: §A.2 · p. 16
  • Each input token is mapped by e: T → X = R^ℓ, and a sequence of embeddings forms a latent window in X^w.The finite token set can be represented as an embedding matrix whose columns assign numerical coordinates to tokens.Source: §A.2 · pp. 16–17
  • A continuous transformer function f: X^w → Y converts each latent window into a probability distribution for the next token.The output space Y has one coordinate per token, so its elements represent unnormalized distributions over T.Source: §A.2 · p. 16
  • The next token is randomly drawn from f’s distribution, then appended to the window while the oldest token is removed.The update F(ψ1, ..., ψw) := (ψ2, ..., ψw, t) creates the subsequent prediction context.Source: §A.2 · p. 17
  • The paper distinguishes individual token input embedding e_w from sequence embeddings exposed by software interfaces, which occur later inside f.This distinction determines which embedding space the paper analyzes.Source: §A.2 · p. 17

A.3 Computational resource requirements

The full method is computationally feasible on CPU, with nearly all runtime concentrated in the dense token pairwise-distance calculation.

A.3 Computational resource requirements

  • Approximately 12 hours per model were required on an Intel Core i7-3820 with 32 GB RAM and no GPU.The subsequent three hypothesis tests ran in only a few minutes on the same machine.Source: §A.3 · p. 17

A.4 Interpretation of fiber bundles in terms of spatial variability

The paper interprets fiber bundles as spaces with distinct long- and short-scale structure, then develops mathematical tests and applies them to token embeddings. Examples and limitations show how slope changes identify irregularities, while context generally cannot persistently resolve token-space singularities.

A.4 Interpretation of fiber bundles in terms of spatial variability

  • For a fiber bundle, ball volumes change from small-radius scaling by the fiber dimension to large-radius scaling by the base dimension, producing a detectable corner.In the illustrated strip, areas scale quadratically at small radii and asymptotically linearly at large radii.Source: §A.4 · p. 18, Figure 5 · p. 18
  • Fiber bundles model measurements with long spatial scales in a base space and short spatial scales in fibers, allowing multiple scales without requiring noise.The base and fiber spaces can represent signal and noise, but the construction also applies to noiseless data when multiple spatial scales are present.Source: §A.4 · p. 18
  • A heteroscedastic fiber structure should reject the fiber-bundle model because its dimension varies across tokens; increasing slopes violate the theorem, while non-rejection remains inconclusive.The example contrasts ψ1, whose slope behavior violates the theorem, with ψ2, for which consistency does not establish that the space is a fiber bundle.Source: §A.4 · p. 18

A.5 Mathematical proof of Theorems 1 and 2

  • The proofs formalize the model using a fibered manifold with boundary, where local neighborhoods split into base and fiber dimensions and pushforward volume supports the radius-scaling analysis.The fiber dimension is dim T − dim S, and the proof uses embeddings, volume forms, compactness, and radius bounds.Source: §A.5 · p. 19
  • Context cannot persistently remove token-space singularities: if the token space is non-manifold, sufficiently longer token-sequence spaces also cannot all be manifolds.The argument uses the contrapositive of Lemma 1; lengthening the context window does not permanently resolve singularities.Source: §A.5 · p. 20

A.6 Case study of some tokens in GPT2

  • In GPT2, # shows no rejection, ¢ rejects the manifold but not fiber-bundle hypothesis near 3.9 units, and $ shows manifold and fiber-bundle violations near radii 3.8 and 4.0.The reported p-values for these illustrative tokens exceed the α = 10^-3 threshold used in the main tables, so they are not listed there.Source: §A.6 · pp. 20–21, Figure 6 · p. 21

A.7 Case study of a neighborhood of ember in Mistral7B

  • In Mistral7B, the neighborhood around ember is visualized after PCA projection, with darker colors indicating lower p-values and putative cusp or pinch structures identified among tokens.The tokens ember and P are described as possible cusp points, while : and ly are described as possible pinch points that manifolds cannot represent without high curvature.Source: §A.7 · p. 21, Figure 7 · p. 22

A.10 Limitations of theory

  • The approach is limited to models whose token embedding vectors are accessible, and its radius choices depend on reach estimates that are difficult to obtain accurately from sampled data.The study therefore focuses on four open-source models; proprietary-model extraction is described as computationally expensive, and underestimated reach can imply high fitted curvature.Source: §A.10 · p. 22, Table 4 · p. 24, Table 5 · p. 25

Related papers

Loading 2504.01002v3…