Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada; Daniel J. Korchinski; Andres Nava; Matthieu Wyart; Yasaman Bahri

Source-linked AI summary

Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri

arXiv:2602.15029v3Feb 16, 2026cs.LGcond-mat.dis-nncs.CL

TL;DR

Language models exhibit structured representations such as circular time, one-dimensional historical manifolds, and linearly decoded coordinates, but the statistical principle behind these patterns is unclear. The paper derives these geometries from translation symmetry in pairwise co-occurrence statistics and shows that the predictions match embedding models and LLMs, while remaining robust to perturbations through shared latent variables.

Problem
The paper addresses the lack of an organizing principle explaining why language-model representations consistently form circles, one-dimensional manifolds, and linearly decodable spatial or temporal coordinates.
Method
The authors analyze word embeddings spectrally, derive representation geometry from translation-symmetric co-occurrence statistics, and model robustness using a shared continuous latent variable.
Results
The predicted manifolds and decoding properties appear in word embeddings and larger sequence models, while geometry persists after substantial co-occurrence perturbations at intermediate dimension.
Takeaways & Limitations
Representational manifolds can originate from statistical symmetries in natural language, with their geometry reflecting collective low-order correlations among many words.
Takeaways & Limitations
The theory is derived for word-embedding algorithms and captures many, but not all, aspects of LLM representations; hierarchical attributes remain unexplained.

Abstract

from arXiv · show

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

1. Introduction

The paper proposes that translation symmetry in pairwise co-occurrence statistics organizes neural representations into predictable geometric manifolds. A unified theory explains circular, curved one-dimensional, and spatial structures, their linear decodability, and their robustness to perturbations.

Motivation: Existing studies find circles, rippled one-dimensional manifolds, and linearly decoded spatial or temporal coordinates in language-model representations, but lack an organizing principle.These geometries appear across architectures and tasks.
Core hypothesis: Translation symmetry in pairwise co-occurrence statistics provides a unified explanation for these representation geometries.The paper argues that co-occurrence probability depending only on temporal or spatial separation produces this symmetry.
Theory: Fourier representations arise when co-occurrence depends on temporal or spatial distance, with long-wavelength modes producing circles and loops and higher harmonics producing ripples.The theory analytically predicts each component’s amplitude and frequency.
Empirical validation: The predicted structures are validated in word embeddings and deep transformer models, including representation manifolds and linear coordinate decoding.The contribution list reports empirical validation across shallow and deep models.
Robustness: Representation manifolds remain robust to substantial co-occurrence perturbations at intermediate embedding dimension because many words share the underlying continuous latent variable.The latent-variable account attributes robustness to large eigenvalues whose eigenvectors are insensitive to noise or perturbation.
Conclusion: The broader conclusion is that symmetry in low-order token correlations shapes learned representations across architectures and learning tasks.The paper characterizes this as a collective phenomenon involving many semantically related words.

2. Preliminaries and Related Work

The paper connects word-embedding geometry to the spectral structure of pairwise co-occurrence statistics and situates its contribution among prior geometric analyses. Its central advance is deriving curved feature manifolds from translation symmetry, including linear decoding and robustness effects.

Preliminaries: Word embedding models learn from pairwise co-occurrence statistics, producing an embedding matrix whose rows represent vocabulary items.The co-occurrence probability Pij measures how often two tokens appear within a shared context window.
Spectral formulation: The learned embeddings’ principal components correspond to the largest eigenmodes of the co-occurrence-derived matrix M⋆.This provides the analytical bridge from data statistics to embedding geometry.
Core approach: The paper extends prior latent-variable analyses by showing that continuous latent variables with translation symmetry generate curved feature manifolds.The same framework predicts linear decoding and robustness to noise.
Geometric representation: The geometry of a vocabulary subset can be recovered from centered embeddings’ PCA coordinates and their associated Gram matrix.Projected embeddings retain the information in the subspace spanned by the centered embeddings.
Novelty: The contribution unifies structured geometries in word embeddings and larger language models by explicitly connecting data symmetry to representation geometry.It also treats open boundaries, two-dimensional lattices, and the collective nature of the phenomenon.

3. Co-occurrence model with symmetry

The paper models semantic continua through translation-symmetric co-occurrence statistics, where pairwise co-occurrence depends only on distance along the continuum. This symmetry determines Fourier-based embedding geometry, including closed loops for periodic concepts, open manifolds for historical time, and analytically predictable coordinate decoding.

Symmetry assumption: Translation symmetry assumes that co-occurrence probability depends only on distance between words on a semantic continuum.The continuum may be periodic, such as calendar time, or open, such as historical time and geography.
Fourier geometry: For uniformly spaced periodic continua, PCA components encode sinusoidal functions whose amplitudes are Fourier coefficients of the co-occurrence kernel.Even kernels produce degenerate sine and cosine pairs.
Predicted manifolds: Calendar-word embeddings form closed loops with integer-frequency oscillations, whereas historical-year embeddings form open loops with non-integer frequencies.These predictions are validated empirically for calendar words and historical years.
Predicted manifolds: Propositions 1 and 3 predict Lissajous curves when representations are projected onto any two principal components.The curve amplitudes, phases, and frequencies are determined analytically by the embedding theory.
Coordinate decoding: Linear coordinate decoding becomes easier as higher-frequency Fourier modes are included, with error scaling asymptotically as ε2 ∼ r−1/D.The predicted scaling is empirically validated on Wikipedia-trained embeddings.

4. Collective effects control the embedding of space and time

The paper explains robust spatial and temporal geometry as a collective effect of shared latent variables influencing many words. Consequently, even removing direct month co-occurrences can leave circular structure recoverable from other seasonal words and dominant low-frequency modes.

Robust reconstruction: After month-month co-occurrences are removed, moderate-dimensional embeddings can still recover the months’ circular ordering and closely approximate their original Gram matrix.The reconstruction uses only a small fraction of the eigenspectrum.
Latent-variable mechanism: A shared seasonal latent variable causes many words to have season-dependent co-occurrence patterns, producing large, perturbation-resistant eigenvalues.Examples include winter-associated “ski” and summer-associated “beach.”
Robust reconstruction: A handful of highly seasonal words suffices for reconstructing month order, whereas highly non-seasonal number words do not enable seasonal reconstruction.The comparison isolates shared seasonal structure as the relevant signal.
Spectral structure: For a smooth even seasonal kernel, Fourier coefficients decay rapidly, so the leading modes represent the global mean and the lowest-frequency seasonal oscillation.Subsequent eigenvectors encode higher-frequency seasonal variations.
Spectral robustness: In the large-vocabulary limit, perturbing a fixed number of PMI entries leaves the top embedding eigenvectors unchanged because eigenvalue gaps grow proportionally with vocabulary size.The argument uses Weyl or Davis–Kahan perturbation bounds.

5. Discussion

The paper argues that geometric structures in representations of time and space arise from a unifying principle based on pairwise co-occurrence statistics. These geometries appear across embedding models and larger sequence models and can support computation and coordinate decoding.

5. Discussion: A single principle links circular, one-dimensional, and geographic representation geometries to pairwise co-occurrence statistics.The paper emphasizes low-order correlational statistics as the foundation of its theory.
5. Discussion: Time and space are represented as circles, curved one-dimensional manifolds, and curved sheets, with high-dimensional overtones enabling linear coordinate decoding.This pattern is reported across word embeddings, text embedding models, and LLMs.
5. Discussion: The theory is proved in word embedding models but its predictions carry over to a diverse set of larger sequence models.Closed-form training dynamics provide the explicit bridge between data statistics and learned representations in the word-embedding setting.
5. Discussion: The same kind of manifold structure may extend beyond time and space to concepts such as color, temperature, and age.The paper relates these structures to perceptual grounding despite text-only training.
5. Discussion: Representational geometry supports forward-pass computations including relational completion, modular addition, and reasoning about line breaks.The paper connects parallelogram, Fourier, and manifold structure to these computational behaviors.

Limitations

The theory is derived for word-embedding algorithms, while LLMs additionally adapt representations to context. The paper also leaves a broader framework for other attribute structures as an open scope boundary.

Limitations: The theory is derived in the setting of word embedding algorithms, although it captures many aspects of LLM representation geometry.LLMs introduce contextual adaptation that is not part of the theory's derivation.
Limitations: Context can disambiguate words such as “May,” causing their representations to become increasingly circular through the forward pass.Explaining this contextual evolution is identified as an interesting direction.
Limitations: A global framework covering continuous, binary, and hierarchical attribute geometries remains desirable.The paper extends prior work on binary attributes but leaves hierarchical structures unexplained.

A. Experimental details

The appendix documents the code and experimental details needed to reproduce and interpret the main-text figures, and provides a public code repository.

A. Experimental details: The appendix summarizes figure-generating code, experimental setup, and evaluation details for reproducing and interpreting the results.The code is publicly available at https://github.com/dkarkada/symmetry-stats-repgeom.

A.1. Main text figures

The main-text figures combine theoretical co-occurrence models, empirical embeddings, eigenmode analyses, and probing experiments across temporal and spatial concepts. Their implementation includes explicit preprocessing, model, prompt, and visualization choices.

A.1. Main text figures: The theoretical model constructs representation vectors from corpus statistics by forming M⋆ and obtaining embeddings through its spectral decomposition.The training procedure then diagonalizes M⋆ and evaluates the resulting embedding expression.
A.1. Main text figures: Calendar-month and historical-year plots use different boundary conditions and centering conventions to visualize their Gram matrices.Months have periodic boundary conditions, whereas years have open boundary conditions; the year color map runs from 1700 to 2020.
A.1. Main text figures: Historical-year embeddings show local Lissajous-curve kinks near the World Wars because war-related articles weakly break time-translation symmetry.The figure compares theoretical and empirical constructions using an exponential kernel.
A.1. Main text figures: A linear probe is evaluated on 1900–2020 year embeddings using 100 random train-test trials with 60 training and 60 test embeddings per trial.The main figure uses ridgeless regression to decode numerical year; regularized performance is reported separately.
A.1. Main text figures: Theoretical state geometry uses latitude-longitude distances with a 0.78 aspect-ratio correction and an exponential co-occurrence kernel.The kernel is S(ij) = 10 exp(−dij/20), with parameters chosen by visual inspection and qualitative eigenmodes insensitive to those choices.
A.1. Main text figures: The experiments use Wikipedia preprocessing, a 25,000-word vocabulary, 2.72 billion tokens, and co-occurrence modeling with L = 16 and f(d) = L + 1 − d.The LLM analysis uses Gemma 2 2B residual-stream activations from transformer blocks with specified prompts and final-token extraction.

B.3. Weaker version of Assumption 3.1

The paper weakens the PSD assumption by modeling translation symmetry in the absolute co-occurrence matrix, while explaining why the simpler assumption remains useful for theoretical predictions. It also documents empirical structure in co-occurrence spectra and representation geometry.

Weaker version of Assumption 3.1: Real data violate the PSD assumption: M⋆ has a spectrum peaked near zero, with rank M+ ≈ rank M−.The fitted exponential kernel therefore does not exactly describe the factored kernel.
Weaker version of Assumption 3.1: The weaker assumption imposes translation symmetry on |M⋆|, allowing separate PSD and NSD convolution kernels.It strictly generalizes the PSD-only case and is supported by empirical data and the collective model.
Weaker version of Assumption 3.1: The weaker formulation is harder to use because estimating |M⋆| requires a global spectral transformation over all vocabulary words.The approximate PSD-based assumption needs only co-occurrence and unigram probabilities for the selected words.
Relation to PMI matrix: M⋆ closely approximates PMI, with the leading correction appearing at third order, while remaining more suitable for bounded least-squares approximation.This motivates using PMI for the theoretical analysis and M⋆ for empirical factorization.
Empirical translation symmetry: Month co-occurrence matrices exhibit circulant structure and agree closely with a periodized exponential kernel based on time interval.The periodized kernel accounts for repeated yearly interpretations of month pairs.
Empirical translation symmetry: Year statistics are fit with a shifted exponential kernel because M⋆ can take negative values; PCA is expected to remove the added constant mode.The spectrum is roughly symmetric, with comparable PSD and NSD ranks and semantic signals beyond the bulk spectral edge.

C.1. Helper-based reconstruction

Helper-word reconstruction tests whether auxiliary vocabulary items can recover seasonal structure in month embeddings. Seasonal helpers succeed, whereas non-seasonal number words do not, and reconstruction improves with more helpers.

Helper-based reconstruction: Number words lack seasonal modulation and fail to reconstruct the correct ordering of calendar months.Their reconstructed PMI and embeddings show no clear month ordering.
Helper-based reconstruction: Seasonal helpers are selected by month-word affinity and seasonality scores derived from complex Fourier-weighted affinities.The top-100 words with strong seasonal signals produce high-quality reconstruction.
Helper-based reconstruction: Reconstruction error decreases as the number of helper words H increases, with seasonal helpers scaling faster and reaching lower error than random helpers.The comparison directly tests whether helpers carry the relevant latent seasonal information.

D. Combined Model of Seasonality and Binary Semantic Attributes

The combined model assigns each word a seasonal phase and binary semantic attributes, yielding a product-space PMI whose eigenstructure separates seasonal and attribute information. This preserves the paper’s geometric constructions when projecting onto the seasonal subspace.

Model construction: The model enriches each word with a seasonal phase and a binary attribute vector, producing a vocabulary of size N2^d.The binary attributes represent additional semantic information such as gender or number.
Model construction: Seasonality and semantic attributes influence co-occurrence independently through multiplicative factors.The resulting joint probability factorizes into time and attribute contributions.
PMI decomposition: The PMI decomposes additively into a temporal term and an attribute term: PMI((x,a),(y,b)) = log Ftime(x,y) + log Fattr(a,b).The constant, semantic, and seasonal components are then separated in the matrix construction.
Spectral decomposition: The model diagonalizes in a product basis combining Fourier modes for seasonality with Walsh characters for binary attributes.The seasonal and attribute operators act on distinct tensor components.
Spectral decomposition: Theorem 5 gives eigenvectors Φk,S(x,a) = ϕk(x)ψS(a) and corresponding eigenvalues for the combined seasonal–attribute PMI.The decomposition identifies how temporal and attribute modes contribute to the spectrum.
Geometric consequence: The PMI splits into orthogonal seasonal and binary-attribute subspaces, so the paper’s geometric structures reappear after projection onto the seasonal subspace.This isolates the seasonal modes from additional semantic structure.

E. Proofs

The proof setup constructs a finite latent semantic lattice and its reciprocal wavevector lattice under periodic boundary conditions. Wavevectors are organized into conjugate pairs, self-conjugate modes, and the constant mode.

Latent semantic lattice: The latent semantic space contains LD lattice sites arranged in D dimensions with L sites along each axis.The index set has cardinality |N| = LD.
Latent semantic lattice: Each word is bijectively assigned a lattice index and coordinate xi = 2ni/L on the interval [−1,1]^D.These coordinates define the latent semantic continuum used in the proof construction.
Wavevectors: Periodic boundary conditions define a reciprocal lattice of wavevectors through positive-half-space ordering and periodized negation.The construction separates wavevectors into disjoint classes before ordering them.
Wavevectors: Lattice coordinates occupy real space, whereas wavevectors occupy reciprocal space and are not commensurate with the coordinate lattice.This distinction underlies the separate indexing of positions and Fourier modes.
Wavevectors: The wavevector set contains conjugate ± pairs, nonzero self-conjugate modes, and the zero mode.For odd L only the zero mode is self-conjugate; even L adds Nyquist-frequency modes on Brillouin-zone faces.

E.2. Proof of Fourier representation geometry (periodic lattice)

For a periodic latent semantic lattice, translation-symmetric co-occurrence statistics make the co-occurrence matrix Fourier-diagonalizable. Consequently, PCA coordinates encode sinusoidal Fourier modes, with amplitudes determined by the kernel’s Fourier coefficients.

The normalized co-occurrence matrix is defined on a latent semantic lattice with allowed wavevectors and associated partition indices.The proposition applies to any vocabulary subset S under the stated lattice notation.
Mean-centering removes the constant mode without mixing the remaining eigenvectors, so the centered Gram matrix retains the Fourier diagonalization.The constant mode is an eigenvector, and centering annihilates it; the remaining sinusoidal modes remain orthogonal.
Periodic lattice symmetry makes plane waves eigenvectors of the co-occurrence matrix, with eigenvalues given by the Fourier transform of the co-occurrence kernel.The change of variables using periodicity yields the discrete Fourier transform and the transfer function ˜m(k).
The Fourier eigenvectors form an orthonormal basis, while evenness of the kernel produces degenerate sine and cosine mode pairs.Conjugate symmetry in Fourier space allows complex eigenvectors to be combined into real-valued Fourier modes.
PCA-projected embeddings are determined by normalized sinusoidal eigenmodes and their transfer-function amplitudes, up to degenerate-subspace rotations and principal-direction permutations.The diagonal form identifies the PCA coordinates through Φ and the eigenvalues ˜m(kµ).

E.4. Proof of Fourier representation geometry under exponential kernel (open BC)

With open boundary conditions and an exponential co-occurrence kernel, the paper derives continuous sinusoidal eigenfunctions and their PCA-aligned embedding geometry. Truncating to finite rank retains the lowest-frequency modes and yields a dimension-dependent coordinate-decoding error.

The open-boundary result assumes a one-dimensional latent semantic lattice, exponential co-occurrence kernel, and embedding dimension at least rank M ⋆.The proposition specifies C(∆x) = exp(−|∆x|/σ), with σ free and d ≥ rank M ⋆.
The PCA-aligned embeddings use these odd and centered-even eigenfunctions with coefficients determined by the corresponding singular values.The stated eigenvalues are λµ = 2σ/(1 + σ^2kµ^2), and the embedding coefficients are their square roots.
In the continuum limit, the exponential kernel becomes an integral operator whose eigenfunctions satisfy a Helmholtz equation and are sinusoidal.The kernel is represented by a Green’s function, so diagonalizing it reduces to solving the associated differential operator.
Mean-centering removes the constant component and modifies the even-sector boundary conditions, while leaving the odd-sector solutions unchanged.The centered operator projects out the DC mode; its eigenfunctions are sinusoidal with the required constant offset in the even sector.
The allowed wavenumbers are strictly ordered, kµ < kµ+1, after combining the odd- and centered-even-sector quantization conditions.The ordering follows from the positive arguments in the quantization formulas.