PaperlayerView exact revision on arXiv

Source-linked AI summary

A theory of platonic representations in language models

Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner, Daniel J. Korchinski, Santiago Acevedo, Matthieu Wyart

arXiv:2610.07168v1cs.LGcs.CLstat.ML

TL;DR

The paper addresses the unexplained similarity of translated sentences’ representations in multilingual models. It models languages with shared abstract hierarchy and language-specific surface structure, using belief propagation to derive layerwise predictions. The framework accounts for middle-layer similarity and distinguishes neighborhood similarity from alignment, while predicting that layerwise novelty can increase cross-lingual similarity.

  • Problem

    Why translated sentences have similar representations in multilingual models remains theoretically unexplained, despite the connection to the Platonic representation hypothesis.

    Source: §1 · p. 1

  • Method

    The paper models data with shared abstract and language-specific surface hierarchy, then analyzes synthetic grammars using belief propagation and layerwise representations.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    The framework explains middle-layer similarity alongside language-specific structure and predicts stronger cross-lingual similarity from layerwise novelty, which the authors confirm in controlled data and pretrained LLMs.

    Source: §1 · p. 3, §6 · p. 8, §8 · p. 9

  • Takeaways & Limitations

    Removing from each layer what is predictable from the preceding layer may provide a depth-resolved way to study abstraction in AI and neuroscience, including brain-activity prediction.

    Source: §8 · p. 10

    The framework’s extension to representation-learning algorithms such as joint embedding methods is left for future work.

    Source: §2 · p. 3

Abstract

from arXiv · show

Representations of translated sentences are similar in the inner layers of multilingual language models -- an observation connected to the platonic representation hypothesis, yet unexplained theoretically. We provide an explanation based on the assumption that data have a hidden hierarchical structure whose abstract levels are shared across languages while surface levels are modality- or language-specific. Concretely, we generate synthetic languages from probabilistic context-free grammars sharing upper-level but not lower-level production rules. In this setting the Bayes-optimal next-token predictor is belief propagation (BP); encoding its messages in successive layers yields analytical predictions that agree well with transformers trained on the same data. The framework explains why cross-lingual similarity peaks in middle layers, coexists with language-specific structure, and strengthens with language proximity, model quality and data exposure. It distinguishes similarity (shared neighborhood geometry) from alignment (shared coordinates), showing that the latter occurs when code-switched data, i.e. mixed-language sentences, are abundant enough. It further predicts that subtracting from each layer the component linearly predictable from the preceding one increases cross-lingual similarity, which we confirm in pretrained LLMs.

1 INTRODUCTION

The paper addresses the missing theory for why translated sentences have similar representations in multilingual models, while retaining language-specific structure. It proposes a shared-hidden-hierarchy account and predicts that removing information carried forward from the preceding layer can reveal stronger cross-lingual similarity.

1 INTRODUCTION

  • Translated sentences are most similar in middle layers, while language-specific processing remains near the input and output (Wendler et al., 2024; Liu & Niehues, 2025; Dumas et al., 2025; Inaba et al., 2025).Similarity here refers to shared local geometry or neighborhood structure, not necessarily shared coordinates.Source: §1 · p. 1
  • Cross-lingual similarity strengthens as models acquire better representations through training and increased capacity, and is weaker for low-resource languages, which exhibit poorer cross-lingual transfer; multilingual models retain language-specific information alongside shared conceptual structure (Peng & Søgaard, 2024; Inaba et al., 2025; Miao et al., 2024; Chang et al., 2022; Dumas et al., 2025).Models can achieve alignment when shared information is encoded along the same directions, allowing a readout learned in one language to operate in another without refitting. Mixed-language contexts, including synthetic code-switching during pretraining, reinforce this alignment (Wang et al., 2025).Source: §1 · pp. 1–2
  • The paper assumes data have a hidden hierarchy with shared abstract levels and language- or modality-specific surface levels, and models this using probabilistic context-free grammars.Synthetic languages share upper production rules and differ in lower rules; BP predictions and trained-transformer results approximately agree.Source: Figure 1 · p. 2, §1 · p. 3
  • Subtracting the component of each layer linearly predictable from the preceding layer yields a residual more suitable for predicting other languages’ neighborhood structure, confirmed in controlled data and pretrained LLMs.The paper favors local similarity measures such as Information Imbalance because global measures provide limited support for the Platonic hypothesis (Gr¨oger et al., 2026; Kornblith et al., 2019; Huh et al., 2024; Koepke et al., 2026; Glielmo et al., 2022).Source: §1 · p. 3

2 RELATED WORK

Prior work characterizes hierarchical algorithms learned by transformers and the tractability of the Random Hierarchy Model. The multilingual setting—and the listed cross-lingual phenomena—remained unaddressed.

2 RELATED WORK

  • Studies of transformers on CFG-generated data characterize parsing and hierarchical inference, but do not address the multilingual setting (Garnier-Brun et al., 2025).The work includes inside-outside-style upward and downward computation, sequential learning of correlations at larger scales, and an encoder-only shortcut for frozen tree topologies.Source: §2 · p. 3
  • The Random Hierarchy Model makes learning tractable and supports predictions about sample complexity, bottom-up rule learning, scaling laws, and belief-propagation denoising (De Giuli, 2019; Mossel, 2016; Malach & Shalev-Shwartz, 2018; Cagnetta et al., 2024; Cagnetta & Wyart, 2024; Cagnetta et al., 2026; Parley et al., 2026; Sclocchi et al., 2025b;a).The multilingual setting had not been considered, leaving the paper’s listed phenomena unexplained.Source: §2 · p. 3
  • Extending the framework to representation-learning algorithms such as joint embedding methods is left for future work.This marks a scope boundary beyond the generative-model setting considered here.Source: §2 · p. 3

3 MULTILINGUAL/MULTIMODAL RANDOM HIERARCHY MODEL (MRHM)

The Multilingual Random Hierarchy Model represents languages as grammars that share upper-level structure but differ in their surface-level rules. Translation pairs share latent variables above the split, and code-switching is modeled by replacing language-specific subtrees.

3 MULTILINGUAL/MULTIMODAL RANDOM HIERARCHY MODEL (MRHM)

  • A probabilistic CFG generates token sequences by recursively expanding latent symbols, while the finite-depth RHM generates an s-ary tree whose leaves are observed tokens.The RHM is specified by depth L, branching factor s, number of rules m, and alphabet size v.Source: §3 · p. 4
  • The two-language model shares production rules and symbols across the upper Lshared levels, then uses distinct rules and symbols across the lower Lsurface levels.At the split, each symbol has m rules for each language.Source: §3 · p. 4
  • Translation pairs share latent variables at levels ℓ≥Lsurface and differ below the split; increasing Lsurface reduces their shared generative structure and serves as a controlled proxy for language distance.The shared variables are interpreted as common semantic content and the lower variables as language-specific realization.Source: §3 · p. 4
  • Code-switching is simulated by replacing each language-specific subtree with its counterpart from the other language with probability ϵ.Replacements are independent across subtrees and are applied analogously in both language directions.Source: §3 · p. 4

4 THEORY: NEXT-TOKEN PREDICTION VIA BELIEF PROPAGATION

Because the model’s dependency graph is tree-structured, belief propagation computes the Bayes-optimal next-token distribution through upward encoding and downward decoding. The paper constructs layerwise networks for these computations and considers how finite data and message retention affect their representations.

4 THEORY: NEXT-TOKEN PREDICTION VIA BELIEF PROPAGATION

  • Belief propagation computes the Bayes-optimal next-token distribution by propagating evidence upward into latent-variable beliefs, then downward to the predicted token’s posterior.The same two-stage computation applies in the multilingual setting.Source: §4 · p. 4
  • The theoretical network represents each latent-symbol belief as a vector and assigns successive layers to sequential BP steps, accumulating messages for later decoding.The complete network has 2L −1 layers, one for each encoder–decoder computation step.Source: §4 · pp. 4–5
  • Below Lsurface, the two languages’ beliefs occupy separate orthogonal blocks, while above it they are shared; coordinate alignment can depend on code-switching rate ϵ and training-set size P.The theoretical networks assume orthogonal representations for simplicity because alignment has little effect on geometric neighborhood similarity.Source: §4 · p. 5
  • For used messages, the network allows gradual decay by γ; γ = 1/2 better matches trained RHM transformers, though γ does not change qualitative findings.Instantaneous removal corresponds to γ = 0.Source: §4 · p. 5
  • With finite data, transformers may recover production rules only to an effective depth ℓ∗, leaving higher-level latents unreconstructed and producing suboptimal predictions.The incomplete-BP model assumes passive middle layers; this affects quantitative but not qualitative theoretical results.Source: §4 · p. 5
  • Depending on training-set size and code-switching rate, transformers may follow either encoder–decoder BP or an encoder-only shortcut; the paper focuses on ϵ = 0.1, where trained transformers consistently exhibit encoder–decoder structure.The encoder-only algorithm collapses downward computation into one final update, while leaving the encoder unchanged.Source: §4 · pp. 5–6

5 GEOMETRIC SIMILARITY

The section compares multilingual transformers trained on synthetic hierarchical grammars with belief-propagation networks. Translation representations become most similar in middle layers, with similarity shaped by shared grammar, language exposure, and training data.

5.1 TRANSFORMER TRAINING SETUP AND SIMILARITY MEASURE

  • The experiments train GPT-style transformers on MRHM data and compare their translation-pair representations with theoretical BP networks.The transformers use eight layers, 512-dimensional embeddings, and four attention heads; conclusions are unchanged for at least 2L−1 layers.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 GEOMETRIC SIMILARITY IN MULTILINGUAL MODELS

  • Translation representations show a non-monotonic layerwise similarity profile, peaking in the middle without becoming perfect.Belief-propagation networks qualitatively reproduce the profile: shared high-level latents support middle-layer similarity, while language-specific information remains relevant during encoding and decoding.Source: §5.2 · pp. 6–7
  • Greater language-specific depth weakens similarity: minimum II rises and maximum mKNN falls as shared depth decreases.The theory attributes this pattern to a smaller shared portion of middle-layer representations.Source: Figure 2 · p. 6, §5.2 · p. 7
  • Similarity also appears between independently trained models for languages A and B, with layerwise trends resembling those of a single multilingual model.The comparison evaluates the models on the same translation pairs.Source: §5.2 · pp. 6–7, Figure 2 · p. 6
  • As training data increase, middle-layer similarity emerges as learned grammar reaches shared hierarchy levels.Imperfect BP reproduces this progression: similarity begins when learned rules reach the first shared level and strengthens as more shared levels are recovered.Source: §5.2 · p. 7, Figure 3 · p. 7
  • With fixed total data, increasing the low-resource language’s share strengthens similarity; minimum II decreases monotonically with that share.The BP account models language A as fully learned and language B as partially learned.Source: §5.2 · p. 7, Figure 4 · p. 8

6 LAYERWISE NOVELTY

The section defines layerwise novelty by removing from each layer the component predictable from the preceding layer. In the tested grammars and pretrained language models, this reveals stronger cross-lingual similarity in intermediate computations than raw hidden states do.

6 LAYERWISE NOVELTY

  • Layerwise novelty subtracts the component linearly predictable from the preceding layer, isolating information newly computed at the current layer.The paper predicts that this residual exposes shared computation more clearly than raw hidden states.Source: §6 · p. 8
  • In MRHM, novelty lowers II in early and middle layers, but raises it later as it removes high-level information carried by skip connections.mKNN shows comparable results, and the findings indicate that language-specific information retained in raw states partly obscures shared intermediate structure.Source: §6 · p. 8
  • In Mistral-7B-v0.1 and OLMo-3-7B, novelty achieves lower II than raw representations across most layers on parallel English–Romanian and English–Polish pairs.The reported tests use Europarl sentence pairs.Source: §6 · p. 8, Figure 6 · p. 9

7 COORDINATE ALIGNMENT

The section distinguishes geometric similarity from coordinate alignment: shared hierarchy can yield similar neighborhood structure without shared coordinates. Cross-probing shows alignment depends jointly on code-switching and training-set size.

7 COORDINATE ALIGNMENT

  • Geometric similarity concerns shared distances and neighborhoods, whereas coordinate alignment lets a readout trained in one language operate in another without refitting.The paper treats alignment, rather than geometric similarity, as the property underlying cross-lingual transfer.Source: §7 · p. 9
  • Cross-probing alignment depends jointly on code-switching rate ϵ and dataset size P; more data alone at ϵ = 0 does not produce substantial accuracy.With ϵ > 0, increasing either the code-switching rate or training-set size strengthens alignment.Source: §7 · p. 9, Figure 7 · p. 9
  • The aligned-region boundary follows Pthreshold ∝ ϵ^-1 because the effective cross-lingual signal scales with ϵP.The paper explains that cross-language features are observed through code-switched subtrees, so a fixed alignment level requires ϵP to remain constant.Source: §7 · p. 9, Figure 7 · p. 9

8 CONCLUSION

A hidden hierarchy with shared abstract levels and language-specific surface levels accounts for multilingual representation patterns, including middle-layer similarity alongside language-specific structure. Alignment requires cross-lingual training signal, while removing layerwise predictable components offers a way to isolate more abstract representations.

8 CONCLUSION

  • The shared-upper, language-specific-lower hierarchy accounts for middle-layer similarity coexisting with language-specific structure and for stronger similarity with language proximity, model quality, and data exposure.Alignment is distinct from similarity and requires cross-lingual signal in training data.Source: §8 · pp. 9–10
  • Mixed-language signal is the proposed efficient lever for purchasing alignment in low-resource languages.The account treats alignment as stronger than shared neighborhood geometry.Source: §8 · p. 10
  • Subtracting from each layer what is predictable from the preceding layer isolates newly computed, higher-level information by eliminating surface details.The authors suggest this layerwise novelty may help study abstraction in AI and neuroscience.Source: §8 · p. 10

A RHM SCHEMATIC

  • The RHM schematic has four levels, with visible tokens at the leaves denoted x_i; a leaf latent can use any of the 2^ℓ subscripts for tokens in its subtree.The caption notes this notation is redundant.Source: Figure 8 · p. 14

B BELIEF PROPAGATION AND THEORETICAL NETWORKS

The theoretical network encodes belief-propagation messages in successive layers to model optimal causal next-token prediction under the hierarchical grammar. Transformers trained on the same data show layerwise representations corresponding to BP messages, with the clearest match at γ = 1/2.

B BELIEF PROPAGATION AND THEORETICAL NETWORKS

  • For causal next-token prediction, BP sends evidence upward from observed subtrees, then propagates messages downward along the path to the next token.Observed leaves provide point-mass evidence, while unobserved leaves are marginalized.Source: B BELIEF PROPAGATION AND THEORETICAL… · pp. 14–15, Figure 9 · p. 14
  • The theoretical network stores BP messages as level-specific blocks, decaying used upward messages by γ; intermediate γ best matches transformers, while γ = 0 deletes them immediately and γ = 1 retains them.The authors attribute the intermediate choice to transformer residual connections.Source: B BELIEF PROPAGATION AND THEORETICAL… · p. 16
  • With limited data, depth-limited BP models rule recovery only through level ℓ*, using uniform unresolved factors above it; cross-lingual similarity emerges at L_surface and grows as shared levels are recovered.The complete BP step sequence remains, but unresolved levels carry uniform, sample-independent messages.Source: B BELIEF PROPAGATION AND THEORETICAL… · p. 16
  • Fixed-topology grammars permit an encoder-only shortcut that combines upward messages into one-step decoding, but code-switching and training-set size influence whether transformers approach it or full encoder–decoder computation.Varying-topology grammars require the full computation; the study focuses on ε = 0.1, where transformers consistently show encoder–decoder structure.Source: §B.1 · pp. 16–17

B.2 TRANSFORMERS APPROXIMATE BP COMPUTATION

  • Transformer layers predict theoretical-network representations of expected BP messages, with clearer correspondence at γ = 1/2 than γ = 0.The transformer-to-theory and theory-to-theory prediction heatmaps show similar layerwise behavior.Source: §B.2 · pp. 17–18, Figure 10 · p. 17

C TRAINING DETAILS

The appendix describes transformer training, representation-similarity measures, and an observable for tracking algorithmic transitions. The layer of highest cross-lingual similarity identifies changes between encoder–decoder and encoder-only behavior as code-switching rate and training-set size vary.

C TRAINING DETAILS

  • Transformer test loss decreases stepwise with training-set size P, following theoretical entropies calculated with depth-limited BP.Figure 13 shows the loss curves for different P.Source: C TRAINING DETAILS · p. 18, Figure 13 · p. 19

D SIMILARITY MEASURES

  • Information imbalance tracks preservation of a nearest-neighbor relationship, while mKNN compares the sets of k nearest neighbors and scores exact neighborhood agreement from 0 to 1.The experiments use cosine similarity after ℓ2-normalization and expected overlap under independent uniform tie breaking.Source: D SIMILARITY MEASURES · pp. 19–20
  • Raw mKNN depends on k and evaluation-set size N, so comparisons across training-set sizes are valid here because both are held fixed.Unlike normalized information imbalance, mKNN is not an intrinsic, parameter-free measure of neighborhood agreement.Source: D SIMILARITY MEASURES · p. 20

E ALGORITHMIC TRANSITION WITH ϵ AND P

  • The layer of lowest information imbalance identifies the algorithmic transition as code-switching rate ε and training-set size P vary: encoder–decoder predicts middle-layer similarity, while encoder-only predicts penultimate-layer similarity.Figure 14 presents this observable across ε and P.Source: E ALGORITHMIC TRANSITION WITH ϵ AND P · p. 20, Figure 14 · p. 20

F VARYING THE NUMBER OF TRANSFORMER LAYERS

Transformers distribute the same belief-propagation computation across available layers, preserving qualitative similarity trends as depth changes. With varying tree topology, similarity still reaches a middle-layer minimum; cross-probing finds coordinate alignment at the first shared latent level and above as code-switching and training data increase.

F VARYING THE NUMBER OF TRANSFORMER LAYERS

G VARYING TOPOLOGY

  • With binary and ternary branching, cross-lingual II reaches an intermediate-layer minimum and rises toward the output, consistent with encoder–decoder processing.The varying-tree multilingual model shares upper rules and tree structure across translations but has distinct lower subtrees.Source: G VARYING TOPOLOGY · pp. 21–22, Figure 17 · p. 22

H COORDINATE ALIGNMENT

  • Cross-probing shows coordinate alignment at the first shared latent level and above, while alignment below the split is impossible because those latent variables are language-specific.Independent probes are trained on each language and tested on the other; alignment increases with code-switching rate and training-set size.Source: H COORDINATE ALIGNMENT · p. 22

I LAYERWISE NOVELTY

Layerwise novelty is the residual after removing the component of a layer affine-predictable from the preceding layer. It generally improves cross-lingual similarity during encoding, but its benefit is less consistent for residual variants and reverses during decoding in the theoretical analysis.

Novelty measure

  • Layerwise novelty is the residual after subtracting an affine prediction based on the preceding layer, with residuals averaged over valid tokens for sentence-level comparison.A single affine map is fit for each layer transition across token positions and fitting sentences using ridge regression.Source: I LAYERWISE NOVELTY · pp. 22–23

Theoretical and trained-network comparison

  • In Figure 18, novelty enhances cross-lingual similarity during encoding but reduces it during decoding relative to raw representations.The result is supported by both II and mKNN for trained transformers and theoretical BP networks; removing retained shared information is consistent with the decoding reduction.Source: Figure 18 · p. 23, I LAYERWISE NOVELTY · p. 23

Natural-language models

  • In Figure 19, layerwise novelty generally gives lower II and higher mKNN than raw representations at intermediate layers across three models and four language pairs.Improvements from the residual contribution and supra-word residual are less consistent across models and language pairs.Source: I LAYERWISE NOVELTY · p. 23, Figure 19 · p. 24
Loading 2610.07168v1…