Source-linked AI summary
A Hyperbolicity Atlas of Large Language Model Hidden States
Zhichao Yang, Yuanze Hu, Gen Li, Qingchen Yu, Shiying Duan, Xinyu Wang, Ye Qiu, Zeming Liu, Guangxu Chen, Zhaoxin Fan
TL;DR
It remains open whether contemporary LLM hidden states exhibit tree-like metric organization, a property existing representation analyses do not directly test. This paper maps Gromov hyperbolicity across models, layers, and domains, finding that depth dominates while scale effects are weak and non-monotonic.
Problem
It remains open whether hidden states in contemporary LLMs exhibit tree-like metric organization; existing representation analyses do not directly test whether their distances behave hierarchically.
Source: §1 · p. 1Method
The paper systematically maps Gromov hyperbolicity of prompt-token hidden states across model scale, layers, families, and input domains.
Source: §1 · p. 1Results
Depth dominates: middle layers usually form a high-relative-hyperbolicity plateau, final layers become more tree-like, scale effects are weak and non-monotonic, and families and domains differ.
Source: §1 · p. 2Takeaways & Limitations
Gromov hyperbolicity is a useful diagnostic when interpreted as a model–layer–domain property, not as a direct claim about symbolic hierarchy.
Source: §7 · p. 9The atlas measures prompt-token states using Euclidean distances and diameter normalization; broader architecture and alternative distance or normalization tests remain needed.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.
1 Introduction
The paper asks whether contemporary LLM hidden states exhibit tree-like metric organization, which existing representation analyses do not directly test. It maps normalized hyperbolicity across models, layers, and domains, finding depth more influential than scale and specialization-dependent domain patterns.
1 Introduction
- The study asks whether hidden states in contemporary LLMs show tree-like metric organization even when models were not trained in hyperbolic manifolds.Prior representation analyses address feature recoverability, spread, local complexity, or recoverable linguistic relations, rather than whether hidden-state distances behave like a hierarchy.Source: §1 · p. 1
- The atlas compares normalized hyperbolicity across parameter scale, layer depth, model family, and input domain because these axes need not move together.The study asks about within-family scale changes, layer trajectories, matched-scale family differences, and domain variation within a model.Source: §1 · pp. 1–2
- The protocol measures prompt-token hidden states for ten open-weight checkpoints across four datasets, sampling token quadruples per prompt and layer and normalizing four-point GH by the largest pairwise distance.Generated responses are excluded so prompt geometry is not confounded with decoding length, answer correctness, or model-specific generation behavior.Source: §1 · p. 2
- Middle layers form a high-δrel plateau while final layers usually become more tree-like; scale effects are weak and non-monotonic, families differ, and domain patterns interact with specialization.HumanEval is high-δrel for Qwen2.5-7B-Base but the lowest-δrel domain for Qwen2.5-7B-Coder-Instruct.Source: §1 · p. 2
- The authors present GH as a diagnostic for locating where prompt-token distances become more tree-like and which model–layer–domain comparisons merit follow-up, not as proof of symbolic reasoning or a causal circuit.This is the paper's stated scope for interpreting its GH map.Source: §1 · p. 2
2 Background
The background defines Gromov hyperbolicity as a four-point measure of how closely a metric resembles a tree, with lower values indicating greater tree-likeness. It then specifies the normalized statistic and the prompt-token hidden-state sets to which the paper applies it.
2.1 Hyperbolicity as Tree-Likeness
- Gromov hyperbolicity measures tree-likeness through distances among four points: a tree metric has δ = 0, and larger values indicate greater deviation from tree-likeness.The paper sorts three sums of pairwise distances and uses the difference between the largest and second-largest values as the four-point hyperbolicity.Source: §2.1 · p. 2
- The paper reports a diameter-normalized four-point estimate because raw δ depends on distance scale; lower δrel means token-state distances are more tree-like after normalization.The normalization uses the maximum observed pairwise distance in the sampled hidden-state set.Source: §2.1 · p. 2
2.2 Prompt Hidden-State Sets
- For each prompt and layer, the analysis collects all prompt-token hidden states and estimates δrel using Euclidean distances.Using prompt tokens alone isolates input-distribution geometry from generated-text length, decoding behavior, and answer correctness.Source: §2.2 · p. 2
3 Experimental Design
The experiments cover ten open-weight checkpoints from five model families and four datasets spanning math, code, commonsense, and truthfulness. A shared prompt-only measurement protocol produces per-example, per-layer estimates, which the authors analyze as distributions using paired or unpaired nonparametric comparisons.
3.1 Models, Data, and Scope
- The study spans ten open-weight checkpoints from Llama, Qwen2.5, Qwen2.5-Math, Qwen2.5-Coder, and DeepSeek-R1-Distill-Qwen, across MATH500, HumanEval, WinoGrande, and TruthfulQA.The four datasets cover mathematical reasoning, code generation, commonsense pronoun resolution, and truthfulness; each checkpoint uses the same dataset composition.Source: §3.1 · pp. 2–3
3.2 Measurement Protocol
- The protocol yields 818,904 sample-layer measurements by extracting prompt-token states at every available layer and sampling 2,000 quadruples per example-layer set.The analysis uses medians and interquartile ranges, Wilcoxon signed-rank tests for paired layer comparisons, and Kruskal–Wallis tests for unpaired family and domain comparisons; results are interpreted as distributions, not yes-or-no labels.Source: §3.2 · p. 3
4 Results
Across the four axes, depth is the clearest regularity: middle layers usually have high relative hyperbolicity, while final layers become more tree-like; scale effects are weak, and family and domain patterns vary.
4.1 RQ1: Scale Effects Are Conditional
- −0.029 (Qwen2.5 base) and +0.035 (Qwen2.5-Math) are the pooled 7B-minus-1.5B shifts, while base-track domain shifts reverse direction.Layer profiles also show that scale changes when models enter or leave the high-δrel middle-layer range, rather than shifting the whole curve uniformly.Source: §4.1 · p. 3, Figure 1 · p. 4, Table 1 · p. 4
4.2 RQ2: Depth Dominates the Atlas
- 0.995–0.996 middle-layer medians across Qwen2.5-7B-Base domains contrast with final medians from 0.549 (MATH500) to 0.676 (HumanEval).Final-minus-middle shifts are negative for every model–domain pair, though their magnitude varies by model and domain.Source: §4.2 · p. 4, Figure 2 · p. 5, Table 2 · p. 5
4.3 RQ3: Matched Families Occupy Different Regimes
- 0.429 (Llama-3-8B-Instruct) to 0.633 (Qwen2.5-Math-7B) spans pooled matched-model medians, exceeding the pooled scale shifts.Specialization separates Qwen variants: instruction tuning lowers δrel broadly, while the coder variant is closer to Llama than to Qwen2.5-7B-Base; the comparisons do not isolate a cause.Source: §4.3 · p. 5, Figure 3 · p. 6, Table 3 · p. 6
4.4 RQ4: Domains Interact with Specialization
- For Qwen2.5-7B-Base, final-layer medians rise from 0.549 (MATH500) through 0.604 (TruthfulQA) and 0.648 (WinoGrande) to 0.676 (HumanEval).HumanEval remains high despite being the smallest dataset, so sample size alone does not explain this ordering.Source: §4.4 · pp. 5–6, Figure 4 · p. 7, Table 4 · p. 7
- HumanEval shifts from 0.676 in Qwen2.5-7B-Base to 0.438 in Qwen2.5-7B-Coder-Instruct, while math specialization retains higher δrel on MATH500.Instruction tuning lowers Qwen2.5-7B-Base from 0.676 to 0.539 on HumanEval and from 0.604 to 0.471 on TruthfulQA; prompt-length controls do not explain the large family and specialization effects.Source: §4.4 · pp. 6–7, Figure 4 · p. 7, Table 4 · p. 7
4.5 Cross-RQ Patterns
- Across RQs, the repeatable pattern is an early-variable, high-δrel middle plateau followed by a model-specific final drop, so scale and family contrasts are best read with layer profiles.Specialization changes which domains receive a tree-like final organization, and matched-scale family differences exceed simple parameter-count explanations.Source: §4.5 · p. 7
5 Discussion
The discussion interprets hyperbolicity as a layerwise trajectory shaped by model and domain, and argues that the diagnostic is most informative in matched comparisons with depth profiles.
5.1 Interpreting the Geometric Patterns
- Middle layers form a high-δrel plateau before final layers often move toward lower values, with the size of that drop depending on family and domain.The proposed interpretation is that middle layers mix organizational principles, while final-layer compression is a possible reading, not a causally identified mechanism.Source: §5.1 · pp. 7–8
- Coder specialization lowers HumanEval δrel, whereas math specialization can raise final-layer δrel relative to the base track, so the same domain can differ across models.The paper suggests training or adaptation changes which domains receive a tree-like final organization.Source: §5.1 · p. 8, §5.2 · p. 8
5.2 Scope and Meaning of the Diagnostic
- Across 50 HumanEval prompts, final-layer Coder–Base differences were −0.242 for prompt-only states, −0.254 for generated-code states, and −0.238 for full sequences.The Coder model was lower in 50/50 prompt sets, 45/50 generated-code sets, and 48/50 full sequences under greedy decoding.Source: §5.2 · p. 8
5.3 Implications for Future Analyses
- Geometric studies should pair final-layer summaries with depth profiles and match model comparisons by family, scale, and domain where possible.Otherwise, domain and specialization effects can be mistaken for scale effects; hyperbolicity is best used as one coordinate in a broader analysis suite.Source: §5.3 · p. 8
6 Related Work
Prior work studies learned hyperbolic representations and other representation-geometry diagnostics; this paper instead measures GH in existing LLM prompt hidden states across several comparison axes.
6 Related Work
- Prior research covers hyperbolic representation learning and geometry-based NLP applications, alongside analyses of anisotropy, syntactic geometry, intrinsic dimension, and neighborhood variation.The paper distinguishes its setting by measuring the same GH statistic in unmodified LLM prompt hidden states across scale, depth, family, and domain.Source: §6 · pp. 8–9
7 Conclusion
The paper presents a prompt-only hyperbolicity atlas across ten models, four datasets, and all layers. Its central conclusion is that depth, family, and specialization structure the measured geometry more clearly than scale alone.
7 Conclusion
- Across ten models, four datasets, and all layers, the atlas finds depth-dominant structure, matched-family differences, domain–specialization interactions, and weak, non-monotonic scale effects.The paper frames hyperbolicity as a model–layer–domain diagnostic, not a direct claim about symbolic hierarchy.Source: §7 · p. 9
Limitations
The atlas is limited to prompt-token hidden states, and its measurements depend on several design choices and sampling conditions.
Limitations
- The atlas measures prompt-token states only, and results may depend on tokenization, prompt templates, quadruple sampling, normalization, distance choice, and fixed sampling budgets.The random-initialization control covers one architecture, one initialization, and two domains; broader experiments are needed to establish how widely that pattern holds.Source: Limitations · p. 9