PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Story

Vladislav Pedashenko, Laida Kushnareva, Yana Khassan Nibal, Eduard Tulchinskii, Kristian Kuznetsov, Vladislav Zharchinskii, Yury Maximov, Irina Piontkovskaya

arXiv:2511.15210v1cs.CLcs.AIcs.LG

TL;DR

Textual determinants of intrinsic dimension remain underexplored, despite ID’s use in LLM analysis. This study combines linguistic analysis with SAE features and steering to examine ID’s relation to text properties. It finds that ID is largely uncorrelated with entropy after length control and varies with style and genre.

  • Problem

    Geometric complexity in text remains comparatively underexplored relative to ubiquitous entropy-like measures.

    Source: §1 · p. 1

  • Method

    The study relates ID to text properties through linguistic analysis, SAE feature correlations, and feature steering during generation.

    Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 3

  • Results

    ID is largely uncorrelated with prediction-based measures after length control and is lower for scientific or informational prose than for personalized, opinionated, or fictional writing.

    Source: §6 · p. 8

  • Takeaways & Limitations

    ID is a domain- and style-sensitive geometric lens that exposes structural regularities not captured by likelihood alone.

    Source: §7 · p. 8

    PHD analysis uses only Gemma, RoBERTa, and Qwen embeddings, and random subsampling may cause estimate variability across runs.

    Source: §8 · p. 8

Abstract

from arXiv · show

Intrinsic dimension (ID) is an important tool in modern LLM analysis, informing studies of training dynamics, scaling behavior, and dataset structure, yet its textual determinants remain underexplored. We provide the first comprehensive study grounding ID in interpretable text properties through cross-encoder analysis, linguistic features, and sparse autoencoders (SAEs). In this work, we establish three key findings. First, ID is complementary to entropy-based metrics: after controlling for length, the two are uncorrelated, with ID capturing geometric complexity orthogonal to prediction quality. Second, ID exhibits robust genre stratification: scientific prose shows low ID (~8), encyclopedic content medium ID (~9), and creative/opinion writing high ID (~10.5) across all models tested. This reveals that contemporary LLMs find scientific text "representationally simple" while fiction requires additional degrees of freedom. Third, using SAEs, we identify causal features: scientific signals (formal tone, report templates, statistics) reduce ID; humanized signals (personalization, emotion, narrative) increase it. Steering experiments confirm these effects are causal. Thus, for contemporary models, scientific writing appears comparatively "easy", whereas fiction, opinion, and affect add representational degrees of freedom. Our multi-faceted analysis provides practical guidance for the proper use of ID and the sound interpretation of ID-based results.

1 Introduction

The introduction frames textual intrinsic dimension (ID) as an underexplored geometric measure complementary to prediction-based complexity metrics. It situates the study amid prior ID research and outlines analyses of genre, linguistic regularities, and interpretable semantic features.

1 Introduction

  • Geometric complexity remains comparatively underexplored, motivating a study of ID alongside ubiquitous entropy-like measures.The introduction distinguishes predictive measures such as entropy, cross-entropy, and perplexity from geometric measures such as intrinsic dimension.Source: §1 · p. 1
  • ID depends on hidden-representation geometry, whereas entropy follows unembedding; after controlling for text length, they are essentially uncorrelated.The paper presents ID and entropy as complementary rather than substitutable measures.Source: §1 · p. 1
  • The study examines ID across domains and styles, linguistic regularities, and semantic axes using sparse autoencoders and feature steering.The introduction reports associations with lexical diversity and cross-sentence repetition, and describes probing causal effects of semantic features on generation.Source: §1 · p. 1

1.1 Related Work

  • Early NLP work connected task-specific variation in pretrained models to low-dimensional subspaces and motivated compact-subspace fine-tuning such as LoRA (Aghajanyan et al., 2021; Hu et al., 2022).Earlier work by Li et al. (2018) studied low-dimensional solution subspaces in parameter space.Source: §1.1 · p. 2
  • Representation-space studies link ID to scaling (Havrilla and Liao, 2024), training dynamics (Razzhigaev et al., 2024), and semantic compositionality (Lee et al., 2025).Other cited applications include architecture search, memorization, and entropy or loss analyses.Source: §1.1 · p. 2
  • Prior PHD work addresses generalization and text detection, while local-ID studies examine hallucination signals, token neighborhoods, and layer-wise variation (Yin et al., 2024; Ruppik et al., 2025).The related work also describes local ID for human–machine segment boundaries and dimensionality reduction for downstream tasks.Source: §1.1 · p. 2

2 Intrinsic Dimension

The paper defines intrinsic dimension as a geometric measure of degrees of freedom and distinguishes it from prediction-based entropy. It argues that the two capture different facets of text complexity, especially after controlling for length.

2 Intrinsic Dimension

  • ID measures a dataset’s degrees of freedom, matching ordinary dimension for linear spaces and smooth manifolds.Extensions to metric spaces aim to remain robust to noise and recoverable from finite samples.Source: §2 · p. 2
  • TwoNN, MLE, TLE, and PHDIM estimate ID using different geometric information and assumptions.TwoNN uses nearest-neighbor distance ratios, MLE fits a local Poisson model, TLE uses extreme-value theory, and PHDIM captures multi-scale connectivity and holes.Source: §2 · p. 2

2.1 Relation between ID and model-prediction entropy

  • ID depends on hidden-representation geometry, while entropy depends on how embeddings align with the vocabulary during unembedding.The same geometric structure can yield different entropies when the density of nearby unembedding vectors differs.Source: §2.1 · p. 3
  • After controlling for text length, ID and entropy are essentially uncorrelated, though residual dependence remains in the very low-ID regime.The paper notes that synonym-rich neighborhoods can raise entropy without changing geometry, while structurally complex but predictable text can have low entropy and high ID.Source: §2.1 · p. 3

3 Intrinsic Dimension through the Lens of Text Semantics

This section investigates which semantic and stylistic properties associate with higher or lower ID, combining interpretable model features, interventions, and independent linguistic diagnostics. Its SAE analyses measure feature–ID relationships and probe feature effects during generation.

3 Intrinsic Dimension through the Lens of Text Semantics

  • The section aims to identify semantic and stylistic properties associated with higher or lower ID using interpretable representations, interventions, and linguistic diagnostics.These approaches combine sparse features learned from activations with causal probes and independent text-property measures.Source: §3 · p. 3

3.1 Sparse Autoencoders (SAEs) and feature steering

  • Steering modifies a hidden state along a selected SAE feature direction during generation to test effects on style, content, and representational complexity.Interventions are assessed for controlled changes while preserving fluency; the feature scale uses its held-out maximum activation and λ sets intervention strength.Source: §3.1 · p. 3
  • SAE features are aggregated from token activations to sequence level, then features with the strongest positive and negative ID correlations are selected.The selected features are interpreted through activation contrasts, domain specialization, and steering experiments.Source: §3.1 · p. 3
  • Independent linguistic diagnostics relate ID to lexical diversity, cohesion, syntactic diversity, and POS-sequence compression.The measures include moving-window type–token ratios, sentence-to-sentence overlap, dependency-graph distances, and gzip compression of POS-tag sequences.Source: §3.1 · p. 3

4 Experimental details

The experiments use a filtered, diverse text corpus and additional proficiency and controlled-generation datasets. SAE analyses use Gemma activations, with steering strengths selected according to output coherence.

4 Experimental details

  • The cleaned COLING development split contains 172,750 texts retained from 261,758 after token-length and PHDIM-stability filtering.The corpus spans multiple domains and human and LLM-generated texts; most semantic interpretation experiments use human-written texts.Source: §4 · p. 4
  • Proficiency analyses use 567 versions of 189 human-authored texts, while controlled generation uses RAID prompts across temperatures and five domains.RAID includes 1,000 prompts for temperature–ID studies and 20 prompts per domain for fine-grained steering.Source: §4 · p. 4
  • The study uses Gemma-Scope residual SAEs on Gemma-2-2B, and reports the largest tested steering strength per feature that preserves coherence.Tested λ values range from 0.25 to 15; very large values can produce repetitive or broken outputs.Source: §4 · p. 4
  • PHD estimators across Gemma, RoBERTa, and Qwen each correlate above 0.5 with the other estimators, making PHD a stated compromise.The figure reports correlations among estimators across the three models.Source: Figure 2 · p. 4

5 Results

Across sources, encoders, and feature analyses, intrinsic dimension (ID) varies with text properties: scientific and informational prose has lower ID than creative and opinionated writing. Estimator comparisons, linguistic correlates, and SAE steering results further characterize how this variation arises.

5.1 Comparison of ID estimators

  • PHDIM, MLE, TLE, and TWONN estimates are substantially correlated; PHDIM tracks the other estimators best, with r > 0.67 for each on the same embeddings.Gemma and Qwen agree more closely than RoBERTa across embeddings; well-formed texts generally fall in an ID band of about 5–15.Source: §5.1 · pp. 4–5

5.2 PHDim and data properties

  • For texts longer than 150 tokens, PHDIM correlates with gzip compression ratio above 0.3, but the wedge-shaped relationship bounds rather than predicts ID.PHDIM is stable for sufficiently long samples, while short texts have high variance; its correlation with explained variance peaks near rank 60 despite local ID around 10.Source: §5.2 · p. 5, Figure 3 · p. 5
  • PHDIM rises with lexical diversity and falls with sentence-level overlap or repetition, while syntax and discourse metrics show weak associations with well-formed text.Perturbations preserving repetition patterns retain this relationship even when semantics are disrupted; broken or disconnected generations can produce extreme ID values.Source: §5.2 · p. 5, Figure 4 · p. 5

5.3 ID divergence by text source and domain

  • Mean PHDIM is 7.5–8 for scientific and technical sources, 8–9 for factual and reporting texts, and 9–10.5 for essays, opinionated writing, and informal or creative content.Fiction from wp has the highest ID among public-domain corpora, while pubmed papers have the lowest; domain ordering is nearly identical across encoders.Source: §5.3 · pp. 5–6, Figure 5 · p. 6
  • Student-story PHDIM rises from a mean of 9.5 for elementary texts to 12.5 for advanced texts, yet genre matters more: even elementary fiction exceeds scientific and informational genres.Simplified syntax at the elementary level has little effect on embedding dimensionality.Source: §5.3 · p. 6

5.4 SAE-based interpretation

  • SAE features associated with higher ID yield more emotional, personalized, rhetorically complex text, while negatively correlated features shift generations toward formal academic and reporting styles.Later layers encode global discourse templates, while layer 16 modulates local stylistic details such as scientific framing and statistics.Source: §5.4 · p. 6, Figure 6 · p. 6

6 Discussion

The discussion frames ID as a geometric measure complementary to prediction-based metrics and sensitive to text style and domain. It cautions against treating ID as a single proxy for difficulty and highlights its relationship to linguistic features and nonlinear embedding structure.

Complementarity and operating range

  • After controlling for length, ID and cross-entropy are largely uncorrelated, so ID captures geometric or structural information not reflected in likelihood.For well-formed text, ID concentrates around 5–15, while the effective global linear embedding dimension is about 60–100, reflecting a highly curved manifold.Source: §6 · pp. 6, 8

Failure detection

  • Extremely low ID flags repetitive or looped generations, whereas extremely high ID flags disconnected fragments or incoherent mixtures.Source: §6 · p. 8

Compression and ID

  • ID correlates moderately with gzip compression ratio, but compression provides bounds on plausible ID rather than accurate pointwise estimates.Source: §6 · p. 8

Linguistic and style correlates

  • ID increases with lexical diversity and decreases with cross-sentence repetition or overlap, while remaining comparatively insensitive to syntax in well-formed text.Scientific and informational prose has lower ID than personalized, opinionated, or fictional writing; personality, emotion, narrative, stance, and informality are associated with higher ID.Source: §6 · p. 8

Implications and cautions

  • Treating ID as a monolithic proxy for difficulty risks misinterpretation; evaluations confined to low-ID corpora can yield misleading conclusions.The discussion recommends targeting higher-ID domains and reporting local ID structure rather than relying only on linear projections or prediction-based metrics.Source: §6 · p. 8

7 Conclusion

The conclusion presents ID as a domain- and style-sensitive property of text representations in large language models, largely orthogonal to prediction-based measures. It emphasizes that ID reveals structural regularities not captured by likelihood alone.

7 Conclusion

  • ID is largely orthogonal to prediction-based measures such as cross-entropy, while scientific and informational texts occupy lower-ID regions than opinionated, narrative, and personalized writing.Source: §7 · p. 8

8 Limitations

The limitations section bounds the PHD analysis by encoder coverage, sampling variability, and differences among estimators. The appendix outlines intrinsic-dimension definitions using uniformly distributed points and describes MLE, TwoNN, and persistent-homology approaches.

8 Limitations

  • PHD uses embeddings from three models, may vary across encoders and random subsamples, and relies on estimators with complementary but non-equivalent assumptions.The estimators are not directly comparable.Source: §8 · p. 8

A.1 Intrinsic dimension definitions

  • MLE dimension uses the exponent governing how the probability of a random point falling within a point’s ε-neighborhood scales.Nε(x) counts points in x’s ε-neighborhood.Source: §A.1 · p. 11
  • TwoNN estimates dimension by fitting a model CDF to the empirical distribution of nearest-to-second-nearest-neighbor distance ratios.For uniform data in Rd, the ratio distribution depends only on d.Source: §A.1 · p. 11
  • PHDim counts nontrivial homological features across scale; an equivalent characterization uses the minimum spanning tree’s total length.The passages describe the asymptotic MST characterization without giving its formula.Source: §A.1 · p. 11

A.2 Three Arguments Against Functional Dependence Between Entropy and Intrinsic Dimension

Three constructive arguments reject a general functional dependence between entropy and intrinsic dimension (ID); appendix analyses also report estimator relationships and a general rise in PHD with model size.

A.2 Three Arguments Against Functional Dependence Between Entropy and Intrinsic Dimension

  • Hidden-state geometry determines ID independently of the unembedding matrix U, while changing token projections can produce different entropy distributions for subsets with unchanged geometry.The paper presents this as a constructive counterexample to a necessary entropy–ID relationship.Source: §A.2 · pp. 11–12
  • Scaling hidden states preserves local dimensionality while changing prediction entropy through inverse-temperature rescaling, so entropy can vary while ID remains invariant.The authors conclude that entropy and ID are affinely independent quantities.Source: §A.2 · p. 12
  • Mean entropy varies continuously under infinitesimal increases in intrinsic dimension, so it cannot be a function of ID alone; observed neural-network correlations are empirical, not necessary.The authors qualify the argument: neural-network embeddings and parameter spaces are constrained by training dynamics and architecture.Source: §A.2 · pp. 11–12

B.1 MLE, TLE and TwoNN upon embeddings from main models

  • MLE and TLE follow similar tracks, TwoNN differs substantially, and PHD remains strongly correlated with the other estimators; RoBERTa shows nonlinear relationships, including non-monotonic TwoNN.For Qwen and Gemma, estimator relationships are nearly linear; RoBERTa is the exception.Source: §B.1 · p. 12, Figure 8 · p. 13, Figure 9 · p. 14

B.2 PHD dependence on model size

  • PHD generally increases with Qwen3 model size, though Qwen3-4B stands out and may differ because of its context-window and embedding-tie setup.The comparison uses Qwen3 base models from 0.6B to 32B on human texts longer than 150 tokens.Source: §B.2 · p. 13, Figure 11 · p. 13

B.3 Intrinsic Dimensionality and data geometry

The section relates intrinsic-dimension estimators to embedding-geometry measures and finds model-dependent correlation patterns; it also examines how text length affects PHD estimates.

B.3 Intrinsic Dimensionality and data geometry

  • ID estimators are highly correlated with one another; they correlate negatively with maximal explained variance for Gemma2 and Qwen2.5, while RoBERTa differs.The authors suggest the encoder-versus-decoder distinction may explain RoBERTa’s different pattern.Source: §B.3 · pp. 15–16
  • The negative correlation with cumulative explained variance strengthens as components are added, then weakens; TwoNN has the weakest correlation, consistent with its reliance on local structure.Principal components are computed over the whole dataset, unlike TwoNN’s primarily local estimate.Source: §B.3 · p. 16

C.1 PHD dependence on text length (tokens)

  • PHD variance is high for short texts and stabilizes beyond about 150 tokens; short texts also show a wide spread of Gemma PHD values.Figures 13 and 14 show the length-dependent standard deviation and scatter, respectively.Source: §C.1 · p. 16, Figure 13 · p. 16, Figure 14 · p. 16

C.2 Syntatic diversity

The section examines syntax, text transformations, token-neighbor distances, proficiency, and generation temperature in relation to PHD. It finds that syntax is not a primary driver for high-quality text, while several other text characteristics and generation conditions coincide with PHD differences.

C.2 Syntatic diversity

  • PHD is largely insensitive to syntax in high-quality texts; lexical and thematic diversity are its primary drivers, while human texts are more syntactically diverse than current LLM outputs.Human-written texts also have relatively high POS compression ratios with notably low variance, though their ratios are not the highest among model groups.Source: §C.2 · pp. 16, 18

C.3 More details on TAACO features and their relations with ID

  • Figure 20 and Table 7 report additional correlations between TAACO features and PHD across embedding models.The section describes features including function-word diversity, lexical repetition, sentence overlap, and discourse connectives.Source: §C.3 · pp. 19–20, Figure 20 · p. 21, Table 7 · p. 21

C.4 Text transformations and their effect on PHD

  • Transformation 3 raises median PHD for Gemma/Qwen but has little effect for RoBERTa; Transformations 1 and 2 lower it for RoBERTa but have no effect or slightly raise it for Gemma/Qwen.The transformations respectively shuffle internal word letters, replace letters independently with homoglyphs, or apply homoglyph replacements consistently by word type.Source: §C.4 · pp. 20–21, Table 8 · p. 21

C.5 Distribution of distances to the nearest neighbor for tokens

  • Tokens sharing a nearest neighbor with themselves have smaller average nearest-neighbor distances than other tokens, indirectly supporting the reported positive correlation with PHD.The authors note that shorter average neighbor distances reduce the minimum spanning tree and may affect its asymptotic behavior.Source: §C.5 · p. 21

C.6 PHD and proficiency level of text

  • PHD is lower for texts written for beginners than for texts written for intermediate and advanced readers.The authors relate this pattern to lower type–token ratio and greater lemma repetition across consecutive sentences in beginner texts.Source: §C.6 · p. 21

C.7 PHD and generation temperature

  • For Qwen-3-8B-base, PHD rises rapidly from temperature 0.2 to 0.8 then levels off; for Qwen-3-8B-instruct, it grows more slowly and almost linearly.Low-temperature base generations show very low lexical diversity and high repetition; the authors suggest repetition may explain the different PHD pattern.Source: §C.7 · p. 22

D SAE-based experiments

The section describes a linear model for predicting PHD, feature-steering examples, a GPU implementation for costly calculations, and the computational resources and licenses used.

D.1 Linear approximation for PHD

  • A simple interpretable model using SAE features and lexical-diversity metrics was trained on human texts longer than 150 tokens to predict Gemma PHD; test-prediction Pearson and Spearman correlations for internal-dimensionality estimates appear in Table 9.Median feature-importance ranks are also reported in Table 9.Source: §D.1 · p. 22, Table 9 · p. 22

E Speeding up the running time of the PHDim calculation.

  • Pairwise-distance calculation is the costly PHDim step; texts over 1024 tokens can take nearly a week on the stated CPU, so the authors moved distance-matrix calculation to GPU.For shorter texts, the speedup is slight because data transfer between CPU and GPU takes time.Source: E Speeding up the running time of the… · pp. 22–23

D.2 Steering examples

  • Table 11 presents illustrative steering effects for selected features across four prompts, with feature sign denoting its positive or negative correlation with ID.The text identifies the examples as demonstrations of the selected features’ steering effects.Source: §D.2 · p. 22, Table 11 · p. 23

F Computational resources

  • Experiments used servers with either two RTX 4090 GPUs and an i7-14700K CPU, or four A100 80 Gb GPUs and a Xeon Gold 6348 CPU.The passage lists both compute configurations.Source: F Computational resources · p. 23

G Artifact licenses

  • The listed artifact licenses include MIT for COLING and RAID, CC BY-NC-SA 4.0 for TAACO, and CC BY-SA 4.0 for OneStopEnglish.These licenses are specified for the named datasets and tool.Source: G Artifact licenses · p. 23
Loading 2511.15210v1…