Language Models are Injective and Hence Invertible

• The result challenges the assumption that Transformer nonlinearities, normalization, and many-to-one attention necessarily discard information and make exact input recovery impossible.The analytic framework identifies collisions as measure-zero exceptions and notes that deliberate non-analytic choices such as quantization or tying can create them.Source: §1 · pp. 1–2
• SIPIT reconstructs the exact input prompt from hidden activations, providing the first provable worst-case linear-time guarantee and often running faster in practice.The algorithm operationalizes injectivity through gradient-based reconstruction.Source: §1 · pp. 1–2
• Injectivity is established at random initialization and preserved through any finite number of gradient-descent steps with step sizes in (0, 1).The theorem assumes real-analytic components, at least one attention head per block, finite vocabulary and context, and initialization distributions with a density.Source: §1 · p. 2
• Standard decoder-only Transformers are almost surely injective: distinct prompts produce distinct last-token representations across all prompts within the finite vocabulary and context length.The guarantee holds for finite width, depth, and training time, except on a measure-zero set of pathological parameters.Source: §1 · p. 2

Proof mechanism

• The proof establishes real-analyticity for embeddings, attention, LayerNorm with ε > 0, MLPs with analytic activations, residuals, and their compositions.A single separating parameter setting rules out identically zero differences between any two prompts; analytic zero sets are therefore measure zero.Source: §2 · p. 3

Initialization

• Distinct prompts almost surely produce distinct last-token representations at initialization, because collision-inducing parameter settings form a measure-zero set.Real-analyticity makes each collision set measure zero, while standard continuous initializations avoid such sets with probability one.Source: §2 · p. 3

Training preservation

• Injectivity persists after any finite sequence of gradient-descent updates, including SGD, mini-batch, and full-batch training.Analytic update maps preserve absolute continuity of the parameter distribution, so training cannot concentrate it on measure-zero collision sets.Source: §2 · p. 4

Finite-set guarantee

• For every finite prompt set, the corresponding representations are almost surely all distinct, strengthening pairwise injectivity to global finite-set distinctness.The guarantee applies across arbitrary finite collections of prompts within the model’s vocabulary and context window.Source: §2 · p. 4

Failure cases

• Collisions remain possible only through deliberately engineered choices, such as identical token or positional embeddings, quantization, or non-smooth components.Under continuous random initialization and standard training, these coincidences have probability zero.Source: §2 · pp. 4–5

Contribution

• The method turns almost-sure representation injectivity into constructive exact recovery, making prompt inversion a linear-time structural consequence of Transformer representations.The paper emphasizes that this property is not a quirk of initialization or training and can often be faster in practice.Source: §3 · pp. 5–6

Threat model

• SIPIT applies when an adversary has access to all per-position hidden states at a fixed layer, such as through a leaked KV-cache or exposed intermediate representations.Exact recovery from only the final embedding is guaranteed in principle but an efficient algorithm for that setting is left to future work.Source: §3 · p. 5

Recovery procedure

• SIPIT sequentially recovers each token by matching its observed hidden state against candidate continuations of the already recovered prefix.Causal computation makes each position depend only on the known prefix and current token; candidate matches may use a tolerance ε.Source: §3 · pp. 5–6
• At each position, SIPIT cycles through vocabulary candidates using a policy, verifies a match, appends the token, and proceeds to the next position.The algorithm accepts a candidate when its predicted state matches the observed state within the verification criterion, then updates the reconstructed prefix.Source: §3 · p. 6

Correctness and guarantees

• SIPIT recovers the true input sequence with probability one in at most T|V| steps.Correctness follows because local injectivity guarantees one matching token at each position, and exhaustive candidate search finds it within |V| trials.Source: §3 · p. 6
• Under the stated perturbed-state conditions, SIPIT still recovers the true sequence with probability one and terminates within T|V| steps.The robustness theorem applies the same recovery and step bound to perturbed hidden states under its defined conditions.Source: §3 · p. 6

4.1 SEARCHING FOR COLLISIONS

• No collisions occurred across all tested models and layers, with distinct prompts producing distinct last-token states and clear separation at first, middle, and final layers.The tests used 100k prompts and around 5 billion pairwise comparisons; Table 1 reports minimum distances well above the 10−6 collision threshold.Source: §4.1 · p. 7, Table 1 · p. 7
• Minimum pairwise distances never approached zero, rose rapidly at short sequence lengths, then stabilized without pathological collapses.Similar behavior was observed for GPT-2 Small and Gemma3, indicating persistent separability across sequence lengths.Source: §4.1 · p. 8, Figure 5 · p. 8
• Exhaustive continuation searches over more than 343 billion prompt pairs per model also found distances far from zero, confirming local injectivity rather than sampling chance.The search focused on the 10 prompts with the smallest last-token distances and identified five highly similar pairs without collisions.Source: §4.1 · p. 8
• Quantization introduced no collisions, more than doubled minimum representation distances, and preserved separation as model size increased substantially.The experiments covered FP4 and INT8 quantization, including Phi-4 (14B) and Llama-3.1-70B.Source: Table 2 · p. 7, Table 3 · p. 7, §4.1 · p. 8

4.2 INVERTIBILITY RESULTS

• 100% accuracy with less than 0.22% of the vocabulary explored: SIPIT remains robust under FP4 quantization and scales linearly across approximately 32K and 128K vocabularies.Inversion runtimes also rise only slightly from the first to final layer, remaining efficient across depth.Source: Figure 6 · p. 8, Table 4 · p. 9, §4.2 · p. 9
• 100% token-level accuracy: SIPIT exactly recovers every 20-token prompt, whereas HARDPROMPTS fails and BRUTEFORCE succeeds only at prohibitive computational cost.The comparison supports SIPIT’s linear-time convergence guarantee.Source: Table 5 · p. 9, §4.2 · p. 9

Analytical properties of Transformers

• Transformer components appear many-to-one analytically because LayerNorm collapses per-example statistics, residuals can cancel, attention-only rank decays doubly exponentially, and the softmax bottleneck limits reachable distributions.The passage notes that completeness and universal-approximation results also permit highly many-to-one Transformer maps in principle.Source: §5 · p. 9
• The paper proves injectivity for the parameter-dependent, task-relevant last-token hidden state, and shows it persists through training rather than holding only at random initialization.This contrasts with Sutter et al. (2025), who establish almost-sure injectivity for the entire hidden-state matrix only at initialization, while Jiang & Haghtalab (2025) study almost-sure surjectivity.Source: §5 · p. 9

Inverse problems in language modeling

• Prior inversion methods reconstruct prompts approximately using trained inverters over outputs, probabilities, logprobs, or encoder logits, without guarantees and not from decoder-only hidden states.These methods include Sun et al. (2021), Zhang et al. (2024), Morris et al. (2023b), Nazir et al. (2025), and Morris et al. (2023a).Source: §5 · p. 10

Automated prompt optimization

• Prompt-optimization methods find effective but approximate prompts, while Thomas et al. (2025) ranks every vocabulary token without injectivity or formal exactness guarantees.These approaches include Guo et al. (2025), Sun et al. (2022), Deng et al. (2022), AutoPrompt (Shin et al., 2020), and Hard Prompts Made Easy (Wen et al., 2023).Source: §5 · p. 10

Novelty

• Unlike all prior work, the approach is training-free, efficient, and offers provable linear-time guarantees for exact recovery from internal states.This positions the method against both trained inversion systems and sequential prompt-recovery methods lacking exactness guarantees.Source: §5 · p. 10

• Decoder-only Transformers are almost surely injective, and SIPIT recovers exact input sequences from hidden activations with provable linear-time guarantees.These results make injectivity a rigorously grounded and operational property rather than an informal belief.Source: §6 · p. 10
• Injectivity reconciles apparently lossy Transformer mechanisms with end-to-end preservation of all input-sequence information, supporting causal and probing analyses of last-token states.SIPIT provides constructive inversion and a baseline for interpretability and auditing when probes or inversion methods fail.Source: §6 · p. 10
• Hidden states encode users’ exact prompts, so systems storing, caching, or transmitting them should face the same data-protection obligations as raw prompts (Miranda et al., 2025; HmbBfDFI, 2024).The passage distinguishes inference-time hidden representations from training data encoded in model weights.Source: §6 · p. 10
• Open directions include extending analysis to multimodal Transformers, studying inversion under noise or quantization, and connecting technical results with evolving regulation.These directions are presented as important for clarifying robustness and supporting safe, responsible deployment.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

• The experiments, proofs, implementation details, and additional experiments are documented across Sections 4 and Appendices A–F.The benchmark uses 100k prompts uniformly sampled from four public datasets, while the appendices cover model analyticity, injectivity, SIPIT, implementation, and activation-function assumptions.Source: REPRODUCIBILITY STATEMENT · p. 11, appendix · p. 11

A PRELIMINARIES

• Appendix A fixes notation and assembles differential, measure-theoretic, and topological tools for later local-invertibility and absolute-continuity arguments.Its scope includes real-analyticity, zero sets, Fréchet derivatives, Hessians, the inverse function theorem, pushforwards, and absolute continuity.Source: A PRELIMINARIES · p. 17

A.1 NOTATION

• The model is a function f : V≤K × R^p → R^d over a finite vocabulary and bounded-length input set, with outputs interpreted as token embeddings.Inputs are discrete sequences, and the finite sequence domain reflects practical context-length limits.Source: §A.1 · p. 18

A.2 REAL-ANALYTICITY

• Real-analyticity is defined for vector- and matrix-valued maps, with multi-index notation supporting the subsequent transformer-layer proofs.The matrix formulation is needed because transformer computations naturally operate on matrices.Source: §A.2 · p. 18

A.2.1 REAL-ANALYTIC FUNCTIONS WITH VECTOR INPUTS

• Vector-space analytic maps are closed under relevant algebraic operations and composition, while any nontrivial analytic map has a measure-zero zero set on connected open domains.Vector-valued analyticity can be established componentwise, and the zero-set result extends from scalar to vector maps.Source: §A.2.1 · pp. 19–20

A.2.2 REAL-ANALYTIC FUNCTIONS WITH MATRIX INPUTS

• Vectorization and matricization reduce matrix analyticity to vector analyticity, preserving equivalence and composition properties for matrix-valued maps.The proof uses that vectorization and matricization are mutual inverses and preserve the relevant norm relationship.Source: §A.2.2 · pp. 20–22

A.2.3 REAL ANALYTICITY OF COMMON COMPONENTS

• The appendix establishes real-analyticity for transformer building blocks including polynomials, exponentials, logarithms, softmax, row normalization, matrix products, Hadamard scaling, and stacking.These operations are intended to be composed in Appendix B to prove analyticity of the full transformer model.Source: §A.2.3 · pp. 22–23
• Transformer primitives, including row normalization and entrywise matrix polynomials, are real-analytic on their stated domains.Row normalization is analytic where row sums are positive, while polynomial component functions are analytic on all matrix space.Source: §A.2.3 · pp. 23–24
• Matrix multiplication, Hadamard scaling, concatenation, and noncommutative matrix polynomials preserve real-analyticity, allowing composition into full transformer layers.The closure arguments use componentwise products, finite sums, fixed-matrix multiplication, and stacking.Source: §A.2.3 · p. 24

A.3 DIFFERENTIAL, MEASURE-THEORETIC, AND TOPOLOGICAL TOOLS

• The appendix defines Fréchet derivatives, Hessians, Euclidean-topological properties, diffeomorphisms, and pushforward measures as foundations for later analysis.These tools include compactness, second countability, Lindelöfness, local compactness, and absolute continuity with respect to Lebesgue measure.Source: §A.3 · pp. 25–26
• The inverse function theorem gives a local C1-diffeomorphism whenever a square Jacobian has nonzero determinant, with a C1 local inverse.This theorem connects nonsingular derivatives to local invertibility and is paired with pushforward arguments for measure transport.Source: §A.3 · p. 26

B TRANSFORMER LANGUAGE MODEL

• Each Transformer block uses pre-layer-normalized multi-head attention and a row-wise MLP, with residual connections preserving the RT ×d shape across L stacked blocks.The complete backbone is defined as TrT := TB(L) ◦· · · ◦TB(1), with dimensionally valid residual additions at every depth.Source: B TRANSFORMER LANGUAGE MODEL · p. 29
• The language model maps token sequences to representations by summing learned token embeddings with learned positional embeddings, then applying stacked Transformer blocks and an unembedding layer.Token embeddings are sequence-length independent, while positional embeddings depend only on positions up to the context bound K.Source: B TRANSFORMER LANGUAGE MODEL · pp. 27, 29
• The embedding map (E, P) 7→Emb(s) = E(s) + PE(s) is real-analytic in the token- and positional-embedding parameters.The proof represents token and positional embeddings as constant-matrix products and uses closure under addition.Source: B TRANSFORMER LANGUAGE MODEL · p. 30

2. Layer Normalization. (x, γ, β) 7→LN(x) = γ ⊙

• Under a real-analytic activation such as tanh or GELU, MLPs, layer normalization, attention modules, residual blocks, and Transformer stacks are jointly real-analytic in inputs and parameters.Layer normalization is analytic because ε > 0 keeps the variance denominator nonzero, while the remaining operations are analytic compositions.Source: B TRANSFORMER LANGUAGE MODEL · p. 30, §2 · p. 30, §4 · p. 31

4. Self-Attention (vanilla or causal). Let Z =

• Vanilla self-attention is real-analytic because matrix products, scaling, row-wise softmax, and the final value multiplication are analytic operations.The result applies to maps of the form (X, Q, K, V) 7→softmax(Z)XV.Source: §4 · p. 30
• Causal self-attention in projection form is real-analytic because the masked exponential rows have strictly positive sums due to the unit lower-triangular diagonal.Row normalization is therefore evaluated within its valid domain before multiplication by XV.Source: §4 · p. 30
• Multi-head attention remains real-analytic for any mixture of vanilla and causal heads, since head concatenation and the output projection preserve analyticity.The conclusion follows by combining the single-head result with analytic horizontal concatenation and multiplication by WO.Source: §4 · p. 30

• The proof combines measure-zero collision arguments based on real-analyticity and witness constructions with preservation of absolute continuity under gradient descent.Gradient-descent updates preserve absolute continuity of the parameter distribution, enabling almost-sure distinctness across the finite sequence set.Source: C ALMOST SURE INJECTIVITY · p. 32
• The result assumes embedding dimension d ≥4, dη ≥1, and at least one attention head in every transformer block.These conditions are described as nonrestrictive for modern large language models.Source: C ALMOST SURE INJECTIVITY · p. 32
• The main theorem states that the last-token, last-layer representation map is injective almost surely after any finite training horizon.The result applies to finite vocabularies and bounded context lengths under the stated architectural assumptions.Source: C ALMOST SURE INJECTIVITY · p. 32

C.1 ABSOLUTE CONTINUITY ENSURES ALMOST SURE INJECTIVITY

• The separating construction handles length or final-token differences through embeddings and positions, and earlier differences through orthogonal embeddings and attention, also under causal masking.The construction requires d ≥4 and propagates a difference at an earlier position to the final output while later blocks remain identity.Source: §C.1 · pp. 33, 35
• Almost surely, distinct inputs have distinct last-token representations under any parameter distribution absolutely continuous with respect to Lebesgue measure, including every finite input family.Real-analyticity and explicit nonzero discrepancy constructions imply measure-zero collision sets; a finite union bound yields global distinctness.Source: §C.1 · pp. 33, 35

C.2 ABSOLUTE CONTINUITY OF THE PARAMETER DISTRIBUTION IS PRESERVED UNDER GD

• After any finite number of gradient-descent steps, the last-token representation map remains injective almost surely because gradient descent preserves absolute continuity of the parameter law.The proof establishes real-analyticity of the update, removes a measure-zero critical set using a nonzero Hessian determinant witness, and applies countable local inverse charts.Source: §C.2 · pp. 35–36

C.2.1 WITNESS CONSTRUCTION

• At the zero-gate witness, the Hessian’s mixed and gate-only blocks vanish, while its spectrum combines the remaining block’s eigenvalues with additional zeros.For the cross-entropy Hessian at the origin, the nonzero eigenvalues include ±∥w∥2, each with multiplicity d, alongside zero eigenvalues.Source: §C.2.1 · p. 37, Then the Hessian of the cross-entropy… · p. 39

Then the Hessian of the cross-entropy loss

• At the zero-parameter witness, the Hessian spectrum contains ±∥w∥2 with multiplicity d and zero with multiplicity p − 2d.The zero-eigenvalue multiplicity follows from p = nd + 2d + p′.Source: Then the Hessian of the cross-entropy… · p. 40
• Theorem C.3 shows that the gradient-descent Jacobian is nondegenerate almost everywhere: det Dϕ(θ) = 0 only on a Lebesgue measure-zero set.This holds for any finite vocabulary, bounded context, sample, and learning rate η ∈ (0, 1).Source: Then the Hessian of the cross-entropy… · p. 40
• The almost-everywhere result follows because the update map is real-analytic and its Jacobian determinant is nontrivial, being positive at the zero-parameter witness.A nontrivial real-analytic function has a Lebesgue measure-zero zero set.Source: Then the Hessian of the cross-entropy… · p. 40

C.2.2 GRADIENT DESCENT PRESERVES ABSOLUTE CONTINUITY

• With stepsize η ∈(0, 1), one gradient-descent step maps every absolutely continuous probability law on Rp to another absolutely continuous law.The proof excludes the measure-zero critical set and uses countably many diffeomorphic charts plus change of variables to show null-set preimages remain null.Source: §C.2.2 · pp. 41–42
• After any finite sequence of gradient-descent updates with stepsizes ηt ∈(0, 1), the composed update map also preserves absolute continuity.The result follows by iterating the one-step null-set-preimage property across all updates.Source: §C.2.2 · p. 42

D LEFT-INVERTIBILITY VIA SIPIT

• The inversion target is to recover the next token st from the layer-ℓ hidden state at position t given the true prefix π = s1:t−1.This formulation applies to causal decoder-only Transformers.Source: D LEFT-INVERTIBILITY VIA SIPIT · p. 43
• For a fixed prefix, the one-step map from candidate next tokens to layer-ℓ representations is almost-surely injective with a positive separation margin.The margin supports a verifier that accepts a candidate when the observed hidden state lies within a sufficiently small ball around its predicted representation.Source: D LEFT-INVERTIBILITY VIA SIPIT · p. 43
• SIPIT exactly and robustly recovers an entire sequence almost surely in worst-case time Θ(T|V|) by scanning the vocabulary once per position.Causality localizes each prediction to the observed hidden state and its true prefix, enabling sequential inversion.Source: D LEFT-INVERTIBILITY VIA SIPIT · p. 43
• Causal self-attention ensures that the hidden state at position t depends only on the prefix through token st, enabling sequential token recovery.The relevant dependence is on ⟨s1, . . . , st⟩ rather than later sequence positions.Source: D LEFT-INVERTIBILITY VIA SIPIT · p. 43
• SIPIT assumes parameters initialized from an absolutely continuous distribution and trained by gradient descent with step sizes in (0, 1), under which the network is almost-surely injective.The injectivity guarantee is attributed to Theorem C.1 under the stated training conditions.Source: D LEFT-INVERTIBILITY VIA SIPIT · p. 43

D.1 ONE-STEP LAST-TOKEN MAPS

• For each fixed prefix, the one-step map from candidate tokens to the current hidden state is almost surely injective and has a strictly positive pairwise margin.Causality makes the current representation depend only on the prefix and current token, while finite distinct outputs imply positive minimum separation.Source: §D.1 · pp. 43–44

D.2 THE CORE ROUTINES: LOCAL VERIFIERS, ACCEPTANCE REGIONS, AND POLICIES

• Local acceptance regions uniquely verify the true token at ε = 0 or whenever ε < ∆π,t/2, and any without-replacement policy encounters it within |V| proposals.Policies enumerate the finite vocabulary exactly once, while gradient-guided ranking improves empirical efficiency without changing worst-case guarantees.Source: §D.2 · pp. 44–46

D.3 GLOBAL INVERSION VIA SIPIT

• SIPIT recovers the exact sequence almost surely, accepting each true token within |V| candidates when ε < ∆πt,t/2, including noiseless and robust observations.The guarantee assumes decoder-only causal self-attention, almost-sure injectivity, and hidden-state perturbations bounded by ε at every position.Source: §D.3 · p. 47
• SIPIT performs at most T|V| verifier tests, but wall-clock time also depends on per-test costs and policy overhead such as proxy updates, scoring, and sorting.With fixed vocabulary and bounded costs, wall-clock time is O(T); larger vocabularies or sorting can add |V| or log |V| factors.Source: §D.3 · p. 48

E IMPLEMENTATION DETAILS AND ADDITIONAL EXPERIMENTS

• The section covers implementation details, collision and SIPIT ablations, controlled identical-prediction experiments, closest hidden-state inspection, and connections to anisotropy and intrinsic dimension.These topics are organized in §§E.1–E.5.Source: E IMPLEMENTATION DETAILS AND… · p. 48

E.1 IMPLEMENTATION DETAILS

• Theoretical collisions mean distinct prompts with identical last-token representations, with proofs requiring exact equality rather than numerical thresholds.This is the collision event controlled by the paper’s theorems and Section C.Source: §E.1 · p. 48
• Experiments replace exact equality with a floating-point numerical proxy because stored representations may not support a meaningful or robust exact-equality test.The proxy uses PyTorch tolerances of 10^-5 relative tolerance and 10^-8 absolute tolerance.Source: §E.1 · pp. 48–49
• No distinct prompt pair satisfied the practical collision criterion across billions to trillions of checks; torch.allclose always returned False.Observed ℓ2 distances remained bounded away from zero at numerical precision.Source: §E.1 · p. 49
• SIPIT follows Algorithm 1 with gradient clipping at norm 1 and projection to the nearest token embedding every K = 50 candidate proposals.Gradients are not taken through the projection.Source: §E.1 · p. 49
• These SIPIT heuristics affect efficiency only; the verifier and all correctness guarantees remain unchanged.Source: §E.1 · p. 49
• HARDPROMPTS [Wen et al. (2023)] is adapted from multimodal optimization by replacing CLIP similarity with SIPIT’s ℓ2 loss and setting T = 1 token·|V|.Other details, including step sizes and stopping rules, mirror SIPIT for fair comparison.Source: §E.1 · p. 49
• Figure 7 summarizes GPT-2 Small, Medium, and Large collision searches using log-scale minimum pairwise ℓ2-distance boxplots and a 10^-6 threshold.Red bars indicate medians, and distributions are shown separately for each layer.Source: Figure 7 · p. 49

E.2.1 COLLISION EXPERIMENTS

• 10−6: minimum pairwise distances stayed far above the collision threshold across GPT-2 and Gemma-3 layers, while separation generally increased with depth and Gemma model size.These results rule out observed collisions across architectures and depths.Source: §E.2.1 · p. 50, Figure 8 · p. 50
• Minimum, mean, and maximum pairwise distances for Gemma-1B rose quickly with short sequence lengths and then plateaued, with the minimum never approaching zero.Margins therefore stabilized rather than collapsed over the explored prompt lengths.Source: §E.2.1 · p. 50, Figure 9 · p. 50

E.2.2 SIPIT

• SIPIT’s inversion time grew linearly with vocabulary size, while fewer than 0.25% of tokens were explored for both Mistral-7B-v0.1 and Llama-3.1-8B.The comparison used approximately 32K- and 128K-token vocabularies with identical token sequences.Source: §E.2.2 · p. 51, Table 6 · p. 51
• Exact recovery was achieved for every 100-token sequence in Train, Test, and OOD datasets, with random-token OOD samples significantly faster to invert than natural-language samples.The authors attribute the speed difference to more dispersed hidden states and clearer gradient directions for random sequences.Source: §E.2.2 · p. 51, Table 7 · p. 51

E.3 IDENTICAL NEXT-TOKEN

• 10−6: separator-token embeddings remained well above the collision threshold across approximately 140K translation and math prompt pairs, despite identical answers.The prompts differed in delimiters such as ->, :, =, and - while preserving the same translations or arithmetic solutions.Source: §E.3 · p. 52, Table 8 · p. 52, Table 9 · p. 52

E.4 PROMPTS WITH SIMILAR REPRESENTATIONS

• 1.274: the closest Llama-3.1-8B pair differed only by trailing newlines, yet its final-layer last-token distance remained far from zero.The nearest pairs for both evaluated models were typically almost identical code snippets differing by whitespace or other minor formatting tokens.Source: §E.4 · p. 53
• 1.146: a closest Mistral-7B-v0.1 pair differing by one appended newline still showed non-negligible last-layer separation, consistent with nearest neighbors being near-duplicate snippets.Across inspected pairs, no qualitatively different prompts were comparably close.Source: §E.4 · pp. 53–54

Experimental setup

• The GPT-2 Small experiment sampled 100 prompts and 1000 single-token continuations per prompt, measuring anisotropy, intrinsic dimension, and minimum pairwise embedding distance at every layer.Each layer–prompt condition produced a 1000×d continuation matrix, with distances averaged across prompts to form the injectivity margin.Source: §E.5 · p. 54

Experiments

• Higher anisotropy tends to coincide with larger injectivity margins, while intrinsic dimension correlates negatively with injectivity margin (Pearson -0.60; Spearman -0.79).Lower intrinsic dimensionality tends to have larger margins, and anisotropy may reinforce numerically robust injectivity.Source: §E.5 · p. 55

Discussion

• Anisotropy and intrinsic dimension follow a reverse-U profile: intermediate layers become more anisotropic and lower-dimensional, then representations partially re-expand near the output.This provides a geometric picture in which structure is carved into aligned directions and low-dimensional manifolds across depth.Source: §E.5 · p. 55
• The observed growth in margins, decrease in intrinsic dimension, and late anisotropy spike support injective information funneling rather than layerwise information-destroying compression.Early layers expand and reorganize, intermediate layers carve low-dimensional directional manifolds, and upper layers sharpen the resulting structure.Source: §E.5 · p. 55

Empirical prevalence

• 18 out of 18 surveyed models use real-analytic feed-forward activations, supporting the relevance of the paper’s theoretical analysis to modern language models.The survey includes SiLU/Swish, SwiGLU, GeGLU, and GELU, which are real-analytic.Source: F REAL-ANALYTIC ACTIVATION FUNCTIONS… · p. 56

Scalar and vector activations

• SiLU and GELU are real-analytic scalar functions, and coordinatewise lifting makes their vector-valued versions real-analytic.The proofs derive SiLU analyticity from sigmoid and GELU analyticity from the error function, compositions, sums, scalings, and products.Source: F REAL-ANALYTIC ACTIVATION FUNCTIONS… · pp. 56–57

Gated feed-forward blocks

• GLU-style blocks built from affine maps, SiLU or GELU, and Hadamard products are real-analytic, covering the SwiGLU and GeGLU constructions.Analyticity follows from closure under affine composition and componentwise products.Source: F REAL-ANALYTIC ACTIVATION FUNCTIONS… · pp. 57–58

Relation to expressivity theory

• Unlike classical expressivity results, the paper studies a fixed decoder-only architecture under standard training and shows that non-injective parameter settings have measure zero and are almost surely avoided.The result concerns finite prompt sets, last-token representations, real-analytic activations, cross-entropy loss, and GD-type training from standard initialization; it does not deny that non-injective Transformers exist.Source: F REAL-ANALYTIC ACTIVATION FUNCTIONS… · p. 58