Forget BIT, It is All about TOKEN: Towards Semantic Information Theory for LLMs

1. Introduction

• The proposed shift from bit to token treats the token as a semantically charged unit and aims to extend Shannon’s framework beyond technical transmission toward semantic and pragmatic questions.The introduction frames this as a path from Weaver and Shannon’s Level A to Levels B and C.Source: §1 · p. 4
• The introduction motivates probabilistic information measures by contrasting their empirical computability with the uncomputability of Kolmogorov complexity.It notes that entropy, mutual information, and KL divergence admit tractable estimators at scale.Source: §1 · p. 5
• A unified mathematical account of Transformer mechanisms remains elusive, leaving architectural progress largely driven by heuristic, large-scale GPU experimentation.The introduction also describes the computational costs of this trajectory as rapidly becoming unsustainable.Source: §1 · p. 6
• The treatise presents Semantic Information Theory as an interdisciplinary framework for explaining LLMs by re-centering analysis around the token.It draws on statistical physics, signal processing, and information theory.Source: §1 · p. 6
• The statistical-physics perspective formalizes Transformer attention and feed-forward networks as continuous-state Hopfield networks and analyzes generalization, memory capacity, and inferential limits.The introduction names local Rademacher complexity and generalized Gardner capacity as tools in this analysis.Source: §1 · p. 6
• The signal-processing perspective studies token-level semantic vectors, their compression and distortion, and an information-theoretically optimal vectorization scheme for next-token prediction.It also relates this scheme to contrastive predictive coding and formulates the Transformer as a specialized non-linear time-varying VAR process.Source: §1 · pp. 5–6
• The information-theoretic perspective models the LLM as a stateful channel with feedback and introduces directed rate-distortion and rate-reward measures for pre-training and post-training.It also describes inference-phase semantic information flow through sub-martingales.Source: §1 · p. 7

2.1. Neural Networks and Statistical Physics

• Continuous-state Hopfield networks extend binary memory models toward semantic NLP, and their generalizations connect Transformer Attention and energy-based sequence processing to energy minimization.The section presents the shift from BIT to TOKEN as enabling statistical physics to analyze Transformer mechanisms.Source: §2.1 · pp. 8–9

2.2. EBM Formulation of the Attention Module

• Attention weights are Boltzmann probabilities over prefix positions, assigning lower energy to historically relevant context; softmax also gives a maximum-entropy distribution under mean constraints.The section interprets this energy-minimization objective as learning parameters that map asymmetric semantic relevance to low thermodynamic energy.Source: §2.2 · pp. 10–11
• Statistical-physics analyses explain deep pure-Attention representation collapse, a positional-to-semantic phase transition, and task-diversity limits in linear Attention.The cited work attributes collapse avoidance to residual connections, layer normalization, and calibrated initialization; spin-glass analysis links increasing task diversity to a phase transition in linear Attention.Source: §2.2 · p. 12

2.3. EBM of the Transformer Architecture

• The global Transformer EBM maps candidate-token energies to vocabulary logits, with low-temperature inference selecting the minimum-energy token and training minimizing expected energy.The passage identifies sampling heuristics such as temperature scaling and top-k as typical inference procedures.Source: §2.3 · pp. 13–14

2.3.1. Generalization Error Bound

• O(1/K) is the generalization bound’s fast convergence rate for the complexity term, under bounded energies and a sufficiently large vocabulary relative to temperature.The theorem bounds cross-entropy uniformly across inference steps; its stated assumptions include log M ≥ 3B/T.Source: §2.3.1 · pp. 16, 18
• The effective loss range G=log M+2B/T and energy-class pseudo-dimension d govern the generalization bound’s leading behavior, with an O(1/K) rate despite a log(N−n) uniform-step cost.The passage says preserving energy scale B preserves the guarantee, while regularizing the parameter class can reduce d below nominal parameter count.Source: §2.3.1 · pp. 18–19

2.3.2. Memory Capacity

• With logarithmic normalization, generalized Gardner capacity coincides with Shannon capacity, and continuous-state Hopfield storage scales exponentially with network size,,.Classical Gardner capacity instead has linear pattern-count scaling; the passage notes that rigorous work re-derived the capacity results after the replica method’s historical lack of mathematical rigor.Source: §2.3.2 · p. 19
• Softmax attention can store P∼O(exp(ηd_rep)) token-embedding patterns, while FFNs encode key-value associations that can be edited through targeted parameter updates [65],,, [82].The passage presents attention’s geometry as enabling vast continuous token-level storage, while noting that compact models near memory saturation may be volatile under continual learning.Source: §2.3.2 · p. 20
• Scaling model dimension can produce a thermodynamic phase transition in which encoded knowledge crosses a critical threshold and inferential capability jumps sharply.The passage presents statistical mechanics as a framework for interpreting parameter-count scaling laws and emergent abilities.Source: §2.3.2 · p. 20

2.4. Inferential Capacity and Granger Causality

• LLM inference satisfies Granger causality at each generation step when the prompt informs the next-token distribution, placing reasoning at the associational level of Pearl’s hierarchy.The account concludes that autoregressive architectures alone cannot abstract symbolic concepts from raw data or achieve genuine causal logical reasoning.Source: §2.4 · p. 21

3. Signal Processing Perspective

• Vectorizing discrete tokens makes semantic affinities computable in continuous space and supports gradient-based optimization, avoiding the combinatorial burden of discrete inference.The paper frames vectorized tokens as inputs, output distributions, and distributed parametric memory.Source: §3 · p. 22

3.1.1. Token-Level Semantic Space

• Exact inference over token sequences has complexity growing with the vocabulary size raised to sequence length, making it unscalable at modern NLP corpus scale.Classical graphical-model methods also face prohibitive pre-training costs and difficulty representing directional dependencies on loopy language graphs.Source: §3 · p. 22, §3.1.1 · p. 23

3.1.2. Token-Level Semantic Vector Space

• The token-level semantic vector space maps tokens to the unit sphere in vocabulary-sized Euclidean space, with a probability measure induced by the embedding map.The section uses the sphere as a representational accuracy and computational efficiency compromise.Source: §3.1.1 · p. 23, §3.1.2 · pp. 24–25

3.1.3. Semantic Compression

• Semantic compression is defined as reducing dimensionality while approximately preserving pairwise inner products, equivalently cosine similarities.JL-style guarantees bound squared distortion by ε^2 under stated projection conditions.Source: §3.1.3 · pp. 26, 28
• The JL-based distortion guarantee is measure-agnostic, so it does not characterize distribution-sensitive limits, especially under aggressive compression below JL-type thresholds.The section identifies a measure-aware dimension-distortion theory as a needed information-theoretic counterpart.Source: §3.1.3 · p. 28

3.1.4. Semantic Vectorization for Next-Token Prediction

• For next-token prediction, an optimal contextual embedding maximizes conditional mutual information with the future given prior embeddings, adding incremental predictive information at each step.This conditioning prevents the embedding from collapsing into a memoryless encoding of the current token.Source: §3.1.4 · p. 30
• CPC’s InfoNCE pipeline relaxes the conditional objective through a pairwise proxy, making its resulting representations information-theoretically suboptimal.The marginal proxy is equivalent under a sufficient-statistic assumption; without it, it is a surrogate rather than a rigorous bound. The paper recommends first aligning embedding objectives with downstream tasks.Source: §3.1.4 · pp. 30–31

3.2. Transformer as a Non-Linear Time-Varying VAR Process

• Attention is a generalized non-linear, time-varying VAR operator because its data-dependent coefficients vary with generation step and prefix.Its deterministic aggregate combines historical states; sampling introduces stochasticity at inference.Source: §3.2 · pp. 31–32
• At low temperature, decoding approaches greedy maximum-likelihood selection from the vocabulary.Practical decoding may use top-k, nucleus, or temperature-based sampling heuristics.Source: §3.2 · pp. 32–33
• The Transformer specializes the generalized recurrence with a shared value map gated by time-varying Attention weights and an FFN non-linearity.The FFN transforms the Attention aggregate into the continuous representation used for the next token.Source: §3.2 · pp. 32–33
• The section identifies LLM-generated token trajectories with the Granger-causal paradigm and places their inferential ceiling at Granger-causal inference in Pearl’s hierarchy.The account treats next-token prediction as the task through which this form of inference arises.Source: §3.2 · p. 34

3.3. ELBO of the Transformer

• Interpreting Attention weights as a variational posterior over latent prefix positions yields an ELBO analysis of the Transformer training objective.The latent variable indexes positions in the prefix, and the posterior is identified with the Attention weights.Source: §3.3 · pp. 34–35
• Maximizing the ELBO maximizes conditional log-likelihood exactly when the Attention-weight posterior equals the true posterior over latent positions.The KL divergence between the variational and true posteriors quantifies the gap.Source: §3.3 · pp. 35–36
• The inference-time Attention distribution is the variational posterior that maximizes the ELBO, making softmax normalization an optimal posterior approximation under the Jensen bound.The residual KL divergence measures posterior mismatch in non-realizable regimes.Source: §3.3 · p. 36
• The framework proposes KL divergence from the true latent-position posterior as a scalar criterion for comparing Attention variants beyond empirical benchmarks.It also connects ELBO maximization to lowering energy at the human-aligned token and sharpening the variational posterior.Source: §3.3 · p. 36

3.4.1. Mamba Series

• Mamba and Mamba2 replace self-Attention with a discrete state-space model that unrolls to a linear time-varying VAR recurrence.The section describes pure SSMs as less expressive on directional dependencies because they lack the bilinear semantic-relevance gate.Source: §3.4.1 · pp. 36–37
• At 1.5B parameters, Mamba3 outperforms the closest sub-quadratic competitor by roughly 1.8 points across downstream benchmarks and matches Mamba2 generation quality with half the state dimension.Its limitations include degraded retrieval on very long, unstructured contexts, increased pre-training cost from MIMO, and design trade-offs from hybrid Attention.Source: §3.4.1 · p. 37

3.4.2. LLaDA

• LLaDA trains by masking a fraction of tokens and predicting their original identities from unmasked context, using a non-causal Transformer in the VAR framework.Its masked-position objective weights the loss by 1/τ to give an unbiased estimator of the discrete-diffusion ELBO.Source: §3.4.2 · p. 37

3.5. Information Theory and Signal Processing

• The BIT is presented as the shared atomic unit bridging computation and communication, while the paper proposes lifting the atomic unit from BIT to TOKEN for Semantic Information Theory.The section frames this shift as a response to classical information theory’s abstraction from semantic meaning and pragmatic effectiveness.Source: Figure 2 · p. 38, §3.5 · p. 39

4.1. Information Theory and AI

• Information-theoretic methods have shaped AI and NLP, but rigorous information-theoretic axiomatization of autoregressive LLM and Transformer generation remains absent.The section situates this gap amid earlier information-theoretic work on AI, statistical NLP, and deep learning.Source: §4.1 · p. 40

4.2. Classical Shannon Information Theory

• Shannon’s canonical schematic depicts a general communication system, while the classical theory models its communication channel probabilistically.The section identifies Figure 3 as Shannon’s 1948 schematic.Source: §4.2 · p. 41, Figure 3 · p. 41, §4.2.1 · p. 41

4.2.1. Three Fundamental Theorems of Information Theory

• Shannon’s theorems bound lossless source coding by entropy, reliable channel rates by capacity, and make separate coding asymptotically optimal when H(S) < C.The noisy-channel theorem makes capacity the critical rate: error tends to zero below it, but not above it.Source: §4.2.1 · pp. 41–42

4.2.2. Methodological Implications

• Shannon derived necessary conditions for reliable communication from runtime behavior without relying on particular hardware, yielding a framework applicable across implementations.His method begins by asking what a perfectly reliable system would have to do.Source: §4.2.2 · pp. 42–43

1. What is reliability in digital communication, mathematically?

• Reliability means asymptotically error-free transmission, analyzed with tools including bounded error probabilities, error exponents, and decoding methods.The passage names maximum-likelihood and joint typicality decoding among the standard tools.Source: §1 · p. 43

2. What is the universal mathematical model of a communication channel?

• A conditional transition law models channel uncertainty probabilistically, independently of whether specific hardware exists.The abstraction is described as implementation-agnostic and universal.Source: §2 · p. 43

3. What are the performance metrics for a communication system?

• Channel capacity bounds achievable rates, and minimizing cross-entropy is equivalent to minimizing KL divergence when the target entropy is parameter-independent.Mutual information is the capacity objective over input distributions.Source: §3 · p. 43

4.3.1. Rate-Distortion Function

• For stationary ergodic sources, rate-distortion minimizes the mutual information rate needed for reconstruction subject to an expected-distortion bound.The objective measures the source information retained in the reconstruction, rather than the reconstruction’s lossless description length; closed-form evaluation is rarely available and numerical computation can be expensive.Source: §4.3.1 · pp. 44–45

4.3.2. Directed Information

• Directed information measures causal, asymmetric dependence across input and output sequences, extending mutual information to channels with memory and feedback.Unlike symmetric mutual information, its causal ordering can represent feedback dependence.Source: §4.3.2 · pp. 45–47
• Massey causality requires that the current channel output depend on the source only through current input and past outputs, not the source’s future.Under this condition, directed information has the causal interpretation described in the section.Source: §4.3.2 · p. 46
• Backward directed information’s stepwise objective is exactly the one maximized by the information-theoretically optimal semantic encoder.At each step, the encoder maximizes conditional mutual information between the future sequence and current embedding.Source: §4.3.2 · p. 47

4.3.3. Directed Information Density

• Directed information density extends information density to causally ordered sequences, and its expectation recovers directed information.It is a sequence-level random variable formed from time-indexed contributions.Source: §4.3.3 · p. 48

4.4. Information-Theoretic Abstraction of LLMs

• The LLM abstraction uses runtime dynamics to remain applicable beyond a particular architecture and poses three questions for LLMs analogous to Shannon’s communication questions.The passage connects this design-time/runtime separation to Sutton’s proposal.Source: §4.4 · pp. 48–49

4.4.1. LLM as a Channel with Feedback and State

• The LLM is modeled as an architecture-agnostic causal communication channel with internal memory and autoregressive feedback, supporting an information-theoretic formalization centered on the TOKEN rather than the BIT.The prompt is encoded as semantic vectors, prior internal states condition the next state, and a decoder emits discrete tokens.Source: §4.4.1 · pp. 49–50

4.4.2. The Theoretical Principles of LLM Training via Semantic Information Theory

• The directed rate-distortion function defines the minimum causal information rate needed to keep step-wise model distributions within KL distortion D of the human reference.The rate term can regularize representations by suppressing prompt correlations unnecessary for matching the reference, a proposed route to mitigating hallucinations.Source: §4.4.2 · pp. 50–51
• At zero distortion, the pre-training rate equals the human reference’s per-position directed information, assuming global convergence under cross-entropy.At ideal convergence, the model’s directed information flow approaches the upper bound set by the human reference.Source: §4.4.2 · pp. 51–52
• Post-training trades directed-information rate against reward, and after alignment the model’s next-token predictions approximate human generation’s Granger-causal structure subject to reward-model preferences.The reward model scores the generated sequence given its prompt.Source: §4.4.2 · pp. 52–53

4.4.3. Semantic Information Theory Principles for Inference

• During inference, cumulative semantic information flow is a sub-martingale; under bounded martingale increments, the paper gives a concentration bound.The increment bound is automatic in the paper’s setting because finite-temperature token probabilities are bounded away from zero and one, making the log-likelihood ratio bounded.Source: §4.4.3 · pp. 54–55
• With an almost-surely bounded stopping time, cumulative directed information through halting lower-bounds the prompt-to-first-token mutual information.Under alignment constraints, optimizing the prompt distribution can enlarge directed semantic information, motivating harness or prompt engineering as a way to improve response quality.Source: §4.4.3 · pp. 55–56

4.4.4. Estimation of Directed Information

• Finite-lag transfer entropy estimates directed-information summands using only strictly past tokens, and a Transformer’s bounded context window matches this truncated regime.Increasing context length moves the estimate closer to full-prefix directed information; TREET is described as its realizable bounded-window counterpart.Source: §4.4.4 · pp. 56–57

4.5. The Causal Ceiling of LLMs: Granger Prediction without Pearl Intervention

• The section asks whether Granger-style predictive dependence constitutes genuine causation, contrasting data-driven temporal prediction with Pearl’s intervention- and counterfactual-based structural reasoning.It frames this distinction as a causal ceiling for LLMs’ inferential capabilities.Source: §4.5 · pp. 57–58
• Pearl’s theorem places Granger causality at Level 1; passive-corpus LLMs cannot structurally execute Level-2 interventions or Level-3 counterfactuals without exogenous assumptions [41],.The passage identifies this as the ceiling of purely data-driven prediction.Source: §4.5 · p. 58
• RL combined with MCTS can improve apparent reasoning, but fixed autoregressive models only imitate intervention and counterfactual patterns in human corpora, not structural causal inference,, [148].The passage attributes these outputs to learned linguistic distributions of human causal narratives.Source: §4.5 · p. 58
• Human-AI collaboration can supply exogenous causal assumptions unavailable from raw text, harnessing LLM predictive bandwidth for human intellectual productivity.The passage describes these assumptions as unidentifiable from the textual manifold alone.Source: §4.5 · pp. 58–59

• The treatise conjectures that TOKEN will bridge empiricism and rationality and concludes it will define generative architectures of the AI Epoch.Source: §5 · p. 60
• The title does not deprecate BIT, which the treatise calls the paramount intellectual achievement of the Information Epoch.Source: §5 · p. 60
• Whether LLMs will reach AGI or ASI remains open, and whether next-token prediction qualifies as thinking is debatable.Source: §5 · p. 60
• Autoregressive architectures have amplified humanity’s capacity to synthesize, integrate, and process semantic information at scale.Source: §5 · p. 60

• Figure 1 schematizes the Transformer architecture and highlights the decoder-only path in a red box.Source: Index of Figures · p. 69
• Figure 2 presents BIT as the shared atomic unit bridging Turing’s computation theory and Shannon’s communication theory, followed by Shannon’s canonical communication-system schematic in Figure 3.Source: Index of Figures · p. 69
• Figure 4 abstracts an LLM as a causal communication channel with state and an autoregressive feedback loop.Source: Index of Figures · p. 69