PaperlayerView exact revision on arXiv

Source-linked AI summary

State over Tokens: Characterizing the Role of Reasoning Tokens

Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, Yoav Goldberg

arXiv:2512.12777v1cs.CLcs.AI

TL;DR

Readable reasoning text can seem to explain model computation, but the paper notes that it may be incomplete or semantically mismatched. It proposes State over Tokens, treating reasoning tokens as persistent computational state across cycles, and concludes that this perspective reframes their role while opening further research questions.

  • Problem

    Readable reasoning text can invite the inference that it faithfully explains computation, despite evidence of incompleteness and mismatch with the model’s actual process.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The paper frames reasoning tokens as State over Tokens: the persistent information carrier that passes state between otherwise stateless computation cycles.

    Source: §3.1 · p. 4, §7 · p. 9

  • Results

    The SoT view characterizes reasoning tokens as computational state rather than a narrative of thought, with human-readable meaning potentially incidental to their computational function.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The framework opens research on how models construct and use token-based state, whether language has computational advantages, and whether SoT can faithfully explain computation.

    Source: §7 · p. 9

    The proposed account that pretraining induces a semantic inductive bias is a hypothesis, and its stated implications are conditional on that hypothesis being correct.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.

1 Introduction

The paper argues that readable reasoning text is not a faithful account of model computation, despite reasoning tokens helping produce answers. It proposes State over Tokens (SoT), treating those tokens as persistent computational state and using that perspective to motivate new research questions.

1 Introduction

  • Readable, useful reasoning text can invite the inference that it faithfully explains model reasoning, but empirical findings contradict that inference.The paper distinguishes the pre-answer token sequence from its interpretation as English reasoning text.Source: §1 · p. 1
  • SoT treats reasoning tokens as externalized computational state and the sole information carrier between otherwise stateless generation cycles.The framework focuses on how tokens function across cycles rather than on their English semantics.Source: §1 · pp. 1–2
  • The whiteboard analogy models tokens as intermediate results or encodings that persist across limited computation cycles, although an observer may misread them.The analogy maps the words to reasoning tokens, the person to the model, and each interval to its limited capacity per cycle.Source: §1 · p. 2
  • The framework motivates research on interpretability across multiple cycles and whether language can function both as computational state and faithful explanation.The paper also proposes investigating how models construct and use token-based state.Source: §1 · p. 2
  • SoT challenges Chain-of-Thought and Scratchpad metaphors: tokens pass state between computations, rather than necessarily expressing the computations themselves.Unlike a human scratchpad that supplements internal memory, tokens are the model’s sole state carrier between cycles.Source: §1 · p. 2

2 The Subject of Inquiry: Reasoning Tokens

Reasoning text can appear plausible without faithfully describing model computation, even as reasoning tokens support final-answer production. The paper uses SoT to characterize their role positively as state carried between computations.

2 The Subject of Inquiry: Reasoning Tokens

  • The paper distinguishes reasoning tokens—the pre-answer sequence—from reasoning text, the same tokens interpreted through English semantics.The discussion includes tokens elicited by examples or instructions and tokens produced through training.Source: §2 · p. 2
  • Reasoning tokens can appear to offer a readable account that builds trust, while functionally serving final-answer production rather than human interpretation.Unlike answer tokens, they need not satisfy the constraint of reflecting the solution for human or external-system interpretation.Source: §2 · p. 3

2.1 Empirical evidence: plausibility without faithfulness

  • Reasoning text can read like a valid derivation without corresponding to the model’s actual computation, so it is not reliably a faithful explanation.The passage distinguishes plausibility to human readers from fidelity to the causal process driving generation.Source: §2.1 · p. 3
  • Evidence describes both omitted factors affecting answers and semantic mismatch between generated text and its computational role.Studies cited in the passage report omissions, disregard of textual details, and accurate answers despite irrelevant reasoning text.Source: §2.1 · p. 3

2.2 What can be said on reasoning tokens

  • SoT is proposed as a metaphor that accounts for tokens’ contribution to answers, their sometimes plausible steps, and their lack of explanatory faithfulness.The paper seeks a positive characterization beyond saying only what reasoning tokens are not.Source: §2.2 · pp. 3–4

3 Conceptual Framework: State over Tokens

The State over Tokens (SoT) framework treats reasoning tokens as computational state that carries information between limited, stateless generation cycles, rather than as a narrative of the model’s reasoning. This view recasts the tokens’ role and identifies properties of that state that shape how it should be understood.

3 Conceptual Framework: State over Tokens

  • SoT frames reasoning tokens as an accumulated computational substrate that encodes state and enables a process to continue across multiple cycles.The framework names this functional view State over Tokens.Source: §3 · p. 4, §3.1 · p. 5

3.1 The formal framework

  • Each cycle applies M to the current token sequence, appends its output, and uses the resulting sequence as the next input.The user input is the initial sequence, and ⊕ denotes concatenation.Source: §3.1 · p. 4
  • The token sequence is the sole persistent information carrier between cycles, allowing accumulated results to use computation across cycles beyond one cycle’s capacity.The paper notes theoretical findings demonstrating this capacity increase for specific constructions (Merrill & Sabharwal, 2024; Li et al., 2024).Source: §3.1 · pp. 4–5

3.1.1 Implications of the State over Tokens view

  • As state, each token prefix is a discrete, forward-looking and necessarily partial input for future computation, interpreted according to the model’s own use rather than its human-readable meaning.Changing the state changes the trajectory of future computations.Source: §3.1.1 · p. 5

4 Two Misconceptions Behind the Illusion of Explanation

The paper distinguishes the intermediate state that helps a computation continue from the computation itself, and argues that its human-readable meaning may not match how the model uses it. Catalan-number examples illustrate both the incompleteness of state as an explanation and the possibility of model-specific encodings.

4.1 The misconception of completeness

  • Intermediate Catalan values allow the sequence to continue but do not reveal the algorithm or computation that produced them.A process could also recompute missing values and output only selected intermediates, so even treating the sequence as an explanation does not guarantee completeness.Source: §4.1 · p. 6, §4 · p. 6

4 Two Misconceptions Behind the Illusion of Explanation

  • The SoT sequence is scaffolding that propels the process forward, not the computation itself or a complete account of it.The Catalan example distinguishes intermediate results from the algorithm that produces them.Source: §4.1 · p. 6

4.2 The misconception of shared meaning

  • A simple additive shift of 10 can make the final state 11, 11, 12, 15, 24, 52 while preserving the computation’s effectiveness, illustrating that human-readable symbols may serve a different model-specific function.The paper extends this possibility beyond numbers: phrases such as “I need to reconsider this approach” may encode something else; real-world encodings may be more intricate (Cloud et al., 2025).Source: §4.2 · pp. 6–7

5 The Ontological Divergence: Text vs. State

The paper argues that reasoning tokens’ human-readable form and their computational function belong to distinct ontological categories, even though they share the same token sequence.

5 The Ontological Divergence: Text vs. State

  • Reasoning tokens are not merely symbols with multiple interpretations: humans read them as text, while the model uses them as computational state for an iterative function.The sequence acts as a communicative medium and a functional state in the same embodiment.Source: §5 · p. 7

6 Questions Raised by State over Tokens

SoT reframes interpretability around how models construct and use token-based computational state, raising questions about its encoding and propagation and whether language is a special medium. It also highlights a tension between state useful for computation and text that faithfully explains it.

6.1 What SoT means for interpretability

  • SoT poses a distinct interpretability challenge: determine how models construct and use state through tokens, including what they externalize, encode, and propagate across cycles.Existing work offers a starting point, but understanding how LLMs manage state through tokens remains largely open (Bogdan et al., 2025; Dutta et al., 2024; Chen et al., 2025a).Source: §6.1 · pp. 7–8

6.2 Is language special for SoT?

6.3 Can SoT ever be a faithful explanation?

  • Using one token sequence both as computational state and as a faithful explanation creates a bottleneck: opaque or redundant state may aid computation, while coherent English constraints may strip needed information.The paper also asks whether reasoning text can concurrently carry state and provide an effective rationalization.Source: §6.3 · pp. 8–9

7 Conclusion

Reasoning tokens are best understood as externalized computational state, not a narrative of thought. This view explains two misconceptions about their explanatory value and opens research directions on how models construct and use token-based state.

7 Conclusion

  • SoT frames reasoning tokens as externalized computational state rather than a narrative of thought.The framework names this role State over Tokens (SoT).Source: §7 · p. 9
  • SoT explains the completeness misconception by distinguishing functionally necessary state from a full account of computation, and the shared-meaning misconception by allowing model-specific encodings.Human-readable meaning may be incidental to computational function, creating two incompatible interpretations of one textual artifact.Source: §7 · p. 9
  • The framework opens research on how LLMs construct and use token-based state, whether natural language offers computational advantages, and whether SoT can faithfully explain underlying computation.Recognizing tokens as computational state also provides a basis for calibrating trust.Source: §7 · p. 9

Related papers

Loading 2512.12777v1…