Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens

1 Introduction

• Because frontier models’ training data and procedures are inaccessible and their natural-language traces are ambiguous, the authors use a controlled setting to investigate trace semantics.Prior interpretations include both semantic accounts of traces and cautions against treating them as faithful indicators of internal computation.Source: §1 · p. 2
• The authors train controlled, open models from scratch on a formally verifiable shortest-path planning task to study the end-user semantics of intermediate tokens.The models are 0.5B-parameter Qwen models trained on grid-maze tasks with A* traces.Source: §1 · pp. 2–3
• The study tests whether solution correctness tracks trace validity, whether trace semantics affect performance, whether GRPO improves trace validity, and whether trace length tracks computational complexity.The training traces are deliberately swapped between problems to remove their connection to the accompanying task.Source: §1 · p. 3
• Models often produce invalid traces despite correct solutions; corrupted-trace models perform comparably to or better than correct-trace models, especially out of distribution.Reinforcement-learning post-training improves solution accuracy but does not consistently improve trace validity, and trace length is largely agnostic to problem difficulty.Source: §1 · p. 3
• The study challenges explanations that attribute trace effectiveness to semantic content, cautioning that human-like or algorithmically interpretable traces may be unnecessary and potentially misleading.The authors say their claim is not that intermediate tokens can never have interpretable meaning, but that they need not have it.Source: §1 · p. 3

2 Related Work

• Trace-augmented training and reinforcement learning improve final-answer accuracy, but prior work did not evaluate trace accuracy or explicitly test incorrect or irrelevant training traces.That leaves open whether accuracy gains come from semantic information in the traces.Source: §2 · p. 3
• The authors extend the question of whether incorrect traces can lead to correct answers by systematically evaluating performance when explicitly incorrect traces are rewarded.They contrast this with R1-Zero, which rewards solution correctness without attending to trace content.Source: §2 · p. 4
• Natural-language trace evaluations are difficult to ground because math problems lack established ground-truth semantics for correct reasoning, while manual judgments and unverified language-model checks can be noisy.Prior faithfulness studies primarily examined large pretrained models.Source: §2 · p. 4
• Neural algorithmic reasoning has examined faithful representations of classical algorithms, but existing work has focused on graph neural networks rather than autoregressive transformers.The cited transformer studies instead trained models to mimic formal procedures such as A*, BFS, MCTS, and DPLL.Source: §2 · p. 4
• Earlier transformer studies trained models to mimic search and other formal procedures, but did not rigorously evaluate generated-trace correctness or its relationship to solutions.This paper presents its work as the first rigorous evaluation of trace correctness in autoregressive transformers.Source: §2 · p. 4

3 Background

• A* traces provide a formal basis for checking whether model-generated traces follow the semantics of the training data and relate to final solutions.The paper argues that this check is possible in small, well-scoped domains with clear ground truth.Source: §3 · p. 4

3.1 The Maze Pathfinding Domain

• The task is to find a legal path between start and goal cells in a 30 × 30 grid, moving orthogonally through free cells; a plan is correct if its actions are legal and it reaches the goal.Maze instances are generated using different algorithms to evaluate performance on maze types unseen during training.Source: §3.1 · pp. 4–5

Acyclic Maze Generation

• Acyclic maze generation includes Wilson’s uniform spanning-tree mazes, Kruskal’s connected loop-free mazes, and DFS mazes with long corridors and low branching.These algorithms differ in their maze distributions and structural biases.Source: Acyclic Maze Generation · p. 5

Cave Generation

• Cave-style settings include Drunkard’s Walk mazes with cycles and open chambers, and Searchformer-style mazes that may contain loops and are out of distribution relative to acyclic training sets.The experiments train separate models on Wilson’s and Searchformer-style datasets.Source: Cave Generation · pp. 5–6

3.2 The A∗Search Algorithm

• A* maintains open and closed lists, expands the open-list node with the lowest f(n) = g(n) + h(n), and returns a reconstructed path when it reaches the goal.The paper uses the Manhattan heuristic, which is consistent for orthogonal unit-cost grid movement and therefore ensures an optimal path.Source: §3.2 · p. 6
• The paper linearizes A* execution by printing create actions when nodes enter the open list and close actions when nodes enter the closed list; its validator simulates those operations.The cG and cH fields encode the exact start cost and heuristic estimate, respectively.Source: §3.2 · p. 6

4 Validating Traces and Solutions

• A formal validator is needed because prior trace-trained-model evaluations assessed final accuracy without assessing the traces themselves.For large RL-post-trained models, trace evaluation is especially difficult because outputs can be lengthy and natural-language interpretations ambiguous.Source: §4 · pp. 6–7
• Figure 2 illustrates sequential validation: assuming parsing succeeds, an invalid action anywhere in the model’s output causes the trace to be flagged invalid.The example uses a maze with start state (1,3) and goal state (2,5).Source: Figure 2 · p. 7
• The A* validator parses each generated trace action and simulates its operations on open and closed lists, marking it valid only if execution proceeds through closing the goal node.This formal procedure checks whether the model’s trace follows the semantics of the algorithm used to generate training traces.Source: §4 · p. 7
• The validator identifies parsing errors, attempts to explore wall cells, and illegal neighbor creation as trace failures.An invalid neighbor is a wall or a cell not adjacent to the last closed node.Source: §4 · p. 7
• Other failures include creating an already closed node, closing a node absent from the open list, a trace-plan mismatch, or failing to close the goal.These errors prevent valid execution or agreement between the trace and its resulting plan.Source: §4 · p. 7

5 Evaluating Solution Accuracy vs. Trace Validity

• The evaluation tests Wilson- and SF-style-trained models across maze-generation distributions, measuring plan validity and trace validity within valid plans, but not optimality.The models are trained on 500k datapoints; evaluation includes Wilson, Kruskal, DFS, Drunkard, and SF-style test sets.Source: Table 1 · p. 8, §5 · p. 8
• The study uses global, formally verifiable trace validity to test whether intermediate-token correctness is tightly coupled to solution correctness.This stricter measure contrasts with prior partial and qualitative natural-language trace evaluations.Source: §5 · p. 8
• Trace validity within valid plans does not consistently reach 100%, so it is not a reliable predictor of plan accuracy, especially for the SF-style model.The table pairs plan validity with trace validity within valid plans across maze distributions.Source: §5 · p. 8, Table 1 · p. 8
• Problem difficulty is defined by the number of A* operations required; training uses 1,000–3,500-token traces, with evaluation spanning easier, held-out, and harder length ranges.The harder evaluation range is split into two subsets, 3,500–4,000 and 4,000–4,500 tokens.Source: §5 · p. 8
• On harder length-generalization problems, trace validity among correct solutions decreases, while the model appears to learn algorithm style without its underlying mechanisms.Longer traces are associated with greater difficulty, and the reported decline suggests the correlation is not fundamental but may reflect the training distribution.Source: §5 · pp. 8–9, Table 2 · p. 9

6 Training with Traces: Does Meaning Matter?

• Swapped traces are randomly reassigned across problems; models otherwise share an architecture, train on 500K samples, and face 1,000-problem tests across maze algorithms.Source: §6 · p. 9
• 11.7% plan accuracy on Drunkards for Wilson-swapped models contrasts with no solved problems for normal models; SF-style swapped models reach 95.4% versus 62.1% for normal models.Source: §6 · p. 9
• Swapped models perform as well as or better than correct-trace models and substantially better than the solution-only baseline, despite 0% trace validity.Table 3 reports results across maze distributions; solution-only reports plan validity rather than trace validity.Source: §6 · p. 9, Table 3 · p. 10
• Plan accuracy follows a U-shaped trend as trace swapping rises, while trace validity steadily falls toward fully swapped training.Source: Figure 3 · p. 10, §6 · p. 10

6.1 Does post-training increase the semantic correctness of the intermediate tokens?

• Across GRPO checkpoints, accuracy improves without consistent trace-validity gains; swapped models still outperform correct-trace counterparts across distributions at 0% validity.Post-training used separate 10k-sample datasets; fine-tuned Qwen3 8B results were largely consistent, with unknown pretraining data and methods noted as confounds.Source: §6 · p. 10, §6.1 · p. 11, Table 4 · p. 11

• The study measures pathfinding difficulty by A* operations to generate an optimal plan, then compares model-generated trace lengths with that measure.Source: §7 · p. 12
• Pearson correlations with A* trace length are r = 0.0081 (p = 0.566) for Wilson and r = 0.197 (p < 0.001) for SF-style models.Generated lengths are highly dispersed; many traces reach the 32k context limit without producing a valid solution.Source: Figure 4 · p. 12, §7 · p. 12
• On 100 trivially simple no-maze instances, the Wilson-trained model solves only 5 and often generates traces exceeding the 32K context limit.The authors report that trace length is more correlated with an instance being out of distribution than with its computational complexity.Source: §7 · pp. 12–13
• Wilson-trained trace lengths align on Wilson mazes (r = 0.40; ρ = 0.98) but not on SearchFormer-style OOD mazes (r = 0.02; ρ = 0.02).The authors suggest the in-distribution alignment may reflect the training distribution rather than genuine adaptivity.Source: §7 · p. 12, Figure 5 · p. 13

8 Conclusion and Implications

• Trace validity is not a reliable predictor of solution correctness; irrelevant traces can support high performance, GRPO can improve solutions without improving trace validity, and trace length is agnostic to problem complexity.These findings challenge interpretations of traces as faithful, problem-adaptive reasoning.Source: §8 · p. 13

8.1 Implications

• The authors argue that intermediate tokens need not have end-user semantics; apparent rationales may reflect training-data coincidence rather than corresponding internal computations.They distinguish this position from claiming that tokens can never be interpretable.Source: §8.1 · p. 13
• Without evidence linking reasoning tokens to internal computations, the authors caution against relying on them for safety-critical monitoring and favor third-party verification of solutions.They note that the connection between intermediate tokens and final solutions may be illusory.Source: §8.1 · p. 14
• Trace length may not measure problem complexity; the authors suggest that output-token-based pricing helped conflate the two.Source: §8.1 · p. 14
• The authors propose prompt augmentation as a speculative view: intermediate tokens may alter context to bring a task closer to the model’s training distribution, rather than reflect its computational difficulty.Learning an augmentation function for each task is identified as a central challenge.Source: §8.1 · p. 14

A.1 Additional experiment details

• The experiments use Qwen-2.5-0.5B decoder-only models and a domain-specific tokenizer that reduces the model to around 380M parameters.Source: §A.1 · p. 19
• Training uses AdamW under bf16 precision, with fixed seeds, 100 warm-up steps, and 100k or 250k steps depending on dataset size.The passage specifies β1=0.9, β2=0.999, weight decay 0.1528, and peak learning rate 2.2758e-4.Source: §A.1 · p. 19
• GRPO uses a sample size of 16, batch size of 256, and entropy coefficient of 0.01 for both models.Source: §A.1 · p. 19

A.2 Additional Seed Run for SF-style Model

• The additional SF-style runs change the weight-initialization seed while keeping the training data and other experimental settings identical to those in Table 3.Source: §A.2 · p. 19
• The test evaluates whether the normal-versus-swapped performance pattern is stable under stochastic variation in initialization.Source: §A.2 · p. 19
• Across all distributions, the swapped model outperforms the normal model with the same initialization seed, and the authors report this pattern as statistically robust across seeds.Source: Table 6 · p. 19, §A.2 · p. 19

A.3 Robustness to Trace Shuffling Seed

• With a different trace-shuffling seed, the swapped model’s performance remains highly consistent with the original across all test domains.Source: Table 7 · p. 19, §A.3 · pp. 19–20

A.4 Breakdown of responses as confusion matrices

• The confusion matrices show that valid traces can precede incorrect plans and invalid traces can precede correct plans, so trace accuracy is not a perfect predictor of plan accuracy.Source: Figure 6 · p. 20, Figure 7 · p. 20, §A.4 · p. 20

A.4.1 Error analysis of Model generated invalid traces

• Normal models’ invalid traces most often involve parsing errors, while 100% of swapped-model traces are invalid and generally correspond to a different maze instance.Source: Table 8 · p. 21, §A.4.1 · p. 21

A.5 Validating Traces and Solutions of Searchformer Models

• Across Searchformer model sizes from 15M to 175M parameters, some responses produce a correct plan with an invalid trace.Source: Figure 8 · p. 22, §A.5 · p. 22

A.6 Evaluating Pre-trained LLMs

• The pre-trained-model study fine-tunes Qwen3-8B-base on natural-language A* traces to test solution-only, correct-trace, and swapped-trace regimes.Source: §A.6 · p. 22

A.6.1 Model and Training Setup; A.6.2 Direct Prompting and SFT evaluations

• The setup uses 50k Wilson-generated maze examples and compares direct prompting, solution-only SFT, correct-trace SFT, and SFT with shuffled traces.The correct-trace regime uses structured natural-language A* traces followed by the correct plan.Source: §A.6.1 · p. 22, §A.6.2 · pp. 22–23

A.6.3 Results

• Preliminary Qwen3-8B results show that models trained with incorrect traces gain plan accuracy over the base model despite zero trace validity.Source: Table 10 · p. 24, §A.6.3 · p. 24

A.7 GRPO on Models Trained with Varying Proportions of Swapped Traces

• GRPO yields plan-validity gains across distributions for swapped or partially swapped models, while trace validity decreases in most cases.Across the swap-spectrum checkpoints, plan accuracy continues to increase after GRPO even when trace validity is flat or decreasing.Source: Table 11 · p. 24, §A.7 · p. 24