PaperlayerView exact revision on arXiv

Source-linked AI summary

The belief-action gap in language model agents

Ayse Sila Okcu, Weisheng Wang, Tongmeng Xie, Niall Dalton, Angelos Nalmpantis, Raghu Arghal, Mario Giulianelli

arXiv:2610.17444v1cs.LGcs.AIcs.CL

TL;DR

Output-only evaluation cannot distinguish why language model agents take suboptimal actions. This paper studies beliefs, preferences, and actions in a text-rendered grid world using probes, inverse reinforcement learning, reasoning tracking, and interventions. It finds that task-relevant information can remain available while behaviour is suboptimal, and the belief–action gap decreases after reasoning despite less precise decoded state beliefs.

  • Problem

    Output-only evaluation cannot diagnose whether suboptimal actions reflect incorrect state beliefs, undesirable preferences, or inconsistency with available information.

    Source: §1 · p. 1

  • Method

    In a controlled text-rendered grid world, the study estimates beliefs and preferences, tracks action preferences through reasoning, and intervenes on reasoning text and hidden representations.

    Source: §1 · p. 2

  • Results

    Task-relevant information can remain available during suboptimal behaviour, and the belief–action gap decreases after reasoning even as decoded state beliefs become less precise.

    Source: §7 · p. 9, §6 · p. 9

  • Takeaways & Limitations

    Decodable state information does not by itself imply that it governs behaviour, supporting a distinction between state estimation, preference, and action selection as sources of failure.

    Source: §7 · p. 10

Abstract

from arXiv · show

Language model (LM) agents can fail for different reasons, requiring different explanations and interventions. We study how state beliefs, action values, and action preferences interact as LM agents reason and commit to decisions in text-rendered grid worlds. We estimate task-relevant beliefs using report-based and representation-based probes, and action values using inverse reinforcement learning. We find that task-relevant state and value information is often represented more reliably than it is reflected in behaviour. Tracking action preferences throughout reasoning further shows that decisions sharpen around a commitment point and reported belief uncertainty decreases near commitment, although individual belief changes do not consistently explain shifts in action preference. Finally, forced actions follow reasoning prefixes from other decisions when transplanted, whereas the tested reasoning-window activation edits provide limited control over the action. Together, these results suggest that some failures arise not from missing task-relevant information, but from how available information is translated into action. Our measurements distinguish information availability from belief-action consistency without by themselves identifying the cause of each failure. Our code is available at: https://anonymous.4open.science/r/belief-action-gap-F745/

1 INTRODUCTION

Similar agent failures can reflect incorrect state beliefs, undesirable preferences, or failures to translate available information into action. This study examines those components separately in a language-model agent acting in a text-rendered grid world.

Study approach

  • The analyses test when task-relevant information is available, how action preferences form, and whether that information causally influences the selected action.The agent operates in a text-rendered grid world with known states, transitions, and optimal actions.Source: §1 · p. 2

Belief–action diagnostics

  • The study combines estimated state beliefs and inferred preferences to assess whether emitted actions are consistent with information available to the agent.Beliefs are estimated from internal activations and explicit reports, while preferences are inferred from behaviour using inverse reinforcement learning.Source: §1 · p. 2

Decision formation during reasoning

  • Action preferences sharpen around a commitment point, which the study tracks alongside changes in beliefs and hidden-state patterns.It examines how beliefs and hidden-state patterns change around the transition at which the preferred action stabilises.Source: §1 · p. 2

Causal tests of information use

  • Reasoning-text interventions can strongly redirect action preferences, whereas targeted edits to highly decodable hidden representations are much less effective.These interventions test whether task-relevant information causally influences the selected action.Source: §1 · p. 2

Empirical evidence for a belief–action gap

  • Task-relevant information can remain available when behaviour is suboptimal, and improved action consistency after reasoning does not simply result from more precise state estimation.The study frames this as a belief–action gap, distinguishing information availability from whether it governs the selected action.Source: §1 · p. 2

2 ESTIMATING BELIEFS AND VALUES

The section defines a framework for separating what the agent believes and values from the action it emits, using state-belief estimates and choice-inferred values to quantify action consistency and uncertainty.

2 ESTIMATING BELIEFS AND VALUES

  • The framework separates state misidentification, wrong outcome preferences, and inconsistent action by combining activation-probed beliefs, choice-inferred values, emitted actions, and decoded-belief entropy.Beliefs are estimated with activation probes, values with one-step inverse reinforcement learning, and the resulting estimates support expected local regret analysis.Source: §2 · p. 2

2.1 GRID WORLD ENVIRONMENT

  • In a fixed 9 × 9 fully observable DoorKey grid, the agent receives current coordinates and key status, reasons, then chooses LEFT, RIGHT, UP, or DOWN; transitions are deterministic.The key must be collected before the locked door can be unlocked, and invalid moves leave the agent in place.Source: §2.1 · pp. 2–3

2.2 ESTIMATING BELIEFS

  • Task-state beliefs come from activation probes over position, key possession, and door status, alongside separately queried reports that are excluded from belief–action-gap calculations.Probe predictions are combined into a factorised distribution over valid states; report accuracy uses the highest-probability answer, with uncertainty measured by Shannon entropy.Source: §2.2 · p. 3

2.3 ESTIMATING VALUES

  • One-step maximum-entropy inverse reinforcement learning fits linear state values from activation-derived state features to emitted actions, evaluating actions by deterministic successor-state value.Separate models are fit for pre-reasoning and post-reasoning regions, and parameters are trained by minimizing the emitted actions’ negative log-likelihood.Source: §2.3 · p. 3

2.4 QUANTIFYING BELIEF–ACTION CONSISTENCY

  • Expected local regret decomposes into the belief–action gap (loss versus the best action under the estimated belief) and value of perfect state information; Shannon entropy separately measures belief dispersion.The value of perfect information measures the gain available if the current state were known before acting.Source: §2.4 · p. 4

3 LINKING BELIEFS AND VALUES TO BEHAVIOUR

Reasoning increases alignment between emitted actions and decoded-belief preferences while making decoded state estimates less precise and more uncertain. The results show that available state and value information does not always translate into behaviour.

3 LINKING BELIEFS AND VALUES TO BEHAVIOUR

  • The proof-of-concept analysis covers 200 gpt-oss-20b grid-world trajectories and 3,194 decision steps, comparing reasoning-generated actions with actions forced before reasoning.The grid world has known states, transitions, and optimal actions; reasoning traces and actions use stochastic decoding at temperature 0.7.Source: §3 · p. 4

3.1 INFERRED STATE BELIEFS AND PREFERENCES

  • After reasoning, held-out has_key and door_open probe accuracy decreases from 100.0% to 89.5% and 88.6%, respectively.Location top-1 accuracy falls from 34.8% to 13.1%, and accuracy within Manhattan distance one falls from 95.5% to 60.8%. The inferred state-preference model's held-out action-prediction accuracy rises from 29.2% before reasoning to 45.2% after reasoning; in three illustrative wall-hit failures, the model reports that its chosen direction is blocked but still selects it.Source: §3.1 · pp. 4–5

3.2 BELIEF–ACTION CONSISTENCY

  • On the held-out set, expected local regret ∆ decreases from 0.700 to 0.670, while reasoning lowers belief–action gap R but raises expected value of perfect state information I and decoded-belief entropy H.The increase in I partly offsets the decrease in R, so the change in ∆ understates the increase in belief–action consistency.Source: Table 1 · p. 5, §3.2 · p. 5

4 DECISION FORMATION THROUGH REASONING

Action preferences sharpen and stabilise during reasoning, often before it ends, while reported belief uncertainty falls modestly and belief corrections rarely coincide with preference changes. Hidden-state patterns also shift around commitment and decision revision, suggesting these are related but distinct components of decision formation.

4 DECISION FORMATION THROUGH REASONING

  • The analysis tracked forced-choice preferences across 95 trajectories and 1,276 decisions, with report-based beliefs available for 7,084 prefixes from 46 decisions across 31 trajectories.Source: §4 · p. 5

4.1 FORMATION OF ACTION COMMITMENT

  • The mean probability assigned to the eventual action rose from 0.35 in the first tenth of reasoning to 0.92 in the final quarter, while inverse normalised action entropy rose from 0.34 to 0.91; final forced actions matched emitted actions in all decisions.Figure 3 shows the evolution across 95 trajectories, 1,276 decisions, and 98,310 evaluated prefixes.Source: §4.1 · p. 6, Figure 3 · p. 6
  • Before any decisional action mention, eventual-action probability still rose from 0.35 to 0.92 and inverse normalised action entropy from 0.33 to 0.89, showing the pattern was not simple repetition.Source: §4.1 · p. 6
  • Median commitment occurred at 0.49 of the reasoning trace, leaving 50.6% afterward; forced-action preference preceded textual commitment in 99.87% of eligible decisions, by a median 523 tokens.Source: §4.1 · p. 7
  • Preference stabilisation was gradual: the next-prefix change probability fell from 0.133 to 0.019, decisions averaged 6.0 switches before commitment, and preferred actions were planner-optimal in 76.7% of prefixes.Source: §4.1 · p. 7

4.2 BELIEFS AROUND COMMITMENT

  • Around commitment, mean reported-belief entropy fell from 0.392 to 0.336 bits, while report accuracy changed only from 82.8% to 83.0%.The entropy change was −0.056 bits, with a 95% trajectory-bootstrap interval of [−0.105, −0.009].Source: §4.2 · p. 7
  • Across 682 action changes, report correctness changed in only 14 cases (2.1%); adding belief features raised three-sentence optimality-loss AUPRC from 0.273 to 0.334 (Δ0.061).All six reports remained correct in 66 of 161 transitions from an optimal to a suboptimal preferred action; the AUPRC improvement’s 95% interval was [0.014, 0.123].Source: §4.2 · p. 7

4.3 HIDDEN STATES AROUND DECISION CHANGES

  • Hidden-state clusters associated with solving and action consolidation became more common after commitment, while reading and environment-reconstruction clusters declined; C10 appeared in 14.0% of commitment sentences versus 5.9% of matched controls.Several clusters were also enriched at action changes in both optimal-to-suboptimal and suboptimal-to-optimal directions, consistent with decision revision rather than error-specific processing.Source: §4.3 · pp. 7–8

5 CAUSAL INFLUENCES ON ACTION SELECTION

Reasoning-text interventions causally shift current action preferences, though their influence weakens as action confidence rises. Action decisiveness is decodable from a reasoning-window activation site, but the tested activation edits provide limited control over action identity.

5.1 REASONING-TEXT PATCHING

  • Across prefixes at approximately 10%, 50%, and 90% of another decision’s reasoning trace, forced-action probability favored the transplanted action: π(aB) ∈[0.598, 0.673] versus π(aA) ∈[0.129, 0.165].The experiment used 396 mismatched prompt–reasoning pairs and 396 congruent controls.Source: §5.1 · p. 8
  • Injected runner-up assertions switched the forced top action in 100%, 92.5%, 90.0%, and 62.5% of cuts across increasing confidence bins, while neutral text changed it in 9.6%.Susceptibility decreased with current action confidence (Spearman ρ = −0.421, p = 5.0×10−10; partial ρ = −0.175, p = 0.013); supporting text raised the mean probability of the current top action from 0.757 to 0.989.Source: §5.1 · p. 8

5.2 ACTIVATION PATCHING

  • A nonlinear readout predicted inverse normalised action entropy from reasoning-window activations with held-out Spearman ρ = 0.771 ± 0.009, above length-oracle 0.644 and positional 0.371 baselines.The fixed window contained up to the final 12 reasoning tokens before the answer-forcing suffix.Source: Figure 4 · p. 8, §5.2 · p. 9
  • Full reasoning-window replacement redirected 20.8% of donor–recipient pairs to the donor-associated action, while steering matched random-direction performance at 3.6% and low-dimensional replacement produced no intended flips.The findings show limited control from the tested activation edits, not that represented information is causally unused.Source: Figure 4 · p. 8, §5.2 · p. 9

6 DISCUSSION & LIMITATIONS

The discussion argues that task-relevant information can be represented without reliably guiding action: action consistency improves even as decoded state estimates become less precise. Text changes can shift action preferences, but tested activation edits do not reliably steer behavior; the findings are limited by the controlled setting and methods used.

6 DISCUSSION & LIMITATIONS

  • The belief–action gap decreases after reasoning even as decoded state distributions become less precise, so improved action consistency does not appear to require sharper environmental estimates.In illustrative wall-hit cases, the model reports that its chosen direction is blocked yet still selects it.Source: §6 · p. 9
  • Action preferences sharpen and usually stabilize around mid-reasoning, but belief corrections rarely coincide with action changes and no consistent correction point emerges.Around commitment, report accuracy changes little; preferences may change repeatedly before commitment.Source: §6 · p. 9
  • Replacing reasoning text or injecting an action suggestion strongly changes current action preference, whereas tested edits to highly decodable reasoning-window representations do not reliably steer behavior.The results distinguish decodability from causal control (Belinkov, 2022).Source: §6 · p. 9
  • The findings are limited to a controlled deterministic grid world and the estimators and interventions tested, motivating tests in richer environments and with stronger causal interventions.Probes establish decodability rather than direct access to internal beliefs (Hewitt & Liang, 2019; Belinkov, 2022); the one-step IRL model infers preferences, not the model’s true value function.Source: §6 · p. 9

7 CONCLUSION … B REPORT-BASED PROBE PROTOCOL

The paper finds that task-relevant information may be represented or reported without governing action, while preferences sharpen during reasoning and reasoning-text edits can redirect them. The merged sections also describe the probes, value-estimation controls, protocol, reproducibility, AI use, and ethical scope behind these analyses.

7 CONCLUSION

  • Task-relevant information can remain available despite suboptimal behavior; preferences sharpen during reasoning, and reasoning-text edits redirect them more than edits to a hidden representation predictive of decisiveness.The belief–action gap decreases after reasoning even as decoded state beliefs become less precise, and explicit belief correctness explains only a small fraction of action revisions.Source: §7 · pp. 9–10

ETHICS STATEMENT; AI USE STATEMENT; REPRODUCIBILITY STATEMENT

  • The work uses no human participants or personal data, treats belief and preference as operational descriptions rather than evidence of human-like cognition, and notes that intervention techniques could steer behavior elsewhere.Authors reviewed AI-assisted code and manuscript content; experiments came from implemented analysis pipelines, and the repository provides those pipelines and specifies required external inputs.Source: ETHICS STATEMENT · p. 10, AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 REPRESENTATION-BASED BELIEF ESTIMATION

  • Representation-based probes decode position, key possession, and door status from concatenated three-token activations using state-disjoint training and evaluation splits.The three-token input has 8,640 dimensions; position uses an MLP classifier, while key possession and door status use linear classifiers, with 80% of unique states for training and 20% for evaluation.Source: §A.1 · p. 12

A.2 REPORT-BASED BELIEF ESTIMATION

  • Report-based beliefs use 38 questions about state features and candidate moves, with parsed answers, empirical or normalized score distributions, and entropy-based uncertainty.Invalid responses are not matched approximately; because queries require a separate forward pass and may elicit new information, they are analyzed separately from the belief–action gap.Source: §A.2 · p. 13
  • Across three blocked-action decisions, the model correctly reported that its chosen direction was blocked but still selected that action.The report-based analysis treats these as cases where task-relevant state information is explicitly recoverable without determining behavior.Source: §A.2 · p. 13

A.3 ONE-STEP IRL ESTIMATION

  • One-step IRL fits linear value functions to deterministic successor states and retains observations only when all four candidate-action successors are represented.Agent-supervised pre- and post-reasoning models use actions from their corresponding decision conditions; optimization uses Adam for 500 epochs with learning-rate annealing, gradient clipping, and L2 regularization.Source: §A.3 · pp. 13–14

A.4 PLANNER-SUPERVISED IRL CONTROL

  • The planner-supervised control replaces realized-action labels with shortest-path optimal actions to test whether activation-derived representations support task-optimal action ordering better than the agent’s policy.Evaluation can count any shortest-path optimal action as correct when multiple such actions are available.Source: §A.4 · p. 14

B REPORT-BASED PROBE PROTOCOL

  • The report protocol specifies 38 questions, categorical and coordinate response formats, parsing and scoring rules, and distinct reasoning-time and endpoint query panels.The reasoning-time panel reads directional walls, key and door status, and action outcomes; the scaled matched-cohort run defers coordinate questions.Source: B REPORT-BASED PROBE PROTOCOL · pp. 14–15

B.1 ADDITIONAL REPORT-BASED BELIEF RESULTS · C ADDITIONAL BELIEF AND VALUE RESULTS

Report-based queries recover task-relevant local state information, including in cases where the agent selects an action that the report identifies as blocked. These results show that information retrievable in a separate pass may differ from what is decoded from decision activations or reflected in behavior.

B.1 ADDITIONAL REPORT-BASED BELIEF RESULTS

  • Greedy report accuracy was 93.9% for walls, 100.0% for goal adjacency, 99.2% for door adjacency, and 67.4% for key adjacency across the 33-state failure set.Each directional query family contained 132 state–question pairs.Source: §B.1 · p. 15
  • Modal answers from ten samples at temperature 0.7 raised report accuracy to 97.0% for walls, 100.0% for goal adjacency, 100.0% for door adjacency, and 97.0% for key adjacency.These are report-based estimates on the 33-state failure set.Source: §B.1 · p. 15
  • In all three wall-hit failures in a 13-decision failure-focused slice, the model correctly reported that its ultimately selected direction was blocked.The slice also contained four avoidable detours, two backtracks, and four optimal baseline decisions; reports varied how much of the reasoning trace was shown without revealing the final action.Source: §B.1 · p. 15
  • On the same 15 decision states, report-based estimates correctly identified wall status for every evaluated direction, while decoding from original decision activations was less reliable.Table 3 reports accuracy for these matched estimates.Source: §B.1 · p. 16, Table 3 · p. 16
  • The report-versus-activation comparison does not establish that report-based beliefs are intrinsically more faithful, because reports come from a separate forward pass and may recover information unused in the original decision.The findings instead show that local state information can sometimes be explicitly retrieved even when it is not perfectly decoded from decision activations or reflected in the selected action.Source: §B.1 · p. 16

C.1 EXPANDED STATE-DECODING RESULTS … D.1 REASONING SEGMENTATION AND COMMITMENT MEASUREMENT

These appendix results show that state and action-preference information is recoverable to varying degrees, while planner-optimal labels yield cleaner task-optimal action structure than the agent’s realized actions. They also specify how reasoning traces, action preferences, and textual commitment were measured.

C.1 EXPANDED STATE-DECODING RESULTS

  • Key possession and door status decoded reliably, but exact position was harder: its probe accuracy was 34.8% before reasoning and 13.1% after, above the 2.56% chance level.Across 39 walkable locations, the post-reasoning readout was less accurate than the pre-reasoning readout.Source: §C.1 · p. 16, Table 4 · p. 16
  • Before reasoning, the most probable decoded location was within Manhattan distance one in 95.5% of examples and distance two in 99.0%, versus 60.8% and 83.5% after reasoning.Local probability mass and binary state-variable decoding also became less precise at the post-reasoning readout.Source: §C.1 · p. 16

C.2 PLANNER-SUPERVISED IRL CONTROL

  • With activation inputs and training fixed, replacing agent-action labels with planner-optimal labels raised test accuracy from 45.4% to 96.0% and optimal-set membership accuracy from 36.9% to 99.2%.This separate four-room control supports the finding that planner-supervised IRL recovers a cleaner task-optimal ordering than agent-supervised IRL.Source: §C.2 · p. 17
  • Planner- and agent-supervised IRL learned different state-preference structures, but absolute value magnitudes and signs should not be read as distances to the goal.One-step IRL constrains comparisons among successor states, so the induced ordering of candidate actions is the meaningful quantity.Source: §C.2 · p. 17, Figure 5 · p. 17

C.3 AGENT-SUPERVISED IRL AND INVERSE-TEMPERATURE ROBUSTNESS

  • On held-out source states, agent-supervised value models predicted 29.2% of pre-reasoning actions and 45.2% of post-reasoning actions.The lower pre-reasoning accuracy coincided with a more imbalanced no-reasoning action distribution, limiting direct comparison of the raw accuracies.Source: §C.3 · p. 17
  • Across tested inverse-temperature settings, the belief–action gap decreased from before to after reasoning while decoded belief entropy increased.Changing β altered policy sharpness and learned-value scale but not this qualitative endpoint pattern; the main analysis uses β = 1.Source: §C.3 · p. 18

D.1 REASONING SEGMENTATION AND COMMITMENT MEASUREMENT

  • Reasoning-time analyses used 95 trajectories, 1,276 decisions, and 2,370,850 tokens, segmented into 153,622 sentence-level units with boundaries and short-span merging rules.Segmentation excluded terminal action JSON but retained embedded action JSON, and was fixed independently of the prefixes used for action elicitation.Source: §D.1 · p. 18
  • At up to 80 prefixes per decision, forced-action commitment marked the start of the final stable preferred-action segment, while textual commitment separately identified a concluding sentence naming the emitted action.The analysis evaluated 98,310 prefixes; textual commitment was identified in 753 of 1,276 decisions, using decisional mentions that excluded descriptive, idiomatic, and negated uses.Source: §D.1 · p. 18

D.2 ADDITIONAL COMMITMENT RESULTS · D.3 BELIEF CHANGES AROUND COMMITMENT AND ACTION REVISION

Action preferences generally sharpen around a mid-trace commitment, while reported belief uncertainty falls near commitment and action revisions rarely coincide with changes in reported local-state correctness. Behavioural and belief readouts predict some later optimality losses, but the observational analyses do not establish why available information is translated into action.

D.2 ADDITIONAL COMMITMENT RESULTS

  • Median action commitment occurs at 0.49 of the reasoning trace, with a median 50.6% of the trace, or 524 tokens, following commitment.Only 37.0% of decisions commit before the first decisional action mention; inferred commitment positions are upper-resolution estimates because commitment is defined over evaluated prefixes. The prompt alone also induces a non-uniform action prior: mean probability of the eventual action is 0.34, versus 0.25 under a uniform four-action distribution.Source: §D.2 · pp. 18–19
  • The probability of a preference switch at the next evaluated prefix falls from 0.133 in the first reasoning decile to 0.019 in the final decile, but confidence does not guarantee stability.Decisions average 6.0 preference switches before commitment, and when eventual-action probability exceeds 0.9, the probability of a later switch remains 0.089. RIGHT decisions commit earliest (median 0.39) and DOWN latest (0.61).Source: §D.2 · p. 19, Figure 6 · p. 19
  • After commitment, world-reading falls from 68% to 29% of sentences and solving rises from 25% to 45%, beyond changes in a position-matched placebo.Concluding or answering rises from 1% to 12%, but its increase is not greater than the placebo; meta-checking is exploratory because annotation precision is low.Source: §D.2 · p. 19, Figure 7 · p. 20

D.3 BELIEF CHANGES AROUND COMMITMENT AND ACTION REVISION

  • Mean report entropy falls from 0.392 to 0.336 bits around commitment, while report accuracy changes only from 82.8% to 83.0%.The entropy change is −0.056 bits (95% trajectory-bootstrap interval [−0.105, −0.009]); values are averaged within decisions before aggregation.Source: Table 5 · p. 20, §D.3 · p. 20

D.2 ADDITIONAL COMMITMENT RESULTS

  • After commitment, grid-related vocabulary falls to 0.37 times its pre-commitment density, while coordinate pairs and checking-related vocabulary increase by factors of 1.41 and 3.73.Explicit direction-word answer mentions occur almost entirely after commitment: 29 before and 1,751 after.Source: §D.2 · p. 20

D.3 BELIEF CHANGES AROUND COMMITMENT AND ACTION REVISION

  • Only 14 of 682 action changes coincide with a change in any current-state report, and all six reports remain correct across 66 optimal-to-suboptimal transitions.Mean report-error differences are close to zero across action changes; action quality can deteriorate without a corresponding reported local-state error.Source: Table 6 · p. 21, §D.3 · p. 21
  • Adding observable belief reports raises held-out AUPRC for predicting an optimality loss within three sentence boundaries from 0.273 to 0.334.The improvement is 0.061 (95% trajectory-bootstrap interval [0.014, 0.123]); next-sentence prediction is inconclusive.Source: §D.3 · p. 21
  • At layer 15, attention to the newest reasoning sentence increases at action changes by 0.0838 percentage points per token, an observational association that does not establish causation.The estimate comes from 67 matched pairs; corresponding changes for the grid, key-status statement, and earlier reasoning include zero.Source: §D.3 · p. 21

D.4 HIDDEN-STATE CLUSTERING ANALYSIS · E ADDITIONAL REASONING-TEXT INTERVENTION RESULTS · E.1 WHOLE-CHAIN REASONING PATCHING

Hidden-state clusters shift around commitment and are enriched when preferred actions change, while transplanted reasoning prefixes steer forced actions toward the action associated with that reasoning. The analyses treat clusters as numeric states rather than semantic categories and do not tie their revision associations specifically to error correction.

D.4 HIDDEN-STATE CLUSTERING ANALYSIS

  • Unsupervised K-means clusters were analyzed as numeric identities, not semantic categories: their normalized mutual information with annotations was 0.131 across 20,366 aligned sentences.Source: §D.4 · p. 21
  • After commitment, C11, C2, and C7 rose by 12.8, 6.3, and 5.6 percentage points, while C5, C6, C0, and C1 fell by 9.8, 7.4, 4.4, and 2.7 points.Post hoc annotations associate the increasing clusters more with solving and answering, and the decreasing clusters more with reading and reconstructing the environment.Source: §D.4 · p. 22
  • At true commitment sentences, C10 appeared in 14.0% versus 5.9% of position-matched controls; C9 and C2 showed smaller excesses of 3.2 and 2.5 points.Source: §D.4 · p. 22, Table 7 · p. 22
  • C10 prefixes showed the largest subsequent increases in eventual-action probability (0.0245), confidence (0.0400), and action margin (0.0402); C11 prefixes began at mean probability 0.933.C11 prefixes showed smaller subsequent changes.Source: §D.4 · p. 22
  • At action revisions, C6, C10, C9, and C5 were enriched by factors of 1.65, 1.64, 1.41, and 1.33 versus shuffled-event controls; all remained significant after FDR correction (q = 0.0024).Source: §D.4 · p. 22, Table 8 · p. 22
  • At optimal-to-suboptimal transitions, C6/C10/C9/C5 enrichments were 2.07/1.48/1.45/1.26; at suboptimal-to-optimal transitions, they were 1.77/1.59/1.44/1.35, suggesting revision rather than error-correction-specific states.Source: §D.4 · p. 22, Table 8 · p. 22

E.1 WHOLE-CHAIN REASONING PATCHING

  • Across cuts, forced actions favored transplanted reasoning; at the late cut, reasoning-chain agreement was 68.9% for mismatched versus 98.5% for congruent pairs.Mean probabilities assigned to the reasoning-chain action were 0.673 for mismatched pairs and 0.971 for congruent pairs.Source: §E.1 · pp. 22–23, Figure 8 · p. 23

E.2 SINGLE-SENTENCE INTERVENTIONS · F ADDITIONAL ACTIVATION INTERVENTION RESULTS · F.1 COMMITMENT READOUT AND INTERVENTION SETUP

Single-sentence counter-evidence can redirect forced actions, but its effect weakens as action confidence and reasoning progress increase; a readout of the final reasoning tokens predicts inverse action entropy better than length and position baselines. The tested activation interventions target the current action, distinguishing any forced-top-action change from a change to the intended target.

E.2 SINGLE-SENTENCE INTERVENTIONS

  • The single-sentence study sampled 240 reasoning prefixes across six bins of current eventual-action probability and evaluated 720 independent forced actions.At each prefix, it appended counter-evidence for the runner-up, a neutral sentence naming no action, or support for the current top action.Source: §E.2 · p. 23
  • Intervention effect declined with reasoning progress (ρ = −0.436), while action confidence retained a smaller independent association after controlling for position (partial ρ = −0.175, p = 0.013).Action confidence itself correlated with reasoning position (ρ = 0.538).Source: §E.2 · p. 23
  • The neutral sentence changed the preferred action in 9.6% of cuts, while the supportive sentence raised mean probability on the current top action from 0.757 to 0.989.These controls indicate the counter-evidence effect primarily reflects the injected sentence’s semantic content, rather than merely appending text.Source: §E.2 · p. 23
  • The forced top action switched to the injected runner-up in 100% of cuts with c_t,k < 0.7, 92.5% for 0.7 ≤ c_t,k < 0.85, 90.0% for 0.85 ≤ c_t,k < 0.95, and 62.5% for c_t,k ≥ 0.95.Across the 201 cases where the injected runner-up differed from the model’s eventual action, susceptibility decreased with action confidence (Spearman ρ = −0.421, p = 5.0 × 10−10).Source: §E.2 · p. 23, Figure 9 · p. 24

F.1 COMMITMENT READOUT AND INTERVENTION SETUP

  • A nonlinear readout of the final up-to-12 reasoning tokens predicted inverse normalised action entropy with held-out Spearman ρ = 0.771 ± 0.009, versus 0.644 for a length-oracle baseline and 0.371 for a positional baseline.Evaluation used trajectory-grouped splits.Source: §F.1 · p. 24
  • The activation interventions tested whether the reasoning-window representation could control the current action, distinguishing any forced-top-action change from a change specifically to the intended target.The main paper summarizes these results in Fig. 4.Source: §F.1 · p. 24

F.2 STEERING, TRANSPLANTATION, AND ABLATION CONTROLS … C After reasoning

Whole-state transplantation changed forced actions more often than low-rank transplantation, while tested steering interventions showed limited directed control. Figure 10 illustrates a belief–action gap and reports aggregate changes in gap components after reasoning.

F.2 STEERING, TRANSPLANTATION, AND ABLATION CONTROLS

  • Targeted and norm-matched random steering each changed the action to the intended target in 3.6% of cases, so targeted steering did not outperform the control.This comparison is at the representative magnitude shown in Fig. 4.Source: §F.2 · p. 24
  • Replacing the full 12-token reasoning-window state changed forced actions in 25.0% of donor–recipient pairs and redirected 20.8% to the intended donor-associated action; low-rank transplantation produced no intended flips.The result applies to the tested settings.Source: §F.2 · p. 24
  • The tested interventions do not show that the reasoning-window representation is causally irrelevant, but they show it was not reliably writable under these edits.Edits covered particular layers, directions, and subspaces, and no known positive-control steering direction was available; sub-block deletion changed actions in up to 29.2% of cases without an intended target, while an explicit instruction changed actions in 5.8% and reached the intended target in 1.0%.Source: §F.2 · p. 24

RIGHT 2.47

  • UP is best under the example’s belief, while RIGHT hits a wall at the outlined actual position.The illustration assigns belief probability to nine positions and values to all 39 non-wall positions.Source: RIGHT 2.47 · p. 25, Figure 10 · p. 25

G ILLUSTRATIVE BELIEF–ACTION DECOMPOSITION

  • In the constructed DoorKey example, the emitted RIGHT action has expected value 2.47, versus 3.95 for UP as the best single action and 5.34 for choosing after learning the state.The resulting belief–action gap is R = 1.48, value of perfect state information is I = 1.39, and expected local regret is ∆ = 2.87; these are illustrative, not fitted observations or task rewards.Source: Figure 10 · p. 25

C After reasoning

  • Across 3,194 decisions, the empirical endpoint means show R decreasing while I and decoded-belief entropy H increase after reasoning.The figure uses representation-based beliefs and agent-supervised value models fitted separately before and after reasoning.Source: Figure 10 · p. 25
Loading 2610.17444v1…