Source-linked AI summary
Therefore I am. I Think
Esakkivel Esakkiraja, Sai Rajeswar, Denis Akhiyarov, Rajagopal Venkatesaramani
TL;DR
The paper asks whether reasoning models decide tool-use actions before visible deliberation. It combines activation probing, steering, and behavioral analysis, finding that decisions are detectable early, steerable, and often rationalized in chain-of-thought after induced flips.
Problem
The paper asks whether reasoning models form action choices during visible reasoning or encode a strong action tendency before reasoning begins.
Method
The study probes and steers binary tool-calling decisions from hidden activations across two benchmarks, then compares baseline and steered reasoning behavior.
Results
Over 95% confidence is achieved in three of four cases before a single reasoning token, while steering produces decision flips and often longer, rationalizing chain-of-thought.
Takeaways & Limitations
The findings suggest that reasoning models can encode action choices before textual deliberation and that chain-of-thought may rationalize externally induced decisions.
Abstract
from arXiv · showhide
We consider the question: when a large language reasoning model makes a choice, did it think first and then decide to, or decide first and then think? In this paper, we present evidence that detectable, early-encoded decisions shape chain-of-thought in reasoning models. Specifically, we show that a simple linear probe successfully decodes tool-calling decisions from pre-generation activations with very high confidence, and in some cases, even before a single reasoning token is produced. Activation steering supports this causally: perturbing the decision direction leads to inflated deliberation, and flips behavior in many examples (between 7 - 79% depending on model and benchmark). We also show through behavioral analysis that, when steering changes the decision, the chain-of-thought process often rationalizes the flip rather than resisting it. Together, these results suggest that reasoning models can encode action choices before they begin to deliberate in text.
1 Introduction
The paper asks whether reasoning models choose actions during visible deliberation or encode action tendencies beforehand. It probes and steers binary tool-calling decisions to test early predictability, causal influence, and subsequent rationalization.
- The central question is whether a reasoning model forms an action choice during deliberate reasoning or before visible reasoning begins.
- Linear probes test whether action choices are encoded in activations before reasoning tokens are generated.
- Activation steering tests whether injecting or suppressing a decision direction changes model behavior without additional fine-tuning.
- The study uses tool-calling decisions as a binary, interpretable exemplar of action selection across two benchmarks.
- Behavioral analysis examines whether chain-of-thought rationalizes steering-induced decision flips rather than resisting them.
2 Related Work
Related work shows that models can internally represent future outputs and latent beliefs before verbalization. These findings motivate studying whether visible chain-of-thought faithfully reflects internally encoded tool-use decisions and whether representations can be controlled.
- Prior studies report that language models can internally commit to future outputs before those outputs appear in text.
- Probing hidden states has been used to detect early answer correctness, belief formation, and signals for early exit or reduced reasoning.
- Work on chain-of-thought faithfulness finds that explanations may rely on hidden cues or report causes selectively rather than fully faithfully.
- Activation steering and representation engineering provide methods for reading and controlling high-level model states without fine-tuning.
- Figure 1 summarizes the pipeline: probe action decisions, apply steering vectors, and measure quantitative and behavioral effects on chain-of-thought.
3 Methods
The methods evaluate binary tool-use decisions in two benchmarks using activation probes, pre-generation steering, held-out examples, and behavioral comparison of baseline and steered responses.
- Benchmarks: The evaluation uses When2Call and BFCL to test whether models should call a tool or not call one.
- Activation extraction: Reasoning traces identify pre_gen, reasoning boundaries, and the decision token for activation extraction.
- Probe training: The binary prediction target is tool versus no tool.
- Probe training: A logistic-regression probe is trained independently for each sampled layer and token position using binary cross-entropy loss.
- Activation steering: Steering vectors are constructed from class-conditional activation means and added or subtracted at pre_gen with strength α.
- Evaluation: Probe accuracy is evaluated with 5-fold stratified cross-validation and AUROC, while steering uses 100 held-out examples per benchmark.
- Evaluation metrics: Suppression and injection flip rates measure tool-to-no-tool and no-tool-to-tool changes, respectively.
- Behavioral analysis: Reasoning-token change compares steered and baseline reasoning lengths, while judges classify behavioral differences across six categories.
4 Results
Linear probes detect tool-calling decisions before visible reasoning, while activation steering changes both decisions and chain-of-thought behavior. Across benchmarks, steering can induce flips, inflate deliberation, or elicit rationalizing responses, although some decisions resist perturbation.
- Pre-generation predictability: Pre-generation probe decisions agree with think_end decisions over 80% of the time, while think_end decisions coincide with actual model decisions with near-perfect accuracy.
- Activation steering: Steering vectors computed from pre-generation activations test whether the early action signal is causal rather than merely predictive.The intervention targets latent intent to make or avoid a tool call, rather than a representation already mixed with visible chain-of-thought.
- Activation steering: Most steering interventions significantly increase chain-of-thought length as models attempt to resist or rationalize the perturbed direction.Some resistant examples retain relatively unaffected reasoning despite the induced extended deliberation.
- Activation steering: A separate unrelated True/False steering direction produces a 0% flip rate across all models and benchmarks, supporting specificity to the tool-call direction.
- Behavioral analysis: On When2Call, inflated deliberation accompanies decision flips 38−48% of the time, while Qwen3 often shows confabulated support and GLM more often shows no meaningful difference.On BFCL, both models flip decisions more than 53% of the time under injection, with flipped decisions primarily rooted in confabulated support and constraint override.
5 Discussion
The results suggest that action choices can be encoded and detected before visible reasoning, while steering can alter decisions and often prompts rationalizing chain-of-thought.
- Action decisions can be encoded before visible reasoning begins and detected with high confidence from model activations.
- Activation steering can influence action decisions through a direction vector derived from pre-reasoning-generation activations.
- Steering sometimes increases reasoning-token generation, while some resistant examples retain their original decisions despite extended reasoning.
- When steering flips a decision, models often invent reasons that rationalize the change rather than resist it.
- The findings raise concerns that chain-of-thought may misleadingly explain decisions and could provide an attack channel for malicious actors.
- The paper suggests that penalizing high pre-generation probe confidence during reinforcement-learning training may support more faithful action-determining reasoning.
A.1 When2Call Layer-Position Heatmaps
For When2Call, layer-position heatmaps show strong action-decision predictability before generation, with the strongest probes in mid-to-late layers and a temporary early reasoning dip.
- Figure 5 compares probe AUROC across sampled layers and generation positions for Qwen3-4B and GLM-Z1-9B on When2Call.
- The strongest probes appear in mid-to-late layers for both models.
- Pre-generation activations show strong action-decision predictability before reasoning tokens are produced.
A.2 BFCL Layer-Position Heatmaps
Figure 6 presents layer-position heatmaps for BFCL across the two main models.
- Figure 6 shows BFCL layer-position heatmaps for the two main models.
A.3.1 Layer-Position Heatmaps
Supplemental GPT-OSS-20B heatmaps cover When2Call and BFCL, displaying medium- and high-reasoning conditions side by side.
- Figures 7 and 8 show GPT-OSS-20B layer-position heatmaps for When2Call and BFCL.
- Each figure places medium- and high-reasoning conditions side by side.
A.3.2 Position Curves
GPT-OSS-20B shows strong early predictability of final tool decisions across When2Call and BFCL, followed by an early reasoning-trace dip and later recovery. Behavioral examples show that steering can inflate deliberation, flip decisions, or leave decisions unchanged while destabilizing reasoning.
- Position Curves: Strong pre_gen predictability appears on When2Call and BFCL under both medium and high reasoning for GPT-OSS-20B.The same overall pattern persists across reasoning settings and benchmarks.
- Position Curves: Agreement with the final think_end probe is initially lower for GPT-OSS-20B, then strengthens toward later reasoning positions on When2Call and BFCL.The later-position strengthening is reported for both benchmarks, while the lower early agreement is specifically noted on When2Call.
- Behavioral Examples: Suppression can produce 2.87× more reasoning and flip a high-confidence tool decision into a no-tool outcome.In the example, the probe assigns 0.9992 tool probability, but steering causes fixation on city formatting and eliminates the tool call.
- Behavioral Examples: A resistant suppression example preserves the tool call despite 3.57× reasoning inflation and visible hedging about function arguments.The final decision survives, but the reasoning trace is described as visibly destabilized.
- Behavioral Examples: An injection-resistant no-tool example leaves the final decision unchanged and produces only a brief, self-corrected reference to an unrelated function.The probe assigns 7.5 × 10−9 tool probability in this example.