Source-linked AI summary
Post-Hoc Reasoning in Chain of Thought: Decoding and Steering Pre-Committed Answers
Kyle Cox, Darius Kianersi, Adrià Garriga-Alonso
TL;DR
The paper asks whether chain-of-thought rationales faithfully reflect model decisions, especially when answers are determined before reasoning. It probes pre-CoT activations and steers them to test causal influence, finding answer control and distinct reasoning failures when incorrect beliefs are induced.
Problem
CoT’s interpretability depends on whether verbalized reasoning reflects the model’s underlying decision process, yet instruction-tuned models may determine answers before generating it.
Method
The authors train linear probes on residual-stream activations before CoT and intervene by steering generations along the resulting directions.
Results
The probes decode final answers and steering along them changes answers, while incorrect steering produces confabulation, non-entailment, and hallucination.
Takeaways & Limitations
The probe directions plausibly recover causal representations of pre-committed answers, suggesting that post-hoc reasoning can produce unfaithful rationales when the prior belief is wrong.
Takeaways & Limitations
The experiments focus mainly on instruction-tuned rather than reasoning models; one reasoning model showed some probe decoding but little steering success.
Abstract
from arXiv · showhide
As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning. However, the utility of CoT toward interpretability depends upon its faithfulness---whether the model's stated reasoning reflects the underlying decision process. We provide mechanistic evidence that instruction-tuned models often determine their answer before generating CoT. Training linear probes on residual stream activations at the last token before CoT, we can predict the model's final answer with >0.9 AUC on most tasks. We find that these directions are not only predictive, but also causal: steering activations along the probe direction often flips model answers, with flip rates substantially exceeding norm-matched orthogonal baselines across most model-dataset pairs. When steering induces incorrect answers, we observe two distinct failure modes: confabulation (fabricating false premises) and non-entailment (stating correct premises but drawing unsupported conclusions). While post-hoc reasoning may be instrumentally useful when the model has a correct pre-CoT belief, these failure modes suggest it can result in undesirable behaviors when reasoning from a false belief.
1. Introduction
The paper examines whether chain-of-thought reasoning faithfully reflects model decisions, focusing on answers determined before CoT and the resulting failure modes. It tests whether pre-CoT answer representations are linearly decodable, causally influential, and associated with confabulation or non-entailment.
- CoT can support interpretability and safety monitoring because natural-language rationales are easier to audit than latent activations.
- Faithfulness is uncertain because models can rationalize biased answers with misleading CoT or ignore their own reasoning when producing final answers.
- Post-hoc reasoning denotes cases where the model determines its answer before generating CoT; that answer is called the pre-committed answer.
- Prior prompt-level studies found that models can preserve answers when CoT is removed, replaced, or contradicted by misleading reasoning.
- Hypotheses: The paper tests whether pre-CoT activations encode the final answer, whether the corresponding probe direction causally shifts answers, and whether incorrect steering produces confabulation or non-entailment.
- Interpretation: The authors treat the probe direction as a plausible causal representation while acknowledging alternative explanations and interpretive uncertainty.
2. Methods
The study evaluates instruction-tuned models on binary reasoning tasks using CoT interventions, pre-CoT probes, activation steering, and trace classification. These methods test post-hoc reasoning, causal answer control, and the structure of reasoning failures after incorrect steering.
- Models and datasets: Five instruction-tuned models are evaluated across four binary classification tasks covering factual, logical, social, and sports reasoning.
- Models and datasets: Each task uses four few-shot CoT demonstrations, randomized A/B answer ordering, 500 training examples, and 500 test examples.
- CoT interventions: The experiments establish that models often exhibit post-hoc reasoning on the selected tasks.
- CoT interventions: The study measures CoT sensitivity by comparing accuracy with and without reasoning and by replacing correct generations with ellipses or misleading CoT.
- Probing for pre-computed answers: Difference-of-means probes predict semantic answers from residual-stream activations at the last pre-CoT token, with AUC measuring linear decodability.
- Flipping answers via activation steering: Activation steering edits the probe direction at the layer with the highest probe AUC during every post-prompt decoding step, sweeping coefficients toward opposite answers.
- Flipping answers via activation steering: A per-example random orthogonal direction with matched norm provides a baseline for distinguishing probe-specific causal effects from generic perturbation effects.
- Classifying CoT traces: Steering-induced incorrect responses are classified by premise truthfulness and logical entailment, using an LLM judge to assign reasoning-trace categories.
3. Results
The results show that CoT usefulness varies by task, while pre-CoT activations often encode the eventual answer and probe-based steering can causally alter it. Steering effects exceed orthogonal baselines, but strong interventions can cause parse failures and reasoning pathologies.
- Task Accuracy: CoT benefits Logical Deduction most, whereas it is less useful and sometimes harmful for Anachronisms.These task differences predict how informative pre-CoT activations will be about the final answer.
- CoT Sensitivity: At or below 20% of answers flipped after CoT removal in 18 of 20 model–dataset pairs, with Gemma 2 9B reaching 52% on Sports Understanding.Substituting misleading CoT caused more task-dependent flips, exceeding 50% on Anachronisms for every model.
- Pre-CoT Probes: Pre-CoT probes generally performed strongly except on Logical Deduction, where answer computation appears to occur during CoT.Average probe scores were anticorrelated with CoT usefulness across tasks.
- Answer Steering: At large steering strengths, parse failures increased, causing many coefficient sweeps to terminate early.The authors interpret this pattern as consistent with off-manifold degeneration.
- Answer Steering: Steering along probe directions flipped answers more effectively than equally sized orthogonal interventions across all model–dataset pairs.The probe–baseline difference was especially pronounced for Qwen 2.5 7B and Gemma 2 9B, partly because their baselines were less effective.
- Reasoning Pathologies: Incorrectly steered answers exhibited confabulation and non-entailment, while hallucination rates generally increased with steering strength.Confabulation states false premises; non-entailment states true premises but draws unsupported conclusions.
4. Discussion
The discussion interprets pre-CoT probe directions as answer-relevant representations supported by predictive and causal evidence, while identifying ambiguity about the intervention pathway and feature ontology. It also distinguishes this work’s scope from reasoning models and outlines future research on failure cases and feature interpretation.
- Feature interpretation: Probe directions predict final answers before CoT and steering along them flips answers more effectively than arbitrary directions.These findings support interpreting the directions as answer-relevant rather than merely correlational.
- Alternative explanations: Confabulation and non-entailment provide evidence against explaining answer flips solely as general reasoning collapse or CoT-mediated causation.Confabulation aligns false premises with the incorrect conclusion, while non-entailment changes conclusions despite largely correct premises.
- Limitations: Because steering is applied through final-answer generation, its effects may reflect direct token biasing rather than editing a pre-CoT belief that propagates through reasoning.Restricting the intervention to the CoT region would be needed to disentangle these pathways.
- Feature interpretation: The probes may recover task-relevant concepts such as plausibility, validity, and temporal consistency rather than explicit answer-label features.Logit-lens results reportedly identify general concepts predictive of answers, such as “impossible” for anachronisms.
- Reasoning models: The experiments focus on instruction-tuned models, while one reasoning-model study found some answer decoding but little success steering answers.The authors therefore leave the faithfulness–usefulness trade-off in reasoning models open.
- Future work: Future work could test post-hoc reasoning in reasoning models, adapt steering to mitigate it, and study failures linked to memorization, miscalibration, or generalization error.Comparisons with Sparse Autoencoder features are also proposed as a way to clarify whether probes represent interpretable features.
5. Related Work
Related work situates this paper among mechanistic studies of CoT interpretability and prompt-level studies of CoT faithfulness. Prior research examines reasoning behaviors, intermediate-answer monitoring, implicit rationalization, causal prompt interventions, and whether models reveal misleading hints.
- CoT interpretability: Prior CoT interpretability work studies behaviors such as example testing, uncertainty estimation, backtracking, intermediate-answer correctness, early stopping, and circuit-level mechanisms.
- Research landscape: This literature combines behavioral prompt interventions with mechanistic analyses of internal directions and features.
- CoT faithfulness: CoT faithfulness research includes implicit post-hoc rationalization, causal prompt models, hint-based evaluations, and interventions that perturb written reasoning.These studies motivate examining whether verbalized reasoning reflects the model’s underlying decision process.
6. Conclusion
The paper verifies post-hoc reasoning, then shows that pre-CoT answer representations are linearly decodable and causally steerable. Steering toward incorrect answers produces confabulation and non-entailment, while the probe direction plausibly represents task-relevant concepts that determine the answer in context.
- The study verifies that models commit to final answers before generating chain of thought on the selected models and datasets.
- The final answer is linearly decodable from pre-CoT residual-stream activations using difference-of-means probes.
- Steering generations along the probe direction causes models to change their answers, supporting a causal influence beyond prediction.
- Incorrect-answer steering produces confabulation and non-entailment, alongside considerable hallucination where premises and conclusions both fail.
- The probe direction plausibly recovers a causal representation of the pre-committed answer through task-relevant concepts activated in context.
B.1. Ellipses
The ellipses intervention removes generated chain of thought while preserving the demonstrations and response format. A separate GPT-5-based procedure generates minimally edited incorrect reasoning traces and classifies reasoning failures with structured judgments.
- The ellipses intervention replaces each correct chain of thought with “...” before the final-answer prompt to test whether the answer changes.
- The intervention samples 50 correct generations per model–dataset pair and preserves the demonstration format while removing generated reasoning.
- Unlike the no-CoT experiment, ellipses retains CoT in demonstrations and the original generation template, so its results are not directly comparable.
- GPT-5 extracts each model response’s CoT and minimally edits it to imply the opposite answer while preserving style and length.
- GPT-5-mini judges whether steered reasoning contains false premises and whether its conclusion follows, using structured outputs and worked examples.
C.2. Disaggregated Classification Results
The disaggregated results classify successful steering examples from Syes and Sno, with consistency checked across repeated Judge runs. Only a small fraction of traces are classified as sound, and some incorrect reasoning is mistakenly judged correct.
- Figures 5 and 6 report CoT classifications for successfully steered examples from Syes and Sno, respectively.
- The study measures classification consistency by reclassifying 200 sampled input-output pairs as Run 2 and comparing them with Run 1.
- The consistency analysis separately evaluates false-premise labels, entailment labels, final labels, and overall classification agreement.
- 2.5% in Run 1 and 1.0% in Run 2 of CoTs are classified as sound, indicating occasional incorrect reasoning judgments.
C.4. CoT Classification Examples
The examples show how incorrect steering produces confabulation, hallucination, and non-entailment. These cases include false premises, incoherent explanations, and conclusions that contradict or exceed the stated reasoning.
- Confabulation: Confabulation can reverse a stated relation, then derive the steered answer from the resulting false premise.
- The examples distinguish confabulation, where false premises may support the conclusion, from non-entailment, where stated premises fail to support it.
- Confabulation: Confabulation can also introduce nonsensical claims and circular reasoning, leaving the conclusion unsupported.
- Non-Entailment: Non-entailment occurs when the model accurately states the premises but concludes the opposite of what they support.
- Non-Entailment: A second non-entailment example correctly derives that a sports car is oldest, then incorrectly calls it newest.
D. CoT Sensitivity Results
CoT interventions show that answers usually persist when the rationale is removed but can be overridden by misleading reasoning, consistent with answers formed before CoT yet held defeasibly.
- Answer flip rates under omission were at or below 20% in 18 of 20 model–dataset pairs.The exceptions were Sports Understanding for Gemma 2 9B at 52% and Qwen 2.5 1.5B at 32%.
- Under substitution with an incorrect CoT, flip rates were highest on Anachronisms, moderate on Logical Deduction and Sports Understanding, and lowest on Social Chemistry.
- Omission suggests answers rarely depend on rationale presence, whereas substitution shows contrary reasoning can often override them.Together, both intervention patterns are consistent with answers formed before CoT but held defeasibly.
- Probe AUC across layers measures the linear decodability of final answers from pre-CoT activations, with higher AUC indicating stronger decodability.
F. Reasoning Model Results
For GPT-OSS 20B, pre-CoT answer representations are weaker than in instruction-tuned models on most datasets, and steering generally fails to exceed the orthogonal baseline.
- GPT-OSS 20B probe AUC was considerably lower than for non-reasoning, instruction-tuned models on all datasets except Anachronisms.On Anachronisms, its probe AUC exceeded 0.9.
- Answer flip rates for GPT-OSS 20B were negligible against the orthogonal baseline under steering.
- The GPT-OSS 20B results were recorded using the same pre-CoT probing and steering methodology.
- The authors hypothesize that large reasoning models determine final answers largely within CoT, unlike instruction-tuned models.
- Steering remained ineffective on Anachronisms despite high AUC, suggesting weaker causal dependence on the pre-committed answer direction.The authors speculate that final answers rely more on CoT tokens, potentially reflecting reinforcement-learning optimization pressure on those tokens.
G. Steering Results with Parse Failure Rate
Steering results are evaluated alongside parse failures, while unembedding probe directions reveals top alphabetic tokens intended to make answer features more interpretable.
- Figure 10 reports steering flip rates together with parse-failure rates across the α sweep for every model–dataset pair.Parse-failure rate is the proportion of generations that could not be parsed.
- The task probe and its negation are unembedded to compute logits, with “+” denoting the probe direction and “−” its opposite-class direction.
- Table 9 lists the five highest-logit tokens for each model–dataset pair after filtering out tokens with non-alphabetical characters.
- Alphabetical-token filtering is intended to increase the chance that tokens have interpretable semantic content and are common in English.Some resulting tokens remain incomprehensible or appear to derive from non-English languages or code.