PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo

arXiv:2603.05488v4cs.CLcs.AIcs.LG

TL;DR

The paper examines whether emitted chain-of-thought faithfully reflects models’ internal reasoning, comparing activation probes, forced answering, and CoT monitoring across easier and harder tasks. It finds performative reasoning on easier tasks but more aligned incremental reasoning on harder tasks, while probe-guided early exit can reduce token use.

  • Problem

    Chain-of-thought traces are not necessarily faithful to models’ internal reasoning, limiting their reliability for safety and interpretability applications.

  • Method

    The study compares attention probes and forced answering with a CoT monitor across MMLU-Redux 2.0 and GPQA-Diamond to compare internal answer predictions with verbalized commitments.

  • Results

    Easier tasks show performative reasoning, whereas harder tasks show probe confidence and CoT monitor accuracy increasing together as answer information emerges through the trace.

  • Takeaways & Limitations

    Attention probing can detect mismatches between internal confidence and emitted reasoning and support calibrated early exits that save tokens with minimal accuracy loss on MMLU-Redux.

  • Takeaways & Limitations

    The relationship between verbalized inflection points and internal confidence shifts is sensitive to window size and confidence threshold and varies across responses, datasets, and models.

Abstract

from arXiv · show

We provide evidence of performative chain-of-thought (CoT) in reasoning models, where a model becomes strongly confident in its final answer, but continues generating tokens without revealing its internal belief. Our analysis compares activation probing, early forced answering, and a CoT monitor across two large models (DeepSeek-R1 671B & GPT-OSS 120B) and find task difficulty-specific differences: The model's final answer is decodable from activations far earlier in CoT than a monitor is able to say, especially for easy recall-based MMLU questions. We contrast this with genuine reasoning in difficult multihop GPQA-Diamond questions. Despite this, inflection points (e.g., backtracking, 'aha' moments) occur almost exclusively in responses where probes show large belief shifts, suggesting these behaviors track genuine uncertainty rather than learned "reasoning theater." Finally, probe-guided early exit reduces tokens by up to 80% on MMLU and 30% on GPQA-Diamond with similar accuracy, positioning attention probing as an efficient tool for detecting performative reasoning and enabling adaptive computation.

1. Introduction

The paper studies when reasoning models’ chain-of-thought diverges from their internal beliefs and proposes activation probes to detect that divergence and support efficient early exit.

  • Motivation: Chain-of-thought can continue as step-by-step reasoning after a model has internally committed to an answer, creating performative reasoning.The paper frames this as a mismatch between external deliberation and internally committed confidence.
  • Approach: Activation probes, forced answering, and CoT monitoring compare when an answer is internally decodable with when it is verbalized.The methods use activations, truncated-trace prompting, and response-text evaluation, respectively.
  • Findings: Performative reasoning is more common on MMLU-Redux, whereas GPQA-Diamond more often exhibits genuine reasoning through gradually emerging answer information.The difficulty-dependent split distinguishes easier tasks from harder tasks requiring more test-time computation.
  • Findings: Backtracking, realizations, and reconsiderations usually coincide with genuine internal belief updates rather than merely performed reasoning.These inflection points occur primarily in responses showing substantial shifts in internal confidence.
  • Implication: Calibrated attention probes reduce generated tokens by 80% on MMLU-Redux and 30% on GPQA-Diamond with comparable performance.The savings support confidence-based early exit during generation.

2. Related Work

Prior work documents unfaithful chain-of-thought and motivates methods that track information flow and latent beliefs across long reasoning traces.

  • Faithfulness: Models can produce plausible rationales that omit the true causes of their decisions, including under natural and explicitly intervened settings.This makes textual CoT a fragile basis for safety monitoring.
  • Open problem: Predicting when CoT will be unfaithful remains difficult because conventional interpretability tools poorly track information flow across long sequences.Single-token or layer-wise analyses scale poorly to long contexts.
  • Open problem: Black-box causal analysis can be prohibitively expensive for very large models and long CoTs.The cost is especially relevant for models with hundreds of billions of parameters.
  • Activation probing: Activation probing extracts latent information with lightweight classifiers, while attention pooling aggregates representations across a reasoning trace.This paper applies attention pooling to track answer information and probe calibration for early exit.
  • Reasoning-model interpretability: Reasoning-model interpretability also examines causal reasoning steps and how reasoning models repurpose pretrained knowledge.The paper complements these lines of work by studying answer information throughout generation.
  • Cooperative communication: The paper frames CoT monitoring through cooperative communication, contrasting informative listeners with reasoning models that may not disclose internal deliberation.This perspective motivates caution about treating emitted reasoning as fully informative.

3. Methods

The study compares activation probes, forced answering, and CoT monitoring across two benchmarks to track when final answers become decodable during reasoning.

  • Methods: The three methods infer final answers from layer activations, truncated-trace prompting, or response text, providing different levels of privileged access.Together they enable comparison of internal answer decoding with verbalized commitment.
  • Models: The analysis focuses on DeepSeek-R1 671B and GPT-OSS 120B, with distilled DeepSeek-R1 models added to examine model-size effects.The distilled models span 1.5B, 7B, 14B, and 32B parameters.
  • Datasets: MMLU-Redux contains 5,700 questions across 57 domains, while GPQA-Diamond contains 198 graduate-level biology, chemistry, and physics questions.Both are four-choice benchmarks, enabling classification probes over answer options.
  • Data collection: Responses are collected under the original inference settings, split into reasoning paragraphs, and represented with activations from every layer and token.Step-level analysis uses text between <think> and </think> tags.
  • Attention probes: Attention probes pool hidden states learned from random prefixes and output an answer-choice distribution at every generation position.One probe is trained per layer and evaluated throughout the reasoning trace.
  • Forced answering: Forced answering truncates a reasoning trace and prompts the same model to predict an answer choice from the preceding context.Softmax probabilities over the four choices quantify the prediction.
  • CoT monitoring: The CoT monitor predicts a choice or N/A from partial reasoning, and separately identifies backtracking, realizations, and reconsiderations.The N/A option discourages solving the question using the monitor’s own knowledge.

4. Attention Probe Results

Attention probes succeed where traditional linear probes fail, extracting final-answer information from pooled activations across long reasoning traces.

  • Evaluation: Fine-tuning on 20 questions provides negligible improvement over direct transfer in overall probe accuracy.The evaluation uses held-out MMLU and GPQA-D sets after training on MMLU questions.
  • Probe comparison: Traditional linear probes perform near chance, whereas attention probes decode final answers by weighting relevant representations across the sequence.The results support pooling across tokens because belief information updates at specific positions.

5. Easier Tasks Exhibit More Performative Reasoning

The gap between internal answer decoding and verbalized conclusions is larger for easier MMLU questions than for difficult GPQA-D questions, while model size also affects when answers become decodable.

  • Dataset-dependent trends: Probes and forced answering often decode answers from the beginning of reasoning, while the CoT monitor identifies them later; this gap is larger for MMLU than GPQA-D.The comparison treats a larger gap as evidence that the model has not revealed its current belief in the CoT.
  • Dataset-dependent trends: GPQA-D accuracy rises gradually for all three methods, with probes and forced answering starting lower but remaining ahead, reflecting the need for CoT on harder questions.MMLU questions mainly require recall, whereas GPQA-D requires multihop reasoning.
  • Dataset-dependent trends: MMLU has a performativity measure of 0.417 in R1, versus 0.012 for GPQA-D, indicating much more performative reasoning on MMLU.A rate near 0 means probe or forced-answer accuracy increases similarly to CoT-monitor accuracy as reasoning proceeds.
  • Model-size-dependent trends: As model size increases, MMLU final answers become decodable earlier; the 1.5B model rises sharply only in the second half, while 7B–32B probes remain broadly similar.All models eventually converge toward perfect probe accuracy during reasoning.
  • Model-size-dependent trends: Smaller models show a smaller probe-to-monitor gap, while the 671B model’s gap drops rapidly as the CoT catches up to internal beliefs.The analysis excludes GPQA-D for smaller models because answer-choice collapse confounds early decoding.
  • Model-size-dependent trends: Highly confident CoTs contain dramatically fewer inflection points per step, and smaller models need more test-time compute to solve the same problem.These observations connect confidence patterns with the amount of reasoning generated across model sizes.

6. Inflection Points Suggest Faithful Reasoning

Inflection points are more common when probes indicate lower confidence, supporting their association with genuine belief updates rather than uniformly performative reasoning. However, local timing between probe shifts and verbalized inflections is inconsistent across settings.

  • Inflection points and confidence: Inflections appear twice as often when the model is not internally confident, with reconsiderations occurring in 3.2% of other steps versus 1.4% of high-confidence steps.The result suggests inflections more often reflect computation resolving or increasing uncertainty.
  • Probe-shift timing: The analysis tests whether 20% probe-confidence shifts predict inflections within a 10-step window, in either temporal direction.A probe shift is defined using the highest-probability answer between consecutive steps.
  • Probe-shift timing: For DeepSeek-R1 on MMLU, reconsiderations occur twice as often after probe shifts as in windows without shifts, but this relationship does not hold for GPQA-D.Across models and datasets, the results are mixed rather than establishing a simple causal pattern.
  • Probe-shift timing: The relationship between verbalized inflections and internal confidence shifts changes with dataset, model, window size, and confidence threshold.The authors conclude that further work is needed to connect internal belief updates to their verbalization timing.
  • Case studies: A recall-based MMLU example keeps probe confidence above 90% while the model reasons separately over all four options without changing its internal confidence.This contrasts with a Python-syntax example where a probe shift accompanies correction after distinguishing Python 2 from Python 3 tuple behavior.

7. Attention Probes for Early Exit

Attention probes can track answer beliefs during reasoning and support calibrated early exits. They predict answers from activations before textual commitment, including contrasting performative and genuine MMLU examples.

  • Probe-based early exit: Attention probes decode final-answer information from internal activations before the model finishes verbalizing its chain of thought.The probes are used to signal when reasoning has effectively finished and to predict the answer choice.
  • Performative versus genuine reasoning: In the performative example, the model internally favors an answer from the first reasoning step while its text says it needs to recall the term.The example concerns the history question about the “New World Order.”
  • Performative versus genuine reasoning: In the genuine MMLU example, probe confidence rises during option analysis and changes after the model identifies a Python-version distinction.The correction appears in both the chain of thought and probe predictions.
  • Calibration: The probes are highly calibrated, closely tracking forced-answer accuracy and transferring to GPQA-Diamond.This calibration supports using probe confidence to choose an exit threshold.
  • Early-exit results: At 95% confidence on MMLU, early exit retains 97% of original performance while saving 68% of tokens; at 80% confidence on GPQA, it retains 97% while saving 50%.The probe directly predicts the answer rather than relying on a chain-of-thought monitor to infer it.

8. Discussion

The discussion frames performative CoT as task-dependent and cautions that emitted reasoning may not faithfully reveal internal commitments. It also identifies activation probes as a route to monitoring and adaptive computation.

  • Performative and genuine reasoning: On MMLU, answer information can be decoded before textual commitment, whereas harder tasks show answer information emerging more gradually with the reasoning trace.This supports treating performative CoT as an interaction between model and task difficulty rather than a uniform property.
  • Inflection points: Backtracking and sudden realizations typically reflect genuine updates to internal belief, although this correspondence is not universal.The paper links these inflection points to question-level uncertainty.
  • Monitoring implications: If internal answer information precedes the trace, text-only monitors may lag behind or misrepresent commitments, uncertainty, and decision rationale.Emitted reasoning can be more informative when answer information emerges through incremental computation.
  • Adaptive computation: Attention-probe calibration can support early exit with minimal accuracy loss and large token savings, especially on MMLU-Redux.This is relevant when reasoning models generate long traces despite having already reached an answer.
  • Interpretive framework: The paper interprets faithfulness through cooperative communication and notes that faithfulness is multifaceted and requires further study.The framing is connected to outcome-oriented training and prior findings on hints and posthoc rationalization.

9. Conclusion

The paper develops attention probes to compare internal answer states with chain-of-thought monitors, identify task-dependent performative reasoning, and enable calibrated early exits. Its experiments use MMLU-Redux and GPQA-Diamond across large reasoning models.

  • Contributions: The paper compares black-box chain-of-thought monitors with white-box attention probes for predicting a model’s eventual answer.It also uses probes for calibrated early exits intended to reduce unnecessary reasoning tokens.
  • Experimental setup: The experiments collect responses on MMLU-Redux 2.0 and GPQA-Diamond under the original authors’ reported inference settings.The datasets provide the evaluation basis for cross-task probe and monitoring comparisons.
  • Experimental setup: DeepSeek-R1 671B and GPT-OSS 120B are evaluated with model-specific providers and reasoning settings, while both use the same question-and-choice prompting structure.The supplied setup includes temperature, top-p, and maximum-token parameters for each model.
  • Forced answering: Forced answering collects one-token answer-choice logits after injecting prompts that elicit one of the four letter choices.The selected prompts produced the highest rate of immediate letter answers among the tested prompts.
  • Chain-of-thought monitoring: The chain-of-thought monitor infers the model’s likely eventual choice from a partial reasoning trace without solving the question itself.It is instructed to return A, B, C, D, or N/A based on directional cues in the trace.
  • Inflection analysis: Inflection-point analysis flags backtracking, realizations, and reconsiderations while instructing the evaluator to exclude ordinary step-by-step reasoning.The output records whether an inflection exists, its type, and what changed.
  • Probe training: Probe training uses macro accuracy, averaging question-level accuracy across positions, with hyperparameters selected through a grid search.The appendix tables report sweeps and results for DeepSeek-R1 and distilled models.

C. Attention Probe vs. Baselines

Attention probes substantially outperform linear and random-label baselines and are evaluated for transfer from MMLU-Redux to GPQA-Diamond. The comparison tests whether probes read existing activation information rather than solve the task independently.

  • Baseline comparison: The baseline comparison includes linear probes and probes trained on random labels, with results aggregated by each probe’s best layer accuracy.The corresponding results are summarized in Table 5.
  • Baseline comparison: Attention probes achieve 87.98% test accuracy versus 31.85% for linear probes under the same training hyperparameters.The linear probe performs only slightly better than chance and cannot accurately predict the model’s final answer.
  • Random-label control: Random-label probes cannot predict the model’s final answer better than chance, supporting the interpretation that attention probes read information already present in the residual stream.This comparison is intended to rule out the probe independently learning the task.
  • Cross-dataset transfer: MMLU-trained probes are tested on GPQA-Diamond using direct transfer and fine-tuning on 19 of 198 questions.The study compares best-layer test accuracy for both transfer settings.

E.1. DeepSeek-R1 Full Probe Accuracy Heatmaps

Across models and tasks, probes decode final answers earlier on MMLU than on GPQA-D, while probe-guided early exit can preserve accuracy with fewer tokens.

  • DeepSeek-R1 Full Probe Accuracy Heatmaps: Final-answer information becomes linearly decodable around layer 20 for DeepSeek-R1 on MMLU, before reasoning begins.Successful probes outperform the 28% chance baseline while the question is still being asked.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: GPQA-D produces a more gradual increase in probe decodability than MMLU, consistent with greater test-time computation for difficult reasoning.The same contrast appears for GPT-OSS, whose GPQA-D probe accuracy also rises gradually over sequence length.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: GPT-OSS likewise decodes MMLU answers before reasoning, with some success even in early layers and shorter reasoning traces than DeepSeek-R1.Its later layers recover final-answer information well before reasoning starts.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: Distilled models generally decode MMLU answers earlier as model size increases, although 14B and 32B probes show layerwise instability.The 14B results may reflect training instability rather than a reliable layerwise signal.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: Probes generally outperform forced answering and CoT monitoring, while forced-answer accuracy can fall later in reasoning for larger distilled models.For 14B and 32B models, probe accuracy remains more stable as forced answering declines in the second half.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: On GPQA-D, both probes and forced answering are often wrong early, whereas on MMLU their disagreement patterns favor forced answering initially for DeepSeek-R1 and probes for GPT-OSS.The GPT-OSS GPQA-D comparison may reflect poor probe transfer from MMLU.
  • DeepSeek-R1 Full Probe Accuracy Heatmaps: GPT-OSS MMLU probes are well calibrated, and 90%-confidence early exit saves most tokens with only a marginal performance drop.Transferred GPQA-D probes are more overconfident; another calibration plot shows less token saving per accuracy drop.

I. Additional Inflection Analysis

Additional analyses link inflection points to probe confidence shifts in some MMLU settings, while high-confidence traces consistently contain fewer inflections; the relationship is weaker or inconsistent on GPQA-D and GPT-OSS.

  • Additional Inflection Analysis: High-confidence GPT-OSS MMLU traces contain fewer inflection points per step than other traces, consistent with inflections reflecting genuine uncertainty.The reported rates are 0.053 versus 0.096 inflections per step.
  • Additional Inflection Analysis: On MMLU, reconsiderations are nearly twice as likely after a 20% probe confidence shift as without one.The reported probabilities are 59% after a shift versus 35% without one; backtracks and realizations show a similar pattern.
  • Additional Inflection Analysis: On GPQA-Diamond, inflections can be more likely without probe shifts, unlike the MMLU reconsideration pattern.This task-dependent reversal limits a uniform interpretation of probe shifts as inflection predictors.
  • Additional Inflection Analysis: The temporal analysis measures whether probe shifts predict later inflections and whether inflections predict subsequent probe shifts across thresholds and windows.It compares conditional-probability differences for both directions of the relationship.
  • Additional Inflection Analysis: For DeepSeek-R1 on MMLU, inflections are about twice as likely after high-threshold probe shifts at moderate two-to-ten-step windows, while reverse correlations vary.GPT-OSS shows a different and inconsistent ordering pattern, especially at smaller windows.
Loading 2603.05488v4…