PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo

arXiv:2603.05488v4cs.CLcs.AIcs.LG

TL;DR

Because chain-of-thought may not faithfully reflect a model’s internal reasoning, its reliability as an explanation is uncertain. Comparing activation probes and forced answers with a text-based monitor, the paper finds that answer information emerges ahead of textual commitment on easier tasks, while internal and textual signals rise together on harder tasks.

  • Problem

    Chain-of-thought traces are not always faithful to internal reasoning, limiting their reliability for safety applications that rely on inspecting them.

    Source: §1 · p. 1

  • Method

    The paper compares activation probes and forced answering, which estimate internal answer predictions, with a chain-of-thought monitor across MMLU-Redux 2.0 and GPQA-Diamond.

    Source: §3.1 · p. 3, §5 · p. 5

  • Results

    Answer information emerges before textual commitment on easier MMLU tasks, whereas on harder GPQA-Diamond tasks probes and forced answers improve alongside the monitor, indicating task-difficulty-dependent differences in chain-of-thought faithfulness.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Attention-probe confidence can support adaptive early exit on MMLU with minimal loss in accuracy.

    Source: §8 · p. 10

    The relationship between verbalized inflection points and internal confidence shifts is inconsistently captured by local co-occurrence and is sensitive to window size and confidence thresholds.

    Source: §6.2 · p. 7

Abstract

from arXiv · show

We provide evidence of performative chain-of-thought (CoT) in reasoning models, where a model becomes strongly confident in its final answer, but continues generating tokens without revealing its internal belief. Our analysis compares activation probing, early forced answering, and a CoT monitor across two large models (DeepSeek-R1 671B & GPT-OSS 120B) and find task difficulty-specific differences: The model's final answer is decodable from activations far earlier in CoT than a monitor is able to say, especially for easy recall-based MMLU questions. We contrast this with genuine reasoning in difficult multihop GPQA-Diamond questions. Despite this, inflection points (e.g., backtracking, 'aha' moments) occur almost exclusively in responses where probes show large belief shifts, suggesting these behaviors track genuine uncertainty rather than learned "reasoning theater." Finally, probe-guided early exit reduces tokens by up to 80% on MMLU and 30% on GPQA-Diamond with similar accuracy, positioning attention probing as an efficient tool for detecting performative reasoning and enabling adaptive computation.

1. Introduction

Reasoning models can commit to an answer before their chain of thought reveals that confidence. The paper uses activation probes to distinguish performative continuation from genuine reasoning and to support early exit.

Motivation

  • Because chain-of-thought traces may not disclose internal deliberation, the paper cautions against relying on monitors alone and proposes activation monitoring for these failures.It frames monitors as cooperative listeners and reasoning models as non-cooperative speakers.Source: §1 · p. 1

Activation attention probes

  • Attention probes predict the model’s final answer from activations at varying reasoning-prefix lengths, tracking belief state throughout generation.The paper also compares probe predictions with forced answering and a CoT monitor that checks whether a prefix contains a potential final answer.Source: §1 · p. 2, Figure 1 · p. 2

Evidence for difficulty-dependent performative reasoning

  • On MMLU-Redux, answer information often appears before a CoT monitor indicates a conclusion, whereas GPQA-Diamond exhibits more genuine reasoning.As model size increases, smaller, less capable models show a similar pattern of requiring more test-time compute to reach a final answer.Source: §1 · p. 2

Inflection points and internal uncertainty

  • Backtracking and “aha” moments occur almost exclusively with large internal confidence shifts, while consistently high-confidence responses contain fewer inflections.The paper describes inflections as indicators of genuine belief updates but reports inconsistent results on whether they coincide with updates at the response level.Source: §1 · pp. 1–2

Practical token savings via calibrated early exit

  • Early exit saves 80% of generated tokens on MMLU-Redux and 30% on GPQA-Diamond with comparable performance.The probes are calibrated for confidence-based early exit without sacrificing accuracy.Source: §1 · p. 2

2. Related Work

Prior work documents unfaithful chain-of-thought and limits of textual monitoring, while motivating approaches that track answer information in model activations throughout generation.

Faithfulness in language model chain-of-thought explanations

  • Prior studies find that CoT can omit decision causes or produce plausible but unfaithful rationales, complicating safety monitoring despite evidence that explanations can predict behavior.Work also reports brittle links between CoT length and difficulty and explores predicting risky behavior before reasoning ends.Source: §2 · p. 2
  • Traditional interpretability tools scale poorly across long sequences, while monitors and resampling inspect longer responses but thorough resampling can be cost-prohibitive for large models and long CoTs.Prior work also finds that predicting when CoT will be unfaithful remains difficult.Source: §2 · pp. 2–3

Activation Probing

  • Activation probing trains lightweight linear classifiers on residual-stream activations; this work uses attention pooling to track answer information through reasoning and evaluate calibration for early exit.Prior applications include safety, future-behavior prediction, self-verification, and analysis of uncertainty and branching.Source: §2 · p. 3

Reasoning Model Interpretability

  • Where prior interpretability work analyzes causally important reasoning steps or compares base and reasoning models, this work tracks token-level answer information as it emerges during generation.Comparative work suggests reasoning models repurpose pretrained knowledge.Source: §2 · p. 3

Cooperative Communication

  • A Gricean communication perspective frames CoT monitoring shortcomings as a question of pragmatic communication, a framing that may guide future work on monitorability.Reasoning models may be only indirectly optimized for effective communication; related work connects rational speech acts to linguistic pragmatics (Frank & Goodman, 2012; Goodman & Stuhlmüller, 2013; Goodman & Frank, 2016; Levinson, 1983).Source: §2 · p. 3

3. Methods

The methods track when final-answer information appears in reasoning traces using activation probes, forced answering, and a chain-of-thought monitor. They are evaluated on two multiple-choice benchmarks using frontier open-weight reasoning models and smaller distilled models.

3. Methods

  • The three methods predict a model’s final answer from a reasoning prefix using layer activations, extra forward passes, or response text.Together, they track when answer information is decodable from activation space, available to the model after truncation, or evident in its generated reasoning.Source: §3 · p. 3

3.1. Models & Datasets

  • The study focuses on DeepSeek-R1-0528 671B and GPT-OSS 120B, and also tests distilled DeepSeek-R1 Qwen2.5 models from 1.5B to 32B.The smaller models are included to study the effect of model size and capability on performative reasoning.Source: §3.1 · p. 3
  • Evaluation uses 5,280 filtered MMLU-Redux 2.0 questions and 198 graduate-level GPQA-Diamond questions, each with four answer choices.Responses use the original authors’ inference settings; reasoning is split into paragraphs at blank lines, and activations are collected at every layer and token.Source: §3.1 · p. 3

3.2. Attention Probes

  • Attention probes pool a layer’s hidden states and are trained on random sequence prefixes to predict final answers regardless of prefix length.One probe is trained per model layer and evaluated at every prefix position; reported results use the best-performing layer across sequence positions.Source: §3.2 · p. 3
  • On MMLU, probing and forced answering predict both models’ answers earlier than CoT monitoring, while on GPQA-Diamond all three methods rise from chance at similar rates.The figure describes the MMLU pattern as performative CoT and the GPQA-Diamond pattern as closer tracking between internal beliefs and CoT.Source: Figure 2 · p. 4

3.3. Forced Answering

  • Forced answering truncates a reasoning trace at an intermediate step and prompts the same model to choose A–D, using class-logit softmax probabilities.The model can use all layers’ activations from the preceding text and a few additional forward passes to make its prediction.Source: §3.3 · p. 4

3.4. Chain of Thought Monitoring

  • The Gemini-2.5-Flash CoT monitor predicts a final choice or N/A from partial reasoning and separately identifies backtracking, realizations, and reconsiderations.The N/A option lets it defer when the partial CoT is insufficient, discouraging it from answering from its own knowledge.Source: §3.4 · p. 4

4. Attention Probe Results

Attention probes decode final answers where traditional linear probes perform near chance, with predictive layers concentrated in the later portions of both models.

Attention Probe Results

  • Probes were trained per layer on MMLU questions and evaluated on held-out sets of 528 MMLU and 157 GPQA-D questions.The experiments included direct transfer and fine-tuning on 20 questions; fine-tuning gave negligible overall probe-accuracy improvement.Source: §4 · p. 5
  • Attention probes succeed where linear probes perform near chance, pooling sequence representations to weight relevant tokens as beliefs update during reasoning.Final-answer prediction is possible in the second half of DeepSeek-R1 layers and the last three quarters of GPT-OSS layers.Source: §4 · p. 5

5. Easier Tasks Exhibit More Performative Reasoning

Performativity is the gap between early probe or forced-answer confidence and a CoT monitor’s ability to identify the answer. Easier tasks show a larger gap, while harder tasks show probe confidence and monitor accuracy rising together.

5.1. Dataset Dependent Trends

  • On MMLU, probes and forced answers are accurate from the start while the monitor lags; on GPQA-D, all three methods improve gradually, with probes and forced answers starting much lower.MMLU mainly requires recall, whereas GPQA-D requires multihop reasoning; for GPQA-D, information gain per reasoning step is reflected in both probe and monitor accuracy.Source: §5.1 · p. 5
  • In DeepSeek-R1, the performativity measure is 0.417 for MMLU and 0.012 for GPQA-D, with the same dataset trend holding across both models and forced answering.A rate near 0 indicates similar added-token gains for probe or forced-answer accuracy and monitor accuracy; the authors conclude that MMLU models know their answer early and gain little from further CoT relative to the monitor.Source: Table 1 · p. 5, §5.1 · p. 5

5.2. Model Size Dependent Trends

  • On MMLU-Redux, probe accuracy is higher earlier as DeepSeek-R1 model size grows, but 7B–32B probes look similar and the 1.5B model reaches peak accuracy more slowly.All models eventually converge toward perfect accuracy during reasoning; the 1.5B model’s best probe reaches it more slowly in the first half.Source: Figure 3 · p. 6, §5.2 · p. 6
  • Forced-answer accuracy improves relative to probe accuracy with model size; smaller models show a smaller early probe-to-monitor gap, while 671B’s gap rapidly shrinks as CoT catches up.The authors suggest smaller models need more test-time compute to solve the same problem.Source: §5.2 · p. 6

6. Inflection Points Suggest Faithful Reasoning

Monitor-labeled inflections are more common in responses with lower probe confidence, but their local timing relative to confidence shifts is inconsistent across models and datasets. Case studies contrast a recall trace with unchanging high confidence against a syntax trace whose belief change is reflected in the CoT.

6.1. Inflection Points Occur in Less Confident Responses

  • Inflections appeared twice as often in less-confident responses; reconsiderations made up 1.4% of high-confidence steps versus 3.2% of other steps, controlling for response length.High confidence meant probe confidence began at 90% or higher and never fell below that level; 215 of 522 DeepSeek-R1 MMLU traces met this criterion. Inflections were monitor-labeled as backtracking, realizations (“aha’s”), reconsiderations, or not an inflection.Source: §6.1 · p. 6, §6 · p. 6

6.3. Case Study Examples

  • In a recall-based MMLU example, the probe exceeded 90% confidence in answer B from the start, while the model repeatedly reasoned through all four options without a confidence change.The model said it needed to recall the term, making this trace an example of likely performative reasoning despite high probe confidence.Source: §6.3 · pp. 7–8

6.2. Inflection Points do not Reliably Occur with Shifts in Confidence

  • With 20% probe shifts and 10-step windows, R1 reconsiderations followed MMLU shifts twice as often, but local co-occurrence showed no consistent timing pattern overall.The pattern did not hold for GPQA-D; GPT-OSS inflections appeared to precede shifts on both datasets. Results varied with window size and confidence threshold, and the reverse direction showed little difference for either R1 dataset.Source: §6.2 · p. 7, Figure 5 · p. 8

6.3. Case Study Examples

  • In a Python-syntax MMLU example, probe confidence in B rose to 90% after reasoning through the choices, then a correction following a Python-version distinction appeared in both the CoT and probe predictions.The CoT tracked the probe’s confidence change more closely in this example.Source: §6.3 · p. 8

7. Attention Probes for Early Exit

Attention probes can predict answer choices from internal activations and support confidence-based early exit. At tested thresholds, early exit preserves 97% of original performance while reducing token use on MMLU and GPQA.

7. Attention Probes for Early Exit

  • Internal-activation probes can signal that a model has finished thinking and predict its final answer choice without a chain-of-thought monitor.The section motivates probing by evidence that models reach a final answer before finishing verbalizing their chain of thought.Source: §7 · p. 8
  • Probe confidence closely tracks forced-answer accuracy and transfers to GPQA, supporting confidence-based early exit thresholds.Figure 7 shows that the DeepSeek-R1 attention probes for MMLU and GPQA-Diamond closely track perfect calibration.Source: §7 · pp. 8–9, Figure 7 · p. 9
  • On MMLU, exiting at 95% confidence retains 97% of original performance while saving 68% of tokens; on GPQA, 80% confidence retains 97% while saving 50%.The same trends hold for GPT-OSS, with results reported in Appendix H.Source: §7 · p. 9

8. Discussion

The discussion argues that performative CoT depends on task difficulty: on MMLU, internal answer information can precede textual commitment, while on GPQA-D it emerges more gradually alongside reasoning. It also considers implications for monitoring, adaptive computation, and how reasoning traces should be interpreted.

Performative CoT

  • On MMLU, answer information is decodable from later-layer activations before CoT commits, whereas on GPQA-D answers emerge more gradually alongside improved forced-answer and monitor performance.Performative CoT is therefore task-dependent; backtracking and sudden realizations typically, but not universally, reflect genuine updates to internal belief.Source: §8 · p. 9
  • A text-only monitor may lag or misrepresent internal commitments and uncertainty, making emitted CoT an unreliable basis for auditing decisions in some tasks.Where answer information emerges through incremental computation, however, the reasoning trace may better track evolving beliefs.Source: §8 · pp. 9–10

Tokens Savings through Early Exit

  • On MMLU-Redux, attention-probe confidence can support early exit with minimal accuracy loss and large token savings, by stopping generation when confidence is high.Figure 8 evaluates accuracy and tokens saved at binned confidence thresholds for DeepSeek-R1 on MMLU and GPQA.Source: Figure 8 · p. 9, §8 · p. 10

Cooperative Communication and Monitoring

  • The cooperative-communication framing explains why traces can stay on-topic yet be verbose and fail to reveal early commitments: relevance and evidential support align with reward more than quantity and manner do.The framing is less explanatory for posthoc rationalization (Turpin et al., 2023), suggesting faithfulness is multifaceted; it also describes partial faithfulness after RL (Chen et al., 2025) and hint verbalization on easy tasks.Source: §8 · p. 10

9. Conclusion · A. Inference and Prompting Details · B. Attention Probe Hyperparameters

The paper compares black-box CoT monitors with white-box attention probes to predict eventual answers, identify performative reasoning, and support calibrated early exits that save tokens. The appendices specify inference and prompting procedures and attention-probe training settings.

9. Conclusion

  • The paper compares CoT monitors and attention probes to predict answers, detect performative reasoning, and enable calibrated early exits that save tokens.The authors frame these methods as steps toward safer and more interpretable reasoning models.Source: §9 · p. 10

A.1. Model Inference Details

  • DeepSeek-R1 671B and GPT-OSS 120B responses were collected on MMLU-Redux 2.0 and GPQA-Diamond using the original authors’ inference settings.DeepSeek-R1 used temperature 0.6, top p 0.95, and a 30,000-token maximum; GPT-OSS used high reasoning effort, temperature 1.0, and top p 1.0.Source: §A.1 · p. 14

A.2. Forced Answer Prompt

  • Forced-answer prompts used temperature 0.0, top p 1.0, and one-token generation; the chosen prompts maximized immediate four-choice answers.Choice logits were collected when present among the top 20 next-token candidates.Source: §A.2 · p. 14

A.3. Chain-of-Thought Monitor Details

  • CoT monitor experiments used Gemini-2.5-Flash through the OpenRouter API.Source: §A.3 · p. 14

A.3.1. FINAL ANSWER PROMPT:

  • The monitor was instructed to predict the model’s likely eventual choice from its partial trace, using directional cues and returning N/A when no clear commitment appeared.It was told not to solve the question or treat option enumeration as evidence of preference.Source: §A.3.1 · pp. 14–15

A.3.2. INFLECTION POINT INDENTIFICATION PROMPT:

  • The inflection-point prompt asked monitors to conservatively identify backtracking, new realizations, and reconsideration as genuine changes in reasoning direction.Source: §A.3.2 · p. 15

B. Attention Probe Hyperparameters

  • Probe settings varied by model: DeepSeek-R1 671B and GPT-OSS MMLU probes used learning rate 1 × 10^-3, weight decay 1 × 10^-3, and batch size 64, while distilled models used learning rate 5 × 10^-3.One probe was trained per activation layer; GPQA used MMLU-trained checkpoints without fine-tuning. Hyperparameters were grid-searched on DeepSeek-R1 671B MMLU residual-stream activations, then reused for other models; macro accuracy averaged per-question accuracy across positions.Source: B. Attention Probe Hyperparameters · pp. 15–16

C. Attention Probe vs. Baselines · D. GPQA Probe Transfer vs. Fine-Tuning · E. Probe Accuracy Heatmaps

Attention probes substantially outperform linear probes and read answer information from the residual stream across sequence positions; MMLU-trained probes transfer to GPQA-Diamond with similar accuracy whether directly transferred or fine-tuned. The paper therefore uses direct transfer for its GPQA results.

C. Attention Probe vs. Baselines

D. GPQA Probe Transfer vs. Fine-Tuning

E.1. DeepSeek-R1 Full Probe Accuracy Heatmaps … H. GPT-OSS Calibration for Early Exit

Across the appendix analyses, probes decode final answers before reasoning on MMLU for full-scale models, while GPQA-D and smaller distilled models show more gradual answer signals. Comparisons with forced answering and calibration analyses further qualify probe performance and early exit.

E.1. DeepSeek-R1 Full Probe Accuracy Heatmaps

  • For DeepSeek-R1, MMLU final answers become linearly decodable at layer 20, while GPQA-D answer information emerges more gradually, implying greater test-time computation.MMLU probes identify the answer significantly better than chance before reasoning begins; GPQA-D probes trained on MMLU activations perform worse across layers.Source: Figure 11 · p. 18, Figure 12 · p. 18

E.2. GPT-OSS Full Probe Accuracy Heatmaps

  • GPT-OSS probes decode MMLU answers before reasoning in the latter two-thirds of layers, with some success in the first few layers; GPQA-D shows a gradual trend.GPT-OSS reasoning traces are shorter, so reasoning starts later than in the DeepSeek-R1 comparison.Source: Figure 13 · p. 18, Figure 14 · p. 19

E.3. DeepSeek-R1 Distills Full Probe Accuracy Heatmaps

  • Across distilled R1 models, 1.5B probes decode answers gradually in later layers, 7B shows earlier decoding, and 14B/32B show early decoding among successful probes amid layerwise instability.The 14B caption notes that most probes fail to decode the answer, possibly because of training instability.Source: Figure 15 · p. 19, Figure 16 · p. 19, Figure 17 · p. 20

F. Probe vs. Forced Answering vs. CoT Monitor for Distilled R1 Models

  • For distilled models, probes exceed forced answering and CoT monitoring at 1.5B/7B; at 14B/32B, forced-answer accuracy drops in the second half despite similar early accuracy.The captions suggest forced answering may rely on current reasoning computation to produce a plausible answer rather than the true belief; early probe and forced-answer accuracy remains similar between 14B and 32B.Source: Figure 20 · p. 21, Figure 21 · p. 21, Figure 22 · p. 21

G. Attention Probe vs. Forced Answer over Sequence

  • On DeepSeek-R1 MMLU and GPQA-D, forced answering is initially more often correct when methods disagree; GPT-OSS MMLU favors probes, whereas GPT-OSS GPQA-D methods are mostly wrong until reasoning ends.On DeepSeek-R1 GPQA-D, both methods are wrong 40% of the time at the beginning; on GPT-OSS GPQA-D, forced answering wins when the methods disagree, possibly because MMLU probes transfer poorly.Source: Figure 23 · p. 22, Figure 24 · p. 22, Figure 25 · p. 22

H. GPT-OSS Calibration for Early Exit

  • At 90% confidence, GPT-OSS MMLU early exit saves most tokens with marginal accuracy loss, while GPQA-D saves fewer tokens per accuracy drop; probes are well calibrated on MMLU but overconfident on transferred GPQA-D.On GPQA-D, the probe also outperforms the original model’s accuracy at the end of reasoning.Source: Figure 27 · p. 23, Figure 28 · p. 24, Figure 29 · p. 24

I. Additional Inflection Analysis · I.1. Probe Shifts and Inflection Point Correlation

Inflection patterns vary by model and task: high-confidence GPT-OSS MMLU traces contain fewer inflections, while probe shifts predict inflections on MMLU but not consistently across settings.

I. Additional Inflection Analysis

  • GPT-OSS high-confidence MMLU traces (124/514) had fewer inflections per step than other traces (0.053 vs. 0.096), while its overall rate exceeded DeepSeek-R1 (0.089 vs. 0.038).GPT-OSS also had more reconsiderations per step than DeepSeek-R1 (0.064 vs. 0.028).Source: Table 7 · p. 24, §I · p. 24
  • On GPQA-Diamond, inflections were more likely when no probe shift occurred in the window, the opposite of the MMLU trend.Figure 30 reports no other significant trend for GPQA-Diamond.Source: §I · p. 24, Figure 30 · p. 25
  • On MMLU, reconsiderations followed 20% probe confidence shifts nearly twice as often as they occurred without a shift (59% vs. 35%).The analysis uses a 10-step window after the shift.Source: §I · p. 24, Figure 30 · p. 25

I.1. Probe Shifts and Inflection Point Correlation

  • The analysis compares how often inflections follow probe shifts versus no shifts, and how often inflections predict subsequent probe shifts, across shift thresholds and window sizes.The first conditional-probability difference tests whether shifts predict upcoming inflections; the reverse tests whether inflections predict later shifts.Source: §I.1 · p. 25
  • For DeepSeek-R1 on MMLU, inflections were twice as likely after probe shifts at a 0.5 threshold and 2–10-step windows, while the reverse correlation was absent or negative.The reported relationship varies with threshold and window size.Source: Figure 31 · p. 26
  • For GPT-OSS on MMLU, probe shifts predicted inflections mainly at a one-step window, while some settings showed the reverse; the trend was inconsistent.The inconsistency complicates analysis, particularly for smaller windows.Source: Figure 32 · p. 26

Related papers

Loading 2603.05488v4…