Source-linked AI summary
Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification
Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, He He
TL;DR
Reasoning models may overthink because it is unclear whether they can evaluate intermediate-answer correctness during search. This paper probes hidden states to extract correctness signals and uses them for early exit, finding a 24% reduction in inference tokens without compromising accuracy. The results indicate that models encode self-verification information but do not efficiently exploit it during inference.
Problem
The paper asks whether reasoning models can evaluate the correctness of intermediate answers during reasoning, which matters for preventing overthinking.
Method
The authors segment long Chain-of-Thought traces into answer-containing chunks and train probes on hidden-state representations to predict correctness.
Results
24% reduction in inference tokens is achieved with confidence-based early exit without compromising accuracy.
Takeaways & Limitations
Reasoning models encode answer correctness and can be given a lightweight hidden-state verifier for more efficient inference.
Takeaways & Limitations
Probe generalization is unstable for out-of-domain transfer between logical and mathematical reasoning, and calibration is weaker on the more difficult AIME dataset.
Abstract
from arXiv · showhide
Reasoning models have achieved remarkable performance on tasks like math and logical reasoning thanks to their ability to search during reasoning. However, they still suffer from overthinking, often performing unnecessary reasoning steps even after reaching the correct answer. This raises the question: can models evaluate the correctness of their intermediate answers during reasoning? In this work, we study whether reasoning models encode information about answer correctness through probing the model's hidden states. The resulting probe can verify intermediate answers with high accuracy and produces highly calibrated scores. Additionally, we find models' hidden states encode correctness of future answers, enabling early prediction of the correctness before the intermediate answer is fully formulated. We then use the probe as a verifier to decide whether to exit reasoning at intermediate answers during inference, reducing the number of inference tokens by 24\% without compromising performance. These findings confirm that reasoning models do encode a notion of correctness yet fail to exploit it, revealing substantial untapped potential to enhance their efficiency.
1 Introduction
Reasoning models search across multiple paths but may continue reasoning after reaching correct intermediate answers. The paper probes hidden states to assess correctness and uses that signal for confidence-based early exit.
- Motivation: Reasoning models explore multiple paths with different intermediate answers, but this search can introduce unnecessary reasoning steps.The section frames overthinking as continued reasoning after a correct answer has been reached.
- Approach: The authors train a binary classifier on hidden states from Chain-of-Thought chunks to predict whether each intermediate answer is correct.Long CoT is segmented into chunks containing intermediate answers before probing.
- Findings: Answer-correctness information is readily encoded in hidden states, and probes perform accurately on both in-distribution and out-of-distribution examples.The probe also achieves expected calibration error below 0.1.
- Efficiency: 24% reduction in inference tokens is achieved by thresholding probe confidence for early exit without compromising accuracy.The probe serves as a verifier that identifies when reasoning can stop at an intermediate answer.
2 Related work
Prior work studies uncertainty estimation, efficient reasoning, and learned verification through external models or prompting. This paper instead leverages correctness information encoded directly in reasoning-model representations for inference-time control.
- Uncertainty estimation in LLMs: White-box uncertainty methods using internal representations have been found to outperform black-box methods based on elicited confidence.Black-box approaches often aggregate self-reported confidence across multiple samples.
- Efficient reasoning during inference: Reasoning models improve task performance through search but can overthink by repeating unnecessary reasoning after reaching a correct answer.Related work explores training methods that make reasoning more concise or reduce overthinking.
- Efficient reasoning during inference: Threshold-based early exit uses encoded correctness information to reduce test-time compute while preserving performance.The paper argues that models encode this information but fail to use it efficiently during inference.
- Learned verifiers: Existing verifiers classify solution correctness or preference, while prompting methods guide models to critique and refine outputs in natural language.This work focuses on correctness information already encoded in reasoning-chain representations.
3 Probing for intermediate answer correctness
The probing pipeline converts long reasoning traces into labeled chunks containing intermediate answers, extracts their hidden-state representations, and trains an MLP probe. Weighted loss addresses the imbalance caused by mostly correct intermediate answers.
- 3.2 Training the probe: The probe is a two-layer multilayer perceptron trained on the collected representation-label dataset.The MLP maps chunk representations to correctness predictions.
- 3.1 Data collection: Long CoT is split into reasoning-path chunks, each paired with an intermediate answer and a binary correctness label.Keywords identify new reasoning paths; answerless adjacent chunks are merged with the nearest answer-containing chunk.
- 3.2 Training the probe: Each chunk representation is the last-layer hidden state at its last token position, paired with its correctness label in the probing dataset.The probing dataset depends on both the original task dataset and the reasoning model generating the representations.
- 3.2 Training the probe: Weighted binary cross-entropy is used because most intermediate answers from strong reasoning models are correct, creating class imbalance.The imbalance weight is scaled by a hyperparameter α.
- 3.2 Training the probe: The probe output applies a sigmoid to produce a correctness probability, with parameters defined by the MLP’s input and hidden dimensions.The loss uses the predicted probability and binary label, with weighting for negative and positive samples.
4 Experiments
Experiments test whether probes can extract answer-correctness signals from reasoning-model hidden states, across datasets, model types, and positions within reasoning chunks. They find strong in-distribution performance, limited cross-domain transfer, weaker signals in non-reasoning models, and predictive information before answers are fully generated.
- Experimental setup: The experiments evaluate probes across mathematical and logical reasoning datasets, several reasoning models, in-distribution and out-of-distribution settings, and intermediate-answer positions.The setup includes GSM8K, MATH, AIME, KnowLogic, DeepSeek-R1-Distill models, QwQ-32B, and position-wise probing within reasoning chunks.
- In-distribution correctness probing: In-distribution probes achieve ROC-AUC above 0.7 and ECE below 0.1, showing that hidden states encode extractable answer-correctness information.Many probes converge to a linear probe, suggesting that correctness information is linearly encoded in reasoning-model hidden states.
- In-distribution correctness probing: Probe quality varies by task and model: mathematical data generally outperforms logical data, larger models work better, and R1-Distill-Qwen-32B exceeds 0.9 ROC-AUC on AIME.Qwen representations also show stronger correctness signals than the Llama-8B counterpart in mathematical settings.
- Out-of-distribution generalization: Probes transfer well between MATH and GSM8K but do not stably generalize from logical to mathematical reasoning, while AIME transfers are less calibrated.These out-of-distribution results use R1-Distill-Llama-8B representations and compare ROC-AUC and ECE against in-distribution probes.
- Relation to long-CoT reasoning: On MATH, probes trained on non-reasoning Llama-3.1-8B-Instruct representations perform worse than probes trained on reasoning-model representations, with lower classification scores and higher calibration errors.This pattern may suggest that long-CoT supervised training enhances self-verification ability.
- Early prediction within reasoning chunks: Hidden states predict upcoming intermediate-answer correctness before the answer is fully generated, with performance improving near the chunk’s beginning and end.The peak accuracy is 79%; ECE reaches 0.03 around the 60% position, while Brier score continues improving toward the chunk’s end.
5 Probe as a verifier for early-exit
The probe converts hidden-state information about intermediate-answer correctness into confidence-based early exit, enabling shorter reasoning without degrading accuracy. Compared with static stopping, confidence-based exit achieves better accuracy at similar token costs.
- Probe-based early exit: The probe takes each reasoning chunk’s representation and outputs the probability that its intermediate answer is correct.During inference, intermediate answers are evaluated sequentially and reasoning stops when the probed probability exceeds threshold Thr.
- Experimental setup: The experiment evaluates final-answer accuracy and token cost on R1-Distill-Llama-8B using the MATH dataset with a 10K-token generation limit.Static early exit instead stops after a predetermined number of intermediate answers m.
- Figure 5: The Figure 5 curves compare final-answer accuracy against inference token cost while varying confidence threshold or static chunk number m.Confidence-based curves vary the correctness-confidence threshold; static curves vary m.
- Results: At Thr = 0.85, confidence-based early exit reaches 88.2% accuracy while reducing generated tokens by approximately 24%.This is reported as roughly the same reasoning accuracy as no early-exit.
- Results: At Thr = 0.9 or higher, confidence-based early exit matches no early-exit accuracy at 88.6% while reducing generated tokens by 19%.The result indicates that excess tokens are generated even after the probe indicates high confidence.
- Comparison with static early exit: With equivalent token usage, confidence-based early exit achieves up to a 5% accuracy improvement over static early exit.At similar usage, Thr = 0.8 yields 87.4% accuracy versus approximately 82.5% for static early exit; above 85% accuracy, it also consumes significantly fewer tokens than m = 6.
6 Discussion
The discussion interprets hidden-state probing as evidence for self-verification in reasoning models and as a lightweight route to more efficient inference control. It also identifies unresolved questions about the origins and use of this capability.
- Self-verification ability: The probe reveals readily accessible answer-correctness information in reasoning models’ hidden states and can serve as a lightweight verifier.The authors relate this finding to model interpretability and underexplored information in internal representations.
- Self-verification ability: Self-verification abilities are less pronounced in non-reasoning models, while their origins remain unclear because training processes and data are intricate and diverse.The authors identify when and how these abilities emerge during training as a future research direction.
- Internal mechanisms of reasoning models: Reasoning models can self-verify but still overthink by performing unnecessary re-checks after correct answers receive high confidence.The discussion frames this as incomplete use of an intrinsic capability and calls for methods that guide more efficient use during training or inference.
- On-policy control of reasoning models: The hidden verifier is more lightweight and integrated with the reasoning model’s existing architecture than prior LLM-based verifiers.The authors present this as supporting an on-policy perspective for inference control.
- Conclusion: The study presents lightweight probing and on-policy control as directions for improving reasoning efficiency and developing more adaptive inference strategies.Future work should investigate self-verification origins and better harness encoded correctness information.
A.1 Data collection details
The data-collection pipeline segments reasoning traces into chunks, filters truncated completions, and records dataset statistics. It also visualizes chunk representations to examine domain differences across datasets.
- A.1 Data collection details: Reasoning-path keywords identify new paths and support segmentation of traces into chunks.The keyword list is documented in Table 3.
- A.1 Data collection details: KnowLogic is split into 80% training and 20% test data, with probing data collected separately.
- A.1 Data collection details: Completions exceeding the 30K-token maximum are discarded during inference.The probing-dataset statistics account for inconsistencies caused by discarded truncated completions.
A.2 Prompts
The study uses specified inference and evaluation prompts, preserving the inference prompt when extracting chunk representations. A t-SNE figure visualizes representations from sampled training chunks.
- A.2 Prompts: Reasoning models use a common elicitation prompt, with a slight modification for Qwen models.The authors report a small benchmark degradation for Qwen that remains within a reasonable range.
- A.2 Prompts: The same prompt is retained when extracting each reasoning chunk’s representations to keep the extracted feature on-policy.
- A.2 Prompts: Gemini 2.0 Flash receives a separate evaluation prompt for answer extraction and evaluation from reasoning chunks.
- A.2 Prompts: Figure 6 visualizes representations for 1000 randomly sampled chunks from each training set using R1-Distill-Llama-8B.
A.3 Grid search
Classifier hyperparameters are selected by grid search over optimization and architecture choices. The search ranges and optimal settings are reported in Tables 7 and 8.
- A.3 Grid search: Grid search varies learning rate, loss weight scaling factor α, optimizer weight decay, and classifier hidden size d.
- A.3 Grid search: Table 8 reports the resulting optimal hyperparameter settings for each probing dataset.
- A.3 Grid search: Table 7 reports the search range for each classifier-training hyperparameter.
A.4 Further results
Further results report in-distribution probing performance across standard classification metrics and out-of-distribution performance across several reasoning models. The evaluations include both mathematical and logical datasets.
- A.4 Further results: In-distribution probes are evaluated with accuracy, precision, recall, and macro F1 across reasoning models and datasets.
- A.4 Further results: Out-of-distribution probes are evaluated using ROC-AUC and expected calibration error.Tables 11–15 report these results for multiple reasoning models.
- A.4 Further results: The out-of-distribution evaluations cover R1-Distill-Qwen models at 1.5B, 7B, and 32B, R1-Distill-Llama-70B, and QwQ-32B.