PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?

Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, Anirudha Majumdar

arXiv:2506.18183v3cs.AIcs.CL

TL;DR

Reasoning models can give confident but incorrect answers, so the paper examines how well their confidence is calibrated. It evaluates calibration across benchmarks, tests deeper reasoning, and introduces introspection of reasoning traces. The models are generally overconfident, become more so with deeper reasoning, and show mixed calibration changes with introspection.

  • Problem

    Reasoning models can produce confident but incorrect answers, while their uncertainty calibration remains underexplored despite its relevance to knowing when to trust them.

    Source: §1 · p. 2

  • Method

    The paper evaluates self-verbalized uncertainty across reasoning models and benchmarks, tests reasoning depth, and introduces multi-stage introspection of earlier reasoning traces.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Reasoning models are typically overconfident, become more overconfident with deeper reasoning, and show mixed calibration changes with introspection, which helps some models but worsens Claude 3.7 Sonnet’s calibration.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The results identify improving reasoning-model calibration, including when accuracy is low, as an open research challenge for safer adoption.

    Source: §7 · p. 14

    Stable, well-calibrated reasoning models remain an open challenge, particularly preserving calibration when model accuracy is low.

    Source: §7 · p. 14

Abstract

from arXiv · show

Reasoning language models have set state-of-the-art (SOTA) records on many challenging benchmarks, enabled by multi-step reasoning induced using reinforcement learning. However, like previous language models, reasoning models are prone to generating confident, plausible responses that are incorrect (hallucinations). Knowing when and how much to trust these models is critical to the safe deployment of reasoning models in real-world applications. To this end, we explore uncertainty quantification of reasoning models in this work. Specifically, we ask three fundamental questions: First, are reasoning models well-calibrated? Second, does deeper reasoning improve model calibration? Finally, inspired by humans' innate ability to double-check their thought processes to verify the validity of their answers and their confidence, we ask: can reasoning models improve their calibration by explicitly reasoning about their chain-of-thought traces? We introduce introspective uncertainty quantification (UQ) to explore this direction. In extensive evaluations on SOTA reasoning models across a broad range of benchmarks, we find that reasoning models: (i) are typically overconfident, with self-verbalized confidence estimates often greater than 85% particularly for incorrect responses, (ii) become even more overconfident with deeper reasoning, and (iii) can become better calibrated through introspection (e.g., o3-Mini and DeepSeek R1) but not uniformly (e.g., Claude 3.7 Sonnet becomes more poorly calibrated). Lastly, we conclude with important research directions to design necessary UQ benchmarks and improve the calibration of reasoning models.

1 Introduction

Reasoning models can achieve strong benchmark performance while still producing confident errors, making their calibration important to trust. The paper evaluates their calibration, the effect of deeper reasoning, and introspection as a way to improve it.

1 Introduction

  • Reasoning models break complex tasks into subtasks and have achieved state-of-the-art performance across programming, mathematics, science, and arts benchmarks.Examples include Codeforces, AIME, GPQA, and MMLU.Source: §1 · p. 2
  • Confident but factually incorrect responses make knowing when and how much to trust reasoning models important for safe real-world integration.Prior work found non-reasoning models poorly calibrated in general, while chain-of-thought prompting can improve their calibration; reasoning-model UQ remained unexplored.Source: §1 · p. 2
  • Reasoning models are generally overconfident, often assigning confidence above 85% to incorrect answers, and calibration varies between knowledge-retrieval and domain-expertise tasks.They are well-calibrated on knowledge-retrieval tasks but poorly calibrated on tasks requiring domain expertise; prompt engineering is less effective than with non-reasoning models.Source: §1 · p. 2
  • Greater inference-time reasoning effort makes models more overconfident, especially on questions they answer incorrectly.The evaluation compares low, medium, and high reasoning effort.Source: §1 · p. 2
  • The paper introduces introspective uncertainty quantification, in which a second model assesses the first model’s reasoning trace and answer to estimate confidence.More critical introspection improves calibration for o3-Mini and DeepSeek R1, but worsens it for Claude 3.7 Sonnet.Source: §1 · p. 2

2 Related Work

Prior work studies methods for eliciting and structuring language-model reasoning, as well as uncertainty estimation for language models. This paper addresses the relatively unexplored calibration of reasoning models.

Reasoning in LLMs

  • Reinforcement-learning-trained reasoning models achieve higher accuracy on complex tasks, but the effects of deeper reasoning on calibration remain unknown.Earlier approaches include parallel reasoning traces, sequential subproblem solving, and planning with Monte Carlo Tree Search; effective chain-of-thought examples can require domain expertise.Source: §2 · pp. 2–3

Uncertainty Quantification of LLMs

  • Language-model UQ methods are broadly white-box or black-box, and this work evaluates reasoning models without finetuning using self-verbalized confidence.Closed-source reasoning models generally do not expose logits needed by several token-based, semantic-similarity, or mechanistic-interpretability methods.Source: §2 · p. 3

3 Preliminaries

Calibration compares a model’s confidence with the probability that its answer is correct. The section introduces ECE and MCE as empirical measures of the discrepancy.

Calibration of Uncertainty

  • Perfect calibration means that among answers assigned confidence p, the proportion that are correct is p.The definition assumes a deterministic ground-truth answer given the input, while correctness can reflect equivalent meaning rather than exact wording.Source: §3 · p. 3

Calibration Metrics

  • ECE measures the average absolute difference between predicted confidence and empirical accuracy, approximated by grouping samples into equal-width confidence bins.The approximation weights each bin’s accuracy-confidence gap by its share of the total samples.Source: §3 · pp. 3–4
  • MCE captures the worst-case discrepancy between confidence and accuracy by taking the largest bin-wise error.The authors approximate MCE using the largest bin-wise error.Source: §3 · p. 4

4 Method

The method evaluates self-verbalized uncertainty through established prompt strategies and introduces a multi-stage approach that asks a model to inspect earlier reasoning traces. Its introspection prompts vary in how explicitly they direct the model to find flaws.

4 Method

  • The method draws on prompt-engineering strategies and extends them with introspective UQ to study reasoning models’ confidence estimation.The prompts are zero-shot, without examples.Source: §4 · p. 4, §4.1 · p. 4

4.1 Prompt Strategies

  • The prompt strategies include zero-shot basic, chain-of-thought, multi-step, and Top-K prompting for eliciting self-verbalized uncertainty.The multi-step strategy requests confidence at each step; Top-K asks for likely candidate answers and their confidence.Source: §4.1 · pp. 4–5, Table 1 · p. 4

4.2 Introspective Uncertainty Quantification

  • Introspective UQ uses a multi-stage reasoning architecture to reason about and update confidence by inspecting earlier reasoning traces.The illustrated procedure identifies weaknesses in a prior chain of thought and reduces confidence in the original answer.Source: §4.2 · p. 5, Figure 2 · p. 5
  • The second introspection stage varies prompt conservativeness across IUQ-Low, IUQ-Medium, and IUQ-High.IUQ-Low gives no hints about flawed reasoning; IUQ-Medium asks for flaws, including in prior confidence estimates; IUQ-High uses the Medium prompt without those confidence estimates.Source: §4.2 · p. 5
  • Introspective UQ is limited to two stages, and the second stage assesses uncertainty without revising the original answer.This design focuses reflection on prior traces and is intended to guard against hallucinations or vacillation when the model is unsure.Source: §4.2 · pp. 5–6

5 Experiments

The experiments evaluate reasoning-model confidence and calibration across benchmarks, reasoning depths, and prompt strategies. They find systematic overconfidence that varies by task and model, while deeper reasoning and introspection can shift calibration in different directions.

5.1 Evaluation Setup

  • The evaluation covers four reasoning models and five question-answering benchmarks, using four prompt strategies and calibration, accuracy, and completion metrics.Calibration is measured with ECE and MCE; reliability diagrams compare bin-wise confidence with accuracy.Source: §5.1 · pp. 6–7

5.2 Calibration of Reasoning Models

  • Reasoning models are better calibrated on ARC-Challenge and MMLU than on StrategyQA, GPQA, and SimpleQA, and are generally overconfident on challenging datasets.Confidence estimates on challenging datasets are typically above 85%; Claude is best calibrated overall, while Gemini and o3-Mini are among the most overconfident.Source: §5.2 · pp. 7–8, Figure 3 · p. 7, Figure 4 · p. 8

5.3 Correlation Between Accuracy and Calibration of Reasoning Models

  • Accuracy and ECE are strongly negatively correlated across models, while Claude shows a weaker correlation and relatively better calibration.Gemini’s correlation is r = −1; Claude’s is r = −0.69, and accuracy and MCE also have a strong negative correlation.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 Reasoning Depth vs. Calibration

  • Deeper reasoning improves calibration when accuracy rises, but can worsen calibration as accuracy saturates, especially on challenging tasks.On GPQA, low-to-high reasoning raises accuracy by 10.8% and lowers ECE by 5.5%; on SimpleQA, ECE rises more than accuracy as reasoning deepens.Source: §5.4 · pp. 9–11, Figure 6 · p. 10, Figure 7 · p. 10

5.5 Reasoning about Confidence

  • Critical introspection improves calibration for DeepSeek and o3-Mini, especially on SimpleQA, but worsens Claude’s calibration.IUQ-Medium and IUQ-High reduce overconfidence for DeepSeek and o3-Mini, while Claude’s estimates shift toward higher-confidence bins.Source: §5.5 · pp. 12–13, Figure 12 · p. 13

5.6 Calibration and Accuracy of Reasoning Models vs. Prompt Strategies

  • Specialized prompt strategies do not significantly outperform basic prompting for reasoning models in calibration, accuracy, or completion rates.The authors attribute this relative independence to reasoning models generating reasoning traces by default.Source: §5.6 · p. 13

6 Conclusion

The conclusion reports that reasoning models are generally overconfident and become more so with deeper reasoning, especially when added reasoning does not improve accuracy. More critical introspection can improve calibration, but the effect is not uniform across models.

  • Reasoning models are generally overconfident, and deeper reasoning can increase overconfidence when it does not improve accuracy.Source: §6 · p. 14
  • More critical introspection improves calibration overall, but introspection increases Claude’s overconfidence.Source: §6 · p. 14

7 Limitations and Future Work

The paper identifies calibration stability at low accuracy and more informative UQ benchmarks as open challenges, and proposes future work on uncertainty reasoning and prompt strategies. Its appendices provide the full introspective-UQ and standard prompt templates.

7 Limitations and Future Work

  • The paper frames poor calibration despite strong accuracy as a challenge for real-world deployment and points to calibration improvement as important for safe adoption.Source: §7 · p. 14
  • Stable calibration across easy and challenging domains remains an open challenge, especially when model accuracy is low.Future work could train models to recognize challenging problems and provide more conservative confidence estimates.Source: §7 · pp. 14–15
  • The authors propose training reasoning models to reason about uncertainty and knowledge boundaries, with the goal of better estimating answer confidence.They describe this as a future direction enabled by reinforcement-learning-induced multi-step reasoning.Source: §7 · p. 15
  • Existing benchmarks with saturated performance can be uninformative for UQ, motivating benchmarks across disciplines and at intermediate reasoning steps.Intermediate-step benchmarks could provide more insight into the calibration of reasoning traces.Source: §7 · p. 15

A Introspective Uncertainty Quantification

  • Appendix A provides IUQ-Low, IUQ-Medium, and IUQ-High prompts that assess an answer’s reasoning trace, with variants asking the model to identify flaws.It also gives a full IUQ-Medium example prompt.Source: A Introspective Uncertainty… · p. 19

B LLM Prompt Strategies

  • Appendix B supplies prompts for basic, chain-of-thought, multi-step, and top-K strategies, including confidence requests for answers and steps.The appendix also shows an example full prompt using the basic strategy.Source: B LLM Prompt Strategies · pp. 19–20

C Calibration of Reasoning Models.

Across prompt strategies, Claude 3.7 Sonnet has the lowest ECE by substantial margins except under top-K prompting, while reliability diagrams show that high accuracy can obscure overconfidence.

Model Calibration for Each Prompt Strategy

Are Reasoning Models Underconfident or Overconfident?

  • Most confidence estimates fall in the 90%–100% bins, but high model accuracy makes overconfidence harder to detect.The easier datasets ARC-Challenge and MMLU illustrate how accuracy can mask overconfidence.Source: C Calibration of Reasoning Models · p. 20

Model Calibration for Each Prompt Strategy

  • Claude 3.7 Sonnet is generally the best-calibrated and most accurate model considered, with the lowest ECE except under top-K prompting.DeepSeek follows Claude in calibration except for top-K; Gemini and o3-Mini have nearly equal calibration and accuracy, with Gemini narrowly ahead under basic and top-K prompts.Source: C Calibration of Reasoning Models · p. 20

D Coupling between Accuracy and Calibration of Reasoning Models

Model accuracy varies across benchmarks, and calibration errors are strongly associated with accuracy across prompt strategies. High accuracy can also make overconfidence less visible.

Are Reasoning Models Underconfident or Overconfident?

  • Reliability diagrams aggregated across datasets show overconfidence, though high accuracy makes it less apparent on easier datasets.Source: Figure 15 · p. 21

Model Accuracy on Benchmarks

  • All models exceed 90% accuracy on ARC-Challenge and MMLU, but accuracy drops about 10–20% on StrategyQA and GPQA and stays below 40% on SimpleQA.The paper attributes the drop to the newer benchmarks’ emphasis on analytical problem-solving rather than memorization; it notes SimpleQA’s adversarial construction makes it useful for UQ.Source: D Coupling between Accuracy and… · p. 21

Correlation between Accuracy and Calibration

  • Across all prompt strategies, ECE and MCE are strongly negatively correlated with accuracy, with Claude’s correlations generally weaker than Gemini’s.The MCE–accuracy trend is weaker and affected by outliers.Source: D Coupling between Accuracy and… · p. 21

E Effects of LLM Prompt Strategies on Calibration and Accuracy of Reasoning Models

Specialized prompt strategies have little average effect on calibration or accuracy, but some model-specific differences emerge, especially with top-K prompts.

Effects of LLM Prompt Strategies on Calibration and Accuracy of Reasoning Models

F Introspective UQ Reasoning Trace Examples

The examples contrast confidence and correctness across GPQA, StrategyQA, and SimpleQA. On SimpleQA, introspection yields better-calibrated estimates for o3-Mini in particular, while DeepSeek remains overconfident.

F Introspective UQ Reasoning Trace Examples

F.1 GPQA

The GPQA examples show models evaluating a first model’s reasoning trace and confidence, but their assessments of the first reaction’s product diverge. Several evaluations select option A, while another selects option C; the Cope-rearrangement product is assessed as retaining its hexahydro designation.

F.1 GPQA

  • The option-C assessment instead favors (Z)-2-methyl-5-phenylpent-2-en-1-ol for the first reaction while retaining the hexahydro product.It describes the outcome as consistent with the rearrangement pathway and stereoelectronic demands.Source: §F.1 · p. 28
  • Assessments of the first reaction differ on its mechanism: some identify benzylic deprotonation, while another evaluation flags allylic deprotonation as a flaw.Source: §F.1 · pp. 32, 36
  • Several evaluations select option A, pairing 4-methyl-1-phenylpent-3-en-1-ol with the hexahydro Cope product.Other evaluations disagree about the first reaction’s product and select option C.Source: §F.1 · pp. 27, 28, 33, 34
  • The evaluations describe the Cope rearrangement as preserving the degree of unsaturation, supporting a hexahydro rather than tetrahydro product.The supplied option list includes both hexahydro and tetrahydro alternatives.Source: §F.1 · pp. 27, 35

F.2 StrategyQA

The StrategyQA example asks whether Sirius belongs to an animal constellation; the provided reasoning and reviews support “True” while expressing high confidence.

F.2 StrategyQA

  • 95%: The first-round reasoning answers True, linking Sirius to Canis Major, the Greater Dog constellation, and thus an animal.Source: §F.2 · pp. 37, 39
  • A review found the first model’s True answer reasonable but slightly conservative at 95% confidence, and rated its validity 100% based on the stated constellation facts.Source: §F.2 · p. 39
  • A later review acknowledges possible cultural or interpretive ambiguity but says it does not invalidate the answer under standard Western astronomical conventions.Source: §F.2 · p. 40

F.3 SimpleQA

The SimpleQA examples show confidence judgments changing with how specifically a reasoning trace supports its answer. Evaluators distinguish evidence-backed dates from guesses based on broad timelines.

F.3 SimpleQA

  • For the March 2019 guess, evaluators cite the lack of specific evidence and lower the first model’s 40% confidence to 30% or 25%.The trace bases March on general CPEC activity rather than a specific meeting, report, or announcement.Source: §F.3 · pp. 43–44
  • For the December 2018 answer, IUQ-Low accepts the 95% confidence, while IUQ-Medium flags unsourced claims and reduces it to 90%.The medium evaluation calls the JCC explanation plausible but questions the claim that no other announcement occurred.Source: §F.3 · pp. 45–46
  • IUQ-High also endorses the December 2018 answer at 95%, citing Bakhtiar’s tenure, the 8th JCC meeting, and official reports.The first model likewise gives 95% confidence based on the December 2018 meeting.Source: §F.3 · pp. 45–46
  • For a June 2016 answer recalled from memory, evaluators assign confidence estimates of 30%, 20%, and 40% across IUQ-Low, Medium, and High.The evaluations repeatedly note the absence of verifiable evidence or a clear reason for choosing June.Source: §F.3 · p. 47

Related papers

Loading 2506.18183v3…