PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Models Better Express Their Confidence

Dongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim, Soyeon Kim, Yongil Kim, Eunbi Choi, Yireun Kim, Minjoon Seo

arXiv:2505.14489v2cs.AIcs.CL

TL;DR

LLMs often sound confident when wrong, motivating study of whether reasoning models express confidence more accurately. The paper benchmarks reasoning models against non-reasoning counterparts and finds that slow thinking improves calibration by enabling confidence adjustment throughout chain-of-thought reasoning.

  • Problem

    LLMs can sound confident when wrong, creating a persistent weakness in accurately communicating confidence.

  • Method

    The study benchmarks six reasoning models derived from four 32B-scale LLMs against non-reasoning counterparts and analyzes calibration during and across chain-of-thought reasoning.

  • Results

    Reasoning models are consistently better calibrated than their non-reasoning counterparts, with gains linked to slow thinking that dynamically adjusts confidence throughout chain-of-thought reasoning.

  • Takeaways & Limitations

    Slow thinking can enhance the trustworthiness and reliability of LLMs by helping them better align expressed confidence with what they know.

  • Takeaways & Limitations

    Reasoning models still tend to express high confidence and infrequently assign values below 55%, leaving uncertainty expression challenging.

Abstract

from arXiv · show

Despite their strengths, large language models (LLMs) often fail to communicate their confidence accurately, making it difficult to assess when they might be wrong and limiting their reliability. In this work, we demonstrate that reasoning models that engage in extended chain-of-thought (CoT) reasoning exhibit superior performance not only in problem-solving but also in accurately expressing their confidence. Specifically, we benchmark six reasoning models across six datasets and find that they achieve strictly better confidence calibration than their non-reasoning counterparts in 33 out of the 36 settings. Our detailed analysis reveals that these gains in calibration stem from the slow thinking behaviors of reasoning models (e.g., exploring alternative approaches and backtracking) which enable them to adjust their confidence dynamically throughout their CoT, making it progressively more accurate. In particular, we find that reasoning models become increasingly better calibrated as their CoT unfolds, a trend not observed in non-reasoning models. Moreover, removing slow thinking behaviors from the CoT leads to a significant drop in calibration. Lastly, we show that non-reasoning models also demonstrate enhanced calibration when simply guided to slow think via in-context learning, fully isolating slow thinking as the source of the calibration gains.

1 Introduction

The paper studies whether slow thinking in reasoning models improves their ability to express confidence accurately. Across broad benchmarks and analyses, it links better calibration to dynamic confidence adjustment during chain-of-thought reasoning.

  • Motivation: LLMs often sound confident when wrong, motivating study of whether slow thinking helps them communicate the limits of their knowledge.Overconfidence threatens reliability, particularly in high-stakes scenarios.
  • Approach: The study examines verbalized confidence estimation, where models directly express confidence in their outputs.Confidence may be expressed linguistically or numerically.
  • Mechanism: Reasoning models dynamically adjust confidence during chain-of-thought by verifying answers, considering alternatives, rejecting them, and continuing verification.In the illustrated example, confidence rises with verification, falls when an alternative is considered, and rises again when that alternative is rejected.
  • Main findings: 33 of 36 settings showed reasoning models strictly outperforming non-reasoning counterparts across all measured calibration metrics.The comparison covers six reasoning models, six datasets, and both knowledge-focused and reasoning-intensive tasks.
  • Mechanism: Calibration improves steadily as reasoning-model chain-of-thought unfolds, whereas non-reasoning models show no significant improvement and sometimes worsen.The analysis attributes the gains to slow-thinking behaviors such as exploring alternatives and revising reasoning.

2 Related Work

Prior work defines confidence calibration and reviews several confidence-estimation strategies. The paper focuses on verbalized confidence because it is compatible with proprietary, model-agnostic systems while investigating whether reasoning models improve calibration through slow thinking.

  • Confidence estimation: Calibration measures how closely a model’s confidence aligns with whether its answer is correct.Well-calibrated confidence is a reliable indicator of prediction correctness.
  • Confidence estimation: Existing confidence-estimation methods use token probabilities, hidden-state probes, or supervised fine-tuning with labeled confidence data.These approaches generally require internal model access or model-specific retraining.
  • Confidence estimation: Sampling multiple responses provides a model-agnostic alternative but increases computational cost through repeated inference.This approach remains compatible with proprietary models.
  • Verbalized confidence: Verbalized confidence estimation prompts models to express confidence through linguistic phrases or numerical probabilities.The approach is compatible with proprietary models, model-agnostic, and computationally efficient.
  • Reasoning models: Reasoning models generate longer chain-of-thoughts through inference-time scaling, distinguishing them from non-reasoning instruction-tuned models.The paper investigates whether their slow-thinking behaviors improve confidence expression.

3 Reasoning models better express their confidence

The study benchmarks reasoning and non-reasoning models using multiple datasets, confidence-expression procedures, and calibration metrics. Reasoning models generally achieve better calibration, including on reasoning-intensive tasks and settings where their accuracy is not higher.

  • Experiment setup: Six reasoning models from four 32B-scale backbones are compared with non-reasoning counterparts across knowledge-focused and reasoning-intensive datasets.Qwen3 is evaluated within one checkpoint using Thinking and Non-thinking modes.
  • Experiment setup: Models produce a solution, reason about confidence, and verbalize confidence using one of ten linguistic-numeric bins.The procedure extracts answers and confidence values for calibration evaluation.
  • Experiment setup: ECE, AUROC, and Brier Score jointly measure absolute calibration, discriminative ability, and individual-level confidence error.Lower ECE and Brier Scores and higher AUROC indicate better calibration.
  • Experiment result: Reasoning models also calibrate better on reasoning-intensive datasets, although multiple-choice cues can give non-reasoning models an AUROC advantage in some cases.Those cases involve GLM-4-0414 on both Math subsets and EXAONE-3.5-Instruct on SuperGPQA Non-Math; non-reasoning models still receive worse Brier Scores.

4 Analysis: slow thinking enables accurate confidence adjustments

The analysis attributes reasoning models’ calibration gains to slow thinking, which lets them revise confidence as reasoning unfolds. Calibration improves progressively during CoT, while removing non-linear reasoning degrades all measured calibration dimensions; prompting non-reasoning models to slow think produces similar gains.

  • 4.1 CoT progress: Reasoning models become progressively better calibrated as their CoT unfolds, unlike non-reasoning models.The study evaluates cumulative CoT segments and finds statistically significant improvement trends for reasoning models, but generally no significant trends for non-reasoning models.
  • 4.1 CoT progress: On NonambigQA, all three reasoning models show progressively lower relative Brier Scores as CoT progresses, whereas non-reasoning models worsen by the end.For non-reasoning models, the Brier Score is lowest before CoT and shows no consistent intermediate trend.
  • 4.2 Ablation study: Confidence Reasoning has only a minor calibration effect, while removing epistemic phrases worsens ECE but improves AUROC through more concentrated confidence bins.The epistemic-marker ablation tends to produce mainly 95% and 65% predictions, preserving discrimination despite overconfidence.
  • 4.2 Ablation study: Non-linear reasoning has the greatest calibration impact: constraining the reasoning path to a linear trajectory degrades ECE, Brier Score, and discrimination.The ablation removes behaviors such as exploring alternatives, refining, and backtracking while retaining only reasoning directly supporting the final answer.
  • 4.3 In-context slow thinking: Non-reasoning models consistently improve calibration across both datasets and all three metrics when prompted to slow think through in-context learning.The result supports slow thinking itself, rather than reasoning-model-specific properties, as the source of the calibration gains.

5 Discussion

The discussion examines whether longer or larger-scale reasoning strengthens calibration and notes that reasoning models still express uncertainty imperfectly. Calibration gains depend more on the quality of slow thinking than on simply extending its duration.

  • Longer CoT: Budget forcing does not necessarily improve calibration further, suggesting that slow-thinking quality matters more than CoT quantity.The test appends “Wait,” to the CoT and conducts additional inference rounds.
  • Model scale: The calibration gap between reasoning and non-reasoning models widens as model size increases.The analysis compares relative calibration gains across model scales and suggests stronger slow-thinking benefits in larger models.
  • Remaining challenges: Reasoning models still infrequently assign confidence below 55%, and lower NonambigQA accuracy corresponds to higher ECE and Brier Scores than TriviaQA.These observations indicate that expressing uncertainty remains challenging even though slow thinking improves calibration overall.

6 Conclusion

The conclusion finds that reasoning models express confidence more reliably than non-reasoning counterparts across diverse datasets and model families. It attributes these gains to slow thinking that dynamically adjusts confidence throughout CoT, improving LLM trustworthiness and reliability.

  • Conclusion: Reasoning models are consistently better calibrated than their non-reasoning counterparts across diverse datasets and model families.The paper connects this calibration advantage to slow-thinking behaviors during chain-of-thought reasoning.
  • Conclusion: Slow thinking dynamically adjusts confidence throughout CoT, helping LLMs better align expressed confidence with what they know.The conclusion frames this as an improvement beyond problem-solving performance, with implications for trustworthiness and reliability.

A.1 Experiment Results in Discussion

Appendix A.1 reports experiments on forcing longer CoTs and varying model scale. Figure 4 organizes these comparisons as budget forcing on the left and model-scale effects on the right.

  • Figure 4: Figure 4 compares relative Brier Score changes under budget forcing on the left and across model scales on the right.The figure covers the two discussion experiments: extending reasoning and examining scale-dependent calibration.

A.2 Expanded results for robustness

The robustness experiments test alternative prompting, confidence-expression formats, and stochastic decoding. Reasoning models retain calibration advantages under these variations, although direct numerical probabilities generally degrade calibration and AUROC can favor less diverse confidence outputs.

  • Alternative prompting: Reasoning models consistently outperform non-reasoning models even when the latter receive alternative prompting methods.The alternatives include two-turn, top-K, and multi-step prompting.
  • Confidence expression styles: Reasoning models outperform non-reasoning models across all metrics and both datasets when using linguistic confidence descriptors.Directly outputting numerical probabilities without bins instead produces a general calibration degradation for both model types.
  • Sampling instead of greedy decoding: R1-Distill-Qwen-32B outperforms Qwen2.5-32B-Instruct on ECE and Brier Score under recommended sampling at temperature = 0.6.Its slightly lower AUROC is attributed to producing a more diverse range of confidence values, whereas the comparison model mostly outputs two bins.
  • Variability analysis: Bootstrap variability is estimated from five resampled subsets of 1,000 examples per dataset, but the analysis is not run across all models and datasets.The omitted combinations would incur unreasonable computational cost.

A.3 Full Results on SuperGPQA and MMLU-Pro

The appendix provides complete benchmark results for SuperGPQA and MMLU-Pro, reporting both accuracy and ECE.

  • SuperGPQA: Full benchmark results on SuperGPQA are reported with both accuracy and ECE.These results are presented in Table 11.
  • MMLU-Pro: Full benchmark results on MMLU-Pro are reported with both accuracy and ECE.These results are presented in Table 12.
  • Reported metrics: The appendix reports accuracy alongside ECE to present both task performance and calibration results.The passage identifies these as full benchmark results for both datasets.

A.4 Full CoT of the Qualitative Analysis

This appendix includes the full CoT associated with the qualitative analysis and documents the prompts and decoding procedures used to obtain model answers and confidence outputs.

  • Full CoT: Figure 5 presents the full, untruncated CoT from R1-Distill-Qwen-32B shown in Figure 1.It is described as the full-CoT version of Figure 1.
  • Prompting procedure: The experimental prompt asks models to perform SOLUTION REASONING, CONFIDENCE REASONING, and CONFIDENCE VERBALIZATION.The required final format is “Answer:ANSWER Confidence:CONFIDENCE”.
  • Confidence output: Confidence is classified into ten named probability ranges from “Almost no chance” to “Almost certain.”Each category represents the probability that the answer is correct.
  • Output extraction: An additional inference round is used when models fail to produce the required answer-and-confidence format.The recovery procedure differs between non-reasoning and reasoning models.
  • Decoding settings: Greedy decoding uses maximum lengths of 4096 tokens for knowledge-focused datasets and 8192 for reasoning-intensive datasets.Sampling effects are examined separately in Appendix A.2.

B.3 Additional Details on Ablation Study

The ablation study details how CoTs were quality-checked and transformed to isolate confidence phrases and non-linear reasoning. Manual inspection indicates that the transformations generally preserved the intended reasoning content.

  • Confidence reasoning: Only four of 100 manually inspected CoTs contain explicit reasoning about the model’s own confidence.The samples were drawn from both datasets.
  • Epistemic phrases: 91 of 100 examples satisfy both removal of epistemic phrases and faithfulness to the original content.The quality check evaluates absence of epistemic phrases and preservation of truthful content.
  • Non-linear reasoning: All 100 inspected examples satisfy both criteria after non-linear reasoning is removed.The criteria require retaining only content supporting the final answer and remaining faithful to the original.
  • Qualitative example: The before-and-after example changes a hesitant, exploratory answer about the Order of the Elephant into a concise Denmark conclusion.The original includes uncertainty and alternative-country exploration, while the transformed versions remove those elements to different degrees.
  • Epistemic-phrase removal: The confidence-phrase transformation removes explicit uncertainty while preserving logical steps, “Wait,” moments, and the original first sentence.It does not fix factual errors or complete missing reasoning.
  • Non-linear-reasoning removal: The non-linear-reasoning transformation linearizes the CoT by removing backtracking, diversions, and unnecessary side exploration.It retains details directly supporting the final answer and does not fix factual errors.

C.2 Datasets

The paper lists four benchmark datasets with licensing information and documents experimental reproducibility, statistical reporting, and broader evaluation claims.

  • Datasets: The benchmark resources are SuperGPQA, MMLU-Pro, NonambigQA, and TriviaQA, with licenses and access links provided.
  • Reproducibility: The authors state that experimental settings are detailed in Section 3.1 and Appendix B, with code made available for reproduction.
  • Statistical reporting: The main calibration claim is supported with bootstrapping over five resampled subsets and standard deviations reported in Appendix A.2.4.
Loading 2505.14489v2…