Source-linked AI summary
CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?
Qing Zong, Jiayu Liu, Tianshi Zheng, Chunyang Li, Baixuan Xu, Haochen Shi, Weiqi Wang, Zhaowei Wang, Chunkit Chan, Yangqiu Song
TL;DR
Precise gold confidence labels are difficult to obtain, motivating the study of natural-language critiques for LLM confidence calibration. The paper introduces Self-Critique and CritiCal, and reports that CritiCal outperforms baselines, including its teacher on complex reasoning tasks, with robust out-of-distribution improvements.
Problem
Precise gold confidence labels are hard to obtain, motivating investigation of whether natural-language critiques can improve verbalized confidence calibration.
Source: §1 · p. 1Method
The paper proposes Self-Critique and CritiCal, an SFT method that trains on teacher critiques of a student’s question, answer, and confidence score.
Source: §1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
Confidence expressions suit multiple-choice tasks, while uncertainty is better suited to open-ended tasks, informing which calibration target to use.
Source: §1 · p. 2The benchmarks may not represent the wider range of deployment scenarios, including creative writing and multimodal tasks.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Accurate confidence calibration in Large Language Models (LLMs) is critical for safe use in high-stakes domains, where clear verbalized confidence enhances user trust. Traditional methods that mimic reference confidence expressions often fail to capture the reasoning needed for accurate confidence assessment. We propose natural language critiques as a solution, ideally suited for confidence calibration, as precise gold confidence labels are hard to obtain and often require multiple generations. This paper studies how natural language critiques can enhance verbalized confidence, addressing: (1) What to critique: uncertainty (question-focused) or confidence (answer-specific)? Analysis shows confidence suits multiple-choice tasks, while uncertainty excels in open-ended scenarios. (2) How to critique: self-critique or critique calibration training? We propose Self-Critique, enabling LLMs to critique and optimize their confidence beyond mere accuracy, and CritiCal, a novel Critique Calibration training method that leverages natural language critiques to improve confidence calibration, moving beyond direct numerical optimization. Experiments show that CritiCal significantly outperforms Self-Critique and other competitive baselines, even surpassing its teacher model, GPT-4o, in complex reasoning tasks. CritiCal also shows robust generalization in out-of-distribution settings, advancing LLM's reliability.
1 Introduction
The paper investigates whether natural-language critiques can improve verbalized confidence calibration, because precise gold confidence labels are hard to obtain. It distinguishes question-focused uncertainty from answer-specific confidence and proposes Self-Critique and CritiCal as ways to critique calibration.
Motivation
- Confidence calibration matters for reliable, trustworthy LLM use in high-stakes applications, and verbalized confidence can communicate response certainty to users.The paper gives “my confidence is 80%” as an example of verbalized confidence.Source: §1 · p. 1
- Natural-language critiques are proposed for calibration because they explain why answers are correct or incorrect, while precise gold confidence labels are hard to obtain.The paper argues that whether confidence is too high or too low can be assessed from reasoning and answer correctness.Source: §1 · p. 1
What to critique: uncertainty or confidence?
- Verbalized confidence performs better on multiple-choice questions, while uncertainty is better suited to open-ended tasks.The paper distinguishes confidence as answer-specific and uncertainty as question-focused.Source: §1 · pp. 1–2
How to critique: self-critique or critique calibration training?
- CritiCal significantly improves confidence calibration on reasoning-intensive questions, outperforming Self-Critique and other baselines and surpassing GPT-4o’s calibration in complex reasoning tasks.The paper also reports robust improvements out of distribution, including cases where models trained on critique-suited data outperform those trained in distribution.Source: §1 · p. 2
2 Related Works
Prior work estimates confidence using internal model data or model outputs, while critique-learning research applies feedback to accuracy, reasoning, or uncertainty. This paper’s related-work discussion positions natural-language critique for confidence calibration as distinct from those approaches.
2.1 Confidence Calibration
- White-box calibration uses internal signals, while black-box methods use model outputs, including consistency-based estimates and verbalized confidence.SaySelf uses a teacher to generate reflective rationales and confidence scores from inconsistencies across sampled reasoning chains, but relies on diverse outputs and imitates reference expressions rather than critiquing its own confidence.Source: §2.1 · p. 2
2.2 Critique Learning
- Critique learning includes outcome- and process-based feedback, numerical uncertainty critiques, and natural-language critiques aimed at improving accuracy; this paper instead studies calibration.The related work notes that numerical-critique methods target uncertainty and that natural-language-critique training has focused on accuracy.Source: §2.2 · p. 2
3 Method
The method pairs a prompting-based Self-Critique approach with CritiCal, which fine-tunes a model to learn confidence calibration from teacher-generated critiques of its initial confidence.
3 Method
- The paper proposes two critique-based approaches: prompting-based Self-Critique and supervised fine-tuning with CritiCal.CritiCal trains the model using critiques of its initial confidence expressions.Source: §3 · p. 2, §3.2 · p. 3
3.1 Self-Critique
- Self-Critique prompts the model to reassess its question, initial reasoning, ambiguities, and logical gaps, refining both its answer and confidence score.Unlike prior self-improvement work focused on reasoning and answer accuracy, this approach targets confidence aligned with correctness and uncertainty in the reasoning steps.Source: §3.1 · p. 3
3.2 CritiCal
- CritiCal is a sampling-free training approach that learns from critiques of confidence-estimation errors rather than conventional targets derived from answer correctness or repeated response generation.Its input is the question, student answer, and confidence score; GPT-4o critiques the calibration by comparing reasoning with a reference solution.Source: §3.2 · p. 3
- CritiCal training collects student answers and confidence scores for 2K training questions, obtains teacher critiques using benchmark reference solutions, and fine-tunes the student on the critique data.The teacher’s structured critiques use “</think>” tokens to separate explanation from final judgment, a format intended to mitigate knowledge shift in large reasoning models.Source: §3.2 · p. 3
4 Experiments
The experiments compare question-focused uncertainty with answer-specific confidence and evaluate whether multi-turn Self-Critique improves confidence calibration. Results vary by task type and model, with Self-Critique helping math calibration slightly but worsening factuality calibration.
4.2 Uncertainty vs. Confidence
- For open-ended questions, uncertainty achieves lower ECE; for multiple-choice questions, confidence outperforms uncertainty in both ECE and AUROC.The study reports these patterns for both LLMs and LRMs.Source: Figure 3 · p. 4, §4.2 · p. 4
4.1 Experimental Setup
- Five models are evaluated across six benchmarks, with MATH reserved for training; accuracy, ECE, and AUROC measure correctness, calibration, and discrimination.Accuracy uses exact match for open-ended responses; lower ECE and higher AUROC indicate better calibration evaluation outcomes.Source: §4.2 · p. 4, §4.1 · p. 4
4.3.1 Multi-Turn Self-Critique
- LRMs show lower ECE and AUROC variability than LLMs on factuality benchmarks; increased math-task variability reflects progressively refined confidence calibration.The analysis attributes the math-related increase in standard deviation to refinement over iterations.Source: Figure 4 · p. 5, §4.3.1 · pp. 5–6
- Factuality-task calibration worsens under Self-Critique, while math calibration improves slightly; the method produces no notable accuracy gains.Factuality benchmarks show increased average ECE and decreased average AUROC.Source: Figure 5 · p. 5, §4.3.1 · p. 5
- Multi-turn experiments use four reasoning and non-reasoning models across six benchmarks, feeding each iteration the results of all previous iterations as context.The study examines calibration and average confidence scores over multiple iterations.Source: §4.3 · p. 4, §4.3.1 · p. 5
4.3.2 Average Confidence Change during Self-Critique
- Llama grows more uncertain and less confident across benchmarks, while Qwen moves oppositely; distilled models generally maintain more consistent scores than non-reasoning models.The distilled Llama’s uncertainty increases on ComparisonQA, an exception to the general pattern.Source: Figure 6 · p. 6, §4.3.2 · p. 6
4.4 CritiCal Analysis
CritiCal improves calibration on complex reasoning tasks, generalizes to an out-of-distribution setting, and performs similarly with SFT and DPO, while SFT is more efficient. The analysis also finds that critique suitability varies by task and that multi-hop factuality data yields stronger calibration gains than math data.
4.4 CritiCal Analysis
- The analysis evaluates in- and out-of-distribution performance across ComparisonQA, StrategyQA, and MATH-Perturb, sampling 2K training questions for each run.For MATH-Perturb, training data comes from original MATH training questions, and evaluation uses perturbed questions from the original test set to prevent data leakage.Source: §4.4 · p. 6
4.4.1 In Distribution
- CritiCal significantly improves calibration and accuracy on StrategyQA and MATH-Perturb, with lower ECE and higher AUROC than all baselines, including Self-Critique.Its impact on ComparisonQA is limited.Source: Table 1 · p. 7, §4.4.1 · p. 7
- CritiCal-trained models retain the task distinction: uncertainty suits open-ended questions, while confidence suits multiple-choice questions.StrategyQA yields greater calibration improvements than MATH-Perturb, which the authors suggest reflects its explicit multi-hop reasoning steps.Source: §4.4.1 · p. 7
4.4.2 Out of Distribution
- On OOD questions, CritiCal improves calibration with lower ECE and higher AUROC, while SFT_Hard and SFT_Soft degrade; the OOD test trains on StrategyQA and tests on MATH-Perturb.The authors attribute the improvement likely to StrategyQA’s critique-suited multi-hop reasoning data.Source: §4.4.2 · pp. 7–8, Table 2 · p. 8
4.4.3 Analysis of Training Method
- CFT and CPO perform very similarly in one-hop and multi-hop reasoning, while SFT remains a sufficient and more efficient training method because DPO costs more computation.CFT denotes SFT-based CritiCal; CPO denotes DPO-based CritiCal.Source: Table 3 · p. 8, §4.4.3 · p. 8
5 Conclusions
The paper studies natural-language critiques for verbalized confidence calibration, proposing Self-Critique and CritiCal. CritiCal outperforms Self-Critique and other baselines in complex reasoning tasks, with reported generalization across in-distribution and out-of-distribution settings.
5 Conclusions
- CritiCal significantly outperforms Self-Critique and other baselines, with calibration exceeding teacher GPT-4o in complex reasoning tasks.The conclusion also reports strong generalization in both in-distribution and out-of-distribution settings, particularly when trained on critique-suited multi-hop reasoning data.Source: §5 · p. 8
- Confidence expressions are better suited to multiple-choice tasks, while uncertainty is more effective for open-ended tasks.Source: §5 · p. 8
- The paper presents SFT as sufficient and efficient for CritiCal training compared with DPO.Source: §5 · p. 8
Limitations
The authors identify limits to claims about CritiCal’s generalizability from the benchmarks used and to the breadth of their SFT–DPO comparison.
Limitations
- The authors note that the study cannot cover all limitations in a single work.Source: Limitations · p. 9
- The selected benchmarks may not represent the full range of LLM deployment scenarios, including creative writing and multimodal tasks.The authors say evaluation on a wider array of datasets could strengthen robustness claims.Source: Limitations · p. 9
- Computational constraints limited the SFT–DPO comparison to ComparisonQA and StrategyQA, leaving potential variation across other task types unexamined.The authors suggest more compute could support broader benchmark and training-configuration analysis.Source: Limitations · p. 9
Ethics Statement
The ethics statement describes the datasets and models used, notes that the data is anonymized, and cautions users not to treat confidence estimates as more than a reference.
Ethics Statement
- The study uses publicly available datasets, and their data is anonymized, so the authors say it raises no privacy concerns regarding specific entities.The listed datasets are ComparisonQA, TriviaQA, StrategyQA, HotpotQA, MATH, MATH-Perturb, and MATH-500.Source: Ethics Statement · p. 9
- The authors state that the same risks associated with LLM research apply to their use of LLaMA, Qwen, Mistral, DeepSeek-Distill-Llama, DeepSeek-Distill-Qwen, and GPT-4o.Source: Ethics Statement · p. 9
- Users are advised to treat CritiCal’s confidence expressions as a reference because estimates may occasionally be inaccurate and users may over-rely on them.The paper identifies this as a risk of using confidence estimates to increase trust in AI.Source: Ethics Statement · p. 9
A Prompt
The appendix provides vanilla prompts for uncertainty and confidence, plus a multi-turn confidence Self-Critique prompt that asks the model to reassess its earlier reasoning and confidence.
A Prompt
- The vanilla StrategyQA prompts separately elicit uncertainty and confidence, with a question placeholder in each.The figures identify the prompt type and show that [Question] is replaced with the real question.Source: Figure 7 · p. 12, Figure 8 · p. 12
- The multi-turn confidence prompt presents the yes/no question, initial response, and refined responses, then asks the model to reassess its reasoning and provide a new confidence score.It directs the model to consider ambiguities, logical steps, and overlooked aspects that could improve answer accuracy and confidence calibration.Source: A Prompt · p. 13, Figure 9 · p. 13
B Detailed Self-Critique Results
Self-Critique preserves the observed distinction between uncertainty and confidence, with stronger improvements on mathematical reasoning than on factuality-related tasks across six benchmarks.
B Detailed Self-Critique Results
- After Self-Critique, the distinction between uncertainty and confidence remains evident in mean ECE and AUROC results.Figure 11 averages each model’s values across five Self-Critique turns; dark bars represent uncertainty prompts and light bars confidence prompts.Source: B Detailed Self-Critique Results · p. 13, Figure 11 · p. 14
- Self-Critique improves more on mathematical reasoning tasks but falls short on factuality-related tasks, indicating limited robustness across the six benchmarks.Figure 12 shows each model’s performance trajectory across the benchmarks.Source: B Detailed Self-Critique Results · p. 13, Figure 12 · p. 14
- Figure 12’s semi-transparent gray area represents average performance across models plus or minus one-third of a standard deviation.Source: Figure 12 · p. 14