Source-linked AI summary
Are LLM Decisions Faithful to Verbal Confidence?
Jiawei Wang, Yanfei Zhou, Siddartha Devic, Deqing Fu
TL;DR
LLMs can verbalize uncertainty, but it remains unclear whether those estimates guide risk-sensitive decisions. RiskEval tests answering and abstaining under varied error penalties and finds that confidence and policy remain largely decoupled, producing utility collapse; the paper therefore questions whether verbal confidence alone is sufficient for trustworthy decision-making.
Problem
It remains unclear whether LLM verbal confidence estimates are faithful to their actions and decisions, despite evidence that such estimates can be calibrated.
Method
RiskEval evaluates whether models use verbal confidence to choose between answering and abstaining as prompt-specified error penalties vary.
Results
Across models and datasets, increasing penalties from 0.1 to 100 negligibly changes confidence or answer-versus-abstain behavior, while high-penalty answering causes large utility losses.
Takeaways & Limitations
Models may possess useful verbal uncertainty estimates yet fail to convert them into effective, risk-sensitive abstention policies.
Takeaways & Limitations
The analysis uses verbalized confidence as a proxy for internal belief, which may not perfectly capture true epistemic uncertainty.
Abstract
from arXiv · showhide
Large Language Models (LLMs) can produce surprisingly sophisticated estimates of their own uncertainty. However, it remains unclear to what extent this expressed confidence is tied to the reasoning, knowledge, or decision making of the model. To test this, we introduce $\textbf{RiskEval}$: a framework designed to evaluate whether models adjust their abstention policies in response to varying error penalties. Our evaluation of several frontier models reveals a critical dissociation: models are neither cost-aware when articulating their verbal confidence, nor strategically responsive when deciding whether to engage or abstain under high-penalty conditions. Even when extreme penalties render frequent abstention the mathematically optimal strategy, models almost never abstain, resulting in utility collapse. This indicates that calibrated verbal confidence scores may not be sufficient to create trustworthy and interpretable AI systems, as current models lack the strategic agency to convert uncertainty signals into optimal and risk-sensitive decisions.
1 Introduction
RiskEval asks whether LLMs’ verbal confidence is faithful to their decisions under changing error penalties. Across the evaluation, models show little risk sensitivity, degrade utility, and fail to convert uncertainty estimates into effective abstention policies.
- RiskEval framework: RiskEval tests whether models adjust answering versus abstaining when prompts impose different error penalties.The framework elicits verbal confidence alongside decisions to assess whether the two are faithful and consistent.
- Findings: Increasing penalties from 0.1 to 100 has negligible effects on self-evaluated confidence or answer-versus-abstain decisions across models and datasets.This pattern indicates limited risk-aware behavior under the tested prompting conditions.
- Findings: High-penalty regimes produce large losses because models retain an answering-heavy policy instead of using confidence-based abstention.These losses are measured against optimal post-hoc abstention strategies using reported verbal confidence.
- Findings: Models’ verbal confidence can be useful or calibrated while their abstention policies remain poorly aligned with that uncertainty.The results identify a decoupling between uncertainty information and decision execution.
- Implications: The findings motivate evaluating strategic reliability in addition to overall accuracy and question whether verbal confidence is faithful in decision-making contexts.The paper frames this as a concern for the validity of verbal confidence estimation as a decision signal.
2 Problem Setup
The setup models answering and abstaining as utility-maximizing decisions under penalties for incorrect answers. It evaluates whether realized policies use verbal confidence appropriately and measures both policy alignment and confidence-based ranking quality.
- Utility model: The utility framework assigns 0 to abstention, +1 to correct answers, and −λ to incorrect answers.The penalty parameter satisfies λ ≥ 0.
- Utility model: A rational agent answers only when confidence meets the penalty-dependent threshold τ(λ) = λ/(1+λ).This threshold makes expected utility from answering at least as large as the zero utility of abstention.
- Metrics: Policy Consistency measures how often the model’s realized decision matches the optimal policy under its own verbal confidence.Higher Policy Consistency indicates better use of confidence for decisions.
- Metrics: Normalized regret measures decision error in probability space by removing raw regret’s linear dependence on the penalty magnitude.For an incorrect answer-versus-abstention choice, it depends on the distance between confidence and the optimal threshold.
- Metrics: AUARC evaluates whether confidence ranks incorrect answers lower without tying the assessment to a particular penalty value.It integrates retained-set accuracy after discarding increasingly low-confidence fractions.
- Evaluation design: The evaluation varies prompt penalty strength to test both confidence calibration and decision-making strategies.The framework compares model behavior across penalty settings for each model and dataset.
3 Experiments
RiskEval evaluates whether LLMs adapt answering and abstention decisions to changing error penalties across three benchmarks. The experiments show persistent answering, declining policy consistency, utility losses, and negligible effects from prompting or post-hoc scaffolding improvements.
- Experimental setup: RiskEval evaluates diverse models on HLE, GPQA Diamond, and GSM8K using metrics for performance, policy consistency, and calibration.Prompts elicit decisions alone or decisions with verbal confidence scores; GPT-4o-mini serves as judge.
- Main results: Across all benchmarks, LLMs do not adapt their decision policies to changing risk, even when abstention is explicitly incentivized.Increasing penalties has negligible effects on reported confidence or answer-versus-abstain behavior.
- Failure to adapt decision rules: At penalties λ ≥10, models continue answering most questions, causing policy consistency to degrade abruptly after remaining high for λ ≤5.The pattern indicates no coherent penalty-dependent decision threshold.
- Utility degradation: As penalties increase, low abstention with nonzero error rates produces rapidly worsening utility and nearly monotonic growth in mean normalized regret on HLE and GPQA.Penalty-normalized average points become strongly negative under high penalties.
- Prompting ablation: Explicitly instructing models to use confidence for abstention leaves abstention rates, normalized regret, and policy-consistency shifts nearly unchanged from baseline.The ablation supports rigidity beyond underspecified instructions.
- Scaffold with optimal policy: Post-hoc enforcement of the optimal policy using verbal confidence and penalty level improves utility across models and evaluation settings.The reported gains indicate that calibration signals can support better decisions when externally converted into an optimal policy.
4 Related Work
Related work studies verbal uncertainty calibration, abstention, and decision-theoretic behavior in LLMs. This paper differs by testing whether frozen models adapt abstention behavior to changing error penalties at inference time.
- Uncertainty quantification: Prior uncertainty-quantification work evaluates verbal confidence through calibration and correlation metrics, often improving confidence estimates or identifying complementary uncertainty signals.These approaches generally emphasize confidence reliability rather than downstream decision behavior.
- Abstention and selective prediction: Research on LLM abstention addresses safety, hallucination, and reliability through alignment- and inference-stage methods.RiskEval instead examines a frozen LLM’s penalty-sensitive abstention behavior during inference.
- Decision-theoretic evaluation: Decision-theoretic studies measure risk attitudes, loss aversion, and deviations from rational or human benchmarks under varied framings.The paper extends this area by testing strategic adaptation of answering and abstention to changing error penalties.
5 Discussions and Conclusion
The discussion identifies a disconnect between LLMs’ uncertainty information and their actions: models can verbalize uncertainty but fail to use it to minimize loss. The authors argue that trustworthy systems must align confidence with risk-sensitive decisions.
- Information and action: Models often accurately verbalize uncertainty yet fail to use that information to minimize loss, revealing a fundamental disconnect between information and action.Under severe penalties, this disconnect leads to utility collapse.
- Implications: The authors propose that reliable agents should maintain consistency between internal beliefs and external actions, especially when errors carry real-world costs.They identify direct risk-sensitive training and inference-time enforcement of optimal decision boundaries as possible future directions.
Limitations
The evaluation is constrained by reliance on verbalized confidence and by its focus on benchmarks with verifiable correctness. These boundaries limit how broadly the confidence–abstention dynamics can be generalized.
- Dependency on Verbalized Confidence: The analysis uses verbalized confidence as a proxy for the model’s internal belief, which may not perfectly capture true epistemic uncertainty.For proprietary API models, restricted access to raw log-probabilities or activations makes verbal elicitation a necessary constraint.
- Dependency on Verbalized Confidence: A model’s true internal probability could be better aligned with its decision to answer even when its verbalized output is decoupled.
- Scope of Tasks: The evaluation covers HLE, GPQA, and GSM8K, so confidence and abstention dynamics may differ in open-ended or subjective tasks.In those settings, utility and penalty are subjective and correctness boundaries are less defined.
A Details of Evaluation Metrics
This section defines the utility, decision, regret, calibration, and outcome metrics used to evaluate whether model decisions follow confidence-based optimal policies under varying penalties.
- Utility and Optimal Decisions: The optimal policy answers when confidence exceeds τ(λ) = λ/(1 + λ), the point where expected answering utility reaches zero.As penalties increase, the threshold for answering rises.
- Regret and Decision Metrics: Regret measures the difference between optimal expected utility and the utility achieved by the model’s chosen action.The analysis considers wrongful answers from overconfidence and wrongful abstentions from underconfidence.
- Regret and Decision Metrics: Normalized regret removes the linear penalty scaling and captures the confidence distance from the optimal boundary when decisions are suboptimal.It is defined as |c − τ(λ)| when the model’s policy differs from the optimal policy.
- Metric Families: Outcome metrics include abstention rate, answered-query accuracy, and normalized average utility, while decision metrics include policy consistency and regret.Calibration metrics separately assess whether verbalized confidence reflects correctness, using measures such as ECE and Brier score.
- Metric Families: Calibration quality is evaluated independently of the decision threshold using confidence-based measures including ECE-10 and Brier score.ECE-10 partitions [0, 1] into 10 equal-width bins; lower Brier scores indicate better calibration and refinement.
- Illustrative Calibration Analysis: Figure 5 tracks verbalized confidence, AUARC, ECE, and Brier across HLE, GPQA, and GSM8K as λ increases.The figure caption states that confidence and calibration quality remain stable across penalty levels.
B.1 Evaluation Details
RiskEval evaluates penalty-sensitive answering and abstention through isolated solver, parser, and judge stages across multiple models and benchmark datasets. Penalties are explicitly varied in prompts while confidence is parsed and analyzed without post-hoc recalibration.
- Models and Datasets: The evaluation includes API, open instruct, and reasoning models across GPQA Diamond, HLE, and GSM8K.HLE and GSM8K use fixed subsets of 128 examples per penalty setting, while GPQA Diamond is evaluated in full.
- Evaluation Pipeline: The three-stage pipeline separates solver generation, parser extraction, and judge-based correctness assessment to prevent information leakage.The solver never observes ground-truth labels, optimal thresholds, or judge feedback.
- Evaluation Pipeline: The parser extracts the final decision, confidence in [0, 1], and committed answer text from heterogeneous model outputs.Numeric and verbal confidence expressions, including long-form reasoning traces, are supported.
- Prompting and Penalty Conditioning: Each prompt sets correct-answer utility to +1, incorrect-answer utility to −λ, and abstention utility to 0, with λ varied explicitly.The prompt remains fixed apart from the scalar penalty, and two strategies test whether confidence is used for abstention.
- Datasets and Modalities: The protocol excludes non-verifiable items from calibration metrics while retaining them for coverage and abstention statistics.Multimodal items are skipped when the evaluated model lacks vision support.
- Confidence Handling: Confidence may be verbal or numeric, is mapped deterministically to probability, and is treated as c = P(y = y* | x) without post-hoc recalibration.Calibration values are discretized into 10 equal-width bins independently for each penalty.
- Optimal Policy and Analysis: The optimal threshold τ(λ) is withheld from models and computed post hoc to test internal policy adaptation rather than rule following.Tables compare actual utility under πM with potential utility under confidence-based optimal policy π*.
- Execution and Reproducibility: Reproducibility is supported through deterministic settings, incremental per-item outputs, configuration files, raw outputs, and aggregated metrics.
B.2 Confidence and Calibration
Models’ verbal confidence and calibration metrics remain stable as penalties increase, indicating that uncertainty signals do not degrade under risk. This stability contrasts with the models’ failure to abstain appropriately.
- Implication: The failure to abstain is therefore not caused by degradation in the uncertainty signal itself.
- Confidence Stability: Verbalized confidence remains stable across penalty levels, preserving the model’s estimated correctness across HLE, GPQA, and GSM8K.The paper interprets this stability as confidence remaining tied to answer content rather than external incentives.
- Calibration Metrics: AUARC remains consistent as penalties increase, indicating that confidence continues to rank answers effectively under high-stress prompts.
- Calibration Metrics: ECE-10 and Brier scores show no significant degradation as the penalty increases.Frontier models are also described as well calibrated on simpler tasks such as GPQA and GSM8K.
B.3 Utility and Regret
As penalties rise, model utility deteriorates and normalized regret increases relative to the optimal policy. The gap is especially consequential when abstention should dominate but models continue answering.
- Overall Outcome: The worsening utility and increasing regret together expose a fundamental failure of risk-sensitive decision-making.
- Utility: Utility U collapses from near-zero or mildly positive values to strongly negative values as penalties enter moderate-to-high regimes.The collapse is most pronounced on HLE and GPQA Diamond, while GSM8K shows a similar but delayed pattern.
- Regret: Penalty-normalized regret R increases monotonically with λ across all datasets.This indicates progressively greater deviation from the utility-maximizing policy as error costs rise.
- Policy Gap: Optimal confidence-based behavior can achieve near-zero or positive utility where the model’s actual policy incurs large negative values.The comparison shows that much of the observed utility loss is avoidable through post-hoc abstention based on the model’s own confidence.
B.4 Ablation Study
Adding explicit instructions to use confidence for abstention produces negligible changes in confidence, abstention, regret, and policy consistency. The ablation therefore does not induce strategic risk adaptation.
- Prompting Intervention: The ablation adds an instruction telling models to use confidence to decide whether to answer or abstain to avoid penalties.
- Behavioral Outcomes: Abstention rate, normalized regret, and policy consistency remain nearly identical to baseline after explicit confidence instructions.Models continue answering despite the warning to use confidence to avoid penalties.
- Interpretation: The disconnection between confidence and action appears to be a deep behavioral prior that simple prompt engineering cannot easily override.
- Signal and Policy Changes: Changes in average confidence and policy consistency are negligible, mostly near 0.0, indicating little shift in internal thresholds or confidence distributions.