PaperlayerView exact revision on arXiv

Source-linked AI summary

On Verbalized Confidence Scores for LLMs

Daniel Yang, Yao-Hung Hubert Tsai, Makoto Yamada

arXiv:2412.14737v2cs.CL

TL;DR

LLM responses lack trust indicators, while confidence scores could help users judge reliance and agents account for one another’s uncertainty. This paper evaluates verbalized confidence across datasets, models, and prompts, finding that reliability depends strongly on how confidence is elicited. Its scope is limited to objective response correctness and the evaluated settings.

  • Problem

    LLM responses arrive without the trust indicators available for forum answers or search results, motivating confidence scores that could help users assess reliance.

    Source: §1 · p. 1

  • Method

    The paper analyzes LLM-verbalized confidence across datasets, models, and prompt methods, assessing score reliability using calibration, meaningfulness, and informativeness concepts.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Reliability depends strongly on prompting: tiny models favor simple formulations, while large models benefit from more complex methods; the large-model combo averages 7% deviation from empirical accuracy.

    Source: §6 · p. 10, §5.4 · p. 10

  • Takeaways & Limitations

    Verbalized confidence is a promising, low-overhead, prompt- and model-agnostic uncertainty-quantification approach, but its reliability depends on how it is elicited.

    Source: §6 · p. 10

    The evaluation concerns objective response correctness and is limited to the evaluated datasets and models, whose task, prompt-context, answer-type, and subjectivity diversity is limited.

    Source: §3.1 · p. 4, §6 · p. 10

Abstract

from arXiv · show

The rise of large language models (LLMs) and their tight integration into our daily life make it essential to dedicate efforts towards their trustworthiness. Uncertainty quantification for LLMs can establish more human trust into their responses, but also allows LLM agents to make more informed decisions based on each other's uncertainty. To estimate the uncertainty in a response, internal token logits, task-specific proxy models, or sampling of multiple responses are commonly used. This work focuses on asking the LLM itself to verbalize its uncertainty with a confidence score as part of its output tokens, which is a promising way for prompt- and model-agnostic uncertainty quantification with low overhead. Using an extensive benchmark, we assess the reliability of verbalized confidence scores with respect to different datasets, models, and prompt methods. Our results reveal that the reliability of these scores strongly depends on how the model is asked, but also that it is possible to extract well-calibrated confidence scores with certain prompt methods. We argue that verbalized confidence scores can become a simple but effective and versatile uncertainty quantification method in the future. Our code is available at https://github.com/danielyxyang/llm-verbalized-uq.

1 Introduction

LLM responses often lack trust indicators, while uncertainty scores could help people judge reliance and LLM agents make more informed decisions. This work examines verbalized confidence scores and how their reliability varies with the way they are elicited.

1 Introduction

  • Uncertainty scores can help users judge how much to rely on responses and help LLM agents account for other agents’ uncertainty.The introduction identifies reliability, prompt and model agnosticism, and low overhead as desired properties of uncertainty quantification.Source: §1 · p. 1
  • Existing approaches estimate uncertainty using sampled-response consistency, internal token logits, proxy models, or knowledge bases, but none meet all the stated requirements.The introduction says these methods can lack generalization, versatility, or scalability.Source: §1 · pp. 1–2
  • Verbalized confidence asks the LLM to state its confidence in its output, but prior findings disagree about its calibration and prompt effects remain underexplored.The approach needs only a prompt modification and a few extra generated tokens, but its reliability is contested.Source: §1 · p. 2
  • The paper analyzes verbalized-confidence reliability across datasets, models, and prompt methods, and contributes an uncertainty decomposition, a reliability specification, and evaluation code.It also reports insights into how dataset difficulty, model capacity, and prompt methods affect reliability.Source: §1 · p. 2

2 Related work

Prior uncertainty-quantification methods draw on external or internal knowledge, with different trade-offs in generalization, model dependence, and overhead. Verbalized confidence instead asks the LLM to express its own confidence, leaving reliability as an open question.

2 Related work

  • Proxy models, heuristics, human feedback, and knowledge bases are external-knowledge approaches, with trade-offs involving generalization, reliability, scalability, and tool overhead.Proxy-model capacity and training data affect reliability; human feedback is described as difficult to scale, while querying knowledge-base tools can be costly.Source: §2 · pp. 2–3
  • Sample consistency estimates confidence from multiple responses, is model-agnostic, and incurs overhead linear in response length.Its reliability depends on the function used to measure response similarity.Source: §2 · p. 3
  • Internal logits derive confidence from token logits, are not model-agnostic, and require choices about aggregation and which token probabilities to use.The cited discussion questions whether low-level token probabilities capture high-level semantic uncertainty.Source: §2 · pp. 3–4
  • Verbalized confidence asks the LLM to express confidence in its response; it is model-agnostic and adds only a constant amount of input and output tokens.Prior work has applied it to response correctness, “I know the answer,” explanations, and linguistic expressions of uncertainty.Source: §2 · p. 4
  • The paper identifies verbalized confidence as a principled approach whose reliability and potential for improvement still need analysis.Its categorization of uncertainty methods follows the knowledge source used to obtain confidence scores.Source: §2 · pp. 2, 4

3 Uncertainty quantification via verbalized confidence scores

The section defines the uncertainty this work aims to quantify and specifies criteria for assessing the reliability of verbalized confidence scores.

3.1 Uncertainty quantification

  • The paper frames LLM uncertainty as input, model, and output components, while noting this intuitive decomposition lacks rigorous, complete formalization.Source: §3.1 · p. 4
  • The evaluation focuses on output uncertainty in the objective correctness of responses, leaving open-ended, subjective, and long-form tasks outside its scope.Source: §3.1 · p. 4

3.2 Reliability of confidence scores

  • Calibration is the main reliability indicator: ECE averages confidence–accuracy gaps across 20 bins, while SmoothECE addresses bin sensitivity and complementary metrics capture other properties.ECE measures average calibration, not how well an individual score is calibrated.Source: §3.2 · p. 5
  • Informativeness measures the diversity of predicted confidence scores; the authors use score diversity to assess how expressively the LLM verbalizes uncertainty.They also discuss variance as an informativeness measure through the Brier-score decomposition.Source: §3.2 · p. 5
  • Meaningfulness measures how much confidence-score distributions depend on the dataset, using KL divergence under an assumption of diverse dataset difficulties.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate verbalized confidence reliability across datasets, models, and prompt methods. The section describes the evaluation scope and the dataset, model, and prompt choices.

4 Experiments

  • The evaluation spans 10 datasets, 11 LLMs, and 17 prompt methods.Source: §4 · p. 5

4.1 Datasets

  • Datasets use validation splits and are restricted to closed-book, closed-ended objective questions with well-defined correct answers.The restrictions avoid lengthy prompt-context overhead and make response correctness easier to determine.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Models

  • Models include instruction-tuned Gemma 1.1, Llama 3, Qwen 1.5, and OpenAI GPT-family models; Falcon and Mistral were excluded over response-format prompting difficulties.Source: §4.2 · p. 6

4.3 Prompt methods

  • Prompt comparisons vary score range and formulation, description detail, few-shot examples, best-guess or top-four requests, and chain-of-thought use.The evaluation includes 10 custom prompts and seven prompts from prior work.Source: §4.3 · pp. 6–7

5 Results

Across datasets and model sizes, verbalized confidence often remains high as accuracy falls, while reliability varies substantially with prompt method. The best-performing prompt differs by model size, and the combo method yields 7% average deviation from empirical accuracy for evaluated large LLMs.

5.1 Evaluation

  • Evaluation samples 1,000 examples per dataset with replacement over 10 seeds, reports 95% confidence intervals, and separates tiny from large models.gemma1.1-2b is excluded because its confidence is almost independent of accuracy.Source: §5.1 · p. 8, §5.3 · p. 9

5.2 Insights into datasets

  • Across datasets, accuracy ranges from 0.5 to 0.9, but confidence remains high as accuracy decreases, producing high calibration error.This pattern occurs for both tiny and large LLMs.Source: §5.2 · p. 8

5.3 Insights into models

  • Accuracy increases with model capacity, and most calibration improvement comes from higher accuracy rather than reduced overconfidence; among LLMs with at least 70 billion parameters, ECE is around 0.1.The smallest evaluated model, gemma1.1-2b, has verbalized confidence that is poorly calibrated and almost independent of its accuracy.Source: §5.3 · pp. 8–9

5.4 Insights into prompt methods

  • Tian et al. (2023)’s tian2023_top1 scores are better calibrated than Xiong et al. (2023)’s xiong2023_vanilla, particularly for large LLMs.This supports the finding that verbalized-score calibration depends heavily on how the model is asked.Source: §5.4 · p. 9
  • For tiny LLMs, probscore gives the best reliability improvement, while few-shot prompting, guess ranking, and combined methods significantly degrade reliability.The authors suggest tiny models benefit more from simple prompt methods, which slightly improve reliability.Source: §5.4 · p. 10
  • For large LLMs, combining probscore, advanced description, and few-shot prompting gives the best improvement; few-shot and guess ranking outperform simpler methods.The authors suggest large models benefit more from complex prompt methods.Source: §5.4 · p. 10
  • 7% average deviation from empirical accuracy: the combo prompt achieves this for evaluated large LLMs, with qualitative calibration improvement over basic.Figure 5 compares basic (left) and combo (right).Source: §5.4 · p. 10, Figure 5 · p. 10

6 Discussion

The authors conclude that verbalized confidence is a promising, low-overhead, prompt- and model-agnostic uncertainty-quantification approach, although reliability depends greatly on prompting. They identify limits in what uncertainty is measured and in the breadth of the evaluation, and outline abilities and future methods that remain to be developed.

Discussion

  • Verbalized confidence is presented as a versatile, low-overhead approach, but tiny models favor simple prompts while large models benefit from complex ones.Reliability is greatly influenced by how the model is asked.Source: §6 · p. 10

Limitations

  • The evaluation measures uncertainty in objective response correctness, which does not capture the LLM’s total uncertainty, and its findings may not transfer to other LLM families.Datasets lack diversity in task, prompt context, answer type, and answer subjectivity; the effectiveness of meaningfulness and informativeness metrics is also debatable.Source: §6 · p. 10

Future work

  • Future work should improve score-range expression, understanding of confidence relative to prompts and answers, and awareness of knowledge limits; real-world readiness remains unsatisfactory.Whether prompt engineering, prompt optimization, or finetuning can yield stronger improvements remains open.Source: §6 · p. 10

Broader impact statement

Reliable uncertainty quantification can support trust and better-informed decisions, but it also introduces risks that warrant ongoing assessment and mitigation.

Broader impact statement

  • Uncertainty estimates may be exploited by adversarial users or misused by malicious LLMs, although current methods are described as foundational and unlikely to cause significant harm in practice.The authors urge continuous risk assessment and proactive mitigation as the field progresses.Source: Broader impact statement · p. 11

A Prompt formulations

Appendix A gives formulations for task descriptions and uncertainty-quantification prompts, including variants that request confidence or probability alongside an answer.

A.1 Task descriptions

  • Table 4 covers task-description formulations, while Table 5 covers formulations for different prompt methods.The Table 4 text replaces <TASK DESCRIPTION> in Equation (5), and Table 5 text replaces <UQ PROMPT>.Source: Table 4 · p. 15, Table 5 · p. 15

A.2 Uncertainty quantification prompts

  • basic_probscore asks for the probability, from 0.0 to 1.0, that the answer is correct.It also specifies a NO ANSWER response when the model cannot answer.Source: §A.2 · pp. 16–17
  • basic_1shot and basic_5shot ask for a confidence percentage after the answer.The basic_1shot formulation measures confidence in the answer; the basic_5shot prompt uses the same instruction.Source: §A.2 · pp. 16–17
  • The advanced prompts ask for confidence or probability while considering uncertainty in the prompt, task difficulty, and knowledge availability.The advanced confidence formulation says the score should ideally correspond to empirical accuracy.Source: §A.2 · p. 18
  • combo requests only a best guess and its 0.0-to-1.0 probability, with no additional words or explanation..Source: §A.2 · p. 19
  • The Tian top-1 variants request a best guess and probability, with wording that varies across versions.Some versions ask for an answer rather than a guess, or label the probability as a confidence score.Source: §A.2 · p. 20
  • tian2023_top4 requests four best guesses and a separate 0.0-to-1.0 probability for each.The output template labels them G1–G4 and P1–P4.Source: §A.2 · p. 21
  • The Xiong vanilla prompt requests an answer and numerical confidence, while the CoT variant additionally requests step-by-step analysis.Both specify confidence as a percentage in the 0–100 range.Source: §A.2 · p. 21

B.1 Statistics on valid responses

Figure 6 reports valid-response rates across datasets by model and prompt method.

B.1 Statistics on valid responses

  • Figure 6 shows that gemma1.1-2b has low valid-response rates with basic_{1,5}shot and combo after responses copying a few-shot example’s confidence score are removed.The figure reports relative valid responses over all datasets for each model and prompt method.Source: Figure 6 · p. 22

B.2 Insights into datasets

The appendix presents calibration views across datasets and models, and compares prompt methods across several prompt aspects.

B.2 Insights into datasets

  • Figure 7 plots calibration per dataset using ECE, while Figure 8 gives a calibration diagram for gemma1.1-2b.The supplied captions do not state a dataset-specific calibration outcome.Source: Figure 7 · p. 22, Figure 8 · p. 23

B.3 Insights into models

  • Figure 8 notes that gemma1.1-2b accuracy is close to uniform across confidence-score ranges.The figure is a calibration diagram for that model.Source: Figure 8 · p. 23

B.4 Insights into prompt methods

  • Figures 9–13 assess calibration, informativeness, and meaningfulness across score range, score formulation, advanced description, few-shot prompting, and other aspects.The captions identify the metrics and prompt aspects but do not state which method performs best.Source: Figure 9 · p. 23, Figure 10 · p. 24, Figure 11 · p. 24

B.5 Principled calibration diagrams

Smooth calibration diagrams add a density-based view alongside binning-based diagrams, and the authors report a GPT-4o separation for combo.

B.5 Principled calibration diagrams

  • Because binning-based calibration diagrams are sensitive to the binning strategy, the authors also evaluate SmoothECE and provide smooth calibration diagrams.The SmoothECE diagrams use estimated density rather than the classical binned presentation.Source: §B.5 · p. 26, Figure 14 · p. 26
  • For GPT-4o, combo produces significant separation between confidence-score distributions for correct and incorrect responses, supporting the finding that prompt formulation affects calibration.Figure 14 shows accuracy conditioned on confidence in blue, with green and red areas for correct- and incorrect-response confidence densities, respectively.Source: Figure 14 · p. 26, §B.5 · p. 26

B.6 Comparison of calibration metrics and Brier score

The decomposed Brier score shows trends consistent with ECE while separating calibration from confidence informativeness. Most models have low resolution, with a marginal increase for combo prompting.

B.6 Comparison of calibration metrics and Brier score

  • The Brier decomposition's calibration term measures deviation from true conditional probabilities, resolution captures their variation across confidence levels, and uncertainty reflects inherent prediction difficulty.Source: §B.6 · p. 26
  • A lower Brier score need not mean better calibration because resolution also contributes; trivia_qa and truthful_qa-mc2 depart from the binary acc·(1−acc) uncertainty relation.Source: §B.6 · pp. 26–27
  • Most models produce concentrated confidence scores and generally low resolution; the combo prompt's marginal resolution increase suggests more discriminative confidence verbalization.Figure 15 compares ECE with the decomposed Brier score by dataset, model, and prompt method.Source: Figure 15 · p. 27, §B.6 · p. 27
  • Decomposed Brier trends are consistent with ECE across dataset, model, and prompt comparisons, while calibration measures squared rather than absolute deviations.Source: §B.6 · p. 27
Loading 2412.14737v2…