PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, Bryan Hooi

arXiv:2306.13063v2cs.CL

TL;DR

Accurate LLM confidence matters for trustworthy decision-making, but existing methods often require internal access or fine-tuning. The paper evaluates a black-box framework combining prompting, sampling, and aggregation across models and tasks, finding persistent verbalized overconfidence alongside improvements from selected strategies.

  • Problem

    Existing confidence-elicitation methods commonly require internal model information or fine-tuning, limiting their suitability for closed-source LLM APIs.

  • Method

    The study defines a black-box framework combining prompting, sampling, and aggregation strategies, then benchmarks it across confidence calibration and failure prediction tasks.

  • Results

    LLMs are often overconfident when verbalizing confidence, while human-inspired prompting and multiple-response consistency can improve calibration or failure prediction.

  • Takeaways & Limitations

    Black-box confidence elicitation provides a useful baseline, but aggregation choices should reflect whether calibration or failure prediction is prioritized.

  • Takeaways & Limitations

    The methods still struggle with incorrect-prediction detection on challenging tasks requiring professional knowledge, such as professional law.

Abstract

from arXiv · show

Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.

1 INTRODUCTION

Reliable confidence estimates support informed decision-making, but existing approaches often require internal model access or fine-tuning. This study therefore explores black-box confidence elicitation through a systematic framework and broad benchmark.

  • Reliable confidence estimates can support risk assessment, error mitigation, selective generation, and reduced hallucinations.
  • White-box confidence methods commonly use token-likelihoods, calibration techniques, or model fine-tuning.
  • Closed-source commercial APIs restrict access to token-likelihoods and embeddings, while token-likelihood may miss semantic uncertainty and fine-tuning can require substantial resources.
  • The study defines prompting, sampling, and aggregation components for black-box confidence elicitation and benchmarks them on calibration and failure prediction.The benchmark covers five task types and five widely used LLMs.
  • LLMs tend to be overconfident when verbalizing confidence, while human-inspired prompts, multiple-response consistency, and aggregation strategies can mitigate this tendency.The study also reports improved performance with increasing model capacity and metric-specific aggregation effects.

2 RELATED WORKS

Prior confidence-elicitation work has often depended on fine-tuning or inaccessible internal representations. The paper positions its broader black-box framework as an extension of prompting-focused research and illustrates how component choices form algorithms.

  • Confidence elicitation estimates an LLM’s confidence without fine-tuning or access to internal information.
  • Earlier approaches include fine-tuned verbalized confidence and external calibrators based on internal model representations, while zero-shot verbalized confidence remains underexplored.
  • This work broadens prior prompting-focused research by exploring a wider method space, integrating strategies systematically, and evaluating more model types.
  • The framework combines prompt, sampling, and aggregator strategies into distinct confidence-elicitation algorithms.For example, Top-K uses a Top-K prompt, self-random sampling with M = 1, and Avg-Conf aggregation.

3 EXPLORING BLACK-BOX FRAMEWORK FOR CONFIDENCE ELICITATION

The proposed black-box framework decomposes confidence elicitation into prompting, sampling, and aggregation. It combines human-inspired confidence prompts with multiple-response agreement and confidence aggregation to produce alternative algorithms.

  • 3.1 MOTIVATION OF THE FRAMEWORK: The framework organizes black-box confidence elicitation into prompting, sampling, and aggregation components with multiple algorithmic choices.
  • 3.1 MOTIVATION OF THE FRAMEWORK: Prompting strategies elicit semantic verbalized confidence, while response variance provides an additional confidence signal.
  • 3.2 PROMPTING STRATEGY: Human-inspired prompts are designed to address overconfidence in vanilla verbalized confidence, and every prompt explains confidence as the likelihood that the answer is true.
  • 3.2 PROMPTING STRATEGY: Chain-of-Thought adds reasoning processes, whereas Self-Probing asks for confidence in an answer generated in an independent chat session.
  • 3.2 PROMPTING STRATEGY: Multi-Step extracts confidence for individual reasoning steps and aggregates those values into overall verbalized confidence.
  • 3.3 SAMPLING STRATEGY: Top-K prompts the model to generate multiple guesses with corresponding confidence values, while sampling can use self-randomness, paraphrasing, or misleading cues.
  • 3.4 AGGREGATION STRATEGY: Consistency aggregation measures agreement between sampled candidate answers and an original answer to estimate confidence.
  • 3.4 AGGREGATION STRATEGY: Avg-Conf combines verbalized confidence with consistency, while Pair-Rank uses Top-K ordering because rankings may be more accurate than overconfident verbalized scores.

4 EXPERIMENT SETUP

The study evaluates confidence estimates across five reasoning-task types and five LLMs using calibration and failure-prediction metrics. It also visualizes vanilla verbalized confidence on GSM8K.

  • Datasets: The benchmark covers commonsense, arithmetic, symbolic, professional-knowledge, and ethical-knowledge reasoning tasks.Datasets include SportUND, StrategyQA, GSM8K, SVAMP, DateUnd, ObjectCou, Prf-Law, and Biz-Ethics.
  • Vanilla Verbalized Confidence: Figure 2 shows vanilla confidence values concentrated between 80% and 100%, often in multiples of 5, with bin accuracy much lower than confidence.
  • Models: The evaluated models are Vicuna 13B, GPT-3 175B, GPT-3.5-turbo, GPT-4, and LLaMA 2 70B.
  • Evaluation Metrics: Calibration measures alignment between expressed confidence and accuracy, while failure prediction measures the ability to distinguish correct from incorrect responses.

5 EVALUATION AND ANALYSIS

The evaluation finds that verbalized confidence is often overconfident, while model scaling, prompting, response consistency, and aggregation choices improve different aspects of confidence estimation. However, failure prediction remains difficult and no single strategy consistently dominates.

  • 5.1 LLMS TEND TO BE OVERCONFIDENT WHEN VERBALIZING THEIR CONFIDENCE: 22.2% improvement in AUROC is observed from GPT-3 to GPT-4 as model capacity and accuracy increase.ECE decreases while AUROC increases across GPT-3, Vicuna, GPT-3.5, and GPT-4.
  • 5.1 LLMS TEND TO BE OVERCONFIDENT WHEN VERBALIZING THEIR CONFIDENCE: Verbalized confidence is concentrated at 80%–100% and is significantly overconfident, while vanilla failure prediction remains poor.GPT-4’s average AUROC is 62.7%, close to the 50% random-guess threshold.
  • 5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE: Human-inspired prompting consistently improves accuracy and calibration over vanilla prompting, but gains are modest for failure prediction and diminish for GPT-4.
  • 5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE: No prompting strategy consistently outperforms the others across datasets and models; Self-Probing is most consistent on GPT-4, whereas Top-K performs best on GPT-3.5.
  • 5.2 HUMAN-INSPIRED PROMPTING STRATEGIES PARTIALLY REDUCE OVERCONFIDENCE: Reducing ECE does not ensure better failure prediction: AUROC remains near 0.5, and GPT-4 reaches ECE 0.064 on GSM8K while assigning 100% confidence to every sample.
  • 5.3 VARIANCE AMONG MULTIPLE RESPONSES IMPROVES FAILURE PREDICTION: 92.7% AUROC on GSM8K versus 54.8% with one response shows that consistency across five sampled responses substantially improves failure prediction.The same sampling-and-aggregation approach also improves average ECE and AUROC relative to verbalized confidence with M = 1.
  • 5.3 VARIANCE AMONG MULTIPLE RESPONSES IMPROVES FAILURE PREDICTION: Increasing sampled responses improves ECE and AUROC before gains become marginal, creating a trade-off because computation grows linearly with M.
  • 5.4 INTRODUCING VERBALIZED CONFIDENCE INTO THE AGGREGATION OUTPERFORMS CONSISTENCY-ONLY AGGREGATION: Pair-Rank reduces average ECE to 0.028, whereas Avg-Conf performs best for failure prediction.Pair-Rank learns a categorical distribution over observed answers, while Avg-Conf uses verbalized confidence to weight consistency.

6 DISCUSSIONS

The study establishes a black-box framework for confidence elicitation and finds that proposed strategies improve some aspects of confidence estimation, but important challenges remain.

  • The framework combines prompting, sampling, and aggregation strategies to benchmark black-box confidence elicitation across eight datasets and five models.
  • White-box methods generally perform better, but the gap over black-box methods is relatively modest, such as 0.522 to 0.605 in AUROC.
  • The best-performing algorithms can reduce ECE to 0.028 but still struggle to predict incorrect answers, especially on professional-knowledge tasks such as professional law.
  • For practitioners, the authors recommend Top-K prompting with Self-Random sampling and Avg-Conf or Pair-Rank aggregation as a stable-performing combination.
  • The evaluation mainly covers fixed-form and freeform question-answering tasks with unique ground-truth answers, while black-box approaches remain suboptimal.

A PROOF OF PROPOSITION 3.1

The proof analyzes the probability that one candidate answer is sampled before another by conditioning on the earliest position where either appears and summing over preceding sequences.

  • Each response contains K sequentially ordered answers, and the unique-answer set A contains all distinct answers across the N responses.
  • The event (Su ≻ Sv) denotes that Su is ranked above Sv in a generation, while E(i)_uv denotes that at least one of the two appears.
  • The proof conditions on Fj, the earliest position j where either Su or Sv appears, and sums the probability over possible positions.
  • For each position, Hj−1 represents feasible sequences of the first j−1 answers excluding Su and Sv before the pairwise comparison.
  • The derivation decomposes the conditional probability into preceding-sequence and pairwise-order terms, then uses the Law of Total Probability and an optimization objective.

B.1 WHITE-BOX METHODS OUTPERFORM BLACK-BOX METHODS, BUT THE GAP IS NARROW.

The comparison finds that white-box token-probability methods generally outperform black-box verbalized confidence, but the performance gap is relatively modest and both remain imperfect.

  • The analysis compares token-probability white-box methods with black-box models using verbalized confidence.
  • The white-box implementations include sequence probability, length-normalized sequence probability, and key token probability.
  • Length-normalized sequence probability and key token probability are generally the most effective white-box methods across five datasets and four evaluation metrics.
  • White-box methods perform better overall, but the gap is relatively modest, and even the strongest white-box methods do not achieve satisfactory results.
  • Different confident and cautious persona prompts produce minimal differences in confidence distributions and performance.

B.3 HOW IS THE DISTRIBUTION OF VANILLA VERBALIZED CONFIDENCE ACROSS MODELS AND DATASETS?

Vanilla verbalized confidence is concentrated at high, rounded values, while alternative prompting and sampling strategies are evaluated for improving calibration and failure prediction.

  • Across four models and five datasets, vanilla confidence values are mostly multiples of 5 between 80% and 100%, resembling human confidence expressions.
  • Top-K prompting is relatively more effective than Multi-Step prompting for reducing ECE and improving AUROC.
  • Top-K prompting requests multiple guesses, which can prevent overconfidence in a single response and encourage answer re-evaluation.
  • The compared methods include vanilla, CoT, Multi-Step, and Top-K verbalized confidence, evaluated using ECE and AUROC.
  • Weak-Claim misleading prompts perform better, whereas highly confident or externally sourced misleading cues can cause the model to alter predictions.
  • Larger candidate sets improve AUROC and reduce variance, but the K that optimizes ECE varies across datasets.

C RELATED WORKS

Confidence elicitation research has begun addressing zero-shot, black-box settings, but existing work remains limited by reliance on fine-tuning or inaccessible internal representations. Related work also examines calibration and prompting, while this study focuses on directly eliciting confidence.

  • Zero-shot verbalized confidence remains underexplored because prior approaches often rely on fine-tuning or inaccessible model representations.
  • Black-box confidence elicitation estimates LLM confidence without fine-tuning or access to proprietary information.
  • This paper studies prompting strategies for directly eliciting verbalized confidence in black-box settings.
  • Prompting-based related work includes studies of confidence in prompts, although some approaches do not directly provide confidence estimates to users.
  • Calibration research seeks to align model confidence with observed accuracy, addressing the overconfidence often found in modern neural networks.

D.1 WHAT IS THE RECOMMENDATION FOR PRACTITIONERS?

The paper recommends combining Top-K prompting, Self-Random sampling, and Avg-Conf or Pair-Rank aggregation for a balance of effectiveness, simplicity, and efficiency. It also explains why multiple responses and confidence-aware aggregation help, while cautioning that confidence remains difficult to interpret reliably.

  • Recommendation: Top-K prompt + Self-Random sampling + Avg-Conf or Pair-Rank aggregation is recommended as a stable-performing practitioner combination.
  • Recommendation: Top-K outperforms other methods on GPT-3.5 and is comparable to Self-Probing on GPT4, while requiring only one inference phase.
  • Aggregation: Aggregation using answers and verbalized confidences, such as Avg-Conf and Pair-Rank, outperforms answer-only consistency when sampled queries are constrained.
  • Considerations: Confidence reliability requires multiple metrics because a high ECE does not imply that outputs accurately represent correctness.
  • Sampling: Sampling multiple responses is more effective than verbalized confidence with M = 1, particularly on arithmetic tasks, because it approximates the model’s internal distribution.
  • Prompting: Top-K, CoT, and Multi-Step prompts significantly reduce ECE in ChatGPT, largely by improving accuracy rather than substantially improving discrimination.

E.1 DATASETS

The evaluation covers five task types—commonsense, arithmetic, symbolic, professional, and ethical knowledge—using representative datasets and multiple LLMs. Confidence quality is assessed through calibration and failure-prediction metrics under standardized prompting and sampling settings.

  • Datasets: The benchmark includes commonsense, arithmetic, symbolic, professional, and ethical knowledge tasks.
  • Datasets: Commonsense reasoning uses SportUND and StrategyQA, with StrategyQA selected as the representative dataset because it contains more data.
  • Datasets: Arithmetic reasoning uses GSM8K and SVAMP, with GSM9K selected as representative because it is more widely used.
  • Datasets: Symbolic reasoning uses DateUnd and ObjectCou, with Date Understanding selected as representative because it is more difficult.
  • Datasets: Professional and ethical knowledge are represented by Professional Law and business ethics datasets from MMLU.
  • Evaluation: Confidence estimates are evaluated using calibration and failure-prediction metrics.
  • Evaluation: ECE measures confidence–accuracy alignment, while AUROC and AUPRC assess discrimination between correct and incorrect answers.
  • Models: Experiments include Vicuna, GPT3, GPT3.5, and GPT4, spanning models from 13 billion parameters to larger commercial systems.
Loading 2306.13063v2…