Source-linked AI summary
A Survey of Confidence Estimation and Calibration in Large Language Models
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, Iryna Gurevych
TL;DR
LLMs can generate factual errors, and existing surveys had not comprehensively organized technical work on confidence estimation and calibration. This survey reviews challenges and recent methods across generation and classification, including semantic and linguistic approaches, and discusses applications and future directions. It does not provide empirical validation because it contains no original experiments.
Problem
Existing surveys focused mainly on hallucination and factuality, leaving no comprehensive systematic overview of technical advances in LLM confidence estimation and calibration.
Method
The survey synthesizes concepts, challenges, methods, applications, and future directions for confidence estimation and calibration in LLM generation and classification.
Results
The survey identifies distinctive LLM challenges and reviews recent methods including semantic confidence estimation, linguistic confidence, and calibration approaches for generation and classification.
Takeaways & Limitations
Confidence estimation and calibration are presented as important components for addressing errors and biases and supporting more reliable AI applications.
Takeaways & Limitations
Without original experiments, the survey cannot offer empirical validation of its theories or concepts.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks in various domains. Despite their impressive performance, they can be unreliable due to factual errors in their generations. Assessing their confidence and calibrating them across different tasks can help mitigate risks and enable LLMs to produce better generations. There has been a lot of recent research aiming to address this, but there has been no comprehensive overview to organize it and outline the main lessons learned. The present survey aims to bridge this gap. In particular, we outline the challenges and we summarize recent technical advancements for LLM confidence estimation and calibration. We further discuss their applications and suggest promising directions for future work.
1 Introduction
LLMs are capable but can still produce biased, hallucinated, and unstable outputs, making trustworthy response evaluation difficult. This survey organizes confidence estimation and calibration research for LLM generation and classification tasks to address this gap.
- LLMs exhibit biases and hallucinations, while evaluating the trustworthiness of their responses remains challenging.
- Confidence estimation supports tasks such as out-of-distribution detection and selective prediction, while calibration aligns confidence with actual accuracy.
- LLM confidence methods must handle exponentially large output spaces, semantic equivalence among expressions, verbalized confidence, and prompt sensitivity.
- The survey fills a gap by systematically reviewing LLM confidence estimation and calibration techniques, applications, and future research directions.
- The survey distinguishes confidence estimation and calibration research for generation and classification tasks.
- These topics are presented as contributing to more reliable applications.
2 Preliminaries and Background
The preliminaries define confidence, uncertainty, and calibration, then introduce metrics and methods for discriminative models and semantic token sequences. They also note that standard calibration measures have important limitations when applied to LLM outputs.
- Basic Concepts: Higher confidence corresponds to lower uncertainty, and confidence scores may be interpreted as relative or absolute measures.
- Basic Concepts: Absolute confidence is calibrated when predicted probabilities align with observed frequencies under similar circumstances.
- Basic Concepts: Aleatoric uncertainty reflects irreducible system variability, whereas epistemic uncertainty arises from limited knowledge and can diminish with more data or better modeling.
- Metrics and Methods: Expected calibration error estimates calibration by binning similar confidence scores and averaging discrepancies between mean confidence and accuracy.
- Metrics and Methods: ECE is sensitive to bucket width and within-bucket sample variance, motivating alternatives such as SCE, ACE, and classwise ECE.
- Metrics and Methods: For generated sequences, calibration metrics must process token sequences with semantics rather than treating outputs as isolated labels.
- Metrics and Methods: Discriminative-model confidence estimation includes logit-based, ensemble or Bayesian, density-based, and confidence-learning methods, with calibration applied during or after training.
3 LLMs for Generation Tasks
LLM confidence estimation methods are organized into white-box and black-box families, depending on whether model parameters or internal states are accessible. Recent approaches combine information sources to improve estimation, while trading off computational cost and assumptions about model similarity.
- White-Box Methods: White-box methods assume access to the LLM’s state at every inference position.
- White-Box Methods: P(True) estimates confidence from the probability assigned when the LLM evaluates its generated answer as “True” or “False”.The method relies on the model’s ability to self-assess whether answers are correct.
- White-Box Methods: White-box methods use logits, internal states, and semantics as confidence-information sources.Logit methods are easy to implement but can conflate low probabilities with language properties; internal-state methods typically require supervised annotated data, while semantics add interpretability.
- White-Box Methods: Combining logit-based estimation with semantic tools such as bidirectional entailment or sentence encoders achieves strong uncertainty estimation for open-domain question answering.Other combinations use rephrasing, round-trip translation, or self-evaluation, but broader information use increases computational demands and requires balancing performance with resource efficiency.
- Black-Box Methods: Black-box methods use generations without access to model parameters, relying on linguistic confidence, consistency, or surrogate models.Consistency methods compare generations and can detect potential hallucinations, while surrogate models apply white-box techniques to target models under approximate parameter-distribution assumptions.
- Calibration: Calibration work covers generation tasks and classification tasks, with the first half of Table 2 focused on generation-task calibration.
4 LLMs for Classification Tasks
LLMs support classification through in-context learning, but confidence calibration must address context and label-position biases, surface-form competition, and differing evaluation protocols.
- In-Context Learning: In-context learning performs classification from a few input-label demonstrations and their surrounding context.
- Calibration Methods: Contextual calibration estimates bias from context-free prompts and transforms predictive scores to offset miscalibration.
- Calibration Methods: Few-shot performance can be improved by combining k-nearest-neighbor classifiers with cached representations or Gaussian mixture models for decision boundaries.
- Multiple-Choice Question Answering: Multiple-choice question answering evaluates prompted LLMs over predefined options, often using demonstrations as instructions.
- Multiple-Choice Question Answering: Evaluation protocols can substantially change model rankings because implementations score option identifiers, the full vocabulary, or length-normalized probabilities differently.
- Summary: Classification calibration primarily mitigates label or choice-position biases, while current work increasingly examines in-context learning and semantics; systematic benchmarking remains missing.
5 Applications
Confidence estimation and calibration support applications that detect hallucinations, handle ambiguity, guide data use, and improve knowledge retrieval for LLMs.
- Hallucination Detection and Mitigation: Confidence or uncertainty can signal hallucinations, with methods using self-consistency, cross-model consistency, or logits followed by validation and repair.
- Hallucination Detection and Mitigation: Reliable LLMs are expected to refrain from arbitrary answers on ambiguous or unanswerable questions, motivating selective generation and clarification methods.
- Uncertainty-Guided Data Exploitation: Uncertainty measurements can select representative instances for few-shot learning or human annotation.
- Uncertainty-Guided Data Exploitation: Low-confidence upcoming tokens can stimulate adaptive multi-retrieval of relevant documents for knowledge enhancement.
6 Future Directions
Future work requires broader evaluation, attention to long-form generation, validation for multimodal models, and calibration that reflects human disagreement.
- Comprehensive Benchmarks: Comprehensive benchmarks across tasks and domains are needed to evaluate the robustness and utility of confidence estimation and calibration.
- Comprehensive Benchmarks: Annotating long-form LLM responses requires extensive human effort, motivating confidence estimation and calibration by parts rather than over entire generations.
- Multi-modal LLMs: It remains unclear whether existing confidence estimation methods work effectively or yield well-calibrated predictions for multimodal large language models.
- Calibration to Human Variation: LLM calibration measures can misalign with human disagreement arising from task ambiguity, annotator subjectivity, and input ambiguity.
7 Conclusion
The survey organizes confidence estimation and calibration for LLMs by introducing foundations, identifying LLM-specific challenges, reviewing methods, and discussing applications and future directions.
- Confidence estimation and calibration are presented as important for addressing errors and biases in LLMs.
- The survey introduces fundamental concepts, metrics, estimation methods, and calibration techniques from traditional discriminative models before identifying their challenges for LLMs.
- It reviews recent methods for generation and classification tasks, including their principles, advantages, and drawbacks.
- The survey concludes by discussing current applications and future research directions.
Limitations
The survey has no original experimental benchmarks, limiting empirical validation and the production of new verified knowledge. It also may omit important work because the field is developing rapidly.
- Without original experiments, the survey cannot empirically validate its theories or concepts.
- The absence of experimental benchmarks limits the paper’s contribution of new, verified knowledge.
- Rapid development in the field means that some important work may have been overlooked despite efforts to compile recent advances.
Ethical Considerations and Potential Risks
The authors anticipate no significant ethical concerns because the survey involved neither experiments nor risky datasets, and used no manual-annotation workers.
- The authors report no significant ethical concerns for this review.
- The work did not conduct experiments, use risky datasets, or employ workers for manual annotation.
A.1 Confidence Estimation Methods
Confidence estimation methods include logit-based, ensemble or Bayesian, density-based, and confidence-learning approaches, while calibration methods operate during training or after training. The survey highlights trade-offs involving interpretability, computational cost, distributional assumptions, model modification, and calibration performance.
- Confidence Estimation Methods: Logit-based estimation directly uses the predicted probability associated with the model’s highest-probability output.
- Confidence Estimation Methods: Probability transformations estimate confidence using quantities such as the gap between the top two probabilities or entropy.
- Confidence Estimation Methods: Ensemble and Bayesian methods estimate uncertainty from prediction variability across independently trained models or repeated dropout-enabled forward passes.
- Confidence Estimation Methods: Ensemble and Bayesian approaches impose heavy computational overhead, motivating sampling-efficiency improvements such as determinantal point processes for MCDropout.
- Confidence Estimation Methods: Density-based methods associate dense training-data regions with higher confidence and sparse regions with greater uncertainty.
- Confidence Estimation Methods: Confidence learning adds a network branch that learns task-relevant confidence, requiring network modification and specialized training.
- Model Calibration: Temperature scaling learns a scalar τ on a validation set to re-weight logits before softmax, offering low complexity and efficiency.
- Model Calibration: Post-hoc calibration is generally model-independent and can adjust probabilities without affecting model performance.