PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions

Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, Anirudha Majumdar

arXiv:2412.05563v2cs.CLcs.AI

TL;DR

LLMs can produce plausible factual errors with high confidence, motivating the need for reliable uncertainty estimates. This survey organizes LLM UQ methods into a taxonomy, reviews applications and challenges, and identifies limitations including model dependence and continued failures of verbal confidence. It also discusses how repeated incorrect answers can remain consistent rather than factual.

  • Problem

    LLMs generate plausible, factually incorrect responses, while existing confidence expression can remain unreliable, raising questions about their trustworthiness.

  • Method

    The survey comprehensively reviews LLM UQ methods, categorizes them into four broad classes, and covers applications, benchmarks, and challenges.

  • Results

    The survey’s supported conclusion is an organized account of LLM UQ methods spanning token-based, black-box semantic-similarity, and mechanistic-interpretability approaches, alongside applications and challenges.

  • Takeaways & Limitations

    UQ is presented as relevant to assessing factuality and to applications ranging from textual systems to embodied robotics.

  • Takeaways & Limitations

    Semantic-similarity UQ outputs are generally model-dependent, posing a central challenge.

Abstract

from arXiv · show

The remarkable performance of large language models (LLMs) in content generation, coding, and common-sense reasoning has spurred widespread integration into many facets of society. However, integration of LLMs raises valid questions on their reliability and trustworthiness, given their propensity to generate hallucinations: plausible, factually-incorrect responses, which are expressed with striking confidence. Previous work has shown that hallucinations and other non-factual responses generated by LLMs can be detected by examining the uncertainty of the LLM in its response to the pertinent prompt, driving significant research efforts devoted to quantifying the uncertainty of LLMs. This survey seeks to provide an extensive review of existing uncertainty quantification methods for LLMs, identifying their salient features, along with their strengths and weaknesses. We present existing methods within a relevant taxonomy, unifying ostensibly disparate methods to aid understanding of the state of the art. Furthermore, we highlight applications of uncertainty quantification methods for LLMs, spanning chatbot and textual applications to embodied artificial intelligence applications in robotics. We conclude with open research challenges in uncertainty quantification of LLMs, seeking to motivate future research.

1 Introduction

LLMs can generate highly capable but factually incorrect responses while expressing unwarranted confidence, creating risks for trustworthy deployment. This survey organizes uncertainty quantification (UQ) methods to help estimate response confidence and guide safer use across applications.

  • LLMs produce plausible, factually incorrect responses, or hallucinations, while users may mistake them for factual, creating safety risks.
  • LLMs often fail to communicate uncertainty accurately and remain overconfident when response factuality is doubtful.
  • Uncertainty quantification estimates an LLM’s confidence in a response, supporting factuality analysis and decisions such as human guidance or further review.
  • The survey addresses the fragmented UQ literature with a taxonomy intended to clarify research scope, available guarantees, and useful methods for practitioners.
  • It groups LLM UQ methods into token-level, self-verbalized, semantic-similarity, and mechanistic-interpretability classes, covering multi-claim and multi-sentence responses.
  • The survey also identifies shared features within categories and open research challenges intended to motivate future work.

Comparison to other Surveys

Existing surveys provide limited coverage of LLM uncertainty quantification, while this survey offers a broader taxonomy tailored to LLM architectures and applications. It additionally emphasizes mechanistic interpretability and embodied applications such as robotics.

  • Earlier hallucination surveys discuss hallucination types, causes, and mitigation but provide little discussion of LLM uncertainty quantification.
  • The earlier UQ survey organizes confidence estimation and calibration by generation versus classification tasks, emphasizing calibration more than confidence estimation.
  • This survey provides an extensive review of UQ methods with briefer coverage of calibration techniques.
  • It discusses mechanistic interpretability in detail and considers its potential applications to LLM uncertainty quantification.
  • Its application coverage extends to embodied settings such as robotics and is organized around LLM transformer and autoregressive token-based characteristics.

Organization

The survey progresses from foundational uncertainty concepts and metrics through four UQ method categories, calibration, benchmarks, applications, and open challenges. Figure 2 summarizes this taxonomy and the survey’s broader organization.

  • The survey begins with background on uncertainty, deep-learning UQ categories, and metrics used by many LLM UQ methods.
  • Later sections discuss calibration, datasets and benchmarks, applications, open challenges, and future research directions.
  • Figure 2 provides an overview spanning the taxonomy, relevant datasets and benchmarks, applications, and future directions.

2 Background

This background defines uncertainty in LLM responses, distinguishes its major forms, and motivates uncertainty quantification as a way to assess hallucinations and model confidence. It then situates UQ methods along a training-based to training-free spectrum and introduces representative training-based approaches.

  • 2.1 Uncertainty: Uncertainty is treated as closely related to confidence, although some prior work distinguishes prediction-associated confidence from prediction-independent uncertainty.
  • 2.1 Uncertainty: LLMs can produce coherent but false answers, and may remain confidently incorrect even when asked to assess their confidence.Examples include fabricated information about a possibly fictional person and an incorrect cookbook attribution.
  • 2.2 Types of Uncertainty: Uncertainty quantification estimates response confidence using signals such as token-distribution entropy and supports factuality analysis.The survey notes that token-based UQ methods generally yield better-calibrated uncertainty estimates in existing empirical studies.
  • 2.2 Types of Uncertainty: Aleatoric uncertainty reflects irreducible randomness, whereas epistemic uncertainty reflects knowledge gaps and can decrease with additional relevant training data.Predictive uncertainty combines these categories without distinguishing their sources.
  • 2.3 Uncertainty Quantification in Deep Learning: Deep-learning UQ spans training-based methods that probe model internals and training-free methods that use auxiliary models or additional data post-hoc.
  • 2.3.1 Training-Based Methods: Training-based approaches include Bayesian neural networks, Monte Carlo methods, deep ensembles, pruning, and distillation, but approximations can reduce fidelity and full Bayesian methods can be computationally prohibitive.Variational inference approximates the posterior, while pruning and distillation reduce ensemble computation or compress ensemble knowledge into fewer networks.

2.4 Uncertainty Quantification for LLMs

LLM uncertainty quantification is motivated by hallucinations and supports factuality analysis. The survey organizes methods into four categories while distinguishing architecture, NLI foundations, and access requirements.

  • 2.4 Uncertainty Quantification for LLMs: LLM-specific uncertainty metrics are motivated by the tendency of LLMs to hallucinate when uncertain, complicating factuality assessment.The survey links uncertainty quantification to factuality analysis and reviews methods developed for this setting.
  • 2.4.1 LLM Architecture: LLMs commonly use decoder-only transformers that generate outputs autoregressively from token probability distributions.The architecture processes prompts into embeddings and selects output tokens from model distributions before continued generation.
  • 2.4.2 Natural-Language Inference: Natural-language inference classifies premise–hypothesis pairs as entailment, contradiction, or neutral, supporting semantic comparisons between responses.Many UQ methods compare multiple realizations of an LLM response using semantic relationships learned by NLI models.
  • 2.4.3 Metrics for Uncertainty Quantification for LLMs: UQ methods are broadly divided into white-box approaches requiring internal model information and black-box approaches usable with closed-source models.White-box methods can require access to architectures and intermediate outputs, whereas black-box methods can quantify uncertainty for models such as GPT-4 and Claude.
  • 2.4.3 Metrics for Uncertainty Quantification for LLMs: The survey taxonomy contains token-level, self-verbalized, semantic-similarity, and mechanistic-interpretability UQ methods.It includes mechanistic interpretability because the authors expect its insights to be more extensively applied to LLM uncertainty quantification.

3 Token-Level UQ

Token-level UQ estimates uncertainty from conditional token distributions, with extensions addressing sequence length, token dependence, and claim-level granularity. These methods can be calibrated in constrained multiple-choice settings but have important limitations for long or misleading sequences.

  • 3 Token-Level UQ: Token-level UQ estimates uncertainty from probability distributions over output tokens and can measure distributional randomness using entropy or semantic clusters.These methods directly quantify aspects of the model’s generating distribution.
  • 3 Token-Level UQ: Token-level confidence may misrepresent correctness when an initially incorrect token is followed by highly probable tokens.Trainable attention-based dependency models adjust uncertainty estimates using predicted conditional dependence between tokens.
  • 3 Token-Level UQ: SelfCheckGPT variants estimate uncertainty from multiple response samples using n-gram log-probabilities and maximum negative log-probability.These approaches use sampled responses and an n-gram model to assess the original response.
  • 3 Token-Level UQ: Token-level UQ methods can perform poorly on long responses because multiplying token probabilities decreases with response length, even for semantically equivalent outputs.Length-normalized scoring functions are introduced to reduce this dependence.
  • 3 Token-Level UQ: Claim-Conditioned Probability decomposes responses into claims and computes constituent-token uncertainty for finer-grained quantification.CCP uses the OpenAI Chat API to identify the main claims in a response.
  • 3 Token-Level UQ: Multiple-choice and True/False prompting can yield well-calibrated option probabilities, with calibration improving for larger models.A value-head variant predicts whether the model knows an answer, using the final-token probability as response confidence.

4 Self-Verbalized UQ

Self-verbalized UQ trains or prompts LLMs to express uncertainty in natural language or numeric confidence, but these estimates remain imperfectly calibrated. Numeric verbalized probabilities improve interpretability, while token-based methods generally calibrate better.

  • Self-verbalized UQ: Self-verbalized UQ enables LLMs to communicate uncertainty through epistemic markers, confidence values, scaled labels, or abstention.These approaches use natural-language outputs to expose the model’s confidence or decline uncertain answers.
  • Self-verbalized UQ: Poor calibration of self-verbalized confidence is more pronounced in low-data language settings such as Hindi and Amharic.
  • Self-verbalized UQ: Verbalized probability provides a definite numerical confidence value or a scaled characterization such as low, medium, or high.This addresses the difficulty of assessing relative confidence when uncertainty is expressed only linguistically.
  • Self-verbalized UQ: Verbalized probability generalizes well, but best performance is achieved in-distribution on the CalibratedMath benchmark.GPT-3 was fine-tuned on CalibratedMath using supervised learning.
  • Self-verbalized UQ: LLMs still often fail to express confidence accurately, exhibiting overconfidence despite uncertainty, although the weakness decreases with model size and prompting can reduce calibration error.Confidence values are frequently concentrated between 80% and 100%, often in increments of five.
  • Self-verbalized UQ: Verbalized confidence is better calibrated than raw conditional token probabilities, yet token-based UQ generally remains better calibrated than self-verbalized UQ.The survey presents this as a pattern from existing empirical studies.

5 Semantic-Similarity UQ

Semantic-similarity UQ estimates uncertainty from the semantic agreement among multiple responses rather than their exact token sequences. NLI models, embeddings, clustering, and graphs support this approach, but model dependence and single-cluster assignment remain challenges.

  • Semantic-similarity UQ: Semantic-similarity UQ compares the meanings of multiple responses, making uncertainty estimates more robust to token-level wording variations.Responses may differ in form while preserving the same semantic content.
  • Limitations and extensions: Token-level uncertainty can be inaccurate when different token orderings preserve sentence meaning but produce different token probabilities.
  • Limitations and extensions: Semantic similarity is model-dependent, and assigning each response to one cluster can limit effectiveness when responses plausibly belong to multiple classes.Graph-based extensions represent more complex relationships between responses explicitly.
  • NLI-based methods: NLI-based methods estimate entailment among generations to measure agreement, semantic entropy, contradiction, falseness, or confidence.Examples include chain-of-thought agreement, semantic entropy probes, and database-assisted fact checking.
  • NLI-based methods: CoTA semantic uncertainty yields more robust model-faithfulness estimates than self-verbalized or token-level uncertainty estimates.
  • Embedding-based methods: Embedding-based methods cluster responses and quantify uncertainty over semantic groups, including through semantic density, concept entropy, and instruction-guided clustering.ClusterLLM uses triplet queries to guide semantically aligned embeddings.

6 Mechanistic Interpretability

Mechanistic interpretability analyzes features, circuits, and activations inside LLMs to explain behavior and identify sources of uncertainty and hallucination. Its connection to uncertainty quantification remains underexplored, with relevant activations and features still an open challenge.

  • Mechanistic interpretability: Mechanistic interpretability seeks causal understanding of LLM uncertainty by examining internal features, circuits, and activation patterns.The survey frames this as a route to locating potential sources of uncertainty and hallucination.
  • Mechanistic interpretability: MI methods include logit lenses, probing, and sparse auto-encoders, which respectively inspect intermediate predictions, encoded information, and disentangled feature activations.Probing reveals correlations, whereas sparse auto-encoders are used to identify meaningful features and causal relations.
  • Methodological limitations: Probing methods reveal correlations rather than causal relations, limiting their use in mechanistic interpretability.
  • Applications to uncertainty and hallucination: Mechanistic analyses have examined hallucinations, latent knowledge, and model capabilities, including activation-based prediction of a larger LLM’s uncertainty across unseen distributions.Other work uses internal knowledge boundaries to prevent answers on subjects the model does not know.
  • Open challenge: Specific intermediate activations and features relevant to uncertainty quantification remain unidentified, leaving the MI–UQ connection as an open research challenge.

7 Calibration of Uncertainty

Calibration aligns uncertainty estimates with observed accuracy, addressing overconfidence and underconfidence in LLM outputs. The survey covers training-based, training-free, ensemble, few-shot, supervised, and conformal approaches, with conformal prediction offering coverage guarantees.

  • Calibration concepts: Confidence estimates above 0.5 tend to be overconfident, whereas estimates below 0.5 tend to be underconfident in the illustrated example.
  • Calibration concepts: Calibration techniques improve agreement between estimated confidence and observed accuracy, correcting overconfidence and underconfidence.Figure 15 illustrates poorly aligned estimates before calibration and closer alignment afterward.
  • Calibration methods: Calibration methods are categorized as training-based or training-free, depending on whether they modify model weights.Training-based methods use supervised techniques, while training-free methods apply statistical procedures to a frozen model.
  • Training-free calibration: Conformal prediction constructs prediction sets guaranteed to contain the true outcome with a specified probability.Its coverage guarantee is based on nonconformity scores and a calibration dataset.
  • Training-free calibration: Conformal prediction minimizes prediction-set size among set-valued predictors satisfying the requested coverage, yielding less ambiguity on average.
  • Training-based calibration: Training-based calibration includes ensembles, few-shot prompting, and supervised learning, with supervised response classification helping combat overconfidence.These approaches compare model outputs, use sequential reasoning, or modify weights through losses, auxiliary models, or additional data.

8 Datasets and Benchmarks

The survey reviews benchmarks and datasets used to evaluate uncertainty quantification across factuality, reasoning, consistency, ambiguity, and other LLM capabilities. It also notes that many datasets report answering accuracy without evaluating uncertainty itself.

  • Capability benchmarks: Broader capability benchmarks cover domain knowledge, common-sense reasoning, reading comprehension, mathematics, and code generation through datasets such as GPQA, MMLU, HellaSwag, RACE, GSM8K, and APPS.These datasets measure different LLM capabilities that can be paired with uncertainty evaluation.
  • Ambiguity and consistency: Ambiguity-focused datasets model prompts with multiple plausible interpretations, while consistency benchmarks alter prompts without changing their semantic meaning.The survey gives ambiguous cat-location prompts and identifies ParaRel as a 328-paraphrase consistency dataset.
  • Evaluation gap: A major limitation is that experimental results for many datasets do not evaluate uncertainty beyond answer accuracy.The survey also notes that some benchmarks have not been widely adopted for uncertainty quantification.
  • General-purpose and task-specific benchmarks: Uncertainty quantification studies draw on benchmarks spanning reading comprehension, conversational question answering, biographies, arithmetic confidence, consistency, multi-hop reasoning, factuality, hallucination detection, and claim verification.Examples include TriviaQA, CoQA, WikiBio, CalibratedMath, ParaRel, HotpotQA, StrategyQA, TruthfulQA, HaluEval, and FEVER.
  • Factuality and hallucination: Hallucination detection and factuality-oriented evaluation use benchmarks such as HaluEval, FEVER, and TruthfulQA to examine non-factual responses and claim verification.TruthfulQA targets questions that humans might answer incorrectly because of misconceptions.

9 Applications

Uncertainty quantification is applied across textual tasks, safety-related settings, and robotics. These applications use uncertainty to detect hallucinations, analyze generated content, guide interaction, and determine when human assistance or verification is needed.

  • Hallucination detection: Token-level uncertainty can reveal likely hallucinations even when an LLM’s response sounds confident.In the illustrated example, a low token-level confidence estimate indicates a high likelihood of hallucination.
  • Textual and interactive applications: Applications extend to sentiment and content analysis, thematic coding, summarization, jailbreaking analysis, retrieval-augmented generation, and response generation with prediction-set guarantees.Some methods propagate uncertainty across multi-step interactions, while others use conformal prediction for guaranteed or likely prediction sets.
  • Jailbreaking and safety: Jailbreak studies associate higher token entropy with jailbreak prompts, while other work finds that uncertainty can be manipulated and safeguard models can be miscalibrated.These findings motivate caution when interpreting uncertainty estimates during adversarial prompting.
  • Robotics: Robotics applications face safety concerns because embodied LLMs can become agents of physical change, yet relatively few applications explicitly account for LLM uncertainty.Existing work estimates confidence in generated plans to determine when human assistance or verification is required.
  • Robotics: Robotics systems use uncertainty estimates and conformal prediction to identify plausible next actions and request human help when multiple actions remain possible.KnowNo examines token probabilities, forms prediction sets, and prompts for assistance when a set contains more than one action.

10 Open Research Challenges

The survey identifies open challenges in making uncertainty estimates align with factuality, remain reliable under adversarial or interactive conditions, and support broader evaluation. It highlights unresolved issues involving sampling, token-level measures, multi-episode agents, interpretability, and datasets.

  • 10.1 Consistency is not Factuality: Consistency-based uncertainty estimates can be miscalibrated because repeated responses may be identically false, although consistency can help predict factuality with sufficiently many samples.The survey identifies black-box consistency methods as especially susceptible to this limitation.
  • 10.1 Consistency is not Factuality: Consistency methods lack rigorous guidance on how many samples are needed to form reliable response sets and confidence or factuality guarantees.Increasing temperature may add randomness, but excessive randomness undermines confidence assessment.
  • 10.2 Entropy is not Factuality: Token-based uncertainty can be miscalibrated when corrupted or insufficient training data causes an LLM to assign high probability to an incorrect answer, and jailbreaks can manipulate token distributions.The survey calls for robustness to adversarial attacks and better alignment between entropy and factuality.
  • 10.2 Entropy is not Factuality: Token entropy may be low for an incorrect claim because highly probable successive tokens do not guarantee claim-level factuality.Future work should align token entropy with claim factuality and account for training-distribution effects on token probabilities.
  • 10.3 Multi-episode interactions: Many agent applications estimate uncertainty per episode without modeling interaction history, despite multi-episode tasks whose later responses depend on preceding episodes.Assuming episode-wise independence is generally not satisfied in real-world applications.
  • 10.4 Mechanistic interpretability: Mechanistic interpretability remains relatively unexplored for uncertainty quantification, although internal states may help distinguish epistemic from aleatoric uncertainty.The survey identifies this connection as a potentially fruitful direction for future research.

11 Conclusion

The survey organizes LLM uncertainty quantification into four methodological classes and reviews relevant datasets, benchmarks, applications, and open challenges. Its taxonomy distinguishes methods by the model information and outputs they use.

  • Taxonomy: The survey categorizes LLM uncertainty quantification into token-based, self-verbalized, semantic-similarity-based, and mechanistic-interpretability methods.These four classes provide the survey’s organizing taxonomy.
  • Scope and outlook: The survey reviews applications and highlights open research challenges to motivate future work on uncertainty quantification for LLMs.Its scope includes methods, applications, datasets, benchmarks, and research directions.
  • Taxonomy: Token-based methods use intermediate outputs or architecture access, whereas self-verbalized methods ask the LLM to express confidence in natural language.The distinction concerns how confidence is obtained from the model.
  • Taxonomy: Semantic-similarity methods are often black-box approaches that estimate confidence from consistency among natural-language outputs.They require access only to the model’s natural-language output.
  • Taxonomy: Mechanistic interpretability methods analyze internal activations to identify latent features that explain activation patterns.The survey also identifies relevant datasets and benchmarks alongside its methodological review.
Loading 2412.05563v2…