PaperlayerView exact revision on arXiv

Source-linked AI summary

Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators

Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluís Màrquez

arXiv:2406.13415v1cs.CLcs.LG

TL;DR

LLMs can produce unreliable factual answers, while existing confidence estimators lack a systematic comparison. The paper surveys these methods and evaluates them in a common framework spanning fact-verification and question answering. Trained probes are most reliable across the tested models, but confidence is often unstable across semantically equivalent inputs.

  • Problem

    Existing factual-confidence methods lack a unified comparison, leaving their relative reliability across models unclear.

  • Method

    The paper surveys confidence estimators and compares them across fact-verification and question-answering using a fair experimental framework.

  • Results

    Trained probes are the most reliable estimators across eight LLMs, while confidence can be unstable across meaning-preserving input variations.

  • Takeaways & Limitations

    Confidence assessment should consider robustness across alternative phrasings, not only performance on a single way of asserting a fact.

  • Takeaways & Limitations

    The trained probe method requires training data and access to model weights, and its transfer to more complex tasks has limits.

Abstract

from arXiv · show

Large Language Models (LLMs) tend to be unreliable in the factuality of their answers. To address this problem, NLP researchers have proposed a range of techniques to estimate LLM's confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one another. To fill this gap, we present a survey and empirical comparison of estimators of factual confidence. We define an experimental framework allowing for fair comparison, covering both fact-verification and question answering. Our experiments across a series of LLMs indicate that trained hidden-state probes provide the most reliable confidence estimates, albeit at the expense of requiring access to weights and training data. We also conduct a deeper assessment of factual confidence by measuring the consistency of model behavior under meaning-preserving variations in the input. We find that the confidence of LLMs is often unstable across semantically equivalent inputs, suggesting that there is much room for improvement of the stability of models' parametric knowledge. Our code is available at (https://github.com/amazon-science/factual-confidence-of-llms).

1 Introduction

LLM factual-confidence estimators lack a unified comparison, motivating a framework that evaluates methods across fact-verification and question answering. Across eight LLMs, trained probes are most reliable, while confidence can remain unstable across meaning-preserving input variations.

  • Motivation: Factual-confidence estimation addresses unreliable or hallucinated information that can spread misinformation and undermine user trust.Reliable confidence estimates are presented as important for mitigating these risks.
  • Approach: The paper surveys existing estimators and introduces a framework for fair comparison across models and datasets.The framework covers five method groups and two factual-confidence formulations.
  • Results: Trained probes provide the most reliable factual-confidence estimates across eight LLMs, but require training data and access to model weights.For instruction-tuned LLMs, verbalization and consistency-based methods are viable alternatives.
  • Robustness: Confidence estimates can be misleading when evaluated on only one phrasing because models may contradict themselves under alternative formulations of the same fact.The experiments find instability across meaning-preserving input variations.

2 Factual Confidence: Key Concepts

The paper distinguishes factual confidence in statements from confidence that a model will answer factual questions correctly. It also treats consistency across meaning-preserving variations as a test of whether knowledge is represented abstractly rather than tied to particular wording.

  • 2.1 Definition of a Fact: A fact may be expressed by multiple statements with different wording but the same meaning, so confidence should remain consistent across paraphrases and translations.This motivates testing factual confidence under meaning-preserving linguistic variations.
  • 2.2 Factual Confidence: P(True), or P(T), measures how likely a model considers an input statement to be true in a fact-verification setup.Statements judged truthful should receive high P(T), while false statements should receive low P(T).
  • 2.2 Factual Confidence: P(I Know), or P(IK), measures how likely a model considers it will return the correct answer to a factual query in a question-answering setup.Unlike P(T), P(IK) is estimated from a question without stating the fact in the input.
  • 2.2 Factual Confidence: The two confidence measures are both informative, but the more suitable measure depends on the input format and task, and methods may not be equally reliable across setups.Most methods can be adapted to estimate both P(T) and P(IK).
  • 2.3 Robustness of Factual Knowledge: Consistency testing distinguishes factual knowledge encoded as an abstraction over linguistic forms from memorization of particular statements.Prior work shows that models may behave inconsistently across semantically equivalent inputs, motivating its use for robustness assessment.

3 Factual Confidence: Survey of Methods

The paper groups factual-confidence estimators into five methodological families and reviews their differing assumptions and requirements. These methods vary in whether they use model internals, token probabilities, elicited confidence, or output consistency.

  • Five estimator families are identified: trained probes, sequence probability, verbalization, surrogate token probability, and consistency-based methods.The survey summarizes their functional differences in Table 1.
  • 3.1 Trained Probes: Trained probes extract confidence from model hidden states and can use earlier or final layers, aiming to reduce sensitivity to claim phrasing.Prior work motivates probes as less dependent on surface-level features.
  • 3.1 Trained Probes: Probe methods require model-weight access and supervised labels indicating statement truth or answer correctness.An unsupervised variant is useful when model outputs are available but reliable annotations are not.
  • 3.2 Sequence Probability: Sequence probability averages token log-probabilities, but measures confidence in a claim’s wording rather than in the claim itself.This surface-form dependence makes sequence probability a weak baseline for factual confidence estimation.
  • 3.3 Verbalization: Verbalization prompts the model to output a numeric confidence level, while top-k, Chain-of-Thought, and multi-step variants extend the basic approach.The paper focuses on simplified non-CoT, k = 1 prompting because it is competitive across many settings.
  • 3.5 Consistency: Consistency methods sample multiple completions and use agreement between responses as confidence, with semantic clustering needed when equivalent answers differ in wording.The method treats consistent answers as high confidence and contradictory answers as low confidence.

4 Methodology

The methodology establishes a common evaluation setup across fact verification and question answering, using two datasets, multiple confidence estimators, and diverse LLMs. Robustness is tested through paraphrases and translations that preserve input meaning.

  • 4.1 Datasets: The framework evaluates factual confidence on Lama T-REx for P(T) and PopQA for P(IK), providing a shared baseline across fact verification and question answering.Lama T-REx supplies true and false statements, while PopQA supplies fact-related questions with entity answers.
  • 4.1.1 P(T) in Fact Verification: Lama T-REx: T-REx false statements replace each relation’s object with another object of the same relation, producing a balanced 50/50 true–false evaluation set.The dataset contains 34K true statements; 80% are used for trained-probe training and the remainder for analysis.
  • 4.1.2 P(IK) in QA: PopQA: PopQA includes synonymous correct-answer phrases, making evaluation more robust to answer wording and reducing the risk of underestimating correctness.The experiments use greedy decoding and assign gold P(IK) according to whether the generated answer is correct.
  • 4.2 Scoring Methods: For P(T), trained probes use hidden states, sequence probability averages statement-token log-probabilities, and other methods derive confidence from prompting or surrogate token probabilities.For P(IK), probes predict correctness of greedy-generated answers, while consistency averages pairwise NLI scores across ten sampled responses.
  • 4.4 Evaluating Scoring Methods: AUPRC is the primary metric because it evaluates ranking across decision thresholds; higher values indicate cleaner separation of true versus false or known versus unknown cases.Precision and recall at K are additionally reported for practical interpretation.
  • 4.3 Models: The study compares eight publicly available LLMs spanning 7B–46.7B parameters, different architectures, and instruction-tuning regimes.The model set includes Falcon and Mistral families, with dense and sparse-mixture architectures represented.
  • 4.6 Paraphrasing and Translation: Robustness is assessed with semantically equivalent paraphrases and French or Polish translations, retaining paraphrases only after bidirectional entailment checks.Each input initially receives ten paraphrases, with an average of eight retained after removing repetitions and non-equivalent variants.

5 Empirical Comparison of the Methods

Across fact-verification and question answering, trained probes are generally the most reliable factual-confidence estimators, while non-trained methods often perform near or below chance for P(IK). Probe reliability transfers out of domain but declines from Lama T-REx to PopQA.

  • 5.1 P(T) on Lama T-REx: The trained probe performs best for P(T), exceeding sequence probability by an average AUPRC of .3 and usually surpassing other methods by at least .1.Only the verbalized method is competitive for Mistral 7B instruct; deeper hidden states appear to capture truth information better than output scores.
  • 5.1 P(IK) on Lama T-REx: Instruction-tuned and larger models generally perform better for prompting-based methods, while average sequence probability remains consistently above chance but comparatively weak.All 40B+ models outperform their 7B counterparts except Mistral-7B-Instruct-v0.2; sequence probability beats other non-trained methods mainly on non-instruction-tuned models.
  • 5.2 P(IK) on PopQA: P(IK) is harder to estimate than P(T), with the best trained probe reaching an AUPRC 0.1 lower for P(IK).The difference may reflect query-only confidence estimation or noise from binary future-correctness labels.
  • 5.2 P(IK) on PopQA: The trained probe is again the most reliable P(IK) estimator, while other methods are generally close to or below chance across models.Chance levels vary between 0.11 and 0.27, with differences across models reaching 40%.
  • 5.3 Generalization of the Trained Probe: Probes retain substantial out-of-domain reliability on PopQA, although AUPRC drops by .15 to .31 from Lama T-REx and remains between .62 and .81.The least and most reliable transfer occur for Mistral-7B and Falcon-40B-instruct, respectively, and transfer may depend on dataset difficulty.

6 Robustness to Linguistic Variations

Meaning-preserving paraphrases and translations test whether confidence estimates and factual encodings remain stable across linguistic forms. Methods are robust to paraphrasing and often transfer across languages, but confidence for individual facts can vary substantially with wording and language.

  • 6.1 Robustness of Methods: Paraphrasing changes AUPRC by only 5%–10% across methods, with at most a 3-percentage-point standard deviation for the trained probe in P(IK).The trained probe is the most affected method in that setting, yet remains broadly robust without retraining or prompt adaptation.
  • 6.1 Robustness of Methods: English-trained probes remain reliable on unseen languages, with French AUPRC of .73–.91 and Polish AUPRC of .61–.91.All methods generalize above chance to French and Polish except verbalized confidence and surrogate logits on Mistral models.
  • 6.2 Robustness of Facts Encoding in LLMs: Paraphrases produce little variation for many facts but up to .5 standard deviation in confidence for others, indicating strong sensitivity to wording.Falcon-7B-instruct shows the least stable encoding, whereas Mistral-7B-v0 shows less variation.
  • 6.2 Robustness of Facts Encoding in LLMs: Language-pair confidence scores correlate strongly for 40B+ models, with Falcon 40B correlations reaching .87–.92, yet Friedman tests find significant distribution differences across languages.The results indicate related but not fully systematic multilingual confidence behavior.

7 Discussion & Conclusion

The trained probe is the most reliable estimator across tested models and tasks, but it requires model weights and supervised data. Factual confidence also varies across meaning-preserving input changes, revealing instability in LLM knowledge.

  • 7 Discussion & Conclusion: Reliable confidence estimation remains important because inaccurate estimates can introduce errors and negatively affect user-model interactions.The estimates are intended to help anticipate hallucinations and limit non-factual outputs.
  • 7 Discussion & Conclusion: The trained probe is the most reliable estimator across eight LLMs for both fact-verification P(T) and question answering P(IK), with out-of-domain transfer.It generalizes from T-REx to PopQA and across languages.
  • 7 Discussion & Conclusion: Trained probing requires model-weight access and supervised data, while prompting methods are alternatives for instruction-tuned models when those requirements are unavailable.The method’s fine-tuning gives it an advantage over zero-shot probing methods.
  • 7 Discussion & Conclusion: Factual confidence can change under paraphrases and translations, so a model may behave differently on semantically equivalent inputs.The findings suggest LLMs do not consistently encode facts as abstractions over diverse input forms.

Limitations

The comparison covers broad but relatively simple settings, so the trained probe’s advantage may not extend to more complex tasks or sentences. Method suitability also depends on transfer limits and prompt sensitivity.

  • Limitations: The trained probe is much stronger on T-REx and PopQA, but its advantage is not guaranteed for non-atomic facts, reasoning, or in-context learning.These settings were outside the study’s capacity to evaluate.
  • Limitations: The trained probe requires training data and has limits on the kinds of tasks to which it can transfer.The authors tested transfer within a simple atomic-fact setup.
  • Limitations: Sequence probability can weaken on more complex or longer sentences, while prompt-based methods are sensitive to prompt variations.Sequence probability performed better than both prompt-based methods for non-instruction-fine-tuned models in these experiments.

Ethics and Broader Impact

The work contributes to reducing risks from LLM use by supporting false-statement detection and answer-confidence estimation. However, the studied methods do not completely solve hallucination, and deployment requires specific infrastructure.

  • Ethics and Broader Impact: The methods support false-statement detection and answer confidence, with potential applications for building trust in LLMs.The contribution targets the broader goal of automatically reducing risk when using LLMs.
  • Ethics and Broader Impact: None of the studied methods completely solves hallucination or non-factual model utterances, leaving a need for future work.
  • Ethics and Broader Impact: Deploying the evaluated 7B and 40B+ parameter models requires specific infrastructure.

A Implementation of scoring methods

The implementation operationalizes confidence through verbalized prompts, yes/no/maybe prompts, output consistency, and paraphrase-based stability analyses. Consistency scores average NLI agreement across sampled completions, while paraphrase variation is reported in Figures 5 and 6.

  • A Implementation of scoring methods: Verbalized confidence prompts models to assign a 1.0–10.0 confidence level to a statement or to answering a question.The prompts separately target P(T) and P(IK).
  • A Implementation of scoring methods: Binary verbalization prompts ask whether a statement is true or whether the model knows the answer, with Yes/No/Maybe responses.These prompts operationalize P(T) and P(IK), respectively.
  • A Implementation of scoring methods: Output consistency samples 10 completions of up to 25 tokens at temperature τ=1 and averages their NLI scores.The consistency score measures agreement among generated responses.
  • A Implementation of scoring methods: Table 4 reports Spearman correlations between English-French and English-Polish P(T) scores on translated Lama T-REx statements.

C Method Robustness to Variation

The evaluation tests whether factual-confidence methods remain reliable under paraphrases and translations. Methods are stable across paraphrases, while cross-language evaluation is reported through score correlations and French and Polish AUPRC results.

  • C Method Robustness to Variation: Ten randomly sampled paraphrase sets are created by selecting one paraphrase for every sentence in the original dataset.
  • C Method Robustness to Variation: All methods remain stable under paraphrases, with the trained probe showing the largest variation at only about 3 percentage points.The evaluation samples ten paraphrase sets and recomputes AUPRC without changing the methods.
  • C Method Robustness to Variation: Cross-language robustness is assessed using score correlations and AUPRC on French and Polish translations of Lama T-REx.

D Analysis of specific Precision and Recall

The precision–recall analysis compares confidence methods across recall thresholds and language settings. Trained probes generally perform best, while classification remains limited and verbalized confidence shows threshold-insensitive precision.

  • D Analysis of specific Precision and Recall: The trained probe noticeably outperforms the other methods in precision-at-recall analyses for both P(T) and P(IK).The comparison covers Lama T-REx and PopQA, with precision reported at recall levels of 90, 70, and 50.
  • D Analysis of specific Precision and Recall: For P(IK), Falcon 40B outperforms the other models, whereas most methods perform poorly with Falcon 7B Instruct.
  • D Analysis of specific Precision and Recall: Verbalized confidence precision does not change across recall thresholds for either P(T) or P(IK), suggesting a limit in its estimation ability.
  • D Analysis of specific Precision and Recall: At recall 90, precision is interpreted as the share of all tested sentences correctly classified after thresholding scores to recover 90% of True sentences.
  • D Analysis of specific Precision and Recall: Classification has substantial overlap between True and False sentences and reaches at most about 60% correctly classified at 90 recall in both settings.
Loading 2406.13415v1…