Source-linked AI summary
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Kirill Grishchenkov, Sergey Petrakov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, Artem Shelmanov
TL;DR
LLM UQ research is fragmented, despite the need to address hallucinations and low-quality outputs. This paper presents a unified benchmark for comparing UQ and confidence-normalization methods across text-generation tasks. Its investigation reports different methods as preferable across output lengths and tasks.
Problem
LLMs can hallucinate or produce low-quality outputs, while UQ research for generation has fragmented methods and evaluation protocols that impede consolidating findings.
Source: §1 · pp. 1–2Method
The authors build a benchmark on LM-Polygraph that unifies UQ baselines across selective QA, generation, and claim-level fact-checking, and evaluates confidence normalization.
Source: §1 · p. 2Results
Across selective QA and generation, sample-diversity methods perform well; information-based methods suit shorter answers, SAR stands out across output lengths, and CCP is best for fact-checking.
Source: §7 · p. 12Takeaways & Limitations
The benchmark supports consistent comparison of UQ methods and assessment of whether confidence normalization produces interpretable scores across eleven datasets.
Source: §1 · p. 2A reported discrepancy may reflect ROUGE-L's limitations for measuring summarization quality.
Source: §6.1 · p. 11
Abstract
from arXiv · showhide
The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a key element of machine learning applications in dealing with such challenges. However, research to date on UQ for LLMs has been fragmented in terms of techniques and evaluation methodologies. In this work, we address this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines and offers an environment for controllable and consistent evaluation of novel UQ techniques over various text generation tasks. Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores. Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches. Code: https://github.com/IINemo/lm-polygraph Benchmark: https://huggingface.co/LM-Polygraph
1 Introduction
LLMs can hallucinate or produce low-quality outputs, while UQ research for generation remains fragmented and difficult to compare. This work introduces a benchmark for consistent evaluation of UQ and confidence normalization across text-generation tasks.
1 Introduction
- LLMs can make incorrect predictions, fabricate claims, or generate low-quality outputs, motivating approaches to uncertainty quantification for safer downstream use.The introduction identifies training-objective peculiarities, limited training data, and task ambiguity as sources of these problems.Source: §1 · p. 1
- UQ studies for LLM generation use divergent methods and evaluation protocols, making it difficult to consolidate findings and draw general conclusions.The paper describes the research landscape as fragmented, with concurrent, non-comparable studies.Source: §1 · p. 2
- The benchmark unifies state-of-the-art UQ baselines for consistent comparison across selective QA, selective MT and summarization, and claim-level fact-checking.The fact-checking pipeline covers English, Chinese, Arabic, and Russian; the investigation spans eleven datasets.Source: §1 · p. 2
- The benchmark assesses uncertainty-score calibration and effectiveness in selective QA and generation and claim-level fact-checking, alongside normalized, bounded confidence scores and a multilingual claim-level evaluation pipeline.It also supports evaluating whether confidence normalization preserves raw-score performance while improving calibration and interpretability.Source: §1 · p. 2
2 Uncertainty Quantification Methods
The section frames generation UQ as a problem of measuring uncertainty at token, sequence, and claim levels, then surveys methods by access requirements and approach. It distinguishes white-box, black-box, and claim-level techniques and their practical constraints.
2.1 Background
- Generation UQ must aggregate token-level uncertainty, may require claim-level scores, and should distinguish irrelevant surface-form variation from meaningful uncertainty.Because LLM token predictions are dependent, an incorrect early claim can also lead to subsequent hallucinations.Source: §2.1 · p. 3
2.2 Overview of Uncertainty Quantification Methods for LLMs
- UQ methods differ in whether they require logits or internal states, repeated inference, training data, or support only sequence-level rather than claim-level scoring.The framework overview describes white-box methods as requiring model access and black-box methods as needing only generated text.Source: §2.2 · pp. 3–4
2.2.1 White-Box Methods
- White-box information-based methods use token probabilities, while sample-diversity methods compare multiple outputs; Semantic Entropy groups responses by meaning and SAR aggregates probabilities using semantic similarity.Information-based methods are simple and cost-effective, often performing on par with more computationally demanding techniques.Source: §2.2.1 · p. 4
- Density-based methods estimate input likelihood from training-data embeddings without extra inference, whereas reflexive P(True) asks the model to verify its generated answer in another inference step.Density-based methods require access to training data; prior work reports P(True) outperforming MSP for large models at the cost of that extra step.Source: §2.2.1 · p. 4
2.2.2 Black-Box Methods
- Black-box diversity methods sample responses, build a similarity matrix, and derive uncertainty from it; verbalized methods instead ask instruction-tuned models to state confidence.Number of Semantic Sets counts clusters of similar responses, while the graph-Laplacian method provides a continuous relaxation of that count.Source: §2.2.2 · pp. 4–5
2.2.3 Claim-Level Extensions
- Claim-level extensions score uncertainty for text fragments, but sampling-based methods may fail there because sampled outputs can diverge and omit claims.MSP and several token-level measures can be adapted to claim tokens; CCP instead compares the claim with perturbed versions to assess semantic similarity.Source: §2.2.3 · p. 5
3 Uncertainty Normalization Methods
The section distinguishes bounded confidence normalization from confidence calibration, which also aims to make scores reflect expected output quality. It introduces two Performance-Calibrated Confidence methods alongside simpler scaling approaches.
3 Uncertainty Normalization Methods
- Linear scaling and quantile scaling normalize uncertainty scores to [0, 1] using only uncertainty values.The methods are fitted on a held-out calibration set containing inputs, generated outputs, uncertainty scores, and generation-quality scores.Source: §3 · p. 5
- Performance-Calibrated Confidence (PCC) aims to make bounded confidence scores directly reflect expected output quality, improving their interpretability for end users.The paper distinguishes this goal, termed confidence calibration, from confidence normalization alone.Source: §3 · pp. 5–6
- Binned PCC estimates confidence using mean output quality within non-intersecting bins formed from calibration-set uncertainty values.Its normalized scores can reorder instances relative to raw uncertainty, making UQ quality vary substantially and unpredictably.Source: §3 · p. 5
- Isotonic PCC fits Centered Isotonic Regression to calibration-set uncertainty and quality pairs, producing a monotonic piecewise-linear mapping that preserves input ordering.This addresses the ordering changes possible with Binned PCC.Source: §3 · p. 6
4 Approaches to Evaluating Uncertainty Quantification Methods
The section reviews evaluation measures for UQ, noting limitations of correlation and threshold-based approaches and presenting rejection verification as a more comprehensive alternative. It also highlights concerns about output-quality metrics, baselines, and claim-level evaluation protocols.
4 Approaches to Evaluating Uncertainty Quantification Methods
- Threshold-based ROC-AUC or PR-AUC evaluations require a quality cutoff whose arbitrary selection can make results across studies incomparable.Rank correlation between uncertainty and output quality, though straightforward, does not reveal the performance gain of a UQ method.Source: §4 · p. 6
- Rejection verification evaluates average output quality as progressively more uncertain outputs are discarded, forming a prediction–rejection curve whose area measures UQ performance.Unlike threshold-based binary classification, this approach does not require thresholding the quality metric.Source: §4 · p. 6
- AlignScore is proposed alongside n-gram metrics because it compares outputs with gold answers using an LLM and can capture semantic differences such as negation.The paper notes that n-gram metrics may misrank an answer differing from the gold answer only by a negation token.Source: §4 · p. 6
- MSP can be difficult to surpass on short-output tasks, so omitting it may leave UQ comparisons incomplete.The paper contrasts it with entropy-based metrics that often perform worse in these tasks.Source: §4 · p. 6
- For claim-level UQ, the benchmark uses unrestricted LLM generation and an automated multilingual evaluation pipeline rather than fixed outputs generated by one LLM and manually annotated.The paper argues that the earlier proxy-model approach creates a discrepancy between fixed text and what the proxy model would generate.Source: §4 · p. 6
5 Evaluation Benchmark
The benchmark organizes UQ evaluation across selective QA and generation, claim-level fact-checking, and confidence calibration. Its setup spans multiple datasets, evaluation metrics, model access settings, and languages.
5 Evaluation Benchmark
- The benchmark has three parts: selective QA and generation, claim-level fact-checking, and confidence calibration.These sections assess UQ performance in selective tasks and fact-checking, as well as the calibration of confidence scores.Source: §5 · p. 7
5.1 Selective QA / Generation
- Selective QA uses CoQA, TriviaQA, MMLU, and GSM8k, while generation uses WMT-14 French–English, WMT-19 German–English, and XSum.Evaluation sets are limited to 2,000 instances per dataset except MMLU, which uses 100 questions per subject; quality metrics vary by task, including AlignScore, COMET, accuracy, and ROUGE-L.Source: §5.1 · p. 7
5.2 Claim-Level Fact-Checking
- Claim-level fact-checking evaluates unsupported claims in biographies generated without restrictions, using an automated pipeline for English, Mandarin Chinese, Arabic, and Russian.GPT-4 decomposes outputs into claims and annotates support status; evaluation considers successfully matched claims and uses ROC-AUC and PR-AUC with unsupported claims as the positive class.Source: §5.2 · p. 8
5.3 Effect of Uncertainty Normalization
- Normalization evaluation compares PRR before and after normalization and measures calibration with MSE between normalized quality and confidence scores.Reserved training partitions from selective QA and generation serve as calibration sets; lower MSE indicates better calibration.Source: §5.3 · p. 8
5.4 Models
- Selective QA and generation experiments include white-box models, while selective QA also includes black-box models available only through generated text or an API.White-box experiments use Mistral 7B v0.2 and Stable LM 2 12B; black-box QA includes their instruction-tuned versions and GPT-4o-mini.Source: §5.4 · pp. 8–9
6 Experiments
The experiments compare uncertainty methods across selective QA and generation, multilingual claim-level fact-checking, and confidence normalization. Sample-diversity methods lead on average across selective QA/generation, CCP performs best in most fact-checking settings, and isotonic PCC improves calibration without degrading PRR.
6.1 Selective QA and Generation
- CCP, MSP, HUQ-MD, SAR, and Lexical Similarity are the best white-box selective-QA methods overall; MSP is strong on MMLU and CoQA, CCP on MMLU and GSM8k, and HUQ-MD on GSM8k for Stable LM.Sample-diversity methods lead on CoQA and TriviaQA but trail information-based methods on MMLU, where short multiple-choice responses limit their opportunity to exploit diversity.Source: §6.1 · p. 10
- For short outputs, information-based methods perform best; for longer outputs, especially NLI-based black-box diversity methods perform better, while SAR is robust across output lengths.Table 4 reports AlignScore-based PRR aggregated across selective QA/generation tasks and both white-box models, grouped by output length.Source: Table 4 · p. 11, §6.1 · p. 11
- Across datasets and models, sample-diversity methods rank highest on average; they also lead both text-summarization metrics, while most information-based methods have negative or near-zero PRR.Conditional PMI is an exception for ROUGE-L-based PRR, a result the authors suggest may reflect limitations of ROUGE-L for summarization quality.Source: §6.1 · p. 11
- On machine translation, PRR-COMET has no clear winner; SAR leads for Stable LM and basic information-based methods for Mistral, while NLI-based black-box methods lead on PRR-AlignScore.DegMat with NLI similarity is best on PRR-AlignScore across Stable LM datasets and second-best for Mistral.Source: §6.1 · p. 11
6.2 Fact-Checking
- CCP performs best on claim-level fact-checking for every tested model and language except Chinese, where CCP and P(True) are comparable on both metrics.Table 5 reports ROC-AUC and PR-AUC, with unsupported claims as the positive class.Source: §6.2 · p. 12, Table 5 · p. 12
6.3 Effect of Uncertainty Normalization
- Binned and isotonic PCC outperform linear and quantile normalization on calibration MSE for almost all considered UQ techniques; linear, quantile, and isotonic PCC retain PRR performance similar to raw scores.Figure 4 reports MSE between AlignScore and confidence scores, averaged over combined test partitions after fitting scorers on combined training partitions.Source: §6.3 · p. 12
7 Conclusion
The paper concludes that the benchmark’s empirical results support choosing UQ methods according to task and output length. It recommends isotonic PCC normalization for interpretable confidence scores because it improves calibration without degrading PRR.
7 Conclusion
- Sample-diversity methods perform well across selective QA and generation, while information-based methods are recommended for shorter answers because they often match diversity methods with less computational overhead.For multiple-choice QA, the authors identify MSP, Perplexity, and CCP as substantially superior; for longer-output tasks, they prefer Semantic Entropy, DegMat, or Lexical Similarity.Source: §7 · p. 12
- SAR is one of the most effective methods across short and long outputs, while reflexive methods generally perform poorly and verbalized techniques may be reasonable for large LLMs such as GPT-4o-mini.MSP is also described as a strong, robust baseline across tasks.Source: §7 · p. 12
- For claim-level fact-checking, CCP is the best method and is computationally efficient; isotonic PCC is suggested for human-interpretable confidence scores because it improves calibration without reducing PRR.Figure 4’s MSE compares normalized confidence scores with AlignScore, averaged over combined test partitions after fitting scorers on combined training partitions.Source: §7 · p. 12, Figure 4 · p. 12
A Detailed Description of Uncertainty Quantification Methods
The appendix describes the uncertainty measures implemented in the benchmark, covering information-based scores, sample-diversity methods, and density-based methods. The measures are expressed as functions of an input, though some also depend on generated outputs.
A.1.1 Information-Based Methods
- Information-based methods derive uncertainty from token probabilities or distributions, including length-normalized negative log probability, mean token entropy, PMI/CPMI, divergences, and TokenSAR’s relevance-weighted token scores.TokenSAR weights generated-token negative log probabilities by token relevance to the full text; its relevance weights are normalized across the output.Source: §A.1.1 · p. 19
A.1.2 Methods Based on Sample Diversity
- Sample-diversity methods estimate sequence-level uncertainty from sampled outputs, using Monte Carlo approximations; Semantic Entropy clusters semantically similar sequences before averaging their probabilities.SentenceSAR adjusts probabilities according to sentence relevance, and SAR combines SentenceSAR with TokenSAR.Source: §A.1.2 · pp. 19–20
A.1.3 Density-Based Methods
- Density-based Mahalanobis Distance fits a Gaussian to training representations and scores an instance by its distance from the training centroid using the empirical covariance matrix.The suggested representation averages the encoder’s last hidden state over non-padding tokens or the decoder’s last hidden state over generated tokens.Source: §A.1.3 · p. 20
- Robust Density Estimation reduces representation dimensionality with PCA and estimates covariance using Minimum Covariance Determinant before computing the Mahalanobis distance.Relative Mahalanobis Distance adjusts the score by subtracting a distance computed on a broad general-purpose dataset.Source: §A.1.3 · p. 20
A.2.1 Methods Based on Sample Diversity
The section describes uncertainty methods based on response diversity, black-box approximations of white-box scores, confidence verbalized by the model, and claim-level assessment. These methods use sampled outputs, model probabilities, or claim-focused comparisons to estimate uncertainty.
A.2.1 Methods Based on Sample Diversity
- Sample-diversity methods compare multiple answers to a prompt, using response similarity and semantic clustering to estimate uncertainty from how much the answers vary.Number of Semantic Sets counts clusters but is limited to integer values and assumes semantic equivalences are transitive.Source: §A.2.1 · pp. 20–21
- Graph-based approaches analyze response similarities using Laplacian eigenvalues, while degree-based scores and embedding eccentricity provide alternative measures of response diversity.The degree-based score estimates average pairwise distance; eccentricity uses distances from the center of an embedding constructed from Laplacian eigenvectors.Source: §A.2.1 · p. 21
A.2.2 Empirical Approximations of White-box Methods
- Black-box empirical approximations estimate MSP, Semantic Entropy, and P(True) from sampled-output frequencies or repeated model responses.LabelProb uses the frequency of the most likely sample; Semantic Entropy uses frequencies of semantic clusters, and P(True) uses the relative frequency of “True” responses.Source: §A.2.2 · p. 21
A.2.3 Reflexive Methods
- Reflexive methods elicit confidence from the model through a selected linguistic expression or a numerical confidence reported with the answer, in one or two turns.The two-turn chain-of-thought variant asks for reasoning and an answer first, then numerical confidence.Source: §A.2.3 · pp. 21–22
A.3 Claim-Level Methods
- Claim-level methods estimate uncertainty for individual text claims, adapting token-based scores or prompting the model to judge each claim’s truthfulness.Claim-Conditioned Probability compares the original claim with versions in which a token is replaced by a model-generated alternative, using an NLI model to assess semantic preservation.Source: §A.3 · p. 22
B Detailed Description of Uncertainty Normalization Methods
The section describes four approaches for mapping raw uncertainty scores to confidence scores using calibration data. They range from simple scaling to calibration methods that estimate response quality while preserving the uncertainty ordering.
B Detailed Description of Uncertainty Normalization Methods
- Linear scaling negates uncertainty scores and maps confidence to a normalized range, clipping scores so tested instances remain within [0, 1].The mapping uses the calibration-set minimum and maximum confidence values.Source: B Detailed Description of Uncertainty… · p. 22
- Quantile scaling uses the calibration-set uncertainty cumulative distribution to produce confidence scores naturally bounded between 0 and 1.The score accounts for the distribution of uncertainty values in the calibration set.Source: B Detailed Description of Uncertainty… · p. 22
- Binned PCC estimates confidence from the mean output quality in the calibration bin for a new instance’s raw uncertainty score.The bin-based mapping can change the ordering of instances relative to raw uncertainty, making UQ quality vary substantially and unpredictably.Source: B Detailed Description of Uncertainty… · p. 22
- Isotonic PCC uses fitted Centered Isotonic Regression to predict response quality while maintaining strict monotonicity: higher uncertainty always produces lower confidence.This method is proposed to remedy the ordering problem described for Binned PCC.Source: B Detailed Description of Uncertainty… · p. 22
C Detailed Experimental Results for Uncertainty Quantification Methods
The supplied tables cover PRR at 50% for uncertainty methods in selective question-answering and selective generation tasks. Their captions specify models and, for black-box methods, the relevant subsets, but do not provide cell values here.
C Detailed Experimental Results for Uncertainty Quantification Methods
- Tables 6 and 7 report PRR↑50% across generation-quality metrics for selective QA with Stable LM 2 12B and Mistral 7B v0.2.The captions identify the warmer color as indicating better results.Source: Table 6 · p. 23, Table 7 · p. 23
- Tables 8–10 report PRR↑50% for black-box UQ methods in selective QA with GPT-4o-mini, Stable LM 2 12B Chat, and Mistral 7B v0.2 Instruct.The captions specify various generation-quality metrics and identify warmer colors as better.Source: Table 8 · p. 24, Table 9 · p. 24, Table 10 · p. 24
- Tables 11 and 12 report PRR↑50% across generation-quality metrics for selective generation with Stable LM 2 12B and Mistral 7B v0.2.The captions identify the warmer color as indicating better results.Source: Table 11 · p. 25, Table 12 · p. 25
D Additional Experimental Results with Uncertainty Normalization
The supplied additional tables cover comparisons between raw uncertainty and bounded confidence, and generation-quality metrics for untuned and instruction-tuned LLMs.
D Additional Experimental Results with Uncertainty Normalization
- Table 13 reports the difference between PRR for raw uncertainty and bounded confidence; lower values are better, and negative values mean confidence performs better.The table covers various normalization techniques.Source: Table 13 · p. 26
E LLM Text Generation Quality
- Tables 14 and 15 report generation-quality metrics for LLMs without instruction tuning and for instruction-tuned LLMs, respectively.Source: Table 14 · p. 26, Table 15 · p. 26