Source-linked AI summary
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
Zhen Lin, Shubhendu Trivedi, Jimeng Sun
TL;DR
Trustworthy use of LLM-generated text remains difficult, especially because NLG uncertainty research is limited and often assumes white-box access. The paper develops black-box uncertainty and confidence measures from multiple generations and applies them to selective QA. Similarity-based dispersion measures effectively identify difficult questions and confident answers.
Problem
Reliable uncertainty quantification for NLG remains limited, and existing methods often assume access to token-level numerical outputs unavailable for black-box LLMs.
Method
The paper estimates input uncertainty and response confidence from multiple generated texts and evaluates these measures for selective natural language generation on QA datasets.
Results
Similarity-based dispersion measures effectively identify difficult questions and predict the quality of generated answers across experiments with popular LLMs and QA datasets.
Takeaways & Limitations
Simple NLI-based similarity and dispersion measures can help practitioners manage uncertainty and potentially exclude incorrect answers when using LLMs.
Takeaways & Limitations
Evaluation focuses on QA datasets because open-ended NLG would require difficult-to-scale quality assessment, and human labels could improve evaluation.
Abstract
from arXiv · showhide
Large language models (LLMs) specializing in natural language generation (NLG) have recently started exhibiting promising capabilities across a variety of domains. However, gauging the trustworthiness of responses generated by LLMs remains an open challenge, with limited research on uncertainty quantification (UQ) for NLG. Furthermore, existing literature typically assumes white-box access to language models, which is becoming unrealistic either due to the closed-source nature of the latest LLMs or computational constraints. In this work, we investigate UQ in NLG for *black-box* LLMs. We first differentiate *uncertainty* vs *confidence*: the former refers to the ``dispersion'' of the potential predictions for a fixed input, and the latter refers to the confidence on a particular prediction/generation. We then propose and compare several confidence/uncertainty measures, applying them to *selective NLG* where unreliable results could either be ignored or yielded for further assessment. Experiments were carried out with several popular LLMs on question-answering datasets (for evaluation purposes). Results reveal that a simple measure for the semantic dispersion can be a reliable predictor of the quality of LLM responses, providing valuable insights for practitioners on uncertainty management when adopting LLMs. The code to replicate our experiments is available at https://github.com/zlin7/UQ-NLG.
1 Introduction
The paper studies uncertainty quantification for black-box LLMs in NLG, motivated by the need to assess when generated responses should be trusted. It proposes simple measures and evaluates them for selective generation, finding that they can identify difficult questions and predict answer quality.
- Wider adoption of LLMs for tasks such as question-answering makes quantifying their uncertainty important.
- High-uncertainty or low-confidence responses can be rejected or sent for further evaluation, especially in selective NLG.
- UQ for NLG faces distinct challenges and has received limited attention compared with classification and regression.
- The study estimates uncertainty and generation confidence from multiple outputs while assuming access only to generated text, not token-level logits.
- Experiments on popular LLMs and QA datasets find that the proposed measures identify challenging questions and predict the quality of their answers.
2 Related Works
Prior NLG uncertainty research often adapts classification-style methods, asks LLMs for confidence, or requires white-box outputs. The paper positions black-box selective NLG as an underexplored setting requiring confidence measures tied to individual responses.
- Earlier NLP UQ work often treats question answering as classification rather than directly addressing generative outputs.
- Self-reported confidence approaches can be expensive and difficult to generalize across LLMs because training details and model behavior are opaque.
- Semantic entropy uses answer equivalence classes but requires token-level numerical outputs, limiting its use with black-box LLMs.
- Selective NLG concerns deciding when to trust or reject an LLM response, and confidence depends on both the input and the prediction.
- The paper evaluates ranking measures rather than directly addressing whether LLM confidence is calibrated.
3 Background
The paper distinguishes predictive uncertainty, which depends on the input and reflects distributional dispersion, from confidence, which also depends on a particular generated response. It motivates black-box alternatives because standard probability-based scores require access to the original LLM.
- 3.1 Predictive Uncertainty in NLG: Predictive uncertainty U(x) measures dispersion in the distribution of outputs conditioned on an input and depends only on that input.
- 3.1 Predictive Uncertainty in NLG: NLG predictive entropy sums over all potential response sequences, making direct computation difficult in its exceedingly high-dimensional output space.
- 3.1 Predictive Uncertainty in NLG: Total uncertainty includes epistemic uncertainty, which may be reduced with better models or more data, and aleatoric uncertainty, which is irreducible and tied to data generation.
- 3.1 Predictive Uncertainty in NLG: The study focuses on total uncertainty and uses QA datasets because open-ended tasks make scalable quality evaluation and uncertainty formulation harder.
- 3.2 Uncertainty vs. Confidence: Confidence C(x,y) concerns a particular prediction and therefore depends on both the input x and generated output y.
- 3.2 Uncertainty vs. Confidence: For NLG, the classification-style predicted probability becomes a joint probability over the generated sequence, but Eq. (2) requires white-box LLM access.
- 3.2 Uncertainty vs. Confidence: Using input uncertainty alone to predict a response’s correctness is problematic; confidence is reported as a more reliable indicator for a given response.
4 Quantifying the Uncertainty for NLG
The paper estimates uncertainty and per-response confidence for black-box LLMs by comparing multiple generated responses through similarity-based methods. These methods include discrete semantic grouping, graph-Laplacian spectral measures, degree-based scores, and eccentricity in a similarity-derived embedding.
- Overview: For each input, the framework generates m responses, computes pairwise similarities, and converts them into an uncertainty estimate or response-level confidence score.The approach relies only on generated text and similarity values rather than the black-box LLM’s token-level numerical outputs.
- Response Similarities: Jaccard similarity compares responses as word sets, making it efficient and easy to implement but insensitive to word order and expressions such as negation.The metric divides the intersection size by the union size of the two response word sets.
- Response Similarities: NLI-based similarity uses an off-the-shelf classifier to estimate entailment and contradiction between response pairs, supporting semantic equivalence grouping.The paper uses bidirectional entailment and contradiction comparisons to merge responses into semantic sets, assuming transitivity of semantic equivalence.
- Semantic-Set Measures: The number of semantic sets measures uncertainty by counting semantically distinct response groups, with more diverse answers indicating higher total uncertainty.For the Zeus example, three responses form two semantic sets because two responses share a meaning while the third differs.
- Graph-Laplacian Measures: The graph-Laplacian approach builds a weighted response graph from pairwise similarities and uses eigenvalue structure as a continuous proxy for the number of semantic meanings.The multiplicity of the zero eigenvalue equals the number of connected components for binary graphs, while the smallest eigenvalues help identify clusters in weighted graphs.
- Response-Level Confidence: Degree-based confidence treats highly connected responses as more confident, while eccentricity embeds responses using Laplacian eigenvectors and uses distance from the center as negative confidence.The corresponding uncertainty measures include average pairwise distance or average distance from the embedding center.
5 Experiments
The experiments evaluate black-box uncertainty and confidence measures across QA datasets and LLMs, using correctness prediction and selective-generation metrics. Results show that semantic similarity choices and response-dependent confidence measures are especially effective, with useful performance achievable from only a few sampled generations.
- Experimental setup: Experiments use CoQA, TriviaQA, and Natural Questions with OPT, LLaMA, LLaMA2, and gpt-3.5-turbo.The study evaluates uncertainty and confidence measures across open-book and closed-book QA settings.
- Evaluation: Correctness is evaluated with GPT-3.5-turbo scores, while AUROC and AUARC measure prediction and selective-generation quality.Responses scoring above 0.7 are deemed correct; AUARC measures retained accuracy as high-uncertainty samples are rejected.
- Uncertainty results: aNLI,entail generally outperforms alternative similarity measures, while Deg, Ecc, and EigV perform similarly as uncertainty estimators.The choice of similarity matters more than the choice among these dispersion constructions; entailment can distinguish meaningless non-contradictory answers.
- Selective generation: 62% to around 90%: selecting the top 50% of LLaMA trivia samples with Deg (E) could substantially improve retained accuracy.The figure notes that different aNLI,entail-based constructions differ relatively little but outperform the baselines.
- Sampling trade-offs: m = 3 already yields good selective-generation performance, while increasing sampled generations typically improves uncertainty and confidence quality.For individual-response accuracy, confidence measures Deg and Ecc improve with more samples, whereas response-agnostic uncertainty measures remain unchanged.
6 Conclusion and Discussion
The paper concludes that simple similarity-based dispersion measures can quantify uncertainty and confidence for black-box LLMs and help identify difficult questions and reliable responses. Its evaluation remains limited to QA, and the measures do not address all sources of factual error or overconfidence.
- Conclusion: NLI-based similarity combined with simple dispersion measures can identify difficult questions and confident answers, often outperforming white-box benchmarks.The paper presents these methods as easily implementable tools for managing uncertainty in black-box LLMs.
- Limitations: The evaluation is restricted to question-answering tasks because reliable-response labels are difficult to obtain for open-ended conversations.The authors also note that human labels could improve evaluation beyond GPT-based judging.
- Limitations: At temperature 0, sampling-based measures become degenerate, and the methods may miss training-corpus factual errors or LLM overconfidence.The measures reflect uncertainty in the posterior represented by the LLM and do not directly solve calibration.
A Proof for Theorem 1
The proof establishes properties of eigenvectors associated with connected components of a graph. It constructs orthonormal vectors for the components and shows that no additional zero eigenvector exists.
- Relation to prior work: The theorem is identified with Proposition 4 of Von Luxburg (2007), with L equivalent to Lsym.The proof briefly recovers the result through an intermediate proposition.
- Eigenvector construction: For a graph with k connected components, the proof constructs k orthonormal eigenvectors.The vectors are defined component-wise and have unit norm because the components are disjoint.
- Orthogonality: Vectors supported on distinct connected components are orthogonal because their supports are disjoint.The proof verifies both unit norm and zero inner product for distinct component vectors.
- Completeness: Any zero eigenvector must be a scaled version of the component vector on its connected component, preventing another independent zero eigenvector.A nonzero coordinate identifies a component on which the vector is proportional to the corresponding constructed eigenvector, yielding a nonzero inner product.
B Additional Experiment Details and Ablations
The appendix details dataset-specific prompting and the answer-pair representation used to compute entailment and contradiction scores. Prompts vary by QA dataset, while the similarity model receives both the question and two candidate answers.
- TriviaQA: TriviaQA uses the exact prompt from Touvron et al. (2023a), including an example question-answer pair before the provided question.The reproduced example maps “In Scotland a bothy/bothie is a?” to “House.”
- Natural Questions: Natural Questions uses a 5-shot version of the Touvron et al. prompt because it is more challenging than TriviaQA.The five questions are randomly selected from the training set.
- CoQA: CoQA prompts include the context paragraph, preceding conversational question-answer pairs, and the provided question.The additional pairs represent earlier turns about the same paragraph.
- Similarity scoring: For each pair of answers, DeBERTa-large receives the question and both answers separated by [SEP] to produce entailment or contradiction scores.The input format is [question] [answer_1] [SEP] [question] [answer_2].
B.3 Automatic accuracy evaluation
The study uses GPT-3.5-turbo to automatically judge response correctness against questions and reference answers, with human verification supporting the evaluation. The authors also identify cases where GPT captures semantic equivalence better than lexical ROUGE-L, while noting that GPT evaluation can fail when it relies on its own knowledge.
- Parsing coverage: 99.34% of GPT judgments parse as integers between 0 and 100 on the first attempt using simple string splitting.The remaining samples were skipped, though improved parsing or prompts could increase coverage.
- Comparison with ROUGE-L: GPT identifies paraphrases, abbreviations, and temporally equivalent answers as correct in examples where ROUGE-L fails.Examples include “mean” versus “nice,” “GCC” versus its expanded name, and a qualitative date description versus a specific date.
- Caveat: GPT evaluation can overrate an answer when it relies on its own knowledge and largely ignores the reference answer.The authors describe a case where GPT rates an answer as 100 despite disagreement with the reference, suggesting prompts may need improvement.
B.6 Full Experiment Results
The full experiments compare similarity measures and uncertainty/confidence constructions across black-box LLM generations. Semantic entailment generally performs best, and semantic similarity is more informative than lexical similarity when differently worded answers share the same meaning.
- Similarity measures: aNLI,entail consistently performs best, especially on nq, and similarity choice appears more important than whether uncertainty uses Ecc, Deg, or EigV.The authors attribute this partly to entailment capturing high uncertainty for low-quality answers to “why” questions.
- Similarity measures: For factual questions, entailment-, contradiction-, and Jaccard-based similarities agree on high-uncertainty cases because correct answers have few alternative formulations.The paper contrasts this with questions where semantically poor answers do not explicitly contradict one another.
- Evaluation results: The reported AUARC and AUROC tables compare black-box methods using m = 20 generations, with the best black-box methods bolded and the best overall method underscored.The tables evaluate expected or individual accuracy depending on the metric.
- Similarity measures: Semantic-based similarity is preferred over purely lexical similarity when generations use different wording but convey the same meaning.Figure 7 illustrates the failure mode of lexical similarity in such cases.
C Additional Results
Additional experiments show that the proposed measures remain useful with few generations and generally improve as the number of generations increases. The experiments also evaluate uncertainty against individual answer accuracy using AUROC.
- Number of generations: With m = 3, 5, and 10, aNLI,entail generally performs best and similarity choice matters more than the uncertainty construction.The results are reported for Ecc, Deg, and EigV constructions.
- Number of generations: With three generations, selective generation already produces a noticeable AUARC performance boost.The reported additional results remain similar to those obtained with m = 20.
- Individual accuracy: The individual-accuracy setting uses uncertainty estimated from m samples to predict whether each generated answer is correct.AUROC is computed from the uncertainty scores and per-answer correctness labels.
C.3 Effects of Sampling Temperature of LLM
Sampling temperature affects black-box uncertainty estimation because lower temperature reduces generation divergence. At low temperature, white-box methods perform better with few generations, while black-box methods recover as the number of generations increases.
- Temperature effects: Lower temperature produces a less divergent posterior, making sampling-based uncertainty and confidence harder to estimate.The temperature-0.5 experiments examine this effect across the reported AUARC and AUROC settings.
- Temperature effects: At lower temperature and small m, the white-box method SE generally outperforms the proposed black-box methods.This comparison is reported for the low-temperature experiments.
- Temperature effects: When m ≥10, the black-box methods’ performance picks up relative to their low-temperature performance.This indicates that more sampled generations partly offset the difficulty created by lower temperature.
- Generation without rejection: Even without rejection, most confidence measures select better responses and significantly improve accuracy.Table 14 reports accuracy from choosing the most confident answer among m = 20 generations.
- Evaluation settings: The low-temperature experiments report AUARC, AUROC, selected-answer accuracy, and adaptive calibration error across corresponding comparison tables.Adaptive calibration error is evaluated after histogram binning, with lower values preferred.