PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness

Jiuhai Chen, Jonas Mueller

arXiv:2308.16175v2cs.CLcs.AI

TL;DR

LLMs can generate convincing but unreliable answers, while black-box APIs expose neither the training information nor token probabilities needed by many uncertainty methods. BSDetector estimates confidence by combining repeated-response consistency with self-reflection, and its scores identify unreliable outputs; selecting the highest-confidence response can also improve accuracy without additional training.

  • Problem

    Black-box LLM APIs lack the training information, token probabilities, and validation data required by many existing uncertainty-estimation methods, despite producing unreliable answers.

  • Method

    BSDetector combines observed consistency from repeated varied responses with intrinsic self-reflection certainty to score an LLM answer.

  • Results

    BSDetector confidence scores are lower for bad responses, outperform baselines across benchmark datasets, and can improve answers by selecting the highest-confidence sample.

  • Takeaways & Limitations

    Confidence scores can support safer high-stakes use through human review, alternative-provider routing, abstention, and more accurate answer selection without additional training.

  • Takeaways & Limitations

    Observed consistency can be unreliable when natural-language-inference models encounter single-word answers that are poorly represented in their training data.

Abstract

from arXiv · show

We introduce BSDetector, a method for detecting bad and speculative answers from a pretrained Large Language Model by estimating a numeric confidence score for any output it generated. Our uncertainty quantification technique works for any LLM accessible only via a black-box API, whose training data remains unknown. By expending a bit of extra computation, users of any LLM API can now get the same response as they would ordinarily, as well as a confidence estimate that cautions when not to trust this response. Experiments on both closed and open-form Question-Answer benchmarks reveal that BSDetector more accurately identifies incorrect LLM responses than alternative uncertainty estimation procedures (for both GPT-3 and ChatGPT). By sampling multiple responses from the LLM and considering the one with the highest confidence score, we can additionally obtain more accurate responses from the same LLM, without any extra training steps. In applications involving automated evaluation with LLMs, accounting for our confidence scores leads to more reliable evaluation in both human-in-the-loop and fully-automated settings (across both GPT 3.5 and 4).

1 INTRODUCTION

BSDetector wraps black-box LLM inference with uncertainty estimation because convincing responses can be unreliable and token probabilities or calibration data are unavailable. It produces confidence scores that identify questionable outputs and can improve answers by selecting among sampled responses.

  • Black-box LLMs can produce plausible-sounding but incorrect answers, limiting their use in high-value applications.
  • Traditional uncertainty methods cannot assume access to training data, token probabilities, or a validation set, so black-box methods must wrap inference.
  • BSDetector repeatedly calls the LLM with varied prompts and sampling temperatures, spending extra computation to estimate trustworthiness.
  • Its confidence scores are reliably lower for responses that are more likely to be bad.
  • Sampling multiple responses and selecting the highest-confidence answer can improve Question-Answering accuracy, despite requiring 10x the inference computation of a reference answer.The scores can also support human review, routing to another provider, or responding that the system does not know.

2 RELATED WORK

Prior LLM uncertainty methods often require token-level probabilities, fine-tuning, or limited task settings. BSDetector instead targets black-box access and evaluates confidence estimation across reasoning, arithmetic, and factual-knowledge domains.

  • Semantic entropy and related probability-based methods require token-level probabilities that black-box APIs often do not expose.
  • Some prior approaches fine-tune models or study only limited tasks, leaving their broader applicability unclear.
  • BSDetector requires no additional training and produces effective estimates across reasoning, arithmetic, and factual-knowledge domains.

3 BSDETECTOR UNCERTAINTY ESTIMATION

BSDETECTOR estimates LLM confidence by combining agreement among varied sampled answers with the model’s self-assessment of its original answer. It uses semantic contradiction measures, calibrated sampling, and follow-up confidence judgments to produce an overall score.

  • 3.1 OBSERVED CONSISTENCY: Observed Consistency measures contradiction among multiple varied answers to the same question, extending self-consistency through semantic similarity for open- and closed-form tasks.The method compares sampled outputs with the original answer rather than requiring exact matches alone.
  • 3.1 OBSERVED CONSISTENCY: BSDETECTOR samples varied responses by changing temperatures or prompts, with higher sample counts improving uncertainty estimates; k = 5 worked well in practice.The prompt modification includes Chain-of-Thought and formatting guidelines, while leaving the original response prompt unchanged.
  • 3.1 OBSERVED CONSISTENCY: NLI-based contradiction probabilities can be unreliable for single-word answers, so the method combines semantic contradiction with an exact-match indicator for stability.The NLI computation is negligible compared with sampling each LLM answer, but its training-data limitations motivate the additional indicator.
  • 3.2 SELF-REFLECTION CERTAINTY: Self-reflection Certainty asks the LLM follow-up questions about the correctness of its original answer, providing an intrinsic estimate that can reveal consistently repeated but nonsensical outputs.Multiple-choice confidence questions are preferred because continuous 0–100 ratings tended to exceed 90.
  • 3.3 OVERALL CONFIDENCE ESTIMATE: The overall confidence score combines Observed Consistency and Self-reflection Certainty as C = βO + (1 −β)S, with β fixed at 0.7 in the experiments.The weighting reflects trust in the LLM’s ability to provide calibrated self-reflection assessments.

4 APPLICATION: GENERATING MORE RELIABLE ANSWERS FROM ANY LLM

BSDETECTOR can select a more trustworthy answer from several responses generated by the same LLM. It scores each candidate using shared observed-consistency computations and candidate-specific self-reflection assessments.

  • 4 APPLICATION: GENERATING MORE RELIABLE ANSWERS FROM ANY LLM: The method evaluates multiple candidate answers from the same LLM, including the original response, and returns the candidate with the highest assessed trustworthiness.Candidates are generated with the same prompt through temperature sampling.
  • 4 APPLICATION: GENERATING MORE RELIABLE ANSWERS FROM ANY LLM: For each candidate, BSDETECTOR reuses sampled outputs to compute observed consistency and separately prompts the LLM for self-reflection certainty.Reusing the sampled outputs reduces the computation needed to assess multiple candidates.
  • 4 APPLICATION: GENERATING MORE RELIABLE ANSWERS FROM ANY LLM: A candidate is preferred when fewer likely sampled answers contradict it and the LLM is more certain about its correctness during self-reflection.The selection therefore combines extrinsic agreement with intrinsic confidence.

5 APPLICATION: MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION

LLM-based evaluation of open-domain answers remains difficult to assess reliably, even when advanced evaluators such as GPT-4 are used. BSDetector supports human-in-the-loop review or fully automated filtering by identifying low-confidence evaluations.

  • 5 APPLICATION: MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: Open-domain answer quality is challenging to evaluate automatically, and GPT-3.5-turbo or GPT-4 evaluators still raise reliability questions.These models are used in automated evaluation and in evaluating outputs from instruction-tuned systems.
  • 5 APPLICATION: MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: BSDetector confidence scores identify evaluations that are likely untrustworthy, distinguishing evaluator confidence from the trustworthiness of the answer generator.The low-confidence subset A contains answers whose corresponding LLM evaluations receive the lowest scores.
  • 5 APPLICATION: MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: Under a limited labor budget, humans can inspect only the low-confidence evaluations, while fully automated systems can omit those answers from the evaluation set.Both approaches are presented as alternatives to inspecting every open-domain answer manually.

6 EXPERIMENTS

Experiments evaluate BSDETECTOR’s uncertainty estimates, answer selection, and GPT-4-based evaluation across QA benchmarks and evaluation settings. BSDETECTOR outperforms baseline uncertainty methods, improves answer accuracy, and supports more reliable human-in-the-loop and fully automated evaluation.

  • 6.1 CALIBRATION OF UNCERTAINTY ESTIMATES: AUROC evaluates whether correct answers receive higher uncertainty scores than incorrect answers, with 1.0 ideal and 0.5 random.The study uses AUROC to assess the quality of uncertainty estimates.
  • 6.1 CALIBRATION OF UNCERTAINTY ESTIMATES: BSDETECTOR significantly outperforms all baseline uncertainty estimation methods across the benchmark datasets.The results indicate that its confidence scores align well with response accuracy.
  • 6.2 GENERATING MORE RELIABLE ANSWERS FROM ANY LLM: Selecting the highest-confidence response from five samples improves accuracy across all tasks compared with the single reference answer.The filtering procedure requires 10x more LLM inference computation than the reference answer.
  • 6.3 MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: GPT-4 evaluation achieved 83.67% accuracy against ground-truth correctness, while rating agreement produced approximately 0.707 MSE against human ratings.These results motivate confidence-based methods for improving automated evaluation reliability.
  • 6.3 MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: Confidence-based human selection yields more reliable combined evaluations than random selection across both studied datasets.The human-evaluated subset consists of answers whose GPT-4 evaluations have the lowest BSDETECTOR confidence scores.
  • 6.3 MORE RELIABLE LLM-BASED (AUTOMATED) EVALUATION: Discarding the 20% lowest-confidence evaluations improves average GPT-4 evaluation reliability across both datasets despite increasing variance from the smaller evaluation set.The retained set contains 400 of the original 500 answers.

7 DISCUSSION

BSDETECTOR provides black-box LLM confidence estimates for identifying unreliable outputs and selecting more accurate responses without additional training. The discussion also identifies reducing the computational cost of confidence calibration as an open question.

  • 7 DISCUSSION: BSDETECTOR computes confidence scores for unreliable or speculative outputs from any LLM, including models accessible only through black-box APIs.The method combines confidence estimation with multiple-response selection to improve answers without additional training.
  • 7 DISCUSSION: Sampling multiple answers and selecting the highest-confidence response can produce more accurate outputs from the same LLM without additional training.The paper frames this as an application of its uncertainty estimates beyond merely flagging untrusted responses.
  • 7 DISCUSSION: Minimizing the computation needed to achieve a desired level of confidence-score calibration remains an open question.The paper suggests adaptive procedures that spend more computation on answers whose confidence is hardest to estimate.

A.1 DETAILS ABOUT NLI MODEL

The NLI-based similarity component compares each sampled answer with the reference answer in both orders and averages contradiction-based similarity scores. This bidirectional calculation reduces positional bias in the NLI model.

  • A.1 DETAILS ABOUT NLI MODEL: NLI forms input by concatenating each sampled answer yi with the reference answer y and returns probabilities for three classes.The method uses the non-contradiction probability as the similarity measure.
  • A.1 DETAILS ABOUT NLI MODEL: The similarity score averages 1 − p_contradiction across both answer orderings for each sampled response.Evaluating both (yi, y) and (y, yi) is intended to mitigate positional bias.

A.2 COMPUTE COSTS

BSDETECTOR requires modest extra computation for self-certainty reflection, but its performance remains superior to temperature sampling even when the latter receives twice as many outputs.

  • BSDETECTOR incurs a slight additional cost for self-certainty reflection compared with baseline Temperature Sampling.
  • Even after doubling Temperature Sampling outputs, its performance remains inferior to BSDETECTOR.

A.3 PROMPTS USED IN BSDETECTOR

The ablation studies examine how output diversity, chain-of-thought prompting, and response-similarity metrics affect BSDETECTOR confidence estimation. More outputs and CoT improve diversity, while the similarity metric is essential for distinguishing responses.

  • A.4.1 INCREASING THE NUMBER OF OUTPUTS AND INTEGRATING COT PROMPT INTRODUCE MORE DIVERSITY?: Table 3a compares BSDETECTOR using 5 versus 10 outputs, with 10 outputs outperforming 5 on every dataset.For GSM8K, SVAMP, and TriviaQA, the improvement is marginal.
  • A.4.1 INCREASING THE NUMBER OF OUTPUTS AND INTEGRATING COT PROMPT INTRODUCE MORE DIVERSITY?: Because doubling API calls yields only slight gains, the experiments retain 5 outputs as a cost-performance trade-off.
  • A.4.1 INCREASING THE NUMBER OF OUTPUTS AND INTEGRATING COT PROMPT INTRODUCE MORE DIVERSITY?: Table 3b indicates that CoT prompting is essential for introducing response diversity and achieving good confidence-estimation performance.
  • A.4.1 INCREASING THE NUMBER OF OUTPUTS AND INTEGRATING COT PROMPT INTRODUCE MORE DIVERSITY?: Figure 6 presents the prompts used to produce BSDETECTOR confidence scores.
  • A.4.2 EFFECT OF DIFFERENT SENTENCE SIMILARITY METRICS: The similarity metric used in BSDETECTOR is essential for discerning differences among varied responses.
Loading 2308.16175v2…