PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations

Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, Mark Ibrahim

arXiv:2404.10960v1cs.CLcs.AI

TL;DR

LLMs can answer incorrectly, hallucinate on unanswerable questions, or produce unsafe responses, motivating abstention when uncertain. The study evaluates statistical and verbalized uncertainty for abstention and reports improved correctness, fewer hallucinations, and greater safety across these settings.

  • Problem

    LLMs can answer incorrectly, fabricate answers to unanswerable questions, and produce harmful or unsafe responses, limiting their reliability.

    Source: §1 · p. 1

  • Method

    The study tests uncertainty-based abstention in question-answering using statistical uncertainty measures and verbalized In-Dialogue Uncertainty across models with and without RLHF.

    Source: §1 · p. 2

  • Results

    Statistical uncertainty improved correctness by 2% to 8%, In-Dialogue Uncertainty detected approximately 50% of unanswerable questions, and statistical uncertainty filtered 70% to 99% of unsafe responses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Uncertainty-based abstention can improve LLM reliability across correctness, hallucination, and safety settings, with potential relevance to higher-stakes applications.

    Source: §1 · p. 2

Abstract

from arXiv · show

A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don't know. Inspired by analogous approaches in classification, this study explores the feasibility and efficacy of abstaining while uncertain in the context of LLMs within the domain of question-answering. We investigate two kinds of uncertainties, statistical uncertainty metrics and a distinct verbalized measure, termed as In-Dialogue Uncertainty (InDU). Using these uncertainty measures combined with models with and without Reinforcement Learning with Human Feedback (RLHF), we show that in all three situations, abstention based on the right kind of uncertainty measure can boost the reliability of LLMs. By sacrificing only a few highly uncertain samples we can improve correctness by 2% to 8%, avoid 50% hallucinations via correctly identifying unanswerable questions and increase safety by 70% up to 99% with almost no additional computational overhead.

1 Introduction

LLMs can give incorrect answers, fabricate answers to unanswerable questions, and produce unsafe responses. This study tests abstention based on different forms of uncertainty and reports gains across all three settings.

Uncertainty measures

  • Statistical uncertainty and In-Dialogue Uncertainty (InDU) capture distinct signals: token-probability distributions and verbal expressions such as hedges or “I don’t know.”Source: §1 · p. 2

Study scope

  • The study evaluates uncertainty-based abstention for incorrect answers, hallucinations on unanswerable questions, and unsafe responses in question-answering.It examines statistical and In-Dialogue uncertainty, and the impact of RLHF.Source: §1 · p. 2

Findings

  • Statistical uncertainty improved correctness by 2% to 8%, In-Dialogue Uncertainty detected approximately 50% of unanswerable questions, and statistical uncertainty filtered 70% to 99% of unsafe responses.The safety result used RLHF fine-tuned models.Source: §1 · p. 2

Implication

  • Uncertainty-based abstention is proposed as a way to improve LLM reliability, with potential relevance to higher-stakes applications.Source: §1 · p. 2

2 Related Work

Related work covers probability-based uncertainty estimation for generation, explicit uncertainty elicitation, knowledge boundaries on unanswerable questions, and safety against adversarial prompts.

Uncertainty estimation

  • Probability-based methods adapt predictive entropy from token-wise conditional probabilities [2021], while semantic entropy estimates uncertainty over meaning clusters [Kuhn et al., 2023].Jiang et al. [2021] and Malinin and Gales [2021] compute predictive entropy; Kuhn et al. [2023] use semantic likelihoods.Source: §2 · pp. 2–3

Explicit uncertainty elicitation

  • Prompting or fine-tuning can elicit explicit uncertainty, but these approaches may add computational effort and be sensitive to distribution shifts.Other work uses sampled answers, discrete labels, numeric confidence, or predefined confidence words [2022; 2023].Source: §2 · p. 3

Unanswerable questions

Safety

3 Uncertainty Estimation

The paper estimates uncertainty using probability-based measures and verbalized hedging, then assesses whether these measures rank responses by correctness and support abstention across thresholds.

3.1 Statistical Uncertainty Measures

  • Statistical uncertainty is extracted directly from model probabilities using single-inference negative log-likelihood or multiple-inference predictive and semantic entropy.The study employs all three measures.Source: §3.1 · p. 3

3.1.1 Negative Log-Likelihood

  • Negative log-likelihood scores a generated sequence given its prompt using token-wise conditional probabilities conditioned on the preceding tokens.Source: §3.1.1 · pp. 3–4

3.1.2 Predictive Entropy

  • Predictive entropy is estimated by sampling N sequences from a single model’s output distribution.The underlying entropy is over the output random variable given the input.Source: §3.1.2 · p. 4

3.1.3 Semantic Entropy

  • Semantic entropy computes uncertainty over likelihoods of meaning clusters, whose number reflects the diversity of meanings in the output distribution.Responses are grouped into semantic clusters using bidirectional entailment.Source: §3.1.3 · p. 4

3.2 In-Dialogue Uncertainty

  • In-Dialogue Uncertainty (InDU) is measured by counting hedge words in a response; higher counts indicate greater implicit uncertainty.The measure aims to capture hedging without explicit prompting.Source: §3.2 · p. 4

3.3 Assessing the quality of uncertainty measures

  • AUROC measures how well uncertainty ranks correct versus incorrect responses, while ARC and AUARC assess accuracy as uncertain responses are rejected across thresholds.The same assessment approach is applied to hallucination and safety settings using their corresponding labels.Source: §3.3 · pp. 4–5

4 Experimental Setup

The experiments assess correctness, hallucinations on unanswerable questions, and safety using Llama-family models and setting-specific datasets and outcome measures.

4 Experimental Setup

  • The three experimental settings are truthful answering of answerable questions, abstention on unanswerable questions, and distinguishing safe from unsafe responses.Source: §4 · p. 5

4.1 Correctness Settings

  • Correctness experiments use pretrained and RLHF-fine-tuned Llama 2 models at 7B, 13B, and 70B, with TriviaQA, SciQA, CoQA, StrategyQA, and GSM8K.Ground-truth correctness uses fuzzy exact match, which checks whether the reference answer is encompassed in the response.Source: §4.1 · p. 5

4.2 Hallucination Settings

  • Hallucination experiments use the SelfAware dataset’s unanswerable questions and semantically closest answerable counterparts, with answerability labels used to compute AUROC.The models are the same as in the correctness settings.Source: §4.2 · p. 5

4.3 Safety settings

  • Safety experiments compare pretrained and RLHF Llama 2 models with Vicuna on AutoDAN and AttaQ adversarial prompts.Main-paper safe-response rates use a keyword-based measure; Llama Guard results and comparisons with human evaluations are reported in the appendices.Source: §4.3 · pp. 5–6

5 Results

Across correctness, hallucination, and safety scenarios, the results show that abstaining based on the appropriate uncertainty measure can improve model reliability at minimal computational cost. Statistical uncertainty supports correctness and safety decisions, while In-Dialogue Uncertainty helps identify unanswerable questions.

5 Results

  • Uncertainty-based abstention improves correctness, reduces hallucinations, and boosts safe responses across the three evaluated scenarios.The reported approach uses the uncertainty measure best suited to each scenario and has minimal computational cost.Source: §5 · p. 6, Figure 3 · p. 8

5.1 Statistical uncertainty can indicate when to abstain from responding incorrectly

  • Statistical uncertainty distinguishes correct from incorrect answers across Q&A tasks; RLHF preserves this signal despite higher confidence and less response diversity.For RLHF models, semantic entropy has an average AUROC of 0.69, while In-Dialogue Uncertainty averages 0.53; both measures are close to random on GSM8K.Source: §5.1 · p. 6, Table 1 · p. 7, Figure 2 · p. 7
  • Rejecting the 5% most uncertain TriviaQA samples raises RLHF accuracy from 84.4% to 86.0%; rejecting 25% raises it by 8.2% to 92.6%.The abstention thresholds use semantic entropy.Source: §5.1 · pp. 6–7

5.2 In-Dialogue uncertainty can reduce hallucinations for unanswerable questions

  • In-Dialogue Uncertainty distinguishes answerable from unanswerable questions better than predictive or semantic entropy, especially for RLHF models.It can filter responses to nearly 50% of unanswerable questions while incorrectly refusing 10% of answerable questions.Source: §5.2 · pp. 7–8, Table 2 · p. 8

5.3 Statistical uncertainty can improve the safety of RLHF models

  • Only RLHF models show well-performing statistical-uncertainty AUROCs for identifying unsafe responses; In-Dialogue Uncertainty performs near randomly.The comparison covers pretrained, supervised instruction-tuned, and RLHF models across AttaQ and AutoDAN.Source: §5.3 · p. 9, Table 3 · p. 9
  • Abstaining from the 10% most uncertain RLHF responses raises safe-response rates from 92.1% to 95.1% on AttaQ and from 92.5% to 99.4% on AutoDAN.The paper also reports filtering 99% of unsafe AutoDAN responses with 10% false refusals, and 70% of unsafe AttaQ responses with 30% false refusals.Source: §5.3 · p. 9

6 Discussion

The right uncertainty measure depends on the question-answering use case: statistical uncertainty supports correctness and safety, while In-Dialogue Uncertainty supports abstention on unanswerable questions. Accuracy-Rejection Curves assess how well uncertainty-based rejection improves retained-sample accuracy or safety.

6 Discussion

  • Statistical uncertainty suits correctness and safety, whereas In-Dialogue Uncertainty suits unanswerable questions.The paper emphasizes matching both the uncertainty type and model form to the use case.Source: §6 · p. 9

A Accuracy-Rejection Curves (ARCs) for Correctness and Safety Scenarios

  • Higher AUARC indicates an uncertainty measure better separates incorrect from correct samples, or unsafe from safe samples, as rejection progresses.An Accuracy-Rejection Curve tracks accuracy on the retained subset as samples are rejected from most to least uncertain.Source: A Accuracy-Rejection Curves (ARCs) for… · p. 17

A.1 Correctness Settings

In correctness and safety settings, uncertainty-based rejection improves the quality of responses retained. The reported AUARC comparisons quantify this across TriviaQA, SciQA, AttaQ, and AutoDAN.

A.1 Correctness Settings; A.2 Safety Settings

  • As progressively more samples are rejected, accuracy among retained samples increases for TriviaQA and SciQA, and the proportion of safe responses increases for AttaQ and AutoDAN.The safety ARC uses a keyword-based safety metric.Source: §A.1 · p. 17, §A.2 · p. 18
  • Correctness ARC accuracy is the ratio of correct samples to all remaining samples; safety ARC accuracy is the ratio of safe samples to all remaining samples.The figures show rejection rate against these respective retained-sample measures.Source: §A.1 · p. 17, Figure 4 · p. 17, §A.2 · p. 18
  • 0.94 AUARC on TriviaQA for both models, 0.83 versus 0.77 on SciQA (RLHF versus pretrained), and 0.97 on AttaQ and near-perfect 0.99 on AutoDAN.The correctness values compare pretrained and RLHF-finetuned models; the safety values are for the Llama2 70b RLHF-finetuned model.Source: Figure 4 · p. 17, §A.1 · p. 17, Figure 5 · p. 18

B Additional results for Correctness Settings

Additional fact-based Q&A results show that RLHF finetuning shifts predictive confidence and response diversity, while uncertainty awareness remains largely intact. Comparisons across metrics and datasets show that the awareness advantage is not uniform between model types.

B Additional results for Correctness Settings

C Additional results for Hallucination Settings

For unanswerable questions, In-Dialogue Uncertainty distinguishes responses through hedge-word use, whereas mean statistical uncertainty measures show little separation. InDU identifies about half of unanswerable questions with a limited false-refusal trade-off.

C Additional results for Hallucination Settings

  • Approximately 50% of unanswerable questions are identified by In-Dialogue Uncertainty, with incorrect refusals on only 10% of answerable questions.The reported ROC curve illustrates this trade-off.Source: C Additional results for Hallucination… · p. 19
  • Responses to unanswerable questions contain more hedge words than responses to answerable questions across model sizes and RLHF settings.The hedge-word difference is slightly less pronounced for pretrained models.Source: C Additional results for Hallucination… · p. 19
  • Mean statistical measures differ little between answerable and unanswerable questions, adversely affecting AUROC.This contrasts with the greater hedge-word counts associated with In-Dialogue Uncertainty.Source: C Additional results for Hallucination… · p. 19

D Additional Results for Safety Settings

These additional results show that uncertainty-based filtering can identify unsafe responses, distinguish unanswerable questions from incorrectly answered ones, and use hedge words as an answerability signal.

D Additional Results for Safety Settings

  • 50% of unanswerable questions can be filtered by In-Dialogue Uncertainty at the cost of incorrectly refusing 10% of answerable questions.The passage describes this as the ROC trade-off for distinguishing answerable from unanswerable questions.Source: Figure 9 · p. 20
  • 70% of unsafe responses on AttaQ and almost 100% on AutoDAN can be filtered, at false-refusal costs of 30% and almost none, respectively.The safety ROC results are reported for the RLHF-finetuned Llama2-7b.Source: D Additional Results for Safety Settings · p. 19, Figure 10 · p. 21

E Combining Hallucination & Correctness

  • In-Dialogue Uncertainty performs nearly as well at separating incorrectly answered from unanswerable questions as at separating answerable from unanswerable questions; statistical measures perform no better than random.For the first distinction, Table 4 reports higher In-Dialogue Uncertainty AUROCs after RLHF finetuning than for the base model.Source: E Combining Hallucination & Correctness · p. 20, Table 4 · p. 21

F Further Insights into In-Dialogue Uncertainty

  • 4.79 hedge words per unanswerable response versus 1.67 per answerable response make verbalized uncertainty a useful answerability signal.The RLHF-finetuned model also uses more diverse hedge words than the pretrained model; the broader gap between answerable and unanswerable questions remains largely consistent across model sizes and RLHF settings.Source: Figure 8 · p. 20, F Further Insights into In-Dialogue… · p. 21

G Validity of Fuzzy Exact Match

Fuzzy exact match counts a response as correct when it contains the reference answer, and its accuracy more closely matches human judgments than exact match or F1 on the evaluated samples.

G Validity of Fuzzy Exact Match

  • Fuzzy exact match counts a response as correct when the reference answer is contained in it, rather than requiring a word-for-word match.The authors adopt it as their correctness metric after finding closer alignment with human evaluations than exact match and F1.Source: G Validity of Fuzzy Exact Match · pp. 21–22
  • On TriviaQA, human accuracy was 87.5%, fuzzy exact match 85.0%, exact match 14.5%, and F1 27.8%.These figures are from 200 evaluated samples.Source: G Validity of Fuzzy Exact Match · p. 22
  • On SciQA, human accuracy was 77.0%, fuzzy exact match 65.0%, exact match 2.5%, and F1 7.0%.These figures are from 200 evaluated samples.Source: G Validity of Fuzzy Exact Match · p. 22

H Validity of Safety Metrics

The authors compare two safety metrics with human judgments and report that keyword-based scoring matches those judgments on AutoDAN, while Llama Guard does not.

H Validity of Safety Metrics

  • On AutoDAN, human evaluation and the keyword-based metric both gave a 92.5% safe response rate, while Llama Guard gave 100%.Each response was assessed by three human judges, with majority vote determining safety; the authors therefore use the keyword-based metric in the main paper.Source: H Validity of Safety Metrics · p. 22
  • Only RLHF finetuning produced well-performing safety AUROCs on AttaQ; pretraining and supervised instruction tuning were insufficient.The comparison uses Llama Guard to assess safe versus unsafe responses.Source: Table 6 · p. 22
  • RLHF-finetuned Llama2-70b uses more diverse hedge words than the pretrained model and more hedge words for unanswerable than answerable questions.The passages report these patterns without giving numerical counts.Source: Figure 11 · p. 23, Figure 12 · p. 23
Loading 2404.10960v1…