PaperlayerView exact revision on arXiv

Source-linked AI summary

SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales

Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, Jing Gao

arXiv:2405.20974v3cs.CLcs.AIcs.LG

TL;DR

LLMs can fabricate information and often fail to express uncertainty, while existing confidence methods have limitations in calibration or granularity. SaySelf trains models to generate uncertainty-explaining rationales and fine-grained confidence estimates; evaluations report improved calibration across in-distribution and out-of-distribution datasets while maintaining task performance.

  • Problem

    LLMs can fabricate information and often hesitate to express uncertainty, while prior confidence approaches have calibration and granularity limitations.

    Source: §1 · p. 1

  • Method

    SaySelf builds supervised training data from sampled reasoning chains and uses reinforcement learning to calibrate fine-grained confidence alongside self-reflective rationales.

    Source: §3.1 · p. 4, §5 · p. 9

  • Results

    SaySelf significantly outperforms baselines on calibration error and confidence distinction across in-distribution and out-of-distribution datasets, while maintaining task performance.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    SaySelf’s self-reflective rationales capture uncertainty and can further improve calibration, while enabling low confidence on unanswerable questions.

    Source: §1 · p. 2

    Rationale training depends on multiple sampled reasoning chains, and unfaithful chains can produce unfaithful self-reflective rationales.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) often generate inaccurate or fabricated information and generally fail to indicate their confidence, which limits their broader applications. Previous work elicits confidence from LLMs by direct or self-consistency prompting, or constructing specific datasets for supervised finetuning. The prompting-based approaches have inferior performance, and the training-based approaches are limited to binary or inaccurate group-level confidence estimates. In this work, we present the advanced SaySelf, a training framework that teaches LLMs to express more accurate fine-grained confidence estimates. In addition, beyond the confidence scores, SaySelf initiates the process of directing LLMs to produce self-reflective rationales that clearly identify gaps in their parametric knowledge and explain their uncertainty. This is achieved by using an LLM to automatically summarize the uncertainties in specific knowledge via natural language. The summarization is based on the analysis of the inconsistency in multiple sampled reasoning chains, and the resulting data is utilized for supervised fine-tuning. Moreover, we utilize reinforcement learning with a meticulously crafted reward function to calibrate the confidence estimates, motivating LLMs to deliver accurate, high-confidence predictions and to penalize overconfidence in erroneous outputs. Experimental results in both in-distribution and out-of-distribution datasets demonstrate the effectiveness of SaySelf in reducing the confidence calibration error and maintaining the task performance. We show that the generated self-reflective rationales are reasonable and can further contribute to the calibration. The code is made public at https://github.com/xu1868/SaySelf.

1 Introduction

LLMs can fabricate information and often fail to signal uncertainty, while prior confidence methods have calibration, granularity, or latency limitations. SaySelf combines self-reflective rationales with fine-grained confidence estimates, and its evaluations report improved calibration while maintaining task performance.

Problem

  • LLMs can fabricate information and often hesitate to express uncertainty on unfamiliar questions, making reliable confidence estimates important.The concern also applies when answers contain multiple tokens.Source: §1 · p. 1

Prior work

  • Prior prompting approaches can calibrate poorly or increase inference latency, while training approaches often yield inaccurate or binary confidence estimates.The paper characterizes prior methods as having poor calibration, coarse-grained confidence, or long inference latency.Source: §1 · p. 1

Approach

  • SaySelf uses sampled reasoning chains to create self-reflective rationales that identify knowledge gaps and explain confidence estimates.An LLM compares semantically distinct chains and summarizes uncertainty in natural language from a first-person perspective for supervised fine-tuning.Source: §1 · pp. 1–2
  • Reinforcement learning rewards accurate, high-confidence predictions and penalizes overconfidence in incorrect responses.At inference, rationales and confidence estimates are produced without multiple sampling, significantly reducing inference time.Source: §1 · p. 2

Results

  • SaySelf significantly reduces confidence calibration error while maintaining task performance across knowledge-extensive question-answering tasks.Its rationales capture internal uncertainty, can further improve calibration, and the model can produce low confidence for unanswerable questions.Source: §1 · p. 2

Potential implications

  • The authors suggest that confidence explanations may promote trustworthiness and that rationales may guide later steps such as tool use or clarification questions.They also anticipate that SaySelf-trained models could support proactive learning protocols.Source: §1 · p. 2

2 Related Work

Prior confidence elicitation includes prompting and training approaches, while related work also studies hallucination and natural-language explanations. SaySelf distinguishes its rationales by explaining uncertainty as well as predictions.

LLMs’ Confidence Elicitation

  • Prompting can generate confidence scores or use answer consistency, but may reduce performance or increase inference latency; training methods include group-based estimates and binary R-Tuning labels.SaySelf is described as addressing the limited accuracy and granularity of prior confidence estimates.Source: §2 · pp. 2–3

LLMs’ Hallucination & Uncertainty Expression

  • Hallucination research addresses unsupported outputs, while uncertainty expression matters because models may fabricate answers to unknown questions.The passage links this tendency to instruction fine-tuning that can encourage helpfulness despite inability to solve a problem.Source: §2 · p. 3

LLMs’ Explainability

  • Unlike prediction-justifying explanations or CoT rationales, SaySelf’s self-reflective rationales also explain confidence and sources of uncertainty.SaySelf uses LLMs’ internal reasoning rather than human-annotated explanations, which may not be faithful to a specific model.Source: §2 · p. 3

3 SaySelf

SaySelf trains models to produce fine-grained confidence estimates and self-reflective rationales through supervised fine-tuning followed by reinforcement learning. The stages use sampled responses to construct training data and task supervision to calibrate confidence.

3 SaySelf

  • SaySelf’s two-stage framework combines supervised fine-tuning on rationale-and-confidence data with reinforcement learning to calibrate confidence for each instance.The training samples for both stages come from HotpotQA; trained models answer with confidence estimates and rationales without additional computational overhead.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Supervised Fine-Tuning

  • The supervised dataset pairs each question with a reasoning chain, self-reflective rationale, and confidence score derived from response consistency.For 90K HotpotQA questions, the method samples N responses, clusters them by semantic similarity, and selects one response per cluster; GPT-4 compares selected responses and summarizes uncertainty in first person.Source: §3.1 · pp. 3–5

3.2 Reinforcement Learning from Task Supervision

  • Reinforcement learning uses a reward based on answer accuracy and confidence, rewarding high confidence on correct answers and penalizing overconfidence on incorrect ones.The method uses PPO to optimize the reward; its confidence level is normalized between 0 and 1.Source: §3.2 · p. 5

3.3 Implementation Details

  • Implementation uses 100 sampled responses per question, temperature 1.2, similarity threshold 0.9, and batch size 8 for both training stages.Learning rates are 7e-5 for supervised fine-tuning and 1e-5 for reinforcement learning; response correctness is checked by whether annotated answers appear in the response.Source: §3.3 · p. 5

4 Experiments

Experiments assess SaySelf on knowledge-extensive QA tasks using calibration, accuracy, and rationale-faithfulness measures. SaySelf improves confidence calibration while maintaining task performance; ablations, unanswerable-question tests, and case studies examine its components and uncertainty detection.

4.1 Evaluation Setting

  • Evaluation spans six QA datasets and measures calibration, task accuracy, and rationale faithfulness against prompting- and training-based baselines.Calibration is assessed with ECE and AUROC; task performance uses accuracy, while GPT-4 rates whether rationales faithfully express uncertainty in sampled responses.Source: §4.1 · pp. 5–6

4.2 Main Experimental Results

  • SaySelf significantly reduces ECE and improves AUROC across in- and out-of-distribution datasets while generally maintaining task performance and generating rationales that reflect sampled-response uncertainty.For rationale evaluation, StrategyQA is an exception: its True/False format typically yields only one or two selected responses per question, resulting in high variance.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

4.3 Ablation Study

  • Ablations indicate that reinforcement learning, the reward-function design, self-reflective rationales, and confidence estimates each contribute to calibrated confidence.The study compares removal of reinforcement learning, rationales, and confidence estimates, as well as use of a naive reward function.Source: Table 1 · p. 7, §4.3 · p. 8

4.4 Unanswerable Questions

  • On SQUADRUN, SaySelf lowers confidence on unanswerable questions while maintaining confidence on answerable questions, achieving the best confidence gap between the subsets.The evaluation compares average confidence across the answerable and unanswerable subsets.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 Case Study

  • In a HotpotQA case study, a rationale identifies competing Howard Centre locations and founding-year claims, treating Letchworth as likely and Cambridge as highly improbable.The examples use clustered selections from 100 sampled responses.Source: §4.5 · pp. 8–9

5 Conclusion

SaySelf combines supervised fine-tuning on a model-specific dataset of self-reflective rationales and confidence estimates with reinforcement learning using a designed reward function. Evaluations across diverse datasets show reduced calibration errors, maintained task performance, and insightful rationales.

5 Conclusion

  • Across diverse datasets, SaySelf reduces calibration errors, maintains task performance, and generates insightful rationales.The framework combines supervised fine-tuning with reinforcement learning.Source: §5 · p. 9

Limitations

SaySelf depends on multiple sampled reasoning chains to develop training rationales, and the faithfulness of chain-of-thought reasoning remains debated. Unfaithful chains can yield unfaithful rationales, although the paper’s ablation finds that the rationales substantially enhance calibration.

Limitations

  • SaySelf depends on multiple sampled reasoning chains to develop self-reflective rationales for training.The paper notes an ongoing debate about whether chain-of-thought reasoning faithfully represents a model’s thinking; unfaithful chains can produce unfaithful rationales.Source: Limitations · p. 9
  • Despite this concern, the ablation study finds that self-reflective rationales substantially enhance calibration performance.The authors suggest that methods aimed at improving chain-of-thought faithfulness could further improve SaySelf.Source: Limitations · p. 9

Ethical Considerations

The paper uses Mistral 7B and GPT-4 and notes risks associated with LLM research. It also warns that inaccurate confidence estimates or rationales could contribute to harmful decisions, while describing its data as anonymized and annotator participation as voluntary.

Ethical Considerations

  • The work uses Mistral 7B and GPT-4, so the authors say it shares risks associated with LLM research.Source: Ethical Considerations · p. 9
  • If users over-rely on confidence estimates or rationales that fail to capture true uncertainty, decisions based on outputs could be harmful.The authors advise checking important information before making crucial decisions.Source: Ethical Considerations · p. 9
  • The authors state that the publicly available datasets are anonymized and that their work therefore does not propagate privacy problems about specific entities.The datasets are listed as available under their respective licenses.Source: Ethical Considerations · p. 9
  • Human annotations were voluntary, and the authors discussed their use and potential risks with annotators beforehand.The authors describe the amount of annotation work as small.Source: Ethical Considerations · p. 9

A Prompt

The appendix prompts use sampled answers and their confidence scores to elicit brief uncertainty explanations, then evaluate whether explanations account for inconsistencies. They also specify output constraints and provide examples for both prompting stages.

A.1 GPT-4 Summarization Prompt

  • The summarization prompt asks GPT-4 to explain uncertainty by summarizing inconsistencies among correct and incorrect responses to a question.Source: §A.1 · p. 12
  • The prompt requires a brief first-person explanation grounded in the model’s knowledge, including each response’s confidence score and avoiding claims about correctness.If the responses are consistent with the correct response, it instructs GPT-4 to return N/A.Source: §A.1 · p. 12
  • The example prompt supplies a question about the actress starring in Sky High and the actor she married, alongside a low-confidence correct response and two alternatives.Source: §A.1 · pp. 12–13
  • Table 5 reports AUROC results for baselines, SaySelf, and ablations; Table 6 reports ECE and accuracy for SaySelf and baselines on Llama 3 8B.Source: Table 5 · p. 13, Table 6 · p. 13
  • An example rationale expresses uncertainty among actress identities and confusion between voice-acting roles and on-screen appearances.It gives Kristen Bell as a likely answer, Kelly Preston as an alternative, and also mentions Kelly Clarkson.Source: §A.1 · p. 13

A.2 GPT-4 Evaluation Prompt

  • The evaluation prompt asks GPT-4 to judge whether an explanation accounts for inconsistencies in sampled responses, whose percentages indicate their frequency.Source: §A.2 · p. 13
  • The evaluator must give reasoning and a 0–10 explanation-quality score in a specified format; an example rates a clear, logical rationale 9.Source: §A.2 · pp. 13–14

B Empirical Evidence

Reasoning chains within clusters were highly similar, and sampled clusters showed that remaining differences were mainly minor wording and sentence-structure variations. These findings support selecting one instance per cluster at random.

  • 0.94 average similarity across reasoning chains indicates high consistency within each clustering group.Source: B Empirical Evidence · p. 14
  • The clusters are 58%, 80%, and 74%, respectively; remaining reasoning chains vary primarily in minor wording and sentence-structure differences, supporting selection of one instance per cluster at random.Source: B Empirical Evidence · p. 14
  • The remaining chain variations were mainly minor wording and sentence-structure differences, supporting random selection of one instance per cluster.Source: B Empirical Evidence · p. 14

C Human Annotations for GPT-4 Evaluation

Human annotators assessed 200 questions using a 1-to-10 scale, and their average scores correlated with GPT-4 evaluations at 0.89. The authors report this as evidence of GPT-4 evaluation reliability.

D Experiments of SaySelf on Llama

On Llama 3 8B, SaySelf generalized across base models and outperformed the baselines in ECE without significant accuracy loss.

Loading 2405.20974v3…