PaperlayerView exact revision on arXiv

Source-linked AI summary

Cycles of Thought: Measuring LLM Confidence through Stable Explanations

Evan Becker, Stefano Soatto

arXiv:2406.03441v1cs.CLcs.LG

TL;DR

LLMs can confidently give wrong answers, while common confidence approaches may be miscalibrated or rely on heuristics. This paper estimates confidence from sampled explanations, treating explanation-conditioned models as test-time classifiers. Across five datasets, it reports improved selective uncertainty over baselines on average, especially for GPT-4-Turbo and more complex questions.

  • Problem

    LLMs can confidently provide incorrect answers, and token-logit confidence is not guaranteed to correspond to correctness.

    Source: §1 · p. 1

  • Method

    The method samples explanations, reweights them by entailment probability, and combines explanation-conditioned answer distributions as test-time classifiers.

    Source: §3.1 · p. 4, §3.2 · p. 5

  • Results

    Across five benchmark datasets, the method improves selective uncertainty on average, particularly for GPT-4-Turbo and more complex questions such as MMLU Law.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The results support evaluating uncertainty with respect to test-time model parameters and data, rather than only empirical calibration on previously seen data.

    Source: §5 · p. 7

    The evaluation is limited to multiple-choice datasets, and entailment probability sometimes fails to distinguish logically incorrect explanations.

    Source: §5 · p. 9

Abstract

from arXiv · show

In many high-risk machine learning applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer. While utilizing explanations is not a new idea in and of itself, by interpreting each possible model+explanation pair as a test-time classifier we can calculate a posterior answer distribution over the most likely of these classifiers. We demonstrate how a specific instance of this framework using explanation entailment as our classifier likelihood improves confidence score metrics (in particular AURC and AUROC) over baselines across five different datasets. We believe these results indicate that our framework is both a well-principled and effective way of quantifying uncertainty in LLMs.

1 Introduction

The paper introduces stable explanations confidence, which estimates uncertainty from generated explanations; across five datasets, it improves selective-uncertainty performance on average over baselines. The motivation is that LLMs can be overconfident, while existing confidence approaches have important limitations.

1 Introduction

  • LLMs can confidently give incorrect answers, and even the largest models can fail on some question types.Such responses can mislead non-expert users.Source: §1 · p. 1
  • Softmax token logits are not guaranteed to reflect the probability that an answer is correct.The mismatch between token-logit confidence and correctness has been observed empirically for LLMs.Source: §1 · p. 1
  • Black-box confidence methods avoid internal model access but often rely on self-verbalized confidence or response consistency, and can remain overconfident.Access to model internals may be infeasible because of proprietary restrictions and computational expense.Source: §1 · pp. 1–2
  • The method uses test-time explanations to adjust the decision boundary, aiming to distinguish stable from unstable explanations despite distribution differences and potentially biased chain-of-thought reasoning.The authors note that chain-of-thought explanations can be biased and may not correspond to the correct answer.Source: §1 · p. 2
  • Across five datasets, stable explanations confidence improves selective-uncertainty performance on average over baselines, particularly on more complex questions.The evaluation includes CommonsenseQA, TruthfulQA, MedQA, MMLU Professional Law, and MMLU Conceptual Physics, using GPT-3.5 and GPT-4.Source: §1 · p. 2

2 Related Work

The related work situates uncertainty quantification around assumptions about test data and surveys approaches for LLM confidence. It distinguishes methods by their access to model internals and by how they elicit or aggregate responses.

2.1 Uncertainty Quantification in Machine Learning

  • Uncertainty quantification methods differ in their assumptions about test data, including whether it shares the training distribution or is shifted.Under a same-distribution assumption, Bayesian methods and conformal prediction are examples; prior networks address distributional uncertainty.Source: §2.1 · pp. 2–3

2.2 Uncertainty Quantification in LLMs

  • LLM uncertainty is harder to frame because generation ranges over text sequences, though multiple-choice evaluation permits standard classification methods.Prior work reports that larger models can be better calibrated on multiple-choice datasets yet remain sensitive to question reformulations and RLHF strategies.Source: §2.2 · p. 3
  • LLM confidence methods divide into white-box approaches that inspect internal information and black-box approaches that use model samples.White-box methods use logits or activations; black-box methods sample and aggregate responses or prompt the model to verbalize confidence.Source: §2.2 · p. 3

3 Stable Explanations

Stable explanations confidence estimates an answer's plausibility from explanations sampled for a question and reweighted for logical consistency. The resulting explanation-conditioned answer distributions are treated as test-time classifiers and combined into a posterior predictive distribution.

3 Stable Explanations

  • The method approximates the explanation distribution by sampling explanations for a question, reweighting them for logical consistency, and aggregating the answers they support.This treats explanation-conditioned models as conditional classifiers and estimates a posterior predictive answer distribution.Source: §3 · p. 4

3.1 Answer Likelihood Conditioned on Explanations

  • Rather than rely only on default softmax token probabilities, the method uses the answer distribution conditioned on an explanation to characterize its effect on possible answers.Semantically identical explanations are grouped by the answers they support, and the conditional answer distribution is used in place of pairwise explanation similarity.Source: §3.1 · p. 4

3.2 Determining Likely Explanations

  • Sampled explanations are reweighted by entailment probability because generated explanations need not actually explain the question's data.The authors argue logical consistency can prevent trusting explanations that overfit to the test datum; alternative plausibility calculations are explored in Appendix B.4.Source: §3.2 · p. 5

4 Experiments

The experiments compare stable explanation confidence with baselines on five QA datasets and test whether explanation entailment distinguishes explanations supporting correct versus incorrect answers. They report selective-uncertainty gains and examine the method’s components.

4.1 Setup

  • Experiments compare GPT-3.5-turbo and GPT-4-turbo across five QA benchmarks, evaluating calibration and selective uncertainty against token-probability, linguistic, top-k, and self-consistency baselines.Multiple-response methods use N=K=5 in the main results; the datasets vary in question length and complexity.Source: §4.1 · pp. 5–6, §4.3 · p. 6, Table 1 · p. 6

4.2 Likely Explanations Not Always Correct

  • Correct explanations have slightly higher mean log-likelihood, but their distribution overlaps incorrect explanations; entailment probability distinguishes a tail of incorrect explanations.This suggests entailment can discount well-written but logically unstable explanations when likelihood does not distinguish them.Source: §4.2 · p. 6, Figure 1 · p. 7

4.3 Stable Confidence Improves Selective Uncertainty

  • AURC averaged 0.784 vs. 0.761 and AUROC 0.802 vs. 0.789, with stable explanation confidence outperforming baselines on GPT-3.5-turbo selective uncertainty.For GPT-4-turbo, AUROC improves by about 6% on average, while ECE is roughly the same as baselines.Source: §4.3 · pp. 6–7

4.4 Ablation Study

  • Combining entailment reweighting with explanation-conditioned answer distributions generally improves selective uncertainty over either component alone, especially on MedQA and MMLU Law.Calibration and accuracy typically do not improve for the full method, though averaging the components may make results more consistent across datasets.Source: §4.4 · p. 7

5 Discussion

Discussion reports stronger selective-uncertainty performance for the proposed method, while qualifying that result through ablations, calibration concerns, and scope limitations.

5 Discussion

  • Across five datasets, the method improves selective uncertainty on average; GPT-4-Turbo beats baselines throughout, while GPT-3.5-Turbo is not SOTA on CSQA or MMLU Physics.The discussion also notes especially pronounced effects on complex reasoning tasks such as MMLU Law.Source: §5 · p. 7, Figure 2 · p. 8
  • The full method generally outperforms its individual components on AURC and AUROC, with calibration similar to or slightly worse than distribution only.The ablation isolates entailment reweighting and explanation-conditioned answer distributions on GPT-3.5-Turbo.Source: Table 2 · p. 9
  • The authors connect explanation-based inference to transductive learning, viewing entailment probability as a possible approximation of training loss that may prevent test-datum overfitting.They describe this as a perspective rather than a demonstrated result.Source: §5 · p. 9
  • ECE relies on training, test, and calibration data sharing a distribution, an assumption that cannot be verified or falsified at test time.The discussion also notes that ECE can be biased by accuracy and binning strategy [12].Source: §5 · p. 9
  • The analysis covers multiple-choice datasets only; the authors also report exceptions on simpler questions and identify explanation faithfulness and efficient high-quality sampling as open issues.They hypothesize that the method’s advantage is less evident when questions resemble classical inductive classification tasks.Source: §5 · p. 9

A Evaluation of Uncertainty Metrics

This section defines the confidence and correctness quantities used in evaluation, then describes how ECE, AUROC, and AURC assess calibration or selective uncertainty.

A Evaluation of Uncertainty Metrics

  • Evaluation represents each example with a predicted label and confidence score, and marks correctness by whether the prediction matches its label.The confidence function assigns a score in [0, 1].Source: A Evaluation of Uncertainty Metrics · p. 13
  • ECE groups examples into confidence-based bins, compares each bin’s empirical accuracy with its average confidence, and computes a weighted average error.Lower ECE indicates better calibration relative to the data distribution, but the metric depends on bin count and can be misleading when accuracy is high [12] [46].Source: A Evaluation of Uncertainty Metrics · p. 13
  • AURC evaluates accuracy among the most confident predictions at successive coverage levels, testing whether smaller selected subsets achieve higher accuracy.Predictions are ordered by confidence before forming the subsets used for the risk-coverage curve.Source: A Evaluation of Uncertainty Metrics · p. 13
  • AUROC measures how well confidence scores distinguish correct from incorrect predictions across thresholds, with larger area indicating better discrimination.A perfect classifier maintains a true positive rate of 1 across false positive rates; random scores produce a diagonal ROC curve.Source: A Evaluation of Uncertainty Metrics · p. 14

B Experimental Details

The appendix documents the confidence-elicitation prompts and how their outputs are used to calculate confidence scores.

B.1 Prompts

  • Stable explanations use one prompt to generate explanations and a conditional-answer prompt whose logits provide the posterior answer distribution.The default explanations discard the answer generated after the explanation; the second prompt conditions on each explanation.Source: §B.1 · p. 14, Figure 3 · p. 15
  • Token-probability confidence applies a softmax to next-token logits in a simple question-and-answer format.Source: §B.1 · p. 14, Figure 4 · p. 14
  • The appendix also supplies prompts for linguistic confidence, chain-of-thought consistency, top-k confidence, and conditional explanation generation.Linguistic responses are parsed for answer and confidence; chain-of-thought responses can instead supply explanations for CoT-stability.Source: §B.1 · p. 14, Figure 5 · p. 15, Figure 6 · p. 16

B.2 Dataset Details

The appendix reports evaluation sample sizes and notes that the most-confident stability predictions improve accuracy over default predictions.

B.2 Dataset Details

  • GPT-3.5 confidence scores use 250 questions per dataset, while GPT-4-Turbo uses 100 because of computational cost.CSQA, TruthQA, and MMLU use validation splits; MedQA uses test splits.Source: §B.2 · p. 17
  • Accuracy improves for the most-confident stability predictions over default predictions for both GPT-3.5 and GPT-4.The accuracy boost over the baseline is less pronounced for GPT-4 than for GPT-3.5.Source: Table 3 · p. 17, §B.2 · p. 17

B.3 Evaluation Details

The appendix specifies evaluation choices and tests alternative explanation-plausibility measures and prompting order.

B.3 Evaluation Details

  • GPT-3.5 and GPT-4 evaluations use the same default predictions at temperature zero to make confidence-method comparisons comparable.Source: §B.3 · p. 17
  • Evaluation adds Gaussian noise with σ = 1e −6 for ties in AURC and AUROC, averages over 10 repetitions, and uses 10 bins for ECE.Source: §B.3 · p. 17
  • Generating and evaluating confidence scores on one dataset takes about an hour with GPT-3.5-Turbo and two hours with GPT-4-Turbo.Source: §B.3 · p. 17

B.4 Alternate Explanation Plausibility Measures

  • The two alternate explanation-plausibility questions fail to distinguish explanations, including those conditioned on true versus false answers.The measures ask whether an explanation is most likely or completely describes the question.Source: §B.4 · p. 17, Figure 10 · p. 18

B.5 Sensistivity to Explanation Prompting

  • On MedQA, stability outperforms other confidence metrics, and generating explanations before answers improves selective-uncertainty and calibration performance.Questions in the top 50% by confidence were all correct.Source: §B.5 · p. 17, Figure 9 · p. 18

B.6 Varying Sample Size

The appendix finds that sample-size effects vary across tasks and that calibration error declines as more explanations are sampled.

B.6 Varying Sample Size

  • Selective-uncertainty performance saturates quickly on CommonsenseQA, while MedQA and MMLU Law improve steadily through M = 5 explanation samples.Calibration error gradually decreases with sample size across all examined datasets.Source: §B.6 · p. 18

B.7 Comparison to TTA

The comparison tests TTA on shared CSQA questions and contrasts explanation timing and explanation counts. The implementation reports slightly weaker-than-published selective-uncertainty results, while its ECE is lower in many cases.

B.7 Comparison to TTA

  • TTA conditions explanations on different answers and combines them in one prompt, unlike stable explanations confidence's ensemble of test-time classifiers.The authors implemented TTA using its written procedure and the same prompts because code and dataset splits were unavailable.Source: §B.7 · p. 18
  • Explanations generated before the answer outperform those generated afterward across all reported performance metrics for GPT-3.5.Table 4 compares the two explanation timings for stability confidence.Source: Table 4 · p. 19
  • Using M = 1, 3, or 5 explanations per question yields improving but saturating performance across all five benchmark datasets.Figure 11 reports this comparison on GPT-3.5-Turbo.Source: Figure 11 · p. 19
  • On shared CSQA questions, the TTA implementation's selective-uncertainty results are slightly below the authors' reported AURC and AUROC, while ECE is lower in many cases.The authors suggest differences in the specific test questions may explain the selective-uncertainty discrepancy.Source: §B.7 · pp. 18–19

C.1 Confidence through the Viewpoint of Transductive Inference

The section frames explanations as a way to adjust an LLM's decision boundary at inference time and formalizes explanation choice through a faithfulness-aware objective. It relates the method to Solomonoff induction and interprets answer-explanation cycles through stability.

C.1 Confidence through the Viewpoint of Transductive Inference

  • Table 5 evaluates the TTA confidence metric on GPT-3.5 and marks results that outperform stable explanations confidence with asterisks.The supplied passage does not state the marked entries or their values.Source: Table 5 · p. 20
  • Explanations can change an LLM classifier's decision boundary at inference time by directing attention to concepts around the test question.The discussion connects this view to in-context learning's link between context adjustment and classical fine-tuning behavior [9].Source: §C.1 · pp. 19–20
  • The transductive objective can be viewed as finding an explanation that supports one answer while using a faithfulness measure instead of optimizing over all previous questions.The passage interprets ℓ as a log likelihood, making L the negative log posterior over hypotheses, and notes that the full previous-question loss is expensive at inference time.Source: §C.1 · p. 20

C.2 Confidence throught the Viewpoint of Solomonoff Induction

  • The method considers a distribution of likely test-time classifiers, rather than selecting a single classifier as typical transductive inference does.The section presents this as a special case of Solomonoff induction.Source: §C.2 · p. 21
  • The framework treats explanations as LLM-interpretable programs, using the LLM's prior likelihood of valid sequences and sampling to approximate expectations over that restricted hypothesis class.This tractable restriction is contrasted with Solomonoff induction's uncomputability and prior over program length.Source: §C.2 · p. 21

C.3 Confidence through the Viewpoint of Stability

  • The stability perspective models alternating explanation generation conditioned on an answer and answer generation conditioned on an explanation; a singleton decision distribution forms a stable cycle.The passage describes this as an interpretation through fixed points of an LLM system.Source: §C.3 · p. 21
Loading 2406.03441v1…