Source-linked AI summary
Teaching Models to Express Their Uncertainty in Words
Stephanie Lin, Jacob Hilton, Owain Evans
TL;DR
Language models often produce false statements, creating a need for uncertainty that helps users assess trust. The paper finetunes GPT-3 to express epistemic uncertainty in natural language and evaluates it with CalibratedMath against logit-based approaches. Verbalized probability achieves reasonable calibration in- and out-of-distribution, though the tested shifts leave question content and format largely unchanged.
Problem
Language models can hallucinate, motivating calibrated uncertainty so users can judge how much to trust statements, including claims without known ground truth.
Method
The paper finetunes GPT-3 to output answers and natural-language confidence levels, evaluating verbalized probability on CalibratedMath across arithmetic tasks and distribution shifts.
Results
Verbalized probability generalizes reasonably well across the evaluated distribution shifts, outperforming logit setups on Multi-answer and matching the constant baseline on Multiply-divide.
Takeaways & Limitations
GPT-3 can express calibrated uncertainty in words rather than simply imitating human uncertainty or reproducing uncertainty information from logits.
Takeaways & Limitations
The tested shifts changed label distributions substantially but changed question content and format little, leaving broader generalization untested.
Abstract
from arXiv · showhide
We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language -- without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g. "90% confidence" or "high confidence"). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own answers in natural language. For testing calibration, we introduce the CalibratedMath suite of tasks. We compare the calibration of uncertainty expressed in words ("verbalized probability") to uncertainty extracted from model logits. Both kinds of uncertainty are capable of generalizing calibration under distribution shift. We also provide evidence that GPT-3's ability to generalize calibration depends on pre-trained latent representations that correlate with epistemic uncertainty over its answers.
1 Introduction
Language models can answer difficult questions yet often produce false statements, motivating calibrated uncertainty that helps users judge trust. This paper studies verbalized probability: expressing epistemic uncertainty in natural language rather than relying on token logits.
- Motivation: False statements in long-form model outputs reduce their value because users cannot tell when claims are truthful.Calibrated uncertainty could help users know how much to trust a statement.
- Motivation: Logits measure uncertainty over token sequences, whereas verbalized probability targets epistemic uncertainty about claims themselves.The paper notes that paraphrasable claims can distribute probability across many low-probability wordings.
- Approach: Verbalized probability asks models to express uncertainty in natural language while remaining calibrated to their own answers rather than imitating human uncertainty.The authors argue that GPT-3’s differing strengths across computer security and arithmetic make simple imitation insufficient.
- Implications: The paper frames verbalized probability as related to honesty because calibrated performance after finetuning makes dishonest misrepresentation of confidence less likely.Calibration alone could be achieved by imitating a calibrated individual, so the authors treat diverse-question performance as relevant evidence.
- Evaluation: CalibratedMath evaluates whether models can produce both numerical answers and confidence across arithmetic tasks that vary in content and difficulty.These differences support tests of calibration under question-type distribution shifts and make human imitation difficult.
- Findings: GPT-3 can learn calibrated uncertainty in words, with reasonable in- and out-of-distribution calibration, while superficial heuristics do not explain the performance.The paper compares verbalized probability with finetuned logits and reports that the calibration is not explained by learning to output logit uncertainty.
2 Setup
The setup defines calibration for models’ own answers, introduces three probability sources, and uses CalibratedMath to test generalization across task and difficulty shifts.
- Calibration: A calibrated model assigning 90% probability to an answer should be correct 90% of the time.
- Three Kinds of Probability: The paper compares answer logits, indirect logits, and verbalized probabilities as sources of confidence.Indirect logits and verbalized probabilities can handle questions with multiple correct answers.
- CalibratedMath: CalibratedMath contains 21 programmatically generated arithmetic tasks spanning varied concepts, answer formats, and difficulty levels.Some tasks have multiple correct answers, and multiplication is harder than addition for GPT-3.
- Distribution Shift: Calibration is tested under shifts in task difficulty and mathematical content between Add-subtract training and Multi-answer evaluation.Multiply-divide provides a second evaluation set with unique answers, harder questions, and distinct multiplication and division concepts.
- Metrics: The experiments hold zero-shot answers fixed and evaluate calibration rather than improvements in answer accuracy.Calibration is measured using mean squared error and mean absolute deviation calibration error.
- Metrics: Mean absolute deviation summarizes differences between accuracy and average confidence across equal-count probability bins.It provides a numeric summary of calibration curves but is not a proper scoring rule.
3 Experiments
The experiments finetune GPT-3 to verbalize calibrated confidence, compare it with logit-based and constant baselines, and test calibration under distribution shift and few-shot prompting. Verbalized probability generalizes reasonably across evaluation sets, while results also indicate pre-trained latent representations contribute to calibration.
- 3.1 Supervised finetuning: GPT-3 is finetuned to output verbalized probabilities using supervised labels derived from its empirical accuracy on each question sub-task.The training set samples 100 questions per sub-task, producing approximately 10k question-answer inputs.
- 3.1 Supervised finetuning: The evaluation compares verbalized probability and indirect logit finetuning with zero-shot answer logits and a constant baseline.Calibration metrics are reported in Table 1 and Figure 4, with lower percentage scores indicating better calibration.
- 3.2 Results: Verbalized probability generalizes reasonably well from Add-subtract training to both Multiply-divide and Multi-answer evaluation sets under distribution shift.It outperforms the two logit setups on Multi-answer and matches the constant baseline on Multiply-divide.
- 3.2 Results: Verbalized probability is better calibrated in-distribution and becomes underconfident on Multi-answer, whose answers are more often correct than those in Add-subtract.The reported explanation is that the model’s confidence does not fully adjust to the shifted answer accuracy.
- 3.2 Results: Indirect logit generalizes well to Multiply-divide but performs worse than verbalized probability on Multi-answer.The authors suggest the setup may make overfitting harder to avoid for indirect logit.
- 3.2 Results: The results suggest finetuned GPT-3 may output a verbal probability only when that precise probability token appeared during training.This proposed token-level constraint may explain the nonsmooth calibration curves.
- 3.3 Stochastic Few-shot: At k = 50 stochastic few-shot examples, verbalized-probability calibration is close to that of finetuned models trained on more than 2.5k samples.Improvement begins at k = 25 and above on both evaluation sets.
- 3.4 Explaining the performance of verbalized probability: A linear probe on pre-trained GPT-3 embeddings generalizes to Multiply-divide, supporting the presence of relevant features learned during pre-training.Its performance is worse than the finetuned verbalized model, but the generalization pattern remains.
4 Discussion
GPT-3 shows some ability to generalize verbalized calibration under distribution shift, but the tested shifts were limited and broader validation remains open.
- GPT-3 shows some ability to generalize verbalized calibration under distribution shift.
- The training and evaluation sets differed substantially in label distribution, but not much in question content or format.
- Future work should test other subject areas, formats, models, and reinforcement-learning approaches.
5 Related work
Related work covers calibration in new domains, pre-trained language models, natural-language generation, and prior verbalized uncertainty, alongside the CalibratedMath task split.
- Calibration in new domains: Calibration methods often require target-domain or additional source-domain samples when generalizing to new domains.
- Pre-trained language models: Prior studies find that GPT-3’s answer-logit calibration generalizes fairly poorly in zero-shot and few-shot settings.
- Natural language generation: Natural-language-generation work extracts uncertainty from logits over answer-sequence tokens and uses fine-tuning or post-hoc calibration.
- Verbalized uncertainty: Earlier work demonstrated verbalized uncertainty for simple trivia questions in an in-domain, few-shot setting.
- CalibratedMath: CalibratedMath trains on Add/Sub tasks and evaluates on Mult/Div or Multi-answer groups across difficulty levels.
- CalibratedMath: GPT-3’s zero-shot arithmetic accuracy is distributed across task types and difficulty levels, with many training-set accuracies near zero.
B Experimental setup
The experiments train verbalized-probability models with randomized few-shot prompts and sub-task accuracy targets, using early stopping to limit overfitting.
- Verbalized uncertainty can use ordered confidence words or random names, with random names performing slightly better.
- The few-shot prompt contains five examples, whose answers and target probabilities come from the estimation step and are randomized before each query.
- The supervised fine-tuning dataset contains approximately 10k examples sampled across sub-tasks, and models train for one epoch.
- Sub-task accuracy targets can conflict with per-sample correctness, because a sub-task average is only a proxy for an individual question’s uncertainty.
- Training stops after around n = 2700 examples when training-set per-sample MSE flattens or increases despite continued target-loss reduction.
C.1 Verbalized calibration curves by number of training samples
Calibration improves with more training examples on Multiply-divide and Multi-answer evaluation sets, but gains stop beyond approximately 2700 examples.
- Beyond approximately n = 2700 training examples, additional training does not improve generalization.
- The curves evaluate Multiply-divide and Multi-answer calibration using Expected Value decoding rather than greedy decoding.
C.2 Comparing results using greedy and EV uncertainties
The section compares greedy and expected-value (EV) uncertainty estimates, highlighting the broader probability range enabled by numeric verbalization and token averaging.
- Numeric verbalization covers a wide range of probability values even with greedy decoding.Word-based verbalization is limited to five categories: 10%, 30%, 50%, 70%, and 90%.
- Expected value over output tokens produces intermediate confidence scores from word-based outputs.For example, combining 50% “Medium” and 70% “High” with equal weights yields 60% confidence.
- Table 4 reports performance for finetuned models using greedy and EV uncertainties, while Figure 11 presents their calibration curves.
C.3 Changing the training set from Add-subtract to Multiply-divide
Training on Multiply-divide and evaluating on Add-subtract and Multi-answer reveals weaker calibration on Multi-answer, associated with a larger label-distribution shift.
- Training on Multiply-divide is evaluated on both Add-subtract and Multi-answer.
- Calibration on Multi-answer is worse than when models are trained on Add-subtract.
- A bigger shift in label distribution from training to evaluation is offered as one reason for the weaker Multi-answer calibration.
C.4 Correlations between probability types
The section compares correlations among verbalized and logit-based uncertainty types across tasks and difficulty levels, finding stronger agreement between the two verbalized forms.
- The Multiply-divide evaluation set is used to calculate each setup’s MSE by task and difficulty level.The results are plotted with a colorbar showing GPT-3’s arithmetic-question accuracy.
- Correlation is high between numeric and word-based verbalized uncertainty, but moderate between verbalized uncertainty and the other two types.Numeric verbalization uses values such as 45%, while word-based verbalization uses labels such as “Confidence: Low.”
- The plotted colorbar encodes GPT-3’s accuracy on the arithmetic questions.