Source-linked AI summary
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning
TL;DR
Poorly calibrated confidence can impede reliable prediction and human deferral, and RLHF-LMs may have degraded conditional-probability calibration. This paper evaluates ways to elicit confidence from RLHF-LMs and finds that verbalized probabilities are often better-calibrated than conditional probabilities, though results are mixed for Llama-2-70B-Chat. The study is limited mainly to factual-recall and short-form question-answering settings.
Problem
Widely used RLHF-LMs may produce poorly calibrated conditional probabilities, while existing calibration studies have focused largely on unsupervised or maximum-likelihood models.
Method
The paper evaluates numerical and linguistic verbalized-confidence prompts, including multi-hypothesis prompting and temperature scaling, across several RLHF-LMs.
Results
Verbalized probabilities are often better-calibrated than conditional probabilities across several closed RLHF-LMs, with mixed results for Llama-2-70B-Chat.
Takeaways & Limitations
Verbalizing confidence offers a practical way to elicit calibrated scores from RLHF-LMs when log probabilities are unavailable.
Takeaways & Limitations
The experiments focus on factual-recall problems and short-form question answering, leaving reasoning-heavy and longer-form settings for future work.
Abstract
from arXiv · showhide
A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.
1 Introduction
Calibration matters because confidence estimates can identify likely errors for human review, yet RLHF may worsen language models’ conditional-probability calibration. This paper evaluates verbalized confidence as an alternative and finds it can improve calibration for RLHF-LMs.
- Motivation: Well-calibrated confidence estimates can identify predictions most likely to be incorrect, enabling human review or override.Poor calibration may contribute to confidently asserted incorrect facts or reasoning.
- Motivation: Most widely used language models are fine-tuned with reinforcement learning from human feedback, whereas much prior calibration analysis studied maximum-likelihood or unsupervised models.Existing findings suggest RLHF may trade calibrated predictions for closer adherence to dialogue instructions.
- Approach: The paper evaluates confidence-extraction methods for RLHF-LMs, emphasizing verbalized probabilities because per-token log probabilities are often unavailable.Verbalized confidence is expressed in token space as numerical probabilities or linguistic uncertainty expressions.
- Findings: Figure 2 describes RLHF as generally worsening Llama-70B log-probability calibration, with reversal of some degradation strongest on TruthfulQA.ECE is lower-is-better, while AUC is higher-is-better.
- Findings: Verbalized confidence is often better-calibrated than conditional probabilities for popular RLHF-LMs, without fine-tuning for verbalization.The study examines models including ChatGPT, GPT-4, Claude, and Llama-2-70B-Chat.
- Findings: Prompting models to generate and evaluate multiple answer choices before assigning confidence further improves verbalized probability calibration.This strategy is inspired by psychological findings that considering alternatives can mitigate overconfidence; temperature scaling is also combined with the approach.
2 Evaluating Calibration in RLHF-LMs
The paper evaluates confidence extraction methods for RLHF-LMs, comparing conditional-probability approaches with verbalized numerical or linguistic confidences across multiple calibration measures and QA datasets.
- The evaluation covers ChatGPT, GPT-4, Claude 1, Claude 2, and Llama-2-70B-Chat.
- Calibration is measured with raw ECE, temperature-scaled ECE, temperature-scaled Brier Score, and selective-classification AUC.ECE compares confidence with average accuracy within confidence bins; Brier Score measures squared error against correctness labels.
- On ChatGPT and GPT-4, directly verbalized confidences are better calibrated than model probabilities, including after temperature scaling for GPT-4.ChatGPT’s table contrasts Label prob. and Is True prob. with Verb. 1S, Verb. 2S, and Ling. 1S; GPT-4’s caption reports the same comparison after scaling.
- The datasets assess factual question answering using TriviaQA, SciQ, and TruthfulQA, with correctness computation designed to reduce exact-match false negatives.TruthfulQA specifically tests whether models mimic human falsehoods.
- Conditional-probability baselines estimate label probability or the probability that a sampled answer is true.Because many RLHF-LMs are closed-source, probabilities are estimated from samples, using n = 10 for the main protocol.
- Verbalization methods elicit numerical or linguistic confidence, using one-stage or two-stage prompts and, in one variant, chain-of-thought context.Linguistic expressions are mapped either to human-survey probabilities or values optimized on held-out calibration questions.
3 Results
Across the evaluated RLHF-LMs, verbalized confidence is often better calibrated than conditional probabilities, although the consistency and strength of this pattern vary by prompting strategy and model.
- Llama-2-70B-Chat shows improvement from verbalized calibration on some metrics, but less consistently than GPT and Claude models.
- Large RLHF-LMs can often verbalize better-calibrated numerical or linguistic confidences than their conditional probabilities.
- Generating and evaluating multiple hypotheses improves calibration among methods that verbalize probabilities directly.
- Numerical probabilities calibrate as well as or better than linguistic uncertainty expressions, while chain-of-thought prompting does not improve verbalized calibration.
- Claude 1 is less able to verbalize well-calibrated confidence, whereas Claude 2 is generally somewhat stronger than GPT-3.5-turbo at verbalization.
4 Discussion
The paper studies calibration in widely used RLHF language models and finds that prompting models to verbalize confidence can outperform conditional probabilities, while leaving important scope and model-understanding questions open.
- The study examines calibration in widely used RLHF-LMs and methods for extracting their confidence scores.
- Verbalized probabilities are better-calibrated than conditional probabilities across several closed models, but results are mixed for Llama-2-70B-Chat.
- Future work should explain differences in confidence verbalization across GPT, Claude, and Llama-2 models and reduce sensitivity to prompting strategy.
- The experiments focus on factual-recall problems, so whether the observations hold in reasoning-heavy settings remains open.
- The study is limited to short-form question answering, leaving longer-form generation for future analysis.
A Additional Results
Additional results show that models express lower confidence on TruthfulQA than on standard factual-recall datasets, while chain-of-thought prompting does not noticeably improve calibration.
- gpt-3.5 is systematically less confident on TruthfulQA than on TriviaQA and SciQ, with the contrast even more stark for gpt-4.
- CoT does not noticeably improve calibration across the tested prompt variants or datasets.
B Fitting Procedure for Temperature and Probabilities for Linguistic Expressions
The appendix describes cross-validation procedures for fitting temperature-scaled metrics and linguistic-expression probabilities; CoT calibration results are not consistently improved.
- Expected calibration error is not consistently improved for any CoT prompt variant on gpt-3.5-turbo.
- Temperature-scaled ECE and BS use five-fold data splitting, fitting temperature on one fold and evaluating on the remaining folds.Fitting temperature on 20% of the data yields relatively stable temperatures across folds.
- ECE-t and BS-t are reported as the average temperature-scaled ECE and BS across folds.
- Ling. 1S-opt. fits probabilities for linguistic confidence expressions on four folds and evaluates on the held-out fold.
- For Ling. 1S-opt., temperature fitting uses separate folds for expression probabilities, temperature estimation, and final evaluation.
C Prompt Templates
The appendix provides prompt templates for the evaluated methods and distinguishes sampling-based confidence estimation from verbalized confidence methods that require one sample.
- Prompt templates substitute each evaluation question for the ${THE_QUESTION} variable.
- Methods above the table’s double line use multiple samples to estimate confidence, whereas methods below it use verbalized confidences from a single sample.
- Answer correctness is evaluated by asking a language model whether the ground-truth and generated answers are semantically equivalent.