PaperlayerView exact revision on arXiv

Source-linked AI summary

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning

arXiv:2305.14975v2cs.CL

TL;DR

Poorly calibrated confidence can impede reliable prediction and human deferral, and RLHF-LMs may have degraded conditional-probability calibration. This paper evaluates ways to elicit confidence from RLHF-LMs and finds that verbalized probabilities are often better-calibrated than conditional probabilities, though results are mixed for Llama-2-70B-Chat. The study is limited mainly to factual-recall and short-form question-answering settings.

  • Problem

    Widely used RLHF-LMs may produce poorly calibrated conditional probabilities, while existing calibration studies have focused largely on unsupervised or maximum-likelihood models.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates numerical and linguistic verbalized-confidence prompts, including multi-hypothesis prompting and temperature scaling, across several RLHF-LMs.

    Source: §1 · pp. 1–2, §2 · p. 4

  • Results

    Verbalized probabilities are often better-calibrated than conditional probabilities across several closed RLHF-LMs, with mixed results for Llama-2-70B-Chat.

    Source: §1 · p. 2, §4 · p. 5

  • Takeaways & Limitations

    Verbalizing confidence offers a practical way to elicit calibrated scores from RLHF-LMs when log probabilities are unavailable.

    Source: §1 · p. 2, §4 · p. 5

    The experiments focus on factual-recall problems and short-form question answering, leaving reasoning-heavy and longer-form settings for future work.

    Source: §4 · p. 5

Abstract

from arXiv · show

A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

1 Introduction

Calibration matters because confidence estimates can identify likely errors for human review, yet RLHF may worsen language models’ conditional-probability calibration. This paper evaluates verbalized confidence as an alternative and finds it can improve calibration for RLHF-LMs.

Motivation

  • Well-calibrated confidence estimates can identify predictions most likely to be incorrect, enabling human review or override.Poor calibration may contribute to confidently asserted incorrect facts or reasoning.Source: §1 · p. 1
  • Most widely used language models are fine-tuned with reinforcement learning from human feedback, whereas much prior calibration analysis studied maximum-likelihood or unsupervised models.Existing findings suggest RLHF may trade calibrated predictions for closer adherence to dialogue instructions.Source: §1 · pp. 1–2

Approach

  • The paper evaluates confidence-extraction methods for RLHF-LMs, emphasizing verbalized probabilities because per-token log probabilities are often unavailable.Verbalized confidence is expressed in token space as numerical probabilities or linguistic uncertainty expressions.Source: §1 · pp. 1–2

Findings

  • Figure 2 describes RLHF as generally worsening Llama-70B log-probability calibration, with reversal of some degradation strongest on TruthfulQA.ECE is lower-is-better, while AUC is higher-is-better.Source: Figure 2 · p. 2
  • Verbalized confidence is often better-calibrated than conditional probabilities for popular RLHF-LMs, without fine-tuning for verbalization.The study examines models including ChatGPT, GPT-4, Claude, and Llama-2-70B-Chat.Source: §1 · p. 2
  • Prompting models to generate and evaluate multiple answer choices before assigning confidence further improves verbalized probability calibration.This strategy is inspired by psychological findings that considering alternatives can mitigate overconfidence; temperature scaling is also combined with the approach.Source: Figure 1 · p. 1, §1 · p. 2

2 Evaluating Calibration in RLHF-LMs

The paper evaluates confidence extraction methods for RLHF-LMs, comparing conditional-probability approaches with verbalized numerical or linguistic confidences across multiple calibration measures and QA datasets.

  • The evaluation covers ChatGPT, GPT-4, Claude 1, Claude 2, and Llama-2-70B-Chat.Source: §2 · p. 2
  • Calibration is measured with raw ECE, temperature-scaled ECE, temperature-scaled Brier Score, and selective-classification AUC.ECE compares confidence with average accuracy within confidence bins; Brier Score measures squared error against correctness labels.Source: §2 · pp. 2–3
  • On ChatGPT and GPT-4, directly verbalized confidences are better calibrated than model probabilities, including after temperature scaling for GPT-4.ChatGPT’s table contrasts Label prob. and Is True prob. with Verb. 1S, Verb. 2S, and Ling. 1S; GPT-4’s caption reports the same comparison after scaling.Source: Table 1 · p. 3, Table 2 · p. 3
  • The datasets assess factual question answering using TriviaQA, SciQ, and TruthfulQA, with correctness computation designed to reduce exact-match false negatives.TruthfulQA specifically tests whether models mimic human falsehoods.Source: §2 · p. 3
  • Conditional-probability baselines estimate label probability or the probability that a sampled answer is true.Because many RLHF-LMs are closed-source, probabilities are estimated from samples, using n = 10 for the main protocol.Source: §2 · p. 4
  • Verbalization methods elicit numerical or linguistic confidence, using one-stage or two-stage prompts and, in one variant, chain-of-thought context.Linguistic expressions are mapped either to human-survey probabilities or values optimized on held-out calibration questions.Source: §2 · p. 4

3 Results

Across the evaluated RLHF-LMs, verbalized confidence is often better calibrated than conditional probabilities, although the consistency and strength of this pattern vary by prompting strategy and model.

  • Llama-2-70B-Chat shows improvement from verbalized calibration on some metrics, but less consistently than GPT and Claude models.Source: Table 5 · p. 5, §3 · p. 5
  • Large RLHF-LMs can often verbalize better-calibrated numerical or linguistic confidences than their conditional probabilities.Source: §3 · p. 5
  • Generating and evaluating multiple hypotheses improves calibration among methods that verbalize probabilities directly.Source: §3 · p. 5
  • Numerical probabilities calibrate as well as or better than linguistic uncertainty expressions, while chain-of-thought prompting does not improve verbalized calibration.Source: §3 · p. 5
  • Claude 1 is less able to verbalize well-calibrated confidence, whereas Claude 2 is generally somewhat stronger than GPT-3.5-turbo at verbalization.Source: §3 · p. 5

4 Discussion

The paper studies calibration in widely used RLHF language models and finds that prompting models to verbalize confidence can outperform conditional probabilities, while leaving important scope and model-understanding questions open.

  • The study examines calibration in widely used RLHF-LMs and methods for extracting their confidence scores.Source: §4 · p. 5
  • Verbalized probabilities are better-calibrated than conditional probabilities across several closed models, but results are mixed for Llama-2-70B-Chat.Source: §4 · p. 5
  • Future work should explain differences in confidence verbalization across GPT, Claude, and Llama-2 models and reduce sensitivity to prompting strategy.Source: §4 · p. 5
  • The experiments focus on factual-recall problems, so whether the observations hold in reasoning-heavy settings remains open.Source: §4 · p. 5
  • The study is limited to short-form question answering, leaving longer-form generation for future analysis.Source: §4 · p. 5

A Additional Results

Additional results show that models express lower confidence on TruthfulQA than on standard factual-recall datasets, while chain-of-thought prompting does not noticeably improve calibration.

  • gpt-3.5 is systematically less confident on TruthfulQA than on TriviaQA and SciQ, with the contrast even more stark for gpt-4.Source: A Additional Results · p. 8
  • CoT does not noticeably improve calibration across the tested prompt variants or datasets.Source: A Additional Results · p. 8

B Fitting Procedure for Temperature and Probabilities for Linguistic Expressions

The appendix describes cross-validation procedures for fitting temperature-scaled metrics and linguistic-expression probabilities; CoT calibration results are not consistently improved.

C Prompt Templates

The appendix provides prompt templates for the evaluated methods and distinguishes sampling-based confidence estimation from verbalized confidence methods that require one sample.

  • Prompt templates substitute each evaluation question for the ${THE_QUESTION} variable.Source: C Prompt Templates · p. 8
  • Methods above the table’s double line use multiple samples to estimate confidence, whereas methods below it use verbalized confidences from a single sample.Source: Table 6 · p. 9
  • Answer correctness is evaluated by asking a language model whether the ground-truth and generated answers are semantically equivalent.Source: C Prompt Templates · p. 10

Related papers

Loading 2305.14975v2…