PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Must Be Taught to Know What They Don't Know

Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, Andrew Gordon Wilson

arXiv:2406.08391v3cs.LGcs.AIcs.CLstat.ML

TL;DR

Reliable estimates of whether LLM answers are correct matter for high-stakes use, but the paper finds that out-of-the-box estimates are unreliable for open-ended generation. It fine-tunes models on graded answers and reports uncertainty estimates with generalization across shifts and models.

  • Problem

    The paper asks whether LLMs can reliably estimate the correctness of their outputs, a concern for using them in high-stakes decisions.

    Source: §1 · pp. 1–2

  • Method

    The authors fine-tune models to predict answer correctness using a small labeled dataset of correct and incorrect answers.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    The authors report that fine-tuning yields calibrated uncertainty estimates with practical generalization, including across distribution shifts and to predictions from other models.

    Source: §8 · p. 10

  • Takeaways & Limitations

    In the user study, confidence scores had their biggest effect on improving the lowest-performing users rather than average accuracy.

    Source: §7 · p. 9

    The user-study result is preliminary in a nascent research area, and finer-grained conclusions require a dedicated study.

    Source: §7 · p. 9

Abstract

from arXiv · show

When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study.

1 Introduction

Reliable uncertainty estimates matter because an answer without confidence may be unusable, while unjustified confidence can make acting on it dangerous. The paper investigates whether fine-tuning on graded answers can provide more reliable, efficient estimates than out-of-the-box methods.

1 Introduction

  • In high-stakes decisions, an answer without confidence may not be actionable, and an erroneously confident answer can be dangerous.Whether LLMs can represent uncertainty over output correctness is central to their use in decision-making.Source: §1 · p. 1
  • Prior claims conflict: some report that prompting can elicit calibrated uncertainty, while others find LLMs remain overconfident.Linguistic variation in free-form generation further complicates uncertainty estimation.Source: §1 · p. 1
  • Fine-tuning for uncertainty improves ECE and AUROC over commonly used baselines on open-ended MMLU.The reported comparison averages across three base models and their chat variants.Source: Figure 1 · p. 2
  • Black-box estimates need no training and can work with closed models, whereas white-box methods train on calibration data and are increasingly accessible with open-source models.The paper frames practitioners’ method choice as challenging amid disagreement and variation in free-form outputs.Source: §1 · p. 2
  • The paper studies uncertainty over answer correctness, labeled correctness examples, distribution-shift generalization, and assistance with human decision-making.It compares zero-shot black-box methods with fine-tuning, including parameterizations and data requirements.Source: §1 · p. 2
  • The authors report that fine-tuning produces faster, more reliable estimates with relatively few added parameters and generalizes beyond its training questions and tasks.The introduction also previews investigations of cross-model uncertainty estimation and human-AI collaboration.Source: §1 · p. 2

2 Related Work

Prior work explores token probabilities, prompting, and learned correctness predictors, but findings on prompted uncertainty are mixed. This paper argues that open-ended generation poses distinct challenges and examines fine-tuning as an alternative.

2 Related Work

  • Language-model likelihoods become less reliable for sentence-length answers because probability mass is spread over many phrasings; semantic clustering can address this but incurs substantial computational overhead [30].On multiple-choice tasks, answers are single tokens, and their probabilities can yield calibrated distributions in some non-chat models [43].Source: §2 · p. 2
  • Prior work reports that careful prompting can produce text-based uncertainty estimates that become more calibrated with model capability [25] [51].Other studies use synthetic training data for confidence models or linear probes of hidden representations to classify correctness [54], [9],.Source: §2 · p. 3
  • The authors report that black-box methods often fail on open-ended generation and that fine-tuning can preserve prompting speed while improving performance [60].They distinguish open-ended generation from multiple-choice evaluations and position their findings alongside work on zero-shot discrimination [60].Source: §2 · p. 3
  • The paper builds on graded-data text-completion approaches [33] [63], adding regularization to preserve base-model predictions and ablations on effective training choices.It reports that frozen features are typically insufficient for generalizing uncertainty estimates, while LoRA fine-tuning on as few as 1000 graded examples can generalize across practical shifts; it also tests estimates with actual users.Source: §2 · p. 3

3 Preliminaries

The paper evaluates estimated answer correctness on multiple-choice and open-ended MMLU, using separate measures for calibration and correct-answer filtering. It also describes temperature scaling as a calibration baseline.

3 Preliminaries

  • The evaluation uses greedy, few-shot answer generation, labels answers correct or incorrect, and estimates P(correct); open-ended MMLU removes answer choices and uses an auxiliary language model for grading.The authors report that language-model grading is a cheap, effective proxy for human grading.Source: §3 · p. 3
  • ECE measures the gap between empirical frequencies and estimated probabilities, while AUROC measures how well confidence separates likely-correct from likely-incorrect answers.Lower ECE is better, and ECE 0 indicates perfect calibration; high AUROC supports selective prediction.Source: §3 · p. 3
  • Temperature scaling calibrates a classifier by scaling logits before softmax, with the temperature learned on held-out data typically drawn from the training distribution.Higher temperatures move probabilities toward uniformity; lower temperatures concentrate them around the most probable output.Source: §3 · p. 4

4 Do We Get Good Uncertainties Out-of-the-Box?

Out-of-the-box likelihood and prompting estimates have important limits for open-ended answers, where phrasing and sequence length complicate confidence. The section reports that prompting performs poorly against fine-tuning and that perplexity lacks the favorable scaling seen for multiple-choice token probabilities.

4 Do We Get Good Uncertainties Out-of-the-Box?

  • The black-box comparison focuses on single samples or forward passes because of computational cost, leaving sampling-based methods for the next section.Some models’ likelihoods may be approximated through sampling.Source: §4 · p. 4
  • Unlike multiple-choice answers, open-ended answers have unrestricted phrasing and length, so sequence probability may reflect wording and token count rather than correctness.Probability mass must cover alternative phrasings, and longer sequences can have lower likelihood even when they express the same concept.Source: §4 · p. 4
  • Perplexity is generally not predictive of correctness in open-ended settings and does not scale with model ability as token-level uncertainty does in multiple choice.The paper compares AUROC for likelihood-based estimates on standard and open-ended MMLU.Source: §4 · p. 4
  • Prompting estimates can use language-model logits to score two strings denoting true and false, or sample uncertainty estimates as text and convert them into numbers [51] [60].Kadavath et al. [25] use a modified yes/no prompt and compute confidence as P(“(a)”) / (P(“(a)”) + P(“(b)”)).Source: §4 · p. 4
  • On open-ended MMLU, prompting methods have poor ECE that does not improve with base-model ability, and even the best model’s AUROC trails the worst fine-tuned model.Figure 2 compares common estimates and prompting methods; the right panel marks the LLaMA-2 7B fine-tuned result and shows 95% bootstrapped confidence intervals on regression fits.Source: Figure 2 · p. 5, §4 · p. 5
  • The authors argue that language modeling on human text may not yield calibrated uncertainty because humans can be poorly calibrated on unfamiliar tasks and pretraining data may lack expert answers.They suggest alignment that penalizes poor calibration could resemble fine-tuning on graded data.Source: §4 · p. 5

5 How Should We Use Labeled Examples?

The paper frames uncertainty estimation as predicting whether an answer is correct from a small labeled set, and compares frozen-feature and fine-tuned approaches. Fine-tuning through model features with a language prompt performs strongly, with around 1,000 examples already improving over simpler baselines.

5 How Should We Use Labeled Examples?

  • Predicting P(correct) is a binary classification task learned from a small labeled set of correct and incorrect answers.The experiments compare three parameterizations that vary in trainable parameters and prompting.Source: §5 · p. 5
  • Probe keeps the language model frozen, while LoRA and LoRA + Prompt can adjust its features for uncertainty estimation.LoRA + Prompt frames correctness as a two-choice response, with “i” and “ii” representing no and yes.Source: §5 · pp. 5–6
  • Around 1,000 labeled examples improve over simpler baselines, with further gains in calibration and selective prediction that diminish after about 5,000 examples.Figure 3 shows 1,000 examples are almost as valuable as 20,000.Source: §5 · p. 6, Figure 3 · p. 7
  • On open-ended MMLU, probing alone performs worse than training through model features with a language prompt.Figure 3 compares ECE and AUROC on multiple-choice and open-ended MMLU; its ECE values use temperature scaling on a small hold-out set.Source: §5 · p. 6, Figure 3 · p. 7

6 When and Why Do These Estimates Generalize?

The estimates generalize across subject matter, question format, and solvability, and results suggest they learn more than question difficulty alone. They can also estimate the correctness of other models’ generations.

6 When and Why Do These Estimates Generalize?

  • Despite close to no social-science examples in training, LoRA + Prompt estimates perform similarly across MMLU subject categories.The fine-tuning set contains higher proportions of STEM and humanities questions than MMLU.Source: §6 · p. 7, Figure 4 · p. 8
  • Across question formats, fine-tuned models often beat the zero-shot baseline, and LoRA + Prompt transfers effectively from multiple choice to open-ended questions.Transfer from multiple choice to open-ended is more challenging; Probe does not generalize effectively in that direction.Source: §6 · p. 7, Figure 4 · p. 8
  • On SelfAware unanswerable questions, calibration-tuned models assign lower confidence than black-box methods while remaining calibrated on answerable questions.Lower confidence on unanswerable questions can allow the model to opt out of responding.Source: §6 · p. 7, Figure 4 · p. 8

6.1 What are uncertainty estimates learning?

  • Training on incorrect substituted answers performs surprisingly well but worse than training on the model’s original answers, so question-error correlations do not fully explain LoRA + Prompt estimates.The substituted-answer setup retains correctness labels but removes the original answer correspondence.Source: §6.1 · p. 8, Figure 5 · p. 9
  • Uncertainty estimates are not model-specific: Mistral 7B often has better AUROC on LLaMA-2 7B generations than LLaMA-2 7B does on its own.The findings also suggest one strong base model can estimate other models’ uncertainties when a small labeled dataset is available or can be generated.Source: §6.1 · p. 8, Figure 5 · p. 9
  • Taken together, the results suggest estimates learn question-answer correctness correspondence as well as signals related to question difficulty.This interpretation is consistent with cross-format, subject-matter, and cross-model generalization.Source: §6.1 · p. 8

7 Does Calibrated Confidence Improve Collaboration with AI Assistants?

In a preliminary study, 181 participants answered multiple-choice questions with a Mistral 7B Instruct assistant and confidence scores. Participants used reliable confidence information when deciding whether to rely on the assistant, though stronger conclusions about collaboration require further study.

7 Does Calibrated Confidence Improve Collaboration with AI Assistants?

  • The preliminary study involved 181 participants completing a multiple-choice exam with Mistral 7B Instruct predictions and confidence estimates.The estimates came from either calibrated or uncalibrated methods.Source: §7 · p. 9
  • Participants used reliable confidence scores to modulate whether they agreed with the model, but generally did not adjust reliance for random confidence scores.Confidence scores had their biggest effect on the lowest-performing users rather than on average accuracy.Source: §7 · p. 9
  • The study is preliminary, and finer-grained work is needed to draw stronger conclusions about uncertainty in human-AI collaboration.The authors note that this remains a nascent area of study.Source: §7 · p. 9

8 Discussion

The discussion presents calibration-tuned uncertainty as useful beyond a model’s own predictions, including for human reliance, while the appendices describe how answer correctness is graded for calibration.

8 Discussion

  • Users relied more on confident answers and less on low-confidence answers when scores were calibrated, but did not modulate reliance with uniformly random scores.The variants had approximately the same average participant accuracy.Source: Figure 6 · p. 10
  • Calibrated uncertainties can generalize to new question types and tasks, transfer across models, and remain robust to distribution shifts in the cases considered.The authors report that uncertainty estimators need not rely on representations specific to a model evaluating its own generations.Source: §8 · p. 10
  • The discussion identifies using separate question-answering and uncertainty-estimation models as a current limitation, with a single model proposed as future work.The authors note that uncertainty-aware pre-training or alignment would need to preserve base language-modeling abilities while correctness labels evolve.Source: §8 · p. 10
  • Uncertainty estimates could inform active-learning sample selection and factuality-oriented generation alignment.The discussion gives sample-efficient fine-tuning and reward functions encouraging confident generations as examples.Source: §8 · p. 10

A.1 Evaluating Correctness

  • For multiple-choice answers, correctness is checked by matching the first greedily generated token, while open-ended answers require assessing semantic equivalence.Substring matching can miss rephrasings or over-credit verbose answers; the paper instead uses an auxiliary language model to grade open-ended responses.Source: §A.1 · pp. 15–16

A.2 Grading

  • For open-ended calibration data, a grading prompt supplies the query, reference answer, and model response to an auxiliary LLM; GPT 3.5 Turbo was chosen for human agreement and cost efficiency.The authors report that humans agreed more with GPT 3.5 Turbo than with substring grading; Figure 7 identifies the text included in the calibration loss.Source: §A.2 · p. 16, §A.1 · p. 16, Figure 7 · p. 16

A.3 Comparison of Grading Techniques

The grading comparison finds GPT-4 closest to human-judged correctness, followed by GPT 3.5 Turbo and then substring matching; the appendices also specify evaluation metrics and training-data categorization.

A.3 Comparison of Grading Techniques

  • GPT-4 estimates human-judged correctness better than GPT 3.5 Turbo, which substantially outperforms substring match, though rankings vary across subsets.The authors chose GPT 3.5 Turbo for expediency and cost; Table 2 reports absolute accuracy differences from the human estimate, with lower values closer.Source: §A.3 · p. 17, Table 2 · p. 17
  • The comparison evaluates human, substring, GPT 3.5 Turbo, and GPT-4 judgments against human scores treated as the correctness standard.It uses LLaMA-2 13b-chat responses to MMLU questions stripped of their multiple-choice options.Source: §A.3 · pp. 16–17

A.4 Metrics

  • ECE summarizes the average gap between confidence and accuracy across equally spaced confidence bins.The passage defines bin confidence and bin accuracy as their respective averages; in these experiments confidence is P(correct).Source: §A.4 · p. 17

A.5 MMLU Supercategory Classifier

  • A linear four-way classifier trained on embedded MMLU samples estimates supercategory proportions in the calibration-training data.It uses 10 labeled samples from each of 57 tasks to classify Humanities, STEM, Social Sciences, or Other.Source: §A.5 · p. 17

B.1 Sampling Methods

The appendix details sampling and verbal-elicitation baselines, the regularizer used in calibration-tuning, and the training data and settings.

B.1 Sampling Methods

  • Sampling-based certainty requires O(n) generations per answer and depends on sampling parameters, which the authors fixed at top_p = 0.95 without searching for an optimum.If sampling converges to greedy decoding, all n samples are identical and certainty is always 1.Source: §B.1 · pp. 17–18
  • The counting baseline estimates certainty by the share of sampled answers judged equivalent to the greedy answer, while likelihood accumulation weights those matches by length-normalized likelihood.The authors report similar scores from the two methods and give likelihood-accumulation results in the main text.Source: §B.1 · p. 18

B.2 Verbal Elicitation

  • The authors adopted two-stage verbal elicitation with one guess because no tested prompting strategy consistently outperformed another; parsing failures can force empty answers or probability 0.5.Complex output strings make partial results difficult when parsing fails, though this issue diminishes with larger models.Source: §B.2 · p. 18

C.1 Regularization Term

  • Calibration-tuning regularizes the tuned distribution toward the base model using Jensen-Shannon divergence on target-sequence logits.The authors report that either direction of KL divergence was insufficient for optimal performance and use JSD as a practical choice.Source: §C.1 · p. 19

C.2 Training Data / C.3 Training Hyperparameters

  • The training set reserves 16 named datasets, and the reported setup uses AdamW, a 10^-4 learning rate, batch size 32, and 10,000 training steps.The setup includes a 1,000-step linear warmup and cosine decay.Source: §C.2 · p. 19, §C.3 · p. 19

D Extended MMLU Results

The appendix reports extended MMLU uncertainty-query and calibration results, and examines confidence in relation to target length and coding-task performance.

D Extended MMLU Results

  • The appendix reports uncertainty-query accuracy and ECE for all MMLU tasks in Figures 8 to 11.Figure 8 presents ECE and AUROC for Query, CT-Probe, CT-LoRA, and CT-Query across MMLU subsets in the multiple-choice setting.Source: D Extended MMLU Results · p. 19, Figure 8 · p. 20

E Confidence as a Function of Target Length

  • On livecodebench_generation_lite, supervised training with LoRA plus Prompt seems to improve selective prediction for all three models, but calibration improves heavily only for Mistral-7B and slightly degrades for the other two.Table 3 reports ECE and AUROC after temperature scaling on a small hold-out set from the original dataset mixture.Source: Table 3 · p. 21
  • Across MMLU subsets, target length does not strongly correlate with calibration-tuned confidence, and the confidence estimates show no consistent relationship with target sequence length.The authors identify this as important because token likelihoods decay with generation length, whereas correct longer answers should not be penalized in confidence.Source: E Confidence as a Function of Target… · pp. 20–21

F Generalization to Coding Tasks

This section tests whether uncertainty methods trained without coding examples transfer to coding tasks and reports that transfer varies by base model. It also describes the user-study setup and cautions against treating its results as definitive evidence of confidence scores’ utility.

F Generalization to Coding Tasks

  • On Mistral-7B, LoRA + Prompt dramatically improves calibration and selective prediction over Zero-Shot Classifier, while on Mistral-7B-Instruct and LLaMa-2-7B selective prediction improves but calibration slightly degrades.The evaluation uses 62 LeetCode easy questions; the methods use training or temperature-scaling data from a mixture without coding tasks.Source: F Generalization to Coding Tasks · p. 21

G.1 Additional Details on Setup

  • The user study recruited 181 US-based first-language English speakers to answer 180 MMLU questions.The authors regenerated answers with Mistral 7B Instruct rather than using the earlier GPT-3.5 responses, whose domain performance had been filtered.Source: §G.1 · pp. 21–22

G.2 Important considerations

  • Interpret user-assistance results cautiously: trial-by-trial feedback, challenging MMLU questions, and numeric confidence may limit generalization to practical reliance.The authors note that feedback could let participants infer model reliability without confidence scores, while users may struggle to process numerical confidence.Source: §G.2 · p. 22

G.3 Extended Results

This section presents extended user-study results on accuracy, reliance sensibility, and participants’ uses of confidence, alongside a discussion of the approach’s stated scope and potential misuse. It also documents the models, datasets, model release, and human-subject procedures and ethics approval.

G.3 Extended Results

  • Figure 15 reports user task accuracy and reliance sensibility across variants; Figures 17 and 18 present accuracy and reliance-sensibility results for GPT-3.5 variants.The supplied passages identify the measures and figures but provide no values or comparisons between variants.Source: §G.3 · p. 22, Figure 15 · p. 29, Figure 17 · p. 30
  • Figures 16 and 17 show per-topic user accuracy against model-average accuracy for Mistral and GPT-3.5 variants, respectively.The passages specify the plots’ scope but do not give per-topic values or comparative outcomes.Source: §G.3 · p. 22, Figure 16 · p. 29, Figure 17 · p. 30
  • Participants used confidence variably: some became skeptical below 50% or investigated predictions below 45%, while others trusted confidence to choose answers or relied on AI when personally unsure.One participant said they automatically agreed above 45, though sometimes challenged the prediction even above 45.Source: §G.3 · p. 22

G.4 Interface and Instructions

  • The study materials include a Modiste interface showing questions, model predictions, confidence, and a running score, alongside variant instructions and a post-study questionnaire.Participants in the LLM-only condition did not see confidence instructions, and those in the No-LLM condition were instructed to answer independently.Source: §G.4 · p. 23, Figure 19 · p. 31, Figure 20 · p. 32

H Broader Impact and Implications

  • The authors frame calibrated confidence as potentially improving interpretability and informed decisions, while noting that confidence scores may be misleading or damaging beyond fact-based questions.They caution that generative applications may not suit the paradigm and that even fact-based ground truth can be subjective; they cite AlphaFold as an example of useful calibrated confidence in a complex domain.Source: H Broader Impact and Implications · p. 23

9. Code Of Ethics

  • The authors say models are cited in Section 5, datasets in Appendix C.2, and trained models are released via Hugging Face.Source: §9 · pp. 39–40
  • The authors provide screenshots of participant instructions and details of compensation in Appendix G.Source: §9 · p. 40
  • The user study received prior approval from an institutional ethics review body, and all participants provided consent.Source: §9 · p. 40
Loading 2406.08391v3…