What Large Language Models Know and What People Think They Know

Mark Steyvers; Heliodoro Tejeda; Aakriti Kumar; Catarina Belem; Sheer Karny; Xinyue Hu; Lukas Mayer; Padhraic Smyth

Source-linked AI summary

What Large Language Models Know and What People Think They Know

Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas Mayer, Padhraic Smyth

arXiv:2401.13835v2Jan 24, 2024cs.LGcs.AIcs.CLcs.HC

TL;DR

This paper examines how effectively LLM explanations communicate model uncertainty to users, focusing on calibration and discrimination gaps. Experiments with multiple-choice and short-answer questions found that default and longer explanations can inflate human confidence, while confidence-tailored explanations improved alignment between user perceptions and model confidence.

Problem
Less is known about how effectively LLMs convey their internal uncertainty to users, despite the importance of communicating reliability in decision-making.
Method
The authors conducted experimental studies using multiple-choice and short-answer questions to compare human assessments of LLM confidence with model confidence and accuracy.
Results
Users overestimated LLM accuracy with default explanations, longer explanations increased confidence without improving discrimination, and confidence-tailored explanations narrowed calibration and discrimination gaps.
Takeaways & Limitations
Aligning explanation language with model confidence can support more responsible and trustworthy use of LLMs where generated information affects critical decisions.
Takeaways & Limitations
The study examined multiple-choice questions with few alternatives and short-answer questions, so applicability to longer open-ended questions remains uncertain.

Abstract

from arXiv · show

As artificial intelligence (AI) systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs' internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models' actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models' internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in AI-assisted decision-making environments. Code and Data can be found at https://osf.io/y7pr6/ . Journal publication can be found on Nature Machine Intelligence at https://www.nature.com/articles/s42256-024-00976-7 .

1. Introduction

LLMs can show internal calibration, but users typically infer confidence from generated language, creating a gap between what models know and what people think they know. This paper defines and investigates calibration and discrimination gaps and tests confidence-aligned explanations.

Motivation: LLM reliability is uncertain because fluent outputs can be inaccurate or nonsensical, making model confidence difficult for users to assess.Developers caution against uncritical acceptance of model outputs.
Key concepts: Model confidence is the probability assigned to a selected answer, whereas human confidence is a user’s probability judgment based only on the LLM’s language.Internal model confidence is not usually displayed to users.
Research questions: The calibration gap compares the reliability of model and human confidence, while the discrimination gap concerns their ability to distinguish likely correct from incorrect answers.The paper asks how large these gaps are and whether adapting textual output can reduce them.
Approach: The paper presents experiments and a dataset capturing human assessments of LLM confidence in question answering.Studies use GPT-3.5 and PaLM2 on multiple-choice MMLU questions and GPT-4o on short-answer TriviaQA questions.
Approach: The work tests response-generation methods intended to improve human-confidence calibration relative to model confidence and true LLM accuracy.This targets more reliable interpretation of LLM textual responses.

2. Methodology

The methodology combines internal LLM confidence estimates with behavioral experiments measuring how humans judge answer correctness from explanations alone. It evaluates multiple-choice and short-answer datasets while varying explanation uncertainty and length, using calibration and discrimination metrics.

Behavioral experiments: Participants estimated the probability that an LLM answer was correct from its explanation without seeing the model’s numerical confidence.For multiple-choice questions, participants also answered questions with LLM assistance; the design assumed limited domain expertise would make explanation content especially influential.
Behavioral experiments: The study compared default explanations with explanations manipulated across three uncertainty-language levels and three explanation-length levels.Experiment 1 used default explanations, while Experiment 2 altered uncertainty language and overall length.
Question datasets: The multiple-choice evaluation sampled 350 MMLU questions across 10 topics, approximately balancing model-confidence bins for each LLM.Questions were sampled separately for GPT-3.5 and PaLM2; the lowest confidence bin contained fewer questions because relatively few questions fell there.
Question datasets: The short-answer evaluation used 336 Trivia QA questions sampled across four confidence bins and seven topics after assessing model confidence on 5,000 questions.GPT-4o generated answers, then evaluated whether each proposed answer was true or false; model accuracy on the behavioral subset was 63%.
Confidence measurement: Internal confidence was obtained from token likelihoods for multiple-choice answers and from a pTrue-style true-or-false evaluation for short answers.For multiple-choice questions, the selected answer’s relative token likelihood was used; for short answers, normalized probabilities over true and false were used.
Evaluation metrics: Expected Calibration Error averaged absolute accuracy–confidence differences across equal-width bins, while AUC measured confidence-based discrimination between correct and incorrect answers.ECE may combine overconfidence and underconfidence within its aggregate value, whereas AUC uses pairwise comparisons of correct and incorrect answers.

3. Results

Default LLM explanations produced substantial calibration and discrimination gaps: people often overestimated answer accuracy and were less able than models to distinguish correct from incorrect answers. Matching explanation uncertainty to model confidence narrowed both gaps, although longer explanations raised confidence without improving discrimination or decision accuracy.

Calibration gap: ECE was much lower for model confidence than human confidence across LLMs and question types with default explanations.This indicates misalignment between perceived accuracy and actual LLM accuracy.
Calibration gap: For multiple-choice questions, human miscalibration was primarily overconfidence, with participants assigning high confidence despite lower associated model accuracy.The calibration diagrams show human confidence above expected accuracy in these conditions.
Discrimination gap: Human confidence discriminated less effectively than model confidence between likely correct and incorrect answers under default explanations.Discrimination was assessed with AUC, with the reported gap appearing across the behavioral experiments and LLMs.
Explanation style and length: Uncertainty language strongly shifted human confidence: low-confidence wording produced less confidence than medium wording, which produced less than high wording.These ordered differences were reported across Experiments 2a, 2b, and 2c, with BF > 100 for both comparisons.
Explanation style and length: Long explanations increased human confidence relative to short explanations, while short explanations also exceeded uncertainty-only responses.The combined evidence reports BF = 25 for long versus short explanations and BF > 100 for short versus uncertainty-only responses.
Calibration and discrimination results: Selecting explanation styles according to model confidence narrowed both calibration and discrimination gaps across LLMs and question types.The reported Bayes factors were BF > 100, BF = 6.48, and BF > 100 for Experiments 2a, 2b, and 2c, respectively.
Participants lack specialized knowledge: Aligned explanations improved recognition of likely correct or incorrect answers without improving human decision accuracy in Experiments 2a and 2b.Participants’ independent knowledge was insufficient to improve on the LLM answers despite better calibration and discrimination.

4. Discussion

The study finds that users often overestimate LLM accuracy, especially when explanations are longer or default-generated, but confidence-tailored explanations improve alignment between model confidence and user judgments. These findings support more transparent uncertainty communication while leaving open questions about generalizability and underlying causes.

Users consistently overestimated LLM output accuracy across three models and both multiple-choice and short-answer questions, particularly with default explanations.
Longer explanations increased human confidence without improving discrimination between likely correct and incorrect answers.The authors suggest that users relied on shallow textual cues, such as overall explanation length.
Confidence-tailored explanations improved users’ calibration and discrimination by expressing uncertainty in language aligned with model confidence.Prompts induced low-, medium-, or high-confidence wording, such as expressing uncertainty or confidence explicitly.
The study’s evidence is limited to multiple-choice questions with few alternatives and short-answer questions, leaving applicability to longer open-ended questions unresolved.The confidence-modification approach also required prompting the LLM twice.
The authors identify the causes of uncertainty miscommunication as an important area for future research, including possible effects of human-preference alignment and autoregressive generation.They hypothesize that RLHF may favor detailed, persuasive explanations and that answer commitment may inflate rationales.
The findings motivate transparent communication because aligning model confidence with user perceptions may support more responsible and trustworthy LLM use.

A.1 Demographic Information by Experiment

Table 3 reports demographic information by experiment and notes that sex was not reported by all participants.

Table 3 presents the breakdown of demographic information by experiment.
Sex was not reported by all participants.

A.2 Experiment 3

Experiment 3 used an alternative rewriting prompt with GPT-3.5 and PaLM2, and reproduced the main pattern without manipulating explanation length.

Experiment 3 rewrote baseline explanations as if the model were not sure, somewhat sure, or sure.
Experiment 3 used GPT-3.5 and PaLM2 with 81 participants and omitted the explanation-length manipulation.
Human confidence in Experiment 3 showed smaller calibration error and larger discrimination than the comparison condition.The passage states that this pattern matched Experiments 2a and 2b.

A.3 Overconfidence Error

The overconfidence analysis uses Overconfidence Error alongside calibration and discrimination measures, with model-confidence-modified explanations reducing overconfidence relative to default explanations.

Overconfidence Error adapts the ECE formula to focus specifically on cases where confidence exceeds accuracy.
Table 4 reports ECE, OE, and AUC for model and human confidence across experiments.
Default explanations produced the largest human overconfidence error, whereas model-confidence-modified explanations lowered it.
Figure 5 presents calibration diagrams for GPT-3.5 and PaLM2 on MMLU and GPT-4o on a 5,000-question TriviaQA sample.

A.4 Human Confidence Agreement

Experiment 2 assessed agreement among participants’ confidence ratings, analyzing correlations at the level of experimental manipulations because same-question, same-language, same-length ratings were sparse.

Participant agreement was measured using the mean correlation between every pair of participants.
Because repeated ratings under identical question, uncertainty-language, and explanation-length conditions were uncommon, agreement was analyzed across experimental manipulations.
Figure 6 compares calibration diagrams for GPT-3.5 and PaLM2 on MMLU questions and GPT-4o on Trivia QA questions.
The lowest-confidence multiple-choice bin contains fewer questions because questions were sparse in that confidence range.

Appendix B. Additional Model Confidence Results

Additional model-confidence analyses compare full datasets with the subsets used in behavioral experiments, reporting model accuracy and AUC for MMLU and Trivia QA.

Behavioral experiments used subsets of 350 multiple-choice questions and 336 short-answer questions, compared with the full MMLU and Trivia QA sets.
GPT-3.5 achieved 63% accuracy and an AUC of 0.78 across all 14,042 MMLU questions.When computing model confidence, 8.7% of answers were incomplete and removed.
PaLM2 achieved 51% accuracy and an AUC of 0.73 across the MMLU questions.
GPT-4o achieved 85% accuracy and an AUC of 0.85 across a 5000-question Trivia QA sample.
For the 336-question Trivia QA behavioral subset, GPT-4o achieved 63% accuracy and an AUC of 0.78.Uniform sampling across confidence bins over-represented lower-confidence questions, where the model was less accurate.

Appendix C. Optimization Procedure

The optimization procedure maps model-confidence ranges to low-, medium-, or high-confidence explanation styles using two thresholds, θ1 and θ2, selected separately for each experiment and model.

Explanation style is selected by mapping model confidence p to low, medium, or high confidence using thresholds θ1 and θ2.
Low confidence applies when p ≤ θ1, medium confidence when θ1 < p ≤ θ2, and high confidence when θ2 < p.
The selection rule filters participant estimates when an explanation’s style does not match the selected style and ignores explanation-length variation in Experiments 2 and 3.
Sensitivity analyses varied θ1 and θ2 and evaluated resulting AUC and ECE relating human confidence to actual LLM accuracy.
Thresholds were optimized separately for each experiment and LLM using a combined objective weighting ECE at 33% relative to AUC.
The optimized thresholds were θ1 = 0.65 and θ2 = 0.75 for GPT-3.5, 0.50 and 0.70 for PaLM2, and 0.50 and 0.60 for GPT-4o.

C.1 Sensitivity Analysis

Sensitivity analyses indicate that calibration and discrimination outcomes are not overly sensitive to threshold changes, while supplementary materials document prompts and example explanations.

C.1 Sensitivity Analysis: Calibration and discrimination results were not overly sensitive to parameter changes.
C.1 Sensitivity Analysis: For GPT-3.5 Experiment 2a, mean AUC was 0.649 and mean ECE was 0.203 across all parameter settings.
C.1 Sensitivity Analysis: GPT-3.5 Experiment 2a showed moderate evidence for improved AUC across 52% of parameter combinations, with BF>3.
C.1 Sensitivity Analysis: Supplementary Figures 7 and 8 display AUC and ECE outcomes for each parameter setting.
C.1 Sensitivity Analysis: Supplementary Table 5 provides prompt styles, while Supplementary Table 6 provides example explanations for a multiple-choice question in Experiments 1a and 2a.