Source-linked AI summary
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Maarten Sap
TL;DR
The paper examines how language models communicate uncertainty, how users respond to epistemic markers, and how alignment feedback shapes model confidence. It finds that models are overconfident, users rely heavily on their responses, and human annotators are biased against uncertainty, with human-study findings limited by a U.S.-only sample.
Problem
Reliable human-AI interaction requires evidence about how language models communicate uncertainty and how users interpret those signals.
Method
The authors evaluate model-generated epistemic markers, study user reliance through human experiments, and analyze reward models and human-feedback datasets to trace sources of model confidence.
Results
Models are overconfident and users rely heavily on their responses; confident responses have an average 47% error rate, while annotators show bias against uncertainty.
Takeaways & Limitations
Model confidence language can create downstream safety harms because users rely on responses whether or not they contain explicit certainty markers.
Takeaways & Limitations
The human studies used U.S.-based participants exclusively, so the findings provide a narrow, U.S.-centric view of how people interpret LM-generated epistemic markers.
Abstract
from arXiv · showhide
As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence in responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are reluctant to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (an average of 47%) among confident responses. We test the risks of LM overconfidence by conducting human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in post training alignment and find that humans are biased against texts with uncertainty. Our work highlights new safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.
1 Introduction
This paper examines how language models communicate uncertainty and how users interpret those signals in human-AI interactions. It finds that models often express certainty despite errors, users rely heavily on model outputs, and human-feedback alignment may reinforce uncertainty aversion.
- The study evaluates uncertainty communication by eliciting epistemic markers from publicly deployed GPT, LLaMA-2, and Claude models on multiple-choice questions.The analysis focuses on realistic information-seeking scenarios.
- 47% of high-confidence responses were incorrect on average, showing that prompted confidence often overstates model accuracy.Models were more likely to generate certainty than uncertainty, including for incorrect answers.
- Users were heavily reliant on both marked and unmarked certainty in LM-generated answers.The user studies examined whether people would rely on answers or double-check them in calibrated and miscalibrated settings.
- The authors investigate model artifacts, alignment components, and human-feedback datasets to trace the origins of model confidence.This analysis identifies RLHF as a key contributing factor and examines annotator preferences toward uncertainty.
- The findings expose shortcomings in LM epistemic markers, downstream reliance risks, and potential mitigation strategies for human-LM communication.
2 Epistemic Markers in Language Models
The paper frames natural-language confidence expressions as epistemic markers that communicate speaker stance and commitment. It studies these markers through open-ended generations rather than predefined confidence scales, using a bottom-up analysis of LM behavior.
- The paper studies linguistic calibration, the alignment between LM accuracy and LM-articulated epistemic markers as perceived by users.
- Epistemic markers communicate speaker stance and commitment through weakeners expressing uncertainty and strengtheners expressing certainty.
- Open-ended generations enable a qualitative bottom-up analysis of how LMs produce epistemic markers in realistic language use.This departs from methods that prescribe numerical or ordinal confidence expressions.
3 How do LMs use Epistemic Markers?
Across nine publicly deployed models, epistemic markers are usually absent unless explicitly prompted, and explicit prompting produces a strong bias toward certainty despite frequent errors.
- 3.1 Methods: The study elicited open-ended responses to diverse MMLU questions using epistemic, chain-of-thought, and combined prompts rather than predefined confidence scales.Prompts were expanded through paraphrases gathered from Mechanical Turk workers and GPT-3.5, and responses were manually validated after regex detection.
- 3.2 Findings: 65% of generations included epistemic markers with explicit epistemic prompts, compared with 16% under chain-of-thought instructions and 5% with the base template.The study used 49 prompts on 284 MMLU questions across nine models; markers were identified through regex-assisted qualitative coding.
- 3.2 Findings: 20% of generations used strengtheners versus 14% using weakeners, with six of nine models significantly favoring certainty expressions.The strengthener bias was especially pronounced in GPT and LLaMA-2 chat models, whereas Claude-2 was more balanced.
- 3.2 Findings: 53% of generations with certainty expressions were correct, leaving 47% incorrect despite exceeding the 25% random-accuracy baseline.Weakeners were associated with 32% accuracy, while strengtheners appeared in 17% of all incorrect answers.
- 3.3 Discussion: Overall, models either omit uncertainty or overuse certainty markers, creating a linguistic miscalibration that may increase downstream human overreliance on AI.The paper identifies this miscalibration as an emerging safety risk in human-LM collaboration and calls for cognitive-forcing designs such as verbalized uncertainties.
4 Human Interpretations of Uncertainty
The human experiments measure how users rely on LM responses marked with certainty, uncertainty, or no epistemic markers, both without feedback and across interactive rounds. Users rely heavily on LM outputs, learn calibrated markers, but remain vulnerable to overconfident miscalibration and its lasting effects.
- 4.1 Methods: The task presented challenging country-capital questions, LM answers with epistemic markers, and a choice between relying on Marvin or looking up the answer.Interactive settings included feedback so participants could build a mental model of Marvin’s performance.
- 4.3 Findings: Users relied on plain statements and strengtheners nearly 90% of the time, while approximately 90% chose to look up answers after weakeners.Without epistemic markers, users still interpreted plain answers as signs of model certainty.
- 4.3 Findings: In the calibrated interactive setting, users learned to rely on strengtheners 99% of the time and weakeners nearly 1% of the time after 20 rounds, averaging 97% accuracy.The results indicate that participants primarily relied on epistemic markers rather than their own knowledge.
- 4.3 Findings: Users relied on 81% of strengtheners although only 66% of strengthener-marked generations were correct, mistakenly trusting incorrect confident responses 73% of the time.Performance averaged 76% in miscalibrated rounds versus 86% in calibrated rounds.
- 4.3 Findings: Overconfident responses caused users to rely on weakeners at 9%, compared with 3% in the control setting, even though none of those weakened answers were correct.Miscalibration in strengtheners altered users’ interpretation of other epistemic markers.
- 4.3 Findings: Underconfident models produced 66% performance in miscalibrated rounds and 98% in later calibrated rounds, whereas users’ mental models remained incompletely corrected after overconfidence.The experiments used challenging trivia and feedback-based interaction to measure reliance on LM-generated markers.
5 Origin of Model Overconfidence
The analysis traces LM overconfidence to post-training alignment, finding that RLHF models favor certainty while reward models and human annotators penalize uncertainty.
- Overconfidence in RLHF Models: RLHF models emit more strengtheners than weakeners, reversing the pattern observed in base and instruction-tuned models.This suggests that the preference for strengtheners is introduced during RLHF.
- Reward Modeling: Reward modeling favors plain statements and strengtheners while strongly penalizing weakeners, biasing generation away from uncertainty.Average reward scores are 4.03 for plain statements, 0.82 for strengtheners, and -1.86 for weakeners.
- Human Raters: Human annotators do not prefer strengtheners, but they disfavor weakeners relative to plain and strengthened text.Strengtheners appear in rejected versus chosen texts at 2.95% versus 2.72%, while weakeners appear at 5.02% versus 4.47%; plain text is chosen 9% more often than strengthened text.
- Discussion: The paper identifies annotator bias against uncertainty as a contributor to RLHF overconfidence and proposes dataset-label interventions while warning of new biases.The authors argue that uncertainty bias can increase human overreliance on LMs, but label swapping may introduce other unknown biases.
6 Desired Criteria and Mitigating Solutions Moving Forward
The paper proposes designing LMs to communicate uncertainty more effectively through unsolicited markers, broader epistemic coverage, and calibration that depends on deployment context.
- Unsolicited Epistemic Markers: LMs should emit epistemic markers autonomously because unmarked responses are perceived by humans as tacit certainty.The proposed criterion targets uncertainty communication without requiring users to prompt for it.
- Unsolicited Epistemic Markers: Potential RLHF interventions include uncertainty-focused data augmentation, annotator training, and dataset evaluation.These interventions are presented as implementation options for improving uncertainty communication.
- Comprehensive Coverage: Models should generate and interpret a comprehensive range of epistemic markers rather than relying on limited written-internet training data.The paper suggests adding speech-derived sources such as podcasts, news transcripts, and peer conversations.
- Context-Dependent Calibration: Calibration thresholds should vary with deployment context, allowing different certainty standards for entertainment and mission-critical applications.User calibration or system-level procedures could measure reliance in a target context and guide prompting or fine-tuning.
7 Conclusion
The paper concludes that LM overconfidence and users’ reliance on LM responses create a human-AI communication risk, with overconfidence traced to RLHF and annotator bias against uncertainty.
- Conclusion: LMs are overconfident, users rely heavily on their responses with or without explicit confidence, and RLHF annotators bias models against uncertainty.The conclusion links model behavior, user reliance, and feedback-data bias within the paper’s scope.
8 Limitations
The limitations concern cultural and linguistic scope, ambiguity in weakened-strengthener interpretation, and the gap between recruited participants and real deployed users.
- Cultural Interpretations of Epistemic Markers: The study examines English epistemic markers, so findings may differ for models operating in other languages and cultures.The paper notes that hedges and strengtheners vary across languages, contexts, and cultures.
- Cultural Interpretations of Epistemic Markers: U.S.-only human studies provide a narrow view because reliance on epistemic markers may vary with cultural background and experience.Participants from other cultural backgrounds might show different reliance patterns.
- The Ambiguity of Weakened Strengtheners: Weakened-strengtheners have ambiguous interpretations, with substantial variation in how participants rely on them.Their frequent use could create confusion or misinterpretation, motivating further study of nuanced epistemic markers.
- Gap Between Human Experiments and Real Self-Incentivized Users: Experimental participants may not represent self-incentivized users in deployed settings, whose contexts could change epistemic-marker interpretation.The authors expect harms in real deployments to differ from those observed in their experiments.
9 Ethics Statement
The ethics statement frames the paper around harms from human–LLM interaction and describes prompt-generation procedures intended to preserve neutrality.
- The paper focuses on potential harms and ethical implications arising when humans interact with LLMs.
- The authors report that LLMs systematically fail to represent uncertainty, creating a threat of human overreliance on incorrect generations.
- Prompts were paraphrased by Mechanical Turk workers, expanded by GPT-3.5, and designed to place certainty and uncertainty keywords together in random order.
B Details on Experiments from Section 4
The experiments used screened U.S.-based participants, naturalistic uncertainty expressions, calibrated scoring, and a temperature-based mitigation test.
- Recruitment Process Details: Each human experiment recruited 25 English-speaking, U.S.-based Prolific participants meeting minimum experience and approval criteria.Participants had at least a 97% approval rating and 100 completed tasks.
- Recruitment Process Details: Participants were paid an average of $17.77 USD per hour, and the experiments ran from September through November 2023.
- A separate paraphrasing task recruited 29 Mechanical Turk participants, who were paid $1 USD each.
- The research team received an internal review-board exemption and still obtained informed consent from all participants.
- Uncertainty expressions were deduplicated and excluded when primarily numeric or ordinal so the remaining templates stayed naturalistic.
- Scoring Details: The human-scoring task rewarded relying on Marvin when correct, penalized relying on Marvin when incorrect, and made looking up answers neutral.Because half the answers were wrong, always relying or always looking up produced a total score of 0.
- The study tested increased temperature as a mitigation for model miscalibration but reports observing no improvement from maximum temperatures.
D Reward Model Compared to Human Scoring of Expressions of Uncertainty
This section documents comparisons between human and reward-model scoring, prompt-generation materials, reliability judgments, and common certainty-expression templates.
- Table 3 compares human certainty scores with OpenAssistant’s reward-model scores for expressions of uncertainty.Human scores are defined as the percentage of control-setting responses in which participants relied on an expression.
- The reported artifacts included the OpenAssistant reward model and datasets, MMLU, and LLaMA 2, used for querying, evaluation, and analysis.
- Tables 4 and 5 list prompts and identify their author, crowdworker, or GPT origin within the snowball-sampling process.
- Table 6 reports human judgments of templates based on reliability.
- Tables 7 and 8 list the 20 most common strengtheners and weakeners generated from chat models.