Source-linked AI summary
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky
TL;DR
LLMs can produce persuasive errors, making appropriate reliance difficult. This paper combines a think-aloud study with a preregistered controlled experiment to examine explanations, inconsistencies, and sources. Explanations increased reliance on both correct and incorrect answers, while sources and inconsistent explanations reduced reliance on incorrect answers.
Problem
LLM responses can be inaccurate while sounding convincing, creating a need to understand how response features shape appropriate reliance.
Method
The authors first conducted a think-aloud study, then ran a large-scale preregistered controlled experiment with N=308 manipulating response accuracy, explanations, and clickable sources.
Results
Explanations increased reliance on both correct and incorrect responses, whereas sources and inconsistent explanations reduced reliance on incorrect responses.
Takeaways & Limitations
Accurate, relevant sources and interventions that help users notice inconsistencies are promising directions for fostering appropriate reliance, but should be tested with users before deployment.
Takeaways & Limitations
The studies used objective question-answering tasks, so their findings may not generalize to writing, idea generation, task automation, or other LLM-use contexts.
Abstract
from arXiv · showhide
Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.
1 Introduction
The paper examines how explanations, inconsistencies, and sources shape reliance on LLM responses. A think-aloud study identifies these features, followed by a controlled experiment isolating their effects.
- LLM responses can be inaccurate in subtle ways, potentially misleading users who rely on them.
- The authors define explanations as supporting details or justifications accompanying an answer, without assuming they faithfully describe the model’s reasoning.
- A think-aloud study with 16 participants identified explanations, inconsistencies, and sources as response features likely to influence reliance.
- A preregistered controlled experiment with N=308 manipulated answer accuracy, explanation presence, and clickable-source presence in a 2 x 2 x 2 design.
- Explanations increased reliance on correct and incorrect answers, whereas sources reduced overreliance on incorrect answers and inconsistencies reduced it when explanations were present.
- The authors propose accurate, relevant sources and attention to inconsistencies as promising strategies, while emphasizing user testing before deployment.
2 Related Work
Prior research shows that explanations can increase reliance even on incorrect AI outputs, while the role of sources in LLM applications remains underexamined. This paper addresses that gap by identifying response features qualitatively and testing them experimentally.
- AI-reliance research has mainly studied classical AI models, with a growing literature on LLMs and generative AI systems.
- Explanations and Inconsistencies: Prior explanation studies report mixed results, including increased reliance on incorrect systems when users rely on fluency or superficial expertise cues.
- Explanations and Inconsistencies: LLMs commonly provide supporting details for factual answers and detailed derivations for mathematical answers, often by default.
- Explanations and Inconsistencies: The paper identifies inconsistencies in explanations as an unreliability cue that shapes reliance.
- Sources: Sources are clickable links to external material, but they can be inaccurate, unsupported, or fabricated.
- Sources: The authors use a think-aloud study followed by a controlled experiment to examine whether sources reduce overreliance through verification or increase reliance as credibility signals.
3 Study 1: Think-Aloud Study
Study 1 used naturalistic ChatGPT interactions and think-aloud analysis to examine how people judge and verify LLM answers. Participants often overrelied on incorrect answers, while inconsistencies and source checking supported some corrective behavior.
- Study design: The study observed participants solving objective questions with ChatGPT through multi-turn interactions in a relatively natural setting.
- Study design: Participants answered general, health or legal, and math questions, including binary factual questions and one numerical problem.
- Accuracy, reliance, and confidence: Study 1 collected 48 task instances from 16 participants, with participants reporting confidence on a 1–7 scale.
- Accuracy, reliance, and confidence: Among 13 initially incorrect ChatGPT answers, participants agreed with 9, indicating widespread overreliance.
- Response features shaping reliance: Participants judged explanations as important for reliability, but inconsistencies prompted follow-up questions and additional interaction.
- Response features shaping reliance: Sources helped participants avoid some errors: 7 of 11 correct answers despite incorrect ChatGPT answers involved source checking.
- Response features shaping reliance: Source presence generally increased confidence, but broken or irrelevant links reduced confidence, and checking sources did not always eliminate overreliance.
4 Study 2: Large-scale, Pre-registered, Controlled Experiment
Study 2 used a within-subjects, 2 x 2 x 2 experiment to test how answer accuracy, explanations, and clickable sources shape reliance and task accuracy. Explanations increased reliance on both correct and incorrect answers, whereas sources reduced overreliance on incorrect answers and inconsistencies mitigated explanation-induced overreliance.
- Design: The experiment varied LLM answer accuracy, explanation presence, and clickable-source presence across eight response types.Participants completed eight tasks, seeing one response of each type, without feedback about answer correctness.
- Measures: Agreement with the LLM’s answer measured reliance, while participant-answer accuracy measured task outcome.Complementary measures included confidence, source clicking, time on task, response evaluations, and follow-up questions.
- Explanations: 78.2% versus 67.2%: explanations increased participant accuracy when the LLM answer was correct and no sources were provided.The explanation effect reversed for incorrect answers, reducing accuracy from 21.8% to 17.2%.
- Explanations: Explanations increased agreement with the LLM for both correct and incorrect answers, alongside higher confidence and evaluations and fewer follow-up questions.The pattern suggests explanations made responses feel more satisfying and reliable regardless of answer accuracy.
- Sources: Sources increased agreement for correct answers from 67.2% to 73.4% but reduced agreement for incorrect answers from 78.2% to 68.2%, without explanations.Sources also changed confidence, justification quality, actionability, follow-up behavior, and time on task depending on answer accuracy.
- Interactions: Inconsistent explanations reduced the joint effects of explanations and sources on self-reported measures, making their combined effects subadditive.When both were present, effects were less than the sum of their individual effects.
5 Discussion
The discussion examines how explanations, sources, and inconsistencies shape reliance on LLM responses, while emphasizing limits on interpreting explanations and generalizing the findings. Sources and inconsistency cues can reduce overreliance, but their effectiveness depends on source quality and response context.
- Explanations: Explanations increase reliance on both correct and incorrect LLM responses.Participants often interpreted LLM supporting details as explanations, although explanations may not faithfully describe how the model produced its answer.
- Sources: Sources reduce overreliance on incorrect answers and increase appropriate reliance on correct answers, but their benefits depend on source quality.The study’s sources were real, accurate, and relevant; broken, junk, unsupported, or fabricated sources may not foster appropriate reliance.
- Sources: Sources alone best reduced reliance on incorrect answers, whereas explanations combined with sources produced the highest accuracy on correct answers.For incorrect LLM answers, accuracy was highest with sources only (M=31.8%); for correct answers, it was highest with explanation and sources (M=79.9%).
- Unreliability cues: Inconsistencies in explanations functioned as unreliability cues that reduced reliance on incorrect LLM responses.Participants who noticed unreliability cues engaged more thoroughly, and inconsistent explanations were associated with less reliance on incorrect responses.
- Explanations: The paper defines explanations as supporting details that justify an answer, without assuming they faithfully represent the model’s internal reasoning.Faithfulness is difficult to evaluate without access to model internals, and participants varied in whether they viewed LLM responses as faithful explanations.
- Limitations: The findings may not generalize beyond objective question-answering, and the controlled experiment studied reactions to single prepared responses rather than ongoing interactions.The authors call for studies in other use contexts and more interactive settings.
6 Conclusion
The paper finds that explanations increase reliance on both correct and incorrect LLM responses, while sources and inconsistent explanations reduce reliance on incorrect responses. It recommends evaluating response features with users and developing interventions around sources and unreliability cues.
- Explanations increase reliance on both correct and incorrect LLM responses.
- Sources and inconsistent explanations are associated with less reliance on incorrect responses.
- The authors emphasize evaluating LLM response features with users before deployment.
- Accurate, relevant sources and interventions that help users notice inconsistencies are proposed as directions for fostering appropriate reliance.
A Additional Analyses: Study 2 Follow-up Questions
The follow-up-question analysis examined when participants asked to continue interacting with Theta and categorized the content of those questions. Follow-up-question instances varied substantially in whether responses included explanations or sources.
- 43.8% of follow-up instances had neither an explanation nor sources.
- 17.9% of follow-up instances had an explanation only, 26.1% had sources only, and 12.3% had both.
- Researchers developed 11 follow-up-question codes after coding sampled questions for saturation.
(1) Questions about related content (61.0%)
Participants asked follow-up questions about related task content, requested additional evidence or explanations, and expressed doubt about inconsistencies or conflicts with sources. These behaviors align with the finding that explanations and sources reduce follow-up-question likelihood.
- Questions about related content (61.0%): 36.0% of follow-up questions directly concerned the LLM’s answer to the task question.
- Questions about related content (61.0%): Participants also asked why questions and occasionally re-asked the task question.The reported frequencies were 4.5% for why questions and 1.0% for re-asking the task question.
- Questions about related content (61.0%): 17.5% of follow-up questions requested sources, links, evidence, or proof, while 5.5% requested explanations or additional details.
- Questions about related content (61.0%): Participants pointed out inconsistencies in explanations and contradictions between LLM answers and sources.
- Questions about related content (61.0%): Sources and explanations significantly decreased the likelihood of asking a follow-up question.
C Study 1 Task Questions Used
Study 1 used objective questions spanning general factual, health or legal, and mathematical domains. The question pool included comparisons, domain-specific factual questions, and multi-step arithmetic problems.
- Participants received one general-domain factual question, one health or legal factual question, and one math question.
- General domain factual questions: General-domain questions included comparisons involving Paris and Tokyo, Tesla and Dropbox, and Moderna and Pepsi.
- Health or legal domain factual questions: Health or legal questions covered disease transmission, venom, rainwater collection, animal ownership, and stealing Wi-Fi.
- Math questions: Math questions required solving age relationships, exponential rice doubling, and a Fibonacci-based bird-count problem.
D Study 2 Participant Demographics and LLM Background
Study 2 included 308 participants who reported demographic characteristics and their familiarity with, use of, and attitudes toward LLMs.
- 308 participants completed the final sample, with gender, age, race, ethnicity, and education collected through an optional exit questionnaire.
- Participants reported moderate familiarity with LLMs, averaging 3.02 ± 0.88 on the 5-point LLM_Knowledge scale.The scale’s midpoint represented being moderately familiar and able to explain what LLMs are.
- LLM use averaged 3.04 ± 1.30, corresponding to use about 3–4 times a month.
- Attitudes toward LLMs averaged 3.70 ± 1.03, between neutral and somewhat positive.
E Study 2 Full Wording of the Experiment
The experiment introduced Theta, an LLM-based question-answering prototype, and asked participants to answer questions with its assistance while restricting independent web searches.
- Participants were told to imagine receiving an answer from an internet-connected AI system to a question they did not know.
- They used Theta to answer eight questions in an approximately 15-minute session completed in one sitting.
- Theta was described as an LLM system whose fluent, convincing responses might nevertheless be incorrect.
- Participants could click Theta’s source links but were instructed not to conduct their own internet searches.
Post-task attention check.
The post-task materials assessed question memory, reasons for disagreeing with Theta, perceived justification, helpfulness, and participants’ general LLM background and demographics.
- Participants completed a memory check indicating whether each of 10 randomly selected questions had appeared in the experiment.
- Participants explained any final answer differing from Theta’s and rated the justification and answer-helpfulness of its responses.
- LLM background was measured separately from Theta using 5-point ratings of familiarity, usage frequency, and overall attitude.
- The questionnaire also collected age, highest education, gender identity, and race or ethnicity, including prefer-not-to-answer options.
- Participants were told after the study that the AI answers could be inaccurate, incomplete, or inconsistent despite sounding convincing.
F Study 2 Task Questions and LLM Responses Used
Study 2 varied answer accuracy, explanation presence, and clickable sources, producing eight response types across objective questions and illustrating the correct-answer response materials.
- The experiment manipulated three response features—answer accuracy, explanation presence, and clickable sources—yielding eight response types.
- For each of 12 task questions, the appendix showed correct and incorrect responses containing both explanations and sources.
- One correct response stated that a moon jellyfish was sent to space before a cockroach and described the STS-40 experiment.
- Another correct response compared Mariana Trench visitors with astronauts who walked on the moon.
- Additional correct responses addressed water content in lungs versus skin, mammalian egg-laying beyond platypuses, and a gorilla hair-density question.
Do our eyes have more than a million moving parts?
The response set presents conflicting answers and supporting explanations to objective questions, including whether the human eye has more than a million moving parts. Other examples address population thresholds, fingerprints, tears, food transit, and scuba diving at Port Royal.
- Eye question: The response set includes a correct answer claiming that human eyes have more than a million moving parts, supported by a detailed explanation and sources.A contrasting incorrect response says the total number of distinct moving parts does not reach a million and identifies the main moving parts as the six extraocular muscles.
- Population question: A separate population question is answered with the claim that Brazil contains less than two-thirds of South America’s population.The response estimates Brazil’s population at roughly 213 million versus about 440 million for South America, or approximately 48%.
- Biological questions: Other responses state that not everyone is born with fingerprints and that reflex tears are produced in larger quantities than basal tears.The fingerprint response attributes the exception to adermatoglyphia, while the tears response distinguishes irritant-triggered reflex tears from continuously produced basal tears.
- Other questions: The response set also states that food typically reaches the stomach in about seven seconds through esophageal peristalsis and that Port Royal is accessible for scuba diving.The Port Royal response describes the submerged Jamaican site as mapped and studied, while noting that some areas may be too deep or delicate for casual divers.