PaperlayerView exact revision on arXiv

Source-linked AI summary

Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

Sunnie S. Y. Kim, Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo, Olga Russakovsky

arXiv:2502.08554v1cs.HCcs.AI

TL;DR

LLMs can produce persuasive errors, making appropriate reliance difficult. This paper combines a think-aloud study with a preregistered controlled experiment to examine explanations, inconsistencies, and sources. Explanations increased reliance on both correct and incorrect answers, while sources and inconsistent explanations reduced reliance on incorrect answers.

  • Problem

    LLM responses can be inaccurate while sounding convincing, creating a need to understand how response features shape appropriate reliance.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    The authors first conducted a think-aloud study, then ran a large-scale preregistered controlled experiment with N=308 manipulating response accuracy, explanations, and clickable sources.

    Source: §1 · p. 2, §4.1.1 · p. 6

  • Results

    Explanations increased reliance on both correct and incorrect responses, whereas sources and inconsistent explanations reduced reliance on incorrect responses.

    Source: §1 · p. 2, §6 · p. 15

  • Takeaways & Limitations

    Accurate, relevant sources and interventions that help users notice inconsistencies are promising directions for fostering appropriate reliance, but should be tested with users before deployment.

    Source: §1 · p. 2, §6 · p. 15

    The studies used objective question-answering tasks, so their findings may not generalize to writing, idea generation, task automation, or other LLM-use contexts.

    Source: §5.3 · p. 14

Abstract

from arXiv · show

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.

1 Introduction

The paper examines how explanations, inconsistencies, and sources shape reliance on LLM responses. A think-aloud study identifies these features, followed by a controlled experiment isolating their effects.

  • LLM responses can be inaccurate in subtle ways, potentially misleading users who rely on them.Source: §1 · p. 1
  • The authors define explanations as supporting details or justifications accompanying an answer, without assuming they faithfully describe the model’s reasoning.Source: §1 · p. 1
  • A think-aloud study with 16 participants identified explanations, inconsistencies, and sources as response features likely to influence reliance.Source: §1 · pp. 1–2
  • A preregistered controlled experiment with N=308 manipulated answer accuracy, explanation presence, and clickable-source presence in a 2 x 2 x 2 design.Source: §1 · p. 2
  • Explanations increased reliance on correct and incorrect answers, whereas sources reduced overreliance on incorrect answers and inconsistencies reduced it when explanations were present.Source: §1 · p. 2
  • The authors propose accurate, relevant sources and attention to inconsistencies as promising strategies, while emphasizing user testing before deployment.Source: §1 · p. 2

2 Related Work

Prior research shows that explanations can increase reliance even on incorrect AI outputs, while the role of sources in LLM applications remains underexamined. This paper addresses that gap by identifying response features qualitatively and testing them experimentally.

  • AI-reliance research has mainly studied classical AI models, with a growing literature on LLMs and generative AI systems.Source: §2.1 · p. 2

Explanations and Inconsistencies

  • Prior explanation studies report mixed results, including increased reliance on incorrect systems when users rely on fluency or superficial expertise cues.Source: §2.2 · pp. 2–3
  • LLMs commonly provide supporting details for factual answers and detailed derivations for mathematical answers, often by default.Source: §2.2 · p. 3
  • The paper identifies inconsistencies in explanations as an unreliability cue that shapes reliance.Source: §2.1 · p. 2, §2.2 · p. 3

Sources

  • Sources are clickable links to external material, but they can be inaccurate, unsupported, or fabricated.Source: §2.3 · p. 3
  • The authors use a think-aloud study followed by a controlled experiment to examine whether sources reduce overreliance through verification or increase reliance as credibility signals.Source: §2.3 · p. 3, §2.2 · p. 3

3 Study 1: Think-Aloud Study

Study 1 used naturalistic ChatGPT interactions and think-aloud analysis to examine how people judge and verify LLM answers. Participants often overrelied on incorrect answers, while inconsistencies and source checking supported some corrective behavior.

Study design

  • The study observed participants solving objective questions with ChatGPT through multi-turn interactions in a relatively natural setting.Source: §3 · pp. 3–4, §3.1.1 · p. 4
  • Participants answered general, health or legal, and math questions, including binary factual questions and one numerical problem.Source: §3.1.1 · p. 4

Accuracy, reliance, and confidence

  • Study 1 collected 48 task instances from 16 participants, with participants reporting confidence on a 1–7 scale.Source: §3.1.1 · p. 4, §3.2.1 · p. 5
  • Among 13 initially incorrect ChatGPT answers, participants agreed with 9, indicating widespread overreliance.Source: §3.2.1 · p. 5

Response features shaping reliance

  • Participants judged explanations as important for reliability, but inconsistencies prompted follow-up questions and additional interaction.Source: §3.2.2 · pp. 5–6
  • Sources helped participants avoid some errors: 7 of 11 correct answers despite incorrect ChatGPT answers involved source checking.Source: §3.2.2 · p. 6
  • Source presence generally increased confidence, but broken or irrelevant links reduced confidence, and checking sources did not always eliminate overreliance.Source: §3.2.2 · p. 6

4 Study 2: Large-scale, Pre-registered, Controlled Experiment

Study 2 used a within-subjects, 2 x 2 x 2 experiment to test how answer accuracy, explanations, and clickable sources shape reliance and task accuracy. Explanations increased reliance on both correct and incorrect answers, whereas sources reduced overreliance on incorrect answers and inconsistencies mitigated explanation-induced overreliance.

Design

  • The experiment varied LLM answer accuracy, explanation presence, and clickable-source presence across eight response types.Participants completed eight tasks, seeing one response of each type, without feedback about answer correctness.Source: §4.1.1 · p. 6

Measures

  • Agreement with the LLM’s answer measured reliance, while participant-answer accuracy measured task outcome.Complementary measures included confidence, source clicking, time on task, response evaluations, and follow-up questions.Source: §4.1.2 · p. 7

Explanations

  • 78.2% versus 67.2%: explanations increased participant accuracy when the LLM answer was correct and no sources were provided.The explanation effect reversed for incorrect answers, reducing accuracy from 21.8% to 17.2%.Source: §4.2.1 · p. 9
  • Explanations increased agreement with the LLM for both correct and incorrect answers, alongside higher confidence and evaluations and fewer follow-up questions.The pattern suggests explanations made responses feel more satisfying and reliable regardless of answer accuracy.Source: §4.2.1 · p. 10

Sources

  • Sources increased agreement for correct answers from 67.2% to 73.4% but reduced agreement for incorrect answers from 78.2% to 68.2%, without explanations.Sources also changed confidence, justification quality, actionability, follow-up behavior, and time on task depending on answer accuracy.Source: §4.2.2 · pp. 10–11

Interactions

  • Inconsistent explanations reduced the joint effects of explanations and sources on self-reported measures, making their combined effects subadditive.When both were present, effects were less than the sum of their individual effects.Source: §4.2.3 · p. 11

5 Discussion

The discussion examines how explanations, sources, and inconsistencies shape reliance on LLM responses, while emphasizing limits on interpreting explanations and generalizing the findings. Sources and inconsistency cues can reduce overreliance, but their effectiveness depends on source quality and response context.

Explanations

  • Explanations increase reliance on both correct and incorrect LLM responses.Participants often interpreted LLM supporting details as explanations, although explanations may not faithfully describe how the model produced its answer.Source: Explanations · p. 13, §5.2 · p. 14

Sources

  • Sources reduce overreliance on incorrect answers and increase appropriate reliance on correct answers, but their benefits depend on source quality.The study’s sources were real, accurate, and relevant; broken, junk, unsupported, or fabricated sources may not foster appropriate reliance.Source: Sources · p. 13
  • Sources alone best reduced reliance on incorrect answers, whereas explanations combined with sources produced the highest accuracy on correct answers.For incorrect LLM answers, accuracy was highest with sources only (M=31.8%); for correct answers, it was highest with explanation and sources (M=79.9%).Source: Sources · p. 13

Unreliability cues

  • Inconsistencies in explanations functioned as unreliability cues that reduced reliance on incorrect LLM responses.Participants who noticed unreliability cues engaged more thoroughly, and inconsistent explanations were associated with less reliance on incorrect responses.Source: Inconsistencies and other… · pp. 13–14

Explanations

  • The paper defines explanations as supporting details that justify an answer, without assuming they faithfully represent the model’s internal reasoning.Faithfulness is difficult to evaluate without access to model internals, and participants varied in whether they viewed LLM responses as faithful explanations.Source: §5.2 · p. 14

Limitations

  • The findings may not generalize beyond objective question-answering, and the controlled experiment studied reactions to single prepared responses rather than ongoing interactions.The authors call for studies in other use contexts and more interactive settings.Source: §5.3 · pp. 14–15

6 Conclusion

The paper finds that explanations increase reliance on both correct and incorrect LLM responses, while sources and inconsistent explanations reduce reliance on incorrect responses. It recommends evaluating response features with users and developing interventions around sources and unreliability cues.

  • Explanations increase reliance on both correct and incorrect LLM responses.Source: §6 · p. 15
  • Sources and inconsistent explanations are associated with less reliance on incorrect responses.Source: §6 · p. 15
  • The authors emphasize evaluating LLM response features with users before deployment.Source: §6 · p. 15
  • Accurate, relevant sources and interventions that help users notice inconsistencies are proposed as directions for fostering appropriate reliance.Source: §6 · p. 15

A Additional Analyses: Study 2 Follow-up Questions

The follow-up-question analysis examined when participants asked to continue interacting with Theta and categorized the content of those questions. Follow-up-question instances varied substantially in whether responses included explanations or sources.

(1) Questions about related content (61.0%)

Participants asked follow-up questions about related task content, requested additional evidence or explanations, and expressed doubt about inconsistencies or conflicts with sources. These behaviors align with the finding that explanations and sources reduce follow-up-question likelihood.

Questions about related content (61.0%)

C Study 1 Task Questions Used

Study 1 used objective questions spanning general factual, health or legal, and mathematical domains. The question pool included comparisons, domain-specific factual questions, and multi-step arithmetic problems.

General domain factual questions

Health or legal domain factual questions

Math questions

D Study 2 Participant Demographics and LLM Background

Study 2 included 308 participants who reported demographic characteristics and their familiarity with, use of, and attitudes toward LLMs.

E Study 2 Full Wording of the Experiment

The experiment introduced Theta, an LLM-based question-answering prototype, and asked participants to answer questions with its assistance while restricting independent web searches.

Post-task attention check.

The post-task materials assessed question memory, reasons for disagreeing with Theta, perceived justification, helpfulness, and participants’ general LLM background and demographics.

  • Participants completed a memory check indicating whether each of 10 randomly selected questions had appeared in the experiment.Source: Post-task attention check · p. 22
  • Participants explained any final answer differing from Theta’s and rated the justification and answer-helpfulness of its responses.Source: Open-ended questions · p. 22
  • LLM background was measured separately from Theta using 5-point ratings of familiarity, usage frequency, and overall attitude.Source: LLM background · pp. 22–23
  • The questionnaire also collected age, highest education, gender identity, and race or ethnicity, including prefer-not-to-answer options.Source: Demographics · p. 23
  • Participants were told after the study that the AI answers could be inaccurate, incomplete, or inconsistent despite sounding convincing.Source: Debrief · p. 23

F Study 2 Task Questions and LLM Responses Used

Study 2 varied answer accuracy, explanation presence, and clickable sources, producing eight response types across objective questions and illustrating the correct-answer response materials.

Do our eyes have more than a million moving parts?

The response set presents conflicting answers and supporting explanations to objective questions, including whether the human eye has more than a million moving parts. Other examples address population thresholds, fingerprints, tears, food transit, and scuba diving at Port Royal.

Eye question

  • The response set includes a correct answer claiming that human eyes have more than a million moving parts, supported by a detailed explanation and sources.A contrasting incorrect response says the total number of distinct moving parts does not reach a million and identifies the main moving parts as the six extraocular muscles.Source: Do our eyes have more than a million… · pp. 24–25

Population question

Biological questions

Other questions

Related papers

Loading 2502.08554v1…