When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots

• Fluent chatbot responses can still be inaccurate and ungrounded, potentially leading people to follow misleading advice.The paper notes that prior approaches include communicating model confidence and providing checkable sources, but indicator design remains unclear.Source: §1 · p. 1
• Large-scale evidence about everyday users’ verification behaviors and interpretations of confidence- and source-based indicators remains limited.Earlier work often focused on specific fact-checking tasks rather than personal questions about hobbies, work, and health.Source: §1 · p. 1
• The study combined interviews with eight AI users, a nationally representative survey of 1,000 U.S. AI users, and a 5,151-participant experiment varying indicator type, query topic, and response correctness.The experiment measured perceptions before and after participants learned whether responses were correct.Source: §1 · pp. 1–2
• Confidence and source indicators had small effects on perceptions before correctness was revealed, but those effects dissipated afterward; task stakes also shaped indicator effectiveness.Participants viewed non-medical responses more positively than medical ones.Source: §1 · p. 2
• Interviewees valued confidence cues but interpreted confidence in conflicting ways, while survey respondents usually verified through external information or source inspection rather than AI confidence.Users considered sources important evidence but also noted verification effort and other limitations.Source: §1 · p. 2

2.1 Assessing AI factuality

• Factuality means a response is truthful or based on fact; indicators can support evaluation, but their use in everyday chatbot interactions remains unclear.Prior approaches include in-line factuality highlighting [14], marking uncertain text [6], and removing incorrect information [13].Source: §2.1 · p. 2

2.2 Effects of communicating confidence and uncertainty

• Confidence can be communicated numerically, visually, or in natural language, but people may find precise estimates hard to interpret and differ in how they read verbal uncertainty.The paper compares percentages, categorical labels, and natural-language confidence cues.Source: §2.2 · p. 2

2.3 Effects of providing sources, evidence and citations

• Sources can aid verification, yet may be unreliable or go unopened; presentation affects engagement, and the best design for everyday personal queries remains unclear.Prior work found source verification improved judgments about when to rely on AI, but did not eliminate overreliance [20].Source: §2.3 · pp. 2–3

3.1 Method

• The exploratory qualitative study used semi-structured interviews to elicit users’ mental models and reactions to speculative factuality-indicator designs.Eight U.S. participants reported using generative AI weekly or daily.Source: §3 · p. 3, §3.1 · p. 3, §3.1.1 · p. 3

3.1.2 Interview Protocol.

• Interviews asked about accuracy assessment, confidence and its interpretation, indicator utility, source experiences, verification, and reactions to contextual scenarios.Participants compared alternatives and considered cases such as low-accuracy labels or confidence mismatches.Source: §3.1.2 · p. 3

3.1.3 Design Stimuli.

• Design stimuli varied confidence labels, percentages, explanations, conversational language, and source displays, presented in a medical-chatbot scenario.Source designs included citation pop-ups, highlighted response text, and a clickable “Researched” bubble.Source: §3.1.3 · pp. 3–4

3.1.4 Data analysis.

• Interview transcripts were analyzed through iterative bottom-up coding, with the research team reviewing and refining categories and themes until consensus.The first author compared codes after four interviews, then continued coding in batches of one or two interviews.Source: §3.1.4 · p. 4

3.2.1 Confidence indicators.

• Participants assigned confidence multiple meanings, sometimes shifting between interpretations within an interview; high confidence paired with an incorrect answer was seen as especially damaging to trust.Some understood confidence as answer correctness, response accuracy, or source fidelity; others assumed the chatbot had already verified its answer.Source: §3.2.1 · p. 4
• Participants preferred numerical confidence for complex or consequential questions, but some found percentages vague, and confidence alone could not explain an answer or substantiate its credibility.Confidence often served as a cue for allocating verification effort or inspecting supporting evidence.Source: §3.2.1 · pp. 4–5

3.2.2 Source indicators.

• Sources offered inspectable evidence and helped users assess trustworthiness, but participants differed on how many sources were enough and emphasized credibility and diversity.Credibility could rest on reputation or prior experience; some wanted many sources, while others considered one credible source sufficient.Source: §3.2.2 · pp. 5–6
• Participants raised concerns about source verification effort, retrieval accuracy, and fabricated or inaccessible links, while favoring claim-linked excerpts and accessible dropdowns.Healthcare and academic contexts were identified as particularly useful settings for sources; excerpts could reduce verification effort and help identify hallucinations.Source: §3.2.2 · p. 6

4.1.1 Participants.

• The survey sampled 1,000 U.S. adults who used at least one popular AI system monthly or more and asked about verification practices, reliability judgments, and desired chatbot support.Recruitment used a stratified YouGov opt-in panel intended to represent U.S. adults; respondents who used AI less frequently were screened out.Source: §4.1.1 · p. 6, §4.1.2 · p. 6

4.2.1 AI chatbot usage contexts.

• 79% rated confidence information important for health and medical advice, compared with 42% for roleplaying, entertainment, or games.Legal advice (77%) and financial planning (72%) also ranked highly; everyday information needs ranked lower (59%).Source: §4.2.1 · p. 6

4.2.2 Perceptions and experiences of AI chatbots.

• 81% had encountered inaccurate chatbot information in the previous 30 days, while respondents rated prior AI responses as moderately believable, accurate, and trustworthy.Mean ratings were 3.41 for believability, 3.39 for accuracy, and 3.14 for trustworthiness; 95.4% reported at least some confidence in evaluating accuracy, but only 10.1% said chatbots provided evaluation information to a very great extent.Source: §4.2.2 · pp. 6, 8

4.2.3 Verification behaviours, motivations, and attitudes.

• 94.7% of respondents verified chatbot responses at least some of the time, most often by looking online, inspecting chatbot sources, or checking other users’ views.Requesting confidence from the chatbot and comparing multiple chatbots were among the least common approaches.Source: §4.2.3 · p. 8
• 58.5% verified to ensure accuracy for themselves, 52.2% before sharing information, and 38.9% because the subject was high-stakes.Attitudes were mixed: reported double-checking was moderate, while time, effort, and uncertainty about verification varied.Source: §4.2.3 · pp. 8–9
• Survey findings indicate that people value evaluating chatbot factuality, especially for high-stakes queries, but do not verify consistently and are only moderately satisfied with their own ability and chatbot-provided support.Verification effort and motivation may vary with context.Source: §4.2.3 · p. 9

5.1.1 Participants.

• The experiment recruited 5,151 U.S. adults who used generative AI monthly or more, with the sample stratified to represent U.S. adults.Participants were randomly assigned to one between-participants condition.Source: §5.1.1 · p. 9, §5.1.2 · p. 9

5.1.2 Design.

• Conditions varied indicator type, topic (gardening, math, or medical), and response correctness; participants evaluated one chatbot example.The study measured accuracy and information sufficiency, along with trust and sentiment before and after correctness disclosure.Source: §5.1.2 · p. 9

5.1.3 Materials.

• Materials used five indicator combinations plus a no-indicator control, with medical, math, and gardening scenarios representing high-, medium-, and lower-stakes contexts.Confidence labels, bubbles, percentages, source icons, and source-plus-confidence language were among the displayed formats.Source: §5.1.3 · pp. 9–10

5.1.4 Procedure.

• Participants reviewed a chatbot screenshot, rated their perceptions, then learned whether its response was correct and rated their final impression and trust.The experiment used static screenshots rather than interactive chatbot exchanges.Source: §5.1.4 · p. 10

5.2 Results

• After correctness was revealed, indicator effects on trust and sentiment disappeared; response correctness had the largest influence, and non-medical responses drew more positive perceptions.Before disclosure, confidence-label and source-label indicators scored highest across accuracy, sufficiency, trust, and sentiment, although indicator differences were relatively small.Source: §5.2 · pp. 10–12
• Correct responses were rated more accurate than incorrect ones (M=3.776 vs. M=3.628), while no indicator pairwise differences in perceived accuracy were significant after correction.Math responses received higher information-sufficiency ratings than gardening and medical responses; the Sources and Confidence-language condition scored 3.495, significantly above the Confidence-bubble condition at 3.335.Source: §5.2 · p. 10

5.2.1 Effects of confidence levels.

• High-confidence responses received higher perceived-accuracy ratings than low-confidence responses (M=3.77 vs. M=3.62), but confidence level did not affect information sufficiency.Medical responses were rated less accurate than gardening or math responses; the study found no significant confidence-level interactions for perceived accuracy.Source: §5.2.1 · p. 13

5.2.2 Effects of confidence indicator misalignment on subjective trust and sentiment change.

• Trust and sentiment changed more after high-confidence than low-confidence responses, while correctness disclosure produced much larger changes in both judgments.Trust increased for responses revealed as correct (M=.454) and decreased for incorrect responses (M=−1.419); sentiment likewise rose for correct (M=.427) and fell for incorrect (M=−1.848) responses.Source: §5.2.2 · p. 13

6.1 The role of source and confidence indicators in assessing factuality

• Indicator effects were small and largely limited to initial perceptions; differences disappeared after participants learned whether responses were correct.Source indicators improved information-sufficiency judgments and initial sentiment over the Confidence-bubble indicator, while Confidence-label and Source-label indicators raised pre-revelation trust.Source: §6.1 · p. 14

6.2 The importance of aligning and communicating confidence accurately

• High-confidence signals generally improved evaluations, but misaligned high confidence on incorrect answers caused sharper trust drops than misaligned low confidence on correct answers.The authors suggest conservative indicators may be preferable because participants perceived incorrect advice as more consequential than missed correct advice.Source: §6.2 · p. 14
• Participants interpreted confidence inconsistently—as correctness probability, source agreement, or prior verification—creating different expectations and potential overreliance.The authors suggest explaining what confidence scores mean and how they are derived, potentially by connecting scores to evidence or highlighting uncertain response spans.Source: §6.2 · p. 14

6.3 Sources are necessary, but not sufficient alone

• Participants valued inspectable sources, but source provision alone did not ensure verification; integrated on-demand links and relevant excerpts could reduce effort.Source indicators paired with confidence-aligned language received the highest information-sufficiency judgments.Source: §6.3 · pp. 14–15

6.4 Considering stakes when designing factuality indicators

• Medical responses drew lower trust, sentiment, and perceived accuracy than mathematics or gardening, while confidence indicators had weaker effects in medical contexts.The authors suggest high-stakes topics may warrant richer evidence displays and greater uncertainty transparency than everyday questions.Source: §6.4 · p. 15

6.5 Limitations and future work

• Findings may not generalize beyond regular US AI users; opt-in panels, static screenshots, perception-only measures, and single-conversation exposure constrain the study’s reach.The interview sample was under 30, and the experiment did not assess reliance or decisions such as sharing responses.Source: §6.5 · pp. 15–16

7 Conclusion

• The authors studied factuality judgments and indicator effects using interviews, an observational survey, and a large-scale experiment.The studies examined how people integrate confidence cues, sources, and context when judging information reliability.Source: §7 · p. 16
• Confidence and source indicators can support assessment when they are interpretable, accessible, and sensitive to topic and complexity.Source: §7 · p. 16
• People ultimately judge systems by accuracy, so indicators should support calibrated expectations and informed decisions about when and how to verify AI-generated information.Source: §7 · p. 16

A.2 Survey

• The survey asked which chatbot tasks respondents used in the prior 30 days, including coding, studying, research, recommendations, productivity, and role-play.Source: §4 · p. 18
• Respondents rated AI replies for believability, accuracy, trustworthiness, and bias, and rated their confidence in evaluating response accuracy.Source: §4 · p. 18
• Survey questions covered checking sources and other information, requesting confidence, and the importance of uncertainty across everyday, financial, legal, health, civic, and relationship topics.Source: §4 · pp. 18–19

B.1 Interview Design Stimuli

• Figure 13 is labeled “Design stimulus B.”Source: Figure 13 · p. 20

C.1.1 Intent to Use.

• Correct responses prompted greater willingness to use the chatbot again than incorrect responses (M=3.58 vs. M=1.755).Correctness had a main effect, F(1, 5115) = 3297.692, p<.001, η²p=.392.Source: §C.1.1 · p. 20
• Indicator type had no significant effect on intent to use the chatbot again (F(5, 5115) = 1.034, p=.396).Source: §C.1.1 · p. 20
• Topic domain did not significantly affect intent to use (F(2, 5115) = 2.825, p=.059).Source: §C.1.1 · p. 20

C.1.2 Recommendability.

• Correct responses drew higher recommendation likelihood than incorrect responses (M=3.32 vs. M=1.6).Correctness had a main effect, F(1, 5115) = 2829.25, p<.001, η²p=.356.Source: §C.1.2 · p. 20
• Gardening respondents were more likely to recommend the chatbot than medical respondents (M=2.50 vs. M=2.38, p=.011, d=.07).No significant recommendation differences involved the Math condition (both ps >.05).Source: §C.1.2 · p. 20
• Factuality indicator type did not affect recommendation likelihood (F(5, 5115) = 1.09, p=.364).Source: §C.1.2 · p. 20

C.2 Experiment Items

• The item set included design stimulus images C and D, alongside questions about demographics and generative-AI chatbot use frequency.Source: §C.2 · pp. 20, 22, Figure 14 · p. 21, Figure 15 · p. 22
• Participants reviewed a chatbot conversation once for at least 30 seconds, then rated impression, trust, information sufficiency, and perceived accuracy.Source: §C.2 · p. 23
• After an accuracy revelation, participants rated their final impression, trust, likelihood of reuse, and likelihood of recommending the chatbot.The survey presented either a correct or incorrect response revelation.Source: §C.2 · p. 23