PaperlayerView exact revision on arXiv

Source-linked AI summary

How Children Design and Reason about Trustworthy AI Chatbots

Deniz Ozturk, Jiayu Li, Daksh Pratap Singh, Yasitha Rajapaksha, Fasika Melese, Bahare Riahi, Shiyan Jiang, Qiao Jin, Joey Huang, Veronica Cateté, Tiffany Barnes, Xiaoyi Tian

arXiv:2609.25244v5cs.HCcs.AI

TL;DR

Research on children’s AI trust has mostly examined them as users judging systems designed by others, leaving less evidence about how they design trustworthy chatbots. This mixed-methods study examined children’s chatbot designs and trust reasoning, finding that judgments and design choices varied with age and chatbot domain.

  • Problem

    Research has mainly studied children evaluating AI systems built by others, leaving less evidence about how children reason about and design chatbot trustworthiness.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    A mixed-methods study examined 115 students’ chatbot designs, using surveys, system logs, interviews, and design artifacts.

    Source: §1 · pp. 1–2

  • Results

    Some children called deliberately incorrect chatbots trustworthy when they fulfilled their designed purpose, while age and chatbot domain were associated with differences in trust reasoning and trait settings.

    Source: §7 · p. 15

  • Takeaways & Limitations

    The findings suggest that children’s chatbot trustworthiness judgments involve interacting design choices and that trust criteria may differ across ages and domains.

    Source: §1 · p. 2, §7 · p. 15

    Cohort differences are exploratory because cohorts differed in setting, schedule, instruction, interface, activities, and data collection, with modest and unequal age-group samples.

    Source: §6.4 · pp. 14–15

Abstract

from arXiv · show

Children increasingly interact with AI chatbots, making trust calibration essential to AI literacy. Prior research has examined children's trust in AI mainly as users evaluating systems built by others, rather than as designers of their own chatbots. We developed a chatbot-building environment with adjustable trust-relevant traits (e.g., confidence, transparency, formality, assertiveness), rules, and persona. We conducted mixed-methods study with 115 learners (ages 8-18) who made 119 chatbots. We examined how children configured their chatbots, reasoned about trustworthiness, and how closely chatbot behavior aligned with their designs. Younger students (age 10-13) set significantly higher confidence than older students (age 14-18), and some deliberately built chatbots that gave wrong answers on purpose, yet still called them trustworthy, arguing that a chatbot does what it was built to do. Younger students equated trust with purpose-fulfillment, while older students linked it to transparent, calibrated design. Students also calibrated academic chatbots to be more transparent and formal than hobby chatbots. We identify seven design dimensions describing what children believe makes a chatbot trustworthy, and discuss implications for AI literacy tools.

1 Introduction

Children’s everyday chatbot use makes calibrated trust important, yet research has focused mainly on children judging systems designed by others. This study examines how children translate beliefs about trustworthiness into chatbot design choices.

1 Introduction

  • Children’s everyday chatbot use for homework, entertainment, and information seeking makes trust calibration important because inaccurate responses can lead to overtrust or undertrust.Overtrust may leave children with incorrect information, while undertrust may keep them from useful AI tools.Source: §1 · p. 1
  • Prior studies largely measured children’s trust as users of systems built by others, leaving less evidence about their own trust-related design decisions.The study addresses this gap by examining children’s choices and explanations when they build chatbots themselves.Source: §1 · p. 1
  • The study positions children as designers because translating ideas about trustworthiness into concrete choices can reveal their mental models and areas needing support.Chatbots offer an accessible setting for children to create and customize conversational AI.Source: §1 · pp. 1–2
  • The mixed-methods study involved 115 students who designed, built, and iteratively refined chatbots in a chatbot-building environment.The researchers combined pre/post surveys and tool logs with thematic coding of interviews and design artifacts.Source: §1 · p. 2
  • The paper identifies seven design dimensions and reports exploratory evidence that age and chatbot domain shape children’s trust-related design choices.It proposes supporting children in articulating design intentions, testing resulting behavior, and revising when behavior misses those intentions.Source: §1 · p. 2

2 Related Work

Prior research finds that children’s trust in AI reflects system qualities and how those qualities are communicated, but evidence often comes from children evaluating systems made by others. This paper connects child-AI interaction and AI literacy to studying children as designers of trustworthy chatbots.

2.1 Children’s Trust in AI and Conversational Agents

  • Children may overtrust conversational agents, especially when systems are opaque, while transparency and expressed uncertainty can support better-calibrated judgments.Prior work reports that children adopted incorrect answers less often when a chatbot expressed uncertainty.Source: §2.1 · p. 2
  • Trust studies often rely on post-interaction surveys or binary judgments and position children as users, leaving less insight into their reasoning and design choices.The study addresses this gap by examining children’s explanations and choices as they build agents.Source: §2.1 · p. 2

2.2 Child-AI Interaction

  • Prior child-AI interaction research finds that learning how AI is programmed and processes information can shift trust, but does not examine children’s own design choices across ages.Related work also raises open questions about age differences in collaboration and communication during child-AI co-creation.Source: §2.2 · pp. 2–3

2.3 AI Literacy through Chatbot Development

  • AI literacy combines critical evaluation, communication, and responsible use, while chatbots offer an accessible setting for engaging with hallucinations and other AI concepts.Designing a chatbot links setting rules and constraints with reasoning about trustworthiness.Source: §2.3 · p. 3
  • The paper extends AI literacy toward designing AI responsibly by asking how children adjust chatbot features to enact their beliefs about trustworthiness.This frames hallucination literacy not only as detecting errors but also as understanding design choices that may mislead others.Source: §2.3 · p. 3

3 Chatbot Development Environments

LUMI makes children’s chatbot design choices visible through an iterative configure-test-evaluate-revise workflow. Its controls and inspection features let students shape bot behavior, examine outputs, and revise their designs.

3.1 Design Goals and Interaction Workflow

  • LUMI’s configure-test-evaluate-revise workflow makes trust-related choices and reactions to chatbot behavior visible across design iterations.Students configure a bot, inspect its responses, evaluate them against expectations and peer feedback, then revise and repeat.Source: §3.1 · p. 3

3.2.1 Configuring Chatbot Identity and Behavior.

  • Students configure chatbot identity, purpose, persona, tone, rules, and four communication traits, with optional knowledge sources.Confidence, Transparency, Formality, and Assertiveness sliders each have low, medium, and high levels translated into model instructions; the traits control expression rather than factual reliability.Source: §3.2.1 · pp. 3–4

3.2.2 Testing and Inspecting Responses.

  • Response inspection separates reasoning, answer, and model-confidence information, while Fact Check and Compare Models support evaluation of claims and competing outputs.The confidence bar reflects confidence in an answer, unlike the confidence trait slider, which controls how confidently the chatbot sounds.Source: §3.2.2 · p. 4

3.2.3 Revising and Sharing Chatbots.

  • Version history, saved conversations, and peer sharing let students trace revisions and surface mismatches between design intent and user experience.Configurations, conversations, and interaction events were timestamped for analysis.Source: §3.2.3 · p. 4

3.3 Controlled Hallucination

  • A controlled rewriting pipeline generated fluent but materially incorrect answers when an evaluation model rated a saved configuration low on clarity, constraint specificity, or coherence.Rewriting stopped when the configuration no longer received low ratings; all participants encountered the mechanism, with frequency varying by prompt design and configuration quality.Source: §3.3 · p. 4

3.4 System Variants and Deployment Contexts

  • Elementary learners used ToyTalk, an adapted LUMI with toy-photo and voice features, while retaining the core configure-test-evaluate-revise cycle and four trait sliders.Middle and high school students used the full text-based LUMI interface.Source: §3.4 · p. 4

4 Methods

The methods combine age-adapted workshops, multiple forms of evidence, and quantitative and qualitative analyses to study chatbot design, trust reasoning, and design fidelity. Cohort differences are treated as exploratory because participant groups experienced different contexts and procedures.

4.1 Participants and Settings

  • 115 students participated across three age cohorts and workshop formats, using age-adapted versions of LUMI to specify chatbot behavior and iteratively test and revise responses.Elementary learners used ToyTalk; middle and high school students used LUMI.Source: §4.1 · pp. 4–5

4.2 Workshop Flow

  • Workshops followed a configure-test-evaluate-revise progression, with age-specific schedules and activities culminating in peer testing and chatbot refinement.Middle school students attended six 60-minute sessions over two days, while high school students attended five 90-minute sessions weekly over six weeks; elementary instruction ran over 13 days.Source: §4.2 · p. 7

4.3 Data Collection

  • Data collection combined surveys, interviews, system logs, worksheets, peer evaluations, and teacher reflections, with elementary students represented primarily by teacher reflections and platform logs.Interviews and surveys were not collected from elementary students to reduce developmental and classroom demands.Source: §4.3 · p. 7

4.4 Data Analysis

  • Analyses integrated statistical tests of surveys and logs with thematic coding, using exploratory cohort and chatbot-topic comparisons where data were sufficiently comparable.The team evaluated design fidelity on responsiveness, purpose alignment, rules alignment, and persona alignment; the rubric had 82.8% exact human–AI agreement and quadratic-weighted κ=0.88 across 232 criterion-level ratings.Source: §4.4 · pp. 7–8
  • Researchers developed trust-related codes deductively and inductively, refined them through consensus, and organized 18 codes into seven higher-level design dimensions.Two researchers independently coded an initial 50% subsample of chatbot-purpose and rules artifacts, reaching Cohen’s κ=1.00.Source: §4.4 · p. 8

5 Results

Students shaped chatbot trust through rules, communication traits, and other interacting design choices, with patterns varying by age and chatbot domain. Some younger students considered deliberately incorrect chatbots trustworthy when they fulfilled their intended purpose.

5.1 Overview: Workshop Impact on AI Attitudes

  • Workshop surveys showed increases across both age groups in AI knowledge, hallucination awareness, confidence building trustworthy chatbots, and AI ability beliefs (all p < .001).The section then examines how students translated these gains into chatbot design choices.Source: §5.1 · p. 9

5.2.1 What Students Built.

  • Academic chatbots commonly supported subject tutoring, especially mathematics, while middle schoolers explored more hobby sub-themes than high schoolers (17 vs. 9).Middle schoolers built 12 mathematics tutoring bots, compared with 3 among high schoolers; 10 of 34 middle school hobby bots were deliberately configured for incorrect or nonsensical responses, and some students still called them trustworthy because they fulfilled their purpose.Source: §5.2.1 · p. 9

5.2.2 Modification and Iteration Patterns.

  • Rules and Guidelines accounted for 31.1% of modifications, more than any other field, while 51–66% of bots had no revisions to each other field, including the trait sliders.Among 23 students building both types, academic bots received more total configurations than hobby bots (M = 8.77 vs. M = 4.67, p = .022), including more revisions to Rules and Guidelines and Communication Tone.Source: §5.2.2 · p. 9

5.2.3 Design of Communication Traits.

  • Elementary and middle school bots had higher Confidence settings than high school bots, while high school bots had higher Transparency and middle school bots lower Formality; Assertiveness did not differ significantly.Figure 5 depicts the age-group distributions at the individual-chatbot level.Source: §5.2.3 · p. 10, Figure 5 · p. 11

5.2.4 Domain Differences in Design.

  • Academic bots had higher Formality than hobby bots, and among students who built both types, academic bots also had higher Transparency; Confidence and Assertiveness did not differ.In an exploratory age-by-domain comparison, middle schoolers set higher Confidence than high schoolers for academic bots, while high schoolers set higher Formality for hobby bots; the authors treat this as preliminary given small, uneven subgroups.Source: §5.2.4 · p. 10

5.3 RQ2. What Design Choices Do Students Believe Make an AI Chatbot Trustworthy When Building Their Own Chatbots?

  • Students described trust as arising from multiple interacting choices, especially Response Style and Behavioral Boundaries such as rules, guidelines, and scope.Examples included combining confidence with assertiveness to make a bot more receptive to correction, and combining knowledge sources, scope restrictions, and model selection.Source: §5.3 · pp. 10–11

7. Error Handling

Student-created chatbots showed moderate design fidelity overall, with purpose alignment weakest and middle school hobby bots lowest across the criteria. Students also described trust through error handling, social cues, and alignment with a chatbot’s intended purpose.

7. Error Handling

  • Students described trustworthy bots as admitting errors, correcting mistakes, and not making up facts or hiding errors; this aligns with their frequent revisions to Rules.The authors connect students’ focus on rules with recognizing them as a primary mechanism for improving reliability.Source: §7 · p. 12
  • Students linked trust to personality and tone, noting that high assertiveness and formality could come across as rude, especially for younger students’ social judgments.The paper describes warmth and kindness as social cues younger students considered when judging trustworthiness.Source: §7 · p. 12
  • Younger students emphasized persona, tone, purpose fulfillment, and high Confidence, while older students emphasized Transparency, calibrated Confidence, and bots acknowledging their limits.Some middle schoolers called deliberately wrong bots trustworthy because they fulfilled their intended purpose.Source: §7 · pp. 12–13

5.4 RQ3. How Closely Did Student-created Chatbots Align with Their Intended Designs?

  • Chatbots’ mean design fidelity was 2.84 (SD = 0.42), slightly below the rubric threshold of 3, and 34% averaged at least 3 across the four criteria.Rules Alignment scored highest (M = 3.06), followed by Responsiveness (M = 3.00) and Persona Alignment (M = 2.86); Purpose Alignment was lowest (M = 2.47).Source: §5.4 · p. 13

5.4.1 Purpose Was the Least Faithfully Realized Design Dimension.

  • Only 45% of messages met Purpose Alignment, compared with 62–69% for the other criteria, and purpose alignment had the lowest mean in every cohort-domain group.The reported group means ranged from 2.30 for MS hobby bots to 2.79 for HS academic bots.Source: §5.4.1 · p. 13

5.4.2 Middle School Hobby Chatbots Showed the Lowest Design

  • MS hobby bots had the lowest composite fidelity (M = 2.56), fell below 3 on all four criteria, and scored significantly lower than each other group; the remaining groups did not significantly differ.Academic bots showed similar fidelity across MS and HS cohorts.Source: §5.4.2 · p. 13

6 Discussion

Children’s designs reveal that trustworthiness can mean faithful execution of intent even when answers are wrong, while students also reason about social credibility and calibrated transparency. The discussion reflects on how the design tool shaped what children could express, proposes three AI-literacy directions, and cautions that cohort differences are exploratory.

6.1 Rethinking Trustworthiness Through Children’s Design

  • Some middle schoolers called chatbots trustworthy for faithfully fulfilling their intended purpose even when deliberately designed to give wrong answers.For these students, trustworthiness did not necessarily mean factual accuracy; children who knew about hallucinations also sometimes configured high-confidence replies.Source: §6.1 · p. 13
  • Elementary students emphasized kindness and warmth, while high school students often emphasized transparency about limits and uncertainty.The discussion treats these as distinct dimensions of trustworthiness and recommends helping children distinguish intended behavior, social engagement, and warranted information.Source: §6.1 · p. 13

6.2 Methodological Reflection

  • LUMI made trust reasoning visible through trait controls, configuration fields, testing, version history, and peer feedback on mismatches between intent and experience.The tool may also constrain what children can express; for example, it had no field for privacy-related trust concerns.Source: §6.2 · p. 14

6.3 Design Implications

  • Tools could contrast a chatbot’s linguistic confidence with its accuracy, including through a visual cue for whether confidence matches accuracy.The proposal responds to students who configured highly confident bots despite learning that confident-sounding AI can be wrong.Source: §6.3 · p. 14
  • Tools could display design fidelity separately from factual accuracy because a chatbot can follow its rules while giving wrong answers, or break its rules while answering correctly.The authors say whether this distinction improves children’s trust judgments requires future study.Source: §6.3 · p. 14
  • Tools could let children compare a friendly but inaccurate chatbot with a cold but correct one to examine social credibility against reliable evidence.The authors propose this comparison as a way to prompt reflection, not as a tested intervention.Source: §6.3 · p. 14
  • The three proposed directions are starting points, and whether they change children’s reasoning requires a separate study.The authors also call for independent factual-accuracy measures and interaction logs, and ask whether their framework extends beyond chatbots to other AI systems.Source: §6.3 · p. 14

6.4 Limitations

  • Cohort differences are exploratory because settings, instruction, interfaces, and data collection differed, and elementary students were absent from the RQ2 trustworthiness analysis.Unequal, modest age-group samples and differing interview formats also constrain comparisons; domain findings are largely limited to middle school because high schoolers built one chatbot.Source: §6.4 · pp. 14–15

7 Conclusion

The study examines children’s trust reasoning while they design chatbots, identifying multiple interacting design dimensions and patterns across age and domain. Some children judged purpose fulfillment as trustworthy despite incorrect answers, while the age-related findings remain exploratory because age cannot be separated from context.

7 Conclusion

  • The analysis identified seven design dimensions and nineteen codes, with children often describing confidence, transparency, rules, and persona as interacting aspects of trustworthiness.Some children called a chatbot trustworthy because it fulfilled its intended purpose, even when deliberately designed to give wrong answers.Source: §7 · p. 15
  • Students ages 8–13 set higher confidence and emphasized social and affective trust, while students ages 14–18 set higher transparency and described trust as conditional on acknowledging limits.These age patterns are exploratory because age was coupled with differences in setting, schedule, and instructional content.Source: §7 · p. 15
  • Students ages 10–13 and 14–18 set academic chatbots to be more formal and transparent than hobby chatbots across configuration logs, design rationale, and self-reported trust.Source: §7 · p. 15
  • Middle school students’ trust exceeded their chatbots’ design fidelity, whereas high school students’ trust was more conditional and aligned more closely with higher-scoring chatbots.The authors interpret this as exploratory evidence that trust criteria may shift with age, but age cannot be separated from setting and instructional context.Source: §7 · p. 15

A Appendix: Pre/Post Survey Results

Pre-to-post surveys showed significant increases in four core constructs for both age groups; anthropomorphism increases were not significant after correction, and gains did not significantly differ between groups.

A Appendix: Pre/Post Survey Results

  • 5 constructs were measured: AI knowledge and understanding, hallucination awareness, confidence in creating trustworthy chatbots, ability beliefs, and anthropomorphism.The first four multi-item constructs had acceptable-to-good internal consistency; anthropomorphism was measured with one item.Source: A Appendix: Pre/Post Survey Results · p. 17
  • 4 core constructs increased significantly from pre- to post-survey in both groups, while MS–HS gain differences were nonsignificant (all p_BH>.68).The four constructs were AI understanding and knowledge, hallucination awareness, confidence in building trustworthy chatbots, and ability beliefs.Source: A Appendix: Pre/Post Survey Results · p. 17
  • Anthropomorphism increased in both groups but was not significant after correction (p_MS=.111; p_HS=.125).Source: A Appendix: Pre/Post Survey Results · p. 17

Appendix A: Pre- and Post-Survey Instrument

The appendix presents a five-point agreement survey spanning AI understanding, hallucination awareness, confidence in creating trustworthy chatbots, and ability beliefs about creating AI applications.

Appendix A: Pre- and Post-Survey Instrument

  • Students rated agreement from Strongly Disagree to Strongly Agree, with Not Sure as the midpoint response.Source: Appendix A · p. 17

Section 1: AI Understanding & Knowledge

Section 2: AI Hallucination Awareness

Section 3: Confidence in Creating Trustworthy Chatbots

Section 4: Ability Beliefs of Creating AI Applications

Section 5: AI Anthropomorphism

The pre–post materials measured AI-related constructs and knowledge, alongside students’ reflections on chatbot accuracy and their perceptions of the builder. Figure 6 reports significant improvement across constructs in both age groups.

Section 5: AI Anthropomorphism

  • The knowledge assessment covered generative AI, language-model text generation, limits on access to classroom context, hallucinations, and ways to reduce hallucination risk.It also asked how to verify important AI-generated claims before using them in a report.Source: Section 5: AI Anthropomorphism · pp. 17–18
  • All AI-related constructs improved significantly from pre to post in both age groups (p < .001).The assessment included an item about treating chatbots as if they have human feelings.Source: Section 5: AI Anthropomorphism · p. 17, Figure 6 · p. 18
  • The response-evaluation rubric scored four criteria on a 1–4 scale and defined composite scores as their mean.Source: Table 5 · p. 18
  • Open-ended pre–post reflections asked students how they would check chatbot answers, improve wrong responses, and describe what they learned about chatbots.Post-only questions asked whether the builder helped students understand hallucinations and become more careful about trusting AI-generated answers.Source: Section 5: AI Anthropomorphism · pp. 18–19

C Post-Study Focus Group Interview Protocol

The post-study focus groups used middle- and high-school protocols with the same core content and age-adapted wording. Prompts elicited students’ design reasoning, trust judgments, prior AI use, mental models, and suggestions for the builder.

C Post-Study Focus Group Interview Protocol

C.1 Middle School Interview Protocol

C.2 High School Interview Protocol

  • The high-school protocol included prompts about anthropomorphism and persona, whether students had compared models, and what happens when a chatbot is corrected.It also asked about prior AI use, changes since the workshop, and suggestions for the builder.Source: Part 2. Anthropomorphism, Persona, and… · p. 20

Related papers

Loading 2609.25244v5…