PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-Designing AI For Mental Health Support With Young Adults of Color (YOC): Needs, Expectations, and Implications for AI Literacy

Elaine Dabin Jeon, John Bosco S. Bunyi, Hannah Kim, Renkai Ma, Yaman Yu, Michal Luria, Jason Yip, Alexis Hiniker, Katie Davis, Angel Hsing-Chi Hwang

arXiv:2610.02812v1cs.HC

TL;DR

Young adults of color face identity-specific risks from AI systems and remain ambiguously positioned between youth safeguards and adult-oriented systems. This paper examines their chatbot use through a co-design workshop, finding that participants wanted more identity-informed disclosure rather than generic advice or assumption-making. The findings also identify knowledge gaps relevant to AI literacy education and support-system design.

  • Problem

    Research on AI-mediated mental health support has insufficiently centered cultural and identity contexts, while YOC may face harms when systems reproduce identity-specific biases, stereotypes, and stigma.

    Source: §1 · p. 2

  • Method

    A co-design workshop examined YOC participants’ chatbot conversations about stress, workload, parental pressure, interpersonal conflict, and immigration experience.

    Source: §3.2.2 · p. 9

  • Results

    Participants wanted support to recognize their identities and sociocultural contexts, with more opportunities for identity-informed disclosure to mitigate generic advice or assumption-making.

    Source: §5 · p. 18

  • Takeaways & Limitations

    The findings and proposed designs reveal knowledge gaps relevant to future AI literacy education and supporting-systems design.

    Source: §5 · p. 18

    Findings are contextual to a two-day workshop with 13 YOC in a racially and culturally diverse large U.S. metropolitan area and are not intended to represent all U.S. YOC perspectives.

    Source: §5.4 · p. 23

Abstract

from arXiv · show

Young adults of color (YOC) face heightened mental health challenges and barriers to care while navigating developmental and life transitions. Situated between youth-oriented safeguards and adult-oriented AI systems, little is known about how they use AI chatbots for mental health and well-being support or how sociocultural contexts shape their expectations, concerns, and design preferences. We conducted a two-day co-design workshop with 13 Asian, Black, and Hispanic/Latino/a young adults aged 18--24. Participants found generic chatbot advice to flatten their lived experiences; rather than making incorrect assumptions, they wanted more opportunities for identity-informed disclosure. Preferences for YOC-centered personalization also revealed gaps in privacy and AI literacy. Participants negotiated different therapeutic roles for chatbots and sought greater AI accountability and user agency, highlighting blurred boundaries between clinical and non-clinical AI-mediated support. Findings suggest directions for integrating AI literacy with mental health literacy and centering YOC's experiences in the privacy calculus.

1 Introduction

The paper examines how young adults of color (YOC), positioned between youth-oriented safeguards and adult-oriented AI systems, use and envision chatbots for mental health support. A co-design workshop surfaces desires for culturally responsive support and identity-informed disclosure alongside concerns about privacy, AI literacy, and chatbot roles.

1 Introduction

  • YOC face heightened stressors and structural barriers to appropriate care, while recent data indicate comparatively higher chatbot use for mental health advice among young adults ages 18–21.The paper frames these needs amid emerging adulthood’s identity exploration and major life transitions that can reconfigure social support systems.Source: §1 · pp. 1–2
  • Research has insufficiently centered cultural and identity contexts in AI-mediated mental health support, despite risks from systems reproducing YOC-specific biases, stereotypes, and stigma [34] [79] [83].Cultural backgrounds can also shape attitudes toward conversational agents [36].Source: §1 · p. 2
  • Young adults occupy an ambiguous position: often treated as autonomous adults despite ongoing development and mental health needs, while youth-oriented safeguards primarily frame young people as minors.The paper argues that YOC’s culturally situated knowledge and lived experiences are essential for understanding their chatbot use.Source: §1 · p. 2
  • A two-day, in-person co-design workshop with 13 Asian, Black, and Hispanic or Latino/a young adults combined reflection, chatbot conversations, and iterative design activities.Activities elicited personal and contextual knowledge as well as participants’ understanding and uncertainties about conversational AI.Source: §1 · p. 2
  • Participants found generic advice flattening and less relevant to culturally situated challenges, and preferred intentional identity-informed disclosure over systems filling gaps with assumptions.They rarely discussed the privacy implications of disclosing more personal background information.Source: §1 · pp. 2–3
  • Participants wanted clear communication about chatbot capabilities and limits, culturally resonant therapeutic roles, stronger accountability and user agency, and more integrated AI and mental health literacy.They struggled to agree on AI’s role in formal mental health support and to understand system pushbacks and safeguards.Source: §1 · p. 3
  • The paper contributes an empirical account, design principles for culturally responsive engagement, and a framework connecting AI literacy, mental health literacy, and privacy.The framework extends beyond technical knowledge or prompting to include navigating structural affordances and relational dynamics, assessing harms, and recognizing culturally situated contexts.Source: §1 · p. 3

2 Background and Related Work

This section reviews prior work on AI chatbots for mental health support, focusing on gaps in understanding young adults of color’s experiences, developmentally informed safety, and AI literacy. It also addresses limited integration of mental health support and AI literacy in research that involves young adults as design contributors and attends to demographic factors.

2.1 Young Adults’ AI Chatbot Use for Mental Health Support and Its Limitations

  • Chatbot use is evolving faster than understanding of its appropriateness and effectiveness, making it important to translate users’ experiences into technology design and evaluation.Young adults’ wide-ranging perspectives and practices, especially those of YOC, warrant further exploration.Source: §2.1 · p. 3

2.1.1 Use of Conversational AI for Mental Health and Well-Being Support.

  • YOC report frequent chatbot use for mental health support, but their practices and culturally situated expectations remain poorly understood.Cultural contexts can shape attitudes toward AI and expectations such as autonomy and relational connection.Source: §2.1.1 · p. 4

2.1.2 Emerging Research on Limitations of AI-Mediated Support.

  • General-purpose chatbots are not clinically validated for therapeutic care, and young adults remain understudied in AI safety despite distinctive developmental circumstances and risks of racial bias.Existing interventions and safeguards have rarely been designed with or rigorously evaluated among young adults, particularly YOC.Source: §2.1.2 · p. 4

2.2 Approaches to Supporting AI Use for Mental Health among Young Adults

  • Supporting young adults’ mindful and responsible use of conversational AI for mental health calls for attention to AI literacy and participatory design.Prior work has pursued both AI literacy frameworks and programs and participatory design methods.Source: §2.2 · p. 5

2.2.1 Existing AI Literacy Frameworks for Young Adults.

  • Existing AI literacy education may not equip young adults for emotionally consequential chatbot use or for recognizing interactional risks such as dependence and escalating reliance.Education has mainly focused on K–12 and higher education, while young adults’ access and needs vary.Source: §2.2.1 · p. 5

2.2.2 Participatory Design Methods for AI Literacy and Young Adults.

  • Participatory design foregrounds situated knowledge in AI literacy and fairness work, with emerging applications to teens and young adults’ digital mental health needs.The supplied passage describes participatory approaches in AI fairness and critical AI literacy, alongside co-design in digital mental health.Source: §2.2.2 · p. 6
  • Research rarely brings technology-mediated mental health support and AI literacy together by engaging young adults as active design contributors.Source: §2.2.2 · p. 6
  • Digital mental health research involving young adults has given limited attention to factors such as ethnicity and socioeconomic background.Source: §2.2.2 · p. 6
  • The literature calls for moving beyond designing for young adults toward designing with them through more empathy-driven design.Building on this work, the study takes a participatory design approach.Source: §2.2.2 · p. 6

3 Methods

The study used a two-day co-design workshop with young adults of color to examine chatbot use and collaboratively identify needs, concerns, and design directions. It combined reflective and storyboarding activities with inductive analysis of workshop materials and pre- and post-workshop survey responses, attending to psychological safety, AI literacy, and the influence of workshop context on interpretation.

3 Methods

  • A two-day, in-person co-design workshop engaged 13 Asian, Black, and Hispanic/Latino/a young adults aged 18–24 as co-constituents of mental health and technology research.Five facilitators supported the workshop.Source: §3 · p. 6

3.1 Participants

  • General chatbot use was daily or almost daily for 7 participants (53.8%) and weekly for 6 (46.2%), while mental health-related use was less frequent.For mental health support, 1 participant (7.7%) reported daily or almost-daily use, 3 (23.1%) weekly use, and 5 (38.4%) monthly use.Source: §3.1 · p. 7

3.2 Day 1 – Recognize and Analyze: Reflecting on AI Chatbot Use for Mental Health

  • The workshop adapted Solyst et al. (2025)’s RAD framework—Recognize potential AI harms, Analyze their sociotechnical dimensions, and Deliberate recommendations for improvement —and postponed information on problematic AI behaviors until Day 2 to avoid prematurely framing interpretations.Source: §3.2 · pp. 7–8

3.2.1 Warm-up and Group Reflection on Ideal AI Chatbot for Mental Health.

  • Participants defined mental health and well-being broadly, then used an art-based activity to represent desired chatbot values that they revisited in later design sessions.Definitions included mood regulation, stable emotions, low distress, inner peace, self-confidence, life quality, and holistic health; group examples included a garden, a library with a lighthouse, and a scale.Source: §3.2.1 · p. 8

3.2.2 AI Chatbot Conversations, Journaling, and Group Discussion.

  • Participants held two 15-minute ChatGPT conversations individually or in pairs, exploring prompts for another young person of color and scenarios involving chatbot pushback, then journaled and discussed their observations.Topics included social anxiety, workload stress, parental pressure, interpersonal conflict, and immigration experience.Source: §3.2.2 · p. 9

3.3 Day 2 – Analyze and Deliberate: Translating Gaps into Design Solutions

  • Day 2 focused on analyzing gaps and potential harms, ideating desired chatbot behaviors and features, and surfacing knowledge and behavioral gaps relevant to AI literacy.The sessions addressed the study’s second and third research questions on desired design and AI literacy implications.Source: §3.3 · p. 9

3.3.1 Wishlist Brainstorming and Taxonomy of Problematic AI Behaviors.

  • Participants brainstormed a wishlist of culturally relevant chatbot behaviors and used storyboarding and low-fidelity prototyping to develop design ideas.They first defined mindful and culturally relevant support in their own terms and identified pain points.Source: §3.3.1 · p. 9, §3.3.2 · p. 10
  • Participants reviewed a taxonomy of potentially problematic chatbot behaviors developed from APA guidance and literature and refined with expert and young-adult input.The refinement prioritized observable interactional behaviors and structural factors relevant to mental and emotional support and to YOC.Source: §3.3.1 · pp. 9–10

3.3.2 Co-design of AI Behaviors and Solutions.

  • Four teams turned wishlist behaviors and self-selected scenarios into goal-centered storyboards and low-fidelity mock-ups, then presented them for peer feedback.A design consultant supported visualization, using generative AI tools when appropriate.Source: §3.3.2 · p. 10

3.4 Data Collection and Analysis

  • Workshop data included facilitator audio recordings and field notes, individual warm-up responses, journaling, donated in-session chat logs, and survey responses.Source: §3.4 · p. 10
  • Two authors inductively generated and iteratively refined themes using Braun and Clarke’s six-phase process, meeting weekly to discuss emerging findings.They critically examined interpretations with the broader research team to mitigate potential biases.Source: §3.4 · pp. 10–11
  • The researchers treated participant accounts as workshop-situated rather than exhaustive, since research questions, activities, and prompts shaped what participants discussed.Source: §3.4 · p. 10
  • The authors acknowledged that their lived experiences and professional and educational backgrounds could shape interpretation, and provided a positionality statement in Appendix D.Source: §3.4 · p. 11
  • Pre- and post-participation surveys measured AI attitudes and literacy, with responses from 12 participants who completed both workshop days reported in Appendix E.Source: §3.4 · p. 11

4 Findings

Participants said generic chatbot advice and incorrect assumptions flattened YOC’s lived experiences by overlooking sociocultural contexts. They wanted identity-informed disclosure and more context-sensitive support.

4.1.1 AI’s Generic Advice and Incorrect Assumptions for YOC-Specific Needs (RQ1).

  • During the conversation exercise, broad encouragement did not address a Black participant’s social anxiety connected to past trauma from minority experiences at school.The chatbot advised aiming for connection rather than perfection without asking clarifying questions.Source: §4.1.1 · p. 12
  • Participants wanted chatbots to ask about identity and context rather than assume circumstances or offer one-size-fits-all advice.They described missing context or clarification as a lack of empathy and said neutral responses could feel uninformed in cultural and intersectional situations.Source: §4.1.1 · p. 12
  • Participants found chatbot answers could offer comfort and next steps but missed complex cultural experiences and reduced distinct challenges to typical young-adult problems.Some considered generic advice suitable for temporary relief, not long-term support.Source: §4.1.1 · p. 12
  • Advice to prioritize self-care or normalize distress as an individual problem overlooked family obligations, mental-health stigma, collectivism, and systemic barriers.Participants described these factors as embedded in their experiences, including academic pressure tied to parents’ sacrifices and stigma in Asian households.Source: §4.1.1 · p. 12

away from the lived experience of being POC.”

Participants said chatbot advice about a conflict with a white friend’s anti-immigration views missed the political and identity contexts of the disagreement.

away from the lived experience of being POC.”

  • In a conflict involving a white friend’s anti-immigration views, the chatbot advised participants they could stay quiet and avoid every political discussion.It framed political opinions as separate from judgments about the user.Source: away from the lived experience of… · pp. 12–13
  • Participants described the response as cold and generic because it missed the broader politicized context of their stressors.P12 also disagreed with the chatbot’s framing of politics and character.Source: away from the lived experience of… · pp. 12–13

think political views can shine light on someone’s true character.”

Participants said attempts to tailor chatbot advice sometimes relied on assumptions about their backgrounds and experiences.

think political views can shine light on someone’s true character.”

is easy to talk about.”

Participants wanted intentional identity-informed disclosure to shape chatbot responses over time, but their interest in personalization coexisted with limited privacy discussion.

is easy to talk about.”

  • ChatGPT conflated an Asian international student’s circumstances with immigrant experiences, diminishing the relevance of its advice about stress and relationships.The participant had initially shared age, gender, and country of origin.Source: is easy to talk about.” · p. 13

4.1.2 YOC Seeks More Intentional, Identity-Informed Follow-up Questions (RQ2).

  • Participants proposed structured opportunities to share identity, background, and personal history so chatbots could retain context across future conversations.One design used a pop-up questionnaire at the start of interaction, resurfaced periodically, and saved demographic and other personal information.Source: §4.1.2 · p. 14, Figure 6 · p. 14
  • They wanted chatbots to adapt responses to lived contexts while making disclosure low-friction, with ways to update changing circumstances and state of mind.Participants also wanted chatbots to recall triggers and avoid repeatedly asking users to clarify details or correct assumptions.Source: §4.1.2 · pp. 13–14

4.1.3 Limited Privacy Discourse Amid Identity-Sensitive Disclosure (RQ3).

  • Despite willingness to share identity-related information, privacy concerns were limited during the workshop and some participants later recognized that sensitive inputs could be stored or used for training.One participant said this new awareness would prompt more care before sharing, without decreasing overall AI use.Source: §4.1.3 · pp. 14–15

AI as a tool I control, not a diary I trust.” (P06)

Participants valued chatbots as less socially consequential places to seek support, but their privacy concerns focused on visible leaks rather than risks from accumulated disclosures. They wanted culturally responsive support with clear role boundaries, while retaining agency and expecting accountability from chatbots.

AI as a tool I control, not a diary I trust.” (P06)

  • Participants framed privacy mainly as gossip or other visible leaks, not cumulative disclosures that could enable increasingly detailed inferences.They valued long-term memory for continuity and personalization but did not discuss how retained information could shape privacy risks over repeated interactions.Source: AI as a tool I control, not a diary I… · p. 15
  • Perceived confidentiality made chatbots appealing for sensitive topics, even as their advice could conflict with users’ sociocultural contexts or reinforce stereotypes.Participants described chatbot support as less stressful and consequential than talking to friends or other people.Source: AI as a tool I control, not a diary I… · p. 15

4.2.1 YOC Seek AI Role Clarification and Guided Critical Thinking (RQ1).

  • Participants wanted chatbots to clarify limitations from the outset, since they could offer professional-seeming support before abruptly redirecting users to crisis resources.They called the mismatch an “accountability gap” and wanted a disclaimer that would help users redirect the conversation.Source: §4.2.1 · p. 15
  • Participants envisioned culturally resonant, role-specific support within one system, but rejected chatbots acting as friends or family.Examples included an “Asian American, Christian, or a queer therapist” and support grounded in cultural norms.Source: §4.2.1 · pp. 15–16
  • Participants wanted chatbots to guide critical thinking while leaving users ownership of decisions and autonomy to explore possible solutions.Their “stone path” metaphor depicts guidance toward insight without taking control from the user.Source: §4.2.1 · p. 16, Figure 7 · p. 16

4.2.2 YOC Seek Greater User Agency and AI Accountability (RQ2).

  • Participants proposed conversational flows that help users think through situations, choose alternate paths, and receive risk-tailored guidance, alongside reminders to take breaks.They also wanted systems to communicate limitations and redirect users to appropriate resources.Source: §4.2.2 · p. 16

group: “You should encourage people to have real-life

Participants designed ways to extend user agency beyond a single chatbot conversation, including reflection, behavioral nudges, and engagement with real-life support. Their discussions also exposed unresolved expectations about chatbot roles, pushback, and when human care is needed.

conversations about how they use AI for mental health

  • Participants designed nudges and opportunities for reflection that keep users in control and extend engagement beyond a single conversation, including sharing AI use with friends.One proposal paired a user report with weekly missions to strengthen support systems.Source: conversations about how they use AI… · p. 17, Figure 8 · p. 17

4.2.3 YOC-proposed AI’s Roles Reveal Knowledge Gaps in Clinical Boundaries (RQ3).

  • Participants disagreed about chatbot care: some favored imperfect harm-reduction support when traditional care was inaccessible, while others wanted prompt referral to humans.The harm-reduction view acknowledged that chatbot care could not address root causes.Source: §4.2.3 · p. 17
  • Some participants viewed pushback as reserved for illegal requests and described rewording or restarting conversations to obtain desired answers.These accounts suggest they did not necessarily treat a chatbot boundary as definitive.Source: §4.2.3 · p. 18
  • Participants lacked a shared mental model of chatbot roles and safeguards, despite recognizing that chatbots are not licensed therapists.Some expected counselor-like, objective answers, while others described AI as cooperative and did not know how to prompt pushback.Source: §4.2.3 · p. 18

5 Discussion

The discussion examines how YOC participants’ desire for identity- and context-aware support intersects with privacy, AI literacy, user agency, accountability, and boundaries between clinical and non-clinical support. It also considers the workshop’s limitations and outlines directions for participatory research and future study designs.

5 Discussion

  • YOC wanted identity-informed disclosure to counter generic advice and assumptions, while struggling to locate chatbots’ therapeutic roles and seeking user agency alongside AI accountability.These tensions point to knowledge gaps that AI literacy education and system design should address in mental health contexts.Source: §5 · p. 18

5.1 Negotiating the Fine Line between Privacy and Personalization

  • YOC’s desire for personalized mental health support was in tension with their understanding of privacy in human–AI interaction.Participants expressed willingness to share personal information for culturally relevant support, even as privacy concerns remained.Source: §5.1 · p. 18, §5.1.1 · p. 19

5.1.1 Theoretical Implication: Privacy Calculus is a Tilted Scale for YOC.

  • YOC’s privacy calculus was tilted: disclosure could invite generic responses or biased identity inferences, while participants often focused privacy concerns on interpersonal rather than data lifecycle risks.Participants sought control over what information systems solicit and use, but sharing identity-relevant context did not reliably deliver culturally grounded support.Source: §5.1.1 · p. 19

5.1.2 Design Implication: Supporting Mindful Privacy Calculus by Enhancing AI Literacy.

  • Systems should support mindful disclosure by helping users share relevant sociocultural context without unnecessary identifiers, rather than simply discouraging disclosure.Proposed supports include explaining why information is requested, offering disclosure alternatives, surfacing risks to third parties, and showing how user-provided context informed a response.Source: §5.1.2 · pp. 19–20

5.1.3 AI Literacy Implication.

  • AI literacy should help YOC understand that sharing more information does not guarantee personalization and that personal disclosures can affect responses unpredictably.Participants were surprised that temporary chats produced responses similar to personal accounts; the discussion also identifies prompt sensitivity and shortcut-based inferences as literacy concerns.Source: §5.1.3 · p. 20

5.1.3 AI Literacy Implication

  • AI literacy should help YOC decide whether and what to disclose, reflect on what information AI needs, and develop a critical mental model of disclosure, personalization, and privacy alongside mental-health literacy.Source: §5.1.3 · p. 21

5.2 Bridging AI and Mental Health Literacy among YOC

  • Participants’ clinical-like expectations of consumer chatbots, including diagnosis-like judgments and “objective” answers, exposed uncertainty about AI and therapeutic roles.The authors note that such functions exceed the appropriate scope of consumer-grade AI, while therapists may support clients through exploratory and collaborative processes.Source: §5.2 · p. 21

5.2.1 An Integrated Conceptual Model: AI x Mental Health Literacy

  • The proposed model pairs understanding AI capabilities and limits with mental-health help-seeking, then extends AI literacy beyond skills to navigating relational cues and structural conditions.It also calls for supporting self-efficacy in self-help and seeking professional help.Source: §5.2.1 · p. 21

5.2.2 AI Literacy Implication

  • AI nutrition labels should state professional-role red lines, distinguish technical limits from role or regulatory constraints, explain support scope, and clarify capability versus permission.Making these boundaries and their rationale legible can help manage expectations and connect AI literacy with mental-health literacy.Source: §5.2.2 · p. 22

5.2.3 Design Implication: Designing for Proactive Role Clarification

  • Because users struggled when supportive systems later pushed back and could read delayed boundaries as accountability gaps, designers should clarify roles proactively, maintain consistency, and co-construct support norms.The paper recommends communicating boundaries based on a tiered taxonomy of mental-health and well-being risks.Source: §5.2.3 · pp. 22–23

5.3 Engaging YOC to Actively Shape Human-AI Interactions

  • Co-design surfaced YOC’s knowledge and behavioral gaps in a psychologically safe peer environment; future studies can combine chat-log and diary data [44] with young-adult-centered participatory methods to normalize taboo mental-health chatbot use.Source: §5.3 · p. 23

5.4 Limitations

  • Findings reflect 13 YOC in one U.S. metropolitan area, not all YOC perspectives; the eight-hour workshop compressed material for participants with varying familiarity, warranting refinement of pacing and preparation.Source: §5.4 · pp. 23–24
  • In-session chatbots had limited access to prior conversational contexts, motivating research on YOC’s perceptions and practices with personal accounts using ethically appropriate methods for sensitive interactions.Participants who regularly used ChatGPT personally did not note noticeable differences during the workshop activities.Source: §5.4 · p. 24
  • Facilitator prompts, examples, and responses may have shaped discussion and encouraged socially desirable responses; future work could test peer facilitation or other ways to reduce session power dynamics.Source: §5.4 · p. 24

6 Conclusion

The conclusion calls for YOC-centered support that combines AI and mental-health literacy, co-constructed safety, and connection to broader care. It also frames participatory research as a way to surface YOC’s design priorities and knowledge gaps.

6 Conclusion

  • Responsible support requires addressing AI design, AI literacy, mental-health literacy, and YOC’s broader care ecosystems together.The conclusion calls for treating YOC as situated experts and active partners in defining how AI can participate in their lives.Source: §6 · pp. 24–25
  • AI literacy efforts should help YOC distinguish technical limitations, institutional or regulatory boundaries, and personalization promises, while reflecting on their own well-being goals.Participants often discovered what they wanted from AI as conversations unfolded, while follow-up questions could steer interactions in unanticipated directions.Source: §6 · p. 24
  • Safety should be co-constructed from the outset, with YOC setting expectations and boundaries and interventions helping them notice drift, excess disclosure, or displaced support.The conclusion contrasts this approach with safeguards that intervene only after sensitive information is disclosed.Source: §6 · p. 24
  • Chatbot use should be situated within broader care networks, complementing other support rather than remaining a private dyadic or solitary path.The conclusion names peers as one possible connection for future designs.Source: §6 · pp. 24–25

A Journaling Prompts

  • The journaling prompts asked participants to review their ChatGPT conversations for notable behavior, usefulness, cultural relevance, and concerns.A second prompt asked how participants tried to get ChatGPT to cooperate and whether that approach succeeded.Source: A Journaling Prompts · p. 31

B Group Discussion Questions

  • Group discussion questions revisited usefulness, identity relevance, concerns, and desired behavior changes before design sessions.Participants were asked to give specific examples where possible.Source: B Group Discussion Questions · p. 31

C Taxonomy of Problematic or Inappropriate AI Behaviors

The taxonomy groups potentially problematic chatbot behaviors and designs relevant to mental-health support, including engagement, personalization, human-like cues, and factual or clinical problems. The positionality statement describes the research team’s perspectives and limits on how fully they represent participants’ situated knowledge.

C Taxonomy of Problematic or Inappropriate AI Behaviors

  • The taxonomy covers engagement-seeking design, hyper-personalization, and human-like cues that may encourage users to treat a chatbot as human-like.Examples include autoplay or infinite scroll, tailoring cues to inferred loneliness, and claims that the chatbot understands or has memories.Source: Table 2 · p. 31, C Taxonomy of Problematic or… · p. 32
  • Other listed behaviors include factual and behavioral inconsistency, biased or toxic content, relational transgression, and over-validation that may affirm inappropriate or ethically ambiguous behavior.The taxonomy also identifies limited clinical capabilities and misleading claims about capabilities such as stress management or medical advice.Source: C Taxonomy of Problematic or… · pp. 32–33
  • Privacy baiting describes prompting or coercing users to share more information than needed, or implying privacy despite possible company access to data.The taxonomy gives the example of assurances that information stays between chatbot and user.Source: C Taxonomy of Problematic or… · p. 33

D Positionality Statement

  • The authors acknowledge that their identities, training, and prior experiences shaped the research process, and that their experiences do not fully represent participants’ situated knowledge.The team describes iterative discussion of interpretations and retaining relevant participant quotations to foreground participant perspectives.Source: D Positionality Statement · pp. 33–34

E Pre- and Post-Participation Survey Data

The pre- and post-participation survey measures covered ChatGPT literacy, trust, perceived conversation quality, general attitudes toward AI, and AI self-disclosure. The supplied passages describe the measures and sample size but do not report survey results.

E Pre- and Post-Participation Survey Data

  • The descriptive-statistics table covers 12 participants who completed both workshop days, and higher scores indicate more of the named construct.Values are rounded to two decimal places.Source: Table 3 · p. 34, E Pre- and Post-Participation Survey… · p. 34
  • Other measures asked about trust in chatbot advice, conversation quality compared with real-life friends, general attitudes toward AI, and breadth and depth of self-disclosure.Post-participation self-disclosure questions were rephrased to ask about intended future use.Source: E Pre- and Post-Participation Survey… · p. 34
  • ChatGPT literacy was measured with 10 questions across technical proficiency, critical evaluation, communication proficiency, creative application, and ethical competence.The scale was selected for ChatGPT, the AI system used in the workshop.Source: E Pre- and Post-Participation Survey… · p. 34

F Co-design Outputs from Day 2

The co-design outputs describe features organized around identity and context, structured interaction, user agency, mindful engagement, and accountability. Proposed designs include personalization, opt-outs, usage awareness, redirection, and multiple chatbot roles.

F Co-design Outputs from Day 2

  • Identity- and context-responsive designs included onboarding to collect background information and retain users’ triggers and sensitive topics for future conversations.The table also lists curated questions in a survey-like format to reduce fatigue and intrusiveness.Source: Table 4 · p. 34, F Co-design Outputs from Day 2 · p. 35
  • Structured, low-friction and agency-focused features included curated questions, clear opt-outs, and adaptive re-prompting when users later disclosed more personal information.These features were paired with mindful, longitudinal engagement as a design principle.Source: F Co-design Outputs from Day 2 · p. 35
  • Harm-reduction and engagement ideas included usage summaries, early risk identification, non-intrusive follow-up, role-based sub-chatbots, and prompts to reconnect socially.Role-based sub-chatbots were proposed with different topic-based roles and confidentiality maintained across them.Source: F Co-design Outputs from Day 2 · pp. 35–36
  • Mutual-accountability designs paired a chatbot that acknowledges its capabilities and limitations with user-controlled redirection, timers, and periodic nudges.The designs also included self-reflection features such as mood visualizations and usage archetypes for peer discussion.Source: F Co-design Outputs from Day 2 · p. 36
Loading 2610.02812v1…