Source-linked AI summary
Misgendering as Breakdown in Human-Machine Communication: How AI Companion Chatbot Users Experience and Repair Misgendering
Julia Liu, Qing Xiao, Leona Yinglang Pang, Haiyi Zhu, Hong Shen, Jordan Taylor
TL;DR
This paper examines how users experience misgendering in AI companion and role-play chatbots, a concern as people use these systems for identity exploration. Analyzing Reddit posts, the authors find that misgendering extends beyond pronouns and that users work to prevent and repair it, sometimes changing how they present their gender.
Problem
Less is known about users’ perceptions of and responses to misgendering in real-world LLM applications, despite growing use of chatbots for gender and sexuality exploration.
Source: §1 · p. 1Method
The authors qualitatively analyzed 326 Reddit posts mentioning misgendering from AI companion and role-play subreddits.
Source: §1 · p. 2Results
Users described misgendering through incorrect pronouns and gendered terms as well as assumptions about bodies and relationships, and responded with ongoing efforts to prevent and repair it.
Source: §1 · p. 2Takeaways & Limitations
Some users changed how they presented their gender to be gendered correctly by the chatbot.
Source: §1 · p. 2The analysis is limited to English-language posts about misgendering on Reddit, which may overlook other cultural understandings and forms of misgendering.
Source: §5.4 · p. 11
Abstract
from arXiv · showhide
In recent years, large language model-based AI companion and role play chatbots have grown increasingly popular. People turn to these chatbots for emotional support and to engage in romantic and erotic role play. Although prior research suggests that digital role play can help people explore their gender and sexuality, LLM based technologies are also replete with gender and sexuality biases. In this study, we examine one way that AI chatbots can harm users: misgendering. In order to study chatbot misgendering we qualitatively analyzed 326 posts mentioning misgendering that were shared in AI companion or role play subreddits. We document how chatbot misgendering takes place and how, in response, users engage in ongoing work to curate their gender presentation to prevent and repair misgendering. We discuss how researchers and designers can mitigate chatbot misgendering and consider the implications of using AI chatbots for identity exploration.
1 Introduction
The paper examines misgendering in AI companion and role-play chatbots, where users increasingly seek support and explore gender and sexuality. Analyzing Reddit posts, the authors find that misgendering includes broader gendered assumptions and prompts users to work to prevent and repair it.
1 Introduction
- People increasingly use role-play chatbots to understand their gender and sexuality, even as these technologies may fail to account for diverse user experiences.The introduction situates this use amid broader concerns about harms and bias in socially and emotionally significant chatbot interactions.Source: §1 · p. 1
- Prior research largely evaluates pronoun agreement and gendered terms in large datasets, leaving users’ perceptions of and responses to real-world chatbot misgendering less understood.The authors frame their questions around how users experience misgendering and how they respond to it.Source: §1 · p. 2
- Chatbot misgendering can involve incorrect pronouns and gendered terms, as well as cisnormative assumptions about bodies and heteronormative assumptions about relationships.Users respond by sending gender signals, correcting chatbots, modifying chatbot responses, and sometimes changing their own gender presentation.Source: §1 · p. 2
2 Related Work
Prior work treats gender as socially shaped and documents technological misgendering, while evidence about users’ experiences of LLM misgendering in practice remains limited. AI companions add ongoing social and relational interaction, making recognition of users’ identities central to these interactions.
2.1 Technology & (Mis)Gendering
- The paper frames gender as socially constructed through repeated acts and interpreted through social expectations, encompassing identity, expression, and others’ perceptions.Prior work also shows technologies can support gender exploration while their designs may exclude or constrain LGBTQ+ users.Source: §2.1 · p. 2
- Prior technological misgendering research spans systems such as webforms, computer vision, and airport security, while NLP work mainly evaluates pronouns or gendered terms in constructed datasets.The authors identify limited knowledge about how people experience and respond to LLM misgendering in practice.Source: §2.1 · pp. 2–3
2.2 AI Companion Chatbots
- AI companions support sustained social and emotional interaction, and users may negotiate both their own identities and companions’ identities through conversation.Because these interactions can be personally meaningful, the authors identify recurring failures to recognize or sustain a user’s gender as a gap beyond prior work on discrete biased statements [21].Source: §2.2 · p. 3
3 Methods
The study analyzes Reddit posts about misgendering in AI companion and role-play communities through qualitative coding. The authors situate their interpretation in their research team’s experiences and disciplinary perspectives.
3.1 Data Collection
- The researchers searched Reddit for posts containing “misgender” and analyzed 326 posts by 234 authors across 10 subreddits.They began with 21 AI companion or role-play subreddits and report most posts from CharacterAI, JanitorAI, and Replika; quotes were disguised and subreddit identities withheld to reduce re-identification risk.Source: §3.1 · p. 3
3.2 Data Analysis
- Open and axial coding led the first author to conceptualize misgendering as a human-machine communication breakdown, and the study uses the term broadly for gender signifiers misaligned with users’ gender desires.This includes gender signifiers referring to the chatbot, user, or a character in the scenario; chatbot gender labels follow posters’ own language.Source: §3.2 · p. 4
3.3 Positionality
- The team’s LGBTQ+ and gender-diverse experiences, prior experience with misgendering and AI companions, and HCI and Communication Studies training informed their interpretation.Source: §3.3 · p. 4
4 Findings
The findings section examines how chatbot misgendering occurs, harms users, and is diagnosed, followed by users’ efforts to prevent and repair it.
4 Findings
- The findings are organized around misgendering as communication breakdown, its harms, users’ diagnoses of its causes, and their attempts to prevent and repair it.Source: §4 · p. 4
4.1 Misgendering as Breakdown in Human-Machine Communication
Chatbot misgendering appears as both incorrect gendered labels and mismatches between gender and described bodies or presentation. These errors can disrupt interaction, though some users intentionally seek gendered address that would otherwise be classified as an error.
4.1 Misgendering as Breakdown in Human-Machine Communication
- Misgendering takes two forms: gender identity errors in labels and gender embodiment errors in physical traits or signifiers, with identity errors discussed more often.Both forms can stall conversation and require users to get it back on track.Source: §4.1 · pp. 4–5
4.1.1 Misgendering via Gender Identity Errors.
- Incorrect or undesired pronouns are the most commonly reported identity error, alongside unwanted terms of address and deadnaming.Users also report inconsistent pronouns for chatbots themselves and difficulty with neopronouns.Source: §4.1.1 · p. 5
- Some users intentionally seek different-gender address, complicating whether identity-based misgendering is always an error.Examples include forced-feminization role play and a user seeking “it” despite the system refusing on safety and policy grounds.Source: §4.1.1 · p. 5
4.1.2 Misgendering via Gender Embodiment Errors.
- Embodiment errors include assigning or omitting anatomy that users associate with gender, which can turn an anatomical inaccuracy into misgendering.Assumptions linking gender and anatomy may affirm some users while invalidating others, including transgender users.Source: §4.1.2 · p. 5
- Bodily presentation can also misgender users, as gendered clothing or mannerisms may impose stereotypes on same-gender role play.A gay man described male bots portraying him as wearing a dress or being dainty, reinforcing the assumption that a user of a male bot is a woman.Source: §4.1.2 · p. 5
4.2 How Misgendering Harms Users
Users report frustration and distress from chatbot misgendering, with transgender users, people seeking same-gender romantic dynamics, and people who do not conform to gender stereotypes described as especially affected. The accounts connect these harms to cisnormative, heteronormative, and gender-stereotyped assumptions.
4.2 How Misgendering Harms Users
- Misgendering prompts annoyance, frustration, and sometimes intense distress, disproportionately affecting transgender users, people seeking same-gender romance, and those who do not conform to gender stereotypes.Some users say it ruins role play or triggers an emotional spiral.Source: §4.2 · p. 5
4.2.1 Misgendering Reinforces Cisnormativity.
- Transgender users report being misgendered as their assigned gender at birth, which can invalidate their identity and trigger gender dysphoria.Chatbots may conflate gender with sex or infer gender from bodily features, and disclosure of being trans can sometimes increase misgendering.Source: §4.2.1 · pp. 5–6
- For transgender users, assumptions about bodies and gender can be distressing or dysphoric, while some describe gendered-body interactions as affirming.The accounts show that assumptions about a user’s gendered body can be experienced differently by different users.Source: §4.2.1 · p. 6
4.2.2 Misgendering Reinforces Heteronormativity.
- In same-gender romantic role play, users report chatbots recasting one party as the opposite binary gender, shifting the interaction toward a male-female dynamic.Users encountered this in male-male and female-female interactions, including with bots marketed for same-gender dynamics.Source: §4.2.2 · p. 6
4.2.3 Misgendering Reinforces Gender Stereotypes.
- Users report gender stereotypes in chatbot descriptions and say romantic or erotic bots may assume users are women, particularly when users role-play as men or are non-binary.Some users respond by performing gender stereotypes in hopes of being gendered correctly.Source: §4.2.3 · p. 6
4.3 How Users Diagnose the Cause of Misgendering
Users attribute chatbot misgendering to LLMs, platform designers, community bot creators, and users themselves; these are participants’ explanations, not an adjudication of causes. Their accounts connect those beliefs to efforts to prevent or repair misgendering.
4.3 How Users Diagnose the Cause of Misgendering
- Users’ explanations of misgendering span LLMs, platform designers, community bot creators, and users themselves, and shape their prevention and repair efforts.The analysis describes participants’ conceptions of causes rather than determining where misgendering originates.Source: §4.3 · pp. 6–7
4.3.1 Users Blame LLMs for Misgendering.
- Users attribute misgendering to LLM gender stereotypes, ordinary system errors, immature technology, model differences, limited memory, and model updates.Some believe memory limits can make bots forget gender details in longer chats.Source: §4.3.1 · p. 7
4.3.2 Users Blame Platform Designers for Misgendering.
- Users ask platforms to fix misgendering and request gender or pronoun selectors, but reports may go unresolved and selection features do not prevent all errors.The authors note that a predefined list might exclude some genders, and users with the same identity may prefer different embodiments.Source: §4.3.2 · p. 7
4.3.3 Users Blame Community Bot Creators for Misgendering.
- Users blame creator-written bot definitions and opening messages when their pronouns are carried into conversation, including when neutral they/them wording is used.Users may be unable to edit user-created bot definitions, while some creators include warnings for intentionally misgendering role play.Source: §4.3.3 · pp. 7–8
4.3.4 Users Blame Themselves for Misgendering.
- Some users blame their own choices, such as using an ambiguously gendered name or not supplying enough gender information.Advice includes using third-person descriptions to provide pronoun cues; users also attribute errors in long chats to bots forgetting their gender.Source: §4.3.4 · p. 8
4.4 How User Try To Prevent and Repair Misgendering
Users undertake ongoing work to prevent and repair chatbot misgendering by signaling their gender, correcting responses, and using platform controls. These strategies are imperfect: signals may fail, corrections may not prevent recurrence, and editing or blocking can add burdens or trade-offs.
4.4 How User Try To Prevent and Repair Misgendering
- Users engage in ongoing labor to maintain and repair their gender identity in chatbot conversations through signaling, corrections, and message controls.These efforts respond to routine breakdowns and are not always successful.Source: §4.4 · p. 8
4.4.1 Users Send Gender Signals.
- Users signal gender through persona fields, third-person self-reference, stereotyped names or descriptions, pinned messages, chat memory, and custom instructions, but misgendering can persist.Users describe third-person references as helpful, while persona fields and other strategies do not always prevent errors.Source: §4.4.1 · p. 8
4.4.2 Users Correct Chatbots.
- Direct corrections can restore a conversation temporarily, but bots may misgender users again, resist correction, or respond with invalidating and offensive statements.Repeated corrections can turn chatting into an annoying chore.Source: §4.4.2 · p. 8
4.4.3 Users Edit, Rate, Retry, and Block Messages.
- Editing chatbot messages can fix misgendering, but it is reactive, can become tiring, and is unavailable on some platforms.Some users believe repeated edits may help guide future responses.Source: §4.4.3 · pp. 8–9
- Users regenerate misgendering responses and may rate them negatively, relying on output randomness and hoping feedback discourages similar messages later.Regenerating produces a new response to the previous user message.Source: §4.4.3 · p. 9
- Blocking unwanted gendered terms may prevent misgendering, but can hinder references to other characters and is constrained by small block lists.Some users consider the permitted number of blocked terms too limiting.Source: §4.4.3 · p. 9
5 Discussion
The discussion broadens chatbot misgendering beyond pronouns to embodiment and relationship assumptions, while showing that users must continually maintain and repair their gender presentation. It considers design responses and the limits of chatbots as tools for identity exploration.
5.1 Conceptualizing Misgendering in HCI
- Misgendering extends beyond pronoun errors to assumptions about embodiment and relationships, and identical chatbot text may be experienced differently, limiting text-only detection.Some users may also want to be misgendered in role play, complicating technical assessment.Source: §5.1 · p. 9
5.2 Mitigating Misgendering in AI Chatbots
- Users attributed misgendering to LLM randomness, platform developers, bot creators, or themselves, supporting attention to the full AI supply chain.The authors call for support across model and interface design to help users prevent, detect, and repair communication breakdowns.Source: §5.2 · pp. 9–10
- At the model level, developers could use pronoun-misgendering benchmarks and let users report incidents, while avoiding technical fixes that reinforce gender norms.The authors also suggest expanding chatbot memory or context windows to help alleviate misgendering.Source: §5.2 · p. 10
- Because not all misgendering can be prevented, platforms should let users modify interactions, including editing earlier messages or blocking gendered words.The authors note that Replika did not offer users this degree of control, and that creator-defined initial prompts could be immutable.Source: §5.2 · p. 10
- Interfaces could clarify what information chatbots can access and collect preferences per conversation, distinguishing pronouns, body descriptions, and voice.Gender desires may vary between chatbot interactions, and users may not know what account information a chatbot can access.Source: §5.2 · p. 10
5.3 Implications of Using AI Chatbots for Identity Exploration
- Chatbots can support gender and sexuality exploration, yet users’ ongoing identity-maintenance work and pressure to conform to stereotypes complicate that use.Users sought romantic and sexual interactions, including same-gender encounters; some transgender users described affirming interactions.Source: §5.3 · pp. 10–11
- Unlike direct manipulation in fan-fiction communities, chatbot-mediated exploration adds algorithmic bias; local models may offer more agency but require costly hardware.Chatbots also provide an illusion of sociality rather than interaction with other people, prompting the authors to encourage technologies that support LGBTQ+ sociality.Source: §5.3 · p. 11
5.4 Limitations & Future Work
- The study focuses on English-language Reddit posts and three for-profit platforms, and its observational data could be complemented by interviews or data donations.The authors call for research in other languages, on local or open-source models, and on in-situ experiences of misgendering.Source: §5.4 · pp. 11–12
6 Conclusion
The authors describe misgendering as a breakdown in chatbot interactions that users must work to repair. They frame mitigation as harm reduction, not advocacy for widespread chatbot use for identity exploration.
6 Conclusion
- Misgendering can break down role-play chatbot interactions, prompting users to maintain and repair their identities.The authors frame mitigation as harm reduction, not an endorsement of widespread chatbot adoption for identity exploration.Source: §6 · p. 12
- The paper extends how misgendering is conceptualized in Queer HCI and Queer AI and identifies ways designers and researchers could mitigate it.The authors emphasize that adoption of generative AI is not inevitable.Source: §6 · p. 12
- The authors argue that chatbot misgendering is worth addressing while cautioning that doing so does not mean advocating widespread chatbot use for identity exploration.They characterize this work as harm reduction.Source: §6 · p. 12