Source-linked AI summary
How Does LGBTQIA+ Identity Affect LLM Behavior? Implications for Requirements Engineering of Mental Health AI Systems
Shailyn Callihoo, Karman Singh, Navreet Dhillon, Harkiran Saini, Brody Stuart Verner, Ronnie de Souza Santos
TL;DR
LLM use in mental health raises fairness concerns for LGBTQIA+ users, yet evidence about how explicit identity disclosure affects generated responses remains limited. This study compares ChatGPT responses to 50 Counsel Chat questions under three identity conditions using repeated generation, binary coding, and qualitative comparison. Identity disclosure did not reduce completeness or practical guidance, but LGBTQIA+-explicit prompts produced more contextualization, unsupported assumptions, and occasional stereotyping, motivating broader fairness requirements while leaving methodological validity constraints.
Problem
Limited empirical work has examined how explicit LGBTQIA+ identity disclosure influences ChatGPT responses in mental health contexts, where fairness matters for vulnerable populations.
Method
The study compares ChatGPT responses to 50 Counsel Chat questions under neutral, straight-explicit, and LGBTQIA+-explicit conditions using repeated generation, binary coding, and qualitative comparison.
Results
LGBTQIA+ identity disclosure did not compromise response completeness or practical guidance but altered contextualization through asymmetric identity handling, unsupported assumptions, and occasional stereotyping.
Takeaways & Limitations
Fairness evaluation for mental health conversational AI should address identity handling, contextual interpretation, explanatory consistency, and unsupported inferences beyond overtly harmful outputs.
Takeaways & Limitations
Subjective coding and classifying a question-condition pair as positive when behavior appeared in any one of three runs may introduce interpretive judgment and inflate frequency estimates.
Abstract
from arXiv · showhide
Large Language Models are now part of healthcare and mental health support systems, raising concerns regarding fairness toward vulnerable populations, including LGBTQIA+ individuals. However, limited empirical work has investigated how explicit LGBTQIA+ identity disclosure influences LLM-generated responses in mental health contexts. In this study, we extracted 50 real mental health questions from the Counsel Chat repository and constructed three prompt conditions for each question: no identity disclosure, explicit straight identity disclosure, and explicit LGBTQIA+ identity disclosure. We generated and analyzed 450 ChatGPT responses across these conditions using binary coding and comparative analysis. Our findings indicate that LGBTQIA+ identity disclosure did not substantially affect response completeness or supportive guidance. However, responses in the LGBTQIA+-explicit condition presented substantially more identity acknowledgment, contextual expansion, unsupported assumptions, and occasional stereotypical reasoning compared to both other conditions. These results suggest that fairness-related concerns in conversational AI systems may emerge through subtle differences in contextual interpretation and explanatory reasoning rather than through overtly harmful outputs. We discuss implications for fairness requirements and the development of LLM-based mental health support systems.
I. INTRODUCTION
LLMs are increasingly used in healthcare and mental health support, but their fairness toward LGBTQIA+ individuals remains insufficiently studied. This study examines whether explicit identity disclosure changes ChatGPT’s responses in mental health conversations and informs fairness requirements for such systems.
- LLMs support healthcare and mental health applications including patient communication, psychoeducation, symptom screening, emotional support, and counseling assistance.
- Healthcare and mental health deployments raise concerns about hallucinations, harmful recommendations, misinformation, emotional dependence, and unequal treatment.
- LGBTQIA+ individuals face discrimination, stigma, isolation, and minority stress, while internet-trained systems may reproduce stereotypes or uneven interaction patterns.
- Limited empirical work has examined how ChatGPT behaves in mental health conversations involving explicit LGBTQIA+ identity disclosure.
- The study compares responses to neutral, straight-explicit, and LGBTQIA+-explicit prompts to investigate identity-related variation and fairness implications.
III. METHODOLOGY
The study uses 50 naturally occurring mental health questions from the Counsel Chat dataset, selected and reviewed through a structured preparation process. The methodology emphasizes topical relevance, cleaning, random sampling, and collaborative validation.
- The methodology uses 50 mental health-related questions from the publicly available Counsel Chat dataset.Counsel Chat contains user-submitted questions and responses from licensed therapists and mental health professionals.
- Counsel Chat provides naturally occurring, informal conversations involving requests for psychological guidance in emotionally sensitive situations.
- The researchers reviewed the dataset for duplicates, empty fields, incomplete records, and non-relevant content before selection.
- Questions focused on depression, anxiety, self-esteem, and relationship dissolution to represent emotionally vulnerable advice-seeking situations.
- The final 50 questions were randomly sampled and collectively checked for clarity, topical relevance, and duplicate scenarios.
B. Phase 2: Dataset Construction and Identity Manipulation
Each selected question was converted into three closely matched prompt conditions differing only in explicit identity disclosure. This design isolates how neutral, straight, and LGBTQIA+ identity information relates to generated responses.
- Dataset Construction and Identity Manipulation: Each question was represented in neutral, straight-explicit, and LGBTQIA+-explicit versions to isolate identity disclosure effects.
- Dataset Construction and Identity Manipulation: The original condition preserved the dataset wording without explicit identity disclosure.
- Dataset Construction and Identity Manipulation: Straight-explicit and LGBTQIA+-explicit conditions added the corresponding identity information while preserving the original wording and emotional context.
- Dataset Construction and Identity Manipulation: Identity disclosure was introduced through short contextual statements prepended to each question.
- Dataset Construction and Identity Manipulation: The LGBTQIA+ conditions represented gay, lesbian, bisexual, queer, and transgender identities.
- Dataset Construction and Identity Manipulation: Researchers independently drafted modified prompts, compared them in pairs, and used group validation to improve consistency and reduce individual bias.
C. Phase 3: Response Generation
Responses were collected in independent ChatGPT sessions for each question and identity condition, then analyzed through consolidated binary coding and comparative interpretation. The analysis captures both explicit behaviors and broader contextual patterns associated with identity disclosure.
- Response Generation: Responses were generated using unchanged prompts in new chat sessions across all questions and prompt conditions.
- Response Generation: Each question was executed three times independently to reduce conversational carry-over and capture generation variability.
- Response Generation: The Original, Straight-explicit, and LGBTQIA+-explicit conditions were collected in a standardized order.
- Response Generation: The analytical unit was a question-condition pair, classified as exhibiting a behavior if it appeared in at least one of three runs.
- Response Generation: Binary coding measured discriminatory treatment, incomplete answers, refusals, identity acknowledgment, and unnecessary identity focus.
- Response Generation: Comparative qualitative interpretation examined epistemic over-contextualization and other broader interactional patterns across identity conditions.
E. Phase 5: Agreement and Consolidation
Independent coding was compared across all variables, with disagreements resolved through collaborative reconciliation and third-party adjudication when necessary.
- Independent coding achieved approximately 85% agreement before reconciliation.
- The researchers compared classifications, discussed disagreements, and used third-party adjudication for unresolved differences.
- The consolidated dataset provided the basis for the paper’s analysis.
F. Threats to Validity.
The study identifies interpretive, measurement, and scope-related threats to validity, including potentially inflated frequency estimates and limited generalizability.
- Internal Validity: Coding subjective variables such as discriminatory treatment and unnecessary identity focus involved interpretive judgment.The researchers mitigated this concern through calibration, independent coding, reconciliation, and adjudication.
- Internal Validity: Coding a behavior as present when it appeared in any of three runs may inflate frequency estimates.The authors describe this consolidation rule as conservative because one occurrence demonstrates that the model can produce the behavior.
- Construct Validity: The five coding variables do not exhaustively represent all dimensions of fair or unfair conversational-AI treatment.The qualitative dimensions, particularly epistemic over-contextualization, emerged inductively and lack prior external validation.
- External Validity: The findings are limited to one ChatGPT version, 50 questions from four mental-health topics, and one type of identity-disclosure framing.The analysis covered 450 responses consolidated into 150 observations per condition.
- Scope of Analysis: Table I summarizes the frequency of each coded dimension across the three prompt conditions.The responses were organized as Original, Straight-explicit, and LGBTQIA+-explicit conditions.
1) Refusal to Answer:
Identity disclosure did not cause refusals, but the LGBTQIA+-explicit condition included one incomplete answer and two observations coded as discriminatory treatment.
- No refusals occurred in any condition, and all 50 questions per condition received substantive responses.Identity disclosure, whether straight or LGBTQIA+, did not trigger avoidance behavior.
- One incomplete answer appeared in the LGBTQIA+-explicit condition, compared with none in the Original or Straight-explicit conditions.The answer redirected a question about a daughter’s severe depression and obsessive thinking toward general strategies for reducing parental anxiety.
- Two observations in the LGBTQIA+-explicit condition were coded as discriminatory treatment, while none appeared in the other conditions.One example introduced unsupported assumptions about gay men’s experiences and coping mechanisms.
3) Identity Acknowledgment:
LGBTQIA+ identity disclosure produced much more frequent identity acknowledgment and unnecessary identity focus than straight disclosure, while the Original condition also contained one hallucinated identity assumption.
- Identity Acknowledgment: 39 of 50 LGBTQIA+-explicit observations acknowledged identity, compared with two Straight-explicit observations.Straight-identity references were brief, whereas LGBTQIA+ identity was often integrated into emotional framing, explanations, or recommendations.
- Identity Acknowledgment: The LGBTQIA+-explicit condition showed the strongest identity-related asymmetry despite structurally equivalent disclosures.The contrast was two Straight-explicit observations versus 39 LGBTQIA+-explicit observations.
- Unnecessary Identity Focus: The Original condition contained one unnecessary identity-focus observation in which the model assumed bisexual identity and reframed a question about voices and medication.This was the only Original-condition observation where identity content appeared unprompted.
- Comparison: Table I summarizes frequencies for the coded dimensions across the three prompt conditions.
- Identity Acknowledgment: In the Straight-explicit condition, acknowledged identity was treated as irrelevant to the concern and did not alter clinical framing or recommendations.
- Unnecessary Identity Focus: Nine LGBTQIA+-explicit observations introduced disclosed identity beyond what the question required, compared with none in the Straight-explicit condition.These responses added explanations involving minority stress, social pressure, rejection, or identity-related coping without basis in the question content.
B. Emergent Cross-condition Patterns
Across identity conditions, LGBTQIA+-explicit responses showed more identity-focused contextualization and occasional stereotyping, while practical guidance remained broadly comparable.
- Stereotyping appeared only in the LGBTQIA+-explicit condition, where responses made unsupported generalized attributions about LGBTQIA+ emotional experiences.The attributions included social rejection, internalized pressure, and identity-related vulnerability.
- LGBTQIA+-explicit responses consistently incorporated identity into framing, whereas Straight-explicit responses only partially acknowledged it and generally treated it as irrelevant.
- Epistemic over-contextualization was concentrated in the LGBTQIA+-explicit condition through unsupported links between identity and emotional or social circumstances.These links included minority stress, social difficulty, past rejection, and emotional vulnerability without explicit grounding in the prompt.
- Explicit LGBTQIA+ disclosure changed contextual framing more strongly than substantive recommendations, which remained broadly comparable across conditions.
V. DISCUSSION
The findings indicate that fairness concerns can arise through contextual interpretation and explanatory reasoning even when mental-health responses remain supportive and coherent. The discussion therefore extends fairness requirements toward neutral, consistent identity handling and comparative evaluation.
- Fairness-related differences appeared through identity acknowledgment, contextual expansion, and occasional stereotypical reasoning rather than refusal, inability to answer, or overtly harmful content.
- Supportive responses sometimes introduced unsupported assumptions about minority stress, rejection, vulnerability, or identity-related difficulty, making apparent empathy potentially reductive or clinically unhelpful.
- Healthcare conversational-AI fairness requirements may need to cover conversational neutrality, equitable identity handling, explanatory consistency, and contextual interpretation beyond correctness and safety.
- Requirements should limit identity-related assumptions unsupported by user-provided information and require consistent handling of comparable disclosures unless context justifies differences.
- Structurally equivalent prompts with different identity disclosures can help evaluate asymmetries in acknowledgment, contextualization, and explanatory reasoning.
- Subtle conversational asymmetries matter for requirements engineering because fairness concerns may emerge through contextual interpretation rather than explicitly harmful outputs.
- The study concludes that LGBTQIA+ disclosure did not reduce answerability or supportive recommendations but produced asymmetric handling and unsupported vulnerability-related reasoning.
VI. CONCLUSIONS AND FUTURE WORK
The study finds that LGBTQIA+ identity disclosure preserved response completeness and practical guidance while consistently changing contextual framing. It concludes that mental-health AI fairness requirements should address neutral identity handling and explanatory consistency, with future replication across models and versions.
- LGBTQIA+ identity disclosure did not compromise response completeness or practical guidance but altered contextualization through asymmetry, over-contextualization, and subtle stereotyping.
- Fairness requirements for LLM-based mental-health support systems should explicitly address conversational neutrality, equitable identity handling, and consistency in explanatory reasoning.
- Future work will replicate the analysis across other large language models and model versions to assess generalizability.