PaperlayerView exact revision on arXiv

Source-linked AI summary

Gender Associations in LLM-Mediated ADHD Self-Diagnosis

Matilde Elene Hansen, Ross Deans Kristensen-McLachlan

arXiv:2610.33697v1cs.CLcs.CY

TL;DR

Evidence remains limited on how demographic cues shape LLM outputs in self-diagnostic interactions, including ADHD. This study adapts CAT to evaluate GPT-4o with explicit and implicit gender cues against clinically informed reference distributions. Explicit cues amplify female-associated responses in several analyses, while implicit cues produce comparatively less skewed, sometimes proxy-aligned patterns.

  • Problem

    How demographic cues shape LLM outputs in medical self-diagnosis remains poorly understood, despite the potential for biased or clinically misleading responses to shape symptom interpretation.

    Source: §1 · p. 1

  • Method

    The study adapts CAT to compare GPT-4o’s gender associations for ADHD symptom descriptions under explicit and implicit cues against subtype-specific clinical reference distributions.

    Source: §3.1 · p. 4

  • Results

    Female-associated responses were amplified in three of four association tests, while implicit cueing produced comparatively less skewed, near-even gender distributions.

    Source: §4.4 · p. 8

  • Takeaways & Limitations

    Evaluations of diagnostic bias depend on how demographic information and reference distributions are operationalized, so bias-mitigation efforts should also be assessed for overcorrection and differential behavior across explicit and implicit cues.

    Source: §5 · p. 9

    The clinical proxy distributions are strongly female-skewed, and apparent model alignment may reflect distortions in the reference data as well as corrective shifts.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used for medical self-diagnosis. However, the way demographic cues shape outputs in these interactions remains poorly understood. This is particularly consequential for diagnoses like ADHD, where persistent gender disparities are not fully aligned with the clinical evidence. We adapt the Context Association Test (CAT) to evaluate whether GPT-4o responds to gendered ADHD associations in a zero-shot, self-diagnostic setting. We show that explicit cueing consistently amplifies certain ADHD subtype associations while implicit cueing produces near proxy-aligned distributions. These divergences suggest that explicit demographic cues may activate bias-mitigation mechanisms that over-correct rather than eliminate bias, whereas implicit cues appear to reflect the model's default behavior. Our results demonstrate that evaluating gendered bias in LLM-mediated self-diagnosis requires careful attention to the demographic associations a model encodes.

1 Introduction

LLMs are increasingly used for mental-health self-diagnosis, where personalized responses may shape symptom interpretation. This study tests how explicit and implicit gender cues affect GPT-4o’s associations between ADHD symptoms and gender, finding substantially different patterns across cueing conditions.

1 Introduction

  • Biased or clinically misleading outputs in self-directed mental-health interactions may contribute to misdiagnosis, inappropriate self-diagnosis, or psychological distress.Conversational agents offer interactive, context-sensitive responses that can foster perceptions of empathy, social presence, and personalization.Source: §1 · p. 1
  • Evaluating psychiatric model outputs requires distinguishing clinically informed responses to symptoms from the reproduction of training-data biases and stereotypes.Prior work has documented gender and ethnic disparities in generated diagnoses, as well as gendered associations in eating-disorder and personality-disorder assessments.Source: §1 · p. 1
  • ADHD is a useful case because it relies on language-based symptom descriptions and has longstanding complexities surrounding gender and diagnosis.The passage situates this case in work on ADHD gender and diagnostic disparities.Source: §1 · p. 1
  • Across ADHD subtypes, explicit cueing generally amplified female-associated responses, while implicit cueing yielded less skewed, near-even to proxy-aligned associations.The study adapted the Context Association Test to compare direct gender terms with gender-indicative names in GPT-4o against clinically informed, gender-stratified proxy distributions.Source: §1 · pp. 1–2

2 Evaluating LLM-Mediated Diagnostic Inference

The section outlines epistemic, interactional, and systematic-bias risks in LLM-mediated diagnosis, then reviews evaluation approaches and their assumptions. It motivates ADHD as a case where cueing and reference distributions matter because gender patterns are clinically contested.

2.1 Sources of Uncertainty

  • Static training data, outdated or incorrect diagnoses, and hallucinations raise epistemic reliability concerns, while sycophantic responses may reinforce self-diagnostic assumptions.The section also describes risks of unsafe recommendations, overpathologization, and premature adoption of psychiatric labels.Source: §2.1 · p. 2
  • Systematic bias may reproduce socially embedded stereotypes, but translating distinctions among preexisting, technical, and emergent bias into empirical evaluation remains challenging.The cited framework links understanding bias sources and structure to identifying and mitigating potential harms.Source: §2.1 · p. 2

2.2 Bias Evaluation in Related Work

  • Prior psychiatric evaluations use vignettes, manipulated cases, or clinical reference points, while forced-choice designs risk overstating preferences when neither demographic option fits the context.The literature includes evaluations against case-level ground truths and studies of demographic associations in generated clinical vignettes.Source: §2.2 · p. 3

2.3 ADHD as Case Study

  • Self-diagnostic cueing remains underexamined, and ADHD is a relevant case because its language-based symptoms coexist with contested gender distributions and subtype patterns.The section describes historical male-skewed diagnosis alongside evidence of more similar adult symptom patterns across genders, leaving clinical variation difficult to separate from diagnostic and referral practices.Source: §2.2 · p. 3, §2.3 · pp. 3–4

2.2 Bias Evaluation in Related Work

  • CAT-based psychiatric evaluation has used demographic parity as its reference, but equal probabilities may not transfer to clinical domains where demographic distributions differ.The section motivates evaluating meaningful associations against clinically informed reference distributions rather than assuming parity.Source: §2.2 · p. 3

3 Methodology

The methodology adapts CAT-style forced association to compare GPT-4o’s gender associations for ADHD symptom descriptions under explicit and implicit cueing. It evaluates outputs against subtype-specific clinical proxy distributions rather than a 50/50 parity baseline.

3.1 Study Design

  • Each trial pairs an ADHD symptom sentence with two gender-associated targets and an unrelated control, comparing explicit gender terms with implicit gender-indicative names.The four tests cover ADHD-I and ADHD-HI jointly in Study 1 and ADHD-C separately in Study 2.Source: §3.1 · p. 4

3.2 API- and technical setup

  • The study used zero-shot independent API calls with GPT-4o for final prompt generation and evaluation, fixing temperature at 0.7 and top-p at 1.GPT-4o-mini was used during pipeline development for semi-automated mapping and metric validation.Source: §3.2 · p. 4

3.3.1 Target Lists

  • The target lists paired explicit “women” and “men” cues or five male and five female first names with the unrelated “basketball” control.The implicit names were selected through a semi-automated procedure using the U.S. Social Security Administration baby-name dataset.Source: §3.3.1 · p. 5

3.3.2 Context Lists

  • Study-specific symptom contexts were derived from Reddit narratives mapped to DIVA behavioral examples, then manually validated and curated; ADHD-C contexts combined selected inattentive and hyperactive-impulsive symptoms.The ADHD-C sentences were fully crossed from five symptoms of each type, while each clause was expanded into ten variants; those variants were not individually reviewed, and experts did not validate the materials.Source: §3.3.2 · p. 5

3.4 Experimental Designs

  • The explicit condition yielded 360 responses for Study 1 and 500 for Study 2, while the implicit condition yielded 4,500 and 6,250 responses, respectively.An a priori power analysis targeted approximately 241 responses per association test, and response counts were conservatively oversampled to balance statistical power and computational constraints.Source: §3.4 · p. 5

3.5 Evaluation Framework and Metrics

  • The evaluation combined LMS and BS with chi-square goodness-of-fit tests comparing observed gender-association distributions against clinically informed subtype proxies.Observed percentages represent response proportions across trials, not estimated group means.Source: §3.5 · p. 5, §3.5.2 · p. 6

3.5.1 CAT-Metrics

  • LMS measures selection of meaningful gender targets over the unrelated control, while BS measures the share of meaningful associations assigned to the proxy-defined majority gender.BS is interpreted relative to the subtype-specific proxy proportion: proximity indicates alignment, and deviations indicate relative amplification or attenuation.Source: §3.5.1 · p. 6

3.5.3 Proxy ADHD Baseline

  • The proxy baseline uses subtype-specific distributions from a Dutch clinical-care sample of adults with ADHD, rather than population-level prevalence estimates.The final sample included 2,110 adults and was strongly female-skewed; the authors caution that its distributions reflect sample composition, contrasting with traditionally reported male-skewed prevalence patterns.Source: §3.5.3 · p. 6

4 Results

Across the studies, explicit and implicit gender cues elicited different ADHD associations, ranging from amplification to attenuation or close alignment with clinical proxies. Task coherence was high, but the results and their interpretation depend on cueing and evaluation choices.

4.1 Overall Task Compliance

  • Across studies and conditions, only 9–10% of responses selected the control or lacked a valid association, indicating high task-level coherence.The authors therefore focus subsequent analyses on systematic bias rather than response validity; Table 2 summarizes aggregate CAT metrics.Source: §4.1 · p. 6, Table 2 · p. 6

4.2.1 S1 Explicit

  • Under explicit Study 1 cueing, women were selected in 80.4% of ADHD-I and 72.3% of ADHD-HI responses; ADHD-HI matched its proxy, unlike ADHD-I.The overall distribution differed significantly from the proxy, driven by ADHD-I; Figure 1 displays subtype distributions against clinical proxies.Source: §4.2.1 · pp. 6–7, Figure 1 · p. 7

4.2.2 S1 Implicit

  • Implicit Study 1 cueing yielded 49.13% female-associated responses against a 60.85% proxy, a significant deviation despite a near-even overall gender split.The authors describe the resulting responses as stable and near gender-neutral rather than random.Source: §4.2.2 · p. 7

4.2.3 S1 Cueing Conditions

  • Study 1 cueing reversed the proxy-relative direction: explicit responses amplified the female skew by 15.62 pp, while implicit responses attenuated it by 11.72 pp.The reversal was driven mainly by ADHD-I, whose female-associated responses fell from 80.4% to 38.5%; ADHD-HI remained comparatively stable and closer to the proxy.Source: §4.2.3 · p. 7

4.3.1 S2 Explicit

  • Under explicit Study 2 cueing, 93.1% of ADHD-C responses were female-associated, 31.82 pp above the proxy, with higher coherence for female than male responses.The distribution differed significantly from the clinical proxy.Source: §4.3.1 · pp. 7–8

4.3.2 S2 Implicit

  • Implicit Study 2 cueing closely matched the ADHD-C proxy: 60.73% versus 61.31%, a −0.58 pp difference that was not statistically significant.Source: §4.3.2 · p. 8

4.3.3 S2 Cueing Conditions

  • For ADHD-C, implicit cueing aligned with the proxy while explicit cueing amplified its female skew, contrary to the expectation that implicit cues reveal latent associations.The authors suggest explicit cueing may activate distorted bias-mitigation processes; ADHD-C’s diagnostic ambiguity may also encourage heuristic associations.Source: §4.3.3 · p. 8

4.4 Discussion: Cross Study Patterns

  • Explicit cueing amplified female associations in three of four tests, while implicit cueing was less skewed; proxy alignment and amplification coexisted across conditions.The authors caution that the controlled CAT assigns demographic targets to symptom descriptions rather than modeling naturalistic user–LLM interaction, and compare clinically informed references with parity-based CAT evaluation.Source: §4.4 · pp. 8–9

5 Conclusions

Gendered distortions varied substantially with demographic cueing, producing qualitatively distinct patterns. These findings make the evaluation design—including how cues and reference distributions are defined—important to interpreting diagnostic bias.

5 Conclusions

  • Gender associations varied with cueing rather than showing a stable direction, with explicit and implicit conditions yielding distinct representational outcomes.Associations ranged from amplification and attenuation to proxy-aligned patterns.Source: §5 · p. 9
  • The controlled CAT isolates gender–symptom associations but does not approximate naturalistic user–LLM interactions.Whether these patterns generalize to other models, evaluation paradigms, or naturalistic self-diagnostic interactions remains to be determined.Source: §5 · p. 9
  • Diagnostic-bias evaluations depend on how demographic information, bias, and reference distributions are operationalized.The authors recommend evaluating mitigation and alignment for possible overcorrection, representational distortion, and differences between explicit and implicit cueing.Source: §5 · p. 9

Limitations

The study’s scope and design constrain how its findings should be generalized or interpreted: it evaluates one model with a constructed reference sample and controlled symptom-level prompts, using a binary gender operationalization.

Limitations

  • The study evaluates only GPT-4o, so its findings may not generalize to all models.The authors describe the model as widely accessible and informative for understanding widely deployed LLMs in psychologically relevant contexts.Source: Limitations · pp. 9–10
  • The clinical adult ADHD sample used as a reference is female-skewed and may itself reflect selection effects, referral practices, and diagnostic thresholds.Model alignment with this proxy could reflect corrective shifts relative to historical underdiagnosis or distortions in the reference data.Source: Limitations · p. 10
  • Symptom contexts came from Reddit posts by self-identified undiagnosed individuals, lacked expert validation, and represent isolated criteria rather than diagnostic symptom constellations.The resulting gender associations are insufficient for diagnostic inference and should be interpreted as symptom-level associations.Source: Limitations · p. 10
  • The forced-association design differs from naturalistic interaction, and target order was not counterbalanced, leaving positional and prompt-specific effects unresolved.English-only prompts and symptom descriptions also limit cross-linguistic generalization.Source: Limitations · p. 10
  • Gender was operationalized as binary, excluding intersectional dimensions such as race, sexual orientation, and socioeconomic status.The authors note this simplification facilitates experimental control but creates representational limitations.Source: Limitations · p. 10

Ethical Considerations

The authors frame the findings as controlled measurements in a consequential self-diagnostic context, not as clinical guidance or evidence of gender differences in ADHD.

Ethical Considerations

Appendices

The appendix describes the DIVA-5’s diagnostic components and illustrates how concrete behavioral examples accompany symptom criteria.

A ADHD: Clinical Diagnosis

B Main Association (CAT): Task Description

The adapted CAT presents GPT-4o with symptom sentences and forced-choice gender targets in independent calls. GPT-4o was selected as a widely used, publicly available model during data collection.

B Main Association (CAT): Task Description

  • Each independent call paired one symptom sentence with a forced choice among three targets, avoiding conversational carryover between trials.Explicit prompts used “Women,” “Men,” and “Basketball”; implicit prompts substituted gender-indicative names.Source: B Main Association (CAT): Task… · p. 14
  • Inference settings were held constant at temperature 0.7 and top-p 1.0, with each call in a separate context window.These settings standardized inference while preserving controlled stochastic variability and categorical responses.Source: B Main Association (CAT): Task… · p. 14

C Justification of Model Selection

  • The study evaluated GPT-4o because it was widely used and publicly available during data collection.The model choice reflects the assumption that self-diagnostic users prompt a model at inference rather than modifying or fine-tuning it.Source: C Justification of Model Selection · p. 14

D CAT Materials: Target lists

D.1 Implicit Target List: Constructed Name Lists

The study built gender-indicative name targets and adult ADHD symptom contexts from public names, Reddit narratives, and DIVA-5 examples. It aligned narratives to clinical criteria while retaining naturalistic, non-diagnostic wording.

D.1 Implicit Target List: Constructed Name Lists

  • Ten gender-indicative names were sampled from U.S. Social Security Administration name frequencies for adults, then crossed into 25 male-female pairs.The birth-year window was 1975–2005; the top 20 names per sex were excluded before drawing from the next 150 most frequent names.Source: §D.1 · p. 15, Table D.3 · p. 15

E CAT Materials: Context-List Generation (CLG)

  • Reddit narratives were mapped to DIVA-5 behavioral examples for inattentive and hyperactive-impulsive ADHD without explicit diagnostic phrasing.The mapping was intended to reduce reliance on language likely present in model training while preserving clinical validity.Source: E CAT Materials: Context-List… · p. 15

E.1 Reddit Narratives

  • The r/ADHD forum was used as a source of anonymous, first-person accounts relevant to self-diagnosis and symptom interpretation.Posts containing “undiagnosed” were extracted; demographic and explicit gender references were removed or neutralized, leaving 267 potential symptom-clause sentences.Source: §E.1 · pp. 15–16

E.2 DIVA-5

  • The context-generation criteria covered all 18 adult DSM-5 ADHD symptoms, split evenly between inattentive and hyperactive-impulsive symptoms.Childhood criteria and play-activity clauses were omitted because the study focused on adult self-diagnostic contexts.Source: §E.2 · p. 16, Table E.5 · p. 16

E.3 CLG: Mapping Pipeline

The mapping pipeline matched Reddit-derived sentences to adult ADHD symptom examples, using repeated constrained classifications and majority voting before manual validation. It retained a small subset of candidate sentences as symptom clauses.

E.3 CLG: Mapping Pipeline

  • Each sentence was assigned to its best-fitting DIVA-5 behavioral example or to “none” through constrained, single-label classification.Classifications were run five times, aggregated by majority vote, and proportional agreement scores guided manual validation.Source: §E.3 · p. 17, Figure 3
  • 55 of 267 sentences (20.6%) mapped to ADHD symptoms; 212 (79.4%) were excluded, and one representative clause was retained per symptom.The 18 retained clauses covered nine inattentive and nine hyperactive-impulsive symptoms; their mean length was 12.56 words.Source: §E.3 · p. 17, Table E.6 · p. 17, Table E.7 · p. 18
  • Thirteen retained clauses came from model-mapped content, three of which were merged; relative mean agreement was 52.14%.Source: §E.3 · p. 17

E.3.1 CLG: Study-Specific Context Lists

The study generated rephrased context lists for two ADHD subtype studies and used forced-choice CAT prompts. Its proxy-population figure distinguishes overall and gender-stratified subtype distributions.

E.3.1 CLG: Study-Specific Context Lists

  • Each symptom clause was rephrased into ten semantically equivalent variants to manage lexical variability.The rephrasing temperature was set to 0.3 to favor deterministic, coherent outputs and reduce irrelevant variation.Source: §E.3.1 · p. 19, Figure 4 · p. 19
  • Study 1 used 180 sentences evenly split between inattentive and hyperactive-impulsive ADHD, while Study 2 used 250 combined-subtype contexts.The combined-subtype contexts paired the five most frequently endorsed symptoms from each subtype, based on the proxy population.Source: Table E.8 · p. 18, §E.3.1 · p. 19, Table E.9 · p. 19

F CAT: Prompt Templates and Example Output

G Proxy Population Distribution

  • Figure 5 presents overall ADHD subtype proportions and gender-stratified proportions, with percentages calculated within each proxy gender.Source: Figure 5 · p. 20
Loading 2610.33697v1…