Automatic Evaluation of Mental Health Stigma in Online Communication

1 Introduction

• The authors release the publicly available benchmark, annotations, stigma exemplars, and code.The release is at https://github.com/jemimakang/mh_stigma.Source: §1 · p. 1
• Stigma can involve subtle devaluation, perceived or experienced stigma, structural disadvantage, and internalized stereotypes, not only explicit hostility.A single passage may combine distinct components such as social distancing, fear, and moralized attribution.Source: §1 · p. 1
• The benchmark combines binary stigma detection with a multi-level taxonomy of stigma mode, domain, public-stigma types, and components across six conditions.It covers naturally occurring online news and Reddit text, distinguishing expressed/enacted from perceived/experienced stigma and public, structural, associative, and self-stigma.Source: §1 · p. 2
• The authors evaluate whether LLM performance changes with prompt information and whether stigma is captured by sentiment, toxicity, and hate-speech classifiers.They also examine performance across mental health conditions.Source: §1 · p. 2
• LLMs often achieve high recall by substantially overpredicting stigma, especially when operational decision rules are underspecified.Adding explicit operational rules reduces this bias.Source: §1 · p. 2

2.1 Stigma as a multidimensional construct

• Stigma can be cognitive, affective, or behavioral, and may be public, structural, associative, or self-stigma.Examples include stereotypes and causal attributions, fear or pity, and social distancing or discriminatory treatment.Source: §2.1 · p. 2
• Stigma is not reducible to negative sentiment, toxicity, or overt abuse, and its textual forms include blame, dangerousness, social distancing, prognostic pessimism, and condescension.It can also appear in first-person accounts of being feared, excluded, mocked, or discriminated against.Source: §2.1 · pp. 2–3
• Prior findings across conditions motivate evaluating stigma across multiple mental health conditions rather than treating mental illness as one homogeneous target.Reported differences include more negative coverage of some conditions and high public stigma toward alcohol addiction and schizophrenia.Source: §2.1 · p. 3

2.2 Detection of mental health stigma

• Existing supervised stigma classifiers and few-shot LLM studies address particular conditions, text types, or stigma forms.Most existing studies focus on detecting stigma presence rather than its types.Source: §2.2 · p. 3
• The paper addresses these gaps with a fine-grained stigma evaluation framework spanning multiple mental health conditions, news, and social media.Source: §2.2 · p. 3

3.1 Stigma taxonomy

• The taxonomy distinguishes expressed/enacted from perceived/experienced stigma, four stigma domains, and public-stigma cognitive, affective, and behavioral labels.It draws conceptually on Link and Phelan and gives greatest granularity to public stigma.Source: §3.1 · p. 3

3.2 Online communication

• The study uses U.S. news articles from NOW (2010–2025) and Reddit posts (2012–2022) containing target-condition terms.Reddit posts were required to contain at least 30 words and were excluded if they mentioned multiple target conditions.Source: §3.2 · p. 3

3.3 Mental health conditions

• The six target conditions are ADHD, alcoholism, autism, borderline personality disorder, depression, and schizophrenia.They were selected to balance theoretical coverage, corpus frequency, and stigma variation, with domain-expert consultation.Source: §3.3 · pp. 3–4

3.4 Annotation and benchmark construction

• Four annotation rounds used stigma experts, a taxonomy-based codebook, and adjudication, with targeted sampling added after random sampling yielded few stigma-present texts.Agreement was moderate for stigma presence and mode; domain and public-stigma subfactor agreement was high on the subset unanimously judged stigma-present, which was small.Source: §3.4 · p. 4

3.4.1 Benchmark composition

• The final benchmark contains 470 texts, 87 annotated as stigma-present, with expressed/enacted stigma and public stigma the most common mode and domain.Public-stigma cognitive components were assigned far more often than affective or behavioral ones; affective pity was retained theoretically but excluded from category-level evaluation because it occurred in no texts.Source: §3.4.1 · p. 4

3.5 Stigma prototype exemplars

• The separate prototype set supplies clear stigma-present and stigma-absent examples for each category and was used in LLM prompts and embedding-similarity experiments.These examples were not part of the annotated benchmark.Source: §3.5 · p. 4

4.1 Experiment details

• Binary classification uses all 470 annotated samples, while mode and domain tasks use 87 stigma-present samples and public-stigma type/component tasks use 62.Fine-grained results require caution because label counts vary substantially and some categories have few examples.Source: §4.1 · p. 5, §4.2.3 · p. 7
• The LLM evaluation spans four providers and prompt conditions varying definitions, examples, operational rules, and stigma-type information.A Qwen2.5-7B classifier with a classifier head was also LoRA-tuned for binary prediction using adapted labels from Meng et al. (2025).Source: §4.1 · p. 5

4.2.1 Stigma presence classification

• Explicit operational rules substantially increase stigma-detection precision and overall accuracy, especially for Claude Sonnet 4.6, and reduce overprediction.Examples were associated with more accurate predictions for GPT 5.5 and, to a lesser extent, Sonnet 4.6, but do not establish acquisition of conceptual distinctions.Source: §4.2.1 · p. 6

4.2.2 Transfer from existing stigma data

• The adapted classifier transferred poorly from Meng et al. (2025), overpredicted stigma with bias +0.574, and had binary precision 0.165 on this benchmark.Continued training on this benchmark removed overprediction but performed no better than training on the benchmark alone.Source: §4.2.2 · pp. 6–7

4.2.3 Predicting stigma modes, domains, types and subtypes

• Operational rules substantially improve stigma-mode prediction, examples help domain and public-stigma type prediction, and definitions help fine-grained cognitive components.The authors relate the mode result to human annotation, where mode distinctions also required explicit decision rules.Source: §4.2.3 · p. 7
• Some cognitive prediction errors involved theoretically related labels, while affective-component confusion should be interpreted cautiously because examples were few.Reported confusions included personal responsibility with trivialization and dangerousness with uncontrollability.Source: §4.2.3 · p. 7

5.1 Condition-specific variation in stigma detection

• Schizophrenia had Claude Sonnet 4.6’s largest positive stigma-prediction bias and lowest accuracy, while alcoholism and ADHD had higher accuracy and lower bias.Operational rules reduced overprediction bias for schizophrenia most strongly; unequal prevalence and sampling mean condition estimates may be sensitive to individual examples.Source: §5.1 · p. 7, Table 5 · p. 8

5.2 Challenges in detecting mental health stigma

• Hard texts had mean/median error rates of .51/.53, versus .36/.28 for moderate items and .24/.10 for easy items.Error rates are across 20 model–prompt predictions per item; the pattern aligns model errors with cases requiring more annotator deliberation.Source: §5.2 · p. 8, Table 6 · p. 8
• For Claude Sonnet 4.6, adding rules and stigma-type information reduced false positives from 102 to 47 but increased false negatives from 40 to 56.The comparison is between definition and definition+examples+rule prompting, showing that additional prompt information did not uniformly improve detection.Source: §5.2 · pp. 8–9
• False positives often mistook negative clinical, risk-related, or metaphorical contexts for stigma, while false negatives included subtle first-person, institutional, or non-toxic cues.Examples included a cannabis-and-schizophrenia question, self-directed shame about alcoholism, support systems leaving autistic adults behind, and ADHD dismissed as an excuse.Source: §5.2 · p. 9

5.3 Dissecting stigma

• Stigma labels were not reliably associated with predicted sentiment or hate speech, and toxicity scores overlapped substantially across stigma-present and stigma-absent texts.The reported tests found no reliable association for general sentiment, targeted sentiment, or hate speech, and no statistically significant point-biserial correlation for toxicity.Source: §5.3 · p. 9
• Semantic similarity to stigma prototypes was insufficient to select the closest prototype, although prototypes could help models learn categories from examples.The comparison distinguishes shallow similarity-based selection from learning to predict categories using examples.Source: §5.3 · p. 9
• Among examined models, only GPT 5.5 consistently improved across 1- versus 10-example conditions; other models were often confused by additional examples.The examples were intended to represent varied social situations and support extrapolation to new situations.Source: §5.3 · p. 9

• The benchmark combines binary stigma detection with a fine-grained taxonomy across multiple conditions for naturally occurring online news and social media text.Source: §6 · p. 9
• LLMs under minimal prompting often overpredicted stigma, while explicit operational rules improved performance, particularly by reducing false positives.Source: §6 · p. 9
• Model errors concentrated in threshold cases requiring greater annotator deliberation, and the benchmark’s authors call for expanding conditions, annotator perspectives, and rare-category examples.The proposed expansions aim to enable more reliable fine-grained evaluation.Source: §6 · p. 9

Limitations

• The enriched benchmark sample reflects construction choices, so its label distribution cannot estimate population-level stigma prevalence in the NOW or Reddit corpora.Later rounds added stigma-present, boundary, and model-disagreement cases.Source: Limitations · p. 10
• The theory-driven framework does not incorporate lived-experience perspectives and may miss forms of stigma most salient or harmful to affected communities.The authors call for future work to include people with lived experience of mental health conditions.Source: Limitations · p. 10
• Treating each condition as a broad category enables comparison across six conditions but misses within-condition variation by social position and representational frame.The passage names gender, race, class, and age, among other factors.Source: Limitations · p. 10
• Showing annotators each text’s target condition may have shaped judgments; despite withheld metadata, stylistic or contextual cues could reveal news versus Reddit.The concern is especially relevant to judgments of dangerousness, blame, and social distancing.Source: Limitations · p. 10
• Evaluation covers selected LLMs, prompt settings, and adjacent-construct classifiers, not an exhaustive comparison of stigma-detection systems.The benchmark is intended as a basis for future testing of dedicated systems.Source: Limitations · p. 10
• Because the corpora cover only online news and Reddit, the benchmark’s generalizability to other settings and languages is limited.The passage lists clinical records, policy documents, television, interviews, and support forums as other settings.Source: Limitations · p. 10
• The authors position the benchmark as a foundation for protocols to monitor stigma, evaluate public-health campaigns, and support reduction efforts.These are future uses the authors describe despite the study’s limitations.Source: Limitations · p. 10

Ethical Considerations

• Reddit users’ individual consent could not be obtained; protections include omitting usernames and direct links, paraphrasing examples, secure storage, pseudonymous IDs, and application-only access to Reddit excerpts.Reddit data use was approved by the University of Melbourne’s Ethics Committee; annotations, metadata, and news excerpts are openly available.Source: Ethical Considerations · pp. 10–11
• The benchmark is a research resource, not a standalone moderation, clinical, or individual-level tool; automated detection risks false positives, false negatives, and reinforcing diagnosis–harm associations.The authors call for future benchmark design and evaluation to incorporate lived-experience perspectives.Source: Ethical Considerations · p. 11

A.1 Targets

• The six targets—ADHD, autism spectrum disorder, borderline personality disorder, depression, alcoholism, and schizophrenia—were selected for theoretical coverage, corpus frequency, and expected stigma variation.Researchers consulted experts, generated term variants, inspected pilot searches, and removed spurious matches such as “BPD” meaning “barrels per day.”Source: §A.1 · p. 13

A.2 News on the Web

• The NOW corpus covers 166,912 U.S. news articles mentioning the six targets; preprocessing removed ambiguous or incomplete records and retained the target-containing paragraph plus two following paragraphs.The underlying NOW corpus spans 2010–2025 and draws on more than 7,000 outlets across 20 countries.Source: §A.2 · p. 13

A.3 Reddit

• The Reddit corpus contains 1,303,437 posts collected from 2012–2022; posts had at least 30 words, mentioned one target condition, and were retained in full.Posts were collected via Pushshift.io.Source: §A.3 · p. 14

B Stigma taxonomy; C Prototype exemplar sources

• The taxonomy distinguishes stigma mode, domain, and public-stigma sub-factors and components, drawing on established frameworks and measures; prototype examples provide at least ten present and ten absent usages per category.Prototype examples were adapted from validated scale items and expanded through close variants, negation, or reframing.Source: B Stigma taxonomy · p. 15, C Prototype exemplar sources · p. 18

D Annotation protocol and guideline revisions

• Annotation rules defined stigma through mental-health-related negative attributions, emotions, treatment, demeaning language, or intrinsically stigmatizing term use, while excluding unrelated negativity and discussion that merely critiques stigma.Later rounds enriched stigma-present and boundary cases, added an explicit operational decision rule, and clarified expressed/enacted versus perceived/experienced stigma.Source: D Annotation protocol and guideline… · pp. 20–21

D.1 Annotation platform interface; D.1.1 Mode; D.1.2 Domain; D.1.3 Sub-factor and component (public domain only); E.2 “Definition” prompt

• Annotators first made a binary stigma judgment and confidence rating, then selected applicable modes, domains, and—within public stigma—cognitive, affective, or behavioral sub-factors and components.Public-stigma component lists included 12 cognitive, five affective, and two behavioral components, with an “Other” option; the “Definition” prompt used a psychologists’ definition of stigma.Source: §D.1 · p. 21, §D.1.1 · p. 21, §D.1.2 · p. 21

E.3 “Examples” prompt

• The examples-only prompt presented psychologists’ stigma examples before asking whether a target text contained stigma.Examples included labeling, stereotypes linking mental illness with violence, calls for protection from people described as insane, discriminatory employment or housing, and exclusion from treatment decisions.Source: §E.3 · p. 23

E.4 “Definition+examples” prompt

• The definition-and-examples prompt added psychologists’ definition of stigma as labeling, stereotyping, separation, status loss, and discrimination occurring together.It then presented the examples and asked whether the target text contained stigma.Source: §E.4 · p. 23

E.5 “Definition+examples+rule” prompt

• The definition-plus-examples-plus-rule prompt retained the definition and examples, then added an explicit operational decision rule.The examples were split across two passages before the rule was provided.Source: §E.5 · pp. 23–24
• The operational rule marked stigma present for specified negative attributions, emotional reactions, treatment, or demeaning language directed at people with mental health conditions or those associated with them.It instructed annotators to mark no when the negative content was unrelated to the condition or merely mentioned rather than endorsed; descriptions of stigma perpetrated by others were still labeled yes.Source: §E.5 · p. 24

F Model and inference details

• The experiments used public model names, API identifiers, and inference settings because proprietary model parameter counts were not publicly disclosed.Runs were conducted in May 2026, used provider or software-wrapper defaults unless otherwise stated, and shared prompt templates across models within each condition.Source: F Model and inference details · p. 25

G Benchmark composition: Label distributions

• The table’s counts describe benchmark composition rather than real-world stigma prevalence, and taxonomy counts are label assignments that may overlap across texts.The taxonomy panels cover the 87 stigma-present texts.Source: G Benchmark composition: Label… · p. 25, Table 16 · p. 27
• Panels A and B report source and condition coverage, with Panel B also showing stigma-mode coverage.Source: G Benchmark composition: Label… · p. 25, Table 16 · p. 27
• Panels C and D report higher-level taxonomy labels and fine-grained public-stigma components among stigma-present texts.Source: G Benchmark composition: Label… · p. 25, Table 16 · p. 27

H Qualitative model error examples

• A predefined-rule audit grouped errors by text, tracked persistent errors and prompt transitions, then manually inspected selected examples for qualitative patterns.Source: H Qualitative model error examples · p. 28
• Table 17 presents selected stigma-absent texts that one or more models predicted as stigma-present.Reddit examples are paraphrased for privacy; news examples are reproduced in full.Source: Table 17 · p. 29
• Table 18 presents selected stigma-present texts that one or more models predicted as stigma-absent.Reddit examples are paraphrased for privacy.Source: Table 18 · p. 30

I Human annotation agreement

• Across all items, stigma-presence agreement was exact = .66 (AC1 = .40), while mode agreement was exact = .61 (AC1 = .54) including no-stigma and exact = .51 (AC1 = .67) on 19 unanimously stigma-present items.Figures 3 and 4 show pairwise agreement matrices for stigma presence and mode, respectively.Source: I Human annotation agreement · p. 31, Figure 3 · p. 31, Figure 4 · p. 32
• In Round 3, stigma-presence exact agreement fell to .39 (AC1 = .22), versus .62 (AC1 = .62) in Round 2; the new annotator labeled 66.7% of items stigma-present, versus 30.0% and 24.7%.Pairwise AC1 between the new annotator and returning annotators was .08 and .11.Source: I Human annotation agreement · p. 31
• Domain labels had macro pairwise AC1 = .82 and public-stigma subfactors AC1 = .89 among unanimously stigma-present items, but small sample size and rare labels may inflate apparent agreement.Subtype and domain agreement was evaluated only on items all annotators marked stigma-present.Source: I Human annotation agreement · p. 31

J Detailed scores for model predictions

• Detailed model-score tables cover stigma presence, mode, domain, type, and subtypes; Table 26 breaks Claude Sonnet 4.6 stigma-presence performance down by condition.The tables report macro-averaged results and bootstrap confidence intervals for mode, domain, type, and subtype tasks.Source: J Detailed scores for model predictions · p. 33, Table 21 · p. 34, Table 22 · p. 34

K Confusion matrices for fine-grained stigma classification

• Cognitive subtype confusions often involved theoretically related categories, while affective components were also frequently confused; sparse examples limit interpretation.Examples include personal responsibility versus trivialization, controllability versus personal responsibility, segregation endorsement versus incompetence, and dangerousness versus uncontrollability.Source: K Confusion matrices for fine-grained… · p. 38
• Figures 5, 6, and 7 show row-normalized model confusion matrices for cognitive, affective, and behavioral public-stigma components, respectively.The appendix frames these matrices as diagnostic error analyses, not stable estimates for rare labels.Source: K Confusion matrices for fine-grained… · p. 38, Figure 5 · p. 39, Figure 6 · p. 39

L Classifier results for stigma-related constructs

• The neighboring-construct models served as diagnostic baselines for general sentiment, targeted sentiment, hate speech, and toxicity rather than stigma-specific systems.Targeted sentiment was estimated with an aspect-based sentiment model applied outside its conventional setting.Source: L Classifier results for… · p. 40, Table 27 · p. 41
• Categorical classifier outputs were tested for association with gold binary stigma labels using chi-square contingency analyses; toxicity scores were examined by gold label.Figure 8 presents standardized residuals for sentiment and hate-speech categories, while Figure 9 shows toxicity-score distributions by stigma label.Source: L Classifier results for… · p. 40, Figure 8 · p. 41, Table 28 · p. 42
• General sentiment, targeted sentiment, and hate-speech predictions were not significantly associated with stigma presence, while toxicity had substantial score overlap and weak discrimination (AUC = .60).The paper interprets these results as evidence that mental health stigma is related to, but distinct from, negative sentiment, hate speech, and toxicity.Source: L Classifier results for… · p. 40, Figure 8 · p. 41, Figure 9 · p. 42

M Semantic similarity with stigma prototypes

• Stigma-prototype embedding predictions consistently had significantly worse recall than Gemini 4 Flash, especially for stigma domain.The single-label prototype method was evaluated only on samples with exactly one gold label for the target field.Source: M Semantic similarity with stigma… · p. 40, Figure 10 · p. 43