PaperlayerView exact revision on arXiv

Source-linked AI summary

WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models

Virginia K. Felkner, Ho-Chun Herbert Chang, Eugene Jang, Jonathan May

arXiv:2306.15087v2cs.CLcs.CY

TL;DR

LLM bias benchmarks have insufficiently captured LGBTQ+ identities and harms grounded in affected communities’ experiences. WinoQueer builds a community-sourced benchmark from LGBTQ+ survey input and evaluates popular models, finding considerable anti-queer bias that can be somewhat mitigated through relevant finetuning.

  • Problem

    Existing LLM bias research often overlooks LGBTQ+ identity complexity and lacks benchmarks grounded in harms experienced by affected communities.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    WinoQueer uses a community-in-the-loop method to create a human-created, human-audited benchmark from LGBTQ+ community survey responses.

    Source: §1 · p. 1, §3.1 · pp. 2–3, §3.2 · pp. 3–4

  • Results

    Popular off-the-shelf LLMs generally exhibit considerable anti-queer bias, while finetuning on relevant community data somewhat mitigates it.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Community-driven, harms-grounded benchmark development provides a blueprint for studying bias affecting other marginalized communities.

    Source: §1 · p. 1

Abstract

from arXiv · show

We present WinoQueer: a benchmark specifically designed to measure whether large language models (LLMs) encode biases that are harmful to the LGBTQ+ community. The benchmark is community-sourced, via application of a novel method that generates a bias benchmark from a community survey. We apply our benchmark to several popular LLMs and find that off-the-shelf models generally do exhibit considerable anti-queer bias. Finally, we show that LLM bias against a marginalized community can be somewhat mitigated by finetuning on data written about or by members of that community, and that social media text written by community members is more effective than news text written about the community by non-members. Our method for community-in-the-loop benchmark development provides a blueprint for future researchers to develop community-driven, harms-grounded LLM benchmarks for other marginalized communities. Note: This version corrects a bug found in evaluation code after publication. General findings have not changed, but tables 5 and 6 and figure 1 have been corrected.

1 Introduction

WinoQueer addresses gaps in LLM bias evaluation by grounding an anti-LGBTQ+ benchmark in harms identified by LGBTQ+ community members. It reports substantial anti-queer bias across popular models and partial mitigation through community-relevant finetuning.

  • Existing bias benchmarks often overlook LGBTQ+ identity, intersectionality, less observable characteristics, and real-world harms.Prior work frequently treats identities as discrete categories and leaves harms insufficiently defined or grounded in affected communities’ experiences.Source: §1 · p. 1
  • WinoQueer is a community-sourced benchmark designed to detect stereotypes that have caused harm to specific LGBTQ+ subgroups.Its development uses a community-in-the-loop method grounded in expressed community needs.Source: §1 · p. 1
  • Finetuning benchmarked models on corpora about or by the LGBTQ+ community partially mitigates their anti-queer bias.Source: §1 · pp. 1–2
  • Baseline evaluations show significant anti-queer bias across BERT, RoBERTa, ALBERT, BART, GPT2, OPT, and BLOOM models.Source: §1 · p. 2

2 Related Work

Prior NLP bias research has often simplified gender and queerness, overlooked intersectionality, and lacked grounding in harms experienced by affected communities. WinoQueer fills this gap with a benchmark focused specifically on homophobic and transphobic bias and developed through large-scale LGBTQ+ community input.

  • Gender-bias studies frequently omit explicit theorization, intersectionality, inclusivity, and transgender, nonbinary, and intersex people.Source: §2 · p. 2
  • Existing anti-queer benchmarks often treat queerness as binary or assume identical stereotypes harm every LGBTQ+ subgroup.Such designs cannot measure identity-specific harms including transmisogyny, biphobia, and lesbophobia.Source: §2 · p. 2
  • Many NLP bias studies lack normative reasoning about why system behaviors are harmful and to whom, motivating community-informed research.General-public crowdsourcing and volunteer approaches do not necessarily provide perspectives from the specific affected communities.Source: §2 · p. 2
  • WinoQueer addresses these gaps with a homophobic and transphobic bias benchmark developed through a large-scale survey of actual queer and trans people.This distinguishes it from broader participatory datasets whose representation of the LGBTQ+ community is uncertain.Source: §2 · p. 2

3 Methods

The methods build WinoQueer from LGBTQ+ community input, convert reported harms into audited counterfactual sentence pairs, and evaluate and debias diverse language models using community-related text.

3.1 Queer Community Survey

  • 295 LGBTQ+ participants reported harmful stereotypes, with harm defined broadly to include psychological discomfort, violence, discrimination, bullying, financial impacts, and lost opportunities.Source: §3.1 · p. 3
  • Community survey responses were manually coded into discriminatory predicates, grounding benchmark templates in stereotypes reported by affected people.Source: §3.1 · p. 2, §3.2 · p. 3

3.2 WinoQueer Template Creation

  • Each sentence is paired with counterfactuals replacing LGBTQ+ descriptors with corresponding straight, heterosexual, cisgender, or cis descriptors.Source: §3.2 · p. 4
  • WinoQueer combines templates, names or pronouns, identity descriptors, and predicates into 45,540 human-created and human-audited sentence pairs.Source: §3.2 · pp. 3–4

A Note on Terminology

  • The terminology design uses “lesbian” for queer women and “gay” specifically for gay men to examine gendered and intersectional stereotypes.Source: A Note on Terminology · p. 4

3.3 Data Collection

  • For debiasing, models were fine-tuned separately on QueerNews and QueerTwitter, with models evaluated using pseudo-log-likelihood bias scores adapted for masked and autoregressive architectures.Source: §3.3 · p. 4, §3.4 · p. 5, §3.5 · p. 6

4 Results and Discussion

Off-the-shelf models show substantial anti-queer bias across architectures, sizes, and LGBTQ+ subgroups, while finetuning reduces bias overall but unevenly. QueerTwitter generally outperforms QueerNews as a debiasing source, although subgroup effects vary.

4.1 Off-the-shelf WinoQueer Results

  • The average off-the-shelf bias score was 66.50, indicating models associated toxic stereotypes with queer people about twice as often as with straight people.Source: §4.1 · p. 6
  • All 20 models showed anti-queer bias, ranging from 55.93 for ALBERT-xxl-v2 to 79.83 for BART-base.Source: §4.1 · p. 6, Table 5 · p. 7
  • WinoQueer bias was only weakly correlated with model size, with no correlation within masked or autoregressive groups, suggesting architecture was more predictive.Source: §4.1 · p. 7
  • Bias varied across identity groups: Queer had the lowest average score at 60.03, while Asexual had the highest at 75.85.Source: §4.1 · p. 7

4.2 Finetuning for Debiasing Results

  • QueerNews finetuning reduced WQ bias by 10.28 points on average, while QueerTwitter reduced it by 17.98 points across 16 models.Source: Table 6 · p. 8
  • Finetuning was most effective for general LGBTQ+ stereotypes and less effective for smaller groups, while QueerTwitter generally outperformed QueerNews except for queer and nonbinary groups.Source: §4.2 · p. 8
  • Evaluation was separated by identity, but finetuning data was not, and individual-model effects varied widely despite the reported subgroup averages.Source: §4.2 · p. 8

5 Conclusions

WinoQueer is a community-grounded benchmark for measuring anti-queer and anti-trans bias in LLMs, with a participatory development method intended for broader use. Results show substantial bias across models and improvement after finetuning, especially with community-authored Twitter data.

  • WinoQueer was developed through a large LGBTQ+ survey, grounding its benchmark in real-world harms and queer people’s experiences.Source: §5 · p. 8
  • The community-in-the-loop benchmark-development method is presented as extensible to other marginalized communities.Source: §5 · p. 8
  • Across 20 models, off-the-shelf systems showed substantial anti-LGBTQ+ bias, with autoregressive models generally more biased than masked models and no significant parameter-count correlation.Source: §5 · p. 8
  • Finetuning on queer-related news or queer-authored Tweets improved WQ bias scores, with QueerTwitter generally more effective than QueerNews.Source: §5 · p. 9
  • The authors recommend auditing models with both intrinsic WinoQueer-style metrics and extrinsic task-level fairness evaluations.Source: §5 · p. 9

Limitations

The study's limitations constrain how broadly WinoQueer represents LGBTQ+ experiences, the linguistic data, generated stereotypes, and model-evaluation results. These boundaries include sampling bias, incomplete identity coverage, dataset-quality issues, calibration uncertainty, and limited model scale and availability.

Community Survey

  • Community Survey: Volunteer sampling, English-only recruitment, adult-only participation, and underrepresentation of several demographic and geographic groups limit generalizability to the broader LGBTQ+ community.The survey does not represent the entire queer community, and queer youth, queer elders, several racial groups, and the Global South are insufficiently represented.Source: Community Survey · pp. 9–10

Finetuning Data Collection

  • Finetuning Data Collection: Media Cloud produced fewer results than Twitter, excluded January–May 2022 news articles, and hashtag sampling may include irrelevant or spam tweets.Source: Finetuning Data Collection · p. 10

Template Creation

  • Template Creation: WinoQueer covers only broad gender and sexual-orientation categories, omits intersectional biases, underrepresents non-European names and pronouns, and relies on a small template set.A high WinoQueer score indicates encoded homophobic or transphobic stereotypes, but a low score does not establish their absence.Source: Template Creation · p. 10

Evaluation and Finetuning

  • Evaluation and Finetuning: Similar but nonidentical scoring functions may evaluate masked and autoregressive models differently, while scores below 50 have an unresolved interpretation.The study also evaluates only open-source models, with baselines up to 7.1 billion parameters and finetuned models up to 1.5 billion parameters.Source: Evaluation and Finetuning · p. 10

A Demographics of Survey Respondents

The survey-demographics tables summarize respondents' self-identified gender, sexual orientation, race or ethnicity, age, and country of residence. Several questions allowed multiple selections or were answered by fewer than all 295 participants.

  • Table 7 reports respondents' self-identified genders, with totals not summing to 100% because participants could select multiple options.Source: Table 7 · p. 12
  • Table 8 reports respondents' self-identified sexual orientations, also allowing multiple selections so percentages need not sum to 100%.Source: Table 8 · p. 12
  • Table 9 reports self-identified race and ethnicity, with 228 of 295 participants answering the question.Source: Table 9 · p. 12
  • Table 10 reports age ranges for 185 of the 295 survey participants.Source: Table 10 · p. 12
  • Table 11 reports countries of residence for 194 of the 295 survey participants.Source: Table 11 · p. 13
Loading 2306.15087v2…