PaperlayerView exact revision on arXiv

Source-linked AI summary

Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Models

Harnoor Dhingra, Preetiha Jayashanker, Sayali Moghe, Emma Strubell

arXiv:2307.00101v1cs.CLcs.AI

TL;DR

The paper examines whether LLMs produce lower-regard portrayals of queer people when prompted with sexual-identity triggers. It compares generated text across identity prompts and applies SHAP-guided post-hoc rewriting, finding that the method can increase regard while retaining context.

  • Problem

    The extent of bias in LLM-generated portrayals of queer people is not well measured, despite the need to quantify bias to assess reduction efforts.

    Source: §1 · p. 1

  • Method

    The authors compare outputs from gender-neutral biographies prompted with sexual-identity triggers and use a regard classifier and SHAP analysis to identify and rewrite low-regard wording.

    Source: §1 · p. 2, §2.3 · p. 3, §5.1 · p. 5

  • Results

    Queer-triggered outputs have lower regard than straight-triggered outputs, and SHAP-guided post-hoc rewriting can raise regard while preserving contextual information.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The findings suggest post-hoc rewriting is a promising approach to reducing measured representational bias in this setting while retaining contextual meaning.

    Source: §1 · p. 2, §7 · p. 8

    The method targets explicit sexual-identity triggers and may not address implicit bias within prompts.

    Source: §8 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) are trained primarily on minimally processed web text, which exhibits the same wide range of social biases held by the humans who created that content. Consequently, text generated by LLMs can inadvertently perpetuate stereotypes towards marginalized groups, like the LGBTQIA+ community. In this paper, we perform a comparative study of how LLMs generate text describing people with different sexual identities. Analyzing bias in the text generated by an LLM using regard score shows measurable bias against queer people. We then show that a post-hoc method based on chain-of-thought prompting using SHAP analysis can increase the regard of the sentence, representing a promising approach towards debiasing the output of LLMs in this setting.

1 Introduction

The paper asks whether LLMs show measurable representational bias against queer people and whether post-hoc rewriting can mitigate it while preserving context. It compares outputs prompted with gender-neutral biographies and sexual-identity triggers, then uses SHAP analysis and a regard classifier to identify and replace low-regard wording.

1 Introduction

  • The authors identify a measurement gap: bias in LLM outputs is known, but its extent is not well measured, and assessing reduction requires quantifying detected bias.Source: §1 · p. 1
  • The study asks whether a pretrained LLM exhibits quantifiable bias against queer people and whether post-hoc debiasing can reduce it while preserving context.Source: §1 · p. 1
  • The comparison uses gender-neutral biographies with sexual-identity trigger words to prompt the LLM and examine regard in the generated text.The biographies provide varied contextual information about subjects.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Queer identity triggers yield lower-regard outputs than straight triggers, with qualitative differences in which aspects of a subject’s life the LLM describes.The paper notes that queer-triggered outputs may emphasize queer struggle and philanthropy rather than business success, while valuing recognition of queer struggle.Source: §1 · p. 2
  • The proposed post-hoc method uses SHAP analysis and a regard classifier to find low-regard words and rewrite text while seeking to preserve contextual information.The authors frame this as text-to-text style transfer and aim to retain recognition of queer struggle.Source: §1 · p. 2

2 Related Work

The related work motivates a pipeline combining gender-neutral biographies, context-rich identity-triggered prompts, fairness analysis, and post-hoc text rewriting. The authors distinguish their approach from prior work through varied contexts for bias detection and a focus on rewriting outputs with fixed model weights.

2.1 Generating gender-neutral text

  • Prior gender-neutralization methods include rule-based and neural rewriting; this work uses a rule-based method with grammatical and semantic correction to neutralize biographies.Earlier neural style-transfer approaches can alter sentence context, which is undesirable for this task.Source: §2.1 · p. 2

2.2 LLM prompting for bias detection

  • Unlike template-focused bias studies, this work combines identity trigger words with varied contexts and performs text-to-text generation for autoregressive LLMs.Prior prompt datasets address masked or autoregressive models, while this study measures outputs across sexual-identity triggers and contextual information.Source: §2.2 · pp. 2–3

2.3 Fairness analysis of LLM output

  • The fairness analysis examines frequent output words and uses pointwise mutual information to compare queer-triggered outputs with corresponding straight-triggered outputs.It quantifies representational bias with regard, which the authors describe as measuring social perception and as preferable to sentiment analysis for this purpose.Source: §2.3 · p. 3

2.4 Debiasing LLM Output

  • Rather than debiasing through model training, this work targets post-hoc rewriting with fixed weights and frames the task as text-to-text style transfer to raise regard while retaining context.The approach draws on prior style-transfer work and SHAP-guided replacement of words associated with an unwanted style.Source: §2.4 · p. 3

3 Bias Statement

The paper studies representational bias in portrayals of queer people, noting that LLM outputs can emphasize struggle over other facets of an individual. Its analysis covers four specified identity labels, and it observes that regard differences primarily reflect low-regard wording that is not identity-specific.

4 Bias Statement

  • The authors focus on representational bias against queer people, including portrayals that emphasize struggle over other facets of an individual’s personality.Source: §3 · p. 3
  • The study covers straight men, straight women, gay men, and lesbian women as a proof of concept, not the full range of sexual and gender identities.The authors state that this limited scope reflects their computational budget and is not intended to disregard other identities.Source: §3 · p. 4
  • Quantitative regard differences primarily stem from words that diminish regard for queer people and tend not to be identity-specific.Source: §3 · p. 4

4 Data

The data section uses WikiBio biographies to create varied, gender-neutral prompts about individual people. Sexual-identity triggers are appended to those prompts to form the contextual corpus used in the study.

4 Data

  • The authors draw biographies from WikiBio, a dataset of around 700,000 Wikipedia biographies, and randomly select approximately 200 biographies with four to nine sentences.The biographies provide personas and contextual information about individual people.Source: §4 · p. 4
  • The biographies focus predominantly on one individual, allowing general information about that person to serve as prompt context before sexual-identity triggers are appended.Source: §4 · p. 4
  • Appending sexual-identity triggers to gender-neutral biographies creates a diverse contextual prompt corpus for the study.Source: §4 · p. 4

5 Proposed Approach

The proposed approach divides its methodology into bias detection and post-hoc debiasing. It uses gender-neutral biographies to study identity-conditioned outputs, then frames debiasing as text-to-text style transfer.

5.1 Bias Detection

  • The detection pipeline neutralizes WikiBio biographies, adds sexual-identity cues (plus an untriggered control), and analyzes generated completions.Gendered terms are replaced with neutral forms and famous-person names with a <PER> token.Source: Figure 2 · p. 4, §5.1 · pp. 4–5
  • Word clouds show frequent output words by prompt type, while PMI identifies words occurring more often with particular identity labels.PMI is calculated between output words and labels for control, straight-man, straight-woman, gay-man, and lesbian-woman prompts.Source: §5.1 · p. 5
  • TF-IDF sentence embeddings are plotted with t-SNE, where point proximity indicates embedding similarity; average cosine similarity compares triggered outputs with control outputs.Source: §5.1 · p. 5
  • Regard scores compare identity-triggered outputs with control, measuring use of words that convey socially perceived high or low regard.Source: §5.1 · p. 5

5.2 Debiasing the LLM Output

  • Because queer-trigger outputs had lower regard, the style-transfer goal is to retain meaning—including acknowledgment of queer struggle—while raising regard.Source: §5.2 · p. 5
  • SHAP identifies low-regard words; chain-of-thought prompting asks for an explanation, then uses it to rephrase the sentence while preserving meaning and replacing those words.Source: §5.2 · pp. 5–6, Figure 3 · p. 6

6 Results

The results show lower regard for queer-triggered outputs than for straight or control outputs. A post-hoc, SHAP-based text-to-text rewrite raises regard while keeping the resulting sentences nearly similar.

6.1 Detecting Bias in the Language Model

  • For the business-biography example, queer-triggered outputs foregrounded queer struggle, and the lesbian-woman output deemphasized the subject’s business mindset.The lesbian-woman output instead focused on philanthropy and promoting inclusivity.Source: §6.1 · p. 6
  • Straight men and women had regard similar to the control, while gay men and lesbian women had significantly lower regard.Table 1 reports cosine similarity and regard-score differences between sexual-identity-triggered sentences and control sentences.Source: §6.1 · p. 6, Table 1 · p. 6

6.2 Debiasing the LLM Output

  • The debiasing approach uses SHAP to identify low-regard words, with the example figures showing words that push regard higher or lower.The paper gives net regard examples of 0.95 and -0.85; discrimination and challenges are cited as words that lower regard.Source: §6.2 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7
  • After the LLM’s explanation for a low-regard sentence was judged sensible, the text-to-text style-transfer formulation changed its words or phrases accordingly.The passage describes this as a response to the explanation of the low-regard sentence.Source: §6.2 · p. 6
  • Debiasing significantly increased regard while the original low-regard and debiased sentences remained nearly similar and overlapped in Figure 7.The baseline was simply prompting the LLM to increase regard.Source: §6.2 · p. 6

6.1 Detecting Bias in the Language Model

  • In the embedding plot, control and straight outputs were close, whereas gay-men and lesbian-women outputs were closer to each other and farther from control.The paper reports average cosine similarity between identity-triggered outputs and control in Table 1.Source: §6.1 · p. 6, Figure 6 · p. 7

7 Discussion

The discussion reports that sexual-identity trigger words are associated with lower regard for queer outputs, even across changed contexts. It describes post-hoc rewriting aimed at replacing low-regard words while largely preserving sentence meaning.

7 Discussion

  • The measurement method requires explicit sexual-identity trigger words, limiting its analysis to language-marked bias.The authors say additional research is needed to validate the assumption that implicit identity cues would also yield lower regard for queer people.Source: §7 · p. 7
  • Figure 7’s substantial overlap between pre- and post-debiasing embeddings suggests high lexical similarity.The discussion also notes that trigger-word bias associations persist when context changes.Source: Figure 7 · p. 7, §7 · p. 7
  • Straight-identity points lie near controls, while queer points are farther away, supporting a heteronormative pattern in the output embeddings.The authors describe this as further evidence that the prevailing norm is heteronormative.Source: §7 · p. 7
  • Chain-of-thought prompting with SHAP analysis targets low-regard words for replacement while largely preserving sentence meaning.Table 2 illustrates words identified as driving lower regard and corresponding rephrased text.Source: §7 · p. 8, Table 2 · p. 8

8 Limitations

The limitations concern prompt scope and quality, the regard classifier used for SHAP analysis, and the study’s focus on four sexual identities.

8 Limitations

  • The method targets explicit trigger-word bias and may not address implicit bias already present in prompts.Gender-neutralized prompts can also remain semantically or grammatically incorrect and may retain implicit bias.Source: §8 · p. 8
  • The prompt premise that a biography should discuss only one person may not hold in some cases.The authors also describe the gender-neutralization method as imperfect.Source: §8 · p. 8
  • The regard classifier may exhibit shortcut learning, with “gay” sometimes correlating with a low-regard sentence.The authors identify examination of SHAP-detected low-regard words as an area for improvement.Source: §8 · p. 8
  • The study examines four sexual identities, which the authors say should not be treated as a new norm for a diverse, fluid spectrum.They emphasize respect for each individual’s unique experiences and identities.Source: §8 · p. 8

A Appendix

The appendix includes word clouds for control and sexual-identity-triggered outputs, as well as example LLM outputs.

A Appendix

Loading 2307.00101v1…