Source-linked AI summary
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, Kai-Wei Chang
TL;DR
Language technologies commonly treat gender as binary, leaving non-binary identities and associated harms inadequately represented. The paper explains gender and language complexity, surveys non-binary people familiar with AI, and examines GloVe and BERT representations. It documents representational and allocational harms, including misgendering and cyclical erasure, and identifies system-wide involvement and monitoring as needed responses.
Problem
Research on language-model gender bias and task evaluation has primarily focused on binary gender, while non-binary harms and representation challenges remain insufficiently understood.
Method
The paper explains gender and language complexity, surveys non-binary people familiar with AI, and analyzes GloVe and BERT representations and downstream tasks.
Results
The paper identifies representational and allocational harms, documents misgendering and cyclical erasure, and demonstrates cases where popular language models reflect these concerns.
Takeaways & Limitations
Addressing these harms requires affected persons to be involved system-wide, alongside ongoing monitoring, transparent communication, feedback, human intervention, and recourse.
Takeaways & Limitations
The survey reaches specific demographic groups and should expand to diverse people, including those unfamiliar with AI or not fluent in English.
Abstract
from arXiv · showhide
Gender is widely discussed in the context of language tasks and when examining the stereotypes propagated by language models. However, current discussions primarily treat gender as binary, which can perpetuate harms such as the cyclical erasure of non-binary gender identities. These harms are driven by model and dataset biases, which are consequences of the non-recognition and lack of understanding of non-binary genders in society. In this paper, we explain the complexity of gender and language around it, and survey non-binary persons to understand harms associated with the treatment of gender as binary in English language technologies. We also detail how current language representations (e.g., GloVe, BERT) capture and perpetuate these harms and related challenges that need to be acknowledged and addressed for representations to equitably encode gender information.
1 Introduction
Language-model gender-bias research has largely centered binary gender, while non-binary people and harms remain insufficiently represented. This paper investigates those harms and how they appear in language representations and tasks.
- Research gap: Most social-bias studies focus on masculine and feminine stereotypes, and gender-sensitive task evaluations generally measure binary gender.This pattern appears across language-model studies and tasks including named entity recognition, coreference resolution, and machine translation.
- Research gap: Binary-gendered models and datasets can perpetuate cyclical erasure of non-binary identities through tainted examples, limited features, and sample-size disparities.The paper links these biases to societal non-recognition and limited understanding of non-binary genders.
- Prior mitigation: Existing mitigation efforts broaden task-specific datasets and metrics, but may mischaracterize non-binary genders as a single gender.The paper argues that these efforts would benefit from perspectives on harms reported by non-binary people.
- Paper approach: The paper investigates representational and allocational harms by explaining gender and language complexity and surveying non-binary people familiar with AI.It addresses challenges from limited or tainted data that were previously not well understood.
- Paper approach: It examines GloVe and BERT representations of non-binary-associated words and pronouns, showing how representational disparities can propagate misrepresentation, misgendering, and erasure.The analysis connects representation quality with downstream harms in language technologies.
2 Gender, Language, and Bias
Gender identity, expression, and sex are distinct, while gender and its linguistic expression are complex, variable, and culturally situated. English language technologies therefore need to represent greater diversity and flexibility than a binary framework permits.
- Gender concepts: The paper distinguishes gender identity from gender expression and sex, defining its focus as how individuals experience their own gender.These aspects do not always align according to Western cisnormative expectations.
- Gender concepts: Western gender discourse can describe similarity or difference from binary genders and relation to gender assigned at birth, including cisgender and transgender identities.Gender may fluctuate over an individual’s lifetime, making biologically essentialist assumptions problematic.
- Cultural scope: Non-binary genders include genders that do not conform to the Western binary, but Western English frameworks cannot accurately describe all non-Western non-cis identities.The paper explicitly limits its treatment of non-binary genders to an English-language framework.
- Language: English pronouns can be central to gender identity, but pronouns do not map bijectively to gender identities.Non-binary people may use multiple pronoun sets or vary their pronouns across contexts.
- Language: Lexical gender conveys gender nonreferentially through terms such as “mother” and “Mr.”, while non-binary honorifics and neutral terms remain recent and sparse in text.The paper gives “Mx.” and “partner” as examples of non-binary or gender-neutral language.
- Implications: Equitable language technologies must capture the full diversity and flexibility of gender and the language surrounding it.The paper describes bias as skewed, undesirable associations that can contribute to representational or allocational harms.
3 Harms
The paper surveys non-binary people about harms across language tasks and applications, focusing on misgendering, erasure, and barriers to inclusive modeling. Respondents identified undesirable outcomes across tasks, with machine translation perceived as especially severe.
- Harms: The paper focuses on misgendering and erasure as primary harms from binary constructions of gender in language technologies.These harms are examined in language-processing tasks and human-centered applications.
- Misgendering: Misgendering occurs when a gendered term does not match someone’s gender identity, including forced binary choices and model defaults to binary terms.Models can misgender non-binary people even when their pronouns are explicitly provided.
- Erasure: Erasure can invalidate or obscure non-binary identities, including by assigning people to binary gender categories based on names.The paper also describes erasure through stereotypes about non-binary communities.
- Erasure: Binary-trained language applications can cyclically reproduce real-world misgendering and erasure, which may then be treated as truth and reinforced in later writing.The paper presents this as a cycle linking training data, model outputs, perceived authority, and subsequent authorship.
- Survey findings: Above 84% of respondents identified undesirable outcomes for non-binary genders in each of NER, coreference resolution, and machine translation.Respondents perceived the highest harm severity in machine translation, a task more commonly used by the population at large.
- Applications: Respondents described risks including social-media outing or deadnaming, healthcare failures involving gender history and identity, and systems that cannot handle non-binary language or neopronouns.Examples included automated summarization, language generation, speech-to-text, machine translation, and automated gender recognition.
- Barriers and limitations: Survey respondents identified tainted examples, discarded or unsampled data, unavailable labels, and pressure to simplify systems as barriers to inclusion.The survey involved 19 AI-familiar participants and had limited demographic diversity, constraining conclusions from the sample.
4 Data and Technical Challenges
Language technologies inherit severe data skews and representational gaps around non-binary gender, which appear in GloVe and BERT and can contribute to misgendering and erasure.
- 4.1 Dataset Skews: Non-binary pronouns have sparse or semantically unsuitable corpus evidence, while available Non-Binary Wiki narratives are short, structurally limited, and Western-culture dominated.These properties further sparsify diverse narratives of non-binary persons.
- 4.2 Text Representation Skews: GloVe gives xe and ze nearest neighbors consisting largely of acronyms and Polish words rather than semantically meaningful pronoun neighbors.This lack of meaningful encoding reflects disparities in their occurrences in the training data.
- 4.2 Text Representation Skews: A WEAT score of 0.916 indicates disparate sentiment associations, with non-binary-associated words receiving negative or derogatory nearest neighbors.Examples include dishonest, careless, unkind, arrogant, negrito, and Fasiq near non-binary-associated terms.
- 4.2 Text Representation Skews: BERT’s pronoun-prediction evaluation shows high accuracy for he and she, lower accuracy for they, and an even larger drop for xe and ze.The evaluation varies templates, names, verbs, subjects, objects, purposes, and possessive-pronoun cues before predicting a masked pronoun.
5 Discussion and Conclusion
The paper documents harms arising from binary gender modeling and shows how language technologies can reproduce discrimination, misgendering, and cyclical erasure. It argues that inclusive solutions require affected non-binary people to participate throughout system development and ongoing oversight.
- Discussion and Conclusion: The work documents representational and allocational harms voiced by non-binary participants and demonstrates cases where popular language models reproduce these concerns.The paper examines how such harms arise in language representations and downstream tasks.
- Discussion and Conclusion: Current efforts remain limited because neopronouns may lack sufficient real-world data and treating non-binary genders as one monolithic category can be harmful.The authors also question whether gender should be modeled as discrete quantities.
- Discussion and Conclusion: The paper frames inclusively modeling gender in language representations and tasks as an interdisciplinary challenge rather than a problem solvable by a quick patch.It calls for approaches that address representation and task-level harms together.
- Discussion and Conclusion: Bottom-up participation by affected people, including annotation and human-in-the-loop mechanisms, is presented as necessary for viable solutions.The proposed system-wide involvement extends beyond model training alone.
- Discussion and Conclusion: The authors identify monitoring, transparent communication of performance and harms, recourse, feedback, and human intervention as critical safeguards.These measures are intended to support non-binary people when language technologies cause harm.
6 Broader Impact and Ethics
The paper treats fairness and inclusivity in NLP as ethical requirements because gender technologies can cause severe harms, especially for people outside binary categories. Its survey procedures emphasized informed consent, non-leading questions, and protection of participant identities.
- Broader Impact and Ethics: The survey received Institutional Review Board Exempt status and required signed informed consent because it posed minimal risk and was conducted online.The study did not involve treating human subjects, according to the passage.
- Broader Impact and Ethics: Survey questions were non-leading, and responses were analyzed in aggregate to protect participants’ identities.Quoted or analyzed text could not be traced back to an individual.
- Broader Impact and Ethics: The paper states that gender modeling in language applications should fairly reflect gender identity and expression.It links failure to do so with severe harms for people who do not subscribe to binary gender.
- Broader Impact and Ethics: The paper provides the full survey, explains the rationale for each question, and qualitatively analyzes the responses.This documentation supports scrutiny of how participant perspectives were elicited and interpreted.
A.1 Demographic information
The survey sampled gender-diverse respondents, many of whom used multiple pronoun sets, but its demographic composition was heavily Western, white, and AI-familiar. The authors therefore caution that the sample severely limits the conclusions that can be drawn.
- Pronouns: 31.6% of respondents used more than one set of pronouns, and the survey allowed multiple pronoun selections.Examples included she/her combined with xe/xem.
- Pronouns: The survey collected non-English pronoun information because pronouns can be less central to gender identity in languages without referential gender or with infrequent pronoun use.The authors identify these data as useful for future research on non-English language technologies.
- Pronouns: Respondents reported pronouns in several languages, including Swedish, French, Mandarin, and German.One response listed “hen” in Swedish, “hän” in Finnish, and no pronouns in Japanese.
- Gender: The sample included respondents identifying with multiple genders and achieved a Western gender-diverse sample.Free-response answers included agender, nonbinary, genderqueer, genderfluid, and identities between categories.
- Ethnicity and nationality: All but two respondents identified as white or Caucasian, no respondents were Black, Indigenous, or Latinx, and two were people of color.The demographic distribution was therefore not racially representative of broader non-binary communities.
- Limitations: The authors state that this sample composition severely limits the conclusions that can be reached and plan to diversify future outreach.They also note that all respondents were familiar with AI and that this may correlate with privilege and socioeconomic status.
A.2 Harms in Language Tasks
Survey responses and task demonstrations identify representational and allocational harms when NLP systems mishandle non-binary names, pronouns, and gendered language. These harms include misgendering, dehumanization, erasure, and exclusion from opportunities or services.
- Named Entity Recognition: NER can misrecognize non-binary names and pronouns, reinforcing stereotypes that their names or pronouns are strange, difficult, or non-human.Chosen names may overlap with common nouns, use uncommon orthography, or consist of a single letter.
- Named Entity Recognition: NER failures can cause allocational harm when resume screening rejects applications or identity verification incorrectly labels non-binary people.Respondents described systems failing to recognize chosen names or treating identities as invalid during verification.
- Coreference Resolution: Coreference systems may misgender people, erase neopronouns, and link singular “they” to the wrong entity instead of the person named earlier.One demonstration linked “they” with “team” rather than “Alice,” despite the intended singular-person reference.
- Coreference Resolution: Coreference errors can affect rankings, housing, financial aid, and legal evidence by losing or altering references to non-binary people.Respondents described missed citations, incorrect pronouns in leases, ineligible aid flags, and undercounted discrimination cases.
- Machine Translation: Machine translation can impose binary gender, misgender non-binary people, treat names or pronouns as objects, and erase neopronouns.Gender-neutral source forms may become stereotyped “he” or “she” translations, while neopronouns may be represented as unknown tokens.
Social Media
Respondents described social-media and related language-model applications that can misclassify, misrepresent, or exclude non-binary people. Reported risks span content moderation, identity inference, verification, and downstream access to services.
- Social Media: Social-media moderation may flag LGBTQ+ content at higher rates while failing to identify hateful language targeting non-binary people.Respondents also connected gender inference from names or other attributes with incorrect pronouns.
- Social Media: Entity-linking systems may out or deadname people, while identity verification can interpret non-binary identities as fake or non-human.These risks were described for social-media and finance-related verification contexts.
- Social Media: Autocomplete may suggest only binary pronouns or predictions aligned with gender stereotypes.This can shape generated language around binary assumptions.
- Social Media: Language models could misgender patients or deny insurance claims when diagnosis and gender or pronouns appear mismatched.Respondents emphasized that healthcare requires engaging with both gender history and current identity.
- Social Media: Incorrect handling of singular “they” could cause communications to be flagged, achievements to be misattributed, or government applications to be rejected.The examples include false or incomplete communications, attribution to group work, and government services rejecting applications based on language analysis.
Education
The survey identifies educational, institutional, and societal pathways through which language technologies can erase non-binary people or intensify transphobia. Respondents also connect these harms to limited representation, skewed data, and intersecting identities.
- Education: Educational tools may mark singular “they,” neopronouns, and creatively gendered language as wrong or ungrammatical.This risk concerns automated grading and educational language-model applications.
- Education: Automated summarization may tag non-binary people as non-human, making their achievements less likely to be summarized accurately.Respondents described this as a form of erasure and invisibility.
- Education: Current models may generate binary pronouns and gendered statements while omitting non-binary language, contributing to non-binary erasure.Examples include generating “he/him” and “she/her” while not generating “they/them,” “ze/hir,” or “She is nonbinary.”
- Education: Classifying gender as binary, a third gender, or an ambiguous category can erase the diversity of non-binary identities.Respondents specifically rejected treating “nonbinary” as a single entity or identity type.
- Barriers and Data: Respondents attributed limited data partly to insufficient developer diversity, limited knowledge, hegemonic internet sources, and historical erasure.They also reported that non-binary data may be discarded as outliers or omitted from sampling.
- Intersecting Identities: 89.5% of respondents said harms can be compounded for non-binary people with intersecting identities.Examples include non-Western names, race, immigration status, English fluency, neurodivergence, and disability.
B Dataset Skews
Pronoun usage is not a reliable proxy for gender, and non-binary pronouns occur far less often in web-derived text than binary pronouns. These distributional differences create a dataset-skew challenge for language representations.
- Dataset Skews: Pronoun usage is not always meaningful for gender, and pronoun distributions differ across genders.The passage cautions that pronouns cannot be treated as a direct gender mapping.
B.1 Representation Skews
The paper examines gender representation skews in GloVe and tests whether PCA can capture a gender subspace spanning binary and non-binary terms. The resulting all-gender subspace remains more aligned with binary than non-binary gender.
- GloVe representation skews: GloVe nearest neighbors often reproduce social biases, including derogatory associations for non-binary terms.The analysis examines nearest neighbors for binary and non-binary possessive pronouns and terms.
- Occupation associations: Binary-gender occupation associations are not very relevant to non-binary-gendered persons and their biases.
- Subspace analysis: PCA is extended across binary, non-binary, and combined word sets to test whether a general all-gender subspace can be captured.The binary set includes gendered pronouns and terms; the non-binary set includes they, them, xe, ze, and related forms.
B.3 BERT experiments
The BERT experiments construct controlled pronoun datasets and evaluate representations using masked-token contexts, sentence templates, classifiers, and similarity-based word sets. They distinguish singular and plural uses of they while examining binary and non-binary language patterns.
- Dataset construction: The experiments create a balanced, labeled dataset by varying names and pronouns across constructed sentences.The name set contains over 900 names.
- Dataset construction: Plural they examples come from Wikipedia contexts mentioning two or more persons, while singular they examples are sampled from the Non-Binary Wiki.Sentences are manually annotated to confirm correct pronoun usage.
- BERT representation analysis: BERT predicts masked-token representations for sentences containing he, she, or singular they.The procedure follows corresponding datasets collected from Wikipedia.
- Misgendering experiments: Sentence templates test whether BERT propagates misgendering across appointments, classes, homes, possessions, and interpersonal contexts.The templates vary names and pronouns while using masked positions for prediction.
- Evaluation: The experiments also compare occupation similarities, pleasantness associations, BERT representations, and classifier confusion matrices.These analyses are represented through adjective sets, word-set definitions, WEAT scores, and classifier evaluations.