PaperlayerView exact revision on arXiv

Source-linked AI summary

They, Them, Theirs: Rewriting with Gender-Neutral English

Tony Sun, Kellie Webster, Apu Shah, William Yang Wang, Melvin Johnson

arXiv:2102.06788v1cs.CL

TL;DR

Much prior work on gender identity assumes binary gender, while singular they is increasingly used in connection with non-binary identity. This paper defines gendered-to-neutral rewriting and builds a benchmark; it reports training a model to produce they-forms without human-labeled data.

  • Problem

    Much prior work on gender identity assumes binary gender, while the singular they is emerging as a way to associate with non-binary identity.

    Source: §1 · p. 1

  • Method

    The paper defines a gendered-to-neutral rewriting task and builds a human-curated evaluation benchmark of 500 sentence pairs from five domains.

    Source: §1 · p. 1

  • Results

    The authors show how to train a model to produce they-forms in English without human-labeled training data.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The task aims to support users in self-identifying preferred pronouns and interacting with technology consistently with their social identity.

    Source: §7 · p. 5

    Domain mismatch between clean Wikipedia training sentences and noisy Twitter and Reddit tests may exacerbate Seq2Seq models’ known robustness problems.

    Source: §5 · p. 4

Abstract

from arXiv · show

Responsible development of technology involves applications being inclusive of the diverse set of users they hope to support. An important part of this is understanding the many ways to refer to a person and being able to fluently change between the different forms as needed. We perform a case study on the singular they, a common way to promote gender inclusion in English. We define a re-writing task, create an evaluation benchmark, and show how a model can be trained to produce gender-neutral English with <1% word error rate with no human-labeled data. We discuss the practical applications and ethical considerations of the task, providing direction for future work into inclusive natural language systems.

1 Introduction

Much prior work on gender identity assumes a binary, while the singular they is increasingly used in connection with non-binary identity. This paper studies gendered-to-neutral rewriting, builds an evaluation benchmark, and describes applications and ethical considerations.

1 Introduction

  • The broader motivation is to consider how language models are used and who uses them, alongside research on encoded social biases, downstream harms, and training-corpus distributions.Source: §1 · p. 1
  • The task asks a model to understand and fluently change gendered forms into neutral forms, including changing a sentence from a he-pronoun form to its they-pronoun counterpart.The example changes “His dream” to “Their dream” and “he grows” to “they grow.”Source: §1 · p. 1
  • The task excludes sentences with more than one human entity to avoid introducing ambiguity.Source: §1 · p. 1
  • The authors build a human-curated benchmark of 500 gendered and gender-neutral sentence pairs across five domains, and train a model to produce they-forms without human-labeled training data.The domains are Twitter, Reddit, news articles, movie quotes, and jokes.Source: §1 · p. 1
  • The underlying goal is to support users in self-identifying their preferred pronouns and interacting with technology consistently with their social identity.Source: §1 · p. 1
  • The paper identifies machine translation and augmented writing as applications, and considers ethical aspects of real-world deployment.Source: §1 · p. 2

2 Applications

A model that understands reference forms and can move between them could support machine translation, writing tools, and data augmentation.

2 Applications

  • For translation from a gender-neutral language into English, a gender-neutral alternative may be more appropriate when the source offers no information to choose he or she.The passage cites providing multiple alternatives as one approach.Source: §2 · p. 2
  • Augmented writing tools could use the model to suggest wording that reduces unintended targeting in job listings.Source: §2 · p. 2
  • The rewriter's sentences could serve as data augmentation input for models, given the reported efficacy of augmentation techniques for improving robustness.Source: §2 · p. 2

3 Challenges

Rule-based rewriting can handle predictable gendered words, but context-dependent pronouns and verb agreement make simple substitution inadequate. Seq2Seq models offer a sentence-level alternative, yet training them is difficult without naturally occurring parallel gendered and neutral text.

Rule-based Approach

  • Simple find-and-replace struggles with pronouns that have multiple neutral forms and with identifying and rewriting verbs.For example, “her” can become “their” or “them” depending on its grammatical role.Source: §3 · p. 2
  • Gendered-subject verbs may need pluralization while verbs tied to non-gendered subjects should remain unchanged.The example changes “grows” to “grow” but leaves “is” unchanged.Source: §3 · p. 2
  • A list of rules could handle verb changes, but would likely require many handcrafted features that are expensive to create and maintain.Source: §3 · p. 2

Seq2Seq Model

  • Seq2Seq rewriting is a natural fit because the task can be viewed as machine translation, but few naturally occurring parallel sentence pairs are available.The shortage of large parallel datasets complicates both model training and evaluation.Source: §3 · p. 2

4 Methodology

The methodology builds training pairs by rewriting filtered Wikipedia sentences, then trains a Seq2Seq model and evaluates both approaches on a manually annotated, balanced, five-domain test set.

4 Methodology

  • The pipeline filters 100 million Wikipedia sentences to obtain 15 million gendered sentences, rewrites them into parallel pairs, and trains a Seq2Seq model on those pairs.The end-to-end model is intended to generalize to unseen examples and turn the multi-step algorithm into one step.Source: §4 · p. 3
  • The evaluation uses 500 manually annotated sentence pairs drawn equally from five domains and balanced for masculine and feminine pronouns.The balance is intended to prevent a rewriter from scoring well by handling only one pronoun gender.Source: §4 · p. 3

4.1 Rewriting Algorithm

  • The rewriting algorithm combines regular expressions, dependency parsing for verb changes, and GPT-2 perplexity ranking to choose among context-dependent pronoun variants.It converts verbs linked to “(s)he” subjects to third-person plural while leaving verbs linked to non-gendered subjects unchanged.Source: §4.1 · p. 3

4.2 Model Details

  • The Transformer model uses augmented training data, and the authors report that both identity pairs and gender-inflected pairs improve performance.The training set uses a 70–30 split of rewritten and identity pairs.Source: §4.2 · p. 4

5 Results

On the annotated test set, both systems achieve over 99 BLEU and under 1% word error rate, though the algorithm is marginally better overall. Their error patterns and performance also vary with sentence gender, domain, and the algorithm’s component errors.

5 Results

  • Over 99 BLEU and less than 1% word error rate: both the algorithm and model reach this level, with the algorithm marginally better on the test set.The source sentence serves as an identity-function baseline: only certain gendered terms and corresponding verbs should change, while other tokens remain unchanged.Source: §5 · p. 4
  • Nearly three-quarters of algorithm mistakes involve pronouns and verbs, compared with less than half of the model’s mistakes.Nearly half of the model’s mistakes instead involve rare tokens such as whitespace, emojis, and symbols.Source: Figure 3 · p. 4, §5 · p. 4
  • The algorithm can err when its dependency parser misclassifies a target verb’s part of speech or its language model ranks the wrong neutral variant lower in perplexity.Source: §5 · p. 4
  • The model may be disadvantaged by training on clean Wikipedia sentences and testing on noisy Twitter and Reddit text; it performs better on cleaner movie-quote and joke domains.The passage identifies rare-word robustness and domain mismatch as known Seq2Seq problems.Source: §5 · p. 4
  • Both systems tend to make more mistakes on feminine than masculine sentences; the authors say more naturally occurring feminine sentences would likely help the model.Around 70% of gendered Wikipedia sentences were masculine, and the algorithm can struggle to resolve “her” as “their” or “them” in context.Source: §5 · p. 4

6 Conclusion

The conclusion presents rewriting English gendered sentences into gender-neutral forms, focusing on gendered pronouns and singular they. It reports training without human-labeled data and points to further work on identity markers and morphologically complex languages.

6 Conclusion

  • The paper proposes rewriting gendered sentences into gender-neutral forms, focusing on English gendered pronouns and singular they.Source: §6 · p. 4
  • The authors show that a model can be trained without human-labeled data by automatically generating large amounts of data by rule.Source: §6 · pp. 4–5
  • The authors see potential for their lightweight methodology to scale to identity markers across languages, while identifying morphologically complex languages as a challenge for future work.Source: §6 · p. 5

7 Ethical Considerations

The stated aim is to support users in self-identifying their preferred pronouns, including non-binary pronouns.

7 Ethical Considerations

  • The task aims to support users’ self-identification of preferred pronouns, including non-binary pronouns, and explicitly does not aim to infer preferences that users have not stated.The authors do not prescribe one form as correct or preferred; they describe the desired capability as understanding and fluently changing between forms as needed.Source: §7 · p. 5

A Appendix

The appendix reports variation in rewriting results, describes the automatically generated training data and manually curated evaluation sentences, and lists evaluation metrics and a scope limitation.

A Appendix

  • The system is not suitable for generic text unless it has been preselected to contain a single individual entity.Source: A Appendix · p. 7
  • Results for converting he-pronoun and she-pronoun sentences to neutral forms vary with training data and task difficulty, as documented in Figure 3.The passage does not specify the direction or magnitude of the differences.Source: A Appendix · p. 7
  • Evaluation metrics include BLEU and WER.Source: A Appendix · p. 7
  • 15 million sentences filtered from 100 million English Wikipedia sentences containing single gendered entities were automatically rewritten using manual rules, SpaCy, and GPT-2.Source: A Appendix · p. 7
  • The evaluation uses manually curated gendered sentences represented equally across Twitter, Reddit, news articles, movie quotes, and jokes.Source: A Appendix · p. 7

Ethical Considerations

The ethical considerations frame rewriting as supporting a user's chosen form, not identifying gender or prescribing a form, while noting limits in entity and pronoun coverage.

Ethical Considerations

Loading 2102.06788v1…