PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Gender Augmented Data for NLP

Nishtha Jain, Maja Popovic, Declan Groves, Eva Vanmassenhove

arXiv:2107.05987v1cs.CL

TL;DR

Gender-ambiguous conversational sentences in gender-inflected languages can have multiple equivalent alternatives, while NLP systems may present only one likely form. The paper builds gender-parallel Spanish data and trains an NMT rewriter to generate alternatives; automatic and manual analyses report promising performance, with tags shown as potentially helpful.

  • Problem

    In gender-inflected languages, short conversational sentences can leave gender unspecified, yet NLP systems often present only the most probable variant.

    Source: §1 · p. 1

  • Method

    The authors build gender-parallel examples with rules and train an NMT rewriter to generate gender alternatives for short conversational sentences.

    Source: §1 · p. 2, §3 · p. 3, §5 · p. 5

  • Results

    The approach showed promising performance in automatic and manual evaluation, and additional tags could help the NMT system; gold labels performed better than classifier-generated labels.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The rewriter is intended to enrich short conversational sentences with alternatives beyond first-person, pronoun-swapping, or animate-noun cases, while keeping reliance on linguistic tools low.

    Source: §7 · pp. 8–9

    The lack of large training sets, including insufficient industry-partner data, prevented training a state-of-the-art automatic gender rewriter.

    Source: §7 · p. 9

Abstract

from arXiv · show

Gender bias is a frequent occurrence in NLP-based applications, especially pronounced in gender-inflected languages. Bias can appear through associations of certain adjectives and animate nouns with the natural gender of referents, but also due to unbalanced grammatical gender frequencies of inflected words. This type of bias becomes more evident in generating conversational utterances where gender is not specified within the sentence, because most current NLP applications still work on a sentence-level context. As a step towards more inclusive NLP, this paper proposes an automatic and generalisable rewriting approach for short conversational sentences. The rewriting method can be applied to sentences that, without extra-sentential context, have multiple equivalent alternatives in terms of gender. The method can be applied both for creating gender balanced outputs as well as for creating gender balanced training data. The proposed approach is based on a neural machine translation (NMT) system trained to 'translate' from one gender alternative to another. Both the automatic and manual analysis of the approach show promising results for automatic generation of gender alternatives for conversational sentences in Spanish.

1 Introduction

The paper proposes generating gender-balanced alternatives for short conversational sentences, where sentence-level NLP systems may offer only the most probable gender variant. It uses rule-built gender-parallel data to train an NMT rewriter intended to cover several gender phenomena with little linguistic-tool and human effort.

1 Introduction

  • The approach aims to provide missing gender variants for conversational sentences and to create gender-balanced data for training NLP models.The authors connect this goal to systems that present only the most probable option when gender is unspecified in the sentence.Source: §1 · p. 1
  • The target includes alternatives involving both a person's gender and grammatical gender agreement with an object, rather than only one gender phenomenon.The introduction illustrates agreement with an object referred to earlier and agreement with the speaker or writer.Source: §1 · pp. 1–2
  • The pipeline extracts candidate segments using POS sequences, creates word variants with rules, then trains an NMT rewriter on the gender-parallel data.The rewriter is intended to generate gendered variants on unseen data without requiring preprocessing at use time.Source: §1 · p. 2
  • The rule-based data-building steps address the lack of readily available open-source gender-parallel training data, with minimal human effort and external linguistic tools.These steps still require language knowledge and a POS tagger.Source: §1 · p. 2

2 Related Work

Related work includes data augmentation and gender-aware machine translation, but prior re-gendering approaches target narrower phenomena or depend on extensive linguistic resources. The paper positions its approach as covering a broader set of gender alternatives in short sentences.

2 Related Work

  • Counterfactual data augmentation creates gender-alternative sentences, including through gender-word dictionaries and pronoun swapping.The cited examples include Lu et al. (2018) and Zhao et al. (2018).Source: §2 · p. 2
  • Zmigrod et al. (2019) use Markov Random Fields to enforce agreement after swapping animate nouns, relying on dependency and morphosyntactic tools.Their approach is described as limited to stereotypes related to animate nouns.Source: §2 · p. 2
  • Gender tags and added sentence context have been used in machine translation, with reported improvements in automatic scores and gender accuracy for English-to-Spanish translation.The cited work includes Vanmassenhove et al. (2019) and Basta et al. (2020).Source: §2 · p. 3
  • Habash et al. (2019) combine a gender classifier and NMT rewriter, using manually produced gender-parallel data for first-person singular alternatives.The present work instead targets adjectives, past participles, and pronouns whose referents are absent from the sentence.Source: §2 · p. 3

3 Generating gender-parallel data

The authors construct gender-parallel data by identifying re-genderable Spanish forms and applying rules to create alternatives. Their rules depend on word class and sentence structure, and target short sentences with at most one referent.

3 Generating gender-parallel data

  • The data-generation goal is to supply masculine-to-feminine and feminine-to-masculine pairs because suitable public gender-parallel datasets are unavailable.The authors first use linguistic knowledge and rules to generate enough parallel data for NMT training.Source: §3 · p. 3

3.1 Re-genderable word classes

  • The approach targets short sentences with at most one referent and cannot generate alternatives when agreement involves multiple entities requiring more than two variants.The paper gives a sentence with separate first-person and third-person referents as an uncovered case.Source: §3 · p. 3, §3.1 · p. 3
  • Past participles and adjectives are re-gendered when they express a referent's property, but not when a nearby noun or neutral demonstrative determines agreement.Participles following auxiliary haber are also excluded; isolated adjectives and participles can be re-gendered.Source: §3.1 · pp. 3–4
  • The rules change gendered clitics such as lo/la but leave neutral forms such as le unchanged, with exceptions for idiomatic uses and indirect objects.When two clitics are attached, only the gendered part is changed; paired adjectives or participles referring to the same entity are both re-gendered.Source: §3.1 · pp. 3–4

3.2 Adding gender variants by rules

  • POS-tag sequences indicate whether a referent is absent and an alternative should be added; rules then change the corresponding word ending and retain neutral sentences unchanged.A present referent noun means only one gender form is grammatically correct, while an absent referent permits an opposite-gender variant.Source: §3.2 · p. 4

4 Gender-parallel data

The authors build a large Spanish gender-parallel corpus from short OpenSubtitles segments and supplement evaluation with industry-partner data. The corpus includes neutral examples as well as re-genderable pairs, while the manually checked development and test data contrast with an unverified, potentially noisy training set.

4 Gender-parallel data

  • The OpenSubtitles pipeline filters segments to at most 10 words, extracts more than 1 million matching sentences, and creates a corpus of about 2.2 million gender-parallel sentences.Neutral sentences are included to teach the rewriter when not to generate alternatives as well as how to generate them.Source: §4 · p. 5
  • Development and test rewrites were manually corrected for about 6% of sentences and 1.5% of words, while the large training set was not manually verified and contained some noise.The development and test sets contain approximately 1,000 and 3,000 sentences, respectively.Source: §4 · p. 5
  • The approach's short-sentence assumption is explicitly limited to sentences of up to 10 untokenized words in the extracted OpenSubtitles candidate set.The paper notes that the POS-sequence approach would not generalize to longer sentences.Source: §4 · pp. 4–5
  • Industry-partner data included about 8,000 sentences with alternatives and 22,000 sentences revised manually, forming structured and unstructured test sets.The unstructured set contains POS sequences the described rules cannot handle and is used to estimate scalability.Source: §4 · p. 5

5 Neural Rewriter

The authors train an NMT rewriter on gender-parallel Spanish sentences and explore source tags for neutral versus re-genderable sentences. The classifier performs better on industry-partner test data than on OpenSubtitles in neutral-class precision and recall.

5 Neural Rewriter

  • The NMT rewriter maps original sentences to re-gendered targets, while neutral sentences have identical source and target text.Source: §5 · p. 5
  • Source tags mark sentences as neutral (N) or re-genderable (G), inferred during training from whether source and target sentences differ.At test time, tags may be supplied or assigned automatically by the gender classifier.Source: §5 · pp. 5–6

5.1 Gender Classifier

  • The classifier uses about 8,000 industry-partner sentences, balanced across both gender-conversion directions, with the remaining sentences labelled neutral.Its development set uses OpenSubtitles sentences.Source: Data · p. 6
  • A support vector machine classifies sentences using character and word n-gram features alongside a feature indicating whether a gendered word is present.The combined representation has around 40,000 features; neural classifiers were ruled out because the training set was limited.Source: Features · p. 6

Precision and Recall

  • Accuracy is comparable at 80% across both test sets, but neutral-sentence precision and recall are higher on industry data than on OpenSubtitles.The classifier has high recall for re-genderable sentences but mislabels substantial numbers of neutral sentences as re-genderable.Source: Table 2 · p. 6, Precision and Recall · p. 6

6 Results for generating gender variants

The results show that tags can reduce rewriter errors, while performance varies by test-set structure and tagging approach. Manual inspection identifies missed changes and inappropriate changes, especially in unstructured sentences.

6 Results for generating gender variants

  • The hand-crafted CDA baseline generates correct variants for 1% of sentences with animate-noun lists and 9% after adding character adjectives, while the latter changes 3% of neutral sentences.Adding past participles or clitic pronouns to the lists would make them grow exponentially and become error-prone, inefficient, and less scalable.Source: §6 · pp. 6–7

6.1 Automatic evaluation of neural rewriter

  • Manual tags reduce error rates on all test sets for both segment types; automatic tags improve re-genderable segments, especially out of domain, but worsen neutral-segment errors.Source: §6.1 · p. 7

6.2 Qualitative manual inspection of errors

  • In structured sentences, common errors are changing neutral pronouns or adjectives and leaving re-genderable sentences unchanged; tags help in some cases.Source: §6.2 · p. 7
  • Unstructured sentences show more error types, including unnecessary changes and non-existing words; tags resolve some examples but can also trigger such words.Source: §6.2 · pp. 7–8

6.1 Automatic evaluation of neural rewriter

  • Template-based in-domain test sets have lower error rates than unstructured out-of-domain sets, while neutral-segment error rates are comparable.Source: §6.1 · p. 7, Table 3 · p. 8

6.2 Qualitative manual inspection of errors

  • Most re-genderable sentence errors are words left unchanged, and tags help with some—but not all—required changes.Classifier-generated tags increase correctly re-gendered structures at the cost of a small number of additions of non-existing words.Source: §6.2 · p. 8

7 Conclusions and Future Work

The paper presents an initial NMT approach for enriching short conversational sentences with gender alternatives, trained using a gender-parallel Spanish dataset extracted from OpenSubtitles. Automatic and manual evaluations are promising, and future work would extend the approach to more sentence structures and languages.

7 Conclusions and Future Work

  • The approach is not limited to first-person singular phenomena, third-person pronouns, or occupational and animate nouns, and keeps reliance on linguistic knowledge and tools minimal.The paper describes this as supporting real-world deployment.Source: §7 · pp. 8–9
  • The main hurdle is the absence of large training sets, and the industry partner's annotated data was insufficient to train a state-of-the-art automatic gender rewriter.Source: §7 · p. 9
  • Spanish training data was compiled by extracting re-genderable and neutral POS patterns from OpenSubtitles and applying rewriting rules to re-genderable sentences.Source: §7 · p. 9
  • The NMT rewriter translates each re-genderable sentence into its gender alternative, with promising automatic and manual evaluation results.The proposed approach is an initial step toward enriching short conversational sentences with gender variants.Source: §7 · pp. 8–9
  • Gold-standard labels perform better than gender-classifier labels, while the classifier is promising given its very small training set; further experiments should use more training data.Source: §7 · p. 9
  • Future work would explore more Spanish sentence structures, other languages with distinct gendering rules, alternative NMT architectures, and improvements to classifier tags.Source: §7 · p. 9

1 Appendix

The appendix describes the data split and the POS sequences and rewriting rules used for the analyzed word categories.

1.1 Training Data Split

  • The appendix details the split of data from the OpenSubs corpus and the industry partner.Source: §1.1 · p. 11
  • Table 1 reports data statistics, including the split of neutral and re-genderable segments in each set.Source: Table 1 · p. 11, §1.1 · p. 11

1.2 POS Sequences and Rewriting Rules

  • The appendix gives POS sequences and rewriting rules for the analyzed word categories in the industry-sourced data.Source: §1.2 · p. 11

Clitic Pronoun Candidates

The clitic-pronoun candidate material concerns POS sequences and rewriting rules for gendered utterances containing past clitic pronouns.

Clitic Pronoun Candidates

Demonstrative Pronoun Candidates

The demonstrative-pronoun candidate material concerns POS sequences and rewriting rules for gendered utterances containing demonstrative pronouns.

Demonstrative Pronoun Candidates

Past Participles Candidates

The section catalogues POS sequences and rewriting rules for gendered utterances containing past participles, adjectives, and clitic pronouns, while also identifying structures that should remain unchanged.

Past Participles Candidates

Adjective Candidates

Clitic Pronouns Attached to Verbs

Neutral Past Participle Structures

Loading 2107.05987v1…