PaperlayerView exact revision on arXiv

Source-linked AI summary

Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Models

Chloe Ciora, Nur Iren, Malihe Alikhani

arXiv:2108.10379v1cs.CL

TL;DR

Gender bias in machine translation is often assessed through overt indicators, but covert asymmetries can be hard to see when translations are considered separately. The paper examines Turkish-English translations for overt stereotypes, personhood effects, and asymmetric gender marking. It reports male bias in neutral contexts and stereotype effects when context is non-neutral.

  • Problem

    Covert gender bias in machine translation has not been adequately addressed, although accurate translations can conceal asymmetrical gender reference unless compared together.

    Source: §1 · p. 1

  • Method

    The study examines commercial MT models translating between gender-neutral Turkish and gendered English, testing stereotypes, personhood references, and asymmetrical gender marking.

    Source: §1 · pp. 1–2, §3.1 · p. 2, §3.3 · p. 3

  • Results

    Models showed male bias in neutral contexts, while stereotypes affected results across experiments when context was non-neutral.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The experiments indicate that gender-bias mitigation requires domain and cultural knowledge, and that techniques are not necessarily transferable across languages.

    Source: §5 · p. 5

    The personhood template may seem unnatural in English, although it is a common Turkish utterance.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

As Machine Translation (MT) has become increasingly more powerful, accessible, and widespread, the potential for the perpetuation of bias has grown alongside its advances. While overt indicators of bias have been studied in machine translation, we argue that covert biases expose a problem that is further entrenched. Through the use of the gender-neutral language Turkish and the gendered language English, we examine cases of both overt and covert gender bias in MT models. Specifically, we introduce a method to investigate asymmetrical gender markings. We also assess bias in the attribution of personhood and examine occupational and personality stereotypes through overt bias indicators in MT models. Our work explores a deeper layer of bias in MT models and demonstrates the continued need for language-specific, interdisciplinary methodology in MT model development.

1 Introduction

The paper argues that gender bias in machine translation includes covert asymmetries that can be hard to notice when translations are viewed separately. It examines overt and covert bias between gender-neutral Turkish and gendered English, including stereotypes, personhood, and gender marking.

1 Introduction

  • Gender asymmetry can be difficult to notice because individually accurate translations may reveal it only when the sentences are compared.The sister example receives an explicit female marker, while the brother example remains neutral.Source: §1 · p. 1, Figure 1 · p. 1
  • Prior mitigation work has focused on debiasing training data, while covert bias remains inadequately addressed and requires linguistic and social context to preserve source meaning.The authors argue that covert bias extends beyond perpetuating outdated stereotypes.Source: §1 · p. 1
  • The study examines overt occupational and personality stereotypes alongside covert personhood bias and asymmetrical gender marking in Turkish-English machine translation.It tests whether gender marking varies with subject gender and whether predicate stereotypes conflict with the subject's gender.Source: §1 · pp. 1–2
  • The authors create a parallel corpus of 1,617 Turkish and English job titles and construct adjective and gendered-subject sentence datasets.The adjective sentences vary whether they include a reference to personhood.Source: §1 · p. 2

2 Related Work

Prior work documents gender bias in models and corpora and commonly evaluates pronoun choices when translating from gender-neutral to gendered languages. This study adapts that approach and draws on Turkish sociolinguistic research about personhood and asymmetrical gender marking.

2 Related Work

  • Prior research found gender bias in MT models and corpora, and a challenge set measured bias when translating from English into languages with morphological gender.Stanovsky et al. (2019) introduced the challenge set.Source: §2 · p. 2
  • Common evaluations translate gender-neutral language into gendered language and examine pronouns for occupations and adjectives; this study uses shared occupations and Turkish stereotypes.The adjective stereotypes are drawn from Sakallı et al. (2018).Source: §2 · p. 2
  • Earlier studies used an intermediary translator to form Turkish datasets, whereas this paper's related-work discussion notes that practice as a dataset distinction.The passage identifies Turkish as a commonly used gender-neutral language in prior studies.Source: §2 · p. 2
  • Turkish sociolinguistic work describes neutral person terms as male-biased and gender marking as asymmetrical, with male gender often unmarked and female gender more explicitly expressed.Braun (2001) describes an exception when a subject is strongly stereotyped as feminine.Source: §2 · p. 2

3 Experiments

The experiments use commercial translation models and Turkish-English materials to examine occupational and adjective stereotypes, personhood, and asymmetrical gender marking. The occupation comparison shows model translations diverging from labor-force shares, while the other experiments specify the sentence designs used to probe bias.

3 Experiments

  • The experiments use Google Translate, Amazon Translate, Microsoft Translator, and SYSTRAN, with translations run in April 2021.The authors state that datasets are available on GitHub.Source: §3 · p. 2

3.1 He is a Doctor, She is a Nurse? Gender Bias in Job Occupation

  • The occupation experiment translates Turkish “He/She is a(n) <occupation>” sentences and adjective-modified variants, comparing selected English pronouns with 2020 Turkish and US workforce statistics.The occupation set matches 1,617 titles across the two countries after exclusions and modifications.Source: §3.1 · p. 2

3.2 He is Smart, She is Beautiful? Bias in Adjective Use

  • The adjective experiment uses 97 Turkish stereotype adjectives, classified as masculine-coded, feminine-coded, or neutral, and tests whether a personhood phrase changes assumed gender.The personhood template is described as a common Turkish utterance despite seeming unnatural in English.Source: §3.2 · p. 3

3.3 Bias Through Asymmetrical Gender Markings

  • The asymmetry experiment translates English sentences with gendered subjects and stereotyped predicates into Turkish, testing gender marking by subject gender and predicate stereotype alignment.The study uses four subject words and predicates across occupation, description, and activity categories, with 120 sentences for each gender.Source: §3.3 · pp. 3–4
  • The figure compares model female-translation rates with women's labor-force shares by occupational group; results do not match either workforce breakdown, and ISCO female-translation rates tend to rise with female participation.The figure presents Turkish labor-force data below and US data above.Source: Figure 2 · p. 3

4 Results

Across the experiments, the models’ gender choices varied with occupation, adjective stereotypes, personhood, and whether gender was expected or unexpected in Turkish. The results also show that female gender was marked more often than male gender in the Turkish translations examined.

4.1 Gender Bias in Occupations

  • Female pronouns accounted for 1.11% with Google, 1.18% with Amazon, 3.83% with Microsoft, and 5.07% with Systran, far below 2020 workplace participation in Turkey and the US.The study reports statistically significant results when translation outcomes in each job category are compared with corresponding labor statistics (p < 0.01).Source: §4.1 · p. 4

4.2 Impact of an Attributive Adjective Preceded by Occupation

  • Adding an attributive adjective significantly shifted female-assigned pronouns toward male pronouns, with more such changes as adjectives moved from “very good” to “very bad.”The reverse pattern occurred for male-assigned pronouns; one example translated “intensive care unit nurse” as “He is a very bad intensive care unit nurse” after adding the negative adjective.Source: §4.2 · p. 4

4.3 Turkish Gender Stereotypes in Person Descriptors

  • Only 6.74% of pronouns assigned for “He/She is <adjective>” were female, and female pronouns were more strongly associated with female-coded adjectives than male pronouns were with male-coded adjectives.The association with female-coded adjectives was significant against both male-coded adjectives (p < .01) and neutral adjectives (p < .05).Source: §4.3 · p. 4

4.4 Analyzing Gendered Personhood

  • Introducing a personhood modifier shifted 74.07% of sentences that received female pronouns in the first template to male pronouns; the reverse shift was 2.76%.Each translator was significantly more likely to assign a male pronoun when the source sentence contained a personhood modifier (p < 0.01).Source: §4.4 · p. 4

4.5 Asymmetrical Gender Analysis

  • Male subject words were translated without overt gender marking 47.7% of the time, while only 25% of female subject words used the neutral case.The neutral “niece/nephew” word remained unmarked in all sentences for both male and female subjects, which the authors theorize reflects spoken Turkish usage.Source: §4.5 · pp. 4–5, Figure 3 · p. 5
  • With male subject words, gender was not preserved 52.1% of the time for stereotypically masculine predicates, but gender was overtly marked in 56.6% of translations with stereotypically feminine predicates.The authors give “The young men are soccer players” and “The young men are secretaries” as contrasting examples.Source: §4.5 · p. 5, Figure 4 · p. 5
  • Except for “niece/nephew,” female subject gender was overtly marked in 75% of translations, while male gender was marked when sentence content deviated from the masculine social norm.The authors describe female gender as marked in the overwhelming majority of cases, regardless of context.Source: §4.5 · p. 5

5 Conclusion

The models show both overt stereotype bias and covert gender and personhood bias, with male assumptions in neutral contexts and stereotype effects in non-neutral ones. The findings point to language-specific and culturally grounded approaches to understanding and addressing MT gender bias.

5 Conclusion

  • Male gender was assumed for gender-equal occupations and stereotype-neutral adjectives, while non-neutral contexts routinely showed stereotype bias.The conclusion also reports that male subjects were more likely to receive neutral overt gender markings.Source: §5 · p. 5
  • The models exhibited overt occupation and adjective stereotypes alongside covert asymmetrical gender marking and personhood bias.Source: §5 · p. 5
  • The authors conclude that domain and cultural knowledge are required, and that bias-mitigation techniques are not necessarily transferable across languages.Source: §5 · p. 5

A Occupation Data Set Details

The occupation dataset pairs occupations from Turkey and the US, including exact and similar matches, with some categories excluded and occupation groups listed in tables.

A Occupation Data Set Details

B Occupations with Female Generated Pronouns

Table 4 lists occupations that at least three of four translators assigned female pronouns to.

B Occupations with Female Generated Pronouns

C Sentence Templates in Turkish

Table 5 presents Turkish sentence templates alongside their English translations, including a template with an adjective slot.

C Sentence Templates in Turkish

  • The table pairs original Turkish templates with English translations.Source: C Sentence Templates in Turkish · p. 8
  • One template is “O <adjective> birisidir,” translated as “He/she is someone who is <adjective>.”Source: C Sentence Templates in Turkish · p. 8
  • The adjective slot in the third template uses one of four options, from “very good” to “very bad.”The options are “c¸ok iyi” (very good), “iyi” (good), “k¨ot¨u” (bad), and “c¸ok k¨ot¨u” (very bad).Source: Table 5 · p. 8
Loading 2108.10379v1…