Source-linked AI summary
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, Ryan Cotterell
TL;DR
Gender stereotypes in language corpora can be propagated or amplified by NLP systems, while English-oriented mitigation methods produce ungrammatical outputs in morphologically rich languages. The paper introduces a Markov random field-based counterfactual augmentation method that converts sentence gender while preserving agreement. Across four languages, it reduces gender stereotyping by a factor of 2.5 without sacrificing grammaticality.
Problem
English-focused gender-stereotype mitigation methods do not transfer to morphologically rich languages because gender changes require coordinated morphological agreement updates.
Method
The paper uses a Markov random field within a four-step process to infer updated morpho-syntactic tags and reinflect complete sentences after changing a noun's gender.
Results
Across four languages, the approach reduces gender stereotyping in neural language models by a factor of 2.5 without sacrificing grammaticality.
Takeaways & Limitations
The approach provides a way to perform counterfactual data augmentation for gendered nouns in morphologically rich languages while preserving morpho-syntactic agreement.
Takeaways & Limitations
The model lacks co-reference information, limiting its ability to update conjoined nouns that refer to the same person.
Abstract
from arXiv · showhide
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems. Although research has focused on mitigating gender stereotypes in English, the approaches that are commonly employed produce ungrammatical sentences in morphologically rich languages. We present a novel approach for converting between masculine-inflected and feminine-inflected sentences in such languages. For Spanish and Hebrew, our approach achieves F1 scores of 82% and 73% at the level of tags and accuracies of 90% and 87% at the level of forms. By evaluating our approach using four different languages, we show that, on average, it reduces gender stereotyping by a factor of 2.5 without any sacrifice to grammaticality.
1 Introduction
NLP systems can replicate or amplify gender stereotypes because training corpora reflect societal imbalances. English-focused mitigation methods do not transfer directly to morphologically rich languages, motivating counterfactual augmentation that preserves agreement.
- NLP systems may replicate or amplify societal biases encoded in human-designed language corpora.
- Figure 1 illustrates transforming Los ingenieros son expertos into Las ingenieras son expertas by updating tags and reinflecting lemmas.
- English-focused methods are insufficient for languages whose morphological endings encode gender agreement across surrounding words.Changing one word's gender requires updating other words, and the required changes vary by language.
- The proposed approach uses a Markov random field to infer how sentences should change when a noun's grammatical gender is altered.It is applied as part of a four-step process for reinflecting entire sentences.
2 Gender Stereotypes in Text
Gender imbalance in text can produce downstream stereotyping in NLP systems, while naive word swapping creates ungrammatical outputs in morphologically rich languages. The paper instead formulates gender transformation as unsupervised tag inference followed by reinflection.
- Unequal mention rates for men and women can cause NLP systems to associate engineering with men and prefer male engineers in applications.
- Naive swapping of gendered words is applicable mainly to English and can yield ungrammatical sentences in languages with gender agreement.
- Morphologically rich languages require coordinated updates because nouns, verbs, and adjectives can carry gender-marking endings.The required changes differ across languages, including whether verbs mark gender.
- Our approach: The approach analyzes sentences with dependency trees, lemmata, POS tags, and morpho-syntactic tags before intervening on a gendered word.It is described as unsupervised because the task lacks direct supervision, although it uses annotated linguistic resources.
- Our approach: The Markov random field infers updated morpho-syntactic tags, after which the system reinflects the lemmata into new forms.Its primary technical contribution is the MRF used for tag inference.
3 A Markov Random Field for Morpho-Syntactic Agreement
The paper models morpho-syntactic agreement with a Markov random field over tags conditioned on dependency structure and POS tags. It supports exact tree inference, configurable parameterizations, and intervention-specific tag constraints.
- Model: The MRF defines a joint distribution over morpho-syntactic tag sequences conditioned on a labeled dependency tree and POS tags.Its binary factors score agreement between connected tags, while unary factors score individual tags.
- Model: A dependency tree represents ordered head-dependent triples, with each position associated with morpho-syntactic and POS tags.For example, ingeniero is represented as a masculine, singular noun.
- Parameterization: The linear parameterization assigns a matrix W(pi, pj, ℓ) to each POS-tag and dependency-label triple, using multi-hot morpho-syntactic tag encodings.A neural alternative shares parameters across edges with different parts of speech and labels.
- Parameterization: Unary factors force or disallow tags during intervention, with α > 1 controlling how strongly non-intervened tags remain unchanged.As α approaches infinity, only the directly intervened tag changes.
- Inference: Because the MRF is tree-shaped, belief propagation performs exact inference in polynomial time, with partition-function computation taking O(n · |M|^2).Max-product belief propagation identifies the highest-probability tag sequence.
- Parameter Estimation: The model parameters are learned by gradient descent on negative log-likelihood, while α is set using development data.The paper considers both linear and neural parameterizations of the binary factor.
4 Intervention
The intervention changes a gendered word's tag and infers the remaining tag updates needed for morpho-syntactic agreement. The resulting tags are then used to reinflect the sentence's lemmata into new forms.
- Intervention: Changing ingeniero from [MSC; SG] to [FEM; SG] requires updating el and experto while leaving unrelated tags unchanged.The model infers high-probability tags for the remaining words after the intervention.
- Intervention: Unary factors minimize unnecessary changes after intervention by making tag changes less likely as α increases.This preserves as much of the original sentence as possible while restoring agreement.
- Reinflection: The final step reinflects lemmata using an inflection model conditioned on each lemma and its new morpho-syntactic tag.For example, experto with [A; FEM; PL] receives high probability for expertas.
5 Experiments
The experiments evaluate the approach intrinsically on gender-tag and reinflection accuracy, then extrinsically on stereotype reduction and grammaticality across languages. Results show stronger recall than language-specific baselines, substantial stereotype reduction, and generally favorable grammaticality trade-offs, with known limitations for conjunctions and ambiguous sentence selection.
- Intrinsic Evaluation: The intrinsic evaluation measures morpho-syntactic tag prediction and reinflected form accuracy after intervening on an animate noun’s gender.All subtags except gender are held fixed, and the intervened nouns are excluded from evaluation.
- Intrinsic Evaluation: The baselines achieve slightly higher precision, but our approach achieves higher recall by capturing subtler gender-agreement relationships.The authors attribute the baselines’ lower recall to their reliance on well-known language-specific rules.
- Intrinsic Evaluation: The neural parameterization does not outperform the linear parameterization, so the simpler linear model is sufficient in practice.The proposed neural version was intended to share parameters across edges with different parts of speech and labels, but that sharing did not make a practical difference.
- Extrinsic Evaluation: Naïve swapping reduces stereotyping for some but not all languages, whereas the proposed approach shows a clear reduction for stereotyped Spanish words.A word is classified as stereotyped toward one gender when 75% of its occurrences have that gender.
- Extrinsic Evaluation: Except in Hebrew, the approach sacrifices less grammaticality than naïve swapping and sometimes increases grammaticality relative to the original corpus.The authors relate the Hebrew result to the model’s lower intrinsic accuracy there.
6 Related Work
The paper situates its work against NLP research focused primarily on detecting and mitigating gender stereotypes in English. It contrasts several English-oriented mitigation efforts with the present focus on morphologically rich languages.
- Prior NLP work primarily addressed detecting and mitigating gender stereotypes in English.
- Bolukbasi et al. mitigated stereotypes in word embeddings while preserving meanings.
- Lu et al. studied gender stereotypes in language models, while Rudinger et al. examined related mitigation work.
7 Conclusion
The paper concludes by introducing a method for converting masculine- and feminine-inflected noun phrases in morphologically rich languages. It evaluates the approach intrinsically and extrinsically, including its effect on gender stereotyping in neural language models.
- The approach converts between masculine-inflected and feminine-inflected noun phrases in morphologically rich languages.
- A Markov random field with optional neural parameterization infers changes needed to preserve morpho-syntactic agreement.
- The task lacked an existing annotated corpus of paired sentences for ground-truth evaluation.
- The authors evaluated the approach intrinsically and extrinsically, demonstrating reduced gender stereotyping in neural language models.
- Future work includes incorporating co-reference information.
A Belief Propagation Update Equations
This section introduces belief-propagation update equations and defines neighboring nodes as the nodes adjacent to a given node. The equations update beliefs using messages passed through the graph.
- The section presents the belief-propagation update equations.
- N(i) denotes the set of neighboring nodes of node i.
- The belief at each node is defined after specifying the neighboring-node set.
B Adjective Translations
The appendix provides feminine and masculine translations for four adjectives used in French, Hebrew, Italian, and Spanish. The translations are organized in separate tables by grammatical gender.
- The appendix includes translations of good, bad, smart, and beautiful.
- Table 6 contains feminine translations in French, Hebrew, Italian, and Spanish.
- Table 7 contains masculine translations in French, Hebrew, Italian, and Spanish.
C Extrinsic Evaluation Example Phrases
The evaluation constructs gendered Spanish phrases for gazetteer nouns and reports their prefix log-likelihoods across the original and two augmented corpora. Table 8 marks ungrammatical phrases for comparison.
- Phrase Generation: For each noun in the animacy gazetteer, the authors generated sixteen phrases for evaluation.The engineer example uses four adjective-based phrases for each translation.
- Phrase Generation: The engineer example includes translations of “The good engineer,” “The bad engineer,” “The smart engineer,” and “The beautiful engineer.”
- Likelihood Comparison: Table 8 compares Spanish phrase prefix log-likelihoods across the original corpus, naïve gender-word swapping, and the authors’ MRF approach.Ungrammatical phrases are denoted by “*”.