Source-linked AI summary
Gender Bias in Coreference Resolution
Rachel Rudinger, Jason Naradowsky, Brian Leonard, Benjamin Van Durme
TL;DR
Coreference resolution is entangled with gender, yet systems may reflect gender patterns in language and employment data. The paper introduces human-validated Winogender schemas that vary pronoun gender and evaluates three public systems. The systems show systematic gender sensitivity and preferences that correlate with occupational gender statistics, while the schemas are better suited to detecting bias than proving its absence.
Problem
Coreference resolution is tightly bound to gender, but evidence is needed on how gender bias manifests in automated systems.
Method
The paper constructs human-validated Winogender schemas varying only pronoun gender and evaluates three publicly available coreference systems across occupation-based cases.
Results
68% of male-female minimal pairs differ under RULE, 28% under STAT, and 13% under NEURAL; male pronouns are also more often resolved as OCCUPATION across all systems.
Takeaways & Limitations
System gender preferences correlate with real-world and textual occupational gender statistics and can reinforce occupational gender disparities.
Takeaways & Limitations
Winogender schemas may demonstrate gender bias when present but cannot prove its absence, and this study focuses on occupational gender bias.
Abstract
from arXiv · showhide
We present an empirical study of gender bias in coreference resolution systems. We first introduce a novel, Winograd schema-style set of minimal pair sentences that differ only by pronoun gender. With these "Winogender schemas," we evaluate and confirm systematic gender bias in three publicly-available coreference resolution systems, and correlate this bias with real-world and textual gender statistics.
1 Introduction
The paper investigates how gender bias affects coreference resolution, motivated by human difficulty linking “mother” and “surgeon” in a classic riddle. It introduces Winogender schemas that vary only pronoun gender to test systems’ gender sensitivity.
- Motivation: The classic surgeon riddle illustrates difficulty assigning both “mother” and “surgeon” to the same person.The father dies, the boy is injured, and the surgeon is the boy’s mother.
- Motivation: Coreference resolution is connected to gender because people and automated systems may associate occupations with particular pronouns.The paper frames this as a concern because data-driven AI may acquire and amplify human-like biases.
- Approach: The study constructs Winogender schemas in which a pronoun must resolve to one of two entities while only pronoun gender varies.The schemas are designed to be easy for humans but challenging for data-driven systems.
- Approach: The study evaluates three publicly available coreference systems and examines whether occupation-specific bias correlates with employment statistics.The systems represent rule-based, feature-driven statistical, and neural approaches.
- Approach: Gender is represented categorically as female, male, and sometimes neutral because of limitations in the textual and real-world datasets.The paper explicitly identifies these choices as dataset limitations.
2 Coreference Systems
The paper compares rule-based, statistical, and neural coreference systems, whose available gender information comes from rules, learned features, and resources such as gendered noun statistics. These design choices can expose systems to gender patterns in their data and representations.
- System paradigms: The evaluation covers three publicly available systems spanning rule-based, feature-driven statistical, and neural paradigms.The systems are treated as off-the-shelf representatives of distinct modeling approaches.
- Rule-based: The rule-based Stanford multi-pass sieve applies deterministic models from highest to lowest precision, with gender identified at the first stage.Gender information is therefore available throughout the sieve.
- Statistical: Statistical systems replace hand-written rules with feature templates whose combinations can capture useful linguistic phenomena and data bias.Occupation-plus-pronoun features may be especially informative for occupations that are infrequent in the data.
- Neural: Neural systems use learned antecedent-scoring functions and global inference, while generally accessing information similar to statistical systems.Pre-trained word embeddings are identified as a notable exception.
- Gender resources: The Bergsma and Lin resource provides gender and number counts for English nouns and phrases from 85GB of web news.For example, it reports that 9.2% of mentions of “doctor” are female, and the paper uses the resource in its analysis.
3 Winogender Schemas
Winogender schemas form a controlled evaluation set for testing whether coreference systems change their predictions when only pronoun gender changes. The authors construct and human-validate 720 occupation-based sentences, then compare their results with gender statistics.
- Schema design: The evaluation set contains 120 hand-written Winograd-style sentence templates designed to expose gender-sensitive occupation resolution.Each sentence contains occupation, participant, and pronoun referring expressions of interest.
- Schema design: The dataset uses 60 occupations, two sentence templates, two participant types, and three pronoun genders to produce 720 sentences.The construction is 60 occupations × 2 templates × 2 participants × 3 pronoun genders.
- Validation: Each template has one intended answer, either OCCUPATION or PARTICIPANT, and was designed to keep human resolution unaffected by pronoun gender.The authors aimed for unambiguous judgments without additional context.
- Validation: All 720 sentences were validated on Mechanical Turk with 10-way redundancy.Tasks included five Winogender and five existing Winograd sentences; raw agreement was higher for Winogender schemas than Winograd schemas.
- Scope: The schemas are not primarily intended to identify sentences that are difficult for humans or require specialized world knowledge.This distinguishes the dataset’s purpose from the original Winograd challenge.
4 Results and Discussion
The three evaluated coreference systems exhibit gender-sensitive errors, with predictions reflecting occupational gender statistics and performing worse on gender-discordant “gotcha” cases.
- Systematic gender bias: 68% of RULE, 28% of STAT, and 13% of NEURAL male-female minimal pairs are resolved differently despite gender-independent correct answers.Human annotators validated that the correct resolution does not depend on pronoun gender.
- Systematic gender bias: Male pronouns are resolved as OCCUPATION more often than female or neutral pronouns across RULE, STAT, and NEURAL.The corresponding rates are RULE: 72% male, 29% female, 1% neutral; STAT: 71%, 63%, 50%; NEURAL: 87%, 80%, 36%.
- Connections to gender statistics: Systems’ occupational gender preferences correlate with both U.S. employment statistics and gender statistics derived from text.Figure 4 compares one point per occupation and reports regression lines with 95% confidence intervals.
- Gotcha cases: Systems perform uniformly worse on “gotcha” sentences where pronoun gender conflicts with the occupation’s majority gender or the correct antecedent.Table 2 defines these cases using the occupation’s BLS gender distribution and whether OCCUPATION or PARTICIPANT is correct.
5 Related Work
Related work examines gender bias in embeddings, datasets, and deployed NLP systems, while concurrent work introduces a complementary Winograd-style coreference evaluation.
- Bias in representations: Prior studies analyze implicit gender bias in modern and historical word embeddings and propose embedding or model debiasing methods.The cited work includes methods based on geometric debiasing and adversarial learning.
- Bias in datasets and systems: Other research studies social and gender stereotyping in NLP and vision datasets, along with gender and dialect effects in deployed speech recognition.Related work also examines author gender and text attributes.
- Coreference evaluations: Zhao et al. (2018) concurrently propose WinoBias, another human-validated Winograd schema-like test for gender bias in coreference resolution.WinoBias includes syntax-resolvable sentences and two occupational mentions, whereas Winogender uses one occupational mention and one other participant.
- Coreference evaluations: The authors encourage future evaluations to use both WinoBias and Winogender because their sentence constructions differ.The datasets therefore provide complementary evaluation coverage.
6 Conclusion and Future Work
The paper introduces Winogender schemas to diagnose gender bias in coreference systems and finds systematic, statistics-linked preferences across three publicly available systems. It cautions that the schemas can reveal bias but cannot establish its absence, while treating human judgments as a lower bound for automated equitability.
- Conclusion: Winogender schemas expose systematic gender bias in three publicly available, off-the-shelf coreference systems.The systems often prefer resolving pronouns of one gender over another for particular occupations.
- Conclusion: These preferences correlate with real-world employment statistics and the textual gender statistics accessed by the systems.The paper posits that systems overgeneralize gender and thereby make errors humans do not make.
- Limitations and future work: Winogender schemas have high positive predictive value but low negative predictive value as a diagnostic of gender bias.They may demonstrate bias when present but cannot prove that a system is unbiased when none is observed.
- Limitations and future work: The evaluation focuses on occupational gender bias, although the schemas may be extended to other manifestations of gender bias.Human judgments are presented as a lower bound for equitability rather than the ultimate objective.