Source-linked AI summary
Measuring and Reducing Gendered Correlations in Pre-trained Models
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, Slav Petrov
TL;DR
Pre-trained models can encode unintended gendered correlations that affect model behavior. The paper develops metrics and evaluates general mitigation techniques, finding that correlations can be reduced while preserving accuracy, though the strategies have trade-offs and scope limitations.
Problem
Pre-trained models can encode artifacts, including gendered correlations, that lead to unwarranted assumptions on new examples.
Method
The paper defines an evaluation framework using metrics for discovering and measuring gendered correlations in models and downstream applications, then evaluates dropout and counterfactual data augmentation.
Results
Models with similar accuracy can differ greatly in gendered correlations, while dropout and counterfactual data augmentation reduce correlations and their benefits carry through fine-tuning.
Takeaways & Limitations
Model development should evaluate unintended correlations, consider configuration differences, and prioritize mitigation methods whose improvements generalize.
Takeaways & Limitations
Dropout and counterfactual data augmentation each have trade-offs; counterfactual augmentation requires word lists whose coverage may be incomplete.
Abstract
from arXiv · showhide
Pre-trained models have revolutionized natural language understanding. However, researchers have found they can encode artifacts undesired in many applications, such as professions correlating with one gender more than another. We explore such gendered correlations as a case study for how to address unintended correlations in pre-trained models. We define metrics and reveal that it is possible for models with similar accuracy to encode correlations at very different rates. We show how measured correlations can be reduced with general-purpose techniques, and highlight the trade offs different strategies have. With these results, we make recommendations for training robust models: (1) carefully evaluate unintended correlations, (2) be mindful of seemingly innocuous configuration differences, and (3) focus on general mitigations.
1 Introduction
The paper studies gendered correlations as a case of unintended model artifacts and uses them to develop evaluation and mitigation recommendations for robust pre-trained models.
- Gendered correlations can allow social stereotypes to affect model decisions.
- The paper proposes metrics for detecting and measuring associations in models and downstream applications.
- Models with similar accuracy can encode correlations at very different rates, motivating richer evaluation when selecting models.
- Dropout regularization and counterfactual data augmentation reduce correlations while maintaining strong accuracy.
- Both techniques offer general-purpose improvements, and their pre-training benefits carry through fine-tuning.Dropout does not target specific correlations, while counterfactual data augmentation can reduce correlations beyond those specified in training.
- The approaches have trade-offs, motivating future research into techniques that broadly address model artifacts.
2 Background and Related Work
Prior work measures gender associations intrinsically and bias propagation in downstream tasks; this paper introduces discovery-oriented analysis and evaluates mitigation methods.
- Intrinsic Measurement: Prior intrinsic studies have found gender stereotypes in word embeddings, but analogy-based measurements may be unstable and fail to generalize.
- Intrinsic Measurement: Recent contextualized-embedding association tests report mixed results, while other probes examine stereotypical beliefs across several social concepts.
- Intrinsic Measurement: Existing intrinsic studies require bias definitions before inspection, limiting discovery of other important model problems.
- Extrinsic Measurement: Downstream research documents racial and gender bias in resume matching, sentiment analysis, coreference resolution, image captioning, and machine translation.
- Extrinsic Measurement: The paper samples three tasks for an evaluation framework intended to overview concerns for natural-language understanding.
- Mitigation: Dropout regularization reduces gendered correlations without manual input, while counterfactual data augmentation is examined because of its popularity.
3 Evaluation Framework
The evaluation framework measures gendered correlations through complementary intrinsic and downstream metrics across multiple task formulations. It combines existing task evaluations with the new DisCo analysis to expose associations in models before and after fine-tuning.
- Framework overview: The framework quantifies gendered correlations using multiple metrics spanning intrinsic model analysis and downstream applications.Its task coverage includes synthetic template-based text and real web text across three pre-trained-model capabilities.
- Downstream metrics: Coreference resolution and Bias-in-Bios measure gendered effects in downstream applications, while STS-B is adapted into a gender-focused similarity metric.Bias-in-Bios reports the slope between TPR gap and profession gender representation rather than Pearson correlation because the latter does not differ significantly between models.
- DisCo: DisCo probes pre-trained representations by testing whether generated top-three fills differ significantly according to the gender association of the input word.It averages the number of gender-associated fills across templates and can discover correlations without prespecifying candidate vocabulary items.
- Coreference resolution: Coreference evaluation measures the Pearson trend between pronoun-to-profession likelihood and the profession’s female representation, with zero indicating no gendered correlation.The evaluation uses WinoGender templates containing a gendered pronoun and a profession antecedent.
- Accuracy and trade-offs: The framework also tracks standard task accuracy to assess interactions and trade-offs between model performance and gendered correlations.Reported accuracy measures are F1 for coreference, Pearson correlation with gold scores for STS-B, and classification accuracy for Bias-in-Bios.
4 Measuring Gendered Correlations
Evaluation of publicly available pre-trained models finds that similar task accuracy can coexist with substantially different gendered correlations. The results therefore support precise, diverse evaluation rather than a simple choice based on architecture or model size.
- Findings: Models with similar accuracy vary widely in their measured gendered correlations, highlighting the need for precise evaluation when selecting a model.The framework is applied to publicly available pre-trained models.
- Model comparisons: ALBERT models have slightly lower intrinsic DisCo values than BERT, while BERT performs substantially better downstream on STS-B and Bias-in-Bios.The reported intrinsic examples include art and music as gendered study subjects and play, cook, and read as gendered activities.
- Model size: BERT model size shows no systematic correlation trend, despite some variation among the evaluated smaller and larger models.The study reports no evidence that smaller models are systematically more susceptible to gendered correlations.
- Implications: No simple recommendation identifies the safest architecture or size, so model selection should use precise and diverse metrics to check expected application behavior.The conclusion follows from accuracy consistency alongside substantial differences in correlation metrics.
5 Reducing Gendered Correlations
The paper evaluates dropout regularization and counterfactual data augmentation as general-purpose ways to reduce gendered correlations while preserving task accuracy, while examining their trade-offs and generalization.
- 5.1 Dropout Regularization: Dropout interrupts attention-based associations and is tested as a configuration-level mitigation for gendered correlations.The study increases BERT’s attention and hidden-activation dropout and reintroduces dropout in ALBERT.
- 5.1 Dropout Regularization: Dropout reduces DisCo, Coref, and STS-B correlation metrics without task-specific changes or altered training-data distributions.The Bias-in-Bios correlation metric does not move perceptibly, and its dataset contains apparent label noise.
- 5.1 Dropout Regularization: High dropout improves correlations without significantly hurting STS-B or Bias-in-Bios accuracy, but coreference accuracy drops at the tested rates.The authors recommend selecting the configuration encoding the fewest correlations for the accuracy a task requires.
- 5.1 Dropout Regularization: ALBERT dropout substantially reduces correlations in all metrics except DisCo without hurting accuracy beyond a 1% change.The best tested ALBERT setting uses dropout value .05.
- 5.2 Counterfactual Pre-training: Counterfactual data augmentation substitutes gender-partner words in Wikipedia sentences and can be applied one-sidedly or two-sidedly during pre-training.Neutral substitutions would require a word list containing neutral terms.
- 5.2 Counterfactual Pre-training: Two-sided CDA improves correlation metrics, is particularly effective on DisCo (Terms), Coref, and STS-B, and preserves accuracy better than dropout.The intervention’s generalization is tested beyond the vocabulary used for mitigation.
- 5.3 Fine-tuning: Mitigated BERT checkpoints maintain lower correlation metrics during STS-B fine-tuning than the public BERT baseline.Fine-tuning reintroduces gendered correlations, but mitigation makes models more resistant to their re-learning.
- 5.2 Counterfactual Pre-training: CDA improves DisCo (Names N-Z) and possibly DisCo (Terms) despite excluding those test vocabularies from mitigation, with greatest improvements under random replacement associations.This pattern suggests CDA may reduce associations between sentence context and a concept of gender rather than only individual tokens.
6 Resilience to Fine-tuning
Pre-training mitigations reduce gendered correlations that can re-emerge during fine-tuning, while partial freezing can preserve mitigation and accuracy under some conditions.
- Partial freezing: Correlations remain consistently lower than baseline across mitigation checkpoints as the number of frozen layers varies, while CDA maintains accuracy.Figure 2 uses the number of frozen layers as its horizontal axis.
- Resilience to fine-tuning: Fine-tuning re-introduces gendered correlations, but mitigation checkpoints retain lower correlation metrics than the public BERT model.Accuracy and correlation metrics increase or remain zero during fine-tuning; mitigated models nevertheless confer resistance.
- Partial freezing: Partial freezing limits how much a pre-trained encoder changes during fine-tuning, preserving more of the pre-trained model.The experiments incrementally freeze more encoder layers while fine-tuning on STS-B.
- Partial freezing: CDA accuracy remains as strong as the public model whether the model is used as a feature extractor or fine-tuned.The experiments are not meaningful for ALBERT because its parameters are shared between layers.
- Partial freezing: When many layers are frozen, dropout-mitigated accuracy declines, although gradual decreases can preserve strong accuracy at points such as freezing 16 layers.Freezing more than 16 layers further reduces correlations beyond Table 4 for the dropout-mitigated model.
7 Recommendations
The paper recommends evaluating unintended associations alongside accuracy, accounting for configuration differences, and favoring general mitigation methods.
- Carefully evaluate unintended associations: Models with similar accuracy can differ greatly on gendered-correlation metrics, so model evaluation should measure unintended associations directly.The analyses can extend to other correlation types by changing the word lists.
- Be mindful of seemingly innocuous configuration differences: When accuracy is comparable, selecting the configuration with fewer encoded correlations can improve robustness; dropout should be retained.The recommendation specifically identifies dropout regularization as an important parameter.
- Focus on general mitigations: Pre-training mitigations can show resilience to fine-tuning, allowing improvements to carry through to multiple downstream tasks.Dropout requires no correlation target, while CDA is attractive when target correlations are known and causes almost no perceptible accuracy change.
- Focus on general mitigations: General-purpose mitigation is recommended because dropout does not target specific correlations and CDA can reduce correlations beyond those specified during training.Both techniques minimize correlations while maintaining strong accuracy.
8 Conclusion
The paper evaluates model accuracy together with gendered correlations and finds evidence that unintended-correlation evaluation and general mitigation are worthwhile in model development.
- Conclusion: The evaluation framework considers both overall accuracy and gendered correlations to study factors shaping unintended correlations.The conclusion presents gendered correlations as the paper’s evaluation focus.
- Conclusion: The results provide evidence that evaluating unintended correlations is critical and that actively mitigating risks is worthwhile when improvements scale generally.The conclusion connects these findings to model development.
A DisCo Evaluation
The DisCo evaluation uses templated sentences with PERSON and BLANK placeholders to probe associations involving people, studies, subjects, preferences, and activities.
- Templates: The evaluation includes templates asking what a PERSON is interested in, studied, or took a course in.These templates probe associations with a blank concept across several educational and interest contexts.
- Templates: The remaining templates probe identity, workplace satisfaction, preferences, and liking activities using PERSON and BLANK.They include positive, negative, habitual, and negated formulations.
- Templates: The set also includes always-like and never-like formulations, adding extreme habitual and negated preference contexts.These templates distinguish unconditional positive and negative activity preferences.
- Templates: Several templates place BLANK in educational roles, including a person’s college major and best school subject.The ordering alternates whether PERSON or BLANK occupies the sentence’s subject position.