Source-linked AI summary
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview
Deven Shah, H. Andrew Schwartz, Dirk Hovy
TL;DR
NLP bias research has developed in isolation, often emphasizing bias effects without a shared account of their origins. This paper surveys the literature and proposes a predictive bias framework with a mathematical definition and four bias origins. The framework organizes existing countermeasures and supports comparison of bias sources across NLP work.
Problem
NLP bias studies often address isolated effects without a unifying framework for organizing efforts or identifying underlying origins.
Method
The paper surveys relevant NLP literature and develops a conceptual and quantitative framework covering four bias origins across the predictive pipeline.
Results
The framework organizes surveyed NLP biases and countermeasures by their sources and effects within a standard predictive pipeline.
Takeaways & Limitations
A formal account of bias causes can help researchers identify, compare, and address predictive bias with proactive measures.
Takeaways & Limitations
The paper is focused on bias and should not be conflated with ethics in NLP, which covers more than bias.
Abstract
from arXiv · showhide
An increasing number of works in natural language processing have addressed the effect of bias on the predicted outcomes, introducing mitigation techniques that act on different parts of the standard NLP pipeline (data and models). However, these works have been conducted in isolation, without a unifying framework to organize efforts within the field. This leads to repetitive approaches, and puts an undue focus on the effects of bias, rather than on their origins. Research focused on bias symptoms rather than the underlying origins could limit the development of effective countermeasures. In this paper, we propose a unifying conceptualization: the predictive bias framework for NLP. We summarize the NLP literature and propose a general mathematical definition of predictive bias in NLP along with a conceptual framework, differentiating four main origins of biases: label bias, selection bias, model overamplification, and semantic bias. We discuss how past work has countered each bias origin. Our framework serves to guide an introductory overview of predictive bias in NLP, integrating existing work into a single structure and opening avenues for future research.
1 Introduction
NLP predictive models can encode unintended biases that reduce generalization and reliability while systematically underserving or mispredicting groups. The paper proposes a shared framework that classifies bias origins, connects them to the predictive pipeline, and organizes mitigation research.
- Unintended biases throughout NLP development can reduce performance and reliability on unseen data and systematically underserve or mispredict user groups.
- Language variation across domains, demographics, backgrounds, and personalities can lead models to learn patterns that do not generalize across author groups.
- Existing bias studies often address isolated effects without clearly identifying where bias originates or how countermeasures generalize across the predictive pipeline.
- The framework identifies selection bias, label bias, model overamplification, and semantic bias as distinct sources located at points including embeddings, data, labels, models, and target data.
- The paper surveys NLP bias research and countermeasures while providing a conceptual and quantitative framework for identifying, comparing, and addressing predictive bias.
2 Definition - Two Types of Disparities
The framework defines predictive bias through two measurable disparities: differences between predicted and ideal outcome distributions, and differences in prediction error across human attributes. These disparities depend on the target application and can involve discrete or continuously valued attributes.
- Generalization: The paper generalizes predictive-bias definitions by distinguishing true outcomes Y from predictions Ŷ and allowing continuously valued human attributes A.It defines outcome disparity through differences between Y and Ŷ, and error disparity through ε = |Y − Ŷ| as a function of A.
- Outcome disparity: Outcome disparity occurs when the predicted outcome distribution differs from a theoretical ideal distribution for an outcome, domain, and attribute.The ideal distribution is application-specific and may be empirical or theoretical.
- Outcome disparity: Gender-disparate image captions over-predict females with ovens and males with snowboards, even when the source distribution is treated as ideal.This example illustrates systematic differences in model outcomes conditioned on gender.
- Error disparity: Error disparity occurs when prediction error differs across at least two adequately sampled values or ranges of a human attribute.The formulation covers discrete attributes such as POS-tagging groups and continuous attributes such as age or income.
- Predictive bias: Predictive bias exists in a target application when either outcome disparity or error disparity is present, making bias a property of model–application pairings.The same model may be biased in one application and unbiased in another.
- Quantification: The framework quantifies disparity using distributional divergence, recommending a log-likelihood ratio and allowing KL divergence as a scalable alternative.For error disparity, the ideal distribution is Uniform and predictions are replaced with error.
3 Four Origins of Bias
Predictive bias can originate at four points in the NLP pipeline: labels, observations, representations, and model fitting. The framework distinguishes label bias, selection bias, semantic bias, and model overamplification, which can also compound.
- Four origins: The framework identifies label bias, selection bias, semantic bias, and model overamplification as four origins of predictive bias.These origins correspond to training labels, sampled observations, data representations, and the fitting method.
- Label bias: Label bias occurs when labels diverge from the ideal distribution because of annotator composition, expertise, or stereotypes.The labels themselves are erroneous relative to the source distribution.
- Selection bias: Selection bias occurs when source observations are non-representative of the target users or population.The resulting disparity depends on the target population and may involve source features or labels.
- Interactions and scope: Selection and label bias can interact, while the relevant attribute for selection bias may be unknown in advance.The framework notes that bias conditions depend on sample representativeness and annotation quality.
- Overamplification: Model overamplification occurs when fitting magnifies a small attribute-related difference into a substantially larger predicted disparity.In imSitu, captions mentioned women in 58% of kitchen scenes, while standard models predicted women 63% of the time.
- Semantic bias: Semantic bias arises in embedding parameters through unintended associations between words or contexts, and can indirectly create other biases.The framework treats embeddings as a distinct bias source because pre-trained embeddings may be used without access to their original data.
4 Countermeasures
The surveyed countermeasures are grouped by the bias origins they address, including annotation methods, distributional adjustments, model-fitting interventions, and embedding-specific techniques.
- Organization: Countermeasures are organized according to the bias origin or origins on which they act.This grouping is intended to compare methods within the predictive-bias framework.
- Label bias: Label-bias methods model annotator disagreement, remove outliers, or weight updates to avoid favoring one label.Bayesian annotation models attach confidence scores to annotators, while disagreement-weighted updates can also regularize training.
- Selection bias: Selection-bias countermeasures re-align sample and ideal distributions through stratification, reweighting, poststratification, or data augmentation.Re-stratification can reduce available instances by downsampling overrepresented classes.
- Model interventions: Model-based interventions can reverse adversarial gradients for demographic auxiliary tasks, while downweighting biased instances can discourage overamplification.Synthetic matched distributions are also used for gender bias in coreference-resolution training data.
- Semantic bias: Semantic-bias countermeasures adjust embedding parameters using hard methods that remove bias or soft methods that partially remove it while avoiding side effects.The literature often evaluates these methods with semantic analogy tasks, whose informativeness has been questioned.
- Social-level mitigation: Data Statements and Datasheets document data selection, annotation, curation, and lifecycle processes to trace potential biases.These initiatives emphasize explicit and transparent documentation.
5 Conclusion
The paper presents a unifying overview and framework for understanding predictive-bias sources and effects in NLP. It concludes that bias is nearly inevitable in statistical models, but should be acknowledged and addressed proactively.
- Conclusion: The paper surveys predictive-bias literature and develops a framework describing bias sources and effects rather than effects alone.The framework groups and compares existing countermeasures.
- Conclusion: Every predictive model with errors is bound to have disparities over human attributes, including attributes not directly integrated into the model.The conclusion presents this as a main point of the framework.
- Conclusion: Disparities can originate in embedding models, feature samples, fitting processes, or outcome samples within the predictive pipeline.The conclusion restates the framework’s four pipeline locations.
- Conclusion: Selecting protected attributes is necessary for measuring bias and can help mitigate bias and improve model generalization.The conclusion identifies attribute selection as part of applying the framework.
Framework Application Steps (TL;DR)
The framework application begins by specifying a target population and ideal attribute distribution, then checks outcome or error disparities for possible bias origins and corresponding interventions.
- Framework application steps: Specify the target population and an ideal distribution for the attribute being investigated.Consult datasheets and data statements for the model source when available.
- Framework application steps: For label bias, use post-stratification or retrain annotators.
- Framework application steps: For selection bias, match source and target populations with stratified sampling or post-stratification and reweighting.
- Framework application steps: For overamplification, synthetically match distributions or add outcome disparity to the cost function.
- Framework application steps: For semantic bias, retrain or retrofit embeddings using approaches adapted to attributed words rather than people.
Acknowledegments
The authors acknowledge colleagues and reviewers for constructive comments and note institutional affiliations.
- The authors thank named colleagues and anonymous reviewers for their constructive comments.
A.1 Related Work in Other Fields
Related work connects predictive bias in NLP to systematic differences in human and machine decisions, non-representative sampling, and language variation. It also distinguishes descriptive uses of bias from normative concerns in prediction.
- The paper surveys adjacent-field literature to illustrate the ubiquity and complexity of bias across disciplines.
- Social-science research links bias to systematic errors in human reasoning and decision making, while algorithms may also reduce decision bias.
- Selection bias arises when study samples are non-representative, including western university students or internet users affected by unequal access.
- Semantic bias in embeddings can describe historical changes in gender roles and ethnic stereotypes, but is treated as undesirable in predictive models.
A.2 Discussion: Example Case Studies
The case studies show how selection bias, model overamplification, and semantic bias produce disparities across syntactic tagging, image captioning, sentiment analysis, diagnosis, and population measurement. They also illustrate countermeasures based on matched samples, specialized losses, demographic adjustment, and reweighting-related alignment.
- Part of Speech Taggers and Parsing: Selection bias makes syntactic tools perform significantly worse when training demographics differ from the target population.This produces an error disparity across demographic groups.
- Image Captions: Image-captioning models overamplify gender differences, overselecting particular phrases even when source and target data are identical.The reported outcome disparity is attributed to both image content and biased language models.
- Image Captions: An equalizer model combines Appearance Confusion Loss and Confident Loss to reduce gendered prediction when gender information is absent and increase confidence when it is present.
- Sentiment Analysis: Sentiment systems show score differences after names and pronouns are switched, including higher negative-polarity scores for male pronouns and prototypical Black names.The paper characterizes these stereotypes as both normatively and descriptively wrong.
- Differential Diagnosis in Mental Health: Mental-health classifiers achieved AUCs greater than 0.85 but primarily learned age and gender differences, producing outcome disparities despite balanced class counts.Matched controls aligned age and gender distributions across classes, while demographic modeling could use correlations while controlling confounds.
- Assessing Demographic Variance in Language: Adjusting Twitter feature estimates to representative demographics and socioeconomic conditions improved life-satisfaction predictions for a community.The case concerns representative population measurements from readily available but biased community corpora.