PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Bias and Fairness in Machine Learning

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, Aram Galstyan

arXiv:1908.09635v3cs.LG

TL;DR

AI systems can produce discriminatory outcomes, while fairness lacks a universal definition. This survey synthesizes sources of bias, fairness definitions, and mitigation research across AI domains, finding that unfairness arises from both data and algorithms.

  • Problem

    AI systems are used in high-stakes decisions yet can produce biased outcomes, and no universal definition of fairness exists for addressing discrimination.

  • Method

    The survey reviews data and algorithmic bias, develops a taxonomy of fairness definitions, and examines fairness research across AI domains and subdomains.

  • Results

    Unfair machine-learning outcomes arise from both biased data and algorithmic behavior, with user experience potentially creating feedback loops that perpetuate or amplify bias.

  • Takeaways & Limitations

    Fairness research must consider application context because some fairness constraints are inherently incompatible except under highly constrained conditions.

  • Takeaways & Limitations

    The survey identifies equity as an underexplored fairness concept whose operationalization and relationship to existing definitions remain future directions.

Abstract

from arXiv · show

With the widespread use of AI systems and applications in our everyday lives, it is important to take fairness issues into consideration while designing and engineering these types of systems. Such systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that the decisions do not reflect discriminatory behavior toward certain groups or populations. We have recently seen work in machine learning, natural language processing, and deep learning that addresses such challenges in different subdomains. With the commercialization of these systems, researchers are becoming aware of the biases that these applications can contain and have attempted to address them. In this survey we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined in order to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and how they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

1 INTRODUCTION

Machine learning systems increasingly influence everyday and high-stakes decisions, but documented cases show that their predictions can discriminate across groups. This survey examines data- and algorithm-based sources of unfairness and reviews how fairness has been defined and studied.

  • Motivation: Machine learning algorithms increasingly make everyday recommendations and high-stakes loan and hiring decisions.They can process many more factors than people, but remain vulnerable to unfair outcomes.
  • Documented Biases: COMPAS produced higher false positive rates for African-American than Caucasian offenders, while other AI systems showed skin-tone and ethnicity-related biases.Examples include beauty-pageant judging biased against darker-skinned contestants and cameras overpredicting Asians as blinking.
  • Survey Scope: The survey identifies unfairness arising from biases in data and from the algorithms themselves.It reviews how data biases skew learning and how algorithmic mechanisms can prevent fair decisions even with unbiased data.
  • Organization: The review covers real-world cases, bias sources within the data-algorithms-users loop, and ways fairness has been operationalized and studied.These topics are organized across Sections 2–4.

2 REAL-WORLD EXAMPLES OF ALGORITHMIC UNFAIRNESS

Real-world AI applications, including criminal-risk assessment, medical and child-welfare systems, autonomous vehicles, advertising, facial recognition, and recommender systems, can produce discriminatory or harmful outcomes. The section illustrates these concerns through COMPAS and other applications, while highlighting tools such as Aequitas for assessing fairness across population subgroups.

  • Applications: AI systems used in courts, medicine, childhood welfare, and autonomous vehicles can directly affect people’s lives and harm society.These applications motivate attention to safety and fairness constraints.
  • Advertising: A supposedly gender-neutral STEM-job advertisement was seen by fewer women than men because of gender imbalance.The passage identifies younger women as a valuable subgroup in the resulting bias discussion.
  • Recognition and recommendation: Facial-recognition and recommender systems have often been evaluated as discriminatory toward certain populations and subgroups.Understanding the sources of these biases is described as important for preventing them.
  • Criminal-risk assessment: COMPAS is presented as a discriminatory commercial risk-assessment system with performance issues relative to non-expert human judgment.A cited study found COMPAS was no better than a normal human, despite using 137 features while study participants saw only 7.
  • Fairness assessment: Aequitas tests models using multiple bias and fairness metrics across population subgroups and generates reports to support decisions intended to avoid harm.Its users include data scientists, machine-learning researchers, and policymakers.

3 BIAS IN DATA, ALGORITHMS, AND USER EXPERIENCES

Bias in AI systems can originate in data, algorithms, and user interactions, with feedback loops intertwining these sources. Biased or unrepresentative data can be learned, amplified, and reflected in algorithmic outcomes, although group differences are not always discriminatory when justified by relevant attributes.

  • Bias in Data, Algorithms, and User Experiences: Biased training data can cause algorithms to learn, reproduce, amplify, and perpetuate existing biases in their predictions and outcomes.Data and algorithm behavior are tightly coupled because most AI systems are data driven.
  • Bias in Data: Measurement bias can arise from using mismeasured proxy variables, as illustrated by COMPAS using arrest-related variables to estimate crime or risk.Minority communities’ more frequent policing can make arrest-based proxies systematically problematic.
  • Bias in Data: Representation bias results from non-representative samples that omit population subgroups, such as geographically imbalanced image datasets that favor Western cultures.The passage links missing diversity during collection to demonstrable cultural bias.
  • Bias in Data: Aggregation bias can produce unsuitable conclusions when models ignore individual differences across ethnic and gender groups, as in clinical diabetes-aid tools.HbA1c levels differ in complex ways across genders and ethnicities.
  • Bias in Data, Algorithms, and User Experiences: Bias definitions are intertwined across data, algorithms, and user interaction because feedback loops connect data with algorithms and algorithms with users.The survey uses this feedback-loop perspective to categorize bias definitions.
  • Bias in Data, Algorithms, and User Experiences: Differences in treatment or outcomes are not illegal discrimination when relevant attributes justify and explain them, such as income differences in the UCI Adult dataset.The survey describes such differences as explainable rather than discriminatory.

4 ALGORITHMIC FAIRNESS

The section surveys competing algorithmic fairness definitions, reflecting the absence of a universal fairness concept. It organizes these definitions into individual, group, and subgroup notions while noting that some fairness constraints are inherently incompatible except in special cases.

  • Fairness lacks a universal definition, making the problem of identifying and mitigating algorithmic discrimination difficult.
  • The survey connects fairness definitions from political philosophy, education, and machine-learning research to algorithmic classification and public perceptions.
  • Group-oriented definitions include equalized odds, equal opportunity, and demographic parity, which respectively equalize true- and false-positive rates, true-positive rates, or positive-outcome likelihoods across groups.
  • The taxonomy distinguishes individual fairness, group fairness, and subgroup fairness, with subgroup fairness testing group constraints across many subgroups.
  • Some fairness constraints cannot be satisfied simultaneously except in highly constrained special cases, demonstrating an inherent incompatibility among fairness requirements.

5 METHODS FOR FAIR MACHINE LEARNING

Bias-mitigation methods in fair machine learning are organized into three categories—pre-processing, in-processing, and post-processing—based on where they intervene in the learning pipeline. These approaches span general machine learning and applications including natural language processing and deep learning.

  • Cross-domain method categories: Bias-mitigation methods generally fall into pre-processing, in-processing, and post-processing categories.The categories distinguish whether methods modify data, the learning algorithm, or outputs after training.
  • Pre-processing: Pre-processing transforms training data to remove underlying discrimination before model learning.It applies when the algorithm can modify the training data.
  • In-processing: In-processing modifies learning algorithms during training to remove discrimination through objective-function changes or constraints.It applies when the learning procedure can be changed.
  • Post-processing: Post-processing adjusts labels assigned by a trained black-box model using a holdout set when data and learning procedures cannot be modified.This intervention occurs after training and uses a function to reassign labels.
  • Cross-domain applications: Debiasing methods extend across AI domains, including fair representations and fair word embeddings in natural language processing and deep learning.The survey describes examples ranging from learning fair representations to learning fair word embeddings.

5.1 Unbiasing Data

Dataset curation decisions shape dataset fairness and can consequently affect algorithmic outcomes. Proposed mitigation practices include documenting dataset creation, characteristics, and motivation through datasheets.

  • Unbiasing Data: Dataset curators’ design decisions have consequences for dataset fairness, which in turn affects resulting algorithms.
  • Unbiasing Data: General bias-mitigation methods advocate good practices when using data.
  • Unbiasing Data: Datasheets can support data reporting by documenting dataset creation methods, characteristics, and motivations.

5.2 Fair Machine Learning

Fair machine learning research proposes fairness-aware methods across classification, regression, representation learning, online networks, and causal modeling. These works address group- and individual-level fairness, dataset and model amplification of bias, unequal representation quality, social-network disadvantage, and causal dependence on sensitive attributes.

  • Classification: Classification methods target subgroup fairness, equality of opportunity, equalized odds, disparate treatment, and disparate impact, with frameworks offering fairness guarantees and discrimination-free classifiers.Some methods also aim to remain stable when the test set changes.
  • Regression: Fair regression research defines individual, group, and hybrid fairness, while related methods address statistical parity, bounded group loss, disparate impact, and disparate treatment.One fair regression method evaluates accuracy–fairness trade-offs using the price of fairness (POF).
  • Representation and multimodal applications: 33% of cooking-image agent roles were men and 67% were women in imSitu, but model training amplified the bias, reducing men’s representation to 16%.The example concerns semantic role-labeling models and the imSitu training dataset.
  • Representation and multimodal applications: Vanilla PCA produced lower reconstruction error for men than women on LFW, motivating Fair PCA to create representations with similar richness across populations.The proposed method relaxes the Fair PCA objective to a semidefinite program and then reduces the solution’s rank through a linear program.
  • Online communities and networks: Online community methods can disadvantage users with fewer friends or followers because community detection may ignore low-connected users or assign them to irrelevant groups.Related fair methods are proposed for community detection, graph embedding, and clustering.
  • Causal modeling: Causal graphs are used to remove unwanted dependence of outcomes on sensitive attributes such as gender or race and to detect proxy or causal discrimination.The survey notes that observational criteria alone cannot determine whether a predictor exhibits unresolved discrimination, motivating causal reasoning frameworks.

5.3 Fair Representation Learning

Fair representation learning mitigates bias by removing sensitive information, disentangling multiple sensitive attributes, or adversarially restricting protected-attribute predictability. These approaches support fairness notions including demographic parity, equality of odds, equality of opportunity, and statistical parity.

  • Fair representation learning: Variational Fair Autoencoders treat sensitive variables as nuisances and use maximum mean discrepancy regularization to make latent representations invariant to them.Removing sensitive-variable information is intended to produce fair representations.
  • Fair representation learning: Flexibly fair representation learning disentangles information from multiple sensitive attributes while remaining adaptable to downstream labels and sensitive-attribute combinations.The approach addresses demographic parity for multiple sensitive attributes or any subset combination.
  • Fair representation learning: Adversarial representation learning maximizes predictor accuracy while minimizing an adversary’s ability to infer the protected or sensitive variable.The model contains a predictor trained to predict Y from X and an adversary designed to expose sensitive information.
  • Fair representation learning: The adversary’s inputs are selected according to the fairness definition, using predicted labels for Demographic Parity and true plus predicted labels for Equality of Odds.Equality of Opportunity is also described as an adversarial fairness target, though its specific input configuration is truncated in the passage.
  • Fair representation learning: For statistical parity, FairGAN training emphasizes distinguishing synthetic samples generated for protected and unprotected groups under matched conditional distributions.The protected variable s distinguishes the groups in the synthetic distributions.

5.4 Fair NLP

Fair NLP research identifies gender bias across word embeddings, coreference resolution, language modeling, sentence encoders, machine translation, and named entity recognition. Proposed remedies include embedding debiasing, data augmentation, bias metrics, and regularization, though reducing language-model bias can compromise perplexity.

  • Word Embeddings: Word embeddings associate “man” with “computer programmer” and “woman” with “homemaker,” motivating hard and soft debiasing methods for gender-neutral words.Hard debiasing neutralizes and equalizes representations, whereas soft correction reduces bias while retaining similarity to the original embedding.
  • Coreference Resolution: Coreference systems exhibit gender bias from training data, word embeddings, and unbalanced gender lists; WinoBias addresses these sources through entity swapping, debiased embeddings, and balancing.The approach generates auxiliary datasets by swapping male and female entities and combines augmentation with word2vec debiasing techniques.
  • Language Modeling: Language-model studies introduce word-level gender-bias metrics and embedding regularization, finding that bias reduction involves a compromise on perplexity.The regularization term minimizes encoder-embedding projections onto the gender subspace, while word-level metrics are reported as more effective than corpus-level metrics.
  • Sentence Encoders: Sentence Encoder Association Test generalizes WEAT-style bias measurement to sentence encoders, including CBoW, GPT, ELMo, and BERT.The study extends bias detection from word embeddings to sentence embeddings using state-of-the-art sentence encoding techniques.
  • Machine Translation: Machine translation studies expose gender bias in English-to-Spanish translation and Google Translate’s male favoritism for stereotypical STEM jobs, while mitigation also boosted performance by one BLUE score.The Spanish example concerns translating “friend” as the feminine form “amiga” in two contexts where the translation should remain the same.
  • Named Entity Recognition: NER systems more often misclassify or omit female names as person entities, and biased training data lacks sufficiently versatile female-name representation.Researchers formalize the observations with six evaluation metrics covering unweighted and frequency-weighted error types.

5.5 Comparison of Different Mitigation Algorithms

Algorithmic fairness is an emerging research area with proposed fair AI algorithms and bias-mitigation techniques evaluated across benchmark datasets. One example uses a geometric approach to learn fair representations by removing correlations between protected and unprotected features.

  • Algorithmic fairness remains a relatively new research area requiring further improvement.
  • Existing studies propose fair AI algorithms and bias-mitigation techniques and compare them using benchmark fairness datasets.
  • A geometric solution learns fair representations by removing correlations between protected and unprotected features.

6 CHALLENGES AND OPPORTUNITIES FOR FAIRNESS RESEARCH

Fairness research remains incomplete, with major opportunities to unify disparate definitions, shift from equality toward equity, and develop methods for detecting unfairness in datasets. Although work spans many AI subareas, research attention remains uneven across domains.

  • Challenges and Opportunities: Synthesizing disparate fairness definitions remains an open problem because it would enable more unified and comparable evaluation.Different definitions cover varied use cases, making it difficult to assess how a fairness solution performs under another definition.
  • Challenges and Opportunities: Current fairness definitions mainly emphasize equality, while equity—providing groups the resources they need to succeed—has received little attention.Operationalizing equity and examining how it augments or contradicts existing fairness definitions remain future research directions.
  • Challenges and Opportunities: Detecting unfairness in particular datasets remains challenging because fairness definitions vary and each requires nuanced detection methods.Prior work has detected Simpson’s Paradox as an instance of data bias, but broader unfairness detection requires further consideration.
  • Challenges and Opportunities: Fairness research spans natural language processing, representation learning, community detection, and other areas, but these subareas have not received equal attention.The survey’s taxonomy summarizes broad research activity while highlighting uneven coverage across domains and fairness definitions.

7 CONCLUSION

The survey examines bias and unfairness in AI through data and algorithmic dimensions, illustrating real-world harms, fairness definitions, and approaches across AI domains.

  • 7 CONCLUSION: The survey frames AI bias and unfairness primarily through two dimensions: data and algorithms.It introduces problems that can adversely affect AI systems and explains why fairness is important.
  • 7 CONCLUSION: Unfair AI applications can cause societal harm in judicial systems, face recognition, and promoting algorithms.These examples demonstrate the potential real-world consequences of unfairness.
  • 7 CONCLUSION: The survey reviews proposed definitions of fairness and bias and work addressing biases across machine learning, deep learning, and other AI domains.It presents methods and research areas intended to stimulate further interest in mitigating AI bias.

9 APPENDIX

The appendix catalogs widely known datasets used to study or mitigate bias and fairness in machine learning. These datasets span income, credit, language, crime, recidivism, parliamentary representation, and face recognition, with some designed to improve balance and diversity.

  • Dataset overview: Fairness datasets target documented bias by enabling evaluation of inequalities or introducing more balanced and representative data.The appendix distinguishes datasets used to address fairness issues from datasets created to target biases observed in older datasets.
  • Tabular fairness datasets: The UCI Adult and German Credit datasets support studies of gender or race inequalities in income and credit-related decisions.UCI Adult uses 1994 census attributes and income above or below $50K/yr, while German Credit contains 1000 credit records.
  • Language fairness datasets: WinoBias evaluates gender bias in coreference resolution through pronoun links to male- or female-stereotypical occupations.Its Winograd-format sentences cover 40 occupations and include two challenge types involving gendered pronouns.
  • Representation and vision datasets: The Pilot Parliaments Benchmark was released to improve gender and race balance, diversity, and representativeness across 1270 parliamentary individuals.The images represent parliaments from three European and three African countries.
  • Representation and vision datasets: Diversity in Faces supports fairness research in face recognition with one million annotations spanning varied facial, demographic, image-quality, and geographic characteristics.The dataset includes variation in craniofacial distances, skin color, symmetry, contrast, age, pose, gender, resolution, areas, and ratios.
Loading 1908.09635v3…