PaperlayerView exact revision on arXiv

Source-linked AI summary

Fairness Testing: A Comprehensive Survey and Analysis of Trends

Zhenpeng Chen, Jie M. Zhang, Max Hort, Mark Harman, Federica Sarro

arXiv:2207.10223v4cs.SE

TL;DR

Unfair ML behavior creates a need for testing methods that assess fairness across demographic groups. This paper surveys 100 fairness-testing studies, organizing them by workflow and testing components while analyzing trends, datasets, tools, and research opportunities. The survey reports that 89% of publications emerged since 2019 and identifies test-oracle selection and tool adoption as ongoing challenges.

  • Problem

    Unfair ML decisions can disadvantage protected groups, while fairness testing faces difficult oracle design because demographic groups and competing fairness definitions must be considered together.

  • Method

    The paper surveys 100 fairness-testing papers, categorizes them by testing workflow and components, analyzes research trends, and compiles datasets and open-source tools.

  • Results

    89% of fairness testing publications emerged since 2019, and the survey identifies test-oracle selection as an important area for developing automatic reliable oracles.

  • Takeaways & Limitations

    The survey provides researchers and practitioners with an overview of current fairness-testing research, accessible datasets, open-source tools, and promising directions.

  • Takeaways & Limitations

    The paper’s literature collection from DBLP may overlook papers published in other venues.

Abstract

from arXiv · show

Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this paper offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely-adopted datasets and open-source tools for fairness testing.

1 INTRODUCTION

ML software can produce unfair outcomes affecting protected groups, creating a need for fairness testing despite difficult oracle design and competing fairness definitions. This survey organizes 100 studies by testing workflow and components, analyzes trends, and identifies datasets and tools.

  • Motivation: Unfair ML decisions can disadvantage minority and protected groups in socially consequential domains such as hiring, credit assessment, and criminal justice.The paper cites a recidivism system that incorrectly assigned higher risk to Black defendants than to White defendants.
  • Fairness in Software Engineering: Fairness is treated as a non-functional software property that should be considered throughout the software engineering process.Prior work frames fairness as a requirement, architecture concern, verification property, testing property, and repair objective.
  • Testing Challenges: Fairness testing is challenging because test oracles must consider inputs and outputs from different demographic groups simultaneously.Different fairness definitions can require different oracles, and some definitions may be mathematically impossible to satisfy concurrently.
  • Research Growth: 89% of fairness testing publications emerged since 2019, indicating the emergence of a new software-testing domain.The cumulative publication trend is reported through 2023.
  • Survey Scope: The survey collects 100 papers, categorizes them by testing workflow and testing components, analyzes trends and opportunities, and summarizes public datasets and open-source tools.The surveyed venues span software engineering, artificial intelligence, computer security, and human-computer interaction.
  • Positioning: Unlike prior surveys covering adjacent fairness topics or only 20 software-fairness testing papers, this work specifically surveys fairness testing of ML software.The authors describe it as the first comprehensive survey dedicated specifically to this literature.

2 PRELIMINARIES

The paper frames fairness through individual and group definitions, using sensitive attributes to specify conditions that ML software should satisfy. It distinguishes widely adopted definitions by how they compare individuals, demographic groups, predictions, and outcomes.

  • Fairness definitions primarily divide into individual fairness, which seeks similar outcomes for similar individuals, and group fairness, which seeks similar treatment across demographic groups.
  • Sensitive attributes, including sex, race, age, and physical ability, are protected characteristics used in fairness assessment.
  • Individual fairness: Individual fairness definitions include unawareness, awareness, counterfactual fairness, and causal fairness, with different treatments of sensitive attributes and their relationships to outcomes.
  • Group fairness: Group fairness definitions include statistical parity, equalized odds, and equal opportunity, which constrain favorable-outcome probabilities or error-related rates across groups.

3 SURVEY METHODOLOGY

The survey defines a focused scope for fairness testing, collects papers through database searching, snowballing, and author feedback, and analyzes them using thematic synthesis. The resulting corpus contains 100 papers organized around testing workflows and components.

  • The survey includes papers introducing fairness testing ideas, approaches, studies, frameworks, or tools for ML software, while excluding unrelated systems and process-fairness topics.
  • Paper collection: The authors searched DBLP using iteratively refined keywords, producing 7,674 hits and selecting 67 relevant papers after manual inspection.
  • Paper collection: Snowballing expanded the collection by identifying transitively related papers, while contacting authors yielded 15 additional suggestions, 8 of which met the inclusion criteria.
  • Paper collection: 100 papers were included in the survey, calculated as 67 keyword-search papers plus 25 snowballed papers plus 8 author-suggested papers.
  • Paper analysis: Thematic synthesis organized the papers by testing workflow and components through full-text analysis, author discussion, double-checking, and feedback from surveyed-paper authors.

4 FAIRNESS TESTING WORKFLOW

Fairness testing research is organized around two key activities: generating test inputs and identifying test oracles. These activities support testing whether ML software satisfies defined fairness conditions.

  • Test oracle identification determines how generated or sampled inputs are evaluated against fairness conditions.
  • The survey organizes existing techniques around these activities as central parts of the fairness-testing workflow.

4.1 Test Input Generation

Fairness test-input generation spans random, search-based, verification-based, and domain-specific techniques, with search-based methods using guided exploration to improve discriminatory-instance discovery. The surveyed approaches commonly separate global exploration from local neighborhood search, while domain-specific methods address input-generation constraints.

  • Fairness test-input generation techniques include random, search-based, verification-based, and domain-specific approaches, organized by their adopted generation strategy.
  • Random generation varies sensitive attributes while assigning non-sensitive values, but it can have a low success rate for producing discriminatory inputs.
  • Search-based generation: Two-phase search methods first explore the input space for discriminatory instances, then search their neighborhoods for additional instances.
  • DNN techniques: ADF, EIDIG, NeuronFair, and DICE are two-phase search-based techniques specifically proposed for DNN fairness testing.
  • Search-based generation: Search-based techniques guide generation with gradients, momentum, biased neurons, information-theoretic clustering, symbolic execution, explanations, or genetic algorithms.
  • Domain-specific generation: Conventional GANs struggle to make precise image changes without affecting other facial features or hairstyles, motivating adapted GAN-based generation for computer-vision testing.

4.2 Test Oracle Identification

Fairness testing uses test oracles to determine whether ML software satisfies fairness requirements, but oracle design is difficult because fairness compares demographic groups and competing definitions may conflict. Existing approaches use metamorphic relations and statistical measurements to operationalize these judgments.

  • Fairness test oracles determine whether software meets fairness requirements and whether a fairness bug exists.
  • Metamorphic Relations as Test Oracles: Metamorphic relations compare outputs across transformed inputs, enabling bug detection without inspecting specific expected outputs.For fairness, transformations commonly alter sensitive attributes while preserving or appropriately adjusting other attributes.
  • Metamorphic Relations as Test Oracles: Classification fairness tests commonly compare otherwise similar individuals who differ in sensitive attributes and expect identical classification outcomes.This relation has been applied across tabular, text, and other data domains.
  • Metamorphic Relations as Test Oracles: Fairness transformations must account for data modality, with templates for text and generative techniques such as GANs for images.For causal or counterfactual fairness, modifying sensitive attributes may also require changing causally affected non-sensitive attributes; otherwise, inputs can become unnatural.
  • Metamorphic Relations as Test Oracles: Regression and generation systems require specialized metamorphic relations because continuous predictions and generated text make outcome differences difficult to judge.Researchers use thresholds for regression and NLP-based similarity measures for natural language generation.
  • Statistical Measurements as Test Oracles: Statistical measurements identify fairness violations by comparing group-level rates or ratios against predefined thresholds.Examples include Statistical Parity Difference and Disparate Impact; the IBM AIF360 toolkit alone provides over 70 measurements, making measure selection a requirements-engineering challenge.

5 FAIRNESS TESTING COMPONENTS

Fairness testing can target multiple parts of an ML-enabled system, including training data, ML programs, models, frameworks, and non-ML components. Existing studies primarily focus on training data, ML programs, and ML models.

  • Fairness testing covers training data, ML programs, ML models, ML frameworks, and non-ML components.The surveyed studies primarily examine training data, ML programs, and ML models.

5.1 Data Testing

Data testing addresses fairness bugs that can be learned from training data and propagated through ML development. Studies examine feature, label, and selection bias using statistical, causal, distributional, and model-based analyses.

  • Training-data fairness bugs can propagate through model development and create biased ML software systems.Data testing therefore examines bias in data features, labels, and distributions.
  • Feature Bias: Feature selection substantially influences model fairness, while correlated non-sensitive attributes can create discrimination even without explicit sensitive attributes.The Amazon delivery case illustrates how zipcode-related correlations could enable racial bias despite race not being directly used.
  • Feature Bias: Feature-bias detection uses association models, causal graphs, latent-space analysis, and counterfactual comparisons to identify direct, indirect, or unknown biased attributes.Causal approaches distinguish direct effects from effects transmitted through non-sensitive features.
  • Label Bias: Label bias arises when irrelevant factors influence outcome-label generation, including through human or algorithmic annotation.Facial-expression datasets showed significant gender-related label bias, especially for happiness and anger, and conventional repair methods did not completely mitigate it.
  • Selection Bias: Selection bias occurs when sampling creates unintended associations between sensitive attributes and outcomes, as illustrated by race-related correlations in the Compas dataset.Researchers use distribution testing to compare demographic representation, favorable rates, and dataset characteristics.
  • Selection Bias: Public face and vision datasets exhibit demographic, racial, gender, geographic, and object-representation skews that can produce downstream performance disparities.Caucasian faces were overrepresented in one face-dataset analysis, and detectors trained on skewed data showed demographic bias.

5.2 ML Program Testing

ML program testing examines implementation choices that can alter fairness, including preprocessing, imputation, hyper-parameters, repair algorithms, and compression. The surveyed evidence shows that fairness interventions often involve trade-offs, variability, or unintended effects on accuracy and other forms of bias.

  • Data processing, decision logic, and runtime configurations can introduce discordance between desired fairness conditions and actual ML software behavior.
  • Testing Data Processing: Preprocessing and missing-value imputation strategies can change fairness outcomes, with some preprocessing methods introducing fairness bugs and others improving fairness.
  • Testing Hyper-parameters: Hyper-parameter testing searches for settings that balance fairness with ML performance, including multi-objective and evolutionary approaches.Fairway and Parfait-ML approximate fairness-performance trade-offs rather than optimizing fairness in isolation.
  • Testing Fairness Repair Algorithms: Fairness repair testing evaluates whether mitigation reduces fairness bugs without reducing accuracy or introducing other side effects.
  • Testing Fairness Repair Algorithms: Most tested repair techniques produced undesirable effects, including reduced accuracy, increased fairness variance, or increased accuracy variance.Other studies found improved fairness could coincide with significant accuracy drops, and sometimes both fairness and accuracy worsened.
  • Testing Fairness Repair Algorithms: Fairea provides a unified baseline for comparing fairness-performance trade-offs, and a large-scale evaluation applied it to 17 mitigation methods across multiple metrics.The evaluation covered 12 ML performance metrics, 4 fairness metrics, and 24 fairness-performance trade-off measurements.
  • Testing Fairness Repair Algorithms: Repair can remove desirable discrimination involving anti-protected attributes, while intrinsic and extrinsic bias may remain uncorrelated and evaluation can depend strongly on the metric and dataset.
  • Testing Compression Algorithms: Compression methods can change model bias inconsistently: pruning and quantization amplified gender bias in one vision task, whereas distilled language models exhibited less bias.Across facial-expression systems, compression algorithms introduced fairness bugs without consistent findings.

5.3 Model Testing

Model fairness testing spans black-box and white-box methods, statistical and metamorphic analyses, input generation, training-data inspection, and neuron-level diagnosis across diverse ML systems.

  • Testing approaches: Fairness testing primarily evaluates individual ML models using black-box or white-box access to model internals and training data.Black-box testing analyzes behavior without internal access, whereas white-box testing examines training data or model structure.
  • Testing approaches: Statistical measurements detect unfair prediction associations, while metamorphic relations test whether predictions change unexpectedly after sensitive-attribute transformations.Themis checks consistency for individuals differing only in sensitive attributes; Aequitas and ExpGA search for discriminatory instances.
  • Application domains: Black-box fairness testing supports NLP, computer vision, and ranking systems through text templates, GAN-generated image mutants, and demographic representation analysis.Ranking studies also compare recommendation performance and alignment across demographic groups.
  • Hybrid and data-based methods: Some methods approximate black-box behavior with white-box models, inspect training-data neighbors, or compare fairness measurements before and after model annotation.These strategies expose discriminatory inputs, explain unfavorable predictions, or identify bias amplification without requiring complete model access.
  • Test generation and subgroup analysis: Fairness testing also uses subgroup discovery and combinatorial t-way sampling to locate group-level disparities and discriminatory protected-attribute mutations.FAIRVIS clusters statistically similar subgroups and ranks them by group fairness metrics, while t-way testing generates constrained diverse datasets.
  • Deep-learning methods: Deep-learning techniques diagnose unfairness through neuron activation distances, sensitive-feature conflict paths, gradient-guided input generation, biased-neuron analysis, significance testing, and causal mediation.FairNeuron selectively retrains conflict paths so identified instances rely on important predictive features rather than sensitive attributes.

6 RESEARCH TRENDS AND DISTRIBUTIONS

The survey finds fairness-testing research expanding across venues, ML categories, data types, fairness notions, and testing manners, with especially strong representation for deep learning and tabular applications.

  • 6.1 Research Venues: 49% of fairness testing papers appear in software engineering venues, while 42% appear in artificial intelligence venues.The collected work also spans computer security, human-computer interaction, and mobile computing communities.
  • 6.2 Machine Learning Categories: 50 papers (50%) focus on deep-learning software, 41 papers (41%) target general ML software, and 8 papers (8%) address both.The survey attributes deep learning’s substantial coverage partly to its widespread adoption and lower interpretability.
  • 6.2 Machine Learning Categories: Since 2019, publications testing deep-learning software have surged past publications testing general ML software.Before 2019, fairness-testing research primarily concentrated on general ML.
  • 6.3 Data Types: 55% of publications test applications with tabular inputs, while approximately 24% address text and 24% address image inputs.In SE venues, tabular applications account for 79%, compared with 17% for text and 8% for images.
  • 6.3 Data Types: Speech and recommendation-system fairness testing each account for approximately 1% of publications and remain relatively underexplored.The survey identifies these data types as needing further research attention.
  • 6.4 Fairness Categories: 46% of papers study group fairness, 46% study individual fairness, and 8% study both fairness categories.The similar overall distribution contrasts with test-generation techniques, which are nearly all proposed for individual fairness.
  • 6.5 Testing Manners: 53% of the 100 papers use black-box testing and 47% use white-box testing.The survey relates black-box predominance to privacy and legal barriers that can restrict disclosure of internal system information.
  • 6.6 Tasks and Sensitive Attributes: Income prediction, credit risk prediction, and recidivism/crime prediction are the three most widely studied tasks.Sex/gender, race/ethnicity, and age are the most common sensitive attributes, appearing in 89%, 57%, and 41% of papers, respectively.

7 DATASETS AND TOOLS

The survey catalogs public fairness datasets and open-source testing tools to support navigation and selection, while noting dataset usage constraints and practitioner adoption difficulties.

  • 7.1 Datasets: The dataset overview records dataset sizes, data types, sensitive attributes, usage scenarios, and access links.Many listed datasets contain tabular data suitable for traditional ML classifiers.
  • 7.1 Datasets: Text and image datasets have become more available alongside the growth of natural language processing and computer vision.Some datasets impose usage restrictions; CelebA is limited to non-commercial research.
  • 7.2 Tools: The paper summarizes 41 open-source fairness testing tools across general ML, deep learning, NLP, and computer vision domains.The catalog is intended to help researchers and practitioners select suitable tools.
  • 7.2 Tools: Practitioners face a steep learning curve and limited guidance when adopting fairness tools.The tool summary is presented as a response to this adoption gap.

8 RESEARCH OPPORTUNITIES

Fairness testing remains embryonic, with open problems spanning sensitive-attribute availability, oracle construction, natural test inputs, test adequacy, and broader fairness scenarios.

  • Sensitive attributes: Existing techniques often require sensitive attributes, yet such information may be unavailable, imperfect, withheld, or restricted by privacy regulations.Inference-based substitutes may be unsatisfactory and limited in application.
  • Sensitive attributes: Intersectional fairness testing remains underexplored because most studies examine one sensitive attribute at a time.Multiple attributes introduce compounded effects and potential over-segmentation concerns.
  • Test oracles: Reliable automatic fairness oracles remain an open challenge because existing metamorphic and statistical oracles require human ingenuity, while fairness measurements can conflict.The IBM AIF360 toolkit alone provides more than 70 fairness measurements, making oracle selection impractical across all contexts.
  • Test inputs: Generated fairness-test inputs may be unnatural because feature perturbations can ignore causal relationships, real-world constraints, and the effects of changing multiple sensitive attributes.Future work should evaluate naturalness automatically and generate inputs for group fairness.
  • Test adequacy: Fairness-testing adequacy remains unaddressed, and traditional or ML coverage metrics lack empirical evidence demonstrating that they detect fairness bugs or establish test sufficiency.Fairness-specific adequacy metrics may therefore be needed.

8.5 Test Cost Reduction

Future fairness testing must reduce testing costs while expanding beyond current data types, tasks, and offline workflows, and accounting for interactions with other software properties.

  • Test cost reduction: Fairness testing can require model retraining, repeated predictions, or extensive data generation, yet no research has focused on reducing its cost.Test selection, prioritization, and minimization are proposed directions.
  • Test cost reduction: Resource-constrained platforms create a challenge for conducting fairness testing with limited computing power, memory, and energy.Mobile and other end devices require methods adapted to their constraints.
  • Other properties: Fairness repair should be tested alongside robustness, security, efficiency, interpretability, privacy, and ML performance because relationships among these properties remain poorly understood.Fairness improvement is often reported to come at the cost of ML performance.
  • Other properties: Explainability can help summarize fairness-bug causes, inform targeted repair, and communicate bias to nontechnical stakeholders.The paper identifies limited guidance for producing targeted fixes.
  • Broader applications: Fairness testing should expand beyond tabular, NLP, and computer-vision classification systems to speech, video, multimodal, recommendation, regression, clustering, and LLM applications.LLM testing additionally faces unexplored question types and limited transparency in many large-scale models.
  • Broader directions: The field needs more online fairness testing, additional testing activities, component-level studies, and tools accessible to nontechnical stakeholders.Current research is predominantly offline, while existing tools generally require programming skills.
Loading 2207.10223v4…