Source-linked AI summary
A survey on datasets for fairness-aware machine learning
Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, Eirini Ntoutsi
TL;DR
Fairness-aware ML needs realistic and diverse benchmark datasets because ML decisions can discriminate through protected attributes and their proxies, even when sensitive attributes are omitted. This survey organizes real-world tabular datasets, analyzes attribute relationships with Bayesian networks and exploratory analysis, and finds bias in data and/or predictive-model outcomes with substantial variation across datasets.
Problem
ML-based decision-making can discriminate through protected attributes and proxy variables, making realistic and diverse benchmark datasets important for evaluating fairness-aware methods.
Method
The survey organizes real-world tabular fairness-aware ML datasets by application and learning characteristics, then analyzes attribute relationships using Bayesian networks and exploratory analysis across three fairness measures.
Results
The analysis finds that bias may appear in the data and/or predictive-model outcomes, while experimental outcomes vary significantly across datasets.
Takeaways & Limitations
Researchers should select diverse datasets and appropriate fairness notions for each domain because no universal fairness notion applies to every problem.
Takeaways & Limitations
There is no universal fairness notion for every problem, so evaluating fairness-aware algorithms remains challenging across domains.
Abstract
from arXiv · showhide
As decision-making increasingly relies on Machine Learning (ML) and (big) data, the issue of fairness in data-driven Artificial Intelligence (AI) systems is receiving increasing attention from both research and industry. A large variety of fairness-aware machine learning solutions have been proposed which involve fairness-related interventions in the data, learning algorithms and/or model outputs. However, a vital part of proposing new approaches is evaluating them empirically on benchmark datasets that represent realistic and diverse settings. Therefore, in this paper, we overview real-world datasets used for fairness-aware machine learning. We focus on tabular data as the most common data representation for fairness-aware machine learning. We start our analysis by identifying relationships between the different attributes, particularly w.r.t. protected attributes and class attribute, using a Bayesian network. For a deeper understanding of bias in the datasets, we investigate the interesting relationships using exploratory analysis.
1 Introduction
Fairness-aware ML evaluation still lacks an overview of the real-world benchmark datasets it uses. This survey organizes such datasets and analyzes their attribute relationships and fairness-related behavior.
- Protected attributes can bias ML models through causal effects and proxy attributes, even when those protected attributes are omitted.Examples include zip code acting as a proxy for race and credit rating as a proxy for safe driving.
- An overview of datasets used to evaluate fairness-aware machine learning remains missing despite existing surveys of fairness methods.
- The survey characterizes fairness-related datasets by application domain, protected attributes, cardinality, dimensionality, and class imbalance.
- A Bayesian network identifies conditional relationships among attributes, especially those involving protected and class attributes, to reduce the search space for bias analysis.The survey then examines relevant relationships through exploratory analysis and evaluates predictive and fairness measures quantitatively.
- The survey gathers scattered datasets and organizes them to support selection by application domain, learning challenge, classification task, and fairness challenge.
- The paper highlights the need for open benchmarks spanning diverse domains, contexts, learning challenges, and fairness notions.
2 Methodology of the survey process
The survey collects public fairness-related datasets, represents their attribute dependencies with constrained Bayesian networks, and evaluates fairness using several measures. Its methodology combines dataset selection, structural analysis, discretization, exploratory analysis, and quantitative fairness evaluation.
- Dataset collection: Public datasets were identified through Google Scholar searches and retained when used in fairness-related classification or clustering experiments.Datasets from sources such as Kaggle were excluded from consideration.
- Bayesian network: A Bayesian network represents dataset attributes as nodes and their conditional relationships as directed edges.
- Bayesian network: Bayesian-network structure learning selects an optimal model by maximizing the likelihood of generating the dataset while favoring lower complexity.
- Bayesian network: The learned network constrains the class attribute to be a leaf node so analysis can focus on factors determining its prediction probability.
- Bayesian network: Direct edges indicate stronger attribute correlation than indirect connections, and a protected-to-class path may indicate dataset bias with respect to that protected attribute.
- Data preparation: Continuous attributes are discretized into meaningful categories to keep Bayesian-network learning computationally manageable.
- Fairness metrics: Because no fairness measure is universally suitable, the survey reports statistical parity, equalized odds, and ABROCA.Statistical parity does not require compliance with ground-truth labels, which can limit its suitability in some ML scenarios.
- Fairness metrics: Statistical parity compares predicted outcomes between protected and non-protected groups, with 0 indicating no discrimination and values in [-1,1].Positive values indicate discrimination against the protected group, while negative values indicate reverse discrimination.
3 Datasets for fairness
The survey examines real-world fairness datasets by modeling attribute dependencies with Bayesian networks and exploring relationships involving protected and class attributes. Across financial, criminological, and other datasets, it identifies demographic disparities and dataset-specific fairness considerations.
- Analysis approach: Bayesian networks model dependencies among dataset attributes, including relationships involving protected attributes and class labels.The survey discretizes numerical and categorical variables where needed before generating the networks.
- Financial datasets: Adult data show strong links between education, sex, working hours, age, relationship status, and income.Highly educated people have high income, while males outnumber females by at least 5 times in the high-education, high-income segment; males working over 40 hours are approximately 7 times more numerous.
- Financial datasets: KDD Census-Income data show sex- and race-associated income disparities and conditional dependence of income on occupation and weeks worked.High-income records include 13,691 males versus 3,711 females; income above 50K occurs for 3.2% of non-white people and 6.7% of white people.
- Other application domains: Other datasets reveal protected-group disparities in credit outcomes, occupations, recidivism scores, and default payments.Young German-credit customers are recognized as bad customers at 42.1% versus 27.2% for older customers, while 62.6% of males versus 32.7% of females have high-level occupations in another dataset.
- Cross-dataset considerations: The survey emphasizes that fairness analysis must account for dataset-specific protected attributes, historical bias, missing data, and relationships among protected variables.Race and sex are commonly protected attributes, race is especially prevalent, and careful attribute selection is required for fairness-aware learning.
4 Experimental evaluation
The survey uses logistic regression to characterize predictive and fairness performance across benchmark datasets, rather than experimentally comparing fairness-aware learning methods. Results vary substantially by dataset, with different datasets excelling on different fairness and predictive measures.
- Evaluation setup: The evaluation uses logistic regression on binary classification tasks to characterize dataset difficulty through predictive and fairness performance.The experiment uses a single 70%:30% training/test split for each dataset.
- Evaluation setup: The reported measures include accuracy, balanced accuracy, statistical parity, equalized odds, ABROCA, and protected- versus non-protected-group TPR and TNR.The confusion-matrix framework distinguishes protected and non-protected groups for group-specific rates.
- Results: Predictive performance and fairness differ significantly across datasets, so no single dataset dominates every evaluation dimension.Ricci peaks on both accuracy and fairness measures, while Communities & Crime has the best accuracy and OULAD the worst accuracy.
- Results: 10/15 datasets have absolute statistical parity below 0.1, with Diabetes, Credit card clients, and OULAD reaching 0.0.Bank marketing has the worst statistical parity value among the reported datasets.
- Results: Equalized odds is 0.0 for Credit card clients, Diabetes, OULAD, and Ricci, while COMPAS recid. performs worst.Law school and Communities & Crime follow COMPAS recid. among the weakest equalized-odds results.
- Results: Ricci has the best ABROCA, followed by OULAD and KDD Census-Income, whereas German credit and COMPAS are worst.The ABROCA slicing plot compares ROC curves for non-protected and protected groups.
5 Open issues on datasets for fairness-aware ML
The survey broadens its dataset discussion beyond popular static benchmarks to newer real, synthetic, and sequential datasets. These datasets support analysis of temporal, spatial, dynamic-population, underrepresentation, and censored-label settings, but several areas require further investigation.
- Scope: The survey extends its review to recently proposed real datasets, synthetic datasets, and datasets for sequential decision making.It advocates developing datasets that represent diverse fairness scenarios alongside new fairness methods and algorithms.
- Adult reconstruction and ACS PUMS datasets: The Adult reconstruction dataset contains 49,531 instances and 14 attributes, with income represented as a continuous variable.It was reconstructed from 1994 Current Population Survey data and matches 13 of 15 Adult attributes.
- Adult reconstruction and ACS PUMS datasets: In California from 2007–2019, ACS PUMS analysis identifies female and non-white as protected values for sex and race.The study uses income as the prediction task.
- Adult reconstruction and ACS PUMS datasets: The ACS PUMS analysis shows lower representation of female and non-white groups, while income above 50K$ increases in both sexes at a slightly higher male growth rate.The analysis examines proportions over time by gender and race.
- Adult reconstruction and ACS PUMS datasets: Gender differences in the positive class increase across many US states from 2011 to 2019, motivating further spatial and temporal analysis.The positive class is income above 50K$.
- Synthetic and sequential datasets: Synthetic data simulate evolving populations for long-term sequential fairness studies or augment underrepresented protected groups through SMOTE interpolation.Other sequential datasets transform Adult and Census into data streams by processing individuals in sequence and ordering them by sensitive attributes.
- Synthetic and sequential datasets: Fair sequential decision-making also includes uncertainty from censored class labels, as illustrated by clinical prediction settings using SUPPORT.This differs from fairness studies that assume certainty about the class label.
- Synthetic and sequential datasets: Sequential counterfactual experiments used Adult but did not investigate fairness, leaving this direction open for further research.The passage connects explainability with debugging models and uncovering biased decision making.
6 Conclusion and outlook
The conclusion presents the survey as a foundation for analyzing fairness datasets and developing evaluation frameworks. It emphasizes scope limitations, dataset bias, domain-specific fairness notions, protected-attribute choices, and the need for newer data.
- Scope and limitations: The survey focuses on tabular data, although fairness-aware machine learning also uses text and images requiring specialized dataset handling.Applying fairness algorithms across these data types may require adapting the algorithms to different datasets.
- Conclusions: Bayesian-network analysis shows conditional attribute relationships, while experiments indicate that bias may occur in datasets and predictive-model outcomes.The survey frames its analysis as a basis for developing fairness-algorithm frameworks or simulation environments.
- Open challenges: Fairness evaluation remains domain-specific because no universal fairness notion applies across finance, education, healthcare, criminology, and other domains.Selecting or defining an appropriate fairness notion for each problem remains a major challenge.
- Open challenges: Protected-attribute selection depends on the domain, problem, and experimental purpose, while proxy-attribute identification and handling require more research.Gender, race, age, and marriage are prevalent protected attributes in the surveyed datasets.
- Open challenges: The surveyed datasets average about 20 years old, range from 48 years old to 7 years old, and mainly come from the US and Europe.The paper calls for newer datasets and further policies on data quality and collection.
Funding Information
The paper acknowledges support from the Ministry of Science and Education of Lower Saxony, Germany.
- Funding: Funding was provided by the Ministry of Science and Education of Lower Saxony, Germany.The project ID is 51410078.
- Funding: The listed funding project has ID 51410078.The acknowledgment identifies Lower Saxony’s Ministry of Science and Education as the funder.
- Funding: The funding acknowledgment names a German Ministry of Science and Education.The specified ministry is located in Lower Saxony.
1. Adult dataset
The supplied passage for the Adult dataset section consists of a citation list to prior fairness-aware machine-learning studies.
- The passage cites Krasanakis et al. (2018) and multiple studies by Kamiran and Calders.
- It also cites work by Calders, Žliobaitė, Luong, Iosifidis, and Ntoutsi.
- Additional cited authors include Calmon, Feldman, Hajian, Zafar, and others.
2. KDD Census-Income dataset
The supplied passage for the KDD Census-Income dataset section consists of citations to studies associated with Iosifidis, Ntoutsi, Ristanoski, and Zhang.
- The passage cites Iosifidis and Ntoutsi in multiple works.
- It also cites Ristanoski et al. (2013).
- The citation list includes W. Zhang and Ntoutsi (2019).
3. German credit dataset
The supplied passage for the German credit dataset section consists of a broad citation list spanning fairness, discrimination, representation, and machine-learning studies.
- The passage cites work by Calders, Luong, Ruggieri, Pedreschi, and Turini.
- It includes studies by Iosifidis and Ntoutsi, Feldman, Hajian, Kamiran, Fish, and Zemel.
- The list also cites Friedler, Mancuhan, Ristanoski, Choi, Ruoss, Ahn, and Chakraborty.
4. Dutch census dataset
The supplied passage for the Dutch census dataset section consists of citations to studies by Kamiran, Calders, Žliobaitė, and other researchers.
- The passage cites Kamiran and Calders across several works.
- It also cites Žliobaitė et al. (2011).
- Additional citations include Xu, Cardoso, Agarwal, and other authors.
5. Bank marketing dataset
The section contains a citation list associated with the Bank marketing dataset.
- The section cites multiple works in connection with the Bank marketing dataset.
- The cited works include Grari et al. (2019) and Zafar et al. (2019).
- The citation list also includes studies by Krasanakis et al. (2018) and Zafar, Valera, Rogriguez, & Gummadi (2017).
6. Credit card clients dataset
The section concerns the Credit card clients dataset and includes a citation list, with a separate reference to the Communities & Crime dataset.
- The cited literature includes work by Yeh & Lien (2009), Berk et al. (2017), and Esmaeili et al. (2020).
- The section includes the heading “Communites & Crime dataset.”
- Additional cited studies include Kamiran and Calders, Feldman, Friedler, Kearns, and Ruoss.
B Datasets’ characteristics
The appendix tables report continued attribute characteristics for KDD Census-Income, COMPAS recid, Communities and Crime, and Diabetes.
- Table 16 reports continued attribute characteristics for KDD Census-Income.
- Table 17 reports continued attribute characteristics for COMPAS recid.
- Tables 18 and 19 report continued attribute characteristics for Communities and Crime.
- Table 20 reports continued attribute characteristics for Diabetes.