Algorithmic Fairness Datasets: the Story so Far

Alessandro Fabris; Stefano Messina; Gianmaria Silvello; Gian Antonio Susto

Source-linked AI summary

Algorithmic Fairness Datasets: the Story so Far

Alessandro Fabris, Stefano Messina, Gianmaria Silvello, Gian Antonio Susto

arXiv:2202.01711v4Feb 3, 2022cs.CY

TL;DR

Fair ML depends on adequately documented datasets, yet the field faces a documentation debt caused by opacity and scattered information. The paper surveys and documents over two hundred datasets, examines Adult, COMPAS, and German Credit, and finds that their limitations call into question their suitability as general-purpose fairness benchmarks while documenting diverse alternatives and curation practices.

Problem
Fair ML research faces a data documentation debt because information about datasets is insufficient and scattered, hindering transparent and responsible data use.
Method
The authors survey over two hundred fairness datasets, produce standardized documentation, deeply document three popular benchmarks, and analyze curation topics including anonymization, consent, inclusivity, sensitive attributes, and transparency.
Results
Adult, COMPAS, and German Credit have limited merits beyond originating from human processes and encoding protected attributes, while their contrived tasks, noisy data, coding mistakes, sensitive-attribute limitations, and age challenge their benchmark status.
Takeaways & Limitations
The survey provides documented alternatives across many domains and fairness tasks, alongside best-practice considerations for curating new resources.
Takeaways & Limitations
Adult's contrived income-prediction task and arbitrary $50,000 threshold make accuracy and fairness highly sensitive to the threshold.

Abstract

from arXiv · show

Data-driven algorithms are studied in diverse domains to support critical decisions, directly impacting people's well-being. As a result, a growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair Machine Learning hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity). In this work, we target data documentation debt by surveying over two hundred datasets employed in algorithmic fairness research, and producing standardized and searchable documentation for each of them. Moreover we rigorously identify the three most popular fairness datasets, namely Adult, COMPAS and German Credit, for which we compile in-depth documentation. This unifying documentation effort supports multiple contributions. Firstly, we summarize the merits and limitations of Adult, COMPAS and German Credit, adding to and unifying recent scholarship, calling into question their suitability as general-purpose fairness benchmarks. Secondly, we document and summarize hundreds of available alternatives, annotating their domain and supported fairness tasks, along with additional properties of interest for fairness researchers. Finally, we analyze these datasets from the perspective of five important data curation topics: anonymization, consent, inclusivity, sensitive attributes, and transparency. We discuss different approaches and levels of attention to these topics, making them tangible, and distill them into a set of best practices for the curation of novel resources.

1 Introduction

The paper addresses algorithmic fairness’s documentation debt by surveying datasets, standardizing their documentation, and analyzing prominent benchmarks and curation practices.

Documentation debt combines opacity in individual datasets with sparsity of information across the research community, contributing to misunderstanding and misuse.
The survey covers over two hundred datasets used in fair ML research and produces compact, standardized data briefs for them.
The authors deeply document Adult, COMPAS, and German Credit, extracting their merits and limitations and questioning their suitability as general-purpose fairness benchmarks.
The dataset survey categorizes resources by domain, supported fairness tasks, and their roles in fairness research, enabling domain-oriented and task-oriented search.
The authors compare anonymization, consent, inclusivity, sensitive-attribute labeling, and transparency practices to derive best practices for new and existing datasets.

2 Related Work

Prior surveys mainly classify fairness measures, mitigation methods, biases, software, or selected subfields, while this work combines broad dataset coverage with deep analysis of popular resources.

Existing fairness surveys typically focus on fairness measures and methods, with some addressing bias sources, software, or selected subfields.
Compared with a concurrent study of 15 tabular fairness datasets, this work emphasizes rigorous inclusion methodology and broader coverage.
Related multi-dataset studies examine specific issues such as computer-vision bias, ethical concerns, labeling transparency, and institutional concentration.
Single-dataset studies provide deeper analyses of consent, problematic content, re-identification, representation, or datasheet documentation.
The paper combines multi-dataset breadth with single-dataset depth by surveying numerous resources and producing datasheets and nutrition labels for the most popular ones.
It adopts existing documentation frameworks for popular datasets while introducing lightweight data briefs for scalable fairness-dataset documentation.

3 Methodology

The methodology screens fairness-related research articles, selects datasets treated as important artifacts, and documents eligible resources through a multi-source review process.

The study considers conference proceedings from fairness, ethics, machine-learning, and data-mining venues, alongside domain-specific workshops.
The fairness-focused subsample is selected using title strings covering fairness, bias, discrimination, equality, equity, disparate impact, and parity.
The screening leaves 558 articles after excluding works focused on other model or performance topics.
Datasets are selected when used to train or test algorithms or when subjected to an in-depth data audit, then documented using articles, references, and external scholarly or official sources.
The inclusion criteria exclude most word embeddings, toy datasets, minor auxiliary resources, and datasets with insufficient available information.
226 datasets satisfy the criteria, each receiving a data brief; the three most popular resources are then identified for detailed analysis.

4 Most Popular Datasets

Adult, COMPAS, and German Credit dominate fairness research usage despite substantial limitations, while broader dataset utilization follows a long-tail distribution. Their documented strengths and weaknesses constrain their suitability as general-purpose fairness benchmarks.

4 Most Popular Datasets: Adult, COMPAS, and German Credit are the most-used fairness datasets, with 118, 81, and 35 reported usages, respectively.The remaining heavily used datasets include Communities and Crime, Bank Marketing, Law School, CelebA, MovieLens, and Credit.
4.1 Adult: Adult combines a population-oriented census sample, consensual self-reported attributes, trained interviewers, and well-documented source variables.The original sample was designed to represent the US population, although later filtering was somewhat arbitrary.
4.1 Adult: Adult’s income-prediction task has limited apparent social utility, and its arbitrary $50,000 threshold makes accuracy and fairness sensitive to threshold choice.Roughly 7% of data points have missing values, and the dataset also contains reporting, top-coding, and outdated-attribute issues.
4.2 COMPAS: COMPAS captures a socially relevant risk-assessment setting, but disparate sources create clerical errors, mismatches, redundant variables, leakage, and preprocessing ambiguity.The dataset records defendants screened in Broward County and includes recidivism targets defined by bookings within two years after screening.
4.2 COMPAS: COMPAS labels re-arrest rather than re-offense, while criminal-history covariates reflect policing practices that may differ across racial groups.This construct–observation mismatch means fairness in the observed data may not represent fairness in the underlying criminal-offense construct.
4.3 German Credit: German Credit reflects real bank credit scoring but is half a century old, affected by selection and oversampling, and historically suffered serious documentation and sex-coding problems.Applicants were already deemed creditworthy and granted loans, while bad credits were oversampled to balance target classes; corrected releases address some coding errors.

5 Existing Alternatives

Existing fairness datasets span many application domains and support diverse tasks, while serving roles such as training, benchmarking, and evaluation in the literature.

5.1 Domain: Fairness datasets span domains including criminal justice, education, search, marketplaces, emergency response, social media, medicine, and hiring.
5.1 Domain: The surveyed resources cover domains from computer science, social sciences, computer vision, health, linguistics, economics and business, and miscellaneous areas.
5.2 Task and setting: Fairness tasks range from generic fair classification to narrower applications such as fair districting and fair truth discovery.
5.2 Task and setting: Tasks are complicated by settings including noisy labels, system dynamics, and privacy concerns, which can cut across classification, regression, and ranking.
5.3 Role: Datasets support training, catalyzing attention, and signalling awareness, but are not generally used to track progress over time because no single fairness measure is agreed upon.
5.3 Role: Overall, fairness datasets originate from varied domains, support diverse tasks, and play different roles across algorithmic fairness research.

6 Best Practices for Dataset Curation

The survey shows that responsible dataset curation requires balancing privacy, consent, inclusivity, sensitive-attribute validity, and transparency rather than treating any single property as sufficient.

6.1 Re-identification: Strong anonymization can protect individuals, but privacy-preserving transformations may reduce social relevance, while dataset age can lessen the likelihood and consequences of re-identification.Toy datasets are safe from re-identification but have low social relevance; semi-synthetic datasets seek a compromise between safety and realism.
6.2 Consent: Consent practices should distinguish privacy from copyright, obtain informed agreement where feasible, and support later withdrawal despite the persistence of derivatives and backchannels.FACES obtained consent from every participant, whereas face-recognition datasets provide recurring examples of non-consensual collection and conflation of licenses with ethical permission.
6.3 Inclusivity: Representation alone does not ensure benefits: collection objectives, downstream uses, and potential harms must be considered alongside demographic balance.Examples include demographic balancing in ImageNet and more diverse recent Framingham cohorts, contrasted with WebText’s skew toward Reddit users and harmful contexts for some ImageNet classes.
6.4 Sensitive Attribute Labelling: Sensitive-attribute labels require careful nomenclature and provenance because external annotation can encode stereotypes, while photographic race or ethnicity labels may capture annotator-perceived rather than self-identified categories.The survey finds that face-dataset documentation often omits how gender and race taxonomies were selected and describes annotation procedures only minimally.

7 Broader Relevance to the Community

The paper releases standardized data briefs and organizes fairness datasets to support dataset selection, search, bias analysis, and identification of understudied areas.

7 Broader Relevance to the Community: Data briefs provide standardized documentation for datasets used in fairness research.The schema is publicly available, and the briefs support a shared documentation format.
7 Broader Relevance to the Community: The documentation supports domain-oriented and task-oriented dataset search.
7 Broader Relevance to the Community: The survey helps identify understudied areas and sub-problems in algorithmic fairness.
7 Broader Relevance to the Community: The dataset collection supports multi-dataset studies organized around shared attributes, domains, or popularity.

8 Conclusions and Recommendations

The paper concludes that documentation debt limits the value of popular fairness benchmarks, while broader dataset documentation and curation guidance provide alternatives and best practices.

8 Conclusions and Recommendations: Adult, COMPAS, and German Credit have limited merits as general-purpose fairness benchmarks.The paper identifies contrived tasks, noisy data, coding mistakes, sensitive-attribute limitations, and age-related concerns.
8 Conclusions and Recommendations: The survey documents over two hundred alternative datasets across many domains and fairness tasks.Examples include criminal justice, education, search, marketplaces, emergency response, social media, medicine, hiring, and finance.
8 Conclusions and Recommendations: The paper analyzes re-identification, consent, inclusivity, labeling, and transparency across fairness datasets.These analyses are distilled into best practices for future dataset curation.
8 Conclusions and Recommendations: Consent should be considered during initial dataset design using established frameworks.The paper specifically references the FRIES framework and emphasizes that consent should not be left to post hoc consideration.
8 Conclusions and Recommendations: Inclusivity broadens representation but must be evaluated alongside dataset uses, costs, and benefits.The paper warns that inclusivity alone does not resolve concerns about impacts on data subjects and wider populations.
8 Conclusions and Recommendations: Sensitive attributes are important for measuring inclusion but should preferably be self-reported and carefully documented when externally assigned.The paper treats labeling as a shortcut whose advantages and disadvantages require explicit consideration.
8 Conclusions and Recommendations: Reliable documentation records curation choices, dataset context, sampling, features, annotations, tasks, and contributors.This information helps users select appropriate datasets and avoid unintentional misuse.
8 Conclusions and Recommendations: Documentation debt results from opacity and sparse information, and it can propagate through dataset derivatives.The authors argue that curators, users, and reviewers can jointly maintain a data documentation culture.

aware reinforcement learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

The supplied passages consist of bibliographic entries from the fairness literature, including works on computer vision, privacy, optimization, and social networks.

aware reinforcement learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): The references include work on fairness and bias mitigation in computer vision.
aware reinforcement learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): The bibliography includes research on fairness with noisy protected groups and decision-making.
aware reinforcement learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): The cited literature spans social networks, health, hate speech, and recommendation systems.

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak

The supplied passages are bibliographic entries covering scientific data stewardship, fairness, privacy, machine learning, and related datasets.

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak: The references include principles for scientific data management and stewardship.
Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak: The bibliography covers fairness risks, candidate screening, and causal fairness frameworks.
Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak: The cited works also include de-anonymization and machine-learning benchmark datasets.

of Black’s stratification hypothesis. Journal of Quantitative Criminology 28(2):265–293

This passage block is a bibliography of work spanning fairness-aware learning, evaluation, representation, ranking, recommendation, and bias mitigation.

Several references study fairness-aware representations, adversarial learning, sensitive-subspace robustness, and trade-offs between fairness and utility.
The cited works address fairness in classification, including disparate treatment, disparate impact, disparate mistreatment, parity, and preference-based objectives.
The bibliography also covers fairness in ranking, recommendation, collaborative filtering, and reciprocal recommendation settings.

and fairness in class incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

This bibliography covers fairness and bias across meta-learning, representation learning, language, vision, recommendation, and sequential decision-making.

The cited works include fair meta-learning, fair representations, and fairness-aware modeling under class imbalance or overlapping groups.
References address gender bias in language and recommendation systems, including corpus-level constraints and coreference-resolution debiasing.
Other works examine bias and fairness in computer vision, face recognition, depth prediction, and long-term qualification or retention decisions.

(2015) Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 1506.06724

The cited works concern fairness-aware recommendation, fairness metrics, and the relationship between accuracy and fairness in binary classification.

One line of work develops fairness objectives for collaborative filtering and recommendation systems.
Another studies how fairness is measured in ranked outputs and binary classification.
The references connect fairness evaluation with accuracy considerations in classification.

Appendix A Data briefs

Data briefs provide concise, standardized documentation of datasets used in fairness research, combining descriptive metadata, taxonomy, sensitive-feature information, and access references.

Appendix A Data briefs: Data briefs are short documentation records for essential properties of datasets used in fairness research, reviewed by dataset contacts for over 95% of surveyed resources.
Appendix A Data briefs: Each brief describes a dataset’s purpose, features, labeling procedure, and envisioned machine-learning task.
Appendix A Data briefs: The schema records creator affiliation, domain, fairness-literature tasks, data format, sample size, sensitive features, links, and further information.
Appendix A Data briefs: Sensitive features include legally salient group attributes and, where context warrants, domain-specific or implicit attributes such as language, brand ownership, or demographic references in text.
Appendix A Data briefs: Domains follow a Scimago-based taxonomy extended with news, social media, social networks, sports, and food, while fairness tasks were labeled through open coding.

B.1 Datasheet

The datasheet documents the provenance, composition, targets, sampling, missingness, confidentiality, and known limitations of the Adult, COMPAS, and German Credit datasets.

Adult: Adult is a filtered sample of 1994 US household survey respondents aged at least 17, earning above $100, and working at least one hour weekly.These filters may introduce sampling effects, while the underlying CPS sample was designed to represent the US population.
Adult: Adult includes self-reported sex, age, and race categories, but over 7% of instances have missing values and household relationships are not reported.The categories are bounded by predefined labels, and respondents may under-report income or provide inaccurate information.
German Credit: German Credit is a bank-selected sample of creditworthy applicants with 30% bad-credit cases oversampled, while its documentation contains errors and sex cannot be cleanly recovered.These properties constrain representativeness and interpretation, especially for applications differing from post-screening credit decisions.