Source-linked AI summary
Aequitas: A Bias and Fairness Audit Toolkit
Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T. Rodolfa, Rayid Ghani
TL;DR
AI fairness research lacks consensus on metrics and has limited operational support for auditing real-world systems. Aequitas provides an open-source, workflow-integrated toolkit for auditing multiple group-based bias and fairness measures, and its case studies report that audited models were often more accurate and no more biased than existing alternatives. The authors conclude that auditing can support more equitable policy decisions, while noting that additional materials and workshops are still needed for practitioners and policymakers.
Problem
There is no consensus on fairness metrics or definitions, and few resources operationalize them, so bias auditing is not yet standard practice.
Method
Aequitas is an open-source toolkit that audits model outputs using multiple bias and fairness metrics across population subgroups within the machine-learning workflow.
Results
Across criminal justice, public health, and public safety projects, deployed models were often more accurate and equally or less biased than policymakers’ existing alternatives.
Takeaways & Limitations
Auditing can support more equitable decisions around developing, deploying, and maintaining machine-learning models in real-world policy settings.
Takeaways & Limitations
The authors state that additional materials and workshops are needed to help data scientists and policymakers understand bias impacts and make informed decisions.
Abstract
from arXiv · showhide
Recent work has raised concerns on the risk of unintended bias in AI systems being used nowadays that can affect individuals unfairly based on race, gender or religion, among other possible characteristics. While a lot of bias metrics and fairness definitions have been proposed in recent years, there is no consensus on which metric/definition should be used and there are very few available resources to operationalize them. Therefore, despite recent awareness, auditing for bias and fairness when developing and deploying AI systems is not yet a standard practice. We present Aequitas, an open source bias and fairness audit toolkit that is an intuitive and easy to use addition to the machine learning workflow, enabling users to seamlessly test models for several bias and fairness metrics in relation to multiple population sub-groups. Aequitas facilitates informed and equitable decisions around developing and deploying algorithmic decision making systems for both data scientists, machine learning researchers and policymakers.
1. Introduction
AI systems increasingly affect high-stakes areas, yet bias auditing is not standard practice. Aequitas addresses limited empirical evaluation and operational support by providing an open-source toolkit for auditing models across metrics and population subgroups.
- Motivation: AI systems optimized for efficiency or effectiveness can produce disparate errors and unfair outcomes across demographic groups.Examples include facial-recognition errors affecting darker-skinned women and gender bias in machine translation.
- Motivation: Bias in general-purpose systems can have dramatic consequences when deployed in policy areas such as employment, surveillance, and law enforcement.The paper cites COMPAS and related discussions as an example.
- Motivation: AI development has advanced faster than policies and practices addressing the ethical implications of ubiquitous AI.The authors describe this mismatch as a barrier to responsible deployment.
- Research gap: Existing research develops bias-detection, mitigation, and fairness methods, but extensive empirical evaluation across real-world policy problems remains limited.The paper also notes a lack of consensus about metrics, definitions, tradeoffs, and practical solutions.
- Contribution: Aequitas is an open-source toolkit integrated into the machine-learning workflow for auditing multiple bias and fairness metrics across population subgroups.The toolkit was released in May 2018 and is intended to support data scientists and policymakers.
- Contribution: The toolkit reached more than 2,400 web-app sessions from 1,300 unique users, while its source code averaged 300 views and 20 clones per week.These usage figures are reported as evidence of immediate impact in data science and public policy communities.
2. Related Work
Prior fairness research offers multiple definitions and tools but limited empirical validation across real-world policy settings. Aequitas differentiates itself by contextualizing audits for policy use and supporting both technical and non-technical users.
- Fairness definitions: Impact fairness concerns prediction disparities, while treatment fairness concerns whether sensitive attributes are explicitly used in modeling.Aequitas focuses on assessing impact fairness with parity-based measures.
- Fairness definitions: Statistical or demographic parity requires equal representation of groups among predicted positives, whereas impact parity compares positive-prediction fractions across groups.Both notions assess distributions of predicted outcomes across groups.
- Fairness definitions: Equalized odds requires equal false-negative and false-positive rates across groups, while equal opportunity relaxes this requirement toward the positive outcome.These definitions are discussed in the context of risk assessment.
- Fairness tradeoffs: Calibration with different group prevalence makes it impossible to achieve both equal false-positive and false-negative rates across groups.The cited work characterizes this as an incompatibility between calibration and balance for both classes.
- Fairness definitions: Individual fairness requires similar entities to receive similar predictions regardless of sensitive attributes.Prior work models similarity using distance measures and Lipschitz conditions.
- Empirical evaluation: Prior work often relied on synthetic datasets and did not emphasize empirical analysis of bias definitions or validation of proposed solutions.COMPAS and a later Pennsylvania child-maltreatment case study are identified exceptions or examples.
- Audit workflow: The toolkit places audits before model selection, before operationalization, and periodically during deployment to detect fairness degradation.The workflow assigns auditing responsibilities to data scientists and policymakers at different stages.
- Toolkit landscape: Aequitas provides policy-context guidance and supports both data scientists and policymakers through workflow integration and a web application.This distinguishes it from repositories focused mainly on ad-hoc analyses or code implementations.
3. Aequitas
Aequitas integrates bias and fairness audits into model development, selection, deployment, and monitoring for data scientists and policymakers. It operationalizes group-based assessments for policy settings where limited resources require prioritizing the top k entities.
- Toolkit and workflow: Aequitas lets users upload audit data, configure protected and reference groups, and generate bias reports through a Python library or command-line interface.Audits can assess models before selection, before operationalization, or in A/B testing with revised algorithms.
- Toolkit and workflow: The toolkit is designed for data scientists comparing models and policymakers assessing bias before acceptance and periodically during production.Its workflow places internal and external audits around model selection, deployment, retraining, and data-collection decisions.
- Policy decision setting: Public-policy systems often use skewed classes and precision at top k because intervention resources limit action to a small number of entities.The model ranks entities by risk score, after which a human expert may select the top k for assistive or punitive intervention.
- Group definitions: Aequitas represents risk assessment with a real-valued score S, a binary decision Ŷ, and a true binary outcome Y, then defines groups by shared attribute values.A reference group is selected for calculating bias measures, while prevalence is the fraction of group members with a positive true outcome.
- Bias and fairness metrics: Its distributional group metrics describe selected-set composition without labels, while error-based metrics compare false-positive and false-negative disparities across groups.Predicted prevalence is the fraction of a group predicted positive; predicted positive rate is the fraction of predicted positives belonging to that group.
- Bias and fairness metrics: Bias is defined as a group metric disparity relative to a reference group, with parity-based fairness controlled by a threshold parameter τ.The toolkit frames auditing as a way to avoid disproportionate benefits or harms and support informed model-selection decisions.
- Toolkit and workflow: Aequitas addresses an adoption barrier by helping users navigate many metrics and connect them to the policy problem through the collaboratively designed Fairness Tree.The authors identify sustained interaction between policymakers and AI developers as a central barrier to routine auditing.
4. Case Studies
Aequitas audits models across real-world public policy projects, comparing performance and fairness across demographic groups. Case studies in criminal justice, public health, and policing reveal group disparities, while comparisons with existing or expert baselines identify models with fewer biases in some settings.
- Cross-case evaluation: Aequitas was evaluated across large-scale public health, criminal justice, and public safety projects using real-world risk prediction systems.The authors describe this as an extensive empirical comparison of several bias and fairness measures across policy domains.
- Criminal Justice: The recidivism project prioritized 150 of roughly 400,000 individuals using Precision at the top 150 and temporal cross-validation.The data comprised 1.5 million individuals over ten years, with more than 3,000 features and several hundred models evaluated.
- Criminal Justice: The selected recidivism model showed no gender disparity but exhibited FOR disparity of 2X for young adults, 1.6x for African Americans, and 1.3x for Hispanics.Young adults had FOR of 6% versus 3% for 36–55 year olds; African Americans had 5.6% versus 3.5% for White individuals.
- Criminal Justice: The recidivism model also showed unsupervised disparities: 79% of the top group were male, 82% were under 35, and African Americans and Hispanics were overrepresented relative to White individuals.Impact disparity was 5X for young people and 2X for African Americans in the top 150.
- Criminal Justice: The city’s existing recidivism approach was more biased than the selected model for both FOR parity and FDR parity.The audit was used to compare the model’s disparities with the current policy based on recent interaction frequency.
- Public Health: In the public health comparison, the Random Forest was less biased than the expert model for both groups, while females had lower FOR than males in each model.The reported values were FOR-Expert = 6.4% versus 10% and FOR-RandomForest = 5.7% versus 8.9% for females and males, respectively.
- Public Health: The public health audit also identified higher FOR for White individuals than African-Americans and highlighted high relative FNR among young adults aged 18–25.The authors note that this age group is high-risk in the target city and therefore especially important to monitor.
- Public Safety and Policing: The policing model showed bias across gender, race, and marital status, including unfavorable FDR bias for divorced officers and, to a lesser extent, African-American officers.It also showed slight unfavorable FOR bias for females, divorced officers, African-American officers, and officers of unknown race.
5. Conclusions and Future Work
Aequitas audits AI systems used in high-impact policy areas to support more equitable model decisions. Across three policy domains, audited ML models were generally more accurate and no more biased than existing policymaker alternatives, while further educational materials remain needed.
- Aequitas is a toolkit for auditing group bias and fairness in AI systems used for criminal justice, education, public health, workforce development, and social services.
- Audits across criminal justice, public health, and public safety and policing found biases in many deployed ML models.
- Many audited ML models were more accurate and equally or less biased than policymakers’ existing alternatives.
- Well-audited ML models were reported as more effective at solving policy problems while reducing inequities.
- The authors identify the work as a start and call for materials and workshops to help developers and policymakers understand bias impacts and make informed decisions.
Appendix A. Reproducible Audit
The reproducible audit appendix demonstrates Aequitas on publicly available COMPAS data because input data from other audits could not be shared. The associated Jupyter notebook is available in the Aequitas GitHub repository.
- Input data from the Center for Data Science and Public Policy audits could not be shared.
- The appendix presents a short COMPAS case study using a publicly available criminal-justice dataset.
- A Jupyter notebook containing the COMPAS audit code is available in the Aequitas GitHub repository.
- COMPAS was designed to identify people at risk of recidivism for pretrial release decisions and was described by ProPublica as biased against Black defendants.
A.1 Data
The COMPAS data come from Broward County records released by ProPublica and combine risk scores, recidivism outcomes, and demographic variables for 7,214 individuals.
- The dataset contains COMPAS recidivism risk decile scores and two-year recidivism outcomes.
- It includes demographic and other variables for 7,214 individuals scored during 2013 and 2014.
- The data are based on Broward County records made publicly available by ProPublica.
A.2 Bias Audit using Aequitas
Aequitas found unfairness in COMPAS across age, gender, and race, using group and disparity metrics evaluated with a fairness threshold of τ = 0.8. The relevant disparity depended on the intervention context and attribute.
- Aequitas detected both unsupervised and supervised unfairness in COMPAS across age, gender, and race.
- Group metrics: Figure 7 organizes rows by attribute-value pairs and columns by group metrics, with green bars indicating no bias and red bars unfavorable bias relative to a reference group.
- Group metrics: Using τ = 0.8, every considered group metric showed some bias toward specific groups.
- Group metrics: Predicted Positive Rate results indicated disproportionate selection of younger people, Native Americans, and African-Americans relative to group sizes.
- Disparity metrics: For punitive COMPAS decisions, Aequitas focuses on False Positive Rate and False Discovery Rate parity because false positives can harm individuals.
- Disparity metrics: Figure 8 reports race, age, and sex disparities, including nearly double African-American versus Caucasian FPR, 1.6X higher FPR for people under 25 versus ages 25–45, and 1.34 times higher female versus male FDR.