Source-linked AI summary
An algorithm for removing sensitive information: application to race-independent recidivism prediction
James E. Johndrow, Kristian Lum
TL;DR
The paper addresses whether recidivism models can make race-independent predictions when re-arrest data and permitted covariates may encode racial bias. It proposes likelihood-based data transformations that remove protected-variable information across mixed-scale variables. Applied to Broward County recidivism data, the method produces nearly identical predictive distributions by race, while the paper notes important dependence on conditional-model quality and biased outcome data.
Problem
Race omission does not ensure fair predictions because permitted covariates can correlate with race, while re-arrest may itself be a biased measure of re-offense.
Method
The paper uses likelihood-based transformations and conditional models to remove protected-variable information and achieve mutual independence from race across mixed-scale covariates.
Results
Adjusted logistic-regression and random-forest models produced nearly identical predictive distributions by race, unlike models trained on unadjusted data.
Takeaways & Limitations
Transforming training data can provide race-independent predictions across models and enable adjusted datasets to be shared for downstream model building.
Takeaways & Limitations
The method requires suitable, carefully specified conditional models and remains difficult to assess when re-arrest is a biased sample of post-release criminal activity.
Abstract
from arXiv · showhide
Predictive modeling is increasingly being employed to assist human decision-makers. One purported advantage of replacing or augmenting human judgment with computer models in high stakes settings-- such as sentencing, hiring, policing, college admissions, and parole decisions-- is the perceived "neutrality" of computers. It is argued that because computer models do not hold personal prejudice, the predictions they produce will be equally free from prejudice. There is growing recognition that employing algorithms does not remove the potential for bias, and can even amplify it if the training data were generated by a process that is itself biased. In this paper, we provide a probabilistic notion of algorithmic bias. We propose a method to eliminate bias from predictive models by removing all information regarding protected variables from the data to which the models will ultimately be trained. Unlike previous work in this area, our framework is general enough to accommodate data on any measurement scale. Motivated by models currently in use in the criminal justice system that inform decisions on pre-trial release and parole, we apply our proposed method to a dataset on the criminal histories of individuals at the time of sentencing to produce "race-neutral" predictions of re-arrest. In the process, we demonstrate that a common approach to creating "race-neutral" models-- omitting race as a covariate-- still results in racially disparate predictions. We then demonstrate that the application of our proposed method to these data removes racial disparities from predictions with minimal impact on predictive accuracy.
1 Introduction
The paper argues that omitting race does not ensure fair recidivism predictions because permitted covariates can encode race. It proposes likelihood-based transformations that remove protected-variable information across mixed-scale data and applies them to produce statistically race-independent predictions.
- Motivation: High-stakes criminal justice models require fairness because their predictions inform decisions about personal liberty.The paper highlights pre-trial release, bail, sentencing, and parole as key applications.
- Motivation: Omitting race is insufficient when permitted covariates are correlated with race, allowing racially disparate predictions.The paper identifies this as omitted-variable bias and demonstrates the concern in recidivism prediction.
- Fairness framework: Existing fairness approaches can impose incompatible accuracy criteria or rely on biased re-arrest outcomes.The paper therefore favors independence from race when racial differences in recidivism cannot be precisely quantified.
- Fairness framework: Statistical parity defines fairness as independence between the predictive distribution and the protected variable.The paper operates within this disparate-impact definition, sometimes called demographic parity.
- Proposed method: The proposed likelihood-based framework adjusts arbitrary numbers of mixed-scale variables toward mutual independence from protected variables.It uses a chain of conditional models and matches conditional quantiles to marginal quantiles.
- Application: In Broward County data, unadjusted models produced racially different predictive distributions, whereas adjusted logistic-regression and random-forest models produced nearly identical distributions.The application predicts re-arrest within two years using demographic and criminal-history covariates.
2 Method for variable adjustment
The paper defines fairness as prediction independence from protected characteristics and transforms covariates to remove their information while minimizing information loss. A chained multivariate construction extends univariate adjustments to mutual independence.
- 2.2 Transportation framework: The proposed method transforms covariates into variables independent of protected characteristics while preserving as much information as possible.Information loss is quantified through a transportation cost between the original and transformed distributions.
- 2.1 Setup: Fairness requires predictions to be independent of protected characteristics, not merely that protected variables be omitted from the model.Because predictors and protected characteristics are usually dependent, excluding the protected variable alone is insufficient.
- 2.2 Transportation framework: The optimal coupling minimizes the total transportation cost between the original and transformed covariate distributions.With Euclidean cost, keeping transformed covariates close to the originals is linked to limiting predictive-accuracy loss for linear models.
- 2.3 Univariate transformations: For univariate variables, the algorithm uses conditional distributions and stochastic maps to produce transformed variables independent of protected characteristics with minimal information loss.The construction handles both continuous and atomic variables, including cases where the conditional distribution has point masses.
- 2.4 Multivariate adjustments via chaining: The method can provide pairwise independence through separate covariate adjustments, but separate adjustments do not guarantee mutual independence of the full transformed covariate set.Pairwise adjustment may nevertheless be useful when preserving covariate interpretability is important.
- 2.4 Multivariate adjustments via chaining: The multivariate transformation adjusts variables sequentially using protected characteristics and previously adjusted covariates, yielding mutual independence from the protected variables.Each transformed component represents the part of its original covariate not predictable from the protected characteristics and earlier adjusted covariates.
3 Simulation Example
The simulation compares unadjusted, pairwise-adjusted, and jointly adjusted covariates while omitting the protected variable from prediction models. Only joint adjustment makes the conditional prediction distributions approximately equal across protected groups.
- Simulation setup: The simulation generates a binary protected variable, two dependent covariates, and an outcome whose distribution differs across protected groups.The sample contains n = 10,000 observations, and the target is equal conditional distributions of fitted predictions.
- Simulation setup: Three procedures are compared: no adjustment, pairwise transformations to independence, and transformations to mutual independence.The protected variable is omitted from every fitted prediction model, and fitted values are compared conditional on the protected variable.
- Adjustment procedures: The continuous covariate is adjusted using a fitted conditional Gaussian distribution, while the count covariate uses a fitted Poisson distribution and stochastic quantile mapping.The joint procedure estimates the second covariate conditionally on the protected variable and the first adjusted covariate.
- Results: The unadjusted model leaves large differences between the conditional predictive distributions, while pairwise adjustment reduces but does not eliminate the discrepancy.Under pairwise adjustment, the distribution for one protected group retains a much longer right tail.
- Results: Joint adjustment yields approximately equal conditional predictive distributions, with ˆf(ˆy | z = 1) ≈ˆf(ˆy | z = 0).This matches the simulation’s stated fairness objective for the fitted prediction.
4 Application: Removing racial bias in recidivism risk assessment
The recidivism application tests whether race remains encoded in covariates and predictions after omitting race, then evaluates a transformation designed to remove that information. In the Broward County data, adjustment largely equalizes predictive distributions across race while preserving similar overall discrimination.
- Data and fairness context: The Broward County dataset uses criminal-history covariates, sex, age, and race to predict re-arrest within two years, with race treated as the protected variable.
- Dependence between race and covariates: G tests found strong evidence of dependence between race and every covariate, indicating that omitting race alone is unlikely to produce fair predictions.
- Transformations to independence: The transformation estimates conditional distributions and samples adjusted covariates so that race-related information is removed, while accounting for the full conditional distribution rather than only its expectation.
- Validation of adjustment: Conditional CDFs were approximately uniform by race, and pairwise Cramer’s V was reduced to near zero in most adjusted data, indicating successful removal of race information up to two-way interactions.
- Prediction results: Adjusted random-forest predictions had nearly identical distributions by race, eliminating almost all racial disparities in predicted re-offense probabilities.
- Prediction results: AUC changed from 0.72 for unadjusted data to 0.71 for adjusted data, while false-positive-rate variation decreased from 0.04 to 0.02.
- Prediction results: Adjustment increased disparities in positive and negative predictive values, but slightly increased accuracy variation while substantially reducing false-positive-rate variation.
5 Discussion
The framework adjusts datasets so models are mutually independent of protected variables, while supporting arbitrary variable types when suitable conditional models can be specified. The discussion also identifies practical boundaries involving risk-score interpretation, sampling bias, and the role of nontechnical experts in defining fairness.
- Method and contribution: The framework adjusts datasets so models trained on them are mutually independent of protected variables.It supports arbitrary numbers and types of protected and adjusted variables when suitable conditional models describe the data.
- Practical implications: Adjusting training data can guarantee fair predictions across multiple organizations, regardless of their modeling choices.The authors contrast this with selecting the top x% within each protected class.
- Practical implications: Protecting continuous variables or multiple protected variables can make within-class ranking impractical because discretization or sparse combinations may be required.These constraints motivate an approach that adjusts the training data directly.
- Open questions: Judges’ interpretations of coarsened risk categories such as “high risk” may differ from interpretations of predicted probabilities.The authors note that some highest-risk groups may still be more likely to behave well after release.
- Open questions: Biased re-arrest sampling makes disparate impact difficult to assess, motivating an independence standard for recidivism in this paper.The authors call for methods that can combine multiple samples where additional data may mitigate sampling bias.
- Open questions: Fairness definitions should be developed with ethicists, policy experts, and affected communities rather than within technical fields alone.The discussion distinguishes mathematical criteria such as independence, false-positive-rate equality, and positive-predictive-value equality.
A Appendix
The appendix compares logistic-regression predictions from unadjusted and transformed data by race. It uses cumulative distributions, densities, and ROC curves to assess prediction distributions and performance across adjustment procedures.
- Logistic-regression analysis: Logistic regression predictions from unadjusted and transformed data are compared by race using cumulative distributions and densities.The analysis mirrors the earlier comparison conducted with random forests.
- Logistic-regression analysis: Figure 8 displays the cumulative distribution and density of logistic-regression predictions by race.These plots support comparison of prediction distributions across the unadjusted and adjusted data conditions.
- Performance comparison: Figure 9 presents ROC curves for the predictive performance of each adjustment procedure.The figure provides a performance comparison across the procedures evaluated in the appendix.