Source-linked AI summary
Algorithmic decision making and the cost of fairness
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq
TL;DR
Pretrial risk algorithms can produce racial disparities, motivating formal fairness interventions. This paper formulates fairness as constrained optimization and finds that fairness constraints generally require race-specific thresholds, whereas public-safety maximization uses one threshold, creating a practical trade-off.
Problem
Pretrial algorithms can incorrectly classify black defendants as high risk more often than white defendants, motivating efforts to reduce racial disparities.
Method
The paper reformulates algorithmic fairness as constrained optimization, maximizing immediate utility measured by crime-prevention benefits minus detention costs under formal fairness constraints.
Results
Fairness-constrained rules generally require race-specific thresholds, while the unconstrained public-safety-maximizing rule uses one uniform threshold; Broward County data show a stark trade-off between these objectives.
Takeaways & Limitations
Choosing between fairness definitions and public safety is consequential, although a single-threshold rule treats equally risky defendants alike and can satisfy an important equality principle.
Takeaways & Limitations
Risk-score fairness cannot be assessed from decisions alone because unintentional omissions or biased features may affect the algorithm itself.
Abstract
from arXiv · showhide
Algorithms are now regularly used to decide whether defendants awaiting trial are too dangerous to be released back into the community. In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk. To mitigate such disparities, several techniques recently have been proposed to achieve algorithmic fairness. Here we reformulate algorithmic fairness as constrained optimization: the objective is to maximize public safety while satisfying formal fairness constraints designed to reduce racial disparities. We show that for several past definitions of fairness, the optimal algorithms that result require detaining defendants above race-specific risk thresholds. We further show that the optimal unconstrained algorithm requires applying a single, uniform threshold to all defendants. The unconstrained algorithm thus maximizes public safety while also satisfying one important understanding of equality: that all individuals are held to the same standard, irrespective of race. Because the optimal constrained and unconstrained algorithms generally differ, there is tension between improving public safety and satisfying prevailing notions of algorithmic fairness. By examining data from Broward County, Florida, we show that this trade-off can be large in practice. We focus on algorithms for pretrial release decisions, but the principles we discuss apply to other domains, and also to human decision makers carrying out structured decision rules.
1 INTRODUCTION
Pretrial risk algorithms can reproduce racial disparities even without explicitly using race, creating a conflict between fairness objectives and public safety. The paper frames this conflict as a constrained optimization problem and compares fairness-constrained rules with a uniform-threshold rule.
- 1 INTRODUCTION: Black defendants were more than twice as likely as whites to be labeled risky despite ultimately not reoffending.Being classified as high risk subjected these defendants to harsher court treatment.
- 1 INTRODUCTION: The paper reformulates algorithmic fairness as constrained optimization that maximizes public safety subject to formal fairness constraints.The analysis focuses on pretrial release decisions and racial disparities.
- 1 INTRODUCTION: Fairness-constrained rules require race-specific thresholds, whereas the optimal unconstrained rule applies one threshold to all defendants.The paper gives an example in which white defendants are detained above 4 and black defendants above 6.
- 1 INTRODUCTION: In Broward County data, fairness constraints could substantially decrease public safety, while optimizing public safety alone could produce stark racial disparities.The paper presents this as a practical trade-off rather than a purely theoretical one.
- 1 INTRODUCTION: The principles extend beyond pretrial algorithms to other domains and to human decision makers using structured decision rules.The authors also note that algorithmic decisions can be supplemented or replaced by social services and non-custodial supervision.
2 BACKGROUND
The paper formalizes decision rules and protected-group comparisons to define several fairness criteria for binary decisions. These criteria include equal detention rates, conditional parity, and equal false-positive rates, alongside broader debates about score calibration and incompatible fairness goals.
- 2 BACKGROUND: A decision rule maps visible attributes x ∈ R^p to the probability of taking action a1, such as detention.The framework allows randomized decisions through probabilities in [0, 1].
- 2 BACKGROUND: The framework treats group membership as a function of observable attributes and models a binary benefit Y that is unknown at decision time.In pretrial decisions, Y = 1 denotes a defendant who would commit a violent crime if released, while Y = 0 denotes otherwise.
- 2 BACKGROUND: Statistical parity requires equal detention proportions across race groups.For example, white and black defendants should be detained at equal rates.
- 2 BACKGROUND: Conditional statistical parity requires equal detention proportions across race groups after controlling for a limited set of legitimate risk factors.The paper gives prior convictions as an example of a legitimate factor.
- 2 BACKGROUND: Predictive equality requires equal false-positive rates across race groups among defendants who would not reoffend if released.A major criticism of COMPAS is that its false-positive rate is higher among black defendants than white defendants.
- 2 BACKGROUND: Calibration requires that defendants with the same risk score have the same reoffending proportion across race groups.Prior work shows that calibration and other fairness properties can be incompatible except in degenerate cases.
3 OPTIMAL DECISION RULES
The paper chooses decision rules by maximizing immediate utility, then characterizes the optimal rules under unconstrained and formal fairness requirements. Fairness constraints generally produce group- or feature-specific thresholds rather than the single uniform threshold of the unconstrained optimum, creating a cost of fairness.
- Utility objective: Immediate utility balances the expected benefit of preventing violent crime against the cost of detaining defendants.For pretrial decisions, the benefit is proportional to violent crimes prevented, while detention cost is proportional to people detained.
- Unconstrained optimum: Absent fairness constraints, utility is maximized by detaining defendants when pY |X exceeds c, yielding a single uniform threshold.The same unconstrained rule applies irrespective of group membership.
- Utility objective: Immediate utility assumes equally costly violent crimes and a common detention cost c for every individual.The paper notes that both restrictions can be relaxed without significantly affecting the formal results.
- Fairness-constrained optima: Under statistical parity and predictive equality, optimal rules detain defendants above group-specific thresholds; conditional statistical parity additionally makes thresholds depend on legitimate factors.The thresholds are deterministic and based on pY |X.
- Technical condition: The positive-density assumption ensures deterministic feasible rules exist and makes the optimal decision rules unique up to probability zero.The assumption requires the risk-score distribution, and for conditional statistical parity the conditional distribution, to have positive density on [0, 1].
- Implication: Threshold rules remain optimal under approximate fairness constraints, but constrained optima generally differ from the unconstrained optimum, so fairness has a cost.The paper identifies this difference as a tension between common fairness constraints and treating all individuals equally irrespective of race.
4 THE COST OF FAIRNESS
Using Broward County data, the paper measures the public-safety cost of fairness-constrained pretrial rules relative to an unconstrained public-safety rule. The analysis finds that fairness constraints detain low-risk defendants and increase violent recidivism among released defendants.
- Interpretation: The constrained and unconstrained optima generally differ, so satisfying common fairness definitions requires sacrificing some public safety.The paper notes that group-specific thresholds can coincide in constructed examples, but exact equality is unlikely in practice.
- Data and evaluation: The study compares fairness-constrained rules with a public-safety-optimizing rule using Broward County defendants and repeated train-test splits.The analysis considers statistical parity, conditional statistical parity, and predictive equality while holding detention at 30%.
- Data and evaluation: Conditional statistical parity results depend significantly on which factors are designated legitimate.Treating all variables as legitimate would impose no constraint; the application considers only prior-arrest count.
- Findings: 17% of defendants detained under statistical parity are relatively low risk, producing an estimated 9% increase in violent recidivism among released defendants.Because the detention total is fixed, detaining low-risk defendants releases an equal number of high-risk defendants.
- Findings: Across fairness constraints, violent recidivism increases while low-risk defendants are detained.The comparison is against a rule that optimizes public safety alone.
5 THE COST OF PUBLIC SAFETY
The paper shows that a single public-safety threshold can produce racial disparities because groups have different risk distributions, even when average risk is equal. Calibrated scores do not eliminate this trade-off, because threshold-based decisions inherit disparities from those distributions.
- Empirical disparities: A public-safety-optimizing rule detaining 30% of defendants detains 40% of black defendants versus 18% of white defendants, violating statistical parity.The disparity follows from different group risk distributions rather than altering the underlying risk scores.
- Empirical disparities: Among defendants who did not commit a violent crime, 32% of black defendants versus 14% of white defendants are detained, violating predictive equality.The cited comparison concerns defendants’ eventual violent-crime outcomes.
- Mechanism: A single threshold generally determines group-specific detention and false-positive rates through the groups’ underlying risk distributions.Fairness guarantees therefore typically deviate from the unconstrained optimum.
- Mechanism: Equal average risk does not prevent fairness conflicts: different-tailed distributions can yield different detention and false-positive rates under one threshold.The paper illustrates this with simulated populations whose means are equal but whose tails differ.
- Calibration: COMPAS scores are calibrated because defendants of different races with the same score are roughly equally likely to reoffend.Figure 2 reports recidivism for any crime and displays 95% confidence intervals.
6 DETECTING DISCRIMINATION
Calibration alone cannot establish that risk scores are informative or nondiscriminatory. Scores can remain calibrated while strategically suppressing information in ways that alter detention outcomes.
- Algorithms may output calibrated scores that are insufficiently informative for detecting discrimination.Calibration concerns agreement between predicted risks and outcomes, not whether scores preserve information needed for fair decisions.
- Given only scores and outcomes, one cannot determine whether scores accurately estimate true risk or were strategically designed to produce racial disparities.
- A facially neutral threshold can still discriminate when the decision maker withholds information that would distinguish risk within one racial group.
- Adding noise to a favored group’s true risk scores and refitting predictions can produce plausible, less informative scores that place no group members above the detention threshold.The transformed scores remain difficult to distinguish from genuinely noisy risk estimates without additional information.
- Because relevant information may be ignored unintentionally, equitable evaluation must inspect the algorithm itself rather than only the decisions it produces.
7 DISCUSSION
The paper argues that public-safety optimization and common fairness metrics can require different decision rules. It also identifies practical and conceptual limits on applying this framework to real policy choices.
- 7 DISCUSSION: Maximizing public safety requires detaining all individuals deemed sufficiently likely to commit violent crime, regardless of race.
- 7 DISCUSSION: Common fairness metrics require multiple race-specific thresholds, creating tension with minimizing expected violent crime.
- 7 DISCUSSION: Broward County analysis finds that public-safety optimization yields stark racial disparities, whereas fairness constraints release more high-risk defendants.
- 7 DISCUSSION: A single-threshold rule applies the same standard across races and avoids releasing an equally risky defendant in one group while detaining another.The paper notes that race-specific classifications may trigger strict scrutiny under the Equal Protection Clause.
- 7 DISCUSSION: Accuracy improvements, higher detention thresholds, and less punitive alternatives such as community supervision can address some fairness concerns without race-specific thresholds.
- 7 DISCUSSION: Policy evaluation is limited when arrests imperfectly measure violent crime, because differential policing can bias observed outcomes across racial groups.
- 7 DISCUSSION: Excluding features with differential predictive power may discard information and inadvertently reproduce redlining-like effects.
- 7 DISCUSSION: The framework may not apply when decisions optimize group composition rather than individual outcomes, as in university admissions.