Source-linked AI summary
It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks
Michelle Bao, Angela Zhou, Samantha Zottola, Brian Brubach, Sarah Desmarais, Aaron Horowitz, Kristian Lum, Suresh Venkatasubramanian
TL;DR
This paper examines the use of RAI datasets, especially COMPAS, as fairness benchmarks despite the contextual complexity of criminal-justice processes. It reviews data biases, real-world discretion, and normative assumptions, concluding that COMPAS is a poor generic benchmark and that fairness findings from RAI data have limited reach to real-world outcomes.
Problem
Method
Results
The authors argue that COMPAS is a poor generic benchmark and that algorithmic fairness results on RAI data alone cannot establish real-world fairness.
Source: §3 · p. 3, §4 · p. 6, §9 · p. 11Takeaways & Limitations
Criminal-justice research using RAI datasets requires contextual understanding and engagement with ethical questions about systemic inequity.
Source: §9 · p. 11The critique focuses on pretrial RAIs; other contexts may share analogous data-bias issues but do not necessarily share pretrial FTA label bias.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Risk assessment instrument (RAI) datasets, particularly ProPublica's COMPAS dataset, are commonly used in algorithmic fairness papers due to benchmarking practices of comparing algorithms on datasets used in prior work. In many cases, this data is used as a benchmark to demonstrate good performance without accounting for the complexities of criminal justice (CJ) processes. However, we show that pretrial RAI datasets can contain numerous measurement biases and errors, and due to disparities in discretion and deployment, algorithmic fairness applied to RAI datasets is limited in making claims about real-world outcomes. These reasons make the datasets a poor fit for benchmarking under assumptions of ground truth and real-world impact. Furthermore, conventional practices of simply replicating previous data experiments may implicitly inherit or edify normative positions without explicitly interrogating value-laden assumptions. Without context of how interdisciplinary fields have engaged in CJ research and context of how RAIs operate upstream and downstream, algorithmic fairness practices are misaligned for meaningful contribution in the context of CJ, and would benefit from transparent engagement with normative considerations and values related to fairness, justice, and equality. These factors prompt questions about whether benchmarks for intrinsically socio-technical systems like the CJ system can exist in a beneficial and ethical way.
1 Introduction
COMPAS became a prominent fairness benchmark, but treating RAI datasets as ground truth overlooks the criminal-justice context and decisions shaping them. The paper examines these complexities and calls for contextual, interdisciplinary engagement with fairness, justice, and equality.
1 Introduction
- COMPAS became a widely used benchmark for developing RAIs and evaluating algorithmic fairness, partly through the practice of comparing methods on datasets used in prior work.The authors caution that treating RAI datasets as ground truth can add further data corruption.Source: §1 · p. 1
- RAI fairness depends on decisions throughout data generation, prediction, evaluation, and deployment, which are shaped by principles drawn from multiple disciplines.A narrow technical focus can miss the broader socio-technical context of racial injustice and incarceration.Source: §1 · pp. 1–2
- The paper reviews RAI dataset biases, real-world fairness claims, disciplinary standards, and practices around using these datasets.It also calls attention to hidden dimensions of justice and equality in technical discourse.Source: §1 · p. 2
2 Background and Related Work
RAIs use scored factors to estimate specified outcomes and inform decisions across the criminal-justice system. Their long-term effects remain inconclusive, while benchmarking raises concerns about provenance and context.
2 Background and Related Work
- RAIs combine scored factors associated with specified outcomes into risk scores that estimate the likelihood of those outcomes.Instruments vary in structure, implementers, queried information, transparency, and intended decision.Source: §2 · p. 2
- RAI predictions inform decisions from pretrial release to sentencing and supervision, but evidence on long-term effects remains inconclusive amid varied deployment and fairness evaluation.Proponents emphasize consistency and transparency; critics raise concerns about racial neutrality, accountability, and applicability.Source: §2 · p. 2
- Benchmarking can conflict with attention to training-data provenance and context, while socio-technical benchmark misalignment receives little discussion.The cited benchmark-evaluation criteria include relevance and solvability.Source: §2 · pp. 2–3
3 Data Biases and Errors in Pretrial RAIs
Pretrial RAI datasets can contain measurement errors and biases in outcomes, protected attributes, covariates, and distributions. Because criminal-justice data lack ground truth and are shaped by prior decisions, the authors argue that generic benchmark evaluations are statistically biased.
3 Data Biases and Errors in Pretrial RAIs
- Because measurement error is inherent and difficult to calibrate without ground truth, the authors argue that generic benchmarking on RAI data is statistically biased.The critique focuses on pretrial RAIs; analogous data-bias issues may apply elsewhere, but other contexts may not share pretrial FTA label bias.Source: §3 · p. 3
- Pretrial RAI measures of rearrest and failure to appear may not validly represent the constructs of re-offense, flight, or concerns that justify detention.Rearrest may differ from re-offense, while nonappearance includes scheduling, work, and transportation barriers; binary FTA measures also omit appearances.Source: §3 · p. 3
- Noisy race reporting and category choices can obscure whether and how an RAI functions differently across racial groups.COMPAS categories omit Native Hawaiian or Other Pacific Islander and treat Hispanic as a race rather than an ethnicity.Source: §3 · pp. 3–4
- Criminal-history covariates can inherit policing disparities, while assessor and booking discretion introduce further variation into recorded data.COMPAS’s neighborhood-crime question can also invite racialized interpretations.Source: §3 · p. 4
- Data-processing choices about historical convictions and failure-to-appear windows have statistical and normative implications that similar-accuracy models may embody differently.The authors note that longitudinal data may be needed to investigate compounded inequities.Source: §3 · p. 4
- Criminal-justice data are collected downstream of prior decisions, creating selection bias and selective labels because outcomes are observed only after selection into the dataset.Detention and bail disparities can make the observed distribution differ from the broader population that interacted with the system.Source: §3 · pp. 4–5
4 Limits of Algorithmic Fairness in RAI Outputs for Real-World Outcomes
Fairness in RAI predictions does not by itself establish fair real-world outcomes, because judges and jurisdictions exercise discretion in interpreting and applying those predictions. The authors argue that application can affect fairness as much as, or more than, the algorithm.
4 Limits of Algorithmic Fairness in RAI Outputs for Real-World Outcomes
- Judges vary in whether they follow RAI recommendations, and studies suggest racial differences in deviations, though their effects on detention disparities remain unclear.Most jurisdictions leave judges final discretion, and oversight is difficult.Source: §4 · p. 5
- Jurisdictions set thresholds mapping risk scores to release recommendations, and reported high-risk re-arrest probabilities range from 10% to 42% across RAIs.Jurisdictions may also implement RAIs contrary to their guidelines.Source: §4 · pp. 5–6
- Human judgment in applying an RAI can affect fairness as much as, or more than, the algorithm itself.A fair algorithm may not yield fair decisions if decision-makers do not use or misuse it.Source: §4 · p. 6
5 Normative Values Embedded in Use of RAI Datasets
Benchmarking RAI datasets embeds value choices about criminal-justice tasks, fairness, and reform. The authors argue that statistical fairness measures do not settle those choices or determine real-world impact.
Tasks and Outcomes
- Choosing reduced crime risk as the outcome primarily favors incapacitation over retribution or rehabilitation and may overvalue incarceration.The passage notes that incarceration itself reduces near-term opportunities to commit crime and can cause lasting harms to incarcerated people and their communities.Source: §5 · p. 6
Statistical Fairness Objectives
- Statistical fairness objectives encode distinct political and ethical values, may not be simultaneously satisfiable, and do not by themselves establish fairness as a social ideal.The authors describe social fairness as shaped through ongoing negotiation among competing needs and visions of the good.Source: §5 · p. 6
Real-World Impact
- Tying algorithmic RAI research to real-world impact assumes RAI reform is a route to criminal-justice reform, although theories of reform and their value judgments are contested.The authors state that their point is not to endorse one position, but that developing algorithmic RAIs places designers in this debate.Source: §5 · pp. 6–7
- In Cook County, the Public Safety Assessment was used to recommend pretrial detention for people with the highest risk scores, despite guidance against using it to recommend detention.The authors give this as an example of how real-world use can diverge from instrument guidance.Source: §5 · p. 6
6 Established Norms in Fields Engaged in Criminal Justice Work
The authors point AI/ML researchers to established criminal-justice research practices and practical guidance. These practices address dataset creation, instrument evaluation and use, and language about justice-involved people.
Data Collection
- Creating criminal-justice datasets can require years of stakeholder access, complex record linkage, and interviews that introduce additional resource demands and potential biases.Administrative records may be manually entered across outdated systems, while interviews can involve self-selection, non-response, rater bias, and attrition.Source: §6 · p. 7
- Researchers should consult and collaborate with criminal-justice researchers to understand dataset-creation decisions and combine contextual expertise with large-dataset management skills.The authors also caution that generalizations from publicly accessible data should account for availability bias.Source: §6 · p. 7
Standards
- Psychology standards assess RAI performance across score ranges and include tests of discrimination and calibration, while the field's calibration definition differs from machine learning's.The Risk Assessment Guidelines for Evaluating Efficacy Statement lists 50 items intended to improve study comparability and reporting transparency.Source: §6 · p. 7
Standards and Guidelines
- AI/ML researchers should learn established RAI standards and guidelines because they clarify research decisions and can provide information about underlying data.Guidelines cover instrument selection, faithful implementation, and decisions informed by results; pretrial studies commonly use one- or two-year follow-up periods because the instruments are intended to be valid for at most two years.Source: §6 · pp. 7–8
Language Guidelines
- Criminal-justice writing guidelines favor person-centered terms such as “incarcerated person” over dehumanizing labels such as “prisoner.”The authors warn that language can perpetuate marginalization even in research intended to make algorithms fairer.Source: §6 · p. 8
7 Mismatch Between AI Fairness Practices and Criminal Justice Research
The authors argue that methods-first AI fairness practices can misalign with criminal-justice research by treating data as a benchmark rather than contextual evidence. Conference workflows and replication incentives can reinforce that mismatch.
Methods-First Focus
- Methods papers often treat datasets as secondary benchmarks for comparing algorithms, rather than using them to produce new criminal-justice science.The authors distinguish these methods papers from substantive papers focused on learning new science from data, including studies of issues within specific RAI datasets.Source: §7 · p. 8
Data Decontextualization
- Optimizing performance can decontextualize RAI data and yield misleading conclusions, such as treating high-risk predictions as preferable because high risk is labeled positive.The authors also note that describing granted bail as the positive decision can overlook that many people cannot afford bail.Source: §7 · p. 8
Conference Workflow
- Rapid conference workflows can encourage researchers to use freely available datasets such as COMPAS, which gained visibility through its connection to Machine Bias.The authors give experiments added during rebuttal periods of less than a week as an example of the time pressure.Source: §7 · p. 9
Benchmark Replication
- Benchmarking incentives can make flawed experiments compound as later work replicates them, while review norms may penalize papers that omit established datasets.The authors identify this feedback loop as one reason for what they consider overuse of COMPAS.Source: §7 · p. 9
Collaboration and Research Norms
- Fast publication cycles can create an appearance of benchmark consensus that is difficult to revise, despite the slower work needed to acquire and understand data.The authors call for recognizing the difference between science and methods contributions, and between treating data as a benchmark and deriving meaning from it.Source: §7 · p. 9
Research Incentives
- Researchers seeking meaningful engagement must align their incentives to avoid the problems the authors enumerate.The authors state that the issues are not insurmountable.Source: §7 · p. 9
8 Call To Arms
The section urges researchers to investigate assumptions and metrics, avoid generic benchmarking or broad claims about criminal justice, and engage with the context of RAI use. It also proposes practical ways to improve research and review practices.
Things not to do
- Researchers should avoid using criminal-justice datasets as generic real-world examples or benchmarks, since removing them from domain context can mislead readers about criminal-justice problems.The section warns that a typical “horse-race” analysis could be misunderstood as making specific claims about criminal justice.Source: §8 · p. 9
- Researchers should avoid broad conclusions about criminal-justice practices from datasets alone because laws, practices, and data acquisition vary across locales.Such conclusions risk being misleading or wrong and becoming political scoring points in debates about reform.Source: §8 · p. 9
Things to be careful about
- The paper does not advocate avoiding criminal-justice datasets or quantitative methods, but encourages context-aware collaboration, clear assumptions, and candid discussion of deployment consequences.Researchers should avoid overselling implications, while describing methodological limitations and potential real-world consequences.Source: §8 · pp. 9–10
- Because criminal-justice data access may require months of advance approval, reviewers should encourage well-designed simulations when COMPAS benchmarking is irrelevant.The section also suggests that conference organizers and reviewers enforce requirements for ethical use, documentation, and dataset release.Source: §8 · p. 10
Things that might be helpful
- Researchers can document dataset limitations and operating conditions with datasheets, and critically examine the value positions and assumptions formalized by fairness interventions.Surfacing contested assumptions can help prevent algorithms from being used blindly in ways that could exacerbate existing biases.Source: §8 · p. 10
Things to investigate
- Researchers should test whether algorithms remain robust when underlying assumptions change slightly, including through distribution shifts or unexamined feature correlations.The section calls this assumption stability and suggests robustness tools can help investigate it.Source: §8 · p. 10
- Researchers should examine whether evaluation metrics capture the disparities they intend to measure, since equal group-conditioned AUCs need not imply equal ranking disparities.Models with the same AUC can also behave differently across demographics or risk thresholds.Source: §8 · p. 10
- The paper frames these practices as ways to expand inquiry into benchmark relevance and whether benchmarks for criminal justice can be beneficial and ethical.The authors focus their concerns on relevance to real-world representativeness and applicability to real outcomes.Source: §8 · pp. 9–10
9 Conclusion
The authors conclude that COMPAS is a poor generic benchmark for technical and normative reasons. They argue that criminal-justice research requires contextual understanding and ethical engagement with systemic inequity to support more informed and just policy decisions.
9 Conclusion
- Using COMPAS as a generic benchmark is a poor choice for technical and normative reasons.The authors hope their analysis informs more informed and just policy decisions.Source: §9 · p. 11
- Conducting criminal-justice research requires contextual understanding of the system’s past and present, including engagement with ethics and entrenched systemic inequity.Source: §9 · p. 11
- The authors aim for their discussion of RAI datasets as real-world fairness benchmarks to inform more informed and just policy decisions.Source: §9 · p. 11
A.1 Comparison of technical approaches for data quality issues
This section reviews technical approaches to data-quality problems and their informational costs and limits in criminal-justice settings. It highlights measurement error, proxy variables, discretion-driven missingness, and the importance of encoding choices.
A.1 Comparison of technical approaches for data quality issues
- Measurement-error strategies include econometric inference, loss-function modifications, and agnostic robustness or sensitivity analysis, but each incurs different informational costs.One cited method modifies the surrogate loss to estimate average misclassification cost without bias, assuming group-dependent label noise is known.Source: §A.1 · p. 20
- Agnostic robustness approaches face criminal-justice translation barriers because robustness can harm other groups’ accuracy and label-noise calibration from validation data is not immediate.The underlying distribution of crime is unknown, complicating calibration.Source: §A.1 · p. 20
- Proxy-based approaches assume structure linking observed measurements to underlying data, but imperfect race proxies can misfit people and reallocate errors.When observed attributes are modeled as noisy versions of true values, descriptive statistics on repeated records can help calibrate mismeasurement probabilities.Source: §A.1 · p. 20
- Missing-at-random methods may fail when judicial discretion depends on unobserved information, leaving administrative data insufficient to adjust for selection bias.The missing-at-random assumption treats decisions causing missing data as conditionally independent of downstream outcomes given observed data; the passage notes unobserved confounders are common.Source: §A.1 · p. 20
- RAI simplicity and interpretability can require discretizing continuous factors, while choices such as how to group prior charges encode consequential timeframes.Examples include distinguishing recent from older than two-year failures to appear, or counting “two or more” charge types.Source: §A.1 · p. 21
A.2 Issues with COMPAS Dataset
The section identifies COMPAS dataset limitations involving classification thresholds, follow-up periods, unequal observation windows, and possible range restriction. It also situates RAI assessments within criminal-justice processes that can generate endogeneity.
A.2 Issues with COMPAS Dataset
- Only 26% of people with the highest-risk Public Safety Assessment scores were re-arrested in a Kentucky validation, raising questions about classification discrepancies and arbitrary thresholds.The passage cautions that classification discrepancies may not be the best way to assess disparity.Source: §A.2 · p. 21
- The reported two-year COMPAS follow-up may include arrests after a case closed, although a later arrest should begin a new prediction period with an updated RAI.Source: §A.2 · p. 21
- The ProPublica subsamples applied a two-year cutoff to non-rearrested people but a longer timeframe to rearrested people, artificially inflating the rearrest rate.Source: §A.2 · p. 21
- Unclear treatment of people incarcerated during the pretrial period may restrict the COMPAS risk-score range and affect predictive-validity estimates.Prior work indicates range restriction may significantly affect validity estimates when moderate- and high-risk individuals are detained at high rates.Source: §A.2 · p. 21
- Static RAI factors such as prior arrests and charges can themselves reflect differential policing and reporting, creating potential endogeneity in RAI assessments.The cited work describes how geospatially differentiated policing can introduce downstream algorithmic bias even when drug-crime information is not used.Source: §A.2 · p. 21