Source-linked AI summary
Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning
Megha Srivastava, Hoda Heidari, Andreas Krause
TL;DR
Machine-learning fairness criteria are incompatible, leaving an open question about which notion best fits a given societal context and lay perceptions. The paper uses adaptive human-subject experiments to compare mathematical notions against participants' judgments. Demographic parity best matches those judgments in two scenarios, and participants value accuracy more when stakes are high.
Problem
Proposed fairness notions cannot all hold simultaneously, and there is no consensus on which notion is appropriate for a particular societal context.
Method
The authors use a descriptive approach with adaptive experiments in which participants compare hypothetical algorithms through at most 20 tests.
Results
Demographic parity best captures participants' perception of fairness in two scenarios, including after alternative definitions are explained and cognitive evaluation is simplified.
Takeaways & Limitations
The findings support involving people affected by algorithmic decisions when formulating fairness and selecting fairness notions for societal contexts.
Takeaways & Limitations
The experiments use a highly stylized setting, whereas real-world judgments involve factors beyond true and predicted labels.
Abstract
from arXiv · showhide
Fairness for Machine Learning has received considerable attention, recently. Various mathematical formulations of fairness have been proposed, and it has been shown that it is impossible to satisfy all of them simultaneously. The literature so far has dealt with these impossibility results by quantifying the tradeoffs between different formulations of fairness. Our work takes a different perspective on this issue. Rather than requiring all notions of fairness to (partially) hold at the same time, we ask which one of them is the most appropriate given the societal domain in which the decision-making model is to be deployed. We take a descriptive approach and set out to identify the notion of fairness that best captures \emph{lay people's perception of fairness}. We run adaptive experiments designed to pinpoint the most compatible notion of fairness with each participant's choices through a small number of tests. Perhaps surprisingly, we find that the most simplistic mathematical definition of fairness---namely, demographic parity---most closely matches people's idea of fairness in two distinct application scenarios. This conclusion remains intact even when we explicitly tell the participants about the alternative, more complicated definitions of fairness, and we reduce the cognitive burden of evaluating those notions for them. Our findings have important implications for the Fair ML literature and the discourse on formalizing algorithmic fairness.
1 Introduction
Fairness criteria for machine learning are mutually incompatible, so this work asks which notion best reflects lay people's judgments in a given societal context. Adaptive experiments find that demographic parity most closely matches participants' perceptions in two scenarios, while high-stakes decisions increase the importance participants place on accuracy.
- Motivation: Multiple proposed fairness notions, including demographic parity, disparate impact, equality of odds, and calibration, cannot all hold simultaneously.Prior work has therefore focused on quantifying tradeoffs among competing criteria.
- Approach: The paper takes a descriptive approach, selecting the fairness notion that most closely matches lay people's perception in the deployment context.It treats fairness as context-dependent rather than requiring all notions to hold together.
- Experimental design: The experiments tested hypotheses about error-rate equality in recidivism, accuracy equality in medical predictions, and accuracy sensitivity under high stakes.These hypotheses were motivated by concerns about algorithmic bias in criminal justice and medical research.
- Experimental design: Participants answered at most 20 adaptively chosen tests comparing two algorithms' predictions with true labels for ten hypothetical subjects.The EC2 active-learning algorithm selected tests to identify the fairness notion compatible with each participant's choices.
- Findings: Demographic parity most closely matched people's idea of fairness in two distinct scenarios, including when participants learned about alternative definitions and received reduced cognitive burden.Experiments were conducted on Amazon Mechanical Turk and compared participants' choices with mathematical fairness notions.
- Implications: The framework is intended to help determine an appropriate fairness notion for a societal context and to involve affected people in formulating algorithmic fairness.The authors describe this as an initial step toward understanding fairness and justice from everyday people's perspective.
2 Study Design and Methodology
The study uses adaptive human-subject experiments to identify which group-fairness notion best matches participants’ judgments across two consequential prediction scenarios. Participants evaluate equal-accuracy hypothetical algorithms through sequential tests, while EC2 adaptively selects tests to distinguish fairness hypotheses.
- Experimental procedure: Participants on AMT compare two hypothetical algorithms’ predictions for ten individuals against true labels in at most 20 tests.The displayed algorithm pairs change between tests, and equal overall accuracy controls for accuracy’s influence on fairness judgments.
- User interfaces: The interfaces elicit either unstructured reasoning or structured choices about the discriminatory demographic characteristic and metric.The structured interface uses two dropdown menus, while the text interface allows free-form explanations; benefit metrics are displayed to reduce evaluation burden.
- Scenarios and contexts: The study examines criminal-risk and skin-cancer-risk prediction as two distinct societal contexts.The scenarios were presented precisely to participants as separate prediction tasks.
- Adaptive experimental design: Around 20 adaptive tests can yield a high likelihood for the participant’s followed fairness notion, compared with at least 600 random tests in simulations.The simulation uses 1000 tests for participants following each of four fairness notions under a noisy response model.
- Adaptive experimental design: EC2 adaptively selects among noisy tests to identify each participant’s most compatible fairness notion while limiting the experiment to at most 20 tests.It represents Demographic Parity, Error Parity, False Discovery rate Parity, and False Negative rate Parity as equivalence classes and chooses tests expected to remove the most remaining edge weight.
- Adaptive experimental design: The Bayesian procedure updates fairness-notion posteriors from observed responses and chooses the next available test by maximizing an objective function.The design assumes a uniform prior and conditional independence, while response probabilities are modeled from group-level benefit vectors and generalized entropy inequality scores.
3 Experimental Findings
AMT experiments matched participants’ fairness perceptions to mathematical notions across crime and cancer-risk scenarios. Demographic parity best captured the majority’s choices, with similar trends across demographic groups and structured explanations.
- The experiments collected responses from 100 participants in the crime-risk scenario and 100 in the skin-cancer scenario.
- Demographic parity best captured the choices of the majority in the crime-risk scenario, with similar trends in the cancer-risk scenario.
- Excluding demographic parity left most participants without confident categorization under the remaining fairness hypotheses.
- The structured explanation interface matched 17 of 20 crime participants and 9 of 20 cancer participants with demographic parity.
- No significant variation appeared across gender, race, age, education, or political views in fairness perceptions.
- Participants’ free-text explanations aligned with mathematical fairness notions, while some uncategorized participants gave inconsistent explanations across tests.
4 Survey Design and Analysis
The survey compared algorithms offering accuracy–inequality tradeoffs in high- and low-stakes medical prediction scenarios. Participants gave greater weight to accuracy when predictions could affect life expectancy.
- Participants compared three hypothetical algorithms offering different tradeoffs between accuracy and equality.
- The medical survey covered high-stakes skin-cancer risk prediction and lower-stakes flu-symptom severity prediction.
- Participants gave higher weight to accuracy and lower weight to inequality when medical predictions could affect patients’ life expectancy.
5 Discussion
The experiments suggest that demographic parity best captures participants’ perception of fairness, while accuracy becomes more important when decision stakes are high. The discussion also emphasizes the study’s stylized setting, engagement safeguards, and directions for broader human-centered fairness research.
- Demographic parity best captures participants’ perception of fairness, while participants prioritize accuracy over equality when stakes are high.The paper reports this as the main takeaway from its experiments.
- Crime-risk participants gave higher weight to accuracy than inequality when predictions could affect defendants’ life trajectories.
- The task encouraged participants to consider factors algorithmic systems should account for and reflect on discriminatory algorithms in society.
- Limitations: The study assumes existing fairness notions can represent human judgment in a highly stylized setting, while real-world judgments involve many additional factors.The authors state that mathematical notions cannot capture all real-world nuances.
- Limitations: Engagement on Amazon Mechanical Turk was difficult to monitor, so the study restricted participation, limited tests, required explanations, and prevented repeat participation.
- Limitations: The findings were robust to two user interfaces, but different experimental settings could still produce different conclusions.
- Limitations: Participants lacked personal stakes, and the study compared only predictive models with similar accuracy; both issues motivate future research.
- Future directions: Future directions include providing more context, comparing algorithmic and human decisions, studying individual- versus group-level fairness, and examining expertise and personal stakes.
7 Reproducibility
The reproducibility materials include adaptive-test logs, processed Amazon Mechanical Turk data, analysis scripts, and a locally runnable interface. Identifying participant information, demographics, and feedback are withheld for privacy.
- Each participant completed 20 adaptively selected tests, with responses linked to demographic information and feedback through Amazon Mechanical Turk return codes.
- The repository contains server code, adaptive test-selection implementation, analysis code, and instructions for reproducing the paper’s plots.
- The data folder includes server interaction logs, participant likelihoods after 20 tests, and pilot-experiment results.
- Identifying information, demographics, and participant feedback are not released to protect privacy.
- The interface supports local execution and test modes for random test selection or random answer generation.
8 Additional Surveys
The paper describes additional application scenarios in which predictive algorithms support courtroom decisions. These include sentencing time in prison and setting bail amounts based on predicted future crime or court appearance risk.
- Sentencing scenarios use algorithms to predict defendants’ likelihood of future crime, informing judges’ decisions about prison time.
- Bail-setting scenarios use algorithms to predict whether defendants will appear at future hearings, informing judges’ bail decisions.
9 Experiments with Other Group Fairness Notions
Additional surveys removed demographic parity and added false positive rate and false omission rate parity as hypotheses in criminal recidivism and skin cancer prediction settings. The results continued to support demographic parity as the most common participant choice.
- The replication excluded demographic parity and included false positive rate parity and false omission rate parity in criminal recidivism and skin cancer prediction settings.Each setting used 50 participants.
- Without demographic parity, the system had greater difficulty matching users to a fairness hypothesis with high likelihood, supporting demographic parity as the most common choice.
- False negatives and false positives had the highest proportion of matches in the additional experiments.
- Figures 10 and 11 report participant matches by fairness notion and likelihood level for crime-risk and skin-cancer-risk prediction scenarios.