Source-linked AI summary
50 Years of Test (Un)fairness: Lessons for Machine Learning
Ben Hutchinson, Margaret Mitchell
TL;DR
Fairness research has lacked unequivocal analyses and clear procedures, while machine-learning work has often left model use and threshold effects implicit. This paper connects historical testing criteria to machine-learning fairness notions and shows that several criteria are equivalent, while others are incompatible except in special cases.
Problem
Fairness research has lacked unequivocal analyses and clear procedures, while machine-learning literature has largely left model-use and threshold effects implicit.
Method
The paper maps historical testing criteria to machine-learning fairness notions and examines correlation-based measurement.
Results
Several historical fairness definitions are equivalent to modern machine-learning criteria, while some criteria are incompatible except in special cases.
Takeaways & Limitations
Future fairness work should address model unfairness, make encoded values explicit, and incorporate technical, cultural, and contextual influences.
Takeaways & Limitations
Some fairness criteria assume the criterion score is perfectly relevant, reliable, and unbiased, despite differing subgroup base rates producing different error rates.
Abstract
from arXiv · showhide
Quantitative definitions of what is unfair and what is fair have been introduced in multiple disciplines for well over 50 years, including in education, hiring, and machine learning. We trace how the notion of fairness has been defined within the testing communities of education and hiring over the past half century, exploring the cultural and social context in which different fairness definitions have emerged. In some cases, earlier definitions of fairness are similar or identical to definitions of fairness in current machine learning research, and foreshadow current formal work. In other cases, insights into what fairness means and how to measure it have largely gone overlooked. We compare past and current notions of fairness along several dimensions, including the fairness criteria, the focus of the criteria (e.g., a test, a model, or its use), the relationship of fairness to individuals, groups, and subgroups, and the mathematical method for measuring fairness (e.g., classification, regression). This work points the way towards future research and measurement of (un)fairness that builds from our modern understanding of fairness while incorporating insights from the past.
1 INTRODUCTION
The introduction traces modern machine-learning fairness concerns to post–Civil Rights Act scrutiny of educational and employment tests, highlighting historical parallels and unresolved measurement challenges. It frames the paper as an effort to learn from that history through structural and terminological correspondences between tests and ML models.
- Historical context: The 1964 Civil Rights Act outlawed discrimination based on race, color, religion, sex, or national origin, including by federally funded government agencies.Title VI prevented federally funded agencies, including universities, from discriminating based on race, color, or national origin.
- Historical context: The Act immediately prompted scrutiny of educational and employment tests and research into mathematically measuring unfair bias and discrimination.Researchers questioned whether ability and fit assessments discriminated on legally forbidden bases.
- Historical parallels: From 1966 to 1976, testing researchers developed subgroup-based fairness notions, recognized incompatible criteria, and challenged quantitative definitions’ limitations, paralleling ML fairness research since 2011.The introduction identifies these as striking parallels between earlier testing research and contemporary machine-learning fairness work.
- Historical parallels: By the 1970s, researchers shifted from defining test unfairness to test fairness and introduced criteria mathematically identical to modern fairness criteria, but the movement largely disappeared by decade’s end.Competing notions of fairness made it difficult to clarify when one notion should be preferred.
- Lessons for ML fairness: The late-1960s fairness research produced no generally agreed method, unambiguous item-level statistic, or broad technical solution for determining fairness.The introduction presents these unresolved outcomes as a reason to learn from the past when studying ML fairness.
- Lessons for ML fairness: Tests and ML models correspond structurally: items resemble features, responses resemble feature activations, and test scoring typically resembles a linear model.The paper also defines demographic variable A, subgroup A = a, target Y, score R, and binary decision D.
2 HISTORY OF FAIRNESS IN TESTING
Historical fairness research moved from defining bias as subgroup-specific prediction error to evaluating fair test use, while revealing incompatibilities among criteria and the importance of context and values. It also developed item-level bias analysis through Differential Item Functioning (DIF), which remains part of test design.
- Cleary’s test-bias criterion: Cleary defined test bias as consistent nonzero prediction errors for a subgroup under a common regression model.Her definition treated bias as a property of the test and formalized fairness concerns around predicting educational outcomes.
- Fairness as test use: Thorndike shifted attention from test scores to fair use, allowing different regression lines and selection thresholds across groups.He argued that fairness should rest on inferences made from tests rather than comparisons of subgroup means.
- Fairness as test use: Thorndike’s criterion required each group’s ratio of predicted positives to ground-truth positives, (TP + FP)/(TP + FN), to be equal.This criterion could judge selection fair with group-specific thresholds when a single threshold would be unfair.
- Incompatible fairness criteria: Cleary’s and Thorndike’s criteria are incompatible except when ρRY = 1 or ρAY = 0.Their definitions converge as test–target correlation approaches 1, while poor test–target correlation can leave no fair solution for nonzero demographic–target correlation.
- Item-level fairness: Testing researchers also developed item-level bias analysis: DIF identifies items whose functioning differs across groups, and flagged items may be investigated or removed.DIF was introduced in 1979 and remains in test designers’ toolboxes.
3 CONNECTIONS TO ML FAIRNESS
Historical testing fairness criteria frequently correspond directly to modern ML fairness definitions, but the comparison also exposes criteria and dimensions that ML fairness does not capture. The literatures share ethical framings and impossibility results while differing notably in their use of regression, correlation, and test-use distinctions.
- Historical criteria and ML categories: Several historical criteria map directly onto ML notions such as sufficiency, equalized odds, conditional procedure accuracy, and conditional independence.Peterson and Novick’s “conditional probability and its converse” is equivalent to these modern definitions.
- Shared theoretical concerns: Both literatures address fairness impossibilities, including the distinction between group and individual fairness and the inability to obtain separation, sufficiency, and independence simultaneously except under special conditions.These results connect fairness criteria to constraints that cannot generally be satisfied together.
- Ethical framings: The two fields also share ethical framings, including utilitarian approaches and analyses linking fairness criteria to costs, value systems, and worldviews.Utility-based testing approaches relate to modern cost-of-fairness formulations, while analyses of underlying value systems resemble work connecting criteria to different worldviews.
- Historical criteria and ML categories: Some testing criteria do not map onto ML independence conditions, while others resemble constrained ranking, infra-marginality, or subgroup-quota formulations.Non-comparative criteria, Thorndike’s criterion, and Darlington’s criterion lack known counterparts, whereas Jones’ and Einhorn’s criteria have conceptual similarities to modern concerns.
- Methods of measuring fairness: Historical testing work used regression models and correlation criteria far more than modern ML fairness, with correlation offering easier computation but weaker guarantees than independence.Correlation requires fewer assumptions and is easier to calculate, whereas estimating independence can require nontrivial inferential statistics.
- Fairness as property or use: Testing’s debate over whether fairness belongs to a test or its use parallels an largely implicit or missing distinction in ML fairness concerning models and decision thresholds.Calibration can be interpreted as a model property because it does not depend on the decision threshold.
4 FAIRNESS GAPS
The historical comparison identifies gaps in machine-learning fairness research, including overlooked bias-oriented analysis and asymmetries between model scores and targets. It also motivates studying compromises between separation and sufficiency without endorsing a particular compromise model.
- Conceptual gaps: Earlier test-fairness work emphasized unfair discrimination and bias, whereas later work reframed these concerns in terms of fairness criteria.The shift also raised the problem of identifying sources of bias.
- Conceptual gaps: Machine-learning fairness lacks a direct analog to Differential Item Functioning, which motivated investigations of bias causes in educational testing.A possible analog would compare P(X_i | R = r,A = a) across input features, model outputs, and subgroups.
- Asymmetries: Swapping model score R and target Y can transform separation into sufficiency, but the comparison also reveals overlooked converse criteria.The converse Cleary criterion predicts R from Y and treats positive errors in the converse regression line as evidence of subgroup unfairness.
- Asymmetries: Converse calibration requires E(R − y|Y = y,A = a) = 0, meaning each subgroup’s expected prediction error is zero at every ground-truth level y.This criterion reverses the conditioning direction of ordinary calibration in regression settings.
- Compromises: Because separation and sufficiency can represent competing interests, machine-learning fairness should consider parameterized compromises rather than advocate a single compromise model.The paper suggests that fair-division economics may offer useful ideas for encoding such compromises.
5 DISCUSSION
The review proposes four directions for future ML fairness research: explain unfairness by examining its causes, expand criteria to model context and use, quantify trade-offs with other goals, and deepen subgroup and intersectionality analysis.
- Future work should explain and reduce model unfairness by examining the causes and full causal pathways producing disparities.This shifts attention from asking what constitutes a fair model to asking what causes disparities, including processes interacting before decision time.
- Fairness criteria should incorporate model context and use, drawing on earlier insights from education and hiring researchers.
- Fairness research should quantify trade-offs between fairness goals and other goals, while clearly articulating assumptions and choices.Relevant considerations include value systems and ethics.
- Researchers should examine how subgroups are defined, including whether they should be discrete categories and how intersectionality can be modeled.One example is quantifying fairness along one dimension conditioned on another, such as age conditioned on skin tone.
6 CONCLUSIONS
The history of test fairness suggests that future ML fairness research should attend closely to legal and public concerns, while making technical values explicit. It should also broaden debate to include cultural causes, model context and use, and systemic unfairness.
- Historical lessons: Test-fairness interest rose amid 1960s social and political upheaval, while contemporary ML-fairness interest reflects public concern about criminal sentencing and predictive policing.The paper connects quantitative fairness definitions in education and employment partly to U.S. federal anti-discrimination legislation.
- Legal and public concerns: Courts may begin ruling on ML-model fairness, and technical definitions that diverge from public perceptions could weaken political support for scientific contributions.The conclusion urges careful attention to legal and public concerns about fairness.
- Use and responsibility: Public concerns center on inappropriate test-score interpretations and test-result misuse, areas where the measurement profession has struggled to define and exercise responsibility.The profession has generally not acted strongly or collectively against misuse.
- Broader fairness debate: Future fairness debates should include technical and cultural causes, model context and use, and whether research variables capture systemic unfairness.The authors endorse making values encoded by technical definitions explicit and relating fairness debates to ethical theories and value systems.
APPENDIX A: ADDITIONAL DEFINITIONS OF TEST FAIRNESS
This appendix provides additional details on fairness definitions listed in Table 2 but not introduced in Section 2.
- The appendix elaborates on fairness definitions included in Table 2 that were not discussed in the main text.
Einhorn and Bass
In 1971, Einhorn and Bass argued that satisfying Cleary’s criterion could still yield different false-positive and false-negative rates across subgroups. They attributed this to differences in standard errors of estimate and proposed equal false discovery rates as a remedy.
- Einhorn and Bass: In 1971, Einhorn and Bass observed that Cleary’s criterion could hold while subgroups experienced different false-positive and false-negative rates.The disparity could arise from differences in subgroups’ standard errors of estimate.
- Einhorn and Bass: Differences in variability around the common regression line were identified as the cause of unequal false-positive and false-negative rates.
- Einhorn and Bass: To address this problem, Einhorn and Bass proposed a criterion based on achieving equal false discovery rates.
Darlington’s “culturally optimum”
Darlington defines a “culturally optimum” test by allowing the acceptable tradeoff between validity and diversity to vary by scenario and maximizing ρX(Y − kC).
- Darlington argues that the subjective value assigned to test validity and diversity can be scenario-specific.
- His elicitation technique produces k, which measures the validity tradeoff acceptable for increasing diversity.
- The “culturally optimum” test maximizes ρX(Y − kC).
Jones
Jones’ 1973 “general standard” defines fairness by requiring subgroup representation to align between rankings by test score and ground truth at every cutoff position.
- Jones: Jones’ criterion declares a test fair at position n when equal subgroup proportions appear in the top n% of score-ranked and ground-truth-ranked candidates.The “general standard” requires this condition to hold for all values of n.