Source-linked AI summary
Biased Programmers? Or Biased Data? A Field Experiment in Operationalizing AI Ethics
Bo Cowgill, Fabrizio Dell'Acqua, Samuel Deng, Daniel Hsu, Nakul Verma, Augustin Chaintreau
TL;DR
To investigate why biased predictions arise and which interventions may reduce them, the paper runs a field experiment with engineers developing worker-performance algorithms. It finds biased predictions mostly arise from biased training data, with data quality also shaping effort and responsiveness to incentives.
Problem
The paper asks why algorithmic predictions are biased and which interventions can reduce them, amid limited empirical evaluation of interventions and links to theory.
Source: §1 · pp. 1–2Method
A field experiment observes approximately 400 engineers developing worker-performance algorithms under randomized conditions and assesses predictions using ground-truth outcomes and randomized input manipulations.
Source: §1 · p. 2Results
Biased predictions were mostly caused by biased training data; one-third of better data’s benefit came through engineers’ greater effort and responsiveness to incentives.
Source: §4 · pp. 3–4Takeaways & Limitations
Biased training data and biased programmers call for different policy solutions, and field experiments can evaluate practical managerial and educational interventions.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Why do biased predictions arise? What interventions can prevent them? We evaluate 8.2 million algorithmic predictions of math performance from $\approx$400 AI engineers, each of whom developed an algorithm under a randomly assigned experimental condition. Our treatment arms modified programmers' incentives, training data, awareness, and/or technical knowledge of AI ethics. We then assess out-of-sample predictions from their algorithms using randomized audit manipulations of algorithm inputs and ground-truth math performance for 20K subjects. We find that biased predictions are mostly caused by biased training data. However, one-third of the benefit of better training data comes through a novel economic mechanism: Engineers exert greater effort and are more responsive to incentives when given better training data. We also assess how performance varies with programmers' demographic characteristics, and their performance on a psychological test of implicit bias (IAT) concerning gender and careers. We find no evidence that female, minority and low-IAT engineers exhibit lower bias or discrimination in their code. However, we do find that prediction errors are correlated within demographic groups, which creates performance improvements through cross-demographic averaging. Finally, we quantify the benefits and tradeoffs of practical managerial or policy interventions such as technical advice, simple reminders, and improved incentives for decreasing algorithmic bias.
1 Introduction
The introduction asks why biased predictions arise and which interventions might reduce them. It studies these questions through a field experiment in which engineers develop algorithms for predicting workers’ performance.
1 Introduction
- The experiment applies machine learning to predict workers’ performance and evaluates interventions in a setting where few studies had assessed such programs or connected them to theory.The paper aims to address that evidence gap.Source: §1 · p. 2
- The paper examines how biased predictions form and whether policy or managerial interventions can change their formation.The questions matter because biased predictions are linked to demographic segregation and outcome disparities in labor markets, criminal justice, and advertising.Source: §1 · pp. 1–2
- The field experiment observes prediction technology as approximately 400 engineers develop algorithms, then assesses their predictions using ground-truth outcomes and randomized input manipulations.Randomized treatments test how policy or managerial interventions change the formation of biased predictions.Source: §1 · p. 2
2 Empirical Setting
The study recruits programmers from a university machine-learning bootcamp and an online freelance platform. They build algorithms to predict math performance for new individuals using biographical features and OECD assessment data.
2 Empirical Setting
- About 80% of participants came from a university machine-learning bootcamp, while 60 freelance machine-learning engineers made up 20% of the subjects.The bootcamp ran the experiment in spring and fall; freelance engineers averaged 3.89 years of prior work experience.Source: §2 · p. 2
- The engineer sample was 71% male and included 28% White, 52% East Asian, 15% Indian, and about 5% Black or Latino/a/x participants.The authors characterize the sample as lacking diversity in key characteristics, though slightly more diverse than the US software-engineering population.Source: §2 · p. 2
- All subjects developed an algorithm to predict math performance from job-application biographical features and applied it to 20,000 individuals absent from the training data.Source: §2 · p. 2
- Engineers received a sample of the OECD Programme for the International Assessment of Adult Competencies dataset as training data.The passage introduces PIAAC as the dataset's abbreviation but does not complete its description.Source: §2 · p. 2
- The paper studies prediction bias, not other possible forms of algorithmic bias.Source: §2 · p. 2
3 Research Design
The research design combines objective performance measures with randomized data and policy treatments. It also audits predictions by changing candidate gender while holding other characteristics constant.
3 Research Design
- Objective performance measures cover both the approximately 400 engineers and OECD test subjects, with outcomes available for every engineer and a professionally weighted sample of the OECD population.The design also permits estimation of team contributions and ensemble performance by aggregating engineers’ predictions.Source: §3 · p. 3
- Randomized treatment arms compare biased training data with data lacking sample-selection problems, reminders about algorithmic bias, and reminders paired with technical guidance.The reminder and technical-guidance groups otherwise received the biased training data.Source: §3 · p. 3
- Within each treatment, participants were randomized to performance incentive schemes to measure whether incentives reduce algorithmic bias.Source: §3 · p. 3
- The audit compares predicted math performance for candidates identical except for gender, which the engineers’ programs could access.The outcome is a continuous predicted-performance measure; the design resembles a two-sided audit using software engineers rather than human screeners.Source: §3 · p. 3
- The engineers’ demographic variation lets the study test whether non-traditional programmers are more likely to notice or reduce bias and whether errors correlate within demographic groups.Source: §3 · p. 3
4 Results
The results identify biased training data as the main source of biased predictions, while showing that training-data quality also affects programmer effort and response to incentives. They find no evidence of lower code bias among female, minority, or low-IAT engineers, but cross-demographic averaging improves performance.
4 Results
- Biased predictions are mostly caused by biased training data, and one-third of the benefit of better data comes through greater programmer effort and responsiveness to incentives.Engineers with better data spent more hours on their algorithms and were especially affected by the randomized incentive manipulation.Source: §4 · pp. 3–4
- The study finds no evidence that female, minority, or low-IAT engineers exhibit lower bias or discrimination in their code.The IAT concerns implicit bias about gender and careers.Source: §4 · p. 4
- Prediction errors are correlated within demographic groups, particularly among male programmers, while cross-gender averaging combines equally predictive but uncorrelated information and improves performance.Averaging two male programmers’ predictions can double down on their correlated errors.Source: §4 · p. 4
- Simple reminders improve algorithm accuracy mainly by reducing variance rather than bias, while technical guidance has mixed results.Programmers who understand the guidance reduce bias, but most do not follow it, and their algorithms perform worse than those of programmers given a simple reminder.Source: §4 · p. 4
5 Conclusion
The paper measures the relative contributions of biased training data and biased programmers, which imply different policy solutions. It argues that field experiments can examine how organizational conditions shape algorithm development and evaluate practical interventions.
5 Conclusion
- The paper measures the relative contributions of biased training data and biased programmers because the two explanations imply different policy solutions.Source: §5 · p. 4
- Production algorithms are developed by people within organizations shaped by training, persuasion, culture, incentives, and implementation frictions.Source: §5 · p. 4
- Algorithms in hiring, lending, and criminal justice offer measurement opportunities unavailable in other economic settings.Source: §5 · p. 4
- The approach resembles empirical software engineering, a field less likely to study algorithmic bias and fairness.Source: §5 · p. 4
- A large, preregistered randomized controlled trial examines discrimination mechanisms more directly and transparently and relates them to realized prediction bias.Source: §5 · p. 5