PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrimination in the Age of Algorithms

Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, Cass R. Sunstein

arXiv:1902.03731v1cs.CYcs.AIcs.LG

TL;DR

The paper asks how discrimination can be detected when human decision-making is opaque and algorithms enter consequential screening processes. It analyzes algorithmic design and decision processes, arguing that transparency, recordkeeping, and scrutiny of human choices can make disparities easier to attribute. With appropriate safeguards, algorithms may become a positive force for equity, although their benefits depend on inspectable data, objectives, and design choices.

  • Problem

    Human motives and decision processes are often opaque, making it difficult to determine whether observed disparities reflect discrimination.

  • Method

    The paper analyzes screening algorithms by separating trainer choices from screener behavior and decomposing disparities across objectives, inputs, training procedures, and structural disadvantages.

  • Results

    The paper concludes that algorithms can provide greater clarity about decision ingredients and motivations, making discrimination easier to investigate when their components and records are available.

  • Takeaways & Limitations

    Safeguards should scrutinize and preserve the human choices, data, objectives, and tradeoffs involved in building algorithms so they can support equity rather than reproduce bias.

Abstract

from arXiv · show

The law forbids discrimination. But the ambiguity of human decision-making often makes it extraordinarily hard for the legal system to know whether anyone has actually discriminated. To understand how algorithms affect discrimination, we must therefore also understand how they affect the problem of detecting discrimination. By one measure, algorithms are fundamentally opaque, not just cognitively but even mathematically. Yet for the task of proving discrimination, processes involving algorithms can provide crucial forms of transparency that are otherwise unavailable. These benefits do not happen automatically. But with appropriate requirements in place, the use of algorithms will make it possible to more easily examine and interrogate the entire decision process, thereby making it far easier to know whether discrimination has occurred. By forcing a new level of specificity, the use of algorithms also highlights, and makes transparent, central tradeoffs among competing values. Algorithms are not only a threat to be regulated; with the right safeguards in place, they have the potential to be a positive force for equity.

I. INTRODUCTION

Human decisions are often opaque even to decision-makers, making discrimination difficult to prove. The paper argues that regulated algorithms can make decision criteria, data, objectives, and disparities more examinable, provided safeguards address human design choices.

  • Discrimination law faces serious proof problems because people may dissemble, lack awareness of their own biases, or be unable to explain their decisions.Human decision-making can be opaque to both outsiders and insiders.
  • Regulating algorithm design and requiring detailed records can make it easier to attribute observed disparities to algorithm components or structural disadvantages.The proposed approach focuses on transparency about design choices and tradeoffs among relevant values.
  • The paper’s framework requires safeguards such as storing algorithm components and training data for examination and experimentation.Without these requirements, unregulated algorithm design could make discrimination harder to detect.
  • Algorithms can make screening decisions more transparent by exposing features, training data, objective functions, and counterfactual changes in outcomes.These elements allow investigators to ask what factors were used and how a candidate’s result would change if features changed.
  • Algorithms are not inherently objective because biased data, objectives, predictors, training samples, and human builders can reproduce discrimination.The paper therefore treats regulation of the people and choices behind algorithms as essential.

A. Implications of Our Framework

The framework decomposes disparities across screening-rule objectives, inputs, training choices, and structural differences. This specificity enables scrutiny and makes tradeoffs among competing values more explicit, while appropriately regulated algorithms may reduce some human discrimination.

  • Attributing disparities: Disparities may reflect screening rules, objectives, predictors, training procedures, or structural differences, so discrimination analysis is fundamentally an attribution problem.Decomposing these sources determines when legal remedies should apply and which remedies are appropriate.
  • Attributing disparities: Algorithms make decomposition easier because their objective functions reveal what outcome the trainer was instructed to predict.This allows scrutiny of whether the selected outcome was reasonable rather than requiring inference about an individual hiring decision.
  • Regulatory implications: Transparency and stored records are necessary to inspect the human choices involved in constructing and training algorithms.The relevant choices include outcomes, candidate predictors, data, and training procedures.
  • Regulatory implications: Algorithms force more explicit judgments when goals conflict, allowing tradeoffs among values to be articulated and examined quantitatively.The paper presents algorithmic design as a way to make contested value choices more visible.
  • Algorithms and equity: With appropriate regulation and suitable objectives and data, algorithms may reduce discrimination by removing bias from unstructured human decision-making.The paper distinguishes this possibility from the risk that builders introduce bias through objectives or data.
  • Algorithms and equity: 42% fewer people could be jailed with no increase in crime by using algorithms to prioritize the highest-risk defendants for detention.The cited New York example says the largest benefits would accrue to African-Americans and Hispanics, who account for nine of every ten jail inmates.

III. ANTI-DISCRIMINATION LAW FOR HUMANS

Human discrimination law must distinguish discriminatory treatment from disparities arising through other causes, but human motives are difficult to uncover. Behavioral research further indicates that people may not know why they choose, complicating proof of discriminatory intent.

  • Human screening decisions: Screening decisions use personal features to choose among people for consequential outcomes such as hiring, promotion, jail, or credit.Decision-makers implicitly or explicitly apply a screening rule linking features to outcomes they care about.
  • Human cognition and biases: Disparate-treatment analysis is difficult because people may dissemble, obfuscate, lie, or lack accurate awareness of their own decision processes.Behavioral science identifies both deliberate thought and automatic responses that may occur without awareness.
  • Human cognition and biases: Experiments found that people denied being influenced by others even when their behavior changed with the presence of other people.This illustrates why self-reports may not reliably reveal the causes of decisions.
  • Human cognition and biases: Statistical evidence of group disparities cannot, by itself, establish disparate treatment in any individual decision.The distinction limits what aggregate evidence can prove about a particular actor or decision.

A. Two Algorithms, Not Just One

The paper distinguishes two algorithms in a screening application: the screener predicts an individual outcome, while the trainer constructs the screening algorithm. This distinction determines how discrimination should be audited and limits the framework to inspectable, sufficiently stable systems.

  • Two algorithms: A screening application contains a screener that predicts an individual’s outcome and feeds that prediction into a decision.Examples include hiring, lending, and jailing decisions.
  • Two algorithms: A trainer constructs the screener by choosing past cases, outcomes to predict, and candidate predictors.These construction choices define the data and target used to produce the screening rule.
  • Why the distinction matters: Distinguishing screener from trainer clarifies both concerns about algorithmic behavior and the different audits required for each.The paper states that monitoring and regulation must account for the two algorithms separately.
  • Scope: The framework assumes repeated decisions provide enough cases for training, making it more suited to micro-prediction tasks such as hiring than macro-prediction tasks such as elections.Its intended applications involve decisions made many times.
  • Scope: The framework applies when the trainer, screener, and training dataset are fixed stored objects that can be inspected.Rapid online-learning settings may not preserve enough information to reconstruct what the system would have done under different choices.

C. How algorithms work

Algorithmic screening is built through human choices about data, outcomes, candidate predictors, and training procedures, after which statistical relationships determine the screener. These choices shape both predictive behavior and potential discrimination, while prediction does not establish causation.

  • C. How algorithms work: Building a screening algorithm requires collecting data, specifying an outcome, selecting candidate predictors, training a prediction procedure, and validating it on held-out data.The dataset records past workers, outcomes, and application characteristics; the screener maps applicant characteristics to predicted outcomes.
  • C. How algorithms work: Human designers decide which workers, outcomes, and candidate predictors enter the training process, so these choices shape what the algorithm can learn.The outcome must be made concrete, and designers decide which variables are available for inclusion.
  • C. How algorithms work: Given the outcome, candidate predictors, and training data, underlying statistical relationships determine which predictors enter the model and how much weight they receive.Machine learning selects predictors from the options made available by human designers, rather than independently creating unavailable information.
  • C. How algorithms work: The training algorithm is designed for prediction rather than causal inference, so included predictors need not be the variables causally affecting the outcome.A protected attribute can be omitted while a correlated proxy remains relevant to prediction.
  • C. How algorithms work: Algorithms can inherit discrimination from biased outcomes, predictors, or training data, including past human judgments embedded in the target being predicted.Human choices in these components can infect the resulting algorithm, and the paper decomposes overall discrimination into contributions from them.

3. Choice of training procedure

Discrimination can enter through the training procedure and especially through the construction of the training sample. Because outcomes are observed only for selected people, biased or sparse samples can reproduce past disparities and reduce prediction accuracy for underrepresented groups.

  • 3. Choice of training procedure: Training procedures can introduce discrimination through deliberate under-optimization or through the selection and construction of training data.A poorly optimized screener may help some groups and hurt others while only weakly approximating the intended outcome.
  • 3. Choice of training procedure: Using data from a setting hostile to women can depress observed performance and create systematic disparities in predictions for future female applicants.The example concerns college-admission predictions trained on records from a hostile campus environment.
  • 3. Choice of training procedure: Training can use only observations with observed outcomes, so groups excluded from applying or hiring may have too little data for equally accurate predictions.The resulting sample imbalance can make predictions for the majority group more accurate than predictions for other groups.
  • 3. Choice of training procedure: Past application and hiring decisions, including information unavailable in the dataset, filter the training sample and create the selective labels problem.The paper notes that this problem has received relatively little systematic investigation and may require combining machine learning with natural-experiment designs.
  • 3. Choice of training procedure: Discrimination from biased training samples reflects skewed data rather than necessarily a defect in what the algorithm does.Humans learning from the same samples would likely form similarly mistaken impressions about relationships between outcomes and predictors.
  • 3. Choice of training procedure: Algorithmic discrimination can be fully decomposed into contributions from the choices of outcome, candidate predictors, and training procedure.This decomposition distinguishes those sources from other dimensions of algorithmic behavior that do not originate discrimination.

1. Discriminatory selection of what candidate predictors to include in prediction function

Once the outcome, available inputs, and training procedure are fixed, selecting the predictive subset is largely a data-driven step rather than a closely programmed human decision. Deliberate intervention remains possible, but access to the inputs and data enables external detection.

  • 1. Discriminatory selection of what candidate predictors to include in prediction function: With the outcome, input variables, and training procedure fixed, machine learning is less likely to introduce discrimination by selecting the wrong variables from the available options.Choosing the optimal subset is data-driven rather than closely programmed by a human.
  • 1. Discriminatory selection of what candidate predictors to include in prediction function: Designers can nevertheless manually introduce discrimination by deviating from the standard training pipeline or embedding discriminatory behavior in code.The paper gives the example of reducing predicted productivity for every female applicant.
  • 1. Discriminatory selection of what candidate predictors to include in prediction function: If the outcome, inputs, and training data are available, third parties can rerun training and test whether they substantially improve the firm’s predictions.A significant improvement would provide a natural way to detect deliberate discrimination in the optimization stage.

2. The screener algorithm

Once the trainer’s human choices are accounted for, the screener is largely the mechanical result of applying the training procedure. Its behavior can therefore be audited, simulated, and tested for deviations from the specified rule.

  • 2. The screener algorithm: The prediction system contains a trainer that constructs the model and a screener that applies the resulting rule to applicants.The discussion of discrimination in this section focuses on the screener after the trainer’s choices have been accounted for.
  • 2. The screener algorithm: After the outcome, candidate predictors, and training procedure are fixed, the screener is essentially determined by those design choices.The paper contrasts this mechanical determination with human ranking decisions in hiring.
  • 2. The screener algorithm: Disparate treatment by the screener should appear as a detectable deviation from the algorithm’s expressed specification when its design and operation can be audited.This requires access to the training algorithm’s design and the ability to interact with the screening algorithm.
  • 2. The screener algorithm: Access to the screener permits simulation of decisions for individual cases, populations, and counterfactual changes to applicant characteristics.These simulations can compare acceptance rates across population subsets and ask whether a candidate would have been hired after changing a characteristic.
  • 2. The screener algorithm: For screening, algorithms can be more transparent than human decision-makers despite being difficult to understand by reading their code alone.The paper distinguishes mathematical opacity from the practical auditability and experimentation enabled by operating the system.

3. Group differences in the raw data and predicted outcomes

Group differences in predictors or outcomes do not by themselves establish legally relevant discrimination. The underlying reality reflected in data may itself differ across groups, and algorithmic adjustment is a separate policy question.

  • Differences in candidate predictors or predicted outcomes are not, by themselves, proof that data use is discriminatory.The relevant inquiry concerns whether the data systematically mismeasure reality or whether protected characteristics played a legally relevant role.
  • Observed group disparities may reflect underlying social differences rather than biased measurement of reality.For example, differences in earnings may reflect unequal child-care burdens or discrimination embedded in the underlying conditions.
  • Adjusting algorithmic predictions or subsequent decisions to address social problems is distinct from eliminating data bias.The paper separates predictive adjustment for policy goals from determining whether the data systematically mismeasure reality across groups.

C. What questions we need to ask

The paper asks how algorithmic hiring decisions should be examined under discrimination law. It focuses on the outcomes, predictors, training choices, and benchmarks that make the construction process more inspectable.

  • The analysis examines how discrimination law can detect discrimination when algorithms enter hiring and other screening decisions.The paper contrasts the desired regulatory framework with the existing framework in a concrete hiring context.
  • Algorithmic screening requires specifying the outcome to predict, making a previously implicit human choice more tractable to examine.A standard prediction algorithm cannot be built without at least implicitly specifying an outcome.
  • The algorithmic paper trail can expose claims about a chosen outcome that conflict with the firm’s documented performance reviews and promotion practices.This creates evidence without requiring a person to confess discriminatory intent.
  • Investigators can compare the firm’s chosen outcome, predictors, and training choices with relevant internal or external benchmarks.These comparisons include alternative outcomes supported by the firm’s data and predictor choices made by comparable firms.
  • A firm’s outlier choices may provide suggestive evidence of disparate treatment or disparate impact, but comparisons alone do not prove discrimination.The firm may then bear a burden to justify choices that generated hiring disparities.

D. Problems of Proof: Human Beings vs. Algorithms

Human screening decisions can make discrimination difficult to prove because motivations and relevant preferences may remain undocumented. Algorithms preserve the same legal standards while making factors, disparities, and tradeoffs more inspectable and quantifiable.

  • Human-screening cases may require difficult statistical or documentary demonstrations to establish that race or gender affected decisions.The proof problem is especially formidable when relevant preferences are not documented.
  • In algorithmic screening, whether training used an illicit factor such as race or sex can be seen directly in the training procedure.The legal analysis remains the same in principle, but the decision process becomes easier to inspect.
  • For disparate-impact cases, algorithmic data can show the existence and magnitude of disparities and help identify the practice producing them.The remaining question is whether the impact can be justified under the applicable legal standard.
  • Algorithms can generate rankings across different tolerances for disparate impact and quantify the effects of each ranking.This makes business-necessity arguments easier to litigate and helps distinguish specious from genuine justifications.
  • Algorithms clarify tradeoffs between racial diversity and predicted academic performance without deciding whether the tradeoff is worthwhile.The paper treats value judgments about affirmative action and nondiscrimination as unresolved by the algorithm itself.

F. Combating Discrimination

Regulating algorithmic discrimination requires transparency about human choices and stored algorithmic components. With safeguards, algorithms may reduce some forms of discrimination and clarify equity-related tradeoffs, but storage, updating, procurement, and compliance costs constrain implementation.

  • A regulatory framework must identify and interrogate choices about the predicted outcome, available inputs, and training procedure.These are the principal human decisions involved in constructing the training algorithm.
  • Transparency is valuable as a means for examining discriminatory choices, not as an end in itself.Algorithmic regulation depends on making the relevant construction decisions available for legal inquiry.
  • Algorithms make transparency more feasible because outcomes, predictors, and training samples are tangible objects that can be stored.These objects can be examined more productively than purely human reasoning that may be difficult even for decision-makers to articulate.
  • Online-learning applications may not store all objects needed for discrimination inquiries, especially when data flow at massive volume.Expanding storage requirements could extend the framework’s reach but impose substantial compliance costs.
  • Outdated systems risk producing “zombie predictions” when algorithms continue using old data under changed conditions.Regular updating can be difficult when outside firms retain leverage after organizations incur the initial installation costs.
  • Algorithms may reduce discrimination relative to human decisions, limit disparate impacts, and produce disproportionate benefits for disadvantaged groups.The paper labels this potential outcome the “disparate benefit” of algorithms.

B. Access to the protected variable promotes equity for the algorithm

Access to protected variables can improve algorithmic predictions when predictor–outcome relationships differ across groups, potentially mitigating bias and increasing equity. The paper also shows that race-aware admissions can improve minority representation at a fixed academic-performance threshold, while legal restrictions on protected attributes remain unsettled.

  • Gender-aware training can avoid using manager ratings for women when those ratings are discriminatory, mitigating biased productivity predictions.Without gender information, the algorithm treats ratings as equally meaningful across workers and understates women’s future productivity.
  • Restricting access to race and related background information can force an algorithm to treat similarly scored applicants as having comparable achievement despite unequal starting conditions.The paper illustrates this with two students who both score 1,100 on the SAT but face sharply different socioeconomic and educational circumstances.
  • Race-aware admissions can increase the share of admitted black students while holding the share with GPA<2.75 constant at 14%.The reported shares are 7% for the race-blind model, 11% for the preprocessed model, and 19% for the race-aware algorithm.
  • In college admissions, race-aware predictions can correct mis-ranking that occurs when race-blind models overlook group-specific relationships between predictors and outcomes.The race-aware algorithm uses applicant race alongside other information to distinguish students whose predicted and actual outcomes diverge under race-blind prediction.
  • Protected-attribute access does not invariably mitigate discrimination, and courts may restrict its use even when it can improve prediction or equity.The paper does not offer a final rule for when algorithms may explicitly consider race.
  • Algorithmic transparency can expose discriminatory patterns in the human decisions used to generate training outcomes.A firm that observes its hiring algorithm selecting mostly men may investigate whether managers’ past hiring decisions reflected bias.

VII. Conclusion

The paper argues that algorithms can make discrimination easier to detect because their inputs, objectives, and decision processes are more specific and auditable than human reasoning. With safeguards governing data, objectives, and auditability, algorithms may help detect, reduce, or eliminate discrimination while making value tradeoffs explicit.

  • Algorithms can provide greater clarity about decision ingredients and motivations than human decision-making, creating more opportunity to detect discrimination.The paper contrasts algorithmic specificity with the ambiguity surrounding human screening decisions.
  • Audit studies document substantial disparities in human decisions, including white applicants receiving callbacks at 34% versus 14% for black applicants.The cited example concerns otherwise-equivalent applicants without criminal records.
  • Women and minorities were 40% less likely than white men to receive recommendations for cardiac catheterization in the cited physician-decision study.
  • Data-driven prediction models may help detect, reduce, or eliminate discrimination as more information becomes available about past decisions and their consequences.The paper identifies hiring, credit, admissions, and criminal-justice decisions as settings where such models may be used.
  • Realizing algorithmic benefits requires transparency and auditability, nondiscriminatory data choices, and reasonable algorithmic objectives.These safeguards also help make the tradeoffs embedded in algorithmic decisions understandable and selectable.

I. APPENDIX: A CHARACTERIZATION OF ALGORITHMIC BIAS

The appendix decomposes algorithmic disparity into structural disadvantage and three design-related sources of bias: outcome choice, input-variable choice, and training-procedure design. It formalizes these components and situates them within prediction and admissions examples.

  • Algorithmic bias is decomposed into bias from input variables, outcome measures, and training procedures, with residual disparity corresponding to structural disadvantage.
  • The framework represents applicants with a full feature vector, group-specific distributions, and a true productivity function, then defines D(v) as the groups’ average difference in v.
  • Because designers lack the true outcome function and full feature vector, they specify an outcome, use a reduced representation, and construct a training procedure that produces a screening rule.
  • The displayed decomposition expresses disparity in assigned scores as structural disadvantage plus outcome-measure, input-variable, and training-procedure effects.Each term is intended to be directly interpretable within the framework.
  • The framework distinguishes the bias introduced by choosing g instead of f, using r(x) instead of x, and estimating t instead of h.
  • In the NELS admissions analysis, race-aware, race-blind, and preprocessed algorithms rank applicants by predicted college performance and produce different fairness–efficiency tradeoffs.The appendix uses these models to simulate admissions decisions and compare predicted college performance outcomes.
Loading 1902.03731v1…