PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on measuring indirect discrimination in machine learning

Indre Zliobaite

arXiv:1511.00148v1cs.CYstat.AP

TL;DR

Predictive models can reproduce discrimination from biased, incomplete, or historically discriminatory data, while the field lacks consistent ways to measure it. This survey organizes discrimination measures, discusses related candidates, and computationally analyzes selected measures. Its analyses favor normalized difference over mutual information and ratio measures, while identifying important scope and measure-specific limitations.

  • Problem

    Predictive models may discriminate because historical data can be biased, incomplete, or encode past discriminatory decisions, while researchers lack consistent measurement approaches.

  • Method

    The survey reviews and categorizes discrimination measures, discusses related measures from other fields, and computationally analyzes selected measures using controlled synthetic data.

  • Results

    The analysis advocates normalized difference over mutual information and ratio measures, while recommending mean difference when groups and outputs are close to balanced.

  • Takeaways & Limitations

    The survey provides a structured basis for comparing discrimination measures and selecting measures for discrimination discovery, prevention, and practical evaluation.

  • Takeaways & Limitations

    Most existing research has focused on binary classification with a binary protected characteristic, leaving more complex settings as future challenges.

Abstract

from arXiv · show

Nowadays, many decisions are made using predictive models built on historical data.Predictive models may systematically discriminate groups of people even if the computing process is fair and well-intentioned. Discrimination-aware data mining studies how to make predictive models free from discrimination, when historical data, on which they are built, may be biased, incomplete, or even contain past discriminatory decisions. Discrimination refers to disadvantageous treatment of a person based on belonging to a category rather than on individual merit. In this survey we review and organize various discrimination measures that have been used for measuring discrimination in data, as well as in evaluating performance of discrimination-aware predictive models. We also discuss related measures from other disciplines, which have not been used for measuring discrimination, but potentially could be suitable for this purpose. We computationally analyze properties of selected measures. We also review and discuss measuring procedures, and present recommendations for practitioners. The primary target audience is data mining, machine learning, pattern recognition, statistical modeling researchers developing new methods for non-discriminatory predictive modeling. In addition, practitioners and policy makers would use the survey for diagnosing potential discrimination by predictive models.

1. INTRODUCTION

The survey addresses inconsistent definitions and measurements of discrimination in predictive models trained on potentially biased historical data. It organizes existing and potentially applicable measures to support systematic comparison and practical guidance.

  • 1. INTRODUCTION: Historical data can cause predictive models to discriminate even when the modeling process is fair and well-intentioned.Machine-learning methods often assume historical data is correct and representative, although it may not be.
  • 1. INTRODUCTION: Researchers lack consensus on how to define fairness and measure discrimination, making results difficult to compare and recommendations difficult to formulate.Different papers often introduce both a new discrimination measure and an algorithm optimized for it.
  • 1. INTRODUCTION: The survey presents a unifying view of discrimination measures and examines implications of optimizing one measure rather than another.It reviews and categorizes measures from machine learning and data mining and discusses measures from other fields that could be applicable.
  • 1. INTRODUCTION: Related surveys provide broader multidisciplinary or legal context but do not analyze and compare discrimination measures in the same way.The cited review by Romei and Ruggieri gives only a brief overview of measures while focusing on solution approaches across disciplines.
  • 1. INTRODUCTION: The article covers legal context, categorized measures, computational analysis, and recommendations for researchers and practitioners.The stated organization includes Sections 2, 4, 5, and 6 for these topics.

2. BACKGROUND

Discrimination in machine learning concerns disadvantageous treatment associated with group membership, including indirect effects from apparently neutral rules. The field develops methods to discover and prevent such patterns, while measures remain central to both activities.

  • 2. BACKGROUND: Discrimination is characterized by the action, situation, and protected ground involved.Examples of grounds include race, disability, age, religion, sexual orientation, gender, and nationality; areas include employment, education, social protection, and goods or services.
  • 2. BACKGROUND: Indirect discrimination arises when an apparently neutral criterion places a protected group at a particular disadvantage.Algorithmic rules can apply such treatment consistently and at larger scale than case-by-case human decisions.
  • 2. BACKGROUND: Historical data may be biased, incomplete, or encode past discriminatory decisions that predictive models can transfer and reinforce.Models are only as objective as their data and the assumptions connecting the model to reality.
  • 2. BACKGROUND: Discrimination-aware machine learning develops models that satisfy non-discrimination constraints despite biased, incomplete, or discriminatory training data.Prevention approaches include data preprocessing, model postprocessing, and model regularization.
  • 2. BACKGROUND: Coherent discrimination measures are needed both to identify discriminatory patterns and to optimize preventive predictive models.The survey therefore reviews measures, analyzes their properties, and studies implications of choosing among them.

3. MACHINE LEARNING SETTINGS, DEFINITIONS AND SCENARIOS

The paper defines fairness in terms of prediction similarity for comparable people and justified differences across groups, then situates these ideas within machine-learning task and model-building settings. It emphasizes that protected attributes may be needed during learning even when excluded from deployment-time decisions.

  • 3.1. Definition of fairness for machine learning: Fairness requires similar predictions for people alike on non-protected characteristics and differences across groups no larger than those justified by such characteristics.The second condition addresses indirect discrimination, including redlining through correlated neighborhood characteristics.
  • 3.1. Definition of fairness for machine learning: The twin test operationalizes direct fairness by requiring identical predictions for individuals who differ only in a protected attribute.The paper states this condition is necessary but not sufficient for nondiscriminatory decision making.
  • 3.2. Machine learning task settings: Machine-learning settings may involve binary, ordinal, or numeric target variables, but discrimination requires outcomes with an ordering of desirability.Examples include loan approval, service tiers, and interest rates.
  • 3.2. Machine learning task settings: Protected characteristics may be binary, categorical, numeric, or combinations of personal attributes, while models use personal characteristics to produce predictions.Examples include gender, ethnicity, and age as protected variables.
  • 3.3. Principles for making machine learning non-discriminatory: Protected characteristics can be used during model learning to enforce constraints, provided the resulting decision model does not require them at decision time.The paper distinguishes algorithms used to construct models from the models used for decisions.
  • 3.3. Principles for making machine learning non-discriminatory: Fairness can be measured on historical data, model predictions, or models themselves, with the appropriate technique depending on the task and application goal.Measures commonly serve as optimization constraints in nondiscriminatory model learning.

4. DISCRIMINATION MEASURES

The survey organizes discrimination measures into statistical tests, absolute measures, conditional measures, and structural measures. These categories differ in whether they detect presence, quantify magnitude, or account for legitimate explanations and individual characteristics.

  • 4. DISCRIMINATION MEASURES: Discrimination measures are categorized as statistical tests, absolute measures, conditional measures, and structural measures.The ordering reflects the historical development of these approaches.
  • 4. DISCRIMINATION MEASURES: Statistical tests identify whether discrimination is present but do not quantify its magnitude or distribution within a dataset.Absolute measures quantify dataset-level discrimination using the protected characteristic and decision, without considering other individual characteristics.
  • 4. DISCRIMINATION MEASURES: Absolute measures provide core principles for conditional measures and statistical tests, although they may ignore legitimate explanations for treatment differences.Conditional measures incorporate additional characteristics to capture discrimination that cannot be explained by non-protected factors.
  • 4. DISCRIMINATION MEASURES: The survey uses notation in which s1 denotes the protected community and y+ denotes the desired decision.It abbreviates p(s = 1) as p(s1) and p(y = +) as p(y+).

4.1. Statistical tests

Statistical tests assess whether observed outcome differences between protected and general groups are unlikely under a no-difference hypothesis, but significance alone does not establish practical magnitude or absence of discrimination.

  • 4.1. Statistical tests: Statistical tests typically test the null hypothesis that protected and general groups have equal outcomes or proportions.Regression, two-sample, proportion, chi-square, and rank tests are described for different data and assumption settings.
  • 4.1. Statistical tests: Statistical significance does not measure difference magnitude, and failing to reject the null does not prove that discrimination is absent.A small sample may lack enough evidence to declare discrimination, while a statistically significant effect may be practically minor.
  • 4.1. Statistical tests: Regression slope tests estimate the protected-variable coefficient and test whether it differs significantly from zero.The regression may include only the protected variable or additional variables from X that explain observed decision differences.
  • 4.1. Statistical tests: Two-sample t-tests compare group means under independent-sample, normality, and equal-variance assumptions.The test statistic uses the difference between expected outcomes and group sample sizes and variances.
  • 4.1. Statistical tests: Proportion and chi-square tests assess equality of positive-outcome rates, including across multiple groups.Rejecting the chi-square null indicates that at least one group has a statistically different proportion.

4.2. Absolute measures

Absolute measures quantify outcome differences between protected and general groups without explaining why those differences occur, using measures based on means, normalized differences, and rankings.

  • 4.2. Absolute measures: Absolute discrimination measures quantify the magnitude of differences between protected and general groups, usually comparing each protected group with the most favored group.They do not incorporate explanations for group differences.
  • 4.2. Absolute measures: Mean difference computes the difference between group means and treats zero difference as no discrimination.For binary classification, it is also called the discrimination score or sliftd.
  • 4.2. Absolute measures: Normalized difference scales binary mean difference by the positive-outcome rate, with δ = 1 for maximum discrimination at that rate and δ = 0 for none.The normalization accounts for the maximum possible discrimination under the given positive-outcome rate.
  • 4.2. Absolute measures: AUC measures discrimination through pairwise ranking comparisons between two groups and linearly relates to binary mean difference as AUC = 0.5d + 0.5.The relation assumes binary targets and adds half a point for ties.
  • 4.2. Absolute measures: AUC becomes time- and memory-intensive on large datasets because it requires a quadratic number of comparisons, although ranking can provide an alternative computation.The ranking-based alternative uses the sum of ranks for the favored group.

4.2.4. Impact ratio.

Impact and related association measures express discrimination through outcome ratios or dependence between protected status and outcomes, with impact ratio using a legal 80% threshold in cited applications.

  • 4.2.4. Impact ratio.: Impact ratio divides the protected group’s positive-outcome rate by the general group’s rate, with r = 1 indicating no discrimination.U.S. courts deem decisions discriminatory when the protected-group rate is below 80% of the general-group rate.
  • 4.2.4. Impact ratio.: Elift ratio instead divides the protected-group positive-outcome rate by the overall positive-outcome rate.It is presented as a measure similar to impact ratio but with a different denominator.
  • 4.2.4. Impact ratio.: The same probability-ratio formulation is referred to as η-neutrality when bounded by 1 + η for all outcome and group values.The cited formulation expresses a neutrality condition through the joint and marginal probabilities.
  • 4.2.4. Impact ratio.: Odds ratios measure association between exposure and outcome and have been used to quantify discrimination through protected-group outcome proportions.Their popularity is linked to the relation between logistic-regression coefficients and odds ratios.
  • 4.2.4. Impact ratio.: Mutual information measures dependence between protected status and outcomes and can be normalized to range from 0 to 1.In discrimination literature it is also called the normalized prejudice index; categorical variables use a probability-based summation.

4.2.8. Balanced residuals.

Balanced residuals evaluate whether model underpredictions and overpredictions are balanced across protected and regular groups, but can disagree with discrimination in the data or with equal treatment.

  • 4.2.8. Balanced residuals.: Balanced residuals compare actual outcomes with model predictions and require average residuals to be equal across protected and regular groups.The proposal is described as a criterion rather than a fully specified measure.
  • 4.2.8. Balanced residuals.: A model can receive no discrimination signal from balanced residuals when it perfectly reproduces discriminatory training outcomes.If all residuals are zero, the measure shows no discrimination even though the predictions inherit discrimination from the original data.
  • 4.2.8. Balanced residuals.: A constant prediction equal to the regular group’s mean can indicate discrimination through residual imbalance even when everyone receives the same prediction.The measure may therefore flag unequal residuals despite equal treatment by the constant-output model.
  • 4.2.8. Balanced residuals.: Relation measures from feature selection could serve as absolute discrimination measures because stronger dependence between protected status and target implies larger absolute discrimination.Candidate families include correlation-based, information-theoretic, and one-class-classifier measures.
  • 4.2.8. Balanced residuals.: For more than two groups, absolute measures can be extended through pairwise, one-versus-rest, or other multiclass-style comparisons.The survey reports a recommendation to use the maximum comparison as the final discrimination score, with a weighted sum as an alternative.

4.3. Conditional measures

Conditional measures assess group differences after accounting for other individual characteristics, separating potentially explainable differences from remaining discrimination. The section discusses unexplained difference, propensity models, and belift ratio, including practical limitations in specifying explanatory variables and estimating probabilities.

  • Conditional measures use input characteristics X to distinguish differences explainable by individual characteristics from remaining discriminatory differences.
  • Unexplained difference subtracts differences explained by legitimate variables from the overall mean difference.The mean difference is d = p(y+|s0) − p(y+|s1), and unexplained difference is du = d − de.
  • Propensity models estimate protected-group membership from input variables, then compare decisions among individuals with similar estimated probabilities.The cited application learns s1 = f(X) using logistic regression.
  • Belift ratio conditions positive-outcome probabilities on protected-group status and input attributes, including attributes correlated with the protected variable.Its probabilities were proposed to be estimated with Bayesian networks.
  • Belift scores can vary across users because Bayesian-network construction depends on assumptions about interactions between variables.The original and simplified versions also cannot explain differences through variables correlated with the protected variable.

4.4. Structural measures

Structural measures quantify direct discrimination by examining affected individuals and comparing their decisions with those of comparable people. Situation testing compares protected individuals with the opposite group, while consistency compares predictions with nearest neighbors but can miss discrimination when group membership tracks legitimate features.

  • Structural measures target direct discrimination by identifying affected individuals and analyzing how many people in the dataset are affected.
  • Situation testing estimates the fraction of protected-group individuals whose decisions would differ when compared with the opposite group.Positive and negative discrimination are handled separately.
  • Consistency compares each individual’s prediction with predictions from nearest neighbors, including neighbors from any group.
  • Consistency can report no discrimination when protected status is highly correlated with legitimate features, despite different outcomes between groups.In the location example, perfect consistency coexists with maximum discrimination because each group’s neighbors receive the same decision.

5. ANALYSIS OF CORE MEASURES

The survey compares core discrimination measures under controlled variations in group balance, outcome rates, and underlying discrimination. Normalized difference is the most robust difference-based measure, whereas ratio measures can be highly sensitive and misleading; all measures still require accounting for legitimate group differences.

  • Experimental design: The analysis examines mean difference, normalized difference, mutual information, impact ratio, elift, and odds ratio, with AUC omitted because it is equivalent to mean difference in classification.Experiments use synthetic data to control protected-group proportion, positive-output proportion, underlying discrimination, and sample size.
  • Measure boundaries: Difference-based measures use 0 as the no-discrimination boundary, whereas ratio-based measures use 1 and AUC uses 0.5.These boundaries occur when one group receives all positive decisions and the other receives all negative decisions.
  • Difference-based measures: Normalized difference captures underlying discrimination across variations in group balance and positive-output rates more reliably than mean difference or mutual information.At p(s1) = 90% and p(y+) = 90%, true discrimination may be near 100%, while mean difference and mutual information indicate nearly zero; normalized difference captures the situation.
  • Recommendations: Mean difference can be used for approximately balanced data, but normalized difference is preferred when group or outcome imbalance may affect measurement.Normalized difference is more complex to compute, which may limit practical use outside research.
  • Ratio-based measures: Ratio measures are sensitive to group and outcome imbalances, asymmetric for reverse discrimination, and therefore difficult to interpret correctly.Odds ratio and impact ratio are especially sensitive; elift is more stable but can still exaggerate discrimination at extreme positive-output rates.
  • Limitations: Directly applying core measures is insufficient because they do not account for legitimate explanations of differences between groups.Stratification can compare people with similar legitimate characteristics, but outcomes may depend on method parameters and task settings.

6. RECOMMENDATIONS FOR RESEARCHERS AND PRACTITIONERS

The survey recommends normalized difference measures for fairness, with simple mean difference suitable for balanced classes and groups, while cautioning against ratio-based measures. It also emphasizes conditioning on qualification-based population segments and extending research beyond simple binary settings.

  • Normalized difference is generally recommended, while simple mean difference may suffice when classes and groups are well balanced.
  • Ratio-based measures are not recommended because their interpretation is challenging across different situations.
  • Core measures should be applied within qualification-based population segments because unconditioned populations rarely have uniform qualifications.The survey notes that justified segmentation is important in both research and practice.
  • Future work should address multiple protected characteristics, multiclass and regression settings, legitimate variables, noisy inputs, and missing protected attributes.
Loading 1511.00148v1…