PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation

Angelina Wang, Vikram V. Ramaswamy, Olga Russakovsky

arXiv:2205.04610v1cs.LGcs.CY

TL;DR

Machine-learning fairness has often treated demographic identity as a single binary attribute, while intersectional evaluation raises questions about training labels, shrinking groups, and fairness metrics. The paper empirically examines these pipeline stages on Census-derived datasets and offers recommendations; for the considered algorithms and datasets, subgroup base-rate rankings are highly correlated with model true-positive-rate rankings, even under fairness constraints.

  • Problem

    Fairness research often considers one demographic axis at a time, although interacting identities can produce distinct forms of discrimination.

    Source: §1 · p. 1

  • Method

    The paper empirically examines identity-label selection, training with smaller groups, and multi-subgroup fairness evaluation, and develops recommendations for each stage.

    Source: §1 · p. 2, §5.2 · p. 10

  • Results

    For the considered algorithms and datasets, subgroup positive-label base-rate rankings are highly correlated with model true-positive-rate rankings, even under fairness constraints.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Use empirical results alongside domain knowledge to select training identities, evaluate across known demographic groups, and consider normative implications before applying imbalance techniques.

    Source: §1 · p. 2, §5 · p. 9, §5.1 · p. 9

    Unsupervised recategorization without human oversight can lead to harmful assignments, such as placing Non-Binary people in gendered groups.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Research in machine learning fairness has historically considered a single binary demographic attribute; however, the reality is of course far more complicated. In this work, we grapple with questions that arise along three stages of the machine learning pipeline when incorporating intersectionality as multiple demographic attributes: (1) which demographic attributes to include as dataset labels, (2) how to handle the progressively smaller size of subgroups during model training, and (3) how to move beyond existing evaluation metrics when benchmarking model fairness for more subgroups. For each question, we provide thorough empirical evaluation on tabular datasets derived from the US Census, and present constructive recommendations for the machine learning community. First, we advocate for supplementing domain knowledge with empirical validation when choosing which demographic attribute labels to train on, while always evaluating on the full set of demographic attributes. Second, we warn against using data imbalance techniques without considering their normative implications and suggest an alternative using the structure in the data. Third, we introduce new evaluation metrics which are more appropriate for the intersectional setting. Overall, we provide substantive suggestions on three necessary (albeit not sufficient!) considerations when incorporating intersectionality into machine learning.

1 INTRODUCTION

Intersectional fairness raises linked questions about which identity labels to use, how to train on progressively smaller groups, and how to evaluate fairness across many subgroups. The paper recommends empirically validating training-label choices, considering data structure when addressing underrepresentation, and extending evaluation beyond pairwise fairness comparisons.

1 INTRODUCTION

  • The work studies five fairness algorithms on five US Census-derived tabular datasets in supervised binary classification of socially important outcomes.Its focus is algorithmic discrimination against demographic subgroups, rather than individuals.Source: §1 · p. 2
  • Supplement domain knowledge with empirical results when choosing training identities, because different algorithms benefit from different levels of granularity; generally evaluate on all known demographic groups.The authors note that this issue also arises within a single axis, such as choosing how granularly to represent Asian Pacific Islander identities.Source: §1 · p. 2
  • As more identities are included, groups shrink; data-imbalance techniques may raise normative concerns, while shared structure between groups may support learning about underrepresented groups.The authors give learning about Black Female patterns from groups such as Black Male as a specific example.Source: §1 · pp. 2–3
  • For the considered algorithms and datasets, subgroup positive-label base-rate rankings are highly correlated with model true-positive-rate rankings, even under fairness constraints.The authors argue that pairwise comparisons can obscure information when extended to more subgroups.Source: §1 · p. 3
  • The authors limit their contribution to intersectional algorithmic fairness, not data-driven insights into societal intersectionality; quantitative studies can flatten groups into monoliths.They also note that identity delineations are unstable and difficult to operationalize.Source: §1 · pp. 2–3

2 RELATED WORK

Prior machine-learning fairness work often used binary demographic attributes, while only a subset of surveyed algorithm papers handled multiple identity axes empirically. Existing work proposed several intersectional methods, but some influential approaches used different terminology.

2 RELATED WORK

  • Of 26 surveyed popular fairness-algorithm papers, 16 could handle attributes beyond binary, and only 7 of those reported empirical results on multiple identity axes.The survey retained highly cited papers from nine conferences before narrowing to papers proposing fairness algorithms.Source: §2 · p. 3
  • Prior work proposed intersectional learning, calibration, regularization, post-processing, and auditing methods, alongside comparisons of existing fairness methods.The cited examples include methods that weight groups by frequency and thus downweight underrepresented groups.Source: §2 · p. 3
  • Some technically advanced intersectional fairness work used terms such as “fairness gerrymandering” rather than “intersectionality,” omitting the history and context associated with the latter term.The authors point specifically to Hebert-Johnson et al. and Kearns et al.Source: §2 · p. 3

3 SETUP

The empirical setup uses Census-derived datasets, fairness algorithms, and a training objective balancing soft accuracy with group fairness. The authors frame the experiments as demonstrations of phenomena in a commonly adapted setting, not as exhaustive evidence across fairness settings.

3 SETUP

  • The experiments use five US Census-derived datasets, selecting California’s 2018 slice for computational feasibility and sufficient sample size.The authors assume each dataset’s positive label is desirable, while acknowledging that this may be unclear for outcomes such as longer travel time.Source: §3 · p. 4
  • Each run uses five trials with different random seeds and training, validation, and test splits to produce 95% confidence intervals.The setup favors straightforward choices aligned with prior work and is intended to demonstrate that phenomena can arise, not to show they occur in every fairness setting.Source: §3 · p. 4
  • The authors use soft accuracy and maximum pairwise TPR difference to balance accuracy with group fairness, optimizing their geometric mean during hyperparameter tuning.They choose a metric analogous to equal opportunity because they treat positive labels as more desirable, while noting that metric choice depends on downstream context.Source: §3 · p. 4
  • The study compares a three-layer neural-network baseline with five fairness algorithms: RWT, RDC, LOS, GRP, and GRY.The methods include reweighting, cost-sensitive classification, an intersectional fairness loss, probabilistic model combinations, and a zero-sum-game approach.Source: §3 · p. 5

4 SELECTING WHICH IDENTITIES TO INCLUDE

The case studies show that the best identity labels for training are not always clear in advance: outcomes vary across datasets and algorithms. The authors recommend combining contextual understanding with empirical validation, while generally evaluating at the finest-grained level possible.

4 SELECTING WHICH IDENTITIES TO INCLUDE

  • Because demographic categories can conceal differing within-group heterogeneity, the authors investigate how label granularity affects model predictions rather than assuming groups are homogeneous.Their case studies examine constituent API identities and the racial category Other.Source: §4 · p. 5

4.1 Case study: heterogeneity within Asian Pacific Islander

  • GRY achieved a max TPR difference of 0.03 ± 0.01 on ACSTravelTime when API was represented by 2 groups, and the most granular training labels did not always perform best.Evaluation used the > 10-group setting, excluding Black and White groups from the subgroup comparison.Source: §4.1 · p. 6, Figure 1 · p. 7

4.2 Case study: heterogeneity within Other

  • On ACSEmployment, treating Other as a separate group often performed best, while on ACSIncome no approach was best across all scenarios.For ACSIncome, Redistribute outperformed Separate for RWT and RDC.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 Constructive Suggestions

  • The case studies suggest that the best identities to include are rarely clear a priori, so selection should combine application context with empirical experiments.The authors note that empirical validation can help identify which grouping works best for a particular application.Source: §4.3 · pp. 7–8
  • Including every available identity label may be computationally difficult, and small groups may benefit from joining similar groups; post-processing approaches may be less susceptible to overfitting.The post-processing exception reflects that such approaches generally learn only one or two group-specific parameters.Source: §4.3 · p. 8
  • The authors advise caution with recategorization because unsupervised grouping without human oversight can produce harmful assignments.They give assigning a person who identifies as Non-Binary to a gendered grouping as an example, while noting that domain expertise may make recategorization more permissible.Source: §4.3 · p. 8
  • Evaluation should generally use the finest-grained identities available, with error bars conveying uncertainty for small groups and larger-group evaluations included as well.The authors also suggest collecting additional demographic labels when domain expertise indicates that relevant axes or identities are missing, while keeping privacy concerns in mind.Source: §4.3 · p. 8

5 HANDLING PROGRESSIVELY SMALLER GROUPS

As intersectional groups become smaller, conventional imbalance techniques raise normative concerns, while shared data structure may help only when groups share predictive patterns. The authors recommend testing that suitability for each algorithm and task, without treating mathematical structure as social structure.

5 HANDLING PROGRESSIVELY SMALLER GROUPS

  • Intersectional long-tail scarcity occurs within a demographic label, unlike the usual machine-learning long tail across labels.The paper illustrates the distinction with a small subset of chairs that are wooden and look different from the others.Source: §5 · pp. 8–9

5.1 Dataset imbalance techniques in machine learning

  • Reweighting and resampling shift attention toward selected individuals, while synthetic and counterfactual examples raise distinct normative concerns, including contested demographic manipulation and unresolved face-generation questions.The paper also notes that GAN-generated face examples can evoke the harms of blackface; whether machine-only use changes that concern remains open.Source: §5.1 · p. 9
  • For fairness applications, simulation may serve better as a testbed for sociotechnical systems than as a source of training data.The paper contrasts these applications’ lower data needs with image-heavy reinforcement-learning settings.Source: §5.1 · pp. 9–10

5.2 Structure in data

  • Shared-identity groups can help reveal an underrepresented group’s predictivity, but the authors use base-rate-agnostic ROC AUC and caution that context may limit such sharing.They operationalize the intersectional effect by comparing Black Female AUC from shared-identity training groups against AUC after adding Black Female samples.Source: §5.2 · pp. 10–11
  • ACSIncome shows no subgroup predictivity difference, ACSPublicCoverage shows subgroup but not intersectional predictivity difference, and ACSTravelTime shows both.The experiments focus on Black Female AUC; the table compares training subgroups, and Fig. 3 compares training with and without Black Female samples.Source: Table 1 · p. 11, Figure 3 · p. 11, §5.2 · p. 11

5.3 Constructive Suggestions

  • The authors recommend analyzing whether structure-based learning suits a particular algorithm and task, and warn against transferring imbalance methods without considering normative implications.Their proposed analysis checks whether a group’s predictivity can be learned from other groups sharing its identities.Source: §5.3 · p. 12
  • Predictive structure does not imply social structure: relying on it too heavily may counteract intersectionality and obscure multiply marginalized groups’ unique effects.The authors caution that gender may be understood only in a racialized way, and vice versa.Source: §5.3 · p. 12

6 EVALUATING A LARGE NUMBER OF GROUPS

Max-difference metrics can miss how all demographic groups are ranked and can give conflicting assessments of which algorithms are fairest. The authors recommend more deliberate comparisons and additional summary statistics for evaluating many groups.

6.1 Weaknesses of existing evaluation approaches

  • Max-difference metrics can conceal subgroup distributions: TPR sets {.1, .2, .8} and {.1, .6, .8} both yield a maximum difference of .7.One-vs-all comparisons are influenced most by the majority group, while pairwise comparisons may incorporate only the maximum and minimum groups.Source: §6.1 · p. 12
  • In ACSIncome, all algorithms ranked the highest-base-rate group above the lowest-base-rate group, even though fairness improved over baseline for all except GRP.The authors note that consistent subgroup ranking can compound discrimination even when a max-difference fairness criterion is met.Source: Table 2 · p. 13, §6.1 · p. 13
  • RWT and RDC were less likely than GRP and GRY to reify rankings; ACSIncome marital status × sex showed more reifying patterns, while ACSMobility marital status × sex × disability showed less.Only statistically significant Kendall’s Tau values are displayed.Source: §6.1 · p. 13, Figure 4 · p. 14
  • Across five of six algorithms, base-rate and TPR rankings were significantly correlated, while max TPR difference and Kendall’s Tau ranked GRY and RWT as fairest, respectively.The authors argue that the metrics offer different information and that application context should guide how their trade-offs are navigated.Source: §6.1 · p. 13, Figure 4 · p. 14

6.2 Constructive Suggestions

  • When using pairwise comparisons, practitioners should choose reference groups deliberately and contextually rather than defaulting to the most or least privileged group.Anchoring comparisons to a dominant group can reinforce the idea that it is the norm.Source: §6.2 · pp. 14–15
  • Disaggregated subgroup analyses should precede deployment, while rank information for extreme-base-rate groups or correlations between base-rate and TPR rankings can supplement pairwise differences during model iteration.The authors note that full disaggregation can be unwieldy during iteration and that small TPR gaps between the same groups across algorithms can compound.Source: §6.2 · p. 15

7 CONCLUSION

The conclusion offers guidance on identity selection, training with small groups, and evaluating many subgroups when incorporating intersectionality into machine learning.

7 CONCLUSION

  • For training, combine domain knowledge with experiments to choose identities, and evaluate on the most granular intersecting identities available.The authors caution that these recommendations are among many steps needed to incorporate intersectionality.Source: §7 · p. 15
  • For increasingly small groups, the authors caution against importing imbalance techniques without considering their normative concerns and suggest leveraging structure between groups sharing an identity.Source: §7 · p. 15
  • For evaluation, the authors recommend more thoughtful pairwise comparisons and additional metrics to capture broader algorithm patterns that existing metrics may obscure.Source: §7 · p. 15

8 POSITIONALITY STATEMENT

The authors disclose that they are computer scientists without traditional social science backgrounds and lack lived experience of some group identities discussed.

8 POSITIONALITY STATEMENT

  • The authors report that they lack traditional social science backgrounds and lived experience for some group identities discussed.Source: §8 · p. 15

A ALGORITHMS AND HYPERPARAMETERS

The appendix describes five fairness algorithms and their implementation and tuning choices, including a three-layer neural-network baseline for three algorithms.

A ALGORITHMS AND HYPERPARAMETERS

  • For the three algorithms where details are provided, the authors use a three-layer neural network as the baseline architecture and tune hyperparameters on the validation set.The data are split 70–30 for training/validation and test, then the training/validation portion is split 70–30 again.Source: A ALGORITHMS AND HYPERPARAMETERS · pp. 19–20
  • RWT reweights group-specific positive and negative instances based on TPR relative to the overall rate, continuing training between reweighting iterations.RDC uses a sequence of cost-sensitive classifications to produce a randomized classifier; the adaptation uses classifier probabilities as continuous outputs.Source: A ALGORITHMS AND HYPERPARAMETERS · pp. 19–20
  • LOS adds an intersectional fairness regularizer to the loss, while GRP combines logistic-regression models and uses the plugin variation because weighted ERM is prohibitively slow.GRY uses cost-sensitive classifications in a learner-auditor zero-sum game and averages the hard outputs of a sequence of classifiers.Source: A ALGORITHMS AND HYPERPARAMETERS · p. 20
Loading 2205.04610v1…