Survey on Causal-based Machine Learning Fairness Notions

1 INTRODUCTION

• Correlation-based notions can miss discrimination when subgroup patterns differ from whole-population patterns, as in Simpson’s paradox and the Berkeley admissions example.The survey also notes that disparate-treatment claims require a causal connection between the decision and the sensitive feature.Source: §1 · p. 1
• Causal fairness notions use a causal model and often require interventions or counterfactuals that cannot always be uniquely computed from observational data.The survey describes SCM identifiability, full-model estimation when identifiability fails, and potential-outcome estimation techniques such as matching and re-weighting.Source: §1 · p. 1
• The survey examines 19 causal fairness notions and offers scenario-based guidelines for choosing among them.The guidance is summarized in a decision diagram navigated using characteristics of the scenario.Source: §1 · p. 1
• It ranks causal fairness notions on Pearl’s observation, intervention, and counterfactual ladder to indicate their practical deployment difficulty.The ranking reflects that computing interventions and counterfactuals depends on identifiability.Source: §1 · p. 2
• The paper surveys fairness notions, causal-inference background, methods for inferring causal quantities from observational data, and suitability and applicability guidance.Its coverage includes identifiability, estimation from a full causal model, and potential-outcome estimation.Source: §1 · p. 2

2 THE NEED FOR CAUSALITY: AN EXAMPLE

• The hypothetical system uses school location (C) and students’ initial average level (A) to predict whether to fire a teacher (Y); C influences A.The example treats all three variables as binary and assumes student level is influenced only by school location.Source: §2 · p. 2
• Equal observed firing rates by initial student level can conceal bias: only 20% of high-income classes have low-level students, where the firing probability is 0.25.The passage concludes that correlation-only statistical fairness notions can fail to detect this bias.Source: §2 · p. 2
• Under interventions setting all teachers’ assigned students to low or high initial levels, the system compares the resulting firing probabilities to assess fairness.The intervention breaks the dependence between student level and school location.Source: §2 · pp. 2–3
• The intervention-based probabilities confirm bias against teachers assigned classes with low initial student levels.Source: §2 · p. 3

3 PRELIMINARIES AND NOTATION

• SCMs and potential-outcome models are formally equivalent, but SCMs more readily represent multiple causal pathways while potential outcomes suit individual-level causal effects.Source: §3 · p. 3, §3.3 · pp. 4–5

3.1 Structural Causal Model (SCM) Framework

• An SCM comprises exogenous variables, observable variables, structural functions, and a probability distribution over latent variables.Its causal diagram is a DAG; dependent unobserved variables can be represented with bidirected edges in semi-Markovian models.Source: §3.1 · p. 3, Figure 2 · p. 3
• An intervention fixes a variable’s value and removes its incident edges, inducing a changed distribution in the other variables.A counterfactual quantity can compare multiple worlds, including an observed world and one in which a variable takes another value.Source: §3.1 · pp. 3–4
• Interventional or counterfactual probabilities are identifiable when their value is uniquely computable from observable data under the causal graph and distributions.Identifiability depends on graph structure, particularly the location of unobserved confounders.Source: §3.1 · p. 4

3.2 Potential Outcome

• Potential outcomes represent each unit’s outcome under a treatment value, but observational studies reveal only the factual outcome, not the other potential outcome.In the hiring example, the unobserved counterfactual is what the same candidate’s hiring outcome would have been under another gender value.Source: §3.2 · p. 4
• Potential-outcome inference typically assumes SUTVA, ignorability, and positivity; ignorability requires no hidden confounding given observed covariates.Positivity requires each sensitive-attribute value to have positive probability for every covariate value.Source: §3.2 · p. 4

3.3 SCM and graphical models vs potential outcome

• Potential outcomes suit narrowly estimating treatment effects and decomposing sources of bias, whereas SCMs suit learning causal relations among variables.Potential outcomes are not well equipped for causal discovery because they do not define effects for variables other than the treatment.Source: §3.3 · p. 5

4 CAUSALITY-BASED FAIRNESS NOTIONS

• Total variation compares observed outcome distributions across groups but cannot reflect the causal mechanism generating or collecting the data.The paper presents total effect as its causal counterpart, defined using intervention probabilities.Source: §4 · p. 5
• In the potential-outcome framework, ATE compares average outcomes under each treatment, while ETT focuses on the observed treatment group and CATE conditions on covariates.Exact ATE computation requires observed and counterfactual outcomes, which are typically not both observable.Source: §4 · pp. 5–6
• The hiring example shows that aggregate gender comparisons can conflict with job-type-adjusted comparisons because applicants are distributed differently across job types.The paper connects this pattern to Simpson’s paradox and explains that assessing each job type separately is needed in this example.Source: §4 · p. 6, Table 3 · p. 7
• Disparate-treatment notions divide causal effects into direct, indirect, explainable, and spurious pathways, with experts judging which pathways are fair or discriminatory.Direct discrimination concerns the direct A→Y edge; indirect discrimination concerns paths through proxy attributes, while spurious effects begin with an edge into A.Source: §4 · pp. 6–7
• Natural direct effect isolates the direct effect by setting A to the disadvantaged value on the direct path and to the reference value on mediated paths.The hiring example treats education as a legitimate explaining variable and hobby as a proxy for gender; natural indirect effect does not distinguish fair from unfair indirect effects.Source: §4 · pp. 7–8
• Path-specific effect can target selected causal paths, allowing the hiring example to include direct and hobby-mediated effects while excluding the explainable education path.The specified path set is A→Y and A→R→Y, whereas A→E→Y is treated as explainable.Source: §4 · p. 8

4.1 No unresolved discrimination

• No unresolved discrimination permits directed paths from A to Y only through a resolving variable accepted as nondiscriminatory.In the hiring graphs, the left includes unresolved paths through hobby and directly to hiring; the right’s effect is through education.Source: §4.1 · p. 8, Figure 4 · p. 8
• The hiring example treats education as a resolving variable, distinguishing its path to the outcome from the direct and hobby-mediated paths.The right-hand graph has no unresolved discrimination because the effect proceeds through education.Source: §4.1 · p. 8
• Applying the notion in real scenarios requires a valid causal graph, and observational data alone may not establish whether a predictor satisfies it.The paper notes that this can remain true even when resolving variables are known in advance.Source: §4.1 · p. 8

4.2 No proxy discrimination

• Potential proxy discrimination exists when a path from A to Y is blocked by a proxy variable R used in deciding the outcome.The proxy is a descendant of A and may have nearly the same impact as using A directly.Source: §4.2 · p. 8
• In the hiring example, the A→R→Y route represents potential proxy discrimination, while intervening on R removes the A–R edge in the comparison graph.The figure contrasts the graph with this route against one without proxy discrimination.Source: §4.2 · p. 8, Figure 5 · p. 8
• The notion requires a valid causal graph, so its application depends on the causal-discovery result being correct.Source: §4.2 · p. 8

4.3 Counterfactual fairness

• Counterfactual fairness requires equal outcome distributions in actual and counterfactual worlds for every individual under any assignment of X.The definition is individual-level because it includes the latent variable U.Source: §4.3 · p. 8
• The paper says counterfactual fairness typically coincides in practice with individual treatment effect.Source: §4.3 · p. 8
• The described test assumes the full causal model, including latent variables, is available and estimates P(U) from observed data using Markov chain Monte Carlo.It then applies abduction, action, and prediction to generate counterfactual samples and compares actual and counterfactual density functions.Source: §4.3 · p. 8

• The three effects distinguish direct and indirect effects for a specific group from spurious effects along back-door paths between A and Y.Source: §4.4 · p. 9
• For female candidates, the direct effect indicates counterfactual discrimination, while the indirect effect indicates no counterfactual indirect discrimination.Source: §4.4 · p. 9
• The spurious effect in the hiring example favors female candidates, and counterfactual indirect effect does not distinguish fair from unfair direct effects.Source: §4.4 · p. 9
• Unlike NDE and NIE, counterfactual effects focus on individuals in a specific group and characterize causal effects through spurious back-door paths.Source: §4.4 · p. 9

• Statistical equalized odds requires equal true-positive and false-positive rates across groups, but does not identify the causes of discrimination.Source: §4.5 · p. 9
• In the hiring example, the counterfactual direct error rate is higher for female candidates who should be hired.Source: §4.5 · p. 9, Table 5 · p. 9
• Counterfactual direct, indirect, and spurious error rates decompose statistical equalized-odds error disparity into causal components.Source: §4.5 · p. 9

4.6 Individual direct discrimination

• Individual direct discrimination compares positive-decision rates among similar individuals in protected and unprotected groups, using a typically 5% threshold.Its distance function uses direct causes of the outcome and their causal effects; it avoids counterfactual outcomes but is complex to compute.Source: §4.6 · p. 10

4.7 Non-Discrimination Criterion

• The non-discrimination criterion assesses direct discrimination with block sets, but does not work in semi-Markovian models with hidden confounders.It avoids nested counterfactual quantities by checking dependence across combinations of block-set values.Source: §4.7 · p. 10

4.8 Equality of Effort

• Equality of effort compares the effort protected and unprotected groups need to reach an outcome level by intervening on a treatment variable.Its main limitation is that a single treatment variable typically does not reflect the discrepancy between groups.Source: §4.8 · p. 10
• Situation testing estimates counterfactual potential outcomes for similar individuals, and the minimum effort to reach a target outcome is defined for each group.Source: §4.8 · p. 10

4.9 Interventional and justifiable fairness

• Interventional fairness requires K-fairness for every set of variables, while justifiable fairness requires it only for supersets of admissible variables.Justifiable fairness assumes admissible and inadmissible variables can be distinguished; it does not require the underlying causal model.Source: §4.9 · p. 11

4.10 Individual equalized counterfactual odds

• Individual equalized counterfactual odds conditions on the same outcome value and is the only criterion requiring interventions on both prediction and actual outcome.Source: §4.10 · p. 11

5.1 Identifiability

• Because sensitive attributes are not randomly assigned and counterfactual outcomes are unobservable, randomized trials can estimate causal effects but are often impractical or unethical in fairness settings.Identifiability asks whether a causal quantity can be expressed using available observational data; the section also considers estimation when quantities are not identifiable.Source: §5.1.1 · p. 11, §5 · p. 11

5.1.1 Identifiability of causal effect (intervention).

• For semi-Markovian models, identifiability of P(y^a) is not guaranteed; Pearl ’s three-rule do-calculus can express interventional probabilities in observational terms, and complete ID returns an expression if identifiable or the reason otherwise.Source: §5.1.1 · pp. 12–13

5.1.2 Identifiability of counterfactuals.

• Counterfactual identifiability is assessed by building a reduced graph of parallel worlds and applying causal-effect criteria; semi-Markovian models lack a single necessary-and-sufficient criterion.ID* and IDC* produce expressions in interventional probabilities, which ID can then express in observational probabilities.Source: §5.1.2 · pp. 13–14

5.1.3 Identifiabilityof direct and indirect effects.

• Natural direct and indirect effects are identifiable from observational data in Markovian models but not generally in semi-Markovian models; path-specific effects fail under the recanting-witness criterion.In semi-Markovian models, experimental identification of natural direct and indirect effects requires a qualifying set W; recanting districts generalize the path-specific criterion.Source: §5.1.4 · p. 14, §5.1.3 · p. 14

5.2 Estimation based on full knowledge of the causal model parameters

• With the causal model structure available, estimation can infer latent variables, generate counterfactual samples through abduction, action, and prediction, and train or assess a counterfactually fair predictor.Training uses non-descendants of the sensitive attribute and estimated latent variables; a modified procedure instead estimates counterfactual bias for an already trained predictor.Source: §5.2 · p. 15
• Counterfactual assessment requires access to the predictor to generate predictions for counterfactual samples, and results show racial unfairness favoring whites, strongest for white–Black comparisons.In Communities and Crime, Hispanic–white and Asian–white comparisons can be counterfactually fair; the law-school finding corroborates Kusner et al..Source: §5.2 · pp. 15–16

5.3 Potential outcome estimation techniques

• Potential-outcome estimation targets the effect of a treatment such as a sensitive attribute on an outcome and commonly assumes SUTVA, ignorability, and positivity.The survey focuses on re-weighting, matching, and stratification approaches relying on these assumptions [16, 59].Source: §5.3 · p. 16

5.3.1 Re-weighting.

• Re-weighting assigns observational units weights to create a pseudo-population with aligned protected and unprotected group distributions, using a balancing score such as the propensity score.In the hiring example, the IPW ATE estimate is 0.25, indicating discrimination favoring women; the normalized IPW estimate is 0.125, matching the ATE. IPW can be unstable with small propensity scores, and misspecification can distort estimates; doubly robust estimation and trimming are described as remedies.Source: §5.3.1 · p. 17

5.3.2 Matching.

• Matching estimates units’ counterfactual outcomes using comparable units in the opposite group, commonly selected by propensity-score similarity; options include nearest-neighbor, caliper, radius, and kernel matching.Exact matching may leave units unmatched, and caliper matching can also fail to find matches within its distance limit.Source: §5.3.2 · p. 17

5.3.3 Stratification.

• Stratification adjusts for confounders by splitting observations into groups and combining group-specific treatment effects; with all needed variables observed and infinite data, it can consistently and unbiasedly estimate ATE.Sparse strata can prevent some group effects from being estimated; combining strata with identical propensity scores can address this problem.Source: §5.3.3 · p. 18

5.4 Estimating causal effects on benchmark datasets

• On the Communities and Crime, COMPAS, and German Credit datasets, statistical parity generally reports higher discrimination than causal-based notions, while Berkeley estimates show no discrimination or discrimination favoring women, consistent with Simpson’s paradox.The exception is ATE_match on German Credit, which reverses the direction to discrimination against male applicants (−0.167). For imputation estimates, total effect is approximately the sum of direct and indirect effects.Source: §5.4 · p. 19
• Across all four benchmark datasets, IPW reduced covariate imbalance by at least an order of magnitude; for unemployment rate in Communities and Crime, abs-SMD fell from 0.68 to 0.078 standard deviations.The benchmark analysis applies seven potential-outcome estimators alongside total variation as a statistical-parity baseline.Source: §5.4 · pp. 18–19

6.1 Suitability

• Six scenario criteria—confounding, explaining variables, intersectionality, masking, known latent variables, and outcome reliability—guide fairness-notion selection.Figure 12 summarizes the guideline as a decision diagram for a real-world scenario.Source: §6.1 · p. 19, Figure 12 · p. 21
• Masking or intersectional bias favors individual rather than group notions; with explaining variables, individual direct discrimination is recommended, while known latent variables support counterfactual fairness or individual equalized counterfactual odds.Individual equalized counterfactual odds is not recommended when ground truth is unavailable or unreliable.Source: §6.1 · p. 20
• Without explaining variables, group-based total effect, ATE, or interventional fairness can be used; with them, semi-Markovian models mislead non-discrimination, and unreliable ground truth rules out counterfactual error rates.When there are no confounding or explaining variables, causal inference is not needed to assess fairness appropriately.Source: §6.1 · p. 20
• ETT, ATT, and ATC are generally unsuitable for fairness scenarios because bias may run in both directions, unlike the one-direction treatment question for which ETT is relevant.The paper gives medicine effects on patients as an example where the reverse treatment direction is not relevant.Source: §6.1 · pp. 20–21

6.2 Applicability

• The ladder ranks causal notions by the required reasoning: higher rungs make identifiability more challenging and notions less applicable; TV is rung 1, while some notions require a causal graph.Non-discrimination requires the Markov property and has exponential complexity; the cited relaxation remains computationally intractable.Source: §6.1 · p. 21, Figure 13 · p. 22
• Rung-2 notions apply when experiments are possible or interventions are identifiable; interventions are identifiable from observational data in Markovian models, while semi-Markovian models require case-specific checks.Individual direct discrimination, for example, depends on identifiability of its counterfactual effect term.Source: §6.2 · p. 21
• In Markovian models, the top five counterfactual notions listed—ETT, NDE, NIE, and counterfactual effects—are identifiable; counterfactual fairness and path-specific effects have additional conditions.Counterfactual fairness requires X ∩ B = ∅; path-specific effects require that the recanting witness criterion is not satisfied.Source: §6.2 · p. 21
• Conditioning counterfactual error rate and individual equalized counterfactual odds on true outcome Y can induce hidden confounding, making even a Markovian model semi-Markovian.The explanation criterion is identified as a criterion for counterfactual error-rate identifiability in Markovian models.Source: §6.2 · p. 22

7 CONCLUSION

• Causal reasoning can address statistical anomalies, support causal interpretations relevant to disparate-treatment claims, and distinguish direct from indirect paths where discrimination may be acceptable or unacceptable.Source: §7 · p. 22
• Most surveyed causal fairness notions rely on a causal graph, and generating graphs consistent with observed data remains a relevant direction for fairness research.Source: §7 · p. 22