Source-linked AI summary
Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias
Aditya Parikh, Eike Petersen, Stella Frank, Enzo Ferrante, Melanie Ganz, Aasa Feragen
TL;DR
The paper examines whether models should be trained not to encode demographic group membership, given concerns that encoded demographics may be used as shortcuts or reproduce biased labels. It distinguishes marginal from class-conditional invariance and analyzes their fairness implications. Its results show that invariance can conflict with popular fairness criteria, while some demographic encoding may be needed when group membership correlates with outcomes.
Problem
Demographic information may be recoverable from model representations and exploited as a shortcut or to reproduce biases in systematically biased labels.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
When group prevalences differ, strict marginal invariance can remove useful information and hurt performance, so representation invariance is neither desirable nor sufficient for fairness.
Source: §6 · p. 8, §7 · p. 9The empirical analysis is limited to classification tasks and simulated prevalence differences that may not reflect real-world extremes.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Encoded demographic information in internal model representations is a commonly assumed risk factor for algorithmic bias, with demographic representation invariance often being touted as the ideal state. However, while demographic shortcut learning is a genuine threat, some degree of encoding is necessary when demographics correlate with target labels. Here, we show, mathematically and empirically, that enforcing demographic invariance can actually hamper bias mitigation and even create new biases. We distinguish marginal from class-conditional representation invariance, and show that they imply the standard group fairness notions of demographic parity and equalized odds, respectively. We evaluate the effects on predictive performance and fairness of enforcing both invariance types, both theoretically and empirically across five tabular and two chest X-ray imaging datasets. Our findings support our mathematical argument that demographic representation invariance is neither desirable nor sufficient for fairness.
1 INTRODUCTION
Although demographic information in representations can be exploited as a shortcut or reproduce biased labels, task-relevant features may also encode group membership. The paper shows that marginal and class-conditional invariance imply different fairness criteria, and that enforcing invariance can create performance and fairness trade-offs.
1 INTRODUCTION
- Demographic information can be decoded from medical-imaging representations even when models were not trained to predict or use demographics.The paper notes the concern that encoded group membership may serve as a shortcut or reproduce biases in systematically biased labels.Source: §1 · p. 1
- Removing all demographic information may also remove task-specific features correlated with group membership and harm performance.This can occur when disease symptoms or prevalence differ between groups.Source: §1 · p. 1
- With unequal group prevalences, enforcing marginal invariance can produce systematic error-rate differences and harm equalized odds.The authors contrast this with class-conditional invariance, which enforces equal error rates but risks leveling down performance for the best groups.Source: §1 · p. 2, Figure 1 · p. 2
- Earlier alignment studies report benefits, but the paper argues that many known methods fail to remove demographic information accurately.The authors suggest those benefits may reflect regularization from incomplete removal rather than successful invariance.Source: §1 · p. 2
- Marginal representation invariance implies demographic parity, while class-conditional representation invariance implies equalized odds.The paper distinguishes identical representation distributions across groups from identical distributions within each label.Source: §1 · p. 2
- The experiments compare methods for enforcing invariance and show that marginal invariance fails to satisfy equalized odds, while class-conditional invariance enforces it with a leveling-down risk.The authors use a method that successfully enforces invariance in their comparison.Source: §1 · p. 2
2 PROBLEM SETTING
The paper formalizes supervised prediction using inputs, outcomes, sensitive group attributes, and learned representations. A feature extractor produces a representation that a second component maps to predictions; demographic decodability is used as a proxy for invariance.
2 PROBLEM SETTING
- The setup allows inputs to be tabular features, images, or other data types, and outcomes to be categorical, continuous, or more complex outputs.Sensitive attributes identify groups such as sex, ethnicity, or age group and may be multivalued or intersectional.Source: §2 · pp. 2–3
- A deterministic feature extractor maps input X to representation Z, and a second component maps Z to risk or score predictions R.Neither component takes the sensitive attribute A as input.Source: §2 · p. 3
- The paper treats group membership as encoded when it is decodable from Z above chance and uses decodability as a proxy for invariance.Decodability depends on the evaluation dataset, not solely on the model.Source: §2 · p. 3
3 WHAT DOES REPRESENTATION INVARIANCE GUARANTEE?
The section connects marginal and class-conditional representation invariance to different fairness guarantees and derives conditions under which those guarantees conflict. It also examines how prevalence shifts and within-class differences affect these invariance targets.
3 WHAT DOES REPRESENTATION INVARIANCE GUARANTEE?
- The section presents its connection between two forms of representation invariance and fairness and performance consequences as its novel contribution.Source: §3 · p. 3
3.1 MARGINAL REPRESENTATION INVARIANCE
- Marginal invariance implies demographic parity; with unequal prevalences, that constraint conflicts with perfect prediction and forces combined binary-group error e0 + e1 ≥ |π0 − π1|.Source: §3.1 · p. 3, UNEQUAL PREVALENCE ACROSS GROUPS… · p. 4
3.2 CLASS-CONDITIONAL REPRESENTATION INVARIANCE
- Class-conditional invariance implies equalized odds while allowing group-level Z distributions to differ with prevalence; under intra-class severity shifts, alignment can erase real presentation differences.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3 INVARIANCE-FAIRNESS INCOMPATIBILITIES
- When A relates to R and Y, marginal invariance conflicts with separation, sufficiency, and group calibration; class-conditional invariance conflicts with independence, sufficiency, and calibration under positivity.Source: §3.3 · p. 5
3.4 WHAT HAPPENS UNDER PREVALENCE SHIFTS?
- With a fixed model and unchanged within-class distributions, group-specific prevalence shifts preserve class-conditional invariance but not marginal invariance.Source: §3.4 · pp. 5–6
4 EXPERIMENTAL DESIGN
The experimental design tests invariance methods against ERM across tabular and chest X-ray datasets and varied prevalence conditions. It measures demographic decodability, predictive performance, fairness, and calibration.
Datasets and prevalence conditions
- The study covers five tabular benchmarks and two CheXpert tasks, using natural or manipulated prevalence conditions that include removing positive examples from lower-prevalence groups.CheXpert uses natural prevalence and a condition removing 50% of female-positive cases.Source: §4 · p. 6
Methods
- Invariant methods are compared with ERM using Fair Normalizing Flows on tabular data and flows, MMD regularization, and adversarial methods on CheXpert.The design varies method settings and uses three training seeds; thresholds are selected on the validation set.Source: §4 · p. 6
Evaluation
- Evaluation measures probe-based demographic decodability, AUROC, group fairness gaps, and calibration; Table 1 reports shift-50 fairness criteria as three-seed means.Table 1 omits standard deviations for space, with these and additional configurations reported in the appendices.Source: §4 · pp. 6–7, Table 1 · p. 7
5 RESULTS
Marginal alignment improves demographic parity but can worsen separation, sufficiency, calibration, and predictive performance. Class-conditional alignment better preserves equalized odds with little AUROC cost, while prevalence shifts expose different stability limits for the two approaches.
Experimental setup
- No alignment method achieves exact statistical invariance; the best marginal-alignment methods reach AUROC 0.52–0.58, while within-class probes reach 0.53–0.62.The reported results therefore reflect approximate invariance, with residual demographic dependence.Source: §5 · p. 7
Marginal alignment
- Marginal alignment reduces ∆DP to 0.01–0.05, but on Dutch ∆EO rises from 0.05 to 0.31 as task-AUROC drops 0.07–0.19 points across datasets.It also increases ∆PPV in four of five datasets and nearly doubles ∆bias on Dutch and Communities & Crime.Source: §5 · pp. 7–8
Class-conditional alignment
- Class-conditional alignment yields the smallest ∆EO on four datasets while keeping task-AUROC within 0.01–0.03 of ERM.Communities & Crime is the exception because its A = 0 stratum has fewer than 80 rows; calibration and sufficiency metrics slightly degrade.Source: §5 · p. 8
Prevalence shifts
- Under shift-75, marginal alignment’s ∆DP rises from 0.02 to 0.19, while conditional alignment keeps ∆EO near-stable at 0.05 to 0.03 but ∆DP rises from 0.01 to 0.32.Figure 4 evaluates models trained on the equalized condition across increasing base-rate gaps, with P(X | Y, A) fixed.Source: §5 · p. 8, Figure 4 · p. 8
6 DISCUSSION
The discussion argues that representation invariance is neither necessary nor sufficient for fairness: marginal alignment can sacrifice useful information and worsen performance, while class-conditional alignment presents a different tradeoff. Its risks depend on prevalence, constraint strength, task setting, and how invariance is defined.
Removing demographic information as a fairness objective?
- Strict marginal invariance enforces demographic parity but can remove useful information and harm performance when outcome prevalence differs.The authors characterize this as a cost beyond leveling down, because marginal invariance is incompatible with commonly desired fairness criteria.Source: §6 · p. 8
Removing unnecessarily strong demographic encoding
- Class-conditional invariance can reduce excess demographic encoding without costing AUROC, but may remove intra-class variation and compromise group calibration.The paper notes that this differs from marginal alignment’s performance and fairness tradeoff.Source: §6 · pp. 8–9
The strength of constraints matters
- Representation-level alignment is stronger than decision-level parity or separation: matching score distributions is sufficient, but not necessary, for the corresponding thresholded criteria.Under unequal prevalences, class-conditional risk-score alignment generally conflicts with group calibration, whereas equal error rates at a chosen threshold do not impose the same restrictions.Source: §6 · p. 9
When can representation invariance help?
- Marginal invariance should generally be avoided when group prevalences are not near equal; weaker regularization or post-hoc thresholds are alternatives in some settings.The discussion also identifies unsupervised domain adaptation, where target-domain labels are unavailable, as a possible use for invariance.Source: §6 · p. 9
Statistical invariance vs. model invariance
- Counterfactual fairness in medical imaging depends on which factors are changed or held fixed, as demographic changes may also alter anatomy, physiology, diseases, or presentation.Its guarantees also depend on modeling choices, unknown confounding, and causal-model validity.Source: §6 · p. 9
Limitations of our work
- Empirical findings cover classification tasks and simulated prevalence differences, which may not reflect real-world extremes.The authors also report approximate rather than exact invariance and note sensitivity of fairness gaps and calibration costs to thresholds, compression, or model selection.Source: §6 · p. 9
7 CONCLUSION
The conclusion rejects representation invariance as a universal fairness target when demographics and outcomes are dependent. In base-rate-gap evaluations, marginal and class-conditional invariance show different fairness trade-offs.
7 CONCLUSION
- Representation invariance is incompatible with most popular fairness criteria when A and Y are not independent.Source: §7 · p. 9
- The authors argue for limiting demographic encoding to the smallest level sufficient for accurate, robust, and fair prediction, rather than removing it entirely.Source: §7 · p. 10
A.1 ADDITIONAL DETAILS: FAIRNESS UNDER INCREASING BASE-RATE GAP AND PREVALENCE SHIFT CONDITIONS
- As the base-rate gap grows, marginal alignment’s ∆EO increases, while ∆PPV and ∆bias grow for every method.The figure reports ∆DP, ∆EO, and ∆bias across prevalence conditions; class-conditional invariance holds ∆EO but not ∆bias.Source: Figure 5 · p. 15, §A.1 · p. 15
A.2 ABLATIONS: PROBE ABLATION
The probe ablation compares demographic decodability in marginal and within-class representations against probe bounds and ERM. Its findings differ by the type of invariance enforced.
A.2 ABLATIONS: PROBE ABLATION
- With marginal alignment, empirical probe AUROC lies below the exact-density bound, while the within-class probe stays high or rises above ERM.Source: §A.2 · p. 15, Table 3 · p. 19
- With class-conditional alignment, the within-class probe stays within its class bound, while the marginal probe rises to P2’s base-rate floor.Source: §A.2 · p. 15, Table 3 · p. 19
- The ablation reports probe-family AUCs alongside a certified balanced-accuracy bound and maximum within-class probe AUC.Source: §A.2 · p. 15, Table 3 · p. 19
A.3 ABLATION: STRENGTH OF REPRESENTATION INVARIANCE, HYPERPARAMETERS AND THRESHOLDS
The strength sweep shows that marginal alignment’s outcomes vary with γ, while changing feature sets does not alter the reported fairness pattern. The default threshold can also produce counterintuitive gap reductions in some low-prevalence runs.
A.3 ABLATION: STRENGTH OF REPRESENTATION INVARIANCE, HYPERPARAMETERS AND THRESHOLDS
- As γ increases, DA, ∆DP, and AUROC decrease, but ∆EO does not.Source: §A.3 · p. 19, Table 4 · p. 19
- On Dutch, ∆EO reaches 0.31 at γ = 1 as DA|Y rises from 0.61 to 0.79; for γ ≤0.1, the method is at par with ERM.Source: §A.3 · p. 19, Table 4 · p. 19
- Across feature configurations, ∆DP ≤0.10 while ∆EO stays at par with or above ERM; at default 0.5, low-prevalence Adult, Credit, and COMPAS runs predict few positives, shrinking both gaps counterintuitively.Source: §A.3 · p. 19, Table 4 · p. 19
A.4 ADDITIONAL DETAILS: DATASETS AND PER-SEED RESULTS
The appendix reports per-seed variability, group calibration, score-level alignment results, and details of the CheXpert cohorts and evaluation. These additional analyses qualify the main fairness findings and specify their experimental scope.
A.4 ADDITIONAL DETAILS: DATASETS AND PER-SEED RESULTS
- Table 6 lists dataset groups, prediction targets, training-split base rates across shift regimes, and selected feature sets.Source: §A.4 · p. 19, Table 6 · p. 21
- Training-seed variation is small relative to method differences on all datasets except Communities & Crime, where ∆EO and ∆PPV variation is comparable to the method effect.Communities & Crime has 400 test rows and 38–151 positives in A=0.Source: §A.4 · pp. 19, 21, Table 7 · p. 22
A.5 ADDITIONAL DETAILS: CALIBRATION BY GROUPS
- Under marginal alignment, calibration slopes become overconfident for the low-prevalence group and underconfident for the high-prevalence group, while ∆cal increases across datasets.Class-conditional alignment restores slopes to ERM levels but leaves the intercept gap at or above ERM; prevalence shifts also increase ∆bias for every method.Source: §A.5 · p. 23, Table 8 · p. 23
A.6 ADDITIONAL DETAILS: SCORE-LEVEL ALIGNMENT
- On Dutch at shift-50, score-level marginal alignment gives ∆DP = 0.006 and ∆EO = 0.26, while within-class alignment gives ∆EO = 0.024.Aligning the scalar score costs less utility than aligning Z: AUROC is 0.81 versus 0.71 for the marginal target, with fairness conclusions unchanged.Source: §A.6 · p. 23, Table 9 · p. 24
- Matching class-conditional score means to 0.000 does not deliver separation: ∆EO remains 0.06–0.08 versus ERM’s 0.055, and ∆bias rises to 0.41.The group-specific score-distribution shape beyond the mean remains untouched.Source: §A.6 · p. 23
B.1 ADDITIONAL DETAILS: DATASET DETAILS AND COHORTS
- The CheXpert shift-50 intervention removes half of female-positive images within each partition, and the experiment does not test a new deployment shift.Evaluation audits frozen representations and scores both marginally and within outcome classes, using a common sex-independent decision threshold.Source: §B.1 · p. 25, Table 10 · p. 25
B.2 ADDITIONAL DETAILS: PROBE RESULTS
The probe and calibration analyses show that reducing demographic information does not guarantee demographic erasure or better fairness. Conditional alignment can improve some outcomes while worsening others, and its apparent EO benefit depends on the operating threshold.
B.2 ADDITIONAL DETAILS: PROBE RESULTS
- Conditional representation probe AUROC fell from ≈.978 under ERM to .590 for Edema and .616 for Cardiomegaly, yet none of 18 representations passed the near-chance AUROC criterion.A single probe family is insufficient to assess the representations.Source: §B.2 · p. 25, Figure 11 · p. 27
- Conditional FNF at γ = .95 passed all three test criteria for Edema in 3/3 seeds, but neither task passed the conditional AUROC criterion.The three Edema runs also passed balanced-accuracy and logloss interval checks.Source: §B.2 · p. 25, Table 13 · p. 27
B.3 ADDITIONAL DETAILS: CALIBRATION AND OPERATING-POINT ANALYSIS
- Strong conditional alignment lowered EO gaps, while marginal alignment produced smaller DP gaps.Extreme thresholds can shrink gaps by predicting the same label for everyone.Source: Figure 10 · p. 26, §B.3 · p. 27
- At γ = .95, conditional FNF improves mean AUROC, EO, and worst-group ECE versus marginal FNF on both tasks, but increases DP gaps.The EO comparison depends on the operating point.Source: §B.3 · p. 27, Table 14 · p. 27, Figure 12 · p. 28
- The EO comparison depends on the threshold, and reduced fairness gaps alone do not establish improved predictions.The threshold plot compares ERM and FNF at γ = .95 on the same test cohorts; extreme thresholds can yield smaller gaps by predicting one label for everyone.Source: §B.3 · p. 27, Figure 13 · p. 28
C APPENDIX 3: FAIRNESS METRICS DO NOT ESTABLISH INVARIANCE
CheXpert experiments show that popular erasure methods did not reliably remove demographic signals, and that fairness metrics alone do not establish invariance. Exploratory comparisons also show trade-offs between demographic decodability and predictive utility, with nonlinear probes detecting information missed by linear probes.
C APPENDIX 3: FAIRNESS METRICS DO NOT ESTABLISH INVARIANCE
- Across eight adversary strengths and five seeds, DANN and CDANN did not erase demographic information from CheXpert representations.Fairness gaps sometimes decreased, including to zero, as diagnostic performance also decreased.Source: C APPENDIX 3: FAIRNESS METRICS DO NOT… · p. 29
- Across 51 single-seed Edema configurations, methods varied: MMD and HSIC achieved near-chance decodability with reduced AUROC, while other approaches retained demographic signals or lost utility.The authors caution that this exploratory analysis does not establish general limitations or failure of the evaluated methods.Source: C APPENDIX 3: FAIRNESS METRICS DO NOT… · p. 29, Table 16 · p. 31
C.1 ADDITIONAL DETAILS: CLARIFICATIONS
- Here, A denotes demographic attributes that apply to people; scanner, site, and protocol are acquisition nuisances handled by domain-invariant methods.The authors note that usage and interpretation may differ for nuisance variables.Source: §C.1 · p. 29
- Nonlinear probes recovered demographic information that linear probes missed; the strongest nonlinear sex-probe AUROC was 0.8.Low linear decodability can conceal nonlinear leakage.Source: §C.1 · p. 30
- Chance probe AUROC does not establish independence, and marginal probes alone do not assess conditional invariance.The figure contrasts linear and nonlinear probes and summarizes DANN and CDANN experiments.Source: Figure 14 · p. 30