Source-linked AI summary
Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, Madeleine Udell
TL;DR
The paper studies how to assess outcome disparities when protected-class labels are unavailable and must be imputed from observable proxies. It decomposes thresholded-estimator bias, proposes a soft-classification weighted estimator, and evaluates both approaches theoretically and with mortgage data. Thresholded estimates are generally upward-biased and threshold-sensitive, while the weighted estimator tends toward simpler negative bias.
Problem
Protected-class labels may be unavailable for fairness assessments, while proxy-based imputation has been observed to exaggerate disparities without a known explanation.
Method
The paper derives bias decompositions for thresholded imputation, develops a weighted estimator using soft proxy classification, and evaluates the methods with theory, simulations, and mortgage data.
Results
Thresholded-estimator bias is generally upward and strongly threshold-sensitive, whereas the weighted estimator has one covariance-based bias source and tends toward overall underestimation.
Takeaways & Limitations
The weighted estimator may be useful for disparity evaluations when proxy models are used, particularly when external knowledge can determine the bias direction.
Takeaways & Limitations
The weighted estimator is asymptotically unbiased only when outcome and protected-class membership are conditionally independent given the proxy.
Abstract
from arXiv · showhide
Assessing the fairness of a decision making system with respect to a protected class, such as gender or race, is challenging when class membership labels are unavailable. Probabilistic models for predicting the protected class based on observable proxies, such as surname and geolocation for race, are sometimes used to impute these missing labels for compliance assessments. Empirically, these methods are observed to exaggerate disparities, but the reason why is unknown. In this paper, we decompose the biases in estimating outcome disparity via threshold-based imputation into multiple interpretable bias sources, allowing us to explain when over- or underestimation occurs. We also propose an alternative weighted estimator that uses soft classification, and show that its bias arises simply from the conditional covariance of the outcome with the true class membership. Finally, we illustrate our results with numerical simulations and a public dataset of mortgage applications, using geolocation as a proxy for race. We confirm that the bias of threshold-based imputation is generally upward, but its magnitude varies strongly with the threshold chosen. Our new weighted estimator tends to have a negative bias that is much simpler to analyze and reason about.
1 INTRODUCTION
Fairness assessment can require imputing protected-class membership when labels are unavailable, creating a need to understand bias in proxy-based disparity estimates. This paper theoretically analyzes thresholded imputation, proposes a weighted alternative, and studies these methods in mortgage data.
- Motivation: High-stakes decisions, including credit, must demonstrate compliance with laws prohibiting discrimination against protected classes.Examples include employment, admissions, sentencing, bail, and credit decisioning.
- Unknown protected-class membership: Fairness assessments may lack protected-class labels because collecting them is illegal or operationally difficult, so observed proxies are used for imputation.The imputed labels support disparate-impact assessments but are not permitted for decision making.
- Contributions: The paper derives bias for thresholded estimators and decomposes it into interpretable sources that explain when outcome disparity is over- or underestimated.Thresholded estimators assign labels only when predicted class probability exceeds a predefined threshold.
- Contributions: The paper proposes a weighted estimator that uses soft proxy outputs instead of hard class imputation and derives its bias.The weighted estimator has only one bias source according to the paper’s theoretical analysis.
- Empirical validation: Using geolocation as the sole race proxy in a public mortgage dataset, the paper identifies a bias source that can account for thresholded-estimator overestimation and finds threshold sensitivity.This analysis is intended to explain overestimation observed in earlier proxy-based studies.
2 EVALUATING THE FAIRNESS OF A BINARY DECISION
The paper defines demographic disparity for binary decisions and examines how proxy-based protected-class imputation estimates it when true memberships are unavailable. It contrasts hard thresholding, which can misclassify or exclude individuals, with a weighted estimator that uses soft classifications.
- Fairness measures: Demographic disparity δ is the difference between favorable-outcome rates for advantaged group a and disadvantaged group b.A positive δ means the advantaged group has a higher favorable-outcome rate.
- Proxy-based assessment: When protected-class labels are unavailable, proxy variables Z such as surname and geolocation provide estimated membership probabilities for groups a and b.The paper presents the binary case A ∈ {a,b} and notes that results generalize pairwise to multiclass protected attributes.
- Thresholded estimator: The thresholded estimator assigns a single predicted label when proxy probability exceeds a threshold, while excluding unclassified observations from disparity evaluation.This hard rule can misclassify protected-class membership because probabilistic proxy predictions are intrinsically uncertain.
- Weighted estimator: The weighted estimator instead incorporates the soft classification probabilities to address misclassification and exclusion caused by hard thresholding.Its mean-group and disparity estimates are defined using probability weights rather than only assigned labels.
- Toy examples: In Example 2.6, the true demographic disparity is 40%, whereas thresholding at q = 0.5 estimates 100% under an income-based lending policy.The overestimate occurs because geolocation is correlated with socioeconomic status and therefore with loan approval, making misclassification outcome-dependent.
- Toy examples: In Example 2.7, the true disparity is 10%, while thresholding at q = 0.5 estimates 46% despite affirmative action favoring the disadvantaged group at each income level.The examples show that systematic, outcome-related misclassification can arise from both inter-geolocation and intra-geolocation outcome variation.
3 BIAS IN THRESHOLDED AND WEIGHTED ESTIMATORS
The paper derives asymptotic bias expressions for thresholded and weighted demographic-disparity estimators when protected-class labels are imputed from proxy probabilities. Thresholded bias has multiple interpretable sources and can over- or underestimate disparity, while weighted bias has a simpler conditional-outcome dependence.
- 3.1 Weighted estimator: The weighted estimator’s asymptotic bias is characterized through the biases of its estimated mean outcomes for protected groups.Its bias is analyzed for binary protected classes and converges almost surely as sample size grows.
- 3.1 Weighted estimator: If Y is conditionally independent of A given Z, the weighted estimator is asymptotically unbiased.This condition can hold when Y is determined by features included in Z, including machine-learning outputs based on those features.
- 3.1 Weighted estimator: When conditional independence fails, weighted-estimator bias depends on within-Z covariance between outcomes and true class membership, producing either overestimation or underestimation.With geolocation as a race proxy, disadvantaged-group outcomes higher within locations imply overestimation, whereas advantaged-group outcomes higher within locations imply underestimation.
- 3.2 Thresholded estimator: The thresholded estimator’s bias decomposes into multiple terms that compare outcome differences across protected groups and across proxy-probability ranges.The Δ1 terms capture between-group outcome discrepancies within proxy ranges, while Δ2 terms capture within-group discrepancies across proxy ranges.
- 3.2 Thresholded estimator: Under sufficient inequalities involving these terms, the thresholded estimator overestimates disparity; reversed inequalities yield underestimation.In the overestimation case, the advantaged-group mean is estimated above its true mean and the disadvantaged-group mean below it; the converse holds for underestimation.
- 3.2 Thresholded estimator: With a high threshold q, unclassified observations make proxy-class proportions smaller than true class proportions, so terms tied to outcome–proxy-probability dependence usually contribute more to overestimation.The paper notes that threshold choice changes which bias sources dominate, and that overestimation can persist even when the intra-proxy-range conditions fail.
4 NUMERICAL RESULTS
Numerical simulations and HMDA mortgage data show that thresholded and weighted disparity estimators exhibit different, interpretable biases when geolocation proxies race. Thresholded estimates generally overstate disparity and are threshold-sensitive, while weighted estimates tend toward underestimation.
- Synthetic experiments: The simulations use geolocation-based race probabilities across three neighborhoods, with neighborhood populations of 3,000, 4,000, and 5,000, repeated 30 times.Average estimates from thresholded, weighted, and true-race calculations are compared.
- Synthetic experiments: In Experiment 4.1.1, income differences between groups drive the thresholded estimator’s bias through the Δ1 terms, while conditional outcome-group covariance can reverse the weighted estimator’s bias.When disadvantaged-group incomes are relatively higher, Δ1 terms counteract overestimation and the weighted estimator becomes negatively biased.
- Synthetic experiments: In Experiment 4.1.2, variation in the Δ2 terms explains the thresholded estimator’s changing overestimation, whereas the weighted estimator is unbiased when income is race-independent given geolocation.Increasing λ raises the relevant Δ2 contributions and thresholded overestimation.
- HMDA data: The HMDA dataset contains around 17 million 2011–2012 mortgage applications with geolocation, race, and loan-origination outcomes.The analysis uses geolocation as a probabilistic race proxy and defines approval or origination as Y = 1.
- HMDA results: In HMDA, thresholding underestimates Black and Hispanic acceptance rates, accurately estimates White acceptance, and therefore overestimates demographic disparity; weighted estimation tends to underestimate disparity.Thresholded overestimation generally decreases as q decreases.
- Bias interpretation: Thresholded bias combines inter-geolocation and intra-geolocation variation, while weighted bias is determined solely by intra-geolocation variation and is easier to analyze.At high thresholds, inter-geolocation variation dominates; thresholded bias can therefore vary strongly with q.
5 CONCLUSIONS
The paper analyzes bias in thresholded proxy-based disparity estimates and proposes a weighted estimator that propagates proxy uncertainty. Thresholded-estimator bias depends strongly on the threshold and multiple interacting sources, whereas weighted-estimator bias has one simpler source and tends to underestimate disparity.
- The paper presents the first theoretical analysis of bias when probabilistic proxy models impute unobserved protected-class membership.
- Thresholded-estimator bias reflects multiple interacting sources, producing strong sensitivity to the chosen threshold.
- The weighted estimator propagates proxy uncertainty to the final estimand and has one bias source tied to approval differences between races within the same geolocation.
- The weighted estimator’s bias led to an overall underestimate and is simpler to analyze than thresholded-estimator bias.
Appendix A PROOFS FOR SECTION 3
The appendix proves convergence and decomposes thresholded-estimator bias into interpretable probability terms. It also connects weighted-estimator bias to conditional covariance and shows that conditional independence removes this covariance.
- The proof of Theorem 3.1 uses the strong law of large numbers to establish almost sure convergence of the weighted estimator.
- The proof regroups terms in the weighted-estimator bias to identify a conditional covariance involving the true class indicator and outcome.
- When the outcome is conditionally independent of the protected class given Z, the relevant conditional covariances are zero.
- Theorem 3.3 decomposes thresholded-estimator bias through three terms, C1(u), C2(u), and C3(u), defined using classification and protected-class probabilities.
- For each class u, C1(u) and C2(u) differ according to the gap between posterior class probability and thresholded-label accuracy.
Appendix B MULTILEVEL UNKNOWN PROTECTED CLASS
The appendix extends the bias analysis to protected classes with more than two values. In the multiclass case, additional class values can make the condition C2(u) > C1(u) hold even when the estimated class is more prevalent than the true class.
- The multiclass setting allows the protected class A to take more than two values and defines A′ as all values other than a and b.
- If the posterior probability of u exceeds that of its opposite class, then C2(u) exceeds C3(u).
- Under both stated conditions, C2(u) exceeds C1(u) and C3(u).
- In the multiclass case, C2(u) > C1(u) can hold even when P(A = u) < P(Â = u), as illustrated for the White group in Figure 5.
C.1 Bias According to Theorem 3.3
Figure 7 compares observed thresholded-estimator bias with the theoretical bias from Theorem 3.3. The theoretical formula closely approximates observed bias in the multiclass setting, indicating that the theorem captures the main bias sources there.
- Theoretical bias from Theorem 3.3 approximates observed thresholded-estimator bias for a multiclass protected class very well.
- The comparison indicates that Theorem 3.3 captures the main bias sources of thresholded estimators in both binary and multiclass settings.
C.2 Results for API
For API, thresholded estimation can either overestimate or underestimate demographic disparity, whereas the weighted estimator produces only a slight underestimation. The difference is linked to API’s distinct geolocation and socioeconomic-status distributions.
- Thresholded estimation bias differs substantially between demographic-disparity estimates for White and API.
- The thresholded estimator can overestimate or underestimate demographic disparity for API.
- The weighted estimator only very slightly underestimates demographic disparity for API.
- API’s distinct geolocation and socioeconomic-status distributions help explain the differing estimation biases.
C.3 Correlation between the race probability and socioeconomic status
Race-proxy probabilities correlate with socioeconomic indicators across census tracts. Tracts with higher White probability tend toward higher income and loan acceptance, while those with higher Hispanic or Black probability tend toward lower values.
- Figure 10 compares census-tract population distributions by race-proxy probability, mean yearly income, and average loan acceptance rate.
- Higher White probability is associated with greater mass at higher yearly incomes and higher loan acceptance rates.
- Higher Hispanic or Black probability is associated with greater mass at lower yearly incomes and lower loan acceptance rates.
- The figure validates that geolocation encodes socioeconomic-status disparities alongside race-proxy information.
C.4 An example of unbiased weighted estimator
A semi-synthetic mortgage dataset constructs loan outcomes from discrete income while using the same income measure to predict race. Under this construction, the weighted estimator is unbiased because race is conditionally independent of the outcome given income.
- The semi-synthetic dataset aggregates applicants’ yearly income into deciles and uses discrete income as the race predictor.
- Loan approval outcomes are simulated from a logistic function of discrete income, so the outcome depends only on income.
- The weighted estimator is unbiased because race is independent of the loan outcome conditional on income by construction.