PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination

Nathan Kallus, Xiaojie Mao, Angela Zhou

arXiv:1906.00285v2stat.MLcs.LGmath.OC

TL;DR

The paper addresses fairness assessment when protected-class membership is unobserved, including in lending and healthcare. It uses auxiliary data and proxy-based models to characterize possible disparities and develops computational and statistical tools for analyzing them, showing that common disparity measures are generally unidentifiable.

  • Problem

    Protected-class membership is often unobserved, hindering fairness auditing when disparity measures depend on that membership.

  • Method

    The paper combines auxiliary datasets and proxy variables to characterize tightest-possible disparity sets, then provides optimization and statistical tools to compute and assess them.

  • Results

    Common disparity measures are generally unidentifiable even with auxiliary data, so the paper characterizes the possible true disparities consistent with observed data and assumptions.

  • Takeaways & Limitations

    The resulting tools support robust disparity assessments by quantifying ambiguity and the value of more informative proxies or assumptions.

  • Takeaways & Limitations

    The approach is limited by the need for auxiliary data where protected-class labels are present.

Abstract

from arXiv · show

The increasing impact of algorithmic decisions on people's lives compels us to scrutinize their fairness and, in particular, the disparate impacts that ostensibly-color-blind algorithms can have on different groups. Examples include credit decisioning, hiring, advertising, criminal justice, personalized medicine, and targeted policymaking, where in some cases legislative or regulatory frameworks for fairness exist and define specific protected classes. In this paper we study a fundamental challenge to assessing disparate impacts in practice: protected class membership is often not observed in the data. This is particularly a problem in lending and healthcare. We consider the use of an auxiliary dataset, such as the US census, to construct models that predict the protected class from proxy variables, such as surname and geolocation. We show that even with such data, a variety of common disparity measures are generally unidentifiable, providing a new perspective on the documented biases of popular proxy-based methods. We provide exact characterizations of the tightest-possible set of all possible true disparities that are consistent with the data (and possibly any assumptions). We further provide optimization-based algorithms for computing and visualizing these sets and statistical tools to assess sampling uncertainty. Together, these enable reliable and robust assessments of disparities -- an important tool when disparity assessment can have far-reaching policy implications. We demonstrate this in two case studies with real data: mortgage lending and personalized medicine dosing.

1. Introduction

The paper addresses fairness assessment when protected-class labels are unobserved, showing that proxy-based disparity estimates are generally unidentifiable. It develops partial-identification tools for principled auditing and demonstrates how informative proxies or assumptions can narrow conclusions.

  • Motivation: Protected-class membership is often unavailable in lending and healthcare, hindering audits of disparate impacts from ostensibly neutral algorithms.Lenders may be unable to collect race and ethnicity information, while healthcare entities often lack these data because of collection costs and disclosure reluctance.
  • Problem: Proxy-based assessments cannot generally identify disparity measures when protected-class labels are unobserved.The paper argues that any single disparity point estimate is fundamentally spurious when the data permit multiple consistent values.
  • Contributions: The paper characterizes the exact partial identification set: all disparity values consistent with the observed datasets and data-generating processes.The characterization is sharp, meaning the set equals the feasible values rather than merely containing them.
  • Implications: Small partial identification sets can support conclusions about disparity signs and magnitudes, whereas large sets warn against substantive conclusions.Set size captures ambiguity; stronger proxies or additional assumptions can produce smaller, more informative sets.
  • Contributions: Optimization-based algorithms and inference procedures compute these sets and account for sampling uncertainty from finite observations.The resulting procedures provide consistency guarantees and support computation of tight disparity bounds.
  • Empirical analysis: The authors apply the approach to real-world analyses of racial lending disparities and disparate impacts of prescriptive algorithms.The contribution passages identify empirical applications in lending and other consequential decision-making settings.

2. Problem Setup

The paper frames disparity auditing when protected attributes are unobserved by combining a main dataset containing decisions, outcomes, and proxies with an auxiliary dataset containing protected attributes and proxies. It defines several disparity measures and assumes the datasets are independent samples from marginal distributions of a common joint distribution.

  • Variables: Protected attributes A, such as race or gender, are unobserved labels whose disparities are assessed using observed proxy variables Z.In the BISG example, Z comprises surname and geolocation; proxies may be categorical, continuous, or mixed.
  • Outcomes: The framework primarily studies binary true and decision outcomes, with results stated to extend straightforwardly to multi-leveled outcomes.The notation also defines conditional outcome probabilities used to express the focal disparity measures.
  • Data combination: The main dataset records (Ŷ,Y,Z), while the auxiliary dataset records (A,Z), so the two datasets provide different marginal views of a common distribution.The setup treats the datasets as separate, unconnected samples rather than directly matched individual records.
  • Data combination: No unique identifier shared across datasets prevents directly joining observations to learn the full joint distribution P(A,Ŷ,Y,Z).The paper therefore formulates proxy-based disparity assessment as a data-combination problem.
  • Disparity measures: The paper audits demographic disparity and classification disparities, focusing on DD, TPRD, and TNRD while extending results to predictive-value and error-rate measures.DD compares approval rates; TPRD and TNRD compare correct positive and negative decisions conditional on true outcomes.
  • Disparity measures: Demographic disparity is δDD(a,b)=P(Ŷ=1|A=a)−P(Ŷ=1|A=b), while TPRD and TNRD condition decisions on Y=1 and Y=0, respectively.The corresponding predictive-value disparities are defined by swapping the roles of Y and Ŷ.

3. Related Literature

The related literature spans proxy-based fairness auditing, partial identification, and data combination. The paper distinguishes its contribution by studying nonlinear disparity functionals under partial identification and integrating conditional information across proxy values.

  • Proxy methods: Prior proxy-based disparity methods analyze protected-class probabilities and procedures, but imputed-race estimates are sensitive to tuning choices such as the imputation threshold.Existing evaluations often emphasize proxy classification accuracy without considering downstream disparity-assessment impacts.
  • Proxy methods: When protected class is unobserved, disparity is generally unidentifiable from proxies, so prior point estimators are generally biased absent very strong assumptions.This reframes documented proxy-method bias as an identification problem rather than only a prediction-accuracy problem.
  • Algorithmic fairness: Fairness research has proposed demographic and classification parity measures, while this paper focuses on auditing rather than adjusting disparities.Whether observed disparities warrant adjustment depends on the legal, ethical, and regulatory context.
  • Partial identification: Partial-identification research addresses parameters that cannot be uniquely recovered because of issues including confounding, missingness, multiple equilibria, or data combination.Data combination, also called the ecological inference problem, reconstructs joint distributions from observed marginal distributions.
  • Partial identification: Fréchet-Hoeffding inequalities provide sharp bounds on joint distributions and certain expectations given marginal information.Related tools are used in settings such as financial risk analysis without knowledge of copulas.
  • Paper distinction: Unlike much prior work, this paper studies nonlinear functionals such as true- and negative-rate disparities and uses conditional information across proxy values with possible smoothness constraints.This extends data-combination ideas toward fairness quantities that depend on partially identified distributions.

4. Unidentifiability of Disparity Measures Under Data Combination

With protected class unobserved, the separate marginals from data combination generally do not determine the full joint distribution or disparity measures. The paper formalizes this ambiguity and characterizes the conditions under which disparities are identifiable or remain set-valued.

  • Identification problem: The marginals P(Ŷ,Y,Z) and P(A,Z) are insufficient to uniquely determine the full joint distribution P(A,Ŷ,Y,Z), even with unlimited data.Identification concerns what the data distribution uniquely determines; infinite observations cannot resolve ambiguity that remains in the distribution itself.
  • Interpretation: Disparity point estimates can be spurious, biased, and sensitive to ad-hoc modeling specifications, so substantive conclusions based on them require caution.This limitation persists even when the observed datasets are known without sampling error.
  • Identification problem: With binary protected class and outcomes, three independent marginal constraints leave four joint-distribution unknowns, so the joint is generally not unique.The same nonuniqueness extends from P(A|Z) and P(Ŷ|Z) to P(A|Z) and P(Ŷ,Y|Z).
  • Disparity measures: When proxies cannot perfectly predict protected class, demographic disparity is unidentifiable without further conditions, and classification disparities TPRD and TNRD are likewise unidentifiable.The result states that multiple full joint distributions can agree with the marginals while producing different disparity values.
  • Partial identification: The partial identification set contains all disparity values generated by valid joint distributions consistent with the observed marginals and any imposed assumptions.Its size captures ambiguity: large sets warn against substantive conclusions, while sets separated from zero can still indicate disparities.

5. Closed-form Partial Identification Sets of Disparities for Binary Protected Class Attribute

For a binary protected class, the paper rewrites disparity measures using conditional protected-class probabilities and derives closed-form partial identification sets. Demographic, true-positive-rate, and true-negative-rate disparities can therefore be represented by computable intervals under the basic data-combination constraints.

  • Weighted representations: The paper expresses demographic and classification disparities through weighted representations involving conditional protected-class probabilities given proxies, predictions, and outcomes.The identifiable components are expectations over P(Ŷ,Y,Z), while the weights depending on the unobserved full joint distribution remain unidentified.
  • Feasible weights: The feasible weight sets are characterized by law-of-total-probability constraints linking conditional protected-class probabilities to the observed marginals.These constraints define W(PD) and its outcome-conditional analogue Ẇ(PD).
  • Closed-form sets: Without extra assumptions, DD, TPRD, and TNRD have closed-form partial identification sets for a binary protected class.The demographic disparity set is an interval; analogous interval formulas are provided for TPRD and TNRD.
  • Interpretation: Every value inside the demographic-disparity interval is compatible with the observed data, so the interval directly represents the range of feasible disparity values.The endpoints are obtained from extremal feasible weights.
  • Interpretation: Perfectly predictive proxies collapse the interval endpoints to a singleton, but this case is described as unrealistic.For classification disparities, the intervals similarly collapse when proxies perfectly predict either outcomes or protected class.

6. Extensions for General Partial Identification Sets

The framework extends partial identification beyond the basic binary setting by incorporating structural assumptions, multiple protected-class levels, and multivariate disparity sets. Smoothness restrictions can narrow feasible sets, while support functions and optimization provide a general route to computation and visualization.

  • Structural assumptions: Additional structural assumptions, including smoothness restrictions, can restrict the set of joint distributions compatible with the separate datasets.The paper uses metric-based bounds on how conditional protected-class probabilities vary across proxy values.
  • Structural assumptions: Smoothness assumptions may narrow disparity sets when defensible from domain knowledge, although their validity cannot be verified from the separate datasets alone.The paper notes that similarity of true joint distributions across similar proxy values is an additional assumption.
  • General optimization: For multiple protected-class levels, the paper computes partial identification sets by solving optimization problems over feasible joint distributions and weights.This general formulation accommodates simultaneously achievable disparities across groups and disparity measures.
  • Multivariate sets: Support functions characterize multivariate partial identification sets by reporting maximal and minimal contrasts and exactly characterizing their convex hull.The support-function approach also supports computation, estimation, and visualization from data.
  • Multivariate sets: For non-binary protected classes, DD sets are convex under the stated weight conditions, whereas TPRD and TNRD sets are generally nonconvex.Taking the convex hull of the latter gives the smallest convex outer approximation.

7. Implementation, Estimation, and Inference

The paper turns partial identification sets into estimable objects using nuisance-probability models, augmented estimators, and cross-fitting. It establishes consistency, asymptotic normality, and confidence intervals under stated sampling and regularity conditions.

  • Implementation and setup: The partial identification sets are population objects reflecting ambiguity from marginal information, while estimation must account for finite-sample variability.The paper develops confidence intervals to characterize this additional uncertainty.
  • Implementation and setup: The required conditional probabilities are estimated from primary and auxiliary datasets using supervised-learning or probabilistic-classification models.Examples include logistic and multinomial logistic regression, with random forests and neural networks as alternatives.
  • General partial identification sets: For general protected classes, the paper estimates a support function with a linear program and uses its outer approximation because it is guaranteed to contain the true partial identification set.The binary-class case has interval bounds with closed-form endpoints.
  • Estimation: Augmentation terms debias partial-identification-bound estimators so nuisance-estimation errors have negligible effect, while cross-fitting prevents evaluation-data overfitting.The resulting estimators are asymptotically equivalent to infeasible estimators that use the true nuisance parameters.
  • Inference: The estimated endpoints are asymptotically normal with closed-form variance, enabling confidence intervals and requiring only slow nuisance-estimator convergence under the stated conditions.The method is largely invariant to how nuisance functions are fitted.

8. Case Studies

Two case studies apply the framework to mortgage lending and personalized Warfarin dosing using proxy variables for protected class membership. The results show that proxy informativeness and added assumptions can narrow disparity sets, but substantial ambiguity often remains.

  • Mortgage credit decisioning: Outcome proxies are similarly uninformative, while geolocation is more informative about race than income and combining them adds very little.The outcome entropy without proxies is around 0.5.
  • Mortgage credit decisioning: All mortgage partial-identification intervals are fairly wide, contain the self-reported-race ground truth, and cannot reject zero disparity at the 5% significance level.Finite-sample uncertainty is described as quite small despite the wide identification intervals.
  • Mortgage credit decisioning: Without smoothness constraints, income alone yields the largest mortgage identification set, while adding income to geolocation barely shrinks the set.Smoothness narrows the income-only set and supports a higher-or-roughly-equal White approval rate relative to Black applicants, but other conclusions remain ambiguous.
  • Personalized Warfarin dosing: In Warfarin dosing, genetic and medication proxies together narrow Black-versus-Rest and White-versus-Rest TPRD bounds, with Black-versus-Rest TPRD positive at 95% confidence.The Asian-versus-other-races TPRD is clearly negative, while White-versus-Asian TPRD is positive and White-versus-Black disparity remains unclear.
  • Personalized Warfarin dosing: The case studies indicate that stronger or combined proxies can clarify disparity direction and, in some cases, its magnitude.The genetic proxy is stronger than the medication proxy in the Warfarin analysis.

9. Conclusion

Fairness assessment is difficult when protected-group membership is unobserved, even though disparate impacts can trigger scrutiny and sanctions. The paper frames proxy-based assessment as a data-combination problem and develops tools to characterize what disparity information can be learned reliably.

  • Even algorithms that omit sensitive inputs can still produce biased outcomes, while the meaning of unfairness depends on context.
  • Protected-group membership is usually absent from decision data, requiring auxiliary data with protected labels for disparity assessment.
  • The paper formulates proxy-based disparity assessment as a data-combination problem and studies its fundamental identification limits.
  • The proposed tools compute the tightest-possible bounds on disparities consistent with the observed data.
  • Partial-identification ambiguity clarifies which fairness quantities can and cannot be learned, supporting more careful interpretation of disparity assessments.

A.1. Failure of Naive Plug-in Estimator

Without augmentation, estimating partial-identification bounds can leave nuisance-estimation errors in the downstream estimators, complicating their convergence and asymptotic distributions.

  • A.1. Failure of Naive Plug-in Estimator: The analysis establishes that adding augmentation terms is necessary to ensure the desired convergence rate for bound estimators based on estimated nuisance quantities.The section contrasts this with the naive plug-in approach, whose remainder term is not automatically negligible.
  • A.1. Failure of Naive Plug-in Estimator: Without augmentation terms, the estimator's asymptotic distribution depends on a remainder term in addition to its main term.The main term is asymptotically normal, but the remainder term can affect the asymptotic distribution.
  • A.1. Failure of Naive Plug-in Estimator: Extra augmentation terms make nuisance-estimation errors have negligible impact on estimators of the lower and upper partial-identification bounds.The resulting asymptotic distributions match those obtained using the true nuisance parameters directly.
  • A.1. Failure of Naive Plug-in Estimator: With augmentation, the final estimators retain n^-1/2 convergence and have well-behaved asymptotic distributions.The unaugmented analysis instead indicates slower convergence rates under the relevant conditions.

A.2. Classification Disparity with Binary Protected Class

The paper constructs estimators and confidence intervals for closed-form partial-identification sets in binary-protected-class disparity measures, including TPRD and TNRD.

  • A.2. Classification Disparity with Binary Protected Class: The section develops estimators for closed-form partial-identification sets using estimated nuisance quantities and proxy-related bounds.The plug-in estimators are formed from lower- and upper-bound nuisance estimates.
  • A.2. Classification Disparity with Binary Protected Class: The bound estimators are asymptotically normal with closed-form asymptotic variance under stated nuisance-estimation conditions.The result is given for the upper and lower bound estimators defined in the preceding equations.
  • A.2. Classification Disparity with Binary Protected Class: Consistent variance estimators support confidence intervals for the partial-identification sets of TPRD and TNRD.Analogous results extend to PPVD and NPVD by exchanging the roles of the predicted and true outcomes.
  • A.2. Classification Disparity with Binary Protected Class: The confidence interval for TPRD and TNRD asymptotically covers the corresponding partial-identification sets with probability at least 1−β.The stated coverage guarantees are expressed through liminf probabilities as n approaches infinity.

A.3. Asymptotic Guarantee without Cross-fitting

Cross-fitting yields asymptotic guarantees for partial-identification bound estimators under high-level nuisance-rate conditions, whereas avoiding it generally requires stronger empirical-process assumptions.

  • A.3. Asymptotic Guarantee without Cross-fitting: Cross-fitting enforces independence between nuisance estimators and the data on which they are evaluated.This permits conditional error control in the asymptotic analysis.
  • A.3. Asymptotic Guarantee without Cross-fitting: With cross-fitting, nuisance-estimation errors have negligible impact on the final asymptotic distribution under high-level convergence-rate conditions.The paper focuses on cross-fitted estimators for generality and simplicity.
  • A.3. Asymptotic Guarantee without Cross-fitting: Without cross-fitting, the analysis generally requires uniform convergence over a sufficiently simple Donsker function class.Establishing this convergence and bounding the remaining error terms involves complicated chaining arguments.
  • A.3. Asymptotic Guarantee without Cross-fitting: Donsker conditions can hold for smooth parametric, finite-dimensional, and VC classes, including nuisance estimators used in the paper's case-study sections.For those simple estimators, cross-fitting is not needed for the closed-form bounds in Sections 8.1 and 8.2.
  • A.3. Asymptotic Guarantee without Cross-fitting: The bound estimators for DD are asymptotically normal, and their variances can be consistently estimated for asymptotically valid confidence intervals.Related results cover TPRD and TNRD, while covariance-based calibrated intervals can attain the confidence level asymptotically.
  • A.3. Asymptotic Guarantee without Cross-fitting: Covariance-adjusted confidence intervals have asymptotic coverage probability exactly equal to the confidence level.The construction uses the covariance between upper- and lower-bound estimators.

A.5. Inference with Known Conditional Probabilities of Protected Class

This section develops inference for demographic-disparity bounds when protected-class probabilities conditional on proxies are treated as known. It establishes asymptotic normality and variance-based confidence intervals under stated conditions.

  • Assumptions: The analysis assumes two datasets and excludes settings where they may share overlapping units.Unknown overlapping structure is beyond the paper’s scope.
  • Assumptions: The auxiliary dataset is assumed to reveal the population distribution, making P(A = α | Z = z) known without finite-sample variability.Here, A denotes protected class and Z denotes proxy variables.
  • Inference: The proposed lower- and upper-bound estimators for binary protected-class demographic disparity are asymptotically normal as n_pri →∞.The result follows under the conditions stated in Theorem 6.
  • Inference: Consistent plug-in variance estimators support confidence intervals for the disparity bounds.The variance estimators are consistent under Theorem 6’s conditions and are used for the intervals described in Theorem 2.

B.1. Proof of Proposition 2

The proof shows that demographic, true-positive-rate, and true-negative-rate disparities are generally not identified from the observed marginals and protected-class probabilities conditional on proxies. It constructs multiple compatible joint distributions that induce different disparity values.

  • Demographic disparity: The observed conditional marginals permit multiple joint distributions for protected class and predicted outcomes.The proof uses Fréchet–Hoeffding endpoints and constructs valid couplings consistent with the given marginals.
  • Demographic disparity: Demographic disparity differs across two valid joint distributions that share the same observed marginals.One construction satisfies conditional independence, while another perturbs a protected-class/outcome joint probability by ϵ on a positive-measure proxy subset.
  • Rate disparities: True-positive-rate disparity is likewise unidentifiable because compatible distributions can alter protected-class assignments within outcome strata.The proof constructs two distributions with different joint probabilities involving A, predicted outcome, and Y = 1.
  • Rate disparities: The true-negative-rate result follows analogously from the true-positive-rate argument.The section explicitly states that the conclusion for true-negative-rate disparity can be proved analogously.
  • Partial identification: For rate disparities, the feasible weighting functions form a compact connected set, and the identified set is an interval obtained by minimizing and maximizing the disparity functional.The proof expresses the interval through optimization over admissible weights.

B.6. Proof for Theorem 1, Theorem 2

These proofs establish the large-sample behavior of cross-fitting estimators for the lower and upper disparity bounds. The estimators are asymptotically equivalent to infeasible estimators using the true nuisance quantities, enabling asymptotic inference and confidence intervals.

  • Theorem 1: Cross-fitting decomposes estimator errors into terms controlled using conditional expectations, moment bounds, Markov’s inequality, and a maximal-inequality lemma.The proof treats nuisance estimates as fixed outside each fold and bounds the resulting remainder terms.
  • Theorem 1: The estimated lower and upper disparity bounds have asymptotic distributions obtained after showing nuisance-estimation remainders are negligible.The lower-bound and upper-bound derivations are presented separately but use analogous arguments.
  • Theorem 1: The proposed lower-bound estimator is asymptotically equivalent to an infeasible estimator that substitutes the true nuisance function η.This representation yields the asymptotic distribution of the estimator.
  • Theorem 2: Plug-in variance estimators converge to the corresponding variances, supporting confidence intervals for the disparity bounds.The proof uses laws of large numbers and Slutsky’s theorem to establish consistency and interval coverage.
  • Theorem 2: With probability at least 1 −β, the resulting confidence intervals contain the relevant bound quantities asymptotically.The stated probability guarantee is obtained through Slutsky’s theorem.

B.7. Proof of Theorems 3 and 4

The proofs establish consistency for optimization-based estimators of disparity bounds by controlling perturbations in estimated constraints and discretized optimization parameters. Linear-system stability supplies the key approximation guarantees under regularity conditions.

  • Theorems 3–4: Theorem 3’s consistency proof relies on a key property of the optimization problem and controls approximation errors from nuisance estimates and discretization.The proof combines approximation-error lemmas with the triangle inequality.
  • Theorem 4: The algorithm approximates the optimization over t by grid-searching an ϵ-net, with feasibility handled through the parametrized linear program.The discretization uses a uniformly covering grid and componentwise inverses of the grid parameters.
  • Stability analysis: Robinson’s linear-system stability theorem bounds how perturbations in constraint coefficients and right-hand sides affect feasible solution sets.The analysis homogenizes the linear system and assumes a Slater point so the stability result applies.
  • Perturbed program: The perturbed optimization system incorporates empirical primary- and auxiliary-dataset probabilities into its equality constraints.The perturbation matrix captures deviations between empirical and population conditional probabilities.
  • Inference: Under regularity conditions, linear-program estimates for demographic disparity are asymptotically normal, and the sample optimal value is asymptotically normal.The section also identifies conditions related to primal and dual solution behavior and variance estimation.
Loading 1906.00285v2…