PaperlayerView exact revision on arXiv

Source-linked AI summary

FADE: FAir Double Ensemble Learning for Observable and Counterfactual Outcomes

Alan Mishler, Edward Kennedy

arXiv:2109.00173v1stat.MLcs.LG

TL;DR

Fairness methods may characterize tradeoffs without helping users improve an existing benchmark or address counterfactual criteria. FADE combines predictors to target multiple observable or counterfactual fairness criteria and explore tradeoffs. The paper gives fast-rate guarantees and reports that multiple disparities can sometimes be minimized with little accuracy loss.

  • Problem

    Existing tradeoff methods often do not accommodate improving an existing benchmark model and typically address observable rather than counterfactual criteria.

    Source: §1 · p. 2

  • Method

    FADE forms weighted combinations of previously trained and newly trained predictors to target multiple fairness criteria or efficiently explore fairness–accuracy tradeoffs.

    Source: §1 · p. 3

  • Results

    Excess risk and unfairness converge to zero at rates up to √n, even when relevant nuisance parameters are estimated at slower rates.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Multiple unfairness measures can sometimes be minimized simultaneously with little impact on accuracy relative to unconstrained predictors or existing benchmarks.

    Source: §9 · p. 36

    The paper studies a regime with basis dimension k < n and generally assumes the basis is eventually fixed; analysis of k ≥ n or growing k is left for future work.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Methods for building fair predictors often involve tradeoffs between fairness and accuracy and between different fairness criteria, but the nature of these tradeoffs varies. Recent work seeks to characterize these tradeoffs in specific problem settings, but these methods often do not accommodate users who wish to improve the fairness of an existing benchmark model without sacrificing accuracy, or vice versa. These results are also typically restricted to observable accuracy and fairness criteria. We develop a flexible framework for fair ensemble learning that allows users to efficiently explore the fairness-accuracy space or to improve the fairness or accuracy of a benchmark model. Our framework can simultaneously target multiple observable or counterfactual fairness criteria, and it enables users to combine a large number of previously trained and newly trained predictors. We provide theoretical guarantees that our estimators converge at fast rates. We apply our method on both simulated and real data, with respect to both observable and counterfactual accuracy and fairness criteria. We show that, surprisingly, multiple unfairness measures can sometimes be minimized simultaneously with little impact on accuracy, relative to unconstrained predictors or existing benchmark models.

1 Introduction

Fair predictors face fairness–accuracy and fairness–fairness tradeoffs, while existing tradeoff methods are less suited to improving a benchmark model or targeting counterfactual criteria. FADE combines user-chosen predictors to target constraints or efficiently explore those tradeoffs, with fast-rate guarantees.

Motivation

  • Existing tradeoff methods do not readily support improving an existing benchmark model and generally address observable rather than counterfactual criteria.Counterfactual criteria depend on counterfactual outcomes, and few methods target counterfactual versions of common criteria such as equalized odds.Source: §1 · p. 2

FADE

  • FADE combines user-chosen basis functions into weighted predictors and offers constrained risk minimization, constrained unfairness minimization, and unfairness-penalized model generation.The penalized method’s weights vary smoothly with penalty parameters, enabling fast exploration of fairness–accuracy paths.Source: §1 · pp. 2–3
  • FADE can target multiple observable or counterfactual criteria, combine previously and newly trained predictors, and explore fairness–accuracy or fairness–fairness tradeoffs.This flexibility also supports improving an existing model’s fairness without sacrificing accuracy, or vice versa.Source: §1 · p. 3

Theory and applications

  • Excess risk and unfairness converge to zero at rates up to √n even when relevant nuisance parameters are estimated at slower, typical nonparametric-machine-learning rates.The paper also reports applications to simulated and real data, including substantial counterfactual improvements for COMPAS and competitive observable results on income prediction.Source: §1 · p. 3

2 Background and related work

Fairness methods differ in how they alter predictors and which outcomes their criteria use. FADE combines existing and newly trained predictors, covers observable and counterfactual criteria, and provides closed-form methods with theoretical guarantees for exploring tradeoffs.

2.1 Ways of achieving fairness

  • Fairness approaches include pre-processing inputs, enforcing constraints or regularization during training, and post-processing existing predictions.FADE combines previously existing predictors with newly trained predictors or other basis functions, blurring the distinction between in-processing and post-processing.Source: §2.1 · p. 4

2.2 Observational and counterfactual fairness

  • Outcome-dependent criteria such as equalized odds can use observable or counterfactual outcomes, while counterfactual risk assessment asks what would happen absent an intervention.FADE handles observable and counterfactual criteria; unlike a cited prior method for counterfactual equalized odds, it applies to classification and regression and combines in-processing with post-processing.Source: §2.2 · pp. 4–5
  • Counterfactual fairness with respect to a sensitive feature, such as a person’s race, is outside this paper’s scope.The paper instead considers counterfactual outcomes under a decision or treatment.Source: §2.2 · pp. 4–5

2.3 Fairness-accuracy and fairness-fairness tradeoffs

  • The size and shape of fairness–accuracy and fairness–fairness tradeoffs depend on the model set, criteria, and data distribution.FADE extends tradeoff exploration to counterfactual measures, a broad class of criteria with closed-form solutions, and theoretical guarantees; its constrained and penalized models can also target specified constraints.Source: §2.3 · pp. 5–6

3 Setup and estimands

FADE predicts observable or counterfactual outcomes with weighted combinations of user-chosen basis functions, then evaluates risk and disparities within that predictor class. Its estimands either optimize risk or unfairness under constraints, or penalize unfairness to trace tradeoff paths.

3.1 FADE summary

  • FADE seeks predictors of observable or counterfactual outcomes that meet user-selected accuracy or fairness constraints or support exploration of tradeoffs.Its “Double” name refers to combining in-processing and post-processing, accommodating both outcome types, optionally learning predictors then ensemble weights, and using doubly robust estimators in the counterfactual setting.Source: §3 · pp. 6–7, §3.1 · p. 7

3.2 Accuracy and fairness measures

  • The framework accommodates unfairness measures expressible as absolute weighted differences of average predictions conditioned on events determined by the sensitive feature and outcome.The paper focuses on rate disparity, generalized false-positive-rate disparity, and generalized false-negative-rate disparity; FPR-diff and FNR-diff together measure violations of equalized odds.Source: §3.2 · pp. 7–8

3.3 Predictor classes

  • FADE predictors are linear combinations of user-chosen basis functions, which may include existing predictors, newly trained predictors, or other functions.The paper analyzes the regime k < n and generally assumes a fixed basis; it leaves analysis of k ≥ n or growing k to future work.Source: §3.3 · p. 9
  • The basis assumptions require its outer-product eigenvalues to stay bounded above and away from zero and its norm to remain bounded.For a fixed basis, the first condition rules out perfect collinearity; the norm condition can be relaxed when basis dimension grows.Source: §3.3 · p. 9

3.4 Estimands

  • A penalized estimand adds weighted unfairness penalties to risk and has a closed-form solution for nonnegative penalty weights.Varying the penalties traces accuracy–fairness paths; under Assumption 1 a constrained risk-min solution has a corresponding penalty, unique when Assumption 3 holds.Source: §3.4 · pp. 10–11
  • The risk-min estimand minimizes risk subject to fairness constraints, while the unfair-min estimand minimizes unfairness subject to a risk constraint.The risk-min problem is feasible for nonnegative fairness tolerances, but unfair-min can be infeasible if no predictor meets the risk bound.Source: §3.4 · p. 10
  • The empirical penalized-model set can be seeded with estimators targeting specified fairness constraints, linking constrained targets to efficient tradeoff exploration.The paper focuses estimation on the penalized risk-min form and does not develop the analogous penalized form for the unfair-min estimand.Source: §3.4 · pp. 11–12

4 Identification

Counterfactual risk and fairness are identified from observed data under consistency, positivity, and ignorability assumptions. The paper uses nuisance regressions and propensity scores in these expressions, but leaves sensitivity analysis for assumption violations to future work.

Identification

  • Identifying counterfactual quantities from observed data requires consistency, positivity, and ignorability.The paper notes that positivity and ignorability may hold in randomized experiments or observational studies with appropriate covariates.Source: §4 · pp. 12–13
  • The assumptions may fail in practice when, for example, measured covariates do not fully deconfound treatment and the potential outcome.The authors leave sensitivity analysis of such violations for future work.Source: §4 · p. 13
  • Under the identifying assumptions, counterfactual risk, FPR-diff, and FNR-diff are identified using observed-data expressions involving propensity scores and outcome regressions.The regressions for the outcome and its square are uncentered influence functions; rate-diff is trivially identified because it involves no outcome.Source: §4 · pp. 12–13

5 Estimation

FADE estimation combines sample splitting, constrained and penalized estimators, and inference for fixed predictors. Its procedures support observable and counterfactual outcomes, with counterfactual guarantees depending on nuisance-estimation assumptions.

5 Estimation

  • Counterfactual estimation splits training and test data to estimate nuisance and target parameters separately; cross-fitting is possible, and a consistency relaxation for nuisance estimates is left unaddressed.New basis predictors require Dlearn, but it is unnecessary when existing basis functions or predictors are used.Source: §5 · pp. 14–15

5.1 Constrained FADE estimators

  • Constrained estimators target minimum risk subject to fairness constraints or minimum unfairness subject to a risk constraint, with asymptotic constraint guarantees under the stated conditions.Because constraints are estimated, a particular estimate may violate its target by an arbitrary amount.Source: §5.1 · pp. 15, 17

5.2 Penalized FADE estimators

  • Penalized FADE evaluates a set of penalty-indexed predictors for selection across fairness-accuracy tradeoffs; its closed-form updates make constructing large sets efficient.The described seeding procedure targets fairness constraints, not risk constraints; the authors note that exploration may still find a desired point if it lies in the chosen basis span.Source: §5.2 · pp. 18–20, Figure 2 · p. 18
  • Penalized estimators have uniform excess-risk and excess-unfairness rates of 1/n or the nuisance rate h(n), whichever is slower; allowing basis dimension k to grow requires k log(k)/n → 0 and yields a k·h(n) rate.These results hold under the assumptions stated for the observable and counterfactual settings.Source: §5.2 · p. 21

5.3 Risk and unfairness of a fixed predictor

  • For fixed predictors and rate, false-positive-rate, or false-negative-rate fairness functions, the estimators support asymptotically valid confidence intervals and hypothesis tests.The theorem gives separate assumptions for observable and counterfactual settings.Source: §5.3 · p. 22

6 Simulations

In simulations, FADE explored fairness–accuracy tradeoffs by penalizing one or several disparities, sometimes reducing multiple disparities with little change in accuracy. The experiments evaluated counterfactual outcomes using predictors trained from observable data.

6.1 Data-generating process

  • The counterfactual Bayes-optimal predictor reached MSE 0.05 and AUC 0.98, but had large rate disparity and small generalized false-positive and false-negative disparities.Predictors were constructed using observable data and evaluated against known Y 0 values.Source: §6 · pp. 22–23, Table 4 · p. 24, §6.1 · p. 24

6.2 Base predictors and nuisance models

  • The OLS aggregation of five base predictors had lower MSE than every base predictor and a performance profile close to the Bayes-optimal predictor.The base predictors included random forest, gradient-boosted, Gaussian Naive Bayes, ridge regression, and a mean predictor.Source: §6.2 · pp. 24–25, Table 5 · p. 25

6.3 Results: one unfairness penalty

  • Penalizing FPR-diff lowered all three disparities while MSE rose from 0.067 to 0.071.For rate-diff and FNR-diff penalties, the targeted disparity also decreased, with only slight or miniscule MSE increases.Source: §6.3 · p. 26, Figure 4 · p. 27

6.4 Results: multiple unfairness penalties

  • Rate-diff 0.06, FPR-diff 0.03, and FNR-diff 0.02 were jointly reached with MSE +0.06 and AUC −0.02 versus OLS.The 1,331 predictors spanned a wide range of performance profiles, and multiple-penalty search found rate-diff and FNR-diff zero-disparity predictors not seen in the single-penalty results.Source: §6.4 · pp. 26–28, Figure 5 · p. 28, Figure 6 · p. 29

7 Results: recidivism risk prediction

In the COMPAS analysis, FADE predictors reduce disparities while maintaining risk performance near the benchmark. All three disparities can be driven to zero with minimal MSE cost relative to OLS.

7 Results: recidivism risk prediction

  • The COMPAS analysis uses general recidivism scores and three covariates, and reports counterfactual results after similar observable and counterfactual conclusions.The dataset covers African-American and Caucasian defendants; the authors note that assessing positivity and ignorability would require domain experts.Source: §7 · pp. 29–30
  • All three disparities can be reduced individually to zero with minimal MSE cost relative to OLS, and improvements extend to pairs of disparities.Nearly all FADE predictors improve on COMPAS in both risk and fairness.Source: §7 · p. 31, Table 7 · p. 31
  • The five base predictors have smaller disparities than COMPAS and generally smaller MSE; OLS does not perform substantially better than the base predictors.COMPAS shows different false-positive and false-negative rates for African-American and Caucasian defendants, as well as a rate disparity.Source: §7 · p. 30, Table 7 · p. 31

8 Results: income prediction

On Adult validation data, FADE predictors are more accurate than the fair-method predictors and have comparable or smaller rate disparity. Including fair predictors in the ensemble allows all three disparities to be reduced to zero with virtually no MSE cost.

8 Results: income prediction

  • The Adult task predicts income above $50K per year from age and education indicators, using sex as the sensitive feature.The data come from demographic variables derived from the 1994 U.S. Census.Source: §8 · p. 31
  • Adding the three fair predictors to the five base predictors makes the tradeoffs essentially disappear: all three disparities reach zero with virtually no MSE cost.With only the five base predictors, fairness-accuracy tradeoffs are most evident for rate-diff and FPR-diff.Source: §8 · pp. 32–33, Figure 8 · p. 34

8.1 Model validation

  • FADE performs comparably to or better than existing fairness methods with base5; including fair predictors in base8 improves both accuracy and fairness.The three fair-method predictors target rate-diff; only two successfully control it, and the Meta predictor's rate-diff is comparable to the base predictors.Source: §8 · p. 32, Table 8 · p. 33, §8.1 · p. 34
  • Training the base predictors and computing and evaluating 1,331 FADE predictors took 25 seconds.A single Meta run took roughly 5 seconds, Reductions approximately 15 seconds, and Adversarial roughly a minute.Source: §8.1 · pp. 34–35
  • The FADE predictors are continuous, but thresholding at 0.5 yields a classifier with classification error 0.24 and disparities of 0 to two digits.That thresholded base5 predictor exactly achieves equalized odds and demographic parity, with slightly higher classification error than Adversarial and Reductions.Source: §8.1 · p. 35
  • All three disparities can be minimized singly or jointly with only small MSE and AUC impacts; aggregated predictors are more accurate than fair predictors and have comparable or smaller rate-diff.Table 9 reports predictors selected on test data and evaluated on validation data.Source: §8.1 · p. 33, Table 9 · p. 35

9 Conclusion

FADE combines predictors to explore fairness-accuracy and fairness-fairness tradeoffs, including for counterfactual outcomes. Its empirical results show that disparities can often be reduced with no tangible accuracy loss, or even with improved accuracy.

9 Conclusion

  • FADE combines arbitrary previously trained and newly trained predictors, whether or not they were designed for fairness, and targets multiple disparities across observable and counterfactual outcomes.The framework collapses the distinction between in-processing and post-processing approaches.Source: §9 · p. 36
  • Its two constrained methods optimize MSE under fairness constraints or unfairness under an MSE constraint; its penalized method efficiently explores tradeoffs and finds favorable risk-fairness profiles.The authors report fast-rate convergence for FADE predictors.Source: §9 · p. 36
  • Disparities can often be substantially reduced with no tangible accuracy loss, or even improved accuracy, relative to unpenalized least squares or an existing benchmark.Source: §9 · p. 36
  • The penalized method's efficiency relies on closed-form parameterized predictors arising from MSE and squared fairness terms.The authors suggest adapting other metrics when the objective can retain a positive-definite quadratic form.Source: §9 · p. 36

Appendix A Proof preliminaries

The proof preliminaries collect assumptions on basis functions and nuisance estimators, define sample and population expectations, and invoke a double-robustness lemma under positivity.

Appendix A Proof preliminaries

  • The listed assumptions include a positive-definite basis second-moment matrix and a uniformly bounded basis norm.Source: Appendix A · p. 37
  • Additional conditions bound the propensity estimator away from one and require consistency of the propensity and outcome nuisance estimators.Source: Appendix A · p. 37
  • The preliminaries define sample and population expectations and state a double-robustness lemma under the positivity assumption.The proof explanation cites iterated expectation, positivity, and Cauchy–Schwarz.Source: Appendix A · pp. 37–38

Appendix B Proofs of propositions

Appendix B proves propositions supporting the fairness constraints, optimization programs, and observable-to-counterfactual results. Its arguments use quadratic-program duality and the consistency and ignorability assumptions.

B.1 Proof of Proposition 1

  • B.1 equates a weighted difference in conditional prediction means with an expectation involving the fairness function.The equality is shown for mappings that select groups through h0 and h1.Source: §B.1 · p. 39

B.2 Proof of Proposition 2

  • B.2 uses strict convexity and feasibility of the risk-min quadratic program to establish strong duality and the KKT conditions.The proof notes that the dual has at least one solution under Assumption 1.Source: §B.2 · p. 39

B.3 Proof of Proposition 3

  • B.3 sets each fairness tolerance to the corresponding empirical squared unfairness to satisfy the risk-min KKT conditions.The resulting constraints bound population squared unfairness by empirical squared unfairness.Source: §B.3 · p. 39

B.4 Proof of Proposition 4

  • B.4 derives the risk and FPR-difference results using iterated expectation, consistency, and ignorability; the FNR result follows by identical reasoning.The proof connects conditional outcome moments to the observable and counterfactual quantities.Source: §B.4 · pp. 39–40

Appendix C Proof of Theorem 5

Appendix C establishes asymptotic normality for risk and unfairness estimators, with counterfactual proofs relying on rate conditions for nuisance estimates. Appendix D also states a convergence-rate result for estimated constrained-program optimal values.

Appendix C Proof of Theorem 5

  • Theorem 5’s observable-setting result follows from the central limit theorem, while its counterfactual proof controls nuisance-estimation remainders under the stated lemmas and assumptions.The risk-estimator argument shows the product of nuisance errors is oP(1/√n).Source: Appendix C · p. 40, §C.1 · p. 40

C.1 Asymptotic normality of the risk estimator

  • The risk estimator’s counterfactual asymptotic-normality proof bounds nuisance-estimation remainder terms before applying the central limit theorem.The proof identifies the product of the outcome-regression and propensity-score errors as oP(1/√n).Source: §C.1 · p. 40

C.2 Asymptotic normality of the unfairness estimators

  • The unfairness-estimator proof obtains asymptotic normality for gFPR and extends the result to gFNR by identical reasoning.It uses the central limit theorem, continuous mapping, Slutsky’s theorem, and the delta method.Source: §C.2 · pp. 40–41

Appendix D Proofs for constrained FADE

  • Under the stated convexity and Slater-condition setup, the estimated constrained-program optimal value differs from its target by OP(1/f(n)).The lemma adapts Shapiro (1991) and gives sufficient conditions for this rate.Source: Appendix D · p. 42

D.1 Intermediate result

The intermediate result establishes convergence properties for the estimated objective and constraint functions, which the constrained-FADE proofs use to bound excess risk and unfairness.

D.1 Intermediate result

  • The asymptotic analysis confines coefficient vectors to a compact set, though this restriction is not imposed in the actual optimization.In practice, predictor outputs are truncated to the specified outcome range.Source: §D.1 · p. 42
  • The estimated objective and constraint functions converge uniformly to Gaussian-process limits under the Donsker condition, with Lemma 4 giving setting-specific rate conditions.The stated assumptions differ between observable and counterfactual settings.Source: §D.1 · pp. 43–44

D.2 Proof of Theorem 1 (Excess risk in constrained FADE)

  • For constrained risk-min and unfair-min problems, the excess risk is OP(1/√n).The proof establishes this rate for both optimization settings.Source: §D.2 · p. 45

D.3 Proof of Theorem 2 (Excess unfairness in constrained FADE)

  • For constrained unfair-min and risk-min problems, the excess unfairness is OP(1/√n).The risk-min result is stated for each fairness constraint.Source: §D.3 · p. 46

Appendix E Proofs for penalized FADE

Appendix E develops proofs for penalized FADE, including arguments for excess risk and unfairness. Appendix F extends the estimators to bases with dimension at least the sample size by adding a penalty matrix term.

Appendix E Proofs for penalized FADE

  • The penalized-FADE proofs use preliminary results under the assumptions of Theorems 3 and 4.The proofs also use the Lipschitz dependence of the estimated coefficient on the penalty parameter and compactness of Λ to obtain a Donsker class.Source: Appendix E · pp. 46–47

E.1 Proof of Theorem 3 (Excess risk in penalized FADE)

  • The E.1 proof uses a Donsker condition to obtain uniform convergence over Λ for the excess-risk argument.The preceding bound includes OP(1) for one factor.Source: §E.1 · p. 48

E.2 Proof of Theorem 4 (Excess unfairness in penalized FADE)

  • The E.2 proof likewise invokes a Donsker condition for uniform convergence over Λ in the excess-unfairness argument.Source: §E.2 · p. 48

Appendix F Bases with dimension k ≥n

  • For k ≥ n, adding a penalty matrix term accommodates large bases and preserves fast computation of many penalized solutions.The term ensures invertibility of the large matrix; the text also notes possible overfitting prevention when k < n and the basis is expressive or k is near n.Source: Appendix F · p. 49
Loading 2109.00173v1…