Fair Adversarial Gradient Tree Boosting

Vincent Grari; Boris Ruf; Sylvain Lamprier; Marcin Detyniecki

Source-linked AI summary

Fair Adversarial Gradient Tree Boosting

Vincent Grari, Boris Ruf, Sylvain Lamprier, Marcin Detyniecki

arXiv:1911.05369v2Nov 13, 2019cs.LGcs.AIcs.CYstat.ML

TL;DR

Fair classification lacks comparable adversarial methods for efficient decision-tree models, despite the importance of mitigating discriminatory predictions. The paper introduces adversarial gradient tree boosting and reports higher accuracy at similar fairness across four datasets and multiple fairness definitions.

Problem
Most bias-mitigation strategies focus on neural networks, leaving fair decision-tree classifiers comparatively underexplored despite tree boosting’s efficiency for tabular data.
Method
The method combines gradient tree boosting with an adversarial neural network whose ability to predict the sensitive attribute is minimized during training.
Results
Across four datasets, the approach achieves higher accuracy while obtaining a similar level of fairness compared with state-of-the-art algorithms.
Takeaways & Limitations
Adversarial debiasing can be applied to gradient-boosted decision trees while retaining competitive accuracy and fairness across demographic-parity and equalized-odds evaluations.

Abstract

from arXiv · show

Fair classification has become an important topic in machine learning research. While most bias mitigation strategies focus on neural networks, we noticed a lack of work on fair classifiers based on decision trees even though they have proven very efficient. In an up-to-date comparison of state-of-the-art classification algorithms in tabular data, tree boosting outperforms deep learning. For this reason, we have developed a novel approach of adversarial gradient tree boosting. The objective of the algorithm is to predict the output $Y$ with gradient tree boosting while minimizing the ability of an adversarial neural network to predict the sensitive attribute $S$. The approach incorporates at each iteration the gradient of the neural network directly in the gradient tree boosting. We empirically assess our approach on 4 popular data sets and compare against state-of-the-art algorithms. The results show that our algorithm achieves a higher accuracy while obtaining the same level of fairness, as measured using a set of different common fairness definitions.

I. INTRODUCTION

Machine-learning systems can reproduce discrimination, creating a need to assess and mitigate bias alongside predictive performance. This paper targets the limited use of adversarial fairness methods with decision-tree classifiers.

The paper addresses discriminatory bias in machine-learning decisions, particularly when models learn or reinforce biases from training data.
Removing protected attributes is insufficient because correlated non-sensitive features can act as proxies for sensitive attributes.
Most existing bias-mitigation strategies focus on neural networks, despite gradient tree boosting’s efficiency and practical use for tabular data.
The paper proposes combining gradient tree boosting with an adversarial fairness constraint for decision-tree classifiers.
Experiments compare the proposal and variants with state-of-the-art approaches using two fairness metrics.

1) Demographic Parity:

Demographic parity requires equal positive-outcome rates across sensitive groups. The paper evaluates this objective using p-rule and disparate impact metrics.

A classifier satisfies demographic parity when its positive prediction is independent of the protected attribute S [7].
The p-rule measures the ratio of positive rates between groups, with 100% indicating total fairness and 0% indicating complete unfairness.
Disparate impact measures the absolute difference between outcome distributions for subpopulations with different sensitive attributes.
Equalized odds requires equal true-positive and false-positive rates across demographic groups.
Disparate mistreatment measures group differences in false-positive and false-negative rates, with values closer to zero indicating less disparity.

B. Related Work

Fairness mitigation methods are commonly organized into pre-processing, in-processing, and post-processing strategies. The paper positions its decision-tree approach within in-processing mitigation and builds on gradient boosting.

Pre-processing methods modify inputs through attribute suppression, label changes, reweighting, or resampling.
In-processing methods mitigate bias during training, including fairness penalties, metric-aware classifiers, and adversarial approaches.
Post-processing methods alter outputs of a trained classifier, avoiding retraining but deriving fairness-adjusted predictions.
Classical gradient boosting sequentially fits weak learners to pseudo-residuals and updates the model iteratively.

III. GRADIENT TREE BOOSTING

Gradient tree boosting builds a strong predictor by sequentially combining weak decision-tree learners. Each learner is fitted to the negative loss gradient and added iteratively to the model.

Gradient tree boosting combines sequentially built weak CART learners into an ensemble strong learner.
The method seeks a function that approximately minimizes empirical loss over the training examples.
At each iteration, pseudo-residuals are computed as the negative gradient of the loss function.
A new weak learner is fitted to the pseudo-residuals and added to the current model until convergence.

IV. FAIR ADVERSARIAL GRADIENT TREE BOOSTING (FAGTB)

FAGTB adapts gradient tree boosting with adversarial learning so prediction accuracy is optimized while limiting sensitive-attribute predictability. The method uses a min-max objective and injects adversarial gradients into successive tree updates.

FAGTB combines gradient tree boosting with adversarial learning to leverage tree-based classification while mitigating sensitive-attribute information in predictions.
The predictor and adversary are optimized in a min-max game, with λ controlling the adversarial loss contribution.
At each boosting step, prediction-loss and adversary-loss gradients are summed to define the target for the next decision tree.
For demographic parity, the adversary receives the classifier’s predicted probability and estimates the sensitive attribute from it.
For equalized odds, the adversary additionally receives the label, allowing sensitive-attribute predictions to depend on the true class.
If perfect prediction and fairness are simultaneously attainable, the adversary reduces to the sensitive-attribute prior; otherwise, λ governs the accuracy–fairness trade-off.

B. Learning

The learning procedure extends classical gradient tree boosting with an adversarial neural network that is updated as the classifier evolves. Each new tree corrects both prediction and adversarial biases before the adversary adapts to the new outputs.

The algorithm initializes the classifier with constant values and initializes the adversarial network parameters before boosting begins.
Each iteration fits a decision-tree regressor to pseudo-residuals designed to correct both prediction and adversarial biases.
A line search selects the tree weight γm before adding the new tree to the classifier.
After the classifier update, the adversary backpropagates on the new classifier outputs to update its weights.

V. EMPIRICAL RESULTS

The experiments evaluate FAGTB on synthetic and real-world classification settings against state-of-the-art methods, while the synthetic study exposes its fairness–accuracy behavior and feature effects.

The study evaluates regression accuracy and fairness on synthetic and real-world data sets, comparing FAGTB with state-of-the-art algorithms.
A. Synthetic Scenario: A classical gradient-tree model achieved a 67% p-rule, despite the label being constructed without dependence on the sensitive attribute.
A. Synthetic Scenario: A FAGTB model with λ = 0.015 was selected to reach an approximately 95% p-rule for demographic parity.
A. Synthetic Scenario: Improving the p-rule by 29 points was accompanied by a 10-point accuracy decrease when bias attenuation occurred suddenly.
A. Synthetic Scenario: The fair model increased the feature-importance gap to 0.145 points and made the indirectly correlated color feature insignificant.

1) Data sets:

The benchmark uses four popular fair-classification data sets spanning income, recidivism, credit default, and bank-marketing prediction tasks. Ten randomized 80/20 train–test repetitions provide averaged test accuracy and fairness metrics.

1) Data sets:: The experiments use Adult, COMPAS, Default, and Bank data sets, covering income, recidivism, payment default, and term-deposit subscription prediction.
1) Data sets:: Sensitive attributes are gender for Adult and Default, race for COMPAS, and age for Bank.
1) Data sets:: Each data set is evaluated in 10 experiments using random 80% training and 20% test subsets.
1) Data sets:: Reported results are averages of accuracy and fairness metrics computed on the test sets.

2) Fairness algorithms:

The evaluation compares FAGTB variants and baselines under fairness objectives, tuning λ to balance classifier accuracy against fairness.

The experiments compare Standard GTB and Standard NN with FAGTB-1-Unit, FAGTB-NN, and several fair in-processing algorithms.FAGTB-1-Unit uses logistic regression as the adversary, whereas FAGTB-NN uses a deep neural network.
Hyperparameters are selected by grid search with five-fold cross-validation for each algorithm and data set.The FAGTB λ value is selected jointly with tree and neural-network hyperparameters.
λ balances classifier accuracy and fairness, with larger values producing fairer predictions and values near 0 prioritizing classifier optimization.For FAGTB, λ balances two cost functions and depends on the target fairness objective.

3) Results:

Across demographic parity and equalized odds, FAGTB-NN achieves strong accuracy while maintaining comparable fairness, though equalized odds is harder to optimize.

FAGTB-NN achieves the highest accuracy among mitigation algorithms while maintaining approximately 90% p-rule equality for demographic parity.Standard GTB and Standard NN have higher accuracy but are the most biased; Standard GTB reaches 32.6% p-rule on Adult UCI.
Higher λ values progressively align predicted-probability distributions across sensitive groups, with λ = 0.15 producing practically aligned distributions.The compared models use λ = 0, 0.06, and 0.15.
Zhang2018’s projection term does not improve demographic-parity results, while Wadsworth2018 obtains similar results without that term.The projection term is intended to prevent predictor updates that could help the adversary.
For equalized odds, FAGTB-NN attains DispFPR and DispFNR below 0.02 on four data sets except Bank, where DispFNR is 0.07.The Bank result is comparable to most state-of-the-art algorithms, which obtain DispFNR values between 0.06 and 0.08.
The method is reported to be more accurate than state-of-the-art algorithms at a similar fairness level.
The current approach uses a neural-network adversary, while deep neural decision forests are proposed as a future tree-only alternative.Future work also includes identifying the optimal λ and detecting bias in specific population subsegments.