Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

David Madras; Toniann Pitassi; Richard Zemel

Source-linked AI summary

Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

David Madras, Toniann Pitassi, Richard Zemel

arXiv:1711.06664v3Nov 17, 2017stat.MLcs.LG

TL;DR

The paper addresses how automated models should make decisions when they operate alongside external human or machine decision-makers. It proposes learning to defer, a two-stage framework that adapts whether the model predicts or passes the case downstream. Across simulated decision-maker settings, the approach improves the accuracy and/or fairness of the overall decision-making system.

Problem
Automated models are often used with other decision-makers, but their interactions and system-level effects on accuracy and fairness are not adequately addressed.
Method
Learning to defer trains an automated model to output predictions and adaptive deferral decisions within a two-stage cascade involving a fixed external decision-maker.
Results
Learning to defer improves the accuracy and fairness of decision-making pipelines across scenarios involving inconsistent, biased, or otherwise resourceful external decision-makers.
Takeaways & Limitations
Adaptive model-decision-maker mixtures can outperform standard rejection learning and optimize the performance of the decision-making pipeline as a whole.
Takeaways & Limitations
The framework focuses on a two-agent, two-stage setup and assumes the downstream decision-maker is a fixed, unknown black box with information unavailable to the model.

Abstract

from arXiv · show

In many machine learning applications, there are multiple decision-makers involved, both automated and human. The interaction between these agents often goes unaddressed in algorithmic development. In this work, we explore a simple version of this interaction with a two-stage framework containing an automated model and an external decision-maker. The model can choose to say "Pass", and pass the decision downstream, as explored in rejection learning. We extend this concept by proposing "learning to defer", which generalizes rejection learning by considering the effect of other agents in the decision-making process. We propose a learning algorithm which accounts for potential biases held by external decision-makers in a system. Experiments demonstrate that learning to defer can make systems not only more accurate but also less biased. Even when working with inconsistent or biased users, we show that deferring models still greatly improve the accuracy and/or fairness of the entire system.

1. Introduction

Learning to defer treats automated predictions as part of a larger human-machine decision system, adapting when to predict or pass based on the downstream decision-maker. The framework is designed to improve system-level accuracy and fairness, including with inconsistent or biased decision-makers.

Automated models in high-stakes domains should be evaluated as components of larger decision-making systems, not only in isolation.The downstream impact of a model depends on how an external decision-maker uses its predictions.
Responsible prediction requires making predictions when they align reliably with system objectives such as accuracy and fairness.
Unlike rejection learning, learning to defer adapts the pass decision to the external decision-maker’s expertise and weaknesses.A model may predict despite uncertainty when the downstream decision-maker is inaccurate or biased for a subgroup.
The paper formulates learning to defer as adaptive rejection learning, allowing the model to work adaptively with the decision-maker.
Theoretical and experimental evidence shows that deferring models can improve the accuracy and fairness of the overall system, even with inconsistent or biased users.Experiments simulate downstream decision-makers that are inconsistent, biased, or equipped with side information.

2. Learning to Defer

Learning to defer models a two-stage decision system in which an automated model may predict or pass to an external decision-maker, and optimizes the final system output rather than the model stage alone. It adaptively uses the decision-maker’s variable strengths and weaknesses, generalizing rejection learning and potentially improving system accuracy and fairness.

2.1 A Joint Decision-Making Framework: Learning to defer optimizes the system’s final output, unlike standard rejection learning, which evaluates the model stage in isolation.The system outputs the model’s prediction when the model predicts and the decision-maker’s prediction when the model passes.
2.1 A Joint Decision-Making Framework: The framework combines an automated model with an external decision-maker: the model predicts or passes, and the decision-maker supplies the final output after a pass.The decision-maker may have auxiliary information unavailable to the model but remains a fixed, unknown black box.
2.1 A Joint Decision-Making Framework: The deferral decision acts as a gating variable that predicts whether the model or decision-maker will incur lower loss on each example.The model learns its predictive probabilities and binary deferral decisions, while the decision-maker’s parameters remain outside its control.
2.3 Learning to Defer is Adaptive Rejection Learning: Learning to defer generalizes rejection learning because its loss uses the decision-maker’s example-specific loss instead of a constant rejection penalty.When the decision-maker has constant loss, suitable penalties make the two objectives equivalent.
2.3 Learning to Defer is Adaptive Rejection Learning: The adaptive formulation better matches systems where every case eventually receives a decision and external decision-makers vary in accuracy, bias, or expertise.Deferral can be reduced for groups facing biased downstream decisions and increased where the decision-maker has higher accuracy from side information.

3. Formulating Adaptive Models within Decision Systems

The paper formulates learning to defer as an adaptive decision-system model that combines a model’s prediction with an external decision-maker’s output. It supports post-hoc thresholds, differentiable gating, and fairness regularization.

Formulating Adaptive Models within Decision Systems: Learning to defer treats the external decision-maker as an expert and learns a pass decision that optimizes the system’s combined output.The model produces predictive probabilities and binary deferral decisions, rather than only expressing uncertainty.
Formulating Adaptive Models within Decision Systems: The deferral probability π determines whether the decision-maker or the model supplies the final prediction.Deferral occurs with probability π, while probability 1 −π selects the model’s decision.
Formulating Adaptive Models within Decision Systems: The framework minimizes expected loss over the pass decision, while rejection learning uses the corresponding non-adaptive formulation.The paper presents learning to defer and rejection learning as related objectives over the downstream decision process.
3.1 Post-hoc Thresholding: Post-hoc thresholding learns two thresholds that turn a binary predictor into a ternary classifier with positive, negative, and pass outputs.The thresholds can be trained after an existing binary classifier and create the pass category between commitment regions.
3.2 Learning a Differentiable Model: A differentiable adaptive model lets π depend on both the model prediction and input X, allowing deferral to respond to heterogeneous decision-maker behavior.This flexibility reflects that the decision-maker’s expected loss may vary with the example differently from the model’s.
3.2 Learning a Differentiable Model: Training estimates the expected loss by sampling s from Ber(π) and uses a Concrete relaxation to estimate gradients through sampling.The paper also notes that stopping gradients from π through the model prediction can preserve the predictor’s independent quality.
3.3 Fair Classification through Regularization: The fair loss adds αfair times a fairness regularizer R to the deferral loss, trading off accuracy and fairness.The regularizer is based on differences in conditional error rates across sensitive-attribute groups, and its binary-classification efficacy is evaluated separately.
3.3 Fair Classification through Regularization: The fairness scheme is related to prior regularization methods, and learning to defer with an oracle decision-maker is experimentally equivalent to rejection learning in the fairness case.This equivalence is reported in Appendix E.

4. Related Work

Related work covers fairness definitions and regularized fair classification, pass or rejection options, and sequential or multi-classifier decision systems. The paper positions learning to defer at the intersection of these areas.

Notions of Fairness: Fairness research includes individual fairness, fair-awareness, statistical parity, calibration, equalized odds, and equal opportunity.The literature uses multiple operational definitions, and some cannot be simultaneously satisfied non-trivially.
Notions of Fairness: Prior algorithms integrate equalized odds into learning through regularization.The cited work develops methods that explicitly incorporate fairness constraints into model training.
Incorporating pass: Rejection-learning research studies error-rate and rejection-rate tradeoffs and integrates a pass option directly into model learning.Related work also examines rare cases, human auditing, and cascading models.
Related decision systems: Other related studies address fair sequential decisions, fair bandits, mixtures of classifiers, and safety-reserve or safe-fail options.The paper distinguishes these efforts from its pass-based learning procedure.

5. Experiments

Experiments evaluate learning to defer with simulated decision-makers that have extra information and different accuracy or fairness properties. Across three scenarios, adaptive deferral improves accuracy-fairness tradeoffs or classification accuracy relative to comparison models.

Experimental scenarios: The experiments use three decision-maker scenarios: high-accuracy but fairness-ignoring, highly biased, and inconsistent across subgroups.The scenarios represent downstream decision-makers that may be more informed, unfair, or unreliable for particular groups.
Datasets and Experiment Details: Semisynthetic experiments combine real datasets with simulated decision-maker outputs trained under different conditions and given extra information unavailable to the model.The datasets are COMPAS for recidivism prediction and Heritage Health for Charlson Index prediction.
Datasets and Experiment Details: The simulated decision-maker receives violent-recidivism information for COMPAS or primary-condition-group information for Health.Bias is induced with αfair = −0.1, while inconsistency is simulated by flipping selected predictions with 30% probability.
Displaying Results: Results are shown across αfair and γdefer/γreject settings using Pareto fronts from held-out test sets, with each plotted point being the median of 5 runs.Figure 3 uses accuracy on the Y-axis and fairness or deferral rate on the X-axis depending on the scenario.
High-Accuracy DM: In the high-accuracy decision-maker scenario, learning to defer achieves a better accuracy-fairness tradeoff than rejection learning.Fair rejection learning also outperforms binary baselines by incorporating extra decision-maker accuracy, while most of the deferring model’s benefit comes from added fairness.
Highly-Biased DM: With a highly biased decision-maker, adaptive deferral can use different deferral rates across sensitive groups and modulate total deferral to counteract bias.The comparison is made against rejection learning in the fairness-accuracy plots.
Inconsistent DM: With an inconsistent decision-maker, learning to defer has higher accuracy at each deferral rate, and the best models outperform both the decision-maker and a binary baseline.The first ∼40% of passed examples improve system accuracy, and adaptive models defer more on reliable than unreliable subgroups.
Subgroup accuracy: Deferring models retain higher accuracy under minimum-subgroup-accuracy requirements, with their most accurate models also having the highest minimum subgroup accuracy.The reported result does not show the accuracy improvement coming at the expense of the least accurate subgroups.

6. Conclusion

The conclusion frames learning to defer as a general multi-agent decision framework that can improve whole-system performance beyond rejection learning. It presents deferring as a way to support fairer and more responsible machine learning when automated models interact with other decision-makers.

6. Conclusion: Learning to defer generalizes rejection learning within a framework for multi-agent decision-making.The framework models a two-stage interaction between an automated model and an external decision-maker.
6. Conclusion: Deferring models can optimize the performance of decision-making pipelines as a whole beyond the improvement provided by rejection learning.
6. Conclusion: The framework is intended for practical systems where automated models interact with multiple decision-making agents.
6. Conclusion: Learning to defer supports predicting responsibly within surrounding systems, with implications for fairer and more responsible machine learning.

Appendix A. Learning to Defer to Three Types of DM: Health Results

Figure 6 compares learning to defer, rejection learning, and binary models on Health across different decision-maker scenarios. It reports accuracy against fairness or deferral rate, using distinct curves and baseline markers for the competing systems.

Appendix A. Learning to Defer to Three Types of DM: Health Results: Figure 6 compares learning-to-defer, rejection learning, and binary models across different Health decision-maker scenarios.
Appendix A. Learning to Defer to Three Types of DM: Health Results: Accuracy is plotted against fairness in the left and centre columns and against deferral rate in the right column.Lower fairness values are better, and accuracy is the y-axis throughout.
Appendix A. Learning to Defer to Three Types of DM: Health Results: The figure includes an accuracy-only binary baseline, a fair rejection model, a fair deferring model, a decision-maker-alone result, and, in some columns, a fairness-optimizing binary model.

Appendix B. Results: Binary Classification with Fair Regularization

The appendix evaluates fairness regularization for binary classifiers on COMPAS and Health, including neural networks with Bayesian weight uncertainty. It reports that regularization reduces disparate impact without causing many additional errors and that the threshold search is efficient.

Appendix B. Results: Binary Classification with Fair Regularization: Regularization effectively reduces DI without making too many more errors on COMPAS, broadly replicating prior results.The replication differs in model architecture and training/test splits, while the main takeaway remains similar.
Appendix B. Results: Binary Classification with Fair Regularization: Figures 7 and 8 examine DI and error rate as functions of α across COMPAS and Health using MLP and Bayesian weight-uncertainty algorithms.
Appendix B. Results: Binary Classification with Fair Regularization: The experiments use COMPAS recidivism prediction with race as the sensitive variable and provide the decision-maker extra violent-recidivism information.
Appendix B. Results: Binary Classification with Fair Regularization: The models are fully connected two-layer neural networks trained with ADAM, with five hidden units for COMPAS and twenty for Health.
Appendix B. Results: Binary Classification with Fair Regularization: Separate group thresholds are selected by random search over 1000 combinations and evaluated on a held-out half of the test set.The search spans several α and γ values in the thresholding procedure.
Appendix B. Results: Binary Classification with Fair Regularization: Random search was significantly faster than grid search and more effective than gradient-based threshold optimization because the loss landscape appeared to contain many local minima.

Appendix E. Comparison of Learning to Defer with an Oracle in Training to Rejection Learning

The appendix compares rejection learning with learning to defer trained using an oracle and tested with the standard decision-maker. Their accuracy–fairness trade-offs are nearly identical, while ignoring the decision-maker can make rejection learning offer little advantage over a binary model.

Appendix E. Comparison of Learning to Defer with an Oracle in Training to Rejection Learning: Figure 9 compares learning to defer with oracle training against rejection learning while using the same decision-maker at test time.
Appendix E. Comparison of Learning to Defer with an Oracle in Training to Rejection Learning: The appendix frames rejection learning and learning to defer with an oracle as theoretically equivalent, while testing their equivalence experimentally for the fair-regularized loss.
Appendix E. Comparison of Learning to Defer with an Oracle in Training to Rejection Learning: Learning to defer trained with an oracle and rejection learning trade off accuracy and fairness in almost an identical manner.
Appendix E. Comparison of Learning to Defer with an Oracle in Training to Rejection Learning: When the decision-maker is not incorporated through learning to defer, rejection learning results mostly overlap binary-model results in the low-accuracy inconsistent-DM setting.

Appendix H. Results: Learning to Defer by Deferral Rate

Learning to defer maintains a fairness/accuracy advantage over rejection learning across deferral rates, with the largest benefit among frequently deferring models.

Learning-to-defer models show a similar fairness/accuracy advantage over rejecting models even at comparable deferral rates.
The advantage of learning to defer is substantially larger for models with high deferral rates than for models with low deferral rates.
The larger high-deferral benefit appears primarily in fairness because the downstream decision-maker already provides high accuracy.
Figure 10 compares accuracy and fairness across binary, rejection, deferring, and decision-maker-only models during hyperparameter sweeps.
Figure 11 reports Pareto fronts separately for low, medium, and high final deferral-rate runs.

Appendix I. Results: Deferral Rates with a Biased DM

On COMPAS, learning-to-defer models adjust their deferral behavior to the decision-maker’s quality and to differences across sensitive-attribute groups, unlike rejection models.

Over 1000 hyperparameter combinations, the COMPAS experiments compare deferral or rejection rates across decision-maker types and sensitive-attribute values.
Learning-to-defer models treat biased and standard decision-makers differently, whereas rejection models do not.
With a lower-quality biased decision-maker, the deferring model almost always defers on fewer than 20% of examples.
Figure 12 plots the percentage of runs below each deferral-rate cutoff, separating rejection and deferring models, decision-maker types, and sensitive-attribute groups.
Learning-to-defer models vary deferral rates across sensitive-attribute values in response to decision-makers who treat groups differently.