Source-linked AI summary
Fairness in Credit Scoring: Assessment, Implementation and Profit Implications
Nikita Kozodoi, Johannes Jacob, Stefan Lessmann
TL;DR
Fair machine learning has limited evidence in credit scoring despite the sector's important lending decisions and fairness concerns. The paper reviews fairness criteria and processors, then benchmarks them in profit-oriented credit scoring. It finds that separation is a suitable fairness criterion, in-processors offer a strong profit-fairness balance, and moderate discrimination reduction can be achieved at relatively low cost.
Problem
Fair-ML research has few connections to credit scoring, especially regarding operational loan approval and the trade-off between fairness and profit.
Method
The paper reviews fairness criteria and processors, and empirically compares them using profit-oriented evaluation across credit-scoring applications.
Results
The study recommends separation for measuring scorecard fairness and finds that in-processors provide a strong profit-fairness balance while reasonable discrimination reduction remains relatively inexpensive.
Takeaways & Limitations
Decision-makers can use processor-specific Pareto frontiers to identify techniques that reduce discrimination to a required level at the smallest monetary cost.
Takeaways & Limitations
Processor choice is constrained by implementation feasibility; with multiple sensitive attributes, separation optimization is available only through adversarial debiasing or reject option classification.
Abstract
from arXiv · showhide
The rise of algorithmic decision-making has spawned much research on fair machine learning (ML). Financial institutions use ML for building risk scorecards that support a range of credit-related decisions. Yet, the literature on fair ML in credit scoring is scarce. The paper makes three contributions. First, we revisit statistical fairness criteria and examine their adequacy for credit scoring. Second, we catalog algorithmic options for incorporating fairness goals in the ML model development pipeline. Last, we empirically compare different fairness processors in a profit-oriented credit scoring context using real-world data. The empirical results substantiate the evaluation of fairness measures, identify suitable options to implement fair credit scoring, and clarify the profit-fairness trade-off in lending decisions. We find that multiple fairness criteria can be approximately satisfied at once and recommend separation as a proper criterion for measuring the fairness of a scorecard. We also find fair in-processors to deliver a good balance between profit and fairness and show that algorithmic discrimination can be reduced to a reasonable level at a relatively low cost. The codes corresponding to the paper are available on GitHub.
1 Introduction
The paper addresses the sparse intersection between fair machine learning and credit scoring, where algorithmic decisions have substantial economic and societal consequences. It reviews fairness criteria and processors, then empirically evaluates their suitability and profit implications in lending.
- Motivation: Retail credit is economically important, and machine-learning scorecards support loan-approval decisions.The paper reports US retail credit outstanding exceeding $4,161 billion in 2020.
- Motivation: Political and regulatory initiatives emphasize equal opportunity and systemic monitoring because automated credit decisions can harm historically disadvantaged groups.
- Fairness research: Fair machine learning evaluates whether predictions satisfy statistical criteria across groups defined by sensitive attributes such as gender, religion, or ethnicity.
- Research gap: Prior fair-ML and credit-scoring research has few points of contact and does not focus on operational loan approval or the fairness-profit trade-off.The paper identifies only three studies at the interface between the two disciplines.
- Paper contributions: The paper revisits fairness criteria, catalogs fairness processors, and empirically compares them using established credit-scoring performance measures, profitability, and seven real-world data sets.The comparison also measures multiple fairness criteria rather than only the criterion optimized by each processor.
2 Theoretical Background
The theoretical background frames fairness processors as interventions at different stages of the machine-learning pipeline and introduces independence, separation, and sufficiency for credit scoring. It emphasizes that strict independence can be impractical, while separation compares group error rates and accommodates correlation between protected attributes and outcomes.
- Fairness optimization in the modeling pipeline: Fairness processors intervene through pre-processing, in-processing, or post-processing, depending on the prediction task's conceptual and technical feasibility.
- Fairness optimization in the modeling pipeline: Pre-processing transforms training data, in-processing adds fairness constraints during training, and post-processing adjusts predictions after model estimation.
- Fairness criteria: A scorecard predicts repayment risk from applicant features and accepts customers whose predicted scores exceed a cutoff.
- Independence: Independence requires equal acceptance rates across sensitive groups, but strict equality can reduce performance enough to threaten credit-scoring sustainability.The paper presents the 80 percent rule as a relaxed alternative to strict independence.
- Separation: Separation requires equal false-positive and false-negative rates across sensitive groups, while allowing the protected attribute to correlate with the true outcome.Its SP measure averages absolute group differences in FPR and FNR; SP = 0 represents perfect separation.
- Sufficiency: Sufficiency requires equal positive-class probabilities across sensitive groups among applicants whose scores exceed the cutoff.
3 Fairness and Credit Scoring
Prior work on fair ML in credit scoring is sparse, while lending decisions involve unequal misclassification costs and substantial social consequences. The section compares independence, separation, and sufficiency, identifying separation as the most suitable criterion for credit scoring.
- Only three prior studies address fair ML and credit scoring, without focusing on operational loan approval or the profit–fairness trade-off.
- Credit scoring involves unequal costs: rejected repayable applicants lose opportunities, while loans to non-solvent applicants can create debt and worsen financial conditions.
- Fairness criteria: Independence equalizes acceptance rates across groups but can increase defaults and perpetuate unfairness when repayment ability differs between groups.
- Fairness criteria: Separation equalizes error rates while allowing different positive classification rates, accounting for group-specific misclassification costs.
- Fairness criteria: Sufficiency can coexist with discriminatory separation outcomes and is therefore considered less suitable for credit scoring.
- Implications: Separation appears preferable because it reflects customer and financial-institution interests, while long-term effects of fairness constraints remain important for societal evaluation.
4 Methodology
The methodology catalogs fairness processors by criterion, implementation method, and classification requirements, then examines selected processors in a profit-oriented credit-scoring setup. It emphasizes that processor choice depends on application and implementation constraints, and that profitability is a relevant fairness cost.
- Catalog: The catalog organizes fairness processors by target criterion, implementation method, and classification-problem requirements, supporting selection of eight processors for empirical study.
- Catalog: Most cataloged processors implement independence, motivating additional methods for separation and sufficiency when independence is unsuitable.
- Implementation constraints: Processor choice is constrained by application and implementation context, including multiple protected attributes, regulatory approval, and access to model inputs or scores.
- Implementation constraints: Post-processors are easier to implement because they are agnostic to input data and scorecards and require only predicted scores.
- Evaluation objective: The comparative analysis evaluates fairness processors under profit maximization rather than relying solely on standard statistical performance measures such as accuracy.
- Pre-processors: Reweighing assigns observations weights based on group–class probabilities, resamples the training data accordingly, and helps fulfill independence.
- Pre-processors: Disparate impact removal transforms features to prevent prediction of the sensitive attribute while preserving within-group rank information.
- In-processors: In-processors incorporate fairness during classifier construction through mechanisms such as fairness-driven regularization and adversarial debiasing.
5 Experimental Setup
The experiment evaluates fairness processors across seven credit-scoring datasets using multiple classifiers, a profit-oriented cost framework, and fairness and discrimination metrics. The setup uses train/test splitting and cross-validation to aggregate processor performance.
- 5.1 Data: Seven credit-scoring datasets represent applications with loan features, repayment outcomes, and age-group sensitive attributes.The datasets combine UCI, competition, and financial-institution sources; age is treated as a protected demographic characteristic under the Equal Credit Opportunity Act.
- 5.2 Experimental Setup: The data are split into training (60%) and test (40%) sets, with five-fold cross-validation applied to the training data.Fairness processors are implemented within each training-fold combination, alongside an unconstrained scoring model.
- 5.2 Experimental Setup: Four base classifiers—logistic regression, artificial neural network, random forest, and XGB—support robustness checks for pre- and post-processors.In-processors are evaluated without a base classifier.
- 5.2 Experimental Setup: Profit is measured per EUR issued using Expected Maximum Profit, with misclassification costs normalized so the base scenario rejects all applications.The cost matrix distinguishes losses from misclassifying bad risks and opportunity costs from rejecting good risks; B is based on loss given default, exposure at default, and principal.
- 5.2 Experimental Setup: Performance combines profitability, AUC, independence, separation, and sufficiency across datasets, folds, and classifiers.Pre- and post-processors yield 140 aggregated estimates each, while in-processors yield 35 estimates.
6 Empirical Results
The empirical analysis compares fairness, predictive performance, and profitability across fairness processors and seven real-world credit-scoring data sets. Results show a clear profit–fairness trade-off, while some processors provide better practical balances than others.
- 6.1 Correlation Analysis: AUC and profit produce similar model rankings, with a Spearman correlation of 0.80, but their disagreement motivates emphasizing profit.The correlations average performance estimates across processor variants and seven credit-scoring data sets.
- 6.1 Correlation Analysis: Independence and separation correlate at 0.95, indicating that optimizing either criterion tends to favor models satisfying both.The paper links lower mutual information between sensitive attributes and predictions with more aligned error-rate parity.
- 6.2 Benchmarking Fairness Processors: Fairness processors generally reduce profit relative to the unconstrained model, while six of eight improve independence and five improve separation.Sufficiency is consistently higher for the unconstrained model, and its agreement with profit suggests profit maximization is compatible with maximizing sufficiency.
- 6.2 Benchmarking Fairness Processors: Reject option classification achieves the strongest independence and separation improvements, but sacrifices more than 30% profit; the prejudice remover reduces profit by less than 5%.The prejudice remover has similar AUC to the unconstrained model but delivers smaller fairness gains than other processors.
- 6.2 Benchmarking Fairness Processors: The prejudice remover offers a better profit–fairness trade-off than the disparate impact remover, while reject option classification is the strongest post-processor for fairness.In-processors are more flexible through meta-parameters but more invasive; post-processors are easier to deploy and may reduce profitability substantially.
- 6.3 The Cost of Fairness: Reducing separation below 0.2 is possible while sacrificing less than e0.01 profit per EUR issued, averaging a 4.91% profit reduction.Completely eliminating unfairness requires separation of 0 and sacrificing more than 35% of profit.
7 Conclusion
The paper combines conceptual analysis and empirical benchmarking to identify suitable fairness criteria and processors for profit-oriented credit scoring. It finds that separation is especially appropriate, in-processors balance profit and fairness, and moderate discrimination reductions can be achieved at relatively low cost.
- Separation is identified as the most appropriate fairness metric for credit scoring because it acknowledges lending’s imbalanced misclassification costs.
- Separation and independence are strongly correlated empirically, while sufficiency may be achievable by any well-trained classifier that predicts the sensitive attribute from other features.
- Reject option classification is easiest to implement but incurs high monetary costs, whereas the prejudice remover offers the best profit-fairness trade-off and greatest calibration flexibility.
- Pareto-frontier analysis enables decision-makers to assess context-specific profit-fairness trade-offs and identify suitable techniques.
- The catalog and empirical findings may also inform fairness interventions in hiring, college admission, and criminal risk assessment.
A Overview of Fairness Criteria
The appendix organizes numerous fairness concepts around independence, separation, and sufficiency, while noting that the paper’s framework is group-based rather than individual or counterfactual.
- Independence, separation, and sufficiency serve as organizing criteria for deriving equivalent or relaxed forms of 19 fairness concepts.
- The paper’s fairness criteria all embody group-based fairness.
B Meta-Parameters of Base Models and Fairness Processors
The empirical experiment tunes fairness processors and base classifiers within a cross-validation framework, using expected monetary profit as the optimization objective.
- Base-classifier meta-parameters are optimized by grid search with EMP as the objective, separately for each relevant training combination.
- Fairness-processor meta-parameters are tuned within a higher-level cross-validation framework using EMP on each validation fold.
- Tables 7 and 8 report candidate meta-parameter values for fairness processors and four base classifiers, respectively.
C Extended Empirical Results
The extended results compare fairness processors and an unconstrained scorecard across seven credit-scoring datasets using predictive performance, profit, and fairness measures.
- Performance is evaluated using AUC, profit per EUR issued, acceptance rate, independence, separation, and sufficiency.
- Pre- and post-processor results are averaged over 25 combinations of five cross-validation folds and five base classifiers.
- In-processor results are aggregated over five training-fold combinations.
- The unconstrained profit-maximization benchmark is reported alongside fairness-processor results for each dataset.
- The extended tables cover the German, Bene, Taiwan, UK, PAKDD, GMSC, and Homecredit datasets.