PaperlayerView exact revision on arXiv

Source-linked AI summary

A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization

Yan Wang, Xuelei Sherry Ni

arXiv:1901.08433v1stat.MLcs.LG

TL;DR

The paper addresses the limited research on combining feature selection and hyperparameter optimization for financial-domain XGBoost risk modeling. It evaluates those choices alongside logistic regression, finding that XGBoost performs significantly better than LR and that TPE outperforms random search on accuracy. The authors present the workflow as a reference for future business-risk modeling studies.

  • Problem

    The paper identifies limited research on the combined effects of feature selection and hyperparameter optimization on XGBoost in the financial domain.

    Source: §1 · p. 2

  • Method

    The study combines feature selection and hyperparameter optimization in XGBoost, compares it with logistic regression, and uses Wilcoxon signed rank tests to assess performance differences.

    Source: §1 · p. 2, §3.6 · p. 8

  • Results

    XGBoost with both random search and TPE significantly outperforms logistic regression on accuracy, AUC, recall, and F1 score; TPE is significantly better than random search.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The authors present the workflow as a reference for future business-risk studies and use XGBoost feature-importance rankings to enhance model interpretation.

    Source: §1 · p. 2, §5 · p. 15

    Results in future business-risk studies might not be consistent when different datasets are used.

    Source: §5 · p. 15

Abstract

from arXiv · show

This paper aims to explore models based on the extreme gradient boosting (XGBoost) approach for business risk classification. Feature selection (FS) algorithms and hyper-parameter optimizations are simultaneously considered during model training. The five most commonly used FS methods including weight by Gini, weight by Chi-square, hierarchical variable clustering, weight by correlation, and weight by information are applied to alleviate the effect of redundant features. Two hyper-parameter optimization approaches, random search (RS) and Bayesian tree-structured Parzen Estimator (TPE), are applied in XGBoost. The effect of different FS and hyper-parameter optimization methods on the model performance are investigated by the Wilcoxon Signed Rank Test. The performance of XGBoost is compared to the traditionally utilized logistic regression (LR) model in terms of classification accuracy, area under the curve (AUC), recall, and F1 score obtained from the 10-fold cross validation. Results show that hierarchical clustering is the optimal FS method for LR while weight by Chi-square achieves the best performance in XG-Boost. Both TPE and RS optimization in XGBoost outperform LR significantly. TPE optimization shows a superiority over RS since it results in a significantly higher accuracy and a marginally higher AUC, recall and F1 score. Furthermore, XGBoost with TPE tuning shows a lower variability than the RS method. Finally, the ranking of feature importance based on XGBoost enhances the model interpretation. Therefore, XGBoost with Bayesian TPE hyper-parameter optimization serves as an operative while powerful approach for business risk modeling.

1. INTRODUCTION

The paper investigates XGBoost for business risk classification, where loan decisions are framed as binary classification. It examines feature selection and hyperparameter tuning together, comparing XGBoost with logistic regression and assessing feature importance.

1. INTRODUCTION

  • Risk modeling supports financial institutions’ decisions about whether to grant loans, framed here as a binary classification task.Logistic regression is commonly used in financial-domain classification for its easy implementation, explainability, and comparable or often better performance than some other binary classifiers.Source: §1 · p. 1
  • Feature selection can alleviate noise and redundant-variable effects, while hyperparameter tuning can help prevent XGBoost failure and reduce overfitting.The paper considers random search and Bayesian tree-structured Parzen Estimator optimization.Source: §1 · p. 2
  • The study examines feature selection and hyperparameter optimization together in XGBoost, comparing its accuracy, AUC, recall, and F1 score with logistic regression.It also uses the Wilcoxon signed rank test to investigate how feature-selection and optimization choices affect performance.Source: §1 · p. 2
  • The paper also ranks features by XGBoost importance scores to enhance model interpretation.The introduction presents this alongside its performance comparisons and analysis of feature-selection and tuning choices.Source: §1 · p. 2

2. ALGORITHMS

This section outlines feature-selection approaches, logistic regression, XGBoost, and the two hyperparameter-optimization strategies used in the study.

2.1. Feature selection methods

  • Four feature-selection methods rank features against the target; Cluster instead groups variables and selects a representative feature within each cluster.Gini, Chi-square, Correlation, and Information use their respective impurity, statistic, correlation, or gain-ratio measures; Cluster selects lower 1-R2-ratio features.Source: §2.1 · pp. 2–3

2.2. Logistic Regression

  • Logistic regression outputs the conditional probability that an observation belongs to a specified class from its input variables and estimated parameters.The section describes LR as a standard binary-classification technique.Source: §2.2 · p. 3

2.3. Extreme gradient boosting along with hyper-parameters

  • XGBoost builds predictions by summing tree-leaf scores and minimizes prediction loss with a regularization term that penalizes model complexity.Tree-structure hyperparameters address overfitting, while learning rate controls the weighting of added trees and slows adaptation to training data.Source: §2.3 · pp. 3–4

2.4. Hyper-parameter optimization methods in XGBoost

  • Random search samples hyperparameters uniformly without using prior boosting results, whereas Bayesian TPE selects promising settings through a probability model.The paper denotes the resulting models XGB_RS and XGB_TPE, respectively.Source: §2.4 · p. 4

3. EXPERIMENTAL DESIGN

The experimental design examines feature-selection methods, XGBoost hyperparameter tuning, comparisons with logistic regression, and important risk-prediction features. It evaluates these questions on a sampled business-risk dataset using cross-validation and statistical tests.

3. EXPERIMENTAL DESIGN

  • The experiments ask which feature-selection methods suit LR and XGBoost, how RS and TPE affect XGBoost, whether XGBoost outperforms LR, and which features predict risk.The study frames these questions for the dataset used.Source: §3 · pp. 4–5

3.2. Data Pre-processing

  • The study uses stratified sampling to select 8,000 observations after retaining 108 of the original 305 variables and removing variables with more than 70% missing data.The source dataset contains over 10 million de-identified U.S. company records from 2006 to 2014, with about 52% risky businesses.Source: §3.2 · p. 5, §3.1 · p. 5

3.3. Searching domain of hyper-parameters in XGBoost

  • Seven XGBoost hyperparameters are optimized within predefined domains, while n_estimators, min_child_samples, and subsample_for_bin retain Python defaults.The optimized parameters are learning rate, subsample, max_leaves, max_depth, gamma, colsample_bytree, and min_child_weight.Source: §3.3 · p. 5

3.4. Performance evaluation criteria

  • Accuracy, AUC, recall, and F1 score evaluate performance, with recall weighted more heavily than precision because false negatives may signify loss.AUC measures discrimination between positive and negative cases; F1 is the harmonic mean of precision and recall.Source: §3.4 · pp. 5–6

3.5. Flowchart of the experiments

  • The workflow uses one fold for validation and nine for training, then applies feature selection, builds LR and tuned XGBoost models, scores validation cases, and evaluates four metrics.Training preprocessing imputes each feature's median and z-score normalizes variables; validation preprocessing uses training-set medians, means, and standard deviations. Feature subsets are fixed at 50 variables for fair comparisons.Source: §3.5 · pp. 6–7

3.6. Statistical significance analysis

  • Ten repetitions of 10-fold cross-validation yield ten average scores per model and feature-selection method; paired Wilcoxon tests compare methods and models, with Bonferroni correction for multiple tests.The nominal significance level is α = 0.1; when comparing LR, XGB_RS, and XGB_TPE, each corrected test uses α/3 = 0.033.Source: §3.6 · pp. 7–8

4. RESULTS

The results identify different best feature selectors for LR and XGBoost and recommend XGB_TPE after model comparisons.

4.1. Parameter settings in FS methods

  • In Cluster feature selection, over 90% of the original dataset’s variation is explained by 50 clusters; one representative feature is selected per cluster, and Gini, Chi-square, Correlation, and Information also use 50 features.Source: §4.1 · p. 8

4.2. Best FS method in XGB_TPE

  • Chi-square leads all four XGB_TPE measures; it and Cluster outperform the other selectors, and Chi-square's AUC advantage over Cluster is significant.Small standard deviations indicate consistent, stable performance across feature-selection methods.Source: §4.2 · pp. 8–10, Figure 3 · p. 9

4.3. Best FS method in XGB_RS

  • Chi-square is selected as the best feature selector for XGB_RS, with effects similar to those observed for XGB_TPE.Source: §4.3 · p. 10

4.4. Best FS method in LR

  • Selector effects are model-dependent; Cluster is chosen for LR because it leads recall despite the lowest accuracy and F1, while Chi-square has the highest accuracy.Source: §4.4 · p. 10, Figure 4 · p. 12

4.5. Final model selection

  • Using each model's best feature selector, XGB_RS and XGB_TPE significantly outperform LR on all four measures; TPE exceeds RS significantly only in accuracy and has lower variability in accuracy, recall, and F1.The comparison covers accuracy, AUC, recall, and F1 score.Source: §4.5 · pp. 10, 11, 13, Figure 5 · p. 12, Table 3 · p. 13

4.6. Rank of variable importance

  • XGB_TPE ranks pctNFChgAccAcc24mon as the most important feature and pctNFPDAmt24mon as less necessary for the model.Higher F scores indicate greater feature importance.Source: §4.6 · p. 13

5. CONCLUSION AND DISCUSSION

The study finds XGBoost with TPE to be its strongest business-risk model, outperforming LR and showing lower variability than random search. Its conclusions also identify feature-selection choices, feature-importance rankings, and dataset dependence as relevant to applying the workflow.

5. CONCLUSION AND DISCUSSION

  • The experiments use commercial information on over 10 million de-identified U.S. companies from 2006–2014, with five feature-selection methods and RS and TPE tuning.The experiments repeat 10-fold cross-validation 10 times.Source: §5 · pp. 13–14
  • XGBoost methods significantly outperform LR across accuracy, AUC, recall, and F1; TPE significantly exceeds random search in accuracy and has lower variability on accuracy, recall, and F1.TPE's advantages over random search on AUC, recall, and F1 are not statistically significant.Source: §5 · p. 14
  • Feature-selection effects depend on model type: Cluster is optimal for LR, while Chi-square is selected for both XGBoost tuning methods.Gini gives LR its lowest recall but an acceptable recall with XGBoost.Source: §5 · p. 14
  • The XGB_TPE feature-importance ranking identifies pctNFChgAccAcc24mon as most important and pctNFPDAmt24mon as having an unclear weight.Figure 6 presents the top 15 features; the paper says its ranking can guide financial institutions' credit-data collection.Source: §5 · p. 14, Figure 6 · p. 14
  • XGBoost is described as capable of handling large-scale data with limited computing resources through parallel and distributed computing, out-of-core computation, and cache-aware learning.Out-of-core computation can use disk space when risk-modeling data exceed main memory.Source: §5 · pp. 14–15
  • Results may not be consistent on different datasets, although the workflow may serve as a reference for future credit-domain studies.The authors also propose it as guidance for jointly exploring feature selection and hyperparameter optimization.Source: §5 · p. 15

Authors

Yan Wang is a Ph.D. candidate in Analytics and Data Science at Kennesaw State University whose research focuses on data mining and machine learning in finance. Xuelei Sherry Ni is a statistics professor and interim department chair at the same university.

Authors

  • Yan Wang is a Ph.D. candidate at Kennesaw State University, with research interests in data mining and machine learning in financial areas.She received an M.S. in Statistics from the University of Georgia and has worked on fraud detection as a data scientist intern.Source: Authors · p. 17
  • Xuelei Sherry Ni is a Professor of Statistics and Interim Chair of the Department of Statistics and Analytical Sciences at Kennesaw State University.She has taught there since 2006 and received M.S. and Ph.D. degrees in Applied Statistics from Georgia Institute of Technology.Source: Authors · p. 17
Loading 1901.08433v1…