Source-linked AI summary
Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data
Patrick Schratz, Jannes Muenchow, Eugenia Iturritxa, Jakob Richter, Alexander Brenning
TL;DR
Ecological model evaluation must account for biased performance estimates, hyperparameter tuning, and spatial autocorrelation. The study compares six models using nested spatial and non-spatial cross-validation for Diplodia sapinea distribution in Spain. GAM and RF show the best predictive accuracy, while the authors recommend spatial partitioning during hyperparameter tuning for spatial data.
Problem
Spatial autocorrelation can make non-spatial resampling performance estimates overoptimistic, while default hyperparameters may not ensure optimal machine-learning performance.
Method
The study compares six statistical and machine-learning models using cross-validation approaches with spatial partitioning and hyperparameter tuning.
Results
GAM and RF showed the best predictive accuracy in the Diplodia sapinea case study, with all machine-learning models outperforming parametric models.
Takeaways & Limitations
Spatial partitioning should also be used for cross-validation hyperparameter tuning when modeling spatial data.
Takeaways & Limitations
The reported methodological conclusions are based on a test case focused on predictive performance.
Abstract
from arXiv · showhide
Machine-learning algorithms have gained popularity in recent years in the field of ecological modeling due to their promising results in predictive performance of classification problems. While the application of such algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages such as R, there are several practical challenges in the field of ecological modeling related to unbiased performance estimation, optimization of algorithms using hyperparameter tuning and spatial autocorrelation. We address these issues in the comparison of several widely used machine-learning algorithms such as Boosted Regression Trees (BRT), k-Nearest Neighbor (WKNN), Random Forest (RF) and Support Vector Machine (SVM) to traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM). Different nested cross-validation methods including hyperparameter tuning methods are used to evaluate model performances with the aim to receive bias-reduced performance estimates. As a case study the spatial distribution of forest disease Diplodia sapinea in the Basque Country in Spain is investigated using common environmental variables such as temperature, precipitation, soil or lithology as predictors. Results show that GAM and RF (mean AUROC estimates 0.708 and 0.699) outperform all other methods in predictive accuracy. The effect of hyperparameter tuning saturates at around 50 iterations for this data set. The AUROC differences between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) performance estimates of the GAM and RF are 0.167 (24%) and 0.213 (30%), respectively. It is recommended to also use spatial partitioning for cross-validation hyperparameter tuning of spatial data.
1. Introduction
Ecological model comparisons face intertwined challenges in estimating predictive performance, tuning hyperparameters, and handling spatial autocorrelation. This study compares spatial and non-spatial validation strategies across six statistical and machine-learning models.
- Contribution: The analysis addresses critical issues in ecological model comparison and provides complete code to support reproducibility and wider methodological adoption.The paper presents the work as an exemplary model-comparison study for spatial data.
- Motivation: Spatial autocorrelation can make regular random-resampling performance estimates biased and overoptimistic.Spatial resampling is adopted to reduce this bias.
- Motivation: Hyperparameter defaults cannot guarantee optimal machine-learning performance, making tuning an important evaluation step.Cross-validation is also used during tuning, raising questions about whether non-spatial techniques suit spatial data.
- Study aims and approach: The study compares spatial cross-validation with random resampling and approaches without hyperparameter tuning to obtain bias-reduced performance estimates.The methodology includes spatial hyperparameter tuning.
- Study aims and approach: Six commonly used models are evaluated: BRT, GAM, GLM, WKNN, RF, and SVM.The comparison covers statistical and machine-learning techniques for supervised predictive modeling.
2. Data and study area
The case study models Diplodia sapinea infection in Basque Country trees using environmental, soil, lithology, and tree-level predictors. The dataset contains 926 observations after removing records with missing information.
- Predictors: Eleven predictors include temperature, precipitation, solar radiation, elevation, slope, hail-damage potential, tree age, soil pH, soil type, lithology, and survey year.Predictors combine numeric and nominal variables from multiple environmental and survey sources.
- Data: The response variable is binary tree infection by the fungal pathogen Diplodia sapinea, with infected and non-infected trees occurring at roughly a 1:3 ratio.The sample includes 223 infected and 703 non-infected trees.
- Predictor preparation: Hail damage was represented as a binary predictor, while hail-damage potential was spatially predicted from climatic variables using a GAM.Soil and pH information were also spatially modeled or mapped from external datasets.
- Data: The dataset contains 926 observations after three records were removed for missing variable information.The methodology is presented for binary classification and can also be adapted to multiclass or quantitative responses.
3. Methods
The study combines nested cross-validation, spatial partitioning, and hyperparameter tuning to compare machine-learning and statistical models for spatial predictive modeling.
- Study design: The study compares RF, SVM, WKNN, BRT, GLM, and GAM using nested cross-validation and spatial performance estimation.Four setups combine spatial or non-spatial performance estimation with spatial or non-spatial tuning, plus spatial estimation without tuning.
- Spatial cross-validation: Spatial cross-validation partitions data into spatially disjoint subsets to reduce the influence of spatial autocorrelation.The spatial approach uses k-means clustering before partitioning.
- Cross-validation procedure: Five-fold partitioning was repeated 100 times for performance estimation, while tuning split each training fold into five additional folds.The procedure was repeated 500 times across five folds and varying random-search iterations to reduce partitioning variance.
- Hyperparameter tuning: Random search evaluated 0, 10, 50, 100, and 200 tuning iterations, selecting the setting with the highest mean AUROC across tuning folds.Hyperparameter tuning was applied to RF, SVM, BRT, and WKNN; GLM has no hyperparameters requiring tuning.
- Performance measure: AUROC was used to assess binary classification performance and combines true-positive and false-positive rates independently of a decision threshold.Values near 0.5 indicate no separation power, whereas 1.0 indicates correct classification of all cases.
- Performance measure: The methodology also supports alternative goodness-of-fit measures suited to binary classification.
4. Results
Results show that hyperparameter tuning benefits some classifiers, while spatial validation reveals substantial optimism in non-spatial performance estimates and identifies RF and GAM among the strongest models.
- Tuning: BRT and SVM improved over default hyperparameters, whereas WKNN and RF showed no relevant AUROC change from tuning.
- Tuning: 50 tuning iterations generally captured nearly all available accuracy gains; additional iterations produced only slight or no improvement.
- Tuning: SVM showed the largest tuning effect, increasing by approximately 0.08 AUROC.
- Predictive performance: RF and GAM achieved the best predictive performance, with a 0.081 mean-AUROC gap separating the best and worst models, RF/GAM and WKNN.
- Predictive performance: 0.661 AUROC for tuned BRT versus 0.587 without tuning, and 0.654 versus 0.574 for SVM, demonstrate clear tuning gains.
- Predictive performance: Non-spatial performance estimates were approximately 24–39% higher than spatial equivalents, indicating overoptimistic validation results.
- Predictive performance: WKNN and RF had the largest spatial-versus-non-spatial differences, at 35% and 39%, while GAM was least affected at 24%.
5. Discussion
The discussion emphasizes computationally practical tuning, spatially consistent validation, and careful specification of search spaces. It also explains how spatial autocorrelation can alter RF hyperparameters and inflate apparent performance.
- Tuning: Random search with at least 50 iterations was sufficient for all considered algorithms, although tuning costs increase with iteration count.
- Tuning: Hyperparameters should generally be tuned because omitting tuning cannot ensure the model achieves its best possible predictive performance on the dataset.
- Tuning: Optimal hyperparameter limits depend on dataset characteristics, so the search space selected here is not universally applicable.
- Spatial validation: Spatial and non-spatial tuning produced no major performance differences, but the authors recommend using spatial cross-validation for both inner tuning and outer estimation.
- Spatial validation: Non-spatial tuning can exploit residual spatial autocorrelation in train/test splits, producing overoptimistic performance estimates.
- Random Forest: RF tuning did not substantially change predictive performance, yet spatial and non-spatial tuning selected different optimal hyperparameters and model structures.
- Random Forest: Spatial tuning commonly selected mtry = 1, reducing RF’s emphasis on spatially autocorrelated predictors and yielding a more general model for heterogeneous datasets.
- Conclusion: Spatial cross-validation is recommended because non-spatial validation would have selected RF over GAM and reported 0.912 rather than 0.708 AUROC.
6. Conclusion
The study compared six statistical and machine-learning models using spatially aware evaluation and recommends matching model choice and cross-validation design to the analytical goal. In the test case, RF and GAM performed best, while spatial cross-validation was favored for bias-reduced estimates.
- Six statistical and machine-learning models were compared in the study.
- RF and GAM showed the best predictive accuracy among the tested models.The conclusion also reports that all machine-learning models outperformed parametric models in this test case.
- Hyperparameter-tuning effects depended on the algorithm and data set, requiring suitable iteration counts and parameter limits.
- The Diplodia sapinea detection accuracy increased by 0.05 AUROC compared with Iturritxa et al. (2014), with hail damage at trees identified as the main driver.
- Spatial cross-validation should replace non-spatial CV for spatial data when estimating predictive performance and tuning hyperparameters.
- Analysis goals should guide model selection: (semi-)parametric models for environmental inference and spatially tuned machine-learning models for highly accurate spatial prediction.
- The authors aim to encourage scientists to report more bias-reduced performance estimates.
Appendix A.1. Random Forest
The Random Forest appendix explains implementation choices centered on runtime and package capabilities. The selected ranger implementation was substantially faster than randomForest under the reported benchmarks.
- The study used ranger for Random Forest because of its fast runtime.
- ranger was up to 25 times faster than randomForest when observations were used as the benchmark criterion.
- ranger was up to 60 times faster than randomForest when the number of trees was the benchmark measure.
- Other Random Forest packages, including RandomForestSRC, bigrf, Random Jungle, and Rborist, fell between the compared implementations.
Appendix A.2. Support Vector Machine
The Support Vector Machine implementation used kernlab because it offered more kernel options than e1071. Other kernels were partly tested but not analyzed in detail.
- kernlab was chosen over e1071 because it provides more kernel options.
- Kernels other than the radial basis function were partly tested but not analyzed in detail.
- The Boosted Regression Trees implementation used the sole known R package available for that method.
Appendix A.4. Generalized Linear/Additive Model
The generalized linear/additive model appendix describes the R implementations and smoothing controls used for GLMs and GAMs. It emphasizes balancing nonlinear flexibility against runtime and convergence constraints.
- Generalized Linear Models: GLMs used the base implementation from R’s stats package.
- Generalized Additive Models: GAMs used mgcv because it supports multiple smoothing-parameter optimization methods and random effects.
- Generalized Additive Models: mgcv allows different smooth terms and limits on predictor nonlinearity.
- Generalized Additive Models: The GAM parameter k limits nonlinearity, with its default upper limit set to k −1 when k is the number of variables.
- Generalized Additive Models: Optimization should ensure k does not reach its upper limit for any optimized term.
- Generalized Additive Models: Restricting predictor smoothness can prevent accurate nonlinearity modeling and reduce model performance.
- Generalized Additive Models: Setting k too high relative to the final smoothing degree increases runtime or can cause convergence problems.
Appendix B. Descriptive summary of numerical and nominal predictor variables
Appendix B summarizes the numerical and nominal predictor variables used in the analysis, including measurement units and descriptive statistics for numerical variables.
- Numerical predictors include precipitation, temperature, solar radiation, and tree age, with their respective measurement units reported.Precipitation is measured in mm/m2, temperature in °C, solar radiation in kW/m2, and tree age in years.
- The numerical-variable summary reports sample size, percentiles, median, mean, maximum, interquartile range, and missing-value count.
- Nominal predictor variables are summarized separately in Table B.4.
Appendix C. Additional hyperparameter tuning results
Appendix C presents fold-specific winning hyperparameter settings from random-search tuning under spatial and non-spatial partitioning setups.
- Figure C.7 shows best hyperparameter settings for 500 folds, each estimated from 400 random-search tuning iterations using five-fold cross-validation.
- The results are split by spatial versus non-spatial partitioning setup and by model type.
- Red crosses mark each model’s default hyperparameter values, while black dots mark the winning setting for each fold.