Source-linked AI summary
Importance of spatial predictor variable selection in machine learning applications -- Moving from data reproduction to spatial prediction
Hanna Meyer, Christoph Reudenbach, Stephan Wöllauer, Thomas Nauss
TL;DR
Spatial machine-learning models may reproduce clustered training data while failing to predict elsewhere because spatial dependencies and autocorrelated predictors are overlooked. Using Random Forests in LULC and LAI case studies, the paper compares spatial with random validation and evaluates spatial predictor selection. It finds that reliable spatial prediction requires accounting for spatial dependencies in both validation and predictor selection.
Problem
Ecological spatial prediction often overlooks spatial dependencies, although clustered samples violate the i.i.d. assumption and random cross-validation can overestimate mapping performance.
Method
The study compares random and spatial cross-validation for Random Forest models predicting LULC and LAI, while evaluating spatial predictor selection.
Results
Reliable spatial predictions require spatial dependencies to be considered during both model validation and selection of appropriate predictor variables.
Takeaways & Limitations
Spatial validation and spatial variable selection should both be considered when producing ecological spatial predictions.
Takeaways & Limitations
Backward recursive feature elimination can retain overfitting predictors because its importance scores are calculated only on training subsets.
Abstract
from arXiv · showhide
Machine learning algorithms find frequent application in spatial prediction of biotic and abiotic environmental variables. However, the characteristics of spatial data, especially spatial autocorrelation, are widely ignored. We hypothesize that this is problematic and results in models that can reproduce training data but are unable to make spatial predictions beyond the locations of the training samples. We assume that not only spatial validation strategies but also spatial variable selection is essential for reliable spatial predictions. We introduce two case studies that use remote sensing to predict land cover and the leaf area index for the "Marburg Open Forest", an open research and education site of Marburg University, Germany. We use the machine learning algorithm Random Forests to train models using non-spatial and spatial cross-validation strategies to understand how spatial variable selection affects the predictions. Our findings confirm that spatial cross-validation is essential in preventing overoptimistic model performance. We further show that highly autocorrelated predictors (such as geolocation variables, e.g. latitude, longitude) can lead to considerable overfitting and result in models that can reproduce the training data but fail in making spatial predictions. The problem becomes apparent in the visual assessment of the spatial predictions that show clear artefacts that can be traced back to a misinterpretation of the spatially autocorrelated predictors by the algorithm. Spatial variable selection could automatically detect and remove such variables that lead to overfitting, resulting in reliable spatial prediction patterns and improved statistical spatial model performance. We conclude that in addition to spatial validation, a spatial variable selection must be considered in spatial predictions of ecological data to produce reliable predictions.
1. Introduction
Ecological spatial prediction often uses flexible machine-learning models while overlooking spatial dependencies, risking validation that measures training-data reproduction rather than prediction beyond sampled locations. The study investigates whether spatial validation and predictor selection can improve reliable mapping.
- Spatial dependencies: Spatial and temporal dependencies violate the assumption that ecological samples are identically and independently distributed.This is especially relevant when ground-truth data are sampled in spatial clusters.
- Validation problem: Random cross-validation can produce considerably overoptimistic error estimates because of spatial autocorrelation.Random splitting may validate reproduction of sampling data without assessing spatial mapping performance.
- Predictor selection: Highly autocorrelated geolocation predictors can be misinterpreted by flexible algorithms, causing failures beyond training locations.Coordinates, elevation, distances, and their derivatives are examples of geolocation predictors.
- Study aim: The authors propose automatic removal of predictors that are counterproductive to spatial mapping.The stated aim is to provide scientifically valuable spatial prediction results.
- Study design: The study examines LULC classification and LAI modelling with Random Forests, comparing spatial and non-spatial cross-validation and applying spatial variable selection.Spectral, terrain-related, and geolocation variables are considered as potential predictors in both case studies.
2. Methods
The study’s processing and modelling were conducted in R Version 3.4, with scripts made publicly retrievable.
- Implementation: Data processing and modelling were performed in R Version 3.4.The analysis scripts can be retrieved from the authors’ GitHub repository.
2.1. Prediction task I: Land use/land cover classification
The first case study classifies ten land-use/land-cover classes around the Marburg Open Forest using aerial imagery, manually digitized reference polygons, and spectral, terrain-related, and geolocation predictors.
- Study area: The LULC study covers approximately 3000 x 2500 m around the Marburg Open Forest in Hessen, Germany.An aerial image provides the basis for classification.
- Reference data: Ten LULC classes were assigned from manually digitized polygons selected through image inspection and field knowledge.The polygons provide reference data for classification-model training.
- Spatial validation: Figure 1 combines the true-color study-area image, LULC training polygons, and the spatial-block grid used for spatial cross-validation.The reference system is UTM 32N (WGS84).
- Predictor variables: The aerial-image predictors include red, green, and blue channels, vegetation indices, and a principal-component-derived variable.The image was resampled to 1 m spatial resolution before predictor preparation.
- Training data: Each pixel within or intersecting a reference polygon was treated as an individual training sample, producing approximately 185000 samples.Every sample contained all potential predictors and its polygon-based LULC class.
2.2. Prediction task II: Leaf Area Index modelling
The second case study models forest LAI using lidar-derived reference data and Sentinel-2 predictors, supplemented by terrain and geolocation variables.
- Prediction task: The LAI task models forested areas of the Marburg Open Forest as an environmental regression problem.The study uses the same research site as the LULC case study.
- Reference data: Lidar data from the 2010 vegetation period provide the LAI reference, calculated from the lidar point cloud.The reference has 15 cm vertical and 30 cm spatial accuracy, despite the time lag from Sentinel-2 predictors.
- Spectral predictors: A Sentinel-2 Level-1C scene acquired on 2017/05/10 supplies the spectral predictor variables.The study excludes bands 1, 9, and 10 because they lack relevant information for this task.
- Predictor variables: The LAI predictor set combines Sentinel-2 channels with elevation, slope, aspect, latitude, and longitude, totaling 15 variables.The non-10 m channels were resampled to match the 10 m channel geometry.
- Training data: Training samples contain predictor values extracted at their locations together with lidar-based LAI values.These samples support model fitting for the LAI prediction task.
2.3. Model training and prediction
The study compares random and spatial validation with several variable-selection strategies for Random Forest models predicting land cover and LAI. Spatial forward feature selection is designed to identify predictors that improve spatial performance rather than merely reproduce clustered training data.
- Model setup: Random Forest models were trained for land-cover classification and LAI prediction using spectral, terrain-related, and geolocation predictors.The response was derived from training polygons for land cover and lidar-derived LAI values for LAI prediction.
- Model comparison: Models using all predictors were evaluated with random and spatial cross-validation to assess how validation strategy changes estimated performance.The study also compared models using variables selected by recursive feature elimination or forward feature selection.
- Cross-validation strategies: Random cross-validation assigns observations randomly to folds, whereas spatial cross-validation assigns them according to spatial location and repeatedly validates on held-out folds.The spatial approach used blocks or clusters, with the number of folds matching the number of spatial units for each case study.
- Variable selection: Forward feature selection tests predictor combinations and adds variables when they improve performance under the specified cross-validation scheme.Spatial FFS was used to identify variables that support spatial mapping, while random FFS provided a comparison based on random validation.
- Variable selection: Recursive feature elimination can retain overfitting predictors because its variable-importance scores are calculated on training subsets rather than spatially held-out data.The authors therefore argue that forward selection combined with spatial cross-validation is required to address this issue.
3. Results
Random validation produced very strong apparent performance, but spatial validation revealed substantially weaker predictive performance for both tasks. Spatial FFS removed geolocation variables, improved spatial metrics slightly, and eliminated visible prediction artefacts.
- 3.1. Statistical performance: Accuracy and Kappa exceeded 0.99 under default random cross-validation for land-cover classification, but spatial validation yielded Kappa = 0.55.For LAI, random validation gave RMSE = 0.96 and R2 = 0.87, versus spatial-validation RMSE = 1.25.
- 3.1. Statistical performance: RFE with spatial validation and FFS with random validation failed to prevent spatial overfitting despite retaining high random-validation performance.RFE produced LULC Kappa = 0.55 and LAI RMSE = 1.22 spatially, while random FFS produced Kappa = 0.14 and RMSE = 1.23 spatially.
- 3.1. Statistical performance: Spatial FFS slightly improved spatial performance to Kappa = 0.56 for LULC and RMSE = 1.20 for LAI compared with the other models.LAI global R2 also increased from 0.58 with all variables to 0.63 with spatial variable selection.
- 3.2. Variable importance and selected variables: Latitude, longitude, and elevation were the most important predictors in full models, while spectral predictors were considerably less important.Random FFS also selected geolocation and terrain variables prominently for both tasks.
- 3.2. Variable importance and selected variables: Spatial FFS identified geographic coordinates and elevation as irrelevant or counterproductive and retained mainly spectral predictors plus slope for LULC.For LAI, it selected bands B05, B07, B03, and B8A, with B03 and B8A adding only slight RMSE reductions beyond B05 and B07.
- 3.2. Variable importance and selected variables: Full and randomly selected models produced linear spatial artefacts, whereas spatial selection produced locally variable predictions driven by spectral information without obvious artefacts.The same pattern appeared for LAI: geolocation-driven linear features disappeared after spatial FFS.
4. Discussion
The discussion shows that spatial cross-validation is necessary but insufficient: misleading, spatially autocorrelated predictors can produce strong apparent performance while undermining spatial prediction. Spatial variable selection and visual assessment help identify and remove these problems.
- Importance of spatial validation: Random cross-validation produces overoptimistic estimates and does not assess predictions beyond the training locations.This is especially problematic for clustered samples and spatially autocorrelated predictor and response variables.
- Relevance of spatial variable selection: Geolocation variables such as latitude and longitude can receive high importance because clustered training samples are not independent, leading to overfitting.The resulting model may reproduce training data while failing to predict elsewhere.
- Relevance of spatial variable selection: Spatial forward feature selection removed elevation when it was identified as unimportant or counterproductive, with improved visual and statistical results.
- Relevance of spatial variable selection: Spatial cross-validation provides reliable performance measures but does not alter the variables selected during Random Forest training.Because internal selection relies on randomly bootstrapped out-of-bag error, misleading predictors can remain in the model.
- Relevance of spatial variable selection: Recursive feature elimination may retain misleading predictors because overfitting variables can rank highly in the Random Forest importance ranking.Latitude and longitude are examples of highly ranked variables that may not support spatial prediction.
- Relevance of spatial variable selection: Removing misleading variables improved statistical spatial performance and produced more reliable spatial patterns than the model using all variables.The increase in statistical performance was less obvious than in a comparable spatio-temporal study with stronger autocorrelation.
- Relevance of spatial variable selection: Spatial cross-validation should guide model validation, tuning, and variable selection rather than being limited to final performance estimation.
- Need for visual assessment in addition to statistical validation: Statistical spatial validation alone can miss substantially different prediction patterns produced by models with and without misleading variables.Visual assessment revealed artefacts, including linear features associated with geolocation predictors in related applications.
5. Conclusions
The conclusions argue that spatial dependencies must be addressed throughout spatial machine-learning workflows. Otherwise, models may reproduce training data yet fail to produce reliable predictions at other locations.
- 5. Conclusions: Spatial validation strategies are particularly important when spatial autocorrelation is strong and training samples are spatially clustered.
- 5. Conclusions: Spatial validation should be used throughout modelling, including hyperparameter tuning, variable selection, and performance estimation.Spatial variable selection is needed to remove predictors that cause overfitting, especially geolocation variables in clustered samples.
- 5. Conclusions: The study suggests revisiting the assumption that Random Forests require no assumptions about data distributions and developing general guidelines for more objective applications.
- 5. Conclusions: Ignoring spatial dependencies risks models that reproduce training data well but make unreliable spatial predictions.