Source-linked AI summary
Estimating the Prediction Performance of Spatial Models via Spatial k-Fold Cross Validation
Jonne Pohjankukka, Tapio Pahikkala, Paavo Nevalainen, Jukka Heikkonen
TL;DR
Spatial autocorrelation violates the independence assumption of ordinary cross validation, making spatial-model performance estimates optimistically biased. The paper proposes spatial k-fold cross validation (SKCV), which removes geographically close training points, and evaluates it across three GIS-based applications. Ordinary CV estimates were up to 40% more optimistic than SKCV, which also supports sampling-density selection for new research areas.
Problem
Standard cross validation assumes independent data, but GIS datasets can contain spatial or temporal autocorrelation that biases spatial-model performance estimates.
Method
SKCV modifies cross validation by excluding training points within a specified spatial or temporal distance of each test set.
Results
Ordinary CV estimates were up to 40% more optimistic than SKCV estimates across experiments involving regression and classification models.
Takeaways & Limitations
SKCV estimates spatial prediction performance without optimistic bias due to SAC and can guide sampling-density selection for new research areas.
Takeaways & Limitations
Very small fold counts can remove most training data, so fold selection must suit the application; K = M is preferred when feasible.
Abstract
from arXiv · showhide
In machine learning one often assumes the data are independent when evaluating model performance. However, this rarely holds in practise. Geographic information data sets are an example where the data points have stronger dependencies among each other the closer they are geographically. This phenomenon known as spatial autocorrelation (SAC) causes the standard cross validation (CV) methods to produce optimistically biased prediction performance estimates for spatial models, which can result in increased costs and accidents in practical applications. To overcome this problem we propose a modified version of the CV method called spatial k-fold cross validation (SKCV), which provides a useful estimate for model prediction performance without optimistic bias due to SAC. We test SKCV with three real world cases involving open natural data showing that the estimates produced by the ordinary CV are up to 40% more optimistic than those of SKCV. Both regression and classification cases are considered in our experiments. In addition, we will show how the SKCV method can be applied as a criterion for selecting data sampling density for new research area.
1. Introduction
Spatial autocorrelation violates the independence assumption behind standard cross validation for GIS data, biasing spatial-model performance estimates. The paper proposes SKCV to estimate prediction performance under these violations and to guide sampling density in new research areas.
- Motivation: GIS data commonly exhibit spatial or temporal autocorrelation, so standard k-fold cross validation’s independence assumption does not hold.Nearby geographic observations are more related than distant observations, making spatial dependence especially relevant to prediction evaluation.
- Motivation: Spatial autocorrelation can produce biased model evaluation and complicate applications such as forestry route selection and natural-resource management.In forestry, predictions of soil carrying capacity support route selection intended to reduce the risk of harvester sinking.
- Contribution: The paper proposes spatial k-fold cross validation (SKCV) to estimate prediction performance without optimistic bias caused by spatial autocorrelation.SKCV omits training samples geographically too close to test data.
- Research questions: SKCV asks how prediction performance changes with the geographical distance to the nearest training measurements.This frames spatial prediction performance in terms of separation between in-sample and out-of-sample data.
- Research questions: SKCV also addresses how densely a new research area should be sampled to achieve a required prediction-performance level while considering data-collection costs.The experiments include soil permeability regression and forest-track-damage classification using public GIS-based data.
2. Spatial k-fold cross validation
SKCV modifies ordinary cross validation by excluding training observations within a specified spatial or temporal dead zone around each test fold. Its design preserves ordinary CV as a special case while exposing practical trade-offs from data removal and fold choice.
- Method: SKCV prevents optimistic performance estimates by retaining only training points at least a specified spatial or temporal distance from the test data.The excluded region is controlled by the dead zone radius rδ.
- Procedure: At each fold, SKCV reduces the training set before fitting the model and predicting the held-out test points.The reduction removes training observations inside the test points’ dead-zone perimeters.
- Special cases: When rδ = 0, SKCV becomes ordinary cross validation; when K = M, it becomes spatial leave-one-out cross validation.The prediction model’s choice does not affect SKCV’s functionality.
- Bias assessment: SKCV-RLO estimates the extra pessimistic bias from reducing training-set size by randomly removing the same number of observations.Its results were more optimistic than SKCV, and the authors report that random-removal performance decreases were negligible compared with SAC-related decreases.
- Limitations and design choices: Very small fold counts can remove most training data because combined test-fold dead zones may have a large effective radius.Fold selection is application-specific; K = M is preferred when computational resources allow it.
3. Data sets
The paper evaluates three spatial prediction cases that differ in data availability, density, resolution, and response measurements. The datasets combine natural-environmental variables from raster and vector sources with water-permeability and harvester-damage responses.
- Data coverage: The three cases represent continuous, scattered, and clustered spatial data availability across typical prediction applications.Case 1 covers the research area except obstacles, case 2 has scattered field measurements, and case 3 contains data clustered into several areas.
- Data formats: The datasets use TIFF raster images and ASCII vector files with different grid resolutions.Table 1 summarizes the response datasets, formats, and grid-resolution sizes used across the three cases.
- Data coverage: Case 1 contains 361201 data points and is much denser than the datasets in cases 2 and 3.Figure 3 reports 1691 points for case 2 and 11795 for case 3.
- Response data: Soil-moisture measurements were obtained gravimetrically by comparing dry and wet sample weights.The measurement followed ASTM D2216-10 (2010).
- Response data: The response data include water permeability measurements and three harvester-track damage classes.Water permeability is represented by a logarithmic exponent derived from permeability speed; damage classes are no, slight, and damage.
4. Experimental analysis with SKCV
Experiments apply SKCV to three GIS prediction cases, examining SAC-aware performance estimates and whether SKCV can guide sampling density for new areas.
- Experimental cases: The experiments cover two water-permeability regression cases and one forest-harvester track-damage classification case using kNN models.The cases differ in data availability and spatial resolution, while kNN is used as the prediction model throughout.
- Case 1: Water permeability: Case 1 contains 361201 data points across an approximately 144 km2 target area, with 49 predictor features including derived features.The water-permeability exponent is the response variable, and Figure 5 confirms SAC in it.
- Case 1: Water permeability: SKCV estimates in case 1 show prediction error (RMSE) rising clearly as the dead zone radius rδ increases, while sparser data reduce its effect.This pattern is consistent with the SAC identified through semivariogram and Moran’s I analysis.
- Case 1: Water permeability: SKCV-RLO is less sensitive than SKCV to removing training points and gives more optimistic results across all spatial densities.The comparison indicates that geographically close training points are preferred, whereas random removal changes prediction accuracy negligibly relative to SAC-based removal.
- Case 1: Sampling-density evaluation: Across 72 area pairs, SKCV is pessimistically biased for rδ ∈[0, 150] meters, almost unbiased for rδ ∈[150, 340] meters, and optimistically biased for rδ ∈[340, 400] meters.Results are stable across spatial densities, which mainly shift the estimates by a constant value.
- Case 2: Field measurements: In case 2, SLOO performance drops radically after approximately 40-50 meters, while SLOO-RLO exhibits a high optimistic bias.The result provides another observed indication of SAC effects in spatial prediction estimates.
- Case 3: Track-damage classification: In case 3, SLOO results decay exponentially with increasing rδ, SLOO-RLO results remain almost unchanged, and the worst-case difference is approximately 40%.This case evaluates classification of forest-harvester track damage.
5. Conclusion
The paper demonstrates that SKCV accounts for SAC when estimating spatial-model prediction performance and can guide sampling-density selection.
- SKCV estimates spatial-model prediction performance without the optimistic bias due to SAC, whereas ordinary CV can produce highly optimistic estimates.
- SKCV can serve as a sampling-density selection criterion for new research areas, reducing data-collection costs.