Source-linked AI summary
Predicting into unknown space? Estimating the area of applicability of spatial prediction models
Hanna Meyer, Edzer Pebesma
TL;DR
Spatial prediction models are frequently applied to environments that differ from their training data, making the scope of cross-validation error uncertain. The paper proposes an area of applicability based on a model-weighted dissimilarity index and cross-validation-aware thresholding. Simulations support the .95 DI quantile as a threshold whose within-AOA prediction error is comparable to cross-validation error.
Problem
Spatial prediction models are applied beyond sampled locations, but common validation and uncertainty estimates do not identify areas whose environmental conditions lack support in the training data.
Method
The method computes a model-weighted dissimilarity index from minimum predictor-space distance and defines the area of applicability by thresholding training-data DI values according to the cross-validation folds.
Results
.95 DI quantile produced an area whose prediction errors were on average similar to the cross-validation error, while cross-validation error should not be considered valid outside it.
Takeaways & Limitations
The AOA complements validation metrics by identifying spatial areas where model predictions are expected to have comparable error and by indicating environments needing additional sampling.
Takeaways & Limitations
The DI–true-error relationship may be weaker for weak prediction tasks because DI does not capture uncertainty from poor predictor ability to model the response.
Abstract
from arXiv · showhide
Predictive modelling using machine learning has become very popular for spatial mapping of the environment. Models are often applied to make predictions far beyond sampling locations where new geographic locations might considerably differ from the training data in their environmental properties. However, areas in the predictor space without support of training data are problematic. Since the model has no knowledge about these environments, predictions have to be considered uncertain. Estimating the area to which a prediction model can be reliably applied is required. Here, we suggest a methodology that delineates the "area of applicability" (AOA) that we define as the area, for which the cross-validation error of the model applies. We first propose a "dissimilarity index" (DI) that is based on the minimum distance to the training data in the predictor space, with predictors being weighted by their respective importance in the model. The AOA is then derived by applying a threshold based on the DI of the training data where the DI is calculated with respect to the cross-validation strategy used for model training. We test for the ideal threshold by using simulated data and compare the prediction error within the AOA with the cross-validation error of the model. We illustrate the approach using a simulated case study. Our simulation study suggests a threshold on DI to define the AOA at the .95 quantile of the DI in the training data. Using this threshold, the prediction error within the AOA is comparable to the cross-validation RMSE of the model, while the cross-validation error does not apply outside the AOA. This applies to models being trained with randomly distributed training data, as well as when training data are clustered in space and where spatial cross-validation is applied. We suggest to report the AOA alongside predictions, complementary to validation measures.
1 Introduction
Spatial prediction models are often extrapolated beyond sampled locations into environmental conditions not represented in training data. The paper motivates estimating an area of applicability (AOA) where cross-validation error remains relevant, using predictor dissimilarity weighted by model importance.
- Motivation: Spatial mapping models trained on field data are commonly used to predict across entire areas, including locations beyond the training samples.Machine-learning methods are especially prevalent because they capture nonlinear and complex relationships.
- Validation gap: Cross-validation provides general error estimates, but spatial dependence and clustered sampling affect how reliably those estimates describe spatial predictions.Spatial cross-validation is used to reduce sensitivity to these sampling and dependency patterns.
- Motivation: Predictions become unreliable when target locations contain environmental-variable combinations not covered by the training data.Such unsuitable environments may be obvious or difficult to identify through expert knowledge alone.
- Validation gap: Common ensemble-based uncertainty estimates describe prediction variation but do not account for missing knowledge about environmental conditions outside the training data.This limitation applies to ensemble variation and quantile-based prediction intervals discussed for spatial mapping.
- Proposed direction: The proposed AOA uses a dissimilarity index based on minimum predictor-space distance to training data, with predictors weighted by their model-derived importance.The AOA threshold is based on training-data DI values calculated according to the cross-validation strategy.
2 Methods
The method quantifies how dissimilar prediction locations are from training data using standardized, importance-weighted predictor distances, then defines the area of applicability from cross-validation-aware dissimilarity thresholds. It evaluates candidate thresholds with simulated spatial prediction tasks.
- Dissimilarity index: The dissimilarity index (DI) is a unitless measure of a new location’s difference from training data, based on its minimum weighted predictor-space distance.DI values increase as the nearest training point becomes more distant; values above 1 indicate above-average dissimilarity among training pairs.
- Predictor preprocessing: Predictors are standardized using training-data means and standard deviations before their distances are weighted by model-derived variable importance.This gives more influential predictors a greater effect on the dissimilarity measure.
- Dissimilarity calculation: For each prediction location, the method calculates distance to the nearest training point and divides it by the average pairwise distance among training data to obtain DI.The Euclidean distance is computed in the normalized, weighted predictor space.
- Area of applicability: The area of applicability (AOA) contains locations whose DI does not exceed a threshold, representing areas where the model’s cross-validation error is expected to apply on average.The threshold is derived from training-point DI values calculated using distances to points outside the same cross-validation fold.
- Threshold evaluation: Candidate DI thresholds were evaluated across 972 simulated spatial prediction realities using varied response functions, sample sizes, and random sampling replicates.The tested cutoffs were the .25, .50, .90, .95, .99, and 1. quantiles of cross-validation-aware training DI values.
3 Results
Across 972 simulations and case-study settings, the .95 DI quantile provided an AOA where prediction error matched cross-validation error, while errors outside the AOA were substantially larger. The case study also showed that importance-weighted DI tracked true prediction error better than Random Forest prediction standard deviations.
- 3.1 Threshold estimation for the AOA: A .95 DI quantile threshold produced a near-zero average difference between cross-validation error and prediction error across 972 simulations.The average mean difference was 0.002.
- 3.1 Threshold estimation for the AOA: Within the AOA, prediction error agreed closely with cross-validation error, whereas outside the AOA prediction RMSE was considerably higher.This pattern was observed across the 972 model scenarios.
- 3.2 Case Study: Importance-weighted DI reflected the spatial pattern of true absolute prediction error with correlation r=0.71, compared with r = 0.62 without variable weighting.Random Forest prediction standard deviations showed different spatial patterns that did not agree with the true error.
- 3.2 Case Study: Using the .95 DI quantile gave a threshold of 0.59; within the AOA, agreement was r=0.97 with RMSE=0.07, versus r=0.16 with RMSE=0.38 outside it.The within-AOA prediction error was in high agreement with the random cross-validation RMSE of 0.08.
- 3.2 Case Study: With clustered training data, spatial cross-validation produced RMSE 0.039 versus 0.021 for random cross-validation, and each corresponding AOA had comparable true prediction RMSE.The true prediction RMSE was 0.043 for the spatial-model AOA and 0.024 for the random-model AOA.
4 Discussion
The AOA links a model’s expected prediction-error range to environmental similarity with cross-validation data. Its interpretation depends on cross-validation and sampling choices, and several methodological assumptions remain open for study.
- The AOA denotes areas where expected prediction error is comparable to the model’s cross-validation error.
- The .95 DI quantile produced AOA errors on average similar to cross-validation error across 972 simulations.
- AOA relevance is greatest for heterogeneous areas or transfers to environments insufficiently represented by field data.
- Ensemble prediction standard deviations do not capture uncertainty caused by missing environmental knowledge or define the AOA.
- For weak prediction tasks, DI relates less strongly to true error because predictor–response modelling ability contributes uncertainty not represented by DI.
- The approach remains a first attempt: weighted distances lack formal statistical justification, Euclidean distance is transformation-sensitive, and nearest-neighbor distance ignores local density.
- AOA depends on the sampling design and cross-validation strategy, which determine the DI threshold and the error estimate it represents.
- The .95 threshold recommendation is reserved because it was derived from similarly designed simulations and needs broader systematic testing.
5 Conclusions
The paper proposes reporting the AOA alongside prediction maps and validation measures because predictions outside it may be unreliable when environmental properties differ strongly from training data.
- The AOA identifies where expected prediction error is comparable to the model’s cross-validation error.
- Predictions outside the AOA should be handled with extreme care or excluded because their environmental properties differ too strongly from training data.
- Reporting the AOA can help avoid decision-making misplanning and propagation of large errors into subsequent models.
- The authors recommend providing the AOA alongside prediction maps and cross-validation performance measures.
Data availability
The AOA estimation methodology and its simulation studies are publicly available for reuse and modification.
- The AOA methodology is implemented in the R package CAST, and the simulation studies are reproducible through published R-markdown scripts.