SAGE: A sampling-aware global evaluation benchmark for species distribution modeling

1 Introduction

• Community-science occurrence records have uneven spatial coverage and taxonomic representation, with underrepresented species often including those of greatest conservation concern.Less than 7% of Earth’s 5 km grid cells contain any occurrence record.Source: §1 · p. 2
• Species-averaged scores conceal which models serve which species, while measured performance also reflects species-specific sampling conditions.SAGE addresses this evaluation gap by relating performance to species data conditions and reporting it in a structured, disaggregated way.Source: §1 · pp. 2–3
• Existing benchmarks do not combine global coverage, many non-anonymized species, plot-based presence-absence evaluation, and explicit evaluation by species data quality.The cited alternatives each lack at least one of these features.Source: §1 · p. 3
• Random Forests and DeepSDMs perform comparably overall, but DeepSDMs benefit infrequently recorded species only when bias-correction practices are adapted to multi-species training.The reported practices include spatial thinning and reweighting.Source: §1 · p. 4
• SAGE pairs 89,768,872 GBIF occurrences with 53,336 sPlotOpen presence-absence plots for 5,771 non-anonymized plant species.The dataset also pairs samples with 52 environmental predictors and supplies thinned training-data versions.Source: Figure 1 · p. 3, §1 · p. 4
• SAGE groups species by sampling effort and relative prevalence, then reports model performance across sparse or dense sampling and infrequent or frequent recording groups.These properties serve as proxies for how thoroughly a range is sampled and how often a species is recorded where sampling occurs.Source: Figure 1 · p. 3, §1 · p. 4

2.1 Datasets

• SAGE pairs GBIF presence-only training records with sPlotOpen presence-absence evaluation data for 5,771 plant species, using 52 environmental predictors.The evaluation data comprise expert-curated vegetation plots; the training set is based on community-science records.Source: §2.1 · p. 4, Figure 2 · p. 5, §2.1.1 · p. 5

2.1.1 Evaluation Data — sPlotOpen Presence-Absence Records

• After filtering and harmonizing sPlotOpen records, the authors retain 53,336 plots for 5,771 species and split them into validation and test sets.The 80% test split is iteratively stratified to retain at least 20 test occurrences per species; validation data are used for tuning.Source: §2.1.1 · pp. 5–6

2.1.2 Training Data — GBIF Presence-Only Records

• GBIF records are cleaned and restricted to native ranges, then prepared as full, 1 km-aggregated, or subsampled training data.The main subsampled variant uses a threshold of 1,000 samples per species; aggregation and subsampling reduce records and samples while retaining all 5,771 species.Source: §2.1.2 · p. 6

2.1.3 Pseudo-absences; 2.1.4 Predictors

• Training contrasts presences with random or target-group background points, alongside 52 tabular environmental predictors spanning climate, soil, topography, and human influence.Target-group background uses other species’ presences and reflects overall survey effort across space.Source: §2.1.3 · p. 7, §2.1.4 · p. 7

2.2 Evaluation Framework

• Species-level AUROC and AUPRG are evaluated only within each species’ native range to avoid counting vast uninhabitable regions as easy absences.The authors exclude suitable locations outside native ranges and invasive or naturalized populations from this evaluation.Source: §2.2 · p. 7
• Sampling effort measures visited grid cells across a species’ range, while relative prevalence measures how often it is recorded within visited cells; median splits define four groups, with lowest-decile groups also distinguished.The two properties are computed from aggregated presence-only data and capture complementary aspects of data availability.Source: §2.2 · pp. 7–9

2.3 Models; 2.3.1 Single-Species Models; 2.3.3 Computational Setup

• The benchmark selects methods that scale to presence-only data and thousands of species, comparing five established single-species approaches with multi-species DeepSDMs.Single-species models include GLM, GAM, MaxEnt, BRT, and Random Forest; reported training times use different CPU and GPU measures and are not directly comparable.Source: §2.3 · p. 9, §2.3.1 · p. 10, §2.3.3 · p. 11

2.3.2 Multi-Species Deep Learning Models

• The optimized DeepSDM incrementally applies aggregation, subsampling, architecture, and loss choices, using a ResNet with 1,024 hidden dimensions and eight blocks for the main results.The main configuration subsamples to 1,000 occurrences per species and uses frequency-based weighting with target-group background only.Source: §2.3.2 · p. 10

3.1 Overall Model Performance

• Random Forest led the single-species approaches, while the fully optimized DeepSDM achieved performance comparable to Random Forest.The design choices also reduced computation by one order of magnitude relative to baseline DeepSDM; training times across model families are not directly comparable because they used different hardware.Source: §3.1 · p. 11

3.2 Model Performance grouped by Species Data Properties

• DeepSDM performed better for infrequently recorded species, whereas Random Forest was stronger in sparsely sampled ranges, especially for sparse but frequently recorded species.Optimized DeepSDM improved over its baseline in every species group, with the largest gain (+6.7%) for species in the lowest deciles of prevalence and sampling effort.Source: Figure 5 · p. 12, §3.2 · p. 12

3.3 Impacts of the Design Choices on Model Performance

• Aggregation benefited low-effort groups, especially infrequent species in sparsely sampled areas, while reducing performance in densely sampled areas; subsampling improved scores consistently across groups.A 1,000-occurrence subsampling threshold retained 1.6M of 4.1M aggregated samples and produced the best overall score; lower thresholds led to sharper declines.Source: Figure 5 · p. 12, §3.3 · p. 13
• Loss configuration had the largest individual impact, followed by aggregation, subsampling, and architecture; the four choices together gained 2.5% AUROC over baseline.Target-group background improved AUROC by +5.3% over random background. Architecture gains were marginal, while the Transformer required about 35× more training time than the ResNet.Source: Figure 5 · p. 12, §3.3 · pp. 13–14
• Across approaches, AUROC scores were broadly similar, but AUROC can reflect factors unrelated to model quality, so the analysis emphasizes differences across species groups.AUPRG results led to the same overall conclusions, except that Random Forest retained a clearer advantage over optimized DeepSDM with that metric.Source: §3.3 · p. 14

3.4 Impact on Prediction Maps

• The four prediction-map examples varied: DeepSDM produced more constrained or localized maps for Azorella pedunculata and Artemisia mesatlantica, while Random Forest scored slightly higher for Aspidistra attenuata.The examples are illustrative, not representative; performance varies considerably within each property-space quadrant.Source: §3.4 · p. 14, Figure 6 · p. 15

3.5 Geographic Distribution of Performance

• Geographic performance gains were heterogeneous: optimized DeepSDM gains over baseline were strongest in low-effort regions, while Random Forest led over DeepSDM across undersampled tropics.The spatial gradient was shaped mainly by sampling effort: about 16% of its variance was among regions, compared with approximately 4% for relative prevalence.Source: §3.5 · pp. 14–15

4.1 Model Performance and Comparison

• Random Forests remain stronger for species in sparsely sampled regions, while DeepSDMs tend to benefit species with low relative prevalence; neither approach dominates.Average AUROC scores are very close, but performance varies across species and regions (Figure 7).Source: §4.1 · pp. 15–16, Figure 7 · p. 16
• The data aggregation, subsampling, architecture, and loss choices collectively improve optimized DeepSDM AUROC by 2.5%, with particular care needed for low-effort species.The individual choices do not uniformly benefit all species.Source: §4.1 · p. 16
• Data aggregation and subsampling achieve equivalent or better performance with less than 6% of the full dataset, reducing computation time.The authors describe these steps as analogous to spatial thinning and note their relevance to local oversampling.Source: §4.1 · pp. 16–17
• Target-group background pseudo-absences improve AUROC over random background, while species-specific weighting improves over uniform weighting more modestly.The loss choices address sampling bias and cross-species imbalance in background-to-observation ratios.Source: §4.1 · p. 17
• AUROC is strongly affected by prevalence and easily classified samples, so absolute scores across species can mislead even when between-model differences remain informative.The authors report that AUPRG yields the same overall conclusions, except that Random Forests retain a clearer advantage over optimized DeepSDM.Source: §4.1 · p. 17

4.2 Benchmark Data and Evaluation Framework

• The released benchmark and code support consistent comparisons, while the current framework restricts predictors to tabular environmental variables and retains fixed data splits for comparability.Future work could explore richer data modalities and additional species properties.Source: §4.2 · pp. 17–18
• SAGE groups species by sampling effort and relative prevalence, but its range-based properties use coarse native-country ranges and capture only part of the possible biases.Finer expert ranges are limited in availability; temporal and other biases could also be considered.Source: §4.2 · p. 17

• SAGE evaluates SDMs across thousands of species with heterogeneous data and is designed to account for severe sampling biases in large-scale biodiversity data.The benchmark supports comparisons of models and design choices across species with varying data bias.Source: §5 · p. 18
• Aggregated metrics can obscure performance disparities between well-sampled and neglected taxa, while stratifying by data properties enables a more fine-grained assessment.The authors say this evaluation may guide model selection according to dataset biases and conservation priorities.Source: §5 · p. 18

• Appendix A covers additional dataset details, including dataset statistics and environmental predictors.Source: Appendices · p. 25
• Appendix B provides additional details on the evaluation framework and species data properties.Source: Appendices · p. 25
• Appendix C covers additional modeling details for single-species and multi-species deep-learning models.Source: Appendices · p. 25
• Appendix D presents additional results, including fine-grained performance heatmaps and the effect of plot size.Source: Appendices · p. 25

A.1 SPlotOpen

• Plots are retained when all vascular plants were recorded or the sampled group was unspecified, and plots with location uncertainty above 1,000 m are excluded.The uncertainty cutoff matches the coarsest spatial resolution of the environmental predictors.Source: §A.1 · p. 25
• Species names are resolved to accepted World Checklist of Vascular Plants names using exact matching followed by strict fuzzy matching, with unresolved names excluded.Infraspecific taxa are assigned through parent links, and duplicate records caused by synonymy are removed.Source: §A.1 · pp. 25–26
• Evaluation records are restricted to reported native ranges, excluding regions flagged as introduced, extinct, or doubtful.The native ranges are constructed from WCVP distribution data and TDWG Level 3 regions.Source: §A.1 · p. 26
• Records are aggregated to a 1 km equal-area grid, where a species is marked present if detected in at least one contributing survey.Species occurring in fewer than 30 unique grid cells are removed, while empty cells remain in the evaluation dataset.Source: §A.1 · p. 26
• The evaluation data are split into 20% validation and 80% test sets using iterative stratification that reserves at least 20 test occurrences per species.Species are processed from rarest to most common before remaining plots are allocated proportionally.Source: §A.1 · p. 26

Occurrence retrieval and cleaning

• The GBIF download contains 326,143,690 georeferenced records for species matched to GBIF backbone taxonomy entries and is restricted to specified observation and specimen records with coordinate uncertainty at most 1,000 m or missing.Records are also limited to species’ native ranges; species without a valid native range or remaining occurrences are excluded.Source: §A.2 · pp. 26–27

Taxonomic harmonization and alignment

• Scientific names are harmonized to accepted WCVP species names, and cleaned GBIF species are aligned with the sPlotOpen evaluation list.Coordinate cleaning removes records at centroids, institutions, oceans, urban areas, duplicate or zero coordinates, and exact within-species coordinate duplicates.Source: §A.2 · pp. 26–27

Data aggregation

• This step reduces the GBIF dataset from 89,768,872 records across 30,138,089 samples (distinct coordinates) to 57,084,622 occurrence records across 4,100,969 samples, while retaining all 5,771 species.Source: §A.2 · p. 27

Subsampling

• The N=1000 subsampled variant contains 43,071,245 records across 1,627,928 samples and retains all 5,771 species.Subsampling selects up to N locations per species from the aggregated dataset; co-occurring species at selected locations also contribute records.Source: §A.2 · p. 27, Figure 8 · p. 28

A.3 Dataset Statistics.

• The dense-frequent group contains 1,422 species and 37,946,692 aggregated PO records, while the sparse-infrequent group has 1,927,774; PA presences are 433,808 and 115,671, respectively.The two groups are equally large; PA presences follow the same ordering, but their imbalance is weaker than for PO occurrences.Source: §A.3 · p. 28, Table 4 · p. 29

A.4 Range Maps

• POWO Level-3 regional information provides global range masks used to restrict prediction and background sampling and to compute species data properties.The benchmark excludes IUCN polygons because fine-grained shapes cover only a small subset of vascular plants, which would introduce systematic availability bias.Source: §A.4 · pp. 28–29
• The range indexes identify species whose native ranges cover each location for training loss masking and identify in-range locations for species-level evaluation.Species outside a location’s range receive an ignore label during training.Source: §A.4 · p. 29

A.5 Predictors

• The benchmark assembles 52 environmental predictors: 19 bioclimatic, 8 soil, 16 topographic, and 9 human-footprint variables.The sources are CHELSA v2.1, SoilGrids250m, EarthEnv global terrain, and the Global Human Footprint dataset.Source: §A.5 · p. 30, Table 5 · p. 30, Table 6 · p. 31

B.1 Species Data Properties

• Sampling effort measures visited cells within a species’ native range, while relative prevalence counts cells where that species was observed; their Spearman correlation is −0.08 across 5,771 species.The properties share less than 1% of their rank variance and span nearly the full percentile-rank space; the grouping scheme uses median-based quadrants and lowest-decile groups.Source: §B.1 · pp. 31–32, Figure 10 · p. 32

C.1 Single-species models

• Single-species baselines are trained independently with standardized predictors and up to 10,000 target-group background points, with per-species hyperparameters selected by validation AUROC.The down-sampled Random Forest uses 1,000 trees and achieves 85.6–85.9 AUROC across tested mtry settings, with √p best.Source: §C.1 · pp. 32–33

C.2 Multi-species deep learning models

• The multi-species models include a four-layer MLP, tabular ResNet, and FT-Transformer, trained with AdamW, a validation-based learning-rate scheduler, and early stopping.The ResNet and FT-Transformer configurations vary hidden dimension and block count; the best ResNet configuration uses dimension 1,024 and 8 residual blocks.Source: §C.2 · pp. 33–34

C.3 Loss functions

• The baseline loss does not account for class imbalance or explicitly control the relative importance of target-group and random background points.The alternative formulation introduces species-specific weights and λ parameters, then adapts weighting to observations within each species’ range.Source: §C.3 · p. 34
• The adapted weights balance contributions from presences and both pseudo-absence types across species, with square-root weights also considered to stabilize training under severe imbalance.The optimized loss includes the baseline as a special case with λ1=1, λ2=0.5, and uniform weights; alternatives tune λ1 alone or alongside species-specific weighting.Source: §C.3 · pp. 34–35

C.4 Order of design choices

• Sensitivity analyses optimize data-related choices first—aggregation and subsampling—then architecture and loss, fixing each best configuration before evaluating the next.This order evaluates later choices on the dataset ultimately used for the final model.Source: §C.4 · p. 35

D.1 Fine-grained performance heatmaps

• Sliding-window performance averages across ranked sampling effort and relative prevalence confirm that the coarse quadrants capture the dominant gradients without major dynamics lost to discretization.Source: §D.1 · p. 35

D.2 AUPRG results

• AUPRG emphasizes the precision–recall trade-off for rare positives and is summarized by the species median because a few extreme low scores distort its macro-average.For Random Forest, the worst species scores −71.6 and the lowest percentile −2.8, against a median of 0.87.Source: §D.2 · p. 35
• Random Forest leads the optimized DeepSDM overall under AUPRG (86.9% vs. 85.4%), although optimization raises DeepSDM from 82.7% to 85.4%.Source: §D.2 · p. 35
• For sparse-infrequent species, design choices yield the largest DeepSDM gain (+6.9%), while loss reweighting raises this group’s scores but slightly lowers dense-group scores.Source: §D.2 · p. 35
• Under AUPRG, GAM ranks ahead of GLM (81.0% against 79.3%), reversing their order under AUROC; Random Forest’s edge over optimized DeepSDM is also clearer than under AUROC.Source: §D.2 · p. 35

D.3 Effect of plot size on the evaluation

• Because small plots sample only part of a 1 km cell, a species present in the cell can be absent from the plot and counted as a false positive.Source: §D.3 · p. 36
• Evaluation cells were weighted by surveyed area or its square root; 15,412 of 42,268 test cells without recorded area received the mean recorded area of 530 m2.Source: §D.3 · p. 36
• 0.1% separates the Random Forest and optimized DeepSDM throughout area-weighted evaluation, and the ranking of all seven models matches the unweighted results.Down-weighting small plots lowers every score slightly, but the authors report that plot-size mismatch does not change model-comparison conclusions.Source: §D.3 · p. 36, Table 11 · p. 38