Source-linked AI summary
FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models
Md Aminur Hossain, Omkumar Vaghasiya, Rajeev Ranjan Dwivedi, Vinod Kurmi, Biplab Banerjee
TL;DR
Aggregate RSFM metrics provide limited evidence about consistency across ecological regions, motivating evaluation by biome-derived groups. FairRSFM benchmarks three RSFMs across four datasets with a frozen-backbone protocol and evaluates mitigation baselines; results show biome-dependent disparities and task-dependent mitigation outcomes.
Problem
Most RSFM evaluations report aggregate dataset-level metrics, leaving performance consistency across ecological regions unmeasured.
Source: §1 · p. 1Method
FairRSFM maps georeferenced samples to biome-derived groups and evaluates three RSFMs across four datasets under a frozen-backbone protocol, alongside mitigation baselines.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Biome-aware evaluation complements aggregate metrics for understanding RSFM reliability, while mitigation outcomes vary by task and model.
Source: §1 · p. 2, §8 · p. 9Biome-aware robustness captures only one aspect of geographic bias and does not fully capture sensor or source shifts, geographic imbalance, resolution differences, temporal effects, or uneven imagery coverage.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions. We introduce FairRSFM, a biome-aware benchmark for evaluating ecological group robustness in RSFMs. FairRSFM maps georeferenced samples from 14 terrestrial biome classes into six ecologically meaningful macro-groups and evaluates models under a unified frozen-backbone evaluation protocol. The benchmark covers four downstream datasets: m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K with Dynamic World label maps. Using Prithvi-EO-2.0, SatMAE, and DOFA across three random seeds, we show that aggregate performance consistently masks biome-dependent disparities across architectures and tasks. For example, Prithvi-EO-2.0 reaches 90.98% overall macro-F1 on m-EuroSAT but a mean worst-group score of only 83.72%, while m-SA-Crop-Type drops from 27.30% overall mIoU to 18.47% in the Xeric and Mineralogical group. We further evaluate Biome-Orthogonal Linear Probing (BOLP), Dynamic Biome Reweighting (DBR), and GroupDRO as complementary mitigation baselines. Their effectiveness is model- and task-dependent; for example, BOLP improves Prithvi-EO-2.0 worst-group F1@opt on m-BigEarthNet from 46.12% to 50.27% without updating the RSFM backbone. FairRSFM provides a reusable protocol for diagnosing and mitigating ecological robustness gaps in remote sensing foundation models. Code and datasets are available at: https://github.com/aminurhossain/FairRSFM.
1. Introduction
FairRSFM addresses the limited ecological detail of aggregate RSFM evaluation by measuring performance across biome-derived groups. It combines a shared frozen-backbone benchmark with mitigation baselines, including BOLP, and complements broader task and corruption benchmarks with ecological robustness analysis.
1. Introduction
- Aggregate RSFM metrics do not show whether performance remains consistent across ecological regions, where the same class can look different and underrepresented regions may be underserved.This matters because geospatial models can degrade under geographic, sensor, scale, and source shifts [5] [22].Source: §1 · p. 1
- FairRSFM maps georeferenced samples to 14 terrestrial biome classes and pools them into six macro-groups to retain ecological distinctions while addressing sparse coverage.The groups represent broad spectral, phenological, hydrological, cryospheric, and surface-property regimes; original biome labels remain available for finer analysis when sample counts permit.Source: §1 · pp. 1–2
- FairRSFM evaluates ERM, DBR, BOLP, and GroupDRO as complementary mitigation baselines, with BOLP removing biome-associated directions from frozen embeddings before task-head training.The framework examines whether disparities can be reduced without full backbone fine-tuning.Source: §1 · p. 2
- The benchmark evaluates Prithvi-EO-2.0, SatMAE, and DOFA across four datasets under a unified frozen-backbone protocol, reporting task metrics and biome-aware robustness measures.The datasets comprise three selected GEO-Bench tasks and an MMEarth20K Dynamic World segmentation subset.Source: §1 · p. 2
2. Related Work
- FairRSFM complements task-focused benchmarks such as GEO-Bench and robustness-to-corruption evaluation such as REOBench [13] by measuring variation across biome-derived groups.Its ecological grouping question is distinct from broader assessment across tasks, resolutions, geographies, or image corruptions.Source: §2 · p. 2
- BOLP extends representation-debiasing methods to ecological robustness by projecting biome-associated directions out of frozen embeddings without backbone fine-tuning or additional trainable parameters.The projection is described as closed-form.Source: §2 · p. 3
3. FairRSFM Benchmark
FairRSFM combines classification and semantic-segmentation datasets with georeferenced biome labels and a common group-robustness evaluation. Its six macro-groups pool sparse biome classes while preserving the raw labels, and its metrics cover task performance, worst-group robustness, calibration, and cross-group disparity.
3.1. Task Families and Datasets
- The benchmark spans classification and semantic segmentation, using three GEO-Bench tasks and an MMEarth20K Dynamic World subset to add dense land-cover evaluation.The suite includes single-label classification, multi-label classification, and crop-type segmentation.Source: §3.1 · p. 3, Table 1 · p. 4
- The MMEarth20K subset contains 20,000 samples with 10 m Dynamic World labels, sampled as 16,000 training, 2,000 validation, and 2,000 test tiles.Macro-group quotas use inverse-frequency weighting of the parent MMEarth distribution, from 28.0% to 6.0%, and are allocated uniformly across the 14 RESOLVE biomes.Source: §3.1 · p. 3
3.2. Biome Label Construction
- Each georeferenced sample receives a raw biome label by joining its patch centroid to an external biome map; segmentation patches use the same centroid-based assignment.The authors treat this as a consistent grouping rule, while noting it can be noisy for ecotone or multi-biome tiles.Source: §3.2 · p. 3
- FairRSFM maps 14 raw biome labels into six fixed macro-groups for primary evaluation and mitigation, retaining raw labels for finer-grained analysis when sample counts allow.Pooling is intended to reduce instability from limited or uneven biome coverage while preserving ecologically meaningful distinctions.Source: §3.2 · pp. 3–4
3.3. Biome Metadata Schema
- The metadata schema stores image, task label, dataset, location, raw biome, and macro-group; the macro-group drives primary robustness metrics and mitigation.Unknown, water-only, or unmatched samples are excluded from primary group summaries unless stated otherwise, and metrics use matched groups present in each split.Source: §3.3 · p. 4, §3.2 · p. 4
3.4. Evaluation Metrics
- Evaluation reports task-specific group metrics, worst-group score, and normalized failure range, alongside calibration and ecological group-disparity summaries.ECE, EOdd, and DPM are descriptive summaries; EOdd and DPM are not claims about protected human attributes. For datasets with only two matched groups, the paper emphasizes worst-group score and cautions about range summaries.Source: §3.4 · p. 4
4. Biome-Aware Debiasing Framework
The framework isolates mitigation effects by keeping the RSFM encoder frozen and training only downstream heads. It compares validation-feedback and loss-based group reweighting with a representation-level method that removes biome-associated directions.
4. Biome-Aware Debiasing Framework
- All strategies keep the RSFM encoder fixed and train only the downstream head, isolating mitigation effects from backbone fine-tuning.The task loss varies by task: cross-entropy for single-label classification, binary cross-entropy for multilabel classification, and pixel-wise cross-entropy for segmentation.Source: §4 · p. 5
4.1. Dynamic Biome Reweighting with Validation Feedback
- DBR initializes group weights using inverse frequency, then adjusts them using validation performance, increasing weights for below-mean groups and decreasing them for above-mean groups.The update strength α and numerical-stability term ϵ govern the update; validation metrics are reused without adding trainable parameters.Source: §4.1 · p. 5
4.2. Biome-Orthogonal Linear Probing
- BOLP recenters frozen embeddings and projects out dominant inter-group mean-difference directions without updating the backbone.Its projector is computed from the top k left singular vectors of the centered group-mean matrix; the resulting transform is fixed after training-set precomputation.Source: §4.2 · p. 5
4.3. Group Distributionally Robust Optimization
- GroupDRO adaptively upweights high-loss macro-groups while training only the downstream head, complementing DBR's validation feedback and BOLP's representation-space intervention.ERM, DBR, BOLP, and GroupDRO retain the full training set and add no trainable parameters beyond the task head.Source: §4.3 · pp. 5–6
5. Experiments
Across frozen-backbone experiments, ecological disparities persisted, and mitigation effects varied by model and task. The experiments compare three RSFMs across four datasets, with biome coverage differing across datasets.
5. Experiments
- Prithvi-EO-2.0, SatMAE, and DOFA were evaluated on four datasets with frozen encoders, updating only task-specific heads or segmentation decoders.The three GEO-Bench tasks use their GEO-Bench splits; MMEarth20K uses the 20,000-sample subset and split defined in Section 3.1.Source: §5 · p. 6
5.1. Biome Distribution Analysis
- Matched-group coverage is 98.8%, 88.5%, 99.9%, and 100% for m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K, respectively.m-EuroSAT and m-BigEarthNet have only three matched macro-groups, m-SA-Crop-Type has two, and MMEarth20K covers all six; the smallest matched-group test sizes are Ng=25 and Ng=14 for the first two datasets.Source: §5.1 · p. 6
5.2. Biome-Wise Performance
- Prithvi-EO-2.0's weakest reported groups include m-EuroSAT Aseasonal High-Biomass at 84.56 ± 0.31 macro-F1 and m-BigEarthNet Transitional Herbaceous and Scrub at 46.04±1.85 F1@opt.The m-EuroSAT Aseasonal High-Biomass group has Ng=25; the m-BigEarthNet result is weaker despite stronger forest-dominated groups.Source: §5.2 · p. 7
- 18.47 ± 0.16 mIoU on Prithvi-EO-2.0 m-SA-Crop-Type's Xeric and Mineralogical group is below 28.55 ± 0.19 on the Transitional Herbaceous and Scrub group.The reported disparity demonstrates that biome-dependent differences also occur in dense prediction tasks.Source: §5.2 · p. 7
5.3. Debiasing and Robustness Results
- Lower NFR, EOdd, and DPM indicate smaller cross-group disparities, while ECE measures calibration and standard deviations quantify variation across seeds.Absolute cross-model scores are interpreted descriptively because downstream probe and decoder implementations differ by model.Source: §5 · p. 6, §5.3 · p. 7
- BOLP raises Prithvi-EO-2.0 m-BigEarthNet worst-group F1@opt from 46.12% to 50.27%, but mitigation effects vary by model and task.On that dataset, BOLP also raises overall F1@opt from 60.09% to 62.75% and reduces NFR from 31.14% to 20.92%; on MMEarth20K it reduces both overall and worst-group performance, while GroupDRO raises worst-group score from 33.75% to 37.16%.Source: §5.3 · p. 7
6. Discussion
Aggregate scores can conceal biome-level performance gaps, and mitigation methods show task-dependent trade-offs between overall and worst-group performance. BOLP improves worst-group scores on classification tasks, while its benefits do not transfer uniformly to dense prediction.
6. Discussion
- 83.72% worst-group performance contrasts with 90.98% overall macro-F1 on m-EuroSAT; m-BigEarthNet and m-SA-Crop-Type also show overall-to-worst-group drops.On m-BigEarthNet, worst-group performance is 46.12% versus 60.09% overall; on m-SA-Crop-Type, worst-group mIoU is 18.47% versus 27.30% overall.Source: §6 · p. 7
- 50.27% worst-group F1@opt on m-BigEarthNet with BOLP improves on ERM's 46.12% without updating the backbone.BOLP also raises overall F1@opt from 60.09% to 62.75% and reduces NFR from 31.14% to 20.92%.Source: §6 · p. 7
- On m-EuroSAT, GroupDRO has the highest overall macro-F1 at 94.14%, while DBR yields the lowest NFR at 5.85%.For m-SA-Crop-Type, all three mitigation methods improve worst-group mIoU over ERM, with DBR reaching 19.81%.Source: §6 · pp. 7–8, Table 7 · p. 8
- On MMEarth20K, GroupDRO reaches 37.16% worst-group mIoU, while BOLP reduces both overall and worst-group mIoU.DBR slightly improves worst-group mIoU to 34.76% from 33.75%.Source: §6 · p. 8, Table 8 · p. 8
7. Limitations
FairRSFM’s ecological group measurements have boundaries related to patch-level biome assignment, dataset coverage, study scope, and the choice of disparity metrics. These constraints call for interpreting group results within the matched groups and settings evaluated.
7. Limitations
- Centroid-based patch labels can be noisy near biome boundaries or in multi-biome tiles; they are a consistent grouping rule, not a pixel-perfect ecological map.An optional majority-pixel alternative is supported.Source: §7 · p. 8
- Worst-group and disparity metrics are dataset-specific and should be interpreted over matched groups; small test groups of 14–25 samples can increase sampling variability.Source: §7 · p. 8
- The empirical study covers three RSFMs and four datasets under frozen-backbone evaluation; broader sensor families, backbones, and macro-group sensitivity remain untested.Segmentation disparity metrics are less standardized, and alternative pixel-, patch-, object-, or class-level formulations remain possible.Source: §7 · p. 8
- Biome-aware robustness captures only one aspect of geographic bias; sensor and source shifts, spatial resolution, temporal acquisition, and uneven imagery coverage are not fully captured.FairRSFM is a complementary ecological robustness benchmark, not a complete remote-sensing bias audit.Source: §7 · p. 8
8. Conclusion
FairRSFM evaluates ecological group robustness in RSFMs across four datasets under a unified frozen-backbone protocol. Results show biome-dependent disparities and task-dependent trade-offs among mitigation baselines.
8. Conclusion
- Across three random seeds, strong aggregate performance can hide substantial biome-dependent disparities in classification and segmentation.The benchmark evaluates Prithvi-EO-2.0, SatMAE, and DOFA using standard task metrics and group-robustness measures.Source: §8 · p. 8
- Mitigation baselines show task-dependent trade-offs between aggregate performance and worst-group robustness; BOLP can improve worst-group classification performance without updating the backbone.BOLP removes biome-associated directions from frozen RSFM embeddings.Source: §8 · p. 9
- Biome-aware evaluation is important for understanding RSFM reliability in global Earth-observation settings.Source: §8 · p. 9
S1. Supplementary Overview
The supplement provides supporting materials for FairRSFM’s group definitions, dataset construction, sensitivity studies, and implementation.
S1. Supplementary Overview
- Supplementary materials include a global visualization and deterministic mapping of six macro-groups, MMEarth20K construction details, DBR and BOLP sensitivity studies, and implementation details.The sensitivity studies vary DBR feedback parameters and BOLP rank.Source: S1. Supplementary Overview · p. 11
S2. Global Biome Macro-Group Map
The supplements define six ecological macro-groups from 14 terrestrial biomes, detail MMEarth20K sampling, and test DBR and BOLP sensitivities. Results show that mitigation settings vary in value across tasks: DBR has a favorable operating range, while BOLP projection rank affects classification and segmentation differently.
S3. Biome Macro-Group Definition
- Table S1 gives a deterministic mapping from 14 terrestrial biome classes to six macro-groups, reducing sparsity while retaining ecologically interpretable distinctions.The six groups represent broad spectral, phenological, hydrological, cryospheric, and surface-property regimes.Source: S3. Biome Macro-Group Definition · p. 11, Table S1 · p. 12
S4. MMEarth20K Construction & Allocation
- MMEarth20K sampled 20,000 tiles using normalized inverse-frequency macro-group quotas based on the parent MMEarth distribution, ranging from 28.0% (5,600) to 6.0% (1,200), with uniform allocation among constituent terrestrial biomes.Source: S4. MMEarth20K Construction & Allocation · p. 11
S2. Global Biome Macro-Group Map
- Figure S1 maps the six macro-groups globally, showing spatially heterogeneous ecological regimes that motivate subgroup evaluation beyond aggregate metrics.The groups pool raw terrestrial biomes into spectral, phenological, hydrological, cryospheric, and surface-property regimes.Source: Figure S1 · p. 11, S2. Global Biome Macro-Group Map · p. 11
S5. Ablation Studies and Hyperparameter Sensitivity
- On Prithvi-EO-2.0 test splits, DBR α∈[0.4, 0.5] yields the lowest NFR; α=0.8 lowers m-EuroSAT worst-group score to 82.38 ± 1.62%, and Tupd=5 reduces worst-group performance by up to 0.74 percentage points.Values are mean ± std across S=3 random seeds.Source: Table S3 · p. 12, S5. Ablation Studies and… · p. 12
S5.2. BOLP Projection Rank (k)
- At k=2, BOLP gives the highest tested overall and worst-group classification scores (84.34 ± 0.00% on m-EuroSAT; 50.27 ± 0.63% on m-BigEarthNet), but full-rank segmentation mIoU falls from 47.99% to 43.31%.For MMEarth20K, full-rank NFR is 30.66 ± 3.17%, above ERM's 28.19 ± 0.64%; intermediate ranks reach 32.85 ± 1.15%.Source: Table S4 · p. 13, S5.2. BOLP Projection Rank (k) · p. 13
S6. Implementation, Architecture & Compute Details
The implementation keeps the RSFM representations frozen while using backbone-specific downstream heads and decoders. It assigns ecological groups through a centroid-based spatial join and reports the associated compute overhead and hardware.
S6.1. Downstream Heads and Decoder Architectures
- With frozen representations ϕ(x), classification probes vary by backbone, while segmentation uses FCN for Prithvi-EO-2.0, convolutional upsampling for SatMAE, and UPerNet for DOFA.Classification uses cross-entropy for m-EuroSAT and multi-label binary cross-entropy with logits for m-BigEarthNet.Source: S6.1. Downstream Heads and Decoder… · p. 13
S6.2. Centroid vs. Majority-Pixel Spatial Join
- Centroid-based assignments agree with majority-pixel assignments on >99.2% of samples across all four datasets.The pipeline uses sample coordinate centroids and a point-in-polygon lookup against the RESOLVE terrestrial ecoregions database.Source: S6.2. Centroid vs. Majority-Pixel… · p. 13
S6.3. Compute Infrastructure and Efficiency
- Experiments ran on A100 80GB and RTX 4090 GPUs; frozen embedding extraction is the primary offline compute cost, while centroid/SVD projection adds zero trainable parameters.After offline embeddings are extracted, group centroids and the closed-form SVD projection matrix produce static linear projection buffers.Source: S6.3. Compute Infrastructure and… · p. 13