PaperlayerView exact revision on arXiv

Source-linked AI summary

A comprehensive and trustworthy benchmark of AI methods for change detection in Earth observation

Tadej Tomanič, Alice Baudhuin, Jan Sotošek, Jure Brence, Panče Panov, Nikola Simidjievski, Dragi Kocev

arXiv:2608.28247v1cs.CVcs.AI

TL;DR

Earth observation change-detection research lacks consistent evaluation protocols and often underreports computational efficiency. This paper introduces an open, standardized benchmark across ten datasets and ten model architectures, finding that classical architectures can offer a superior accuracy–efficiency trade-off while releasing reproducible FAIR resources.

  • Problem

    Earth observation change detection is important for monitoring land-surface transformations, but research is constrained by inconsistent protocols and limited efficiency reporting.

  • Method

    The paper evaluates ten representative architectures across ten heterogeneous datasets using controlled protocols, efficiency profiling, and openly released benchmark artifacts.

  • Results

    Classical architectures such as U-Net SiamConc frequently provide a superior accuracy–computational-cost trade-off compared with complex vision transformers and state-space models.

  • Takeaways & Limitations

    Reliable model selection for Earth observation change detection requires controlled cross-dataset comparisons that account for computational cost and reproducibility.

  • Takeaways & Limitations

    Fixed hyperparameters may disadvantage architectures sensitive to model-specific tuning, and stochastic training can produce variable performance.

Abstract

from arXiv · show

Change detection in Earth observation (EO) is critical for monitoring land surface transformations, yet recent research in the field is constrained by inconsistent evaluation protocols and a narrow focus on predictive accuracy without regard for computational efficiency. To address this, we present a standardized, open-source benchmark for evaluating state-of-the-art (SOTA) deep learning methods for Earth observation change detection. We conduct a comprehensive analysis of ten representative model architectures, ranging from convolutional networks (CNNs) to vision transformers (ViTs), across ten heterogeneous change detection datasets. We rigorously evaluate these models with identical experimental protocols, comparing models trained from scratch against those utilizing pre-trained weights. Furthermore, we evaluate predictive performance alongside computational efficiency, including parameter counts and inference latency. Our findings reveal that well-optimized classical architectures, such as Siamese U-Nets, frequently outperform more complex contemporary models when computational efficiency is factored in, and that pre-training consistently provides a significant performance boost with no additional inference cost. To ensure complete transparency and reproducibility, all experimental resources, including standardized data splits, training scripts, training logs, and model checkpoints are publicly available and adhere to FAIR principles (Findable, Accessible, Interoperable, and Reusable).

change detection in Earth observation

This paper presents a trustworthy, comprehensive benchmark of AI methods for Earth observation change detection. It evaluates diverse models and emphasizes both predictive performance and computational efficiency.

  • Benchmark scope: 10 deep learning models were benchmarked across 10 diverse change detection datasets.The benchmark covers representative architectures and heterogeneous evaluation settings.
  • Benchmark scope: Unified training and evaluation pipelines were used to minimize experimental bias.The study standardizes the comparison conditions across methods.
  • Evaluation focus: Model performance was analyzed alongside computational-efficiency trade-offs.The benchmark treats efficiency as an evaluation dimension rather than focusing only on predictive accuracy.
  • Reproducibility: All benchmark resources were openly released in adherence to FAIR principles.The release supports transparent and reusable research.

1. Introduction

Earth observation change detection supports monitoring land-surface transformations, but comparisons are fragmented by inconsistent protocols and limited efficiency reporting. The paper addresses this gap with a standardized benchmark spanning diverse datasets, architectures, and reproducible resources.

  • Motivation: Earth observation change detection compares co-registered bi-temporal images to identify genuine semantic or structural changes.The task must distinguish real transformations from irrelevant variations such as seasonal shifts.
  • Research gap: Rapid architectural proliferation and inconsistent evaluation protocols have fragmented empirical comparisons.The literature spans CNNs, ViTs, and SSMs, with competing claims of superior performance.
  • Research gap: Controlled evaluations show traditional models can match or outperform complex architectures while efficiency is often underreported.Relevant deployment metrics include FLOPs, parameter counts, and memory footprints.
  • Benchmark design: A reliable benchmark requires identical protocols across heterogeneous datasets while profiling performance and computational efficiency.The proposed scope includes urban, seasonal, disaster, and multi-sensor conditions.
  • Contributions: The benchmark establishes fixed splits, unified pipelines, standardized losses, and openly released artifacts for reproducible comparison.Its resources include data splits, scripts, schemas, checkpoints, and experiment logs.

2. Related work

Prior work identifies evaluation bias, saturated architectural gains, and reproducibility problems in Earth observation change detection. These findings motivate controlled, auditable comparisons supported by reusable data, code, and models.

  • Existing reviews: Reviews describe change-detection methods across sensors, supervision paradigms, algorithms, and learning frameworks.Reported operational challenges include label scarcity, registration noise, scene complexity, and domain shift.
  • Existing reviews: Architectural performance gains have saturated on standard benchmarks, while annotation costs, label availability, and domain shift remain constraints.This meta-analysis shifts attention from model capacity toward data and deployment conditions.
  • Evaluation bias: Controlled studies found pretrained U-Net baselines can match or outperform complex transformers across canonical datasets.This challenges unverified claims of state-of-the-art architectural gains.
  • Evaluation bias: Reported gains can arise from outdated baselines, flawed dataset preprocessing, or modernized training procedures rather than novel modules.These factors make direct comparisons between published results unreliable.
  • Trustworthy benchmarking: Trustworthy benchmarking therefore combines standardized comparison with transparent, accessible, and reusable artifacts.The benchmark operationalizes these principles through controlled protocols and FAIR releases.

3. Datasets & models

The benchmark covers ten heterogeneous Earth observation datasets and ten representative architectures, spanning CNNs, attention-based networks, transformers, and state-space models. The datasets capture varied sensors, resolutions, and environmental conditions, while the models span different computational profiles.

  • Benchmarking datasets: Ten datasets were selected to span spatial resolutions, sensor modalities, geographic regions, and scene dynamics.They cover urban development, disaster mapping, agriculture, land-cover dynamics, and multi-sensor change detection.
  • Benchmarking datasets: The benchmark includes sub-meter aerial imagery, 10 m Sentinel-based data, and multi-sensor inputs.It also tests off-nadir displacement, SAR speckle, seasonal vegetation shifts, soil moisture variation, and illumination differences.
  • Benchmarking datasets: The datasets provide co-registered pre-event and post-event images with binary pixel-level change masks.A mask value of 1 denotes change and 0 denotes background.
  • Model architectures: Ten representative models span classical convolutional, attention and prior-guided, transformer, and state-space architectures.The selection is intended to evaluate generalization across diverse model families.
  • Model architectures: U-Net SiamConc, HRNet SiamConc, SiamCRNN, and TinyCD provide convolutional baselines with different spatial, temporal, and efficiency designs.TinyCD is explicitly optimized for resource-constrained edge computing.
  • Model architectures: STANet and CGNet add attention or change-prior mechanisms to address environmental variation, boundaries, and internal artifacts.STANet models long-range dependencies, while CGNet guides multi-scale fusion with a change prior map.
  • Model architectures: BIT, ChangeFormerV6, and ChangeViT use transformer-based global context modeling, while CSSM uses selective state-space sequence modeling.CSSM targets linear computational complexity through target-specific parameter selection.

4.1. Training and evaluation protocol

The benchmark uses standardized dataset preparation, training, validation, and testing procedures across heterogeneous Earth observation change-detection datasets. Models are compared under random initialization and ImageNet-1K backbone transfer learning, with early stopping based on validation loss.

  • Dataset splits: A 60/20/20 train/validation/test split was applied across the benchmark datasets.Existing splits were used for Season-Varying CDD, SYSU-CD, and CLCD; the remaining datasets were randomly sampled.
  • Preprocessing: Custom transform pipelines normalized channels, reordered tensors, and resized inputs to 256 × 256 with antialiasing.Ground-truth masks were likewise converted to channel-first 32-bit tensors and resized to the same target resolution.
  • Optimization: Training used a unified toolbox with up to 100 epochs, batch size 16, initial learning rate 1 × 10−4, and a hybrid loss with equal component weights.Training data were shuffled each epoch, while validation and test ordering remained deterministic.
  • Initialization regimes: All architectures were evaluated with random initialization and ImageNet-1K transfer learning to measure domain-transfer effects.ChangeFormerV6 and CSSM were exceptions because pre-trained weights were unavailable; transferred weights applied only to backbones.
  • Checkpoint selection: Early stopping after 10 validation epochs selected the checkpoint with the lowest validation loss for final test evaluation.This procedure was applied separately for each dataset.

4.2. Evaluation metrics

The evaluation protocol measures predictive performance together with computational efficiency under standardized testing conditions. Because changed pixels are spatially imbalanced, mIoU and F1-score are prioritized over overall accuracy alone.

  • Evaluation dimensions: Predictive performance and computational efficiency were measured jointly for comprehensive candidate evaluation.Efficiency measures include parameter counts, FLOPs, latency, activation memory, and training time.
  • Mask generation: A fixed threshold of τ = 0.5 converted continuous output probability maps into binary change masks.The threshold was applied during testing.
  • Predictive metrics: mIoU and F1-score served as the primary metrics because severe spatial class imbalance can make overall accuracy misleading.Evaluation used the unseen test split to assess model generalization.
  • Training efficiency: Peak activation memory was measured with batch size 16, while training time tracked convergence under the early-stopping protocol.

4.3. Reproducibility, FAIR release, and trustworthy integration

The benchmark combines reproducible execution, FAIR artifact release, and an ontology-based representation of datasets, methods, experiments, and evaluations. This integration supports traceability from results to their underlying configurations and makes benchmark information machine-actionable.

  • Reproducibility: MLflow logged execution metadata, dataset split identifiers, and hyperparameters for every benchmarking experiment.TensorBoard recorded step-wise loss and metric trajectories across training and validation epochs.
  • Standardization: Identical NVIDIA A100-PCIe GPUs, CUDA 12.9, data loaders, dataset splits, and preprocessing pipelines enforced a common comparison environment.
  • FAIR release: The release applies FAIR principles to standardized splits, benchmarking code, trained checkpoints, and experiment logs.
  • FAIR release: Artifacts are findable through public Zenodo, GitHub, and Hugging Face records, with persistent identifiers and descriptive metadata.They are openly retrievable under an open license, with metadata available independently of the data payload.
  • Semantic integration: An RDF ontology links datasets, methods, runs, and evaluation results using shared vocabularies and makes benchmark results queryable.The schema is organized into Dataset, Method, Experiment, and Evaluation layers.
  • Traceability: The evaluation layer records each measure, data split, aggregation method, and resulting numeric value.Together, the layers trace an evaluation result to the exact dataset, model configuration, and experimental run.

5. Results & discussion

Across ten datasets, pre-training generally improves predictive performance, while classical convolutional architectures remain highly competitive. Efficiency-aware results favor U-Net SiamConc and SiamCRNN as strong accuracy–resource trade-offs, although dataset difficulty and model behavior vary substantially.

  • Predictive performance: 0.6980 to 0.7283 is the benchmark-wide average mIoU increase associated with transfer learning.The comparison is between models trained from scratch and models using pre-trained weights.
  • Predictive performance: 0.7931 average mIoU makes pre-trained U-Net SiamConc the benchmark’s overall leader, ranking first across seven datasets.SiamCRNN follows at 0.7797, while ChangeViT reaches 0.7709.
  • Predictive performance: 0.7392 average mIoU makes SiamCRNN the strongest from-scratch model, while ChangeViT reaches 0.7381 and wins Season-varying CDD.SiamCRNN also performs best on MSBC and EGY-BCD under the reported evaluations.
  • Dataset variation: 0.8369 versus 0.5160 average mIoU separates Season-varying CDD from BANDON when trained from scratch, highlighting dataset-dependent difficulty.The corresponding pre-trained averages are 0.8943 and 0.5442.
  • Predictive performance: 0.02 to 0.085 average mIoU gains occur with ImageNet-1K pre-training across architectures, except CGNet, whose performance degrades.Average F1-score gains range from 0.014 to 0.068; HRNet SiamConc and U-Net SiamConc improve most.
  • Computational efficiency: 0.29 M parameters and 1.46 GFLOPs make TinyCD lightweight, whereas ChangeFormerV6 uses 41.03 M parameters and 138.77 GFLOPs.The benchmarked architectures therefore span a wide computational range.
  • Computational efficiency: 6.02 ms makes CGNet fastest, but pre-trained U-Net SiamConc combines 0.7931 average mIoU with 10.39 ms latency and 8.29 GB memory.SiamCRNN offers a faster 9.02 ms and 6.55 GB alternative with 0.7797 mIoU.
  • Computational efficiency: 1.64 GB memory and 60.63 ms latency characterize CSSM, while heavy models reach 14.78 GB memory and 55.13 ms latency.STANet provides a practical 3.01 GB memory balance for constrained deployments.

6. Conclusions

The benchmark systematically compares deep learning architectures for Earth observation change detection under controlled protocols and computational profiling. Its results favor efficient classical architectures and emphasize transparent, FAIR-aligned resources for reproducible research.

  • Conclusions: Ten architectures were evaluated across heterogeneous Earth observation change detection datasets under a strictly controlled experimental protocol.The evaluation included models trained from scratch and pre-trained models under identical conditions and hardware constraints.
  • Conclusions: The benchmark profiles predictive performance alongside latency, FLOPs, memory footprints, and related computational requirements.These measurements support assessment beyond standard predictive metrics.
  • Conclusions: Classical architectures such as U-Net SiamConc often provide a superior accuracy–computational-cost trade-off compared with complex vision transformers or state-space models.This finding challenges the prevailing narrative that newer, more complex architectures necessarily deliver better overall trade-offs.
  • Conclusions: Study-design details, metadata, training logs, and best-performing model checkpoints are publicly available.The authors present these resources as a basis for more systematic and rigorous future experiments.
  • Conclusions: FAIR-aligned open resources allow researchers to reuse benchmark assets and reduce the computational overhead of redundant model training.The benchmark is positioned as a starting point and guiding design principle for future change-detection benchmarking.
  • Conclusions: The change-detection benchmark is the first step toward FAIR-EO Hub, a broader standardized platform covering fundamental Earth observation applications.Planned extensions include classification, semantic segmentation, and object detection alongside emerging Earth observation foundation models.

CRediT authorship contribution statement

The authors contributed across conceptualization, methodology, software, investigation, visualization, validation, writing, supervision, resources, funding, and project administration.

  • CRediT authorship contribution statement: Tadej Tomanič contributed formal analysis, investigation, methodology, software, validation, visualization, and writing.His listed writing contributions include the original draft and review and editing.
  • CRediT authorship contribution statement: Alice Baudhuin contributed data curation, software, visualization, and writing.Her writing contributions include review and editing.
  • CRediT authorship contribution statement: Jan Sotošek contributed investigation, methodology, resources, software, and visualization.
  • CRediT authorship contribution statement: Jure Brence contributed conceptualization and writing review and editing.
  • CRediT authorship contribution statement: Panče Panov, Nikola Simidjievski, and Dragi Kocev contributed funding, supervision, conceptualization, methodology, and related project oversight.Panče Panov also contributed project administration; Nikola Simidjievski contributed funding acquisition and supervision; Dragi Kocev contributed conceptualization and funding acquisition.

Code and Data Availability

The benchmark is released with open-source code, standardized data-processing resources, trained checkpoints, and complete training logs to support end-to-end reproducibility.

  • Code and Data Availability: The benchmarking framework is fully open-source and integrated into the AiTLAS toolbox.
  • Code and Data Availability: Standardized data loaders, training scripts, annotated turtle files, schemas, and ontologies are available in the FAIR-EO-CD-benchmark repository.
  • Code and Data Availability: Trained model checkpoints and complete TensorBoard training logs are deposited on Hugging Face and Zenodo.Raw data are not hosted directly, but links to the original datasets are available on GitHub.
Loading 2608.28247v1…