PaperlayerView exact revision on arXiv

Source-linked AI summary

EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting

Junwei Luo, Shuai Yuan, Zhenya Yang, Yansheng Li, Zhe Liu, Hengshuang Zhao

arXiv:2606.27277v1cs.AIcs.CV

TL;DR

EO forecasting must handle sparse observations, uncertain land-surface states, and weather-dependent dynamics that existing models and benchmarks only partially capture. EO-WM uses physically structured weather conditioning and diagnostic benchmarks, improving degradation-severity prediction and forcing-response fidelity while remaining competitive on pixel-level metrics.

  • Problem

    Existing EO forecasting methods do not explicitly represent uncertainty, distinguish physical weather roles, or test physically consistent responses to changed meteorological forcing.

  • Method

    EO-WM is a video diffusion transformer that decomposes meteorological forcing into climatological baseline, weather anomalies, and cumulative stress, evaluated with two weather-response benchmarks.

  • Results

    EO-WM improves degradation-severity prediction and forcing-response fidelity across deterministic and generative baselines while remaining competitive on standard pixel-level metrics.

  • Takeaways & Limitations

    The proposed benchmarks evaluate EO forecasts for extreme-weather degradation and response direction and magnitude beyond standard pixel reconstruction.

  • Takeaways & Limitations

    The seasonal forecasting setting does not directly extend to multi-year or decadal simulation because paired satellite and weather records are limited.

Abstract

from arXiv · show

Earth Observation (EO) forecasting aims to predict future Earth surface dynamics from satellite observations under changing meteorological conditions. In this paper, we view this task as a partially observed, weather-driven world modeling problem, in which weather acts as a conditioning signal, while forecasting remains uncertain due to sparse observations and unobserved land-surface states. However, existing methods do not fully capture this setting: deterministic models collapse uncertainty into a single future prediction, while diffusion-based methods typically treat weather variables as undifferentiated conditioning signals, and existing benchmarks focus mainly on reconstruction accuracy rather than whether forecasts respond correctly to changed weather forcing.We introduce EO-WM, a video diffusion transformer for multispectral EO forecasting. EO-WM incorporates a physically informed conditioning framework that represents meteorological forcing through a climatological baseline, weather anomalies, and cumulative physical stress signals. Specifically, it separates baseline and anomaly through distinct conditioning pathways, and accumulates anomalous forcing over time to capture sustained heat and drought stress. To evaluate weather-response behavior beyond standard metrics, we introduce two diagnostic benchmarks: an Extreme Summer Benchmark for severity-aware prediction of vegetation degradation under extreme weather, and a Seasonal Matched-Pair Benchmark for testing response fidelity under changed weather forcing. Experiments show that EO-WM reduces the error in predicted Normalized Difference Vegetation Index (NDVI) decline amplitude by a relative 5.63% and improves directional hit rate by a relative 7.80%, while remaining competitive on standard pixel-level metrics. The benchmarks and model will be made open-source at https://github.com/Luo-Z13/EO-WM.

1 Introduction

The introduction reframes Earth Observation forecasting as a partially observed, weather-driven world-modeling problem and presents EO-WM as a probabilistic video diffusion transformer. It also proposes physically informed weather conditioning and two benchmarks for evaluating weather-responsive vegetation forecasting beyond pixel reconstruction.

  • EO forecasting predicts future satellite observations from weather information and supports extreme-event monitoring, crop-yield prediction, and ecosystem or vegetation forecasting.
  • EO forecasting involves sparse, incomplete observations and stochastic forcing-response mappings because satellite dynamics between valid observations remain unobserved.Satellite revisit intervals span days, while cloud contamination further reduces valid observations.
  • Existing methods either provide deterministic point predictions without explicit uncertainty or use meteorology as generic conditioning without separating physical weather roles.The introduction distinguishes climatological background, anomalous weather events, and accumulated environmental stress as unmodeled conditioning components.
  • EO-WM is a video diffusion transformer that forecasts multispectral satellite imagery from sparse observations and heterogeneous conditions while representing multiple plausible futures.Its diffusion formulation is designed to represent unobserved intermediate dynamics under partial observability.
  • EO-WM decomposes meteorological forcing into climatological baseline, weather anomaly, and cumulative stress, while two benchmarks test vegetation degradation and directional response to changed weather.Cumulative stress indices capture the persistence of abnormal weather, and the benchmarks extend evaluation beyond standard pixel-level metrics.

2 Related Work

Prior EO forecasting methods establish weather-conditioned prediction but are mainly deterministic, while broader video and remote-sensing generative work introduces uncertainty, controllability, and world-modeling. Existing remote-sensing world models, however, do not explicitly structure meteorological forcing or evaluate response fidelity to changed weather.

  • Video Generation and World Modeling: Diffusion-based video prediction offers a natural framework for uncertain future dynamics, while controllable video models improve external-condition injection into large video backbones.STIV and ATI incorporate text, image, trajectory, or motion controls into diffusion transformers; Wan and Open-Sora demonstrate controllable generation quality.
  • Video Generation and World Modeling: Video world-modeling research extends passive generation toward interactive simulation through large world foundation models, open simulators, and action-conditioned video diffusion models.These works show that video generation models can support world simulation when paired with actions or controls.
  • EO Forecasting: EO forecasting was formalized as predicting future satellite observations conditioned on weather, with early methods relying mainly on deterministic prediction.ConvLSTM, Earthformer, and vegetation forecasting studies established the value of explicit weather conditioning, strong spatiotemporal backbones, multimodal context, and weather signals.
  • Generative EO Models: Generative EO research applies diffusion to satellite forecasting and reconstruction, alongside unified remote-sensing time-series, metadata-conditioned, and climate-aware generation frameworks.These methods represent recent efforts to address limitations of earlier deterministic EO forecasting.
  • Generative EO Models: BAGEL and RS-WorldModel bring foundation-model and world-modeling concepts to remote sensing but focus mainly on generic generative quality.They do not explicitly structure weather into climatological baseline, anomaly, and accumulated stress, or evaluate whether futures respond faithfully to changed meteorological forcing.

3 Method

EO-WM formulates multispectral Earth observation forecasting as probabilistic, weather-driven world modeling under sparse observations and partial observability. Its latent diffusion architecture separates seasonal climatology from anomalous forcing and accumulates harmful weather anomalies into physical-stress conditions.

  • Task formulation: EO forecasting uses sparse multispectral satellite observations and dense meteorological forcing aligned across their timestamps.Weather is treated as an exogenous driver of surface-state transitions, while unobserved land-surface conditions make future outcomes probabilistic.
  • Generative architecture: EO-WM encodes multispectral inputs with an EO-specific VAE and generates stochastic futures using an MMDiT trained with flow matching.The diffusion model predicts the flow-matching velocity with mean-squared error while excluding conditioned context frames and invalid pixels.
  • Condition routing: Spatial conditions combine visual context, elevation, time, anomaly, and stress features, with the spatial pathway reinjected every four double-stream blocks.Zero-initialized learned gates support periodic reinjection so observation-aware and forcing-related information remains available through the transformer.
  • Climatology–anomaly conditioning: EO-WM decomposes meteorological forcing into a monthly climatological baseline and a residual anomaly, routing them through distinct conditioning pathways.Climatology provides a seasonal reference at the input token layer, whereas spatially aligned anomalies enter the reinjected spatial pathway.
  • Cumulative physical stress: EO-WM accumulates three weather-stress fields so forecasts represent both instantaneous forcing departures and sustained environmental burden.ReLU gates retain positive temperature anomalies and negative precipitation anomalies; stress values are spatially averaged, log-compressed, projected, and added to the reinjected spatial condition.

4 Benchmarks and Metrics

Section 4 introduces two diagnostic benchmarks built from EarthNet2021 test splits to evaluate weather-driven forecasting behavior beyond standard reconstruction metrics. It also defines standard reconstruction, extreme-summer, and seasonal matched-pair metrics.

  • Benchmark motivation: The two diagnostic benchmarks retain EarthNet2021’s 10-context/20-target protocol while targeting behaviors that standard reconstruction metrics do not capture.The benchmarks assess weather-response behavior rather than only matching one realized future.
  • Extreme Summer Benchmark: The Extreme Summer Benchmark contains 1,440 verified windows from the 2018 European summer heat event, stratified into low-, mid-, and high-severity bins by NDVI decline amplitude.Each 30-frame window places the 10-frame context immediately before vegetation decline and requires valid cloud masks and a baseline-relative NDVI drop during the 20-frame target period.
  • Seasonal Matched-Pair Benchmark: The Seasonal Matched-Pair Benchmark contains 422 pairs from 380 locations, matching geographic cube and seasonal timing across different years.Quality filtering and initial-state matching reduce cloud, phenology, and observed-context confounds before selecting complementary tracks.
  • Metrics: Standard metrics include EarthNetScore, Pixel-MAE, and NDVI-MAE, while Extreme Summer adds Trough NDVI-MAE and Drop Amplitude Error.EarthNetScore combines MAD, OLS, EMD, and SSIM sub-scores; TN-MAE evaluates trough NDVI error, and DAE evaluates baseline-to-trough decline-amplitude error.
  • Metrics: Seasonal Matched-Pair evaluation uses Divergence Reproduction Ratio and paired divergence correlation to compare predicted and ground-truth changes under altered weather forcing.The provided metric description identifies DRRmean as mean DRR and PDCsp as Spearman paired divergence correlation.

5 Experiments

Experiments evaluate EO-WM in the EarthNet2021 forecasting setting and show that physically informed weather conditioning improves weather-response behavior beyond standard reconstruction metrics. Inference ablations further reveal a trade-off between pixel-level stability and pair-specific response fidelity.

  • Experimental setup: EO-WM predicts 20 future 4-channel Sentinel-2 frames from 10 context frames at 128 × 128 resolution under shared weather conditions.The EO-VAE tokenizer is finetuned on the EarthNet2021 training split.
  • Experimental setup: The diffusion backbone is trained from scratch with 387M parameters, and inference uses no classifier-free guidance unless otherwise specified.Anomaly CFG is reported separately as an inference-time sensitivity analysis.
  • Benchmark results: Earthformer achieves the lowest overall NDVI-MAE on Extreme Summer but increasingly under-reproduces vegetation declines as event severity rises.Its behavior indicates conservative forecasts despite strong reconstruction quality.
  • Weather-conditioning ablation: Climatology–anomaly decomposition improves degradation-amplitude and paired-divergence metrics, while cumulative stress further improves DAE, DHR, and PDC.These results support modeling both instantaneous weather anomalies and sustained vegetation stress.
  • Inference-strategy ablation: Five-sample ensembling improves reconstruction-oriented metrics but slightly reduces PDC, whereas stronger anomaly guidance raises DRR and DHR while degrading pixel quality and TN-MAE.The main architectural comparisons therefore use unguided inference.
  • Visual diagnostics: EO-WM shows the steepest fitted slope and highest auxiliary DRA, while generative models detect extreme events more often than deterministic Earthformer, especially at low and medium severity.These visual diagnostics indicate better severity calibration and probabilistic event detection.

6 Conclusion

EO-WM presents physically informed multispectral Earth Observation forecasting with structured meteorological forcing and benchmarks beyond pixel reconstruction. The conclusion highlights seasonal-horizon and partially observed land-surface limitations, while motivating ground-station integration for future applications.

  • Contributions: EO-WM models meteorology as structured exogenous forcing by separating climatological context, weather anomalies, and cumulative stress.It is presented as a physically informed diffusion forecasting model for multispectral Earth Observation under sparse and partial observations.
  • Contributions: The introduced benchmarks evaluate capabilities beyond pixel reconstruction, including degradation prediction under extreme heat and drought.The supplied passage also describes evaluating whether forecasts preserve the correct response, but its sentence is truncated.
  • Limitations: Seasonal forecasting cannot yet extend directly to multi-year or decadal simulation because paired satellite-observation and weather records are limited.Longer horizons would involve hundreds of Sentinel-2 frames, stronger error accumulation, changing seasonal regimes, and slow climate trends.
  • Limitations: Unobserved or partially observed land-surface states include soil moisture, irrigation, and vegetation type.These hidden states constrain the current forecasting setting.
  • Future Work: Combining satellite imagery with same-region ground-station measurements could expose hidden states and improve forecasting accuracy.The passage connects this direction to ecosystem monitoring, crop-growth prediction, and climate-risk assessment.

A Technical appendices and supplementary material

The supplementary material provides implementation, diagnostic, ablation, benchmark-construction, evaluation, and comparison-method training details across Sections A.1–A.3.

  • A.1 EO-WM details: Section A.1 covers EO-WM implementation details, tokenizer reconstruction diagnostics, ablations, and qualitative examples.
  • A.2 Benchmark construction: Section A.2 describes construction of the Extreme Summer and Seasonal Matched-Pair benchmarks, including filtering criteria, sample statistics, and evaluation metrics.
  • A.3 Comparison methods: Section A.3 summarizes adaptation and training details for comparison methods, including Wan2.1, Latte, and OpenSTL.

A.1 EO-WM Training and Ablations · A.1.1 More Training Details

EO-WM training follows the EarthNet2021 10-to-20 forecasting protocol, using Sentinel-2 sequences with visual context and future-frame prediction. Training combines masked latent diffusion optimization, metadata-based auxiliary captions, and distributed mixed-precision optimization.

  • A.1.1 More Training Details: EO-WM uses 30-frame Sentinel-2 sequences at 128×128 resolution with four optical channels, providing 10 context frames to predict 20 future frames.The protocol follows EarthNet2021.
  • A.1.1 More Training Details: The EO-VAE is trained on the EarthNet2021 training split with 2% of samples held out for validation and a base learning rate of 2×10−5.Training continues until convergence.
  • A.1.1 More Training Details: Pixels with cloud probability at or above 0.2 are invalidated, and latent cells enter diffusion loss only when at least 50% of corresponding pixels remain valid.The quality mask is downsampled by area averaging.
  • A.1.1 More Training Details: Loss masking excludes conditioned visual-context frames, restricting optimization to valid target-frame latents rather than copied reference frames.
  • A.1.1 More Training Details: Auxiliary captions use fixed templates describing metadata and observed sequence properties, while excluding target image content and remaining outside the core forecasting task.They populate the video backbone’s inherited text-conditioning interface.
  • A.1.1 More Training Details: Training uses 4 GPUs with bfloat16 mixed precision, ZeRO-2 optimization, gradient checkpointing, and no gradient accumulation.The effective batch size is 256 from batch size 64 per GPU.
  • A.1.1 More Training Details: EO-WM optimization uses AdamW with learning rate 2 × 10−4, weight decay 0.01, cosine scheduling, and 500 warm-up steps.
  • A.1.1 More Training Details: Flow-matching timestep sampling uses shift parameter α = 2.0 for the reported EO-WM training configuration.

A.1.2 Tokenizer Reconstruction Analysis

EO-VAE reconstruction is near-lossless on both benchmarks and more faithful than Wan-VAE and trivial baselines under masked pointwise metrics. However, EarthNetScore’s EMD and OLS components can favor predictors that match non-masked target-subset statistics rather than exact trajectories.

  • Metric interpretation: EarthNetScore aggregates SMAD, SOLS, SEMD, and SSSIM using their harmonic mean, while ENS, P-MAE, and N-MAE serve as standard reconstruction metrics.The proposed metrics additionally target degradation severity and changed-weather response, which EarthNetScore does not isolate.
  • Reconstruction fidelity: EO-VAE reduces MAE on Extreme Summer from 0.0257 (copy-last) and 0.0263 (persistence) to 0.0022.On Seasonal Matched-Pair, it reduces MAE from 0.0161 and 0.0183 to 0.0034.
  • Reconstruction fidelity: EO-VAE achieves lower MAE/MSE than Wan-VAE on both benchmarks, indicating stronger preservation of EarthNet multispectral observations.
  • Metric interpretation: EMD and OLS need not be maximized by exact ground-truth recovery because they compare prediction statistics with non-masked target subsets using asymmetric evaluation intervals.The table notes that GT is diagnostic rather than a theoretical upper bound for these components.
  • Metric interpretation: On Seasonal Matched-Pair, Copy last clear frame and Persistence exceed GT in EMD/OLS despite worse MAE/MSE, while Wan-VAE exceeds EO-VAE on EMD/OLS despite worse masked pointwise errors.

A.1.3 Condition-Injection Ablation

The ablation separates the effects of adding EO side conditions and repeatedly reinjecting spatial condition features within MMDiT blocks. Visual context and masks alone perform poorly, while EO conditioning substantially improves forecasting and deep reinjection further improves reconstruction and response fidelity.

  • Condition-injection ablation: The ablation tests EO side conditions and repeated spatial-condition reinjection inside MMDiT blocks, using visual context and quality masks as the baseline.EO conditions include meteorological, static geographic, and spatiotemporal conditions; deep reinjection repeats spatial-condition injection inside the MMDiT blocks.
  • Condition-injection ablation: The visual-context-and-mask-only configuration performs poorly on both benchmarks, indicating that visual history alone is insufficient for weather-conditioned EO forecasting.The first ablation row uses only visual context and quality masks.
  • Condition-injection ablation: 0.2385 Extreme ENS is achieved after adding EO side conditions at the input, up from 0.1458 for the visual-context-and-mask-only configuration.Input-only EO conditioning recovers most of the performance in the ablation.
  • Condition-injection ablation: 0.6186 Seasonal DHR is achieved after adding EO side conditions at the input, up from 0.4302 for the visual-context-and-mask-only configuration.The input-only EO-conditioning variant improves Seasonal DHR substantially.
  • Condition-injection ablation: Deep spatial-condition reinjection further improves reconstruction and response-fidelity metrics beyond input-only EO conditioning.The default EO-WM setting reinjects spatial-condition features after every four double-stream blocks.

A.1.4 Qualitative Forecasting Results

Qualitative 10-context/20-target forecasts demonstrate the challenge of inferring future vegetation dynamics from sparse observations under dense meteorological forcing. EO-WM captures degradation timing and spatial coherence more faithfully than the conservative Earthformer forecasts, consistent with its physically informed conditioning.

  • Forecasting setting: The 10-context/20-target task requires forecasting vegetation degradation despite cloud-contaminated or missing Sentinel-2 observations and dense future meteorological forcing.White regions in the ground-truth sequence mark cloud-contaminated or missing frames.
  • Model comparisons: Earthformer produces conservative forecasts in which vegetation decline is delayed relative to the observed target sequence, especially under rapid heat- and drought-driven transitions.The transition is from green vegetation to dry or senescent surfaces.
  • Model comparisons: Wan2.1 and EO-WM better capture forcing-response behavior by predicting vegetation degradation earlier than Earthformer.This qualitative pattern is consistent with the quantitative experimental trends.
  • Model comparisons: EO-WM preserves more coherent spatial patterns and more faithful response timing, consistent with explicitly modeling climatological baseline, weather anomaly, and cumulative physical stress.These conditioning components provide a physically informed explanation for the observed qualitative behavior.

A.2 Benchmark Construction and Evaluation Protocols … A.4 Data and Asset Availability

The paper constructs curated Extreme Summer and Seasonal Matched-Pair benchmarks from EarthNet2021, evaluates weather-response fidelity and probabilistic calibration with specialized diagnostics, and documents adaptation of Wan2.1 for multispectral EO forecasting.

  • A.2.1 Extreme Summer Benchmark Construction: Extreme Summer windows are built through NDVI trajectory analysis, candidate prefiltering, anchor-based localization, quality filtering, and verified event detection.Candidate sequences require target valid-pixel ratio >30%, sequence-level NDVI drop ≥0.35, and non-negative trough NDVI; verified events require at least two consecutive and two total target frames below θ = baseline NDVI −0.10.
  • A.2.1 Extreme Summer Benchmark Construction: 1,440 verified Extreme Summer windows remain from 1,447 prefiltered candidates after rejecting seven for cloud coverage or failed extreme-event verification.The benchmark uses 30-frame windows containing 10 context and 20 target frames, and its composite extreme score combines drop amplitude, persistence, consecutiveness, and data quality.
  • A.2.2 Seasonal Matched-Pair Benchmark Construction: The Seasonal Matched-Pair Benchmark pairs same-location, same-season windows from different years to isolate interannual weather variability and test changed-forcing responses.This design asks whether identical initial observation states lead to appropriately different futures under different meteorological conditions.
  • A.2.3 Extreme Summer Benchmark: Evaluation Metrics: Extreme Summer evaluation reports EarthNetScore alongside pixel, NDVI, trough, decline-amplitude, and drop-reproduction metrics on the 20-frame target period.Stochastic methods are evaluated using ensemble-mean predictions, while severity bins are defined by percentiles of the composite extreme score.
  • A.2.4 Seasonal Matched-Pair Benchmark: Evaluation Metrics: Seasonal Matched-Pair evaluation uses DRR, DHR, and PDC to measure divergence magnitude calibration, directional accuracy, and ranking fidelity under changed weather.DRR = 1.0 is ideal, DHR = 0.5 corresponds to random guessing, and higher PDC indicates better cross-pair sensitivity calibration.
  • A.2.5 Probabilistic Calibration Diagnostics: Five-sample probabilistic diagnostics show both ensembles are under-dispersed, while EO-WM improves CRPS, spread-skill ratio, and empirical 90% coverage relative to Wan2.1.Spread-skill ratios remain below 1 and 90% coverages below 0.9, so EO-WM is improved but not perfectly calibrated.
  • A.3.1 Wan2.1 Adaptation: Wan2.1-Inp is adapted from a 1.3B-parameter video diffusion transformer through four-channel VAE expansion, Earth-observation conditioning pathways, and progressive four-stage fine-tuning.The adaptation supports B, G, R, and NIR inputs while injecting geospatial and meteorological information into the DiT.
Loading 2606.27277v1…