PaperlayerView exact revision on arXiv

Source-linked AI summary

RS-WorldModel: a Unified Model for Remote Sensing Understanding and Future Sense Forecasting

Linrui Xu, Zhongan Wang, Fei Shen, Gang Xu, Huiping Zhuang, Ming Li, Haifeng Li

arXiv:2603.14941v1cs.AI

TL;DR

Existing remote sensing methods typically separate change understanding from future forecasting, despite shared spatiotemporal priors. RS-WorldModel unifies both tasks using RSWBench-1.1M and a three-stage training pipeline. With 2B parameters, it surpasses much larger open-source models on most change-QA metrics and achieves an FID of 43.13 for future scene forecasting.

  • Problem

    Remote sensing understanding and forecasting are usually trained separately, limiting use of their shared geographic and spatiotemporal structure.

  • Method

    RS-WorldModel jointly trains change understanding and text-guided future forecasting on RSWBench-1.1M through GAGP, SIT, and VRO.

  • Results

    2B parameters let RS-WorldModel surpass open-source models up to 120× larger on most ST-CQA metrics and achieve an FID of 43.13, outperforming all listed baselines on forecasting FID.

  • Takeaways & Limitations

    A unified remote sensing world model can address observed-change understanding and text-guided future-scene forecasting within one model and benchmark.

  • Takeaways & Limitations

    Future forecasting rewards plausible, condition-consistent outcomes rather than enforcing pixel-level matching to a single ground-truth future scene.

Abstract

from arXiv · show

Remote sensing world models aim to both explain observed changes and forecast plausible futures, two tasks that share spatiotemporal priors. Existing methods, however, typically address them separately, limiting cross-task transfer. We present RS-WorldModel, a unified world model for remote sensing that jointly handles spatiotemporal change understanding and text-guided future scene forecasting, and we build RSWBench-1.1M, a 1.1 million sample dataset with rich language annotations covering both tasks. RS-WorldModel is trained in three stages: (1) Geo-Aware Generative Pre-training (GAGP) conditions forecasting on geographic and acquisition metadata; (2) synergistic instruction tuning (SIT) jointly trains understanding and forecasting; (3) verifiable reinforcement optimization (VRO) refines outputs with verifiable, task-specific rewards. With only 2B parameters, RS-WorldModel surpasses open-source models up to 120$ \times $ larger on most spatiotemporal change question-answering metrics. It achieves an FID of 43.13 on text-guided future scene forecasting, outperforming all open-source baselines as well as the closed-source Gemini-2.5-Flash Image (Nano Banana).

1 Introduction

Remote sensing methods commonly separate change understanding from future scene forecasting, despite their shared geographic and spatiotemporal priors. RS-WorldModel unifies both capabilities with a dedicated dataset and three-stage training pipeline.

  • Existing generative models synthesize satellite imagery without reasoning about changes, while understanding models interpret scenes without forecasting future or counterfactual states.
  • Unified modeling is motivated by shared geographic and acquisition context, including location, seasonality, and sensor characteristics.
  • RSWBench-1.1M contains 1.1M high-resolution samples covering change understanding and text-guided future forecasting, with fine-grained geographic metadata and global diversity from fMoW.
  • The three-stage pipeline uses GAGP for geographic forecasting priors, SIT for joint understanding and generation, and VRO for task-specific verifiable reward refinement.
  • 2B parameters enable RS-WorldModel to surpass open-source models up to 120× larger on most spatiotemporal change QA metrics.
  • RS-WorldModel achieves an FID of 43.13 on text-guided future scene forecasting, outperforming open-source baselines and Gemini-2.5-Flash Image.

2 RSWBench-1.1M Dataset

RSWBench-1.1M is a large-scale benchmark and training corpus for remote sensing understanding and forecasting, constructed from fMoW with automated annotation, physical filtering, semantic refinement, and strict split isolation.

  • The dataset supports Spatiotemporal Change Question-Answering and Text-Guided Future Scene Forecasting through a scalable automated annotation pipeline.
  • Both training and evaluation data derive from fMoW and follow official split protocols to prevent data leakage.
  • Scalable Data Construction Pipeline: Physical standardization pairs observations from matching coordinates, normalizes acquisition metadata, and filters samples using visibility and cloud-ratio criteria.
  • Scalable Data Construction Pipeline: Metadata translation converts numeric sensor information into linguistic cues, such as describing shadows from solar elevation, to reduce overfitting to raw values.
  • The benchmark comparison distinguishes whether datasets support remote sensing understanding, generation, or both, using ✓, ✗, and – labels.
  • The training corpus contains approximately 1.1M samples, including 371K generative-pretraining instances, 742K mixed instruction-tuning instances, and 16K reinforcement-alignment samples.

3 Method

RS-WorldModel unifies remote-sensing change understanding and future-scene forecasting through shared metadata-conditioned, autoregressive modeling. Its three-stage training combines geographic pretraining, joint instruction tuning, and task-specific verifiable rewards.

  • Unified formulation: The model formulates ST-CQA and TFSF as instruction-conditioned sequence-generation tasks with text or visual-token outputs.ST-CQA uses bi-temporal imagery, questions, and metadata; TFSF uses a current image, text instruction, and source and target metadata.
  • Unified formulation: RS-WorldModel encodes satellite images as visual tokens, fuses them with geospatial metadata, and autoregressively generates language or future-image tokens.A shared latent space and next-token objective connect perception, understanding, and forecasting.
  • Training rationale: The three objectives progressively build physical simulation priors, task alignment, and a shared representation of remote-sensing dynamics.The framework is intended to support robust performance across both perception and forecasting tasks.
  • Three-stage training: GAGP trains metadata-conditioned forecasting by sampling source and target observations from the same geographic location and predicting target visual tokens.The text-free forecasting objective conditions predictions on geographic and acquisition metadata.
  • Three-stage training: SIT jointly optimizes ST-CQA and TFSF on a mixed dataset with one next-token objective across text and visual outputs.TFSF prompts specify land-cover transitions, while ST-CQA prompts request changed and unchanged elements and sensor-variation reasoning.
  • Three-stage training: VRO refines outputs with task-specific verifiable rewards for forecasting and change-question answering.TFSF rewards combine instruction faithfulness and spatial consistency, while ST-CQA uses metadata-grounded LLM evaluation of semantic and physical quality.

4 Experiments

The experiments evaluate RS-WorldModel on spatiotemporal change question answering and text-guided future scene forecasting against commercial, open-source, and domain-specific baselines. Results show strong performance in both understanding and forecasting, while ablations support the contributions of reference adherence, staged training, and geographic metadata.

  • Experimental setup: The evaluation covers ST-CQA on a 5K subset and TFSF on a 1.6K subset using language, similarity, perceptual, and GPT-based metrics.ST-CQA reports GPT-Score, BLEU-1, METEOR, ROUGE-L, S-BERT, SimCSE, ST5-SCS, and response length; TFSF reports FID, CosSim, and four GPT-based scores.
  • ST-CQA results: RS-WorldModel ranks first among open-source baselines on BLEU-1, ROUGE-L, and all three contextual similarity metrics.Against same-scale Qwen3-VL-2B, ROUGE-L improves by 21% and S-BERT by 14%.
  • TFSF results: 43.13 FID is obtained on TFSF, while the model ranks first among open-source models on every metric and surpasses Gemini-2.5-Flash Image at 46.14 FID.FID is reduced by 48% relative to CRS-Diff and by 47% relative to FLUX.1-Kontext; CosSim and GPT scores are highest.
  • TFSF results: RS-WorldModel produces photorealistic satellite imagery that preserves layouts, vegetation, rooftops, shadows, and lighting under detailed textual descriptions.The qualitative comparison reports stronger structural fidelity and physical consistency than competing diffusion and autoregressive models.
  • Ablation studies: Increasing λ from 0.0 to 0.2 improves TFSF CosSim from 67.22 to 68.34, lowers FID from 44.19 to 43.13, and raises GPT-based OA from 88.75 to 89.43.On ST-CQA, GPT-Score rises from 86.03 to 86.20 and BLEU-1 from 49.78 to 50.59; λ=0.2 is adopted.
  • Ablation studies: The full GAGP→SIT→VRO pipeline achieves 43.13 FID, 44.59 Similarity, 89.43 OA, and 86.20 GPT-S, outperforming partial configurations.Adding GAGP before SIT reduces FID from 73.55 to 44.23, while VRO provides further improvement.
  • Ablation studies: Geographic and acquisition metadata reduce FID from 53.72 to 50.28 during GAGP and improve alignment with target-region land-cover distributions.The ablation supports location and sensor information as useful conditioning for spatially grounded representations.

5 Conclusion

The paper concludes that RS-WorldModel and RSWBench-1.1M unify remote-sensing change understanding with future-scene forecasting. The 2B-parameter model outperforms much larger models on ST-CQA and achieves leading forecasting FID, while ablations support contributions from each training stage.

  • Conclusion: RS-WorldModel jointly addresses spatiotemporal change understanding and text-guided future scene forecasting for remote sensing.It is paired with RSWBench-1.1M, a 1.1M-sample dataset covering both tasks with fine-grained geographic metadata.
  • Conclusion: The three-stage pipeline consists of Geo-Aware Generative Pre-training, synergistic instruction tuning, and verifiable reinforcement optimization.The stages are presented as the training approach for the unified model.
  • Conclusion: With only 2B parameters, RS-WorldModel surpasses open-source models up to 120× larger on most ST-CQA metrics and outperforms all open-source baselines and Gemini-2.5-Flash Image on forecasting FID.Ablations report distinct contributions from each training stage and benefits from the verifiable reward design across both tasks.

A.1 Unified multimodal understanding and generation

Unified multimodal models combine visual understanding and controllable generation within shared autoregressive frameworks. Prior remote-sensing vision-language models mainly strengthen understanding, while newer systems extend toward task unification and multisensor comprehension.

  • Unified multimodal understanding and generation: Unified multimodal models jointly handle visual understanding and controllable generation through shared representations.The cited work associates this design with semantic consistency, generation controllability, and world-modeling capabilities.
  • Unified multimodal understanding and generation: Remote-sensing vision-language models include grounded spatial reasoning, broad VQA and captioning benchmarks, unified instruction tuning, open-world interpretation, and multisensor comprehension.Examples include GeoChat, RSGPT, SkyEyeGPT, Skysense-o, and EarthGPT.

A.3 Large-Scale Remote Sensing Vision-Language Datasets

Remote-sensing vision-language datasets have expanded multimodal understanding resources, but many remain centered on VQA, captioning, or single-temporal settings. The cited resources also include token-distribution analyses for forecasting and understanding subsets.

  • Large-Scale Remote Sensing Vision-Language Datasets: VRSBench, SkySenseGPT, RS-GPT4V, and SkyScript provide large or fine-grained resources for remote-sensing multimodal understanding.Their cited uses include image understanding, instruction tuning, unified multimodal instruction following, and semantically diverse data.
  • Large-Scale Remote Sensing Vision-Language Datasets: The TFSF and ST-CQA subsets are characterized through token-length distributions and frequent-term word clouds.Figure 6 covers forecasting, while Figure 7 covers understanding.
  • Large-Scale Remote Sensing Vision-Language Datasets: Existing datasets primarily focus on VQA and captioning and are mostly single-temporal.The passage contrasts these resources with settings requiring broader temporal coverage.

B Details about RSWBench-1.1M Dataset

RSWBench-1.1M is characterized through token-length distributions and word clouds covering its forecasting and understanding subsets. These statistics indicate diverse instructions, balanced task complexity, and rich semantic coverage from automated annotation.

  • Token-length distributions and word-cloud visualizations describe both the forecasting and understanding subsets of RSWBench-1.1M.
  • The dataset statistics demonstrate diverse instructions and balanced complexity across tasks.
  • The automated annotation pipeline achieves rich semantic coverage across the benchmark.

C Additional Implementation Details

RS-WorldModel is trained in three stages with frozen vision components and multimodal infrastructure. The stages progress from geo-aware pre-training to mixed-task instruction tuning and reinforcement optimization with task-specific rewards and KL regularization.

  • The model freezes the vision encoder and multimodal projector while training remaining parameters in bf16 on 8 NVIDIA A800 GPUs.DeepSpeed ZeRO-3 and Flash Attention 2 are used across all three stages.
  • Stage 1 uses geo-aware generative pre-training on 371K forecasting samples, while Stage 2 trains on 742K mixed understanding and forecasting samples.Both stages run for 32 epochs; Stage 2 starts from the Stage-1 checkpoint and uses a reduced peak learning rate.
  • Stage 3 applies GRPO to 16K mixed ST-CQA and TFSF samples with task-specific rewards and a judge model for ST-CQA.KL regularization keeps the policy close to the Stage-2 initialization and preserves instruction-following behavior.

D Case Studies

Case studies show RS-WorldModel producing strong qualitative results for both spatiotemporal change understanding and text-guided future scene forecasting. The supporting pipeline uses metadata-aware prompts, automated scoring, and a pure LLM judge for reinforcement optimization.

  • ST-CQA case study: In ST-CQA, RS-WorldModel identifies layout consistency, vegetation growth, and acquisition-time shadow variations while achieving the highest GPT-Score.Baselines either overlook subtle changes or hallucinate major modifications.
  • TFSF case study: In TFSF, RS-WorldModel consistently attains the highest GPT-based Similarity and Quality scores across three textual scenarios.The generated images show superior structural fidelity, shadow consistency, and text adherence among strong open-source baselines.
  • Prompted pipeline: The data construction and evaluation pipeline provides prompt templates for drafting, refinement, GPT-Score computation, and model training behavior.These templates support scalable annotation, automatic scoring, and stage-specific system prompts.
  • Prompted pipeline: VRO uses a pure LLM judge rather than a VLM to provide more stable, semantically rich, and metadata-grounded reward signals.The Stage-1 and Stage-2/3 system prompts define behavior across training stages.
  • Prompted pipeline: The refinement prompt combines temporal captions with geospatial metadata to correct inaccuracies and describe both changed and unchanged aspects.Metadata is converted into natural descriptive language rather than exposed as raw numerical values or technical terms.
  • Prompted pipeline: The image-generation prompt supplements refined captions with predicted changes and visual effects while requiring alignment with observed differences and avoiding speculation.Its response format requires a structured JSON object containing the refined temporal caption.
Loading 2603.14941v1…