PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generation

Zhiwang Zhou, Yuandong Pu, Xuming He, Yidi Liu, Yixin Chen, Junchao Gong, Xiang Zhuang, Wanghan Xu, Qinglong Cao, Shixiang Tang, Yihao Liu, Wenlong Zhang, Lei Bai

arXiv:2512.21643v3cs.CV

TL;DR

Weather models have generally separated predictive generation from diagnostic understanding, despite the need for both. Omni-Weather unifies these tasks with a shared model and a Chain-of-Thought dataset for weather generation. The paper reports stronger performance across generation and understanding, with joint training benefiting both.

  • Problem

    Weather generation and understanding remain separate, although weather modeling needs both predictive accuracy and mechanistic interpretation.

    Source: §1 · p. 2

  • Method

    Omni-Weather uses a shared multimodal foundation model for generation and understanding and a Chain-of-Thought dataset for causal reasoning in generation.

    Source: §1 · p. 2

  • Results

    Omni-Weather surpasses task-specific baselines, and joint generation-understanding training improves performance on both sides.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results indicate that weather generation and understanding can mutually enhance each other within a unified model.

    Source: §1 · p. 2

    Broader validation remains necessary, including medium-range forecasting and typhoon track prediction.

    Source: §5 · p. 10

Abstract

from arXiv · show

Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-attention mechanism. Moreover, we construct a Chain-of-Thought dataset for causal reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.

1 INTRODUCTION

Weather generation and understanding have advanced separately, leaving prediction and diagnostic interpretation disconnected. Omni-Weather proposes a shared model and a weather-generation Chain-of-Thought dataset to bring these aims together.

1 INTRODUCTION

  • Weather models remain split: forecasting and downscaling systems lack interpretation, while diagnostic models cannot synthesize physical fields.The paper frames this gap as consequential because weather hazards call for both prediction and explanations of underlying drivers.Source: §1 · p. 2
  • Omni-Weather consolidates generation and understanding in a shared backbone and uses a Chain-of-Thought dataset for causal reasoning in generation.The dataset supports explicit reasoning supervision and thinking inference.Source: §1 · p. 2
  • Omni-Weather is introduced as the first multimodal weather foundation model jointly addressing generation and understanding tasks.The listed tasks include nowcasting, inversion, diagnostic reasoning, and question answering.Source: §1 · p. 2
  • Joint generation-understanding training provides complementary supervision, improves transferable storm-evolution representations, and benefits performance on both task types.The paper identifies these effects as evidence that the two weather task families can mutually enhance each other.Source: §1 · p. 2
  • The proposed Chain-of-Thought dataset targets causal reasoning in weather generation and is reported to enhance perceptual quality and interpretability.The authors position it as a first step toward explainable generative modeling in weather.Source: §1 · p. 2

2 RELATED WORK

Related work covers weather generation and understanding as separate areas, alongside broader unified multimodal architectures. The paper identifies their integration for weather as still unresolved.

Weather generation models

  • Weather-generation research includes radar reconstruction and precipitation forecasting, but its models leave understanding, reasoning, and interpretability largely unexplored.Examples include DiffSR reconstructing radar reflectivity from satellite and lightning inputs, and CasCast predicting precipitation evolution from past VIL sequences.Source: §2 · p. 3

Weather understanding models

  • Weather-understanding systems analyze reports or multimodal atmospheric inputs, but remain limited to textual analysis or visual reasoning without predictive generation.WeatherQA predicts convection-impacted regions from atmospheric images, while RadarQA produces quality-assessment reports using radar observations and numerical forecasts.Source: §2 · p. 3

Unified multimodal models

  • Unified multimodal architectures combine visual understanding and generation through varied designs, including shared-transformer, parallel-branch, and diffusion-decoder approaches.Examples described include Transfusion, LMFusion, MetaMorph, and MetaQuery.Source: §2 · p. 3

3 METHOD

The method frames weather generation and understanding as related sequence-to-sequence tasks, then handles them with one multimodal model and shared backbone. It adds causal Chain-of-Thought supervision and inference prompts for weather generation.

3.1 UNIFIED REPRESENTATION OF WEATHER GENERATION AND UNDERSTANDING TASKS

  • Nowcasting, satellite-to-radar inversion, and radar understanding are formulated as mappings from meteorological inputs to target outputs, using time-aligned SEVIR radar and satellite sequences.Nowcasting predicts the next twelve VIL frames from ten; inversion reconstructs VIL from IR069 and IR107; understanding produces reports from single frames or sequences.Source: §3.1 · p. 4

3.2 OMNI-WEATHER: FOUNDATION MODEL FOR WEATHER GENERATION AND UNDERSTANDING

  • Task prompts condition a shared transformer backbone, while task-specific visual encoders and decoders support text understanding, radar inversion, and nowcasting.Understanding uses an understanding encoder and text decoder; inversion uses a generation encoder and VAE decoder; nowcasting uses EarthFormer temporal tokens and predicts twelve future VIL frames.Source: §3.2 · pp. 4–5
  • Omni-Weather starts from pretrained Bagel-7B-MoT and is supervised-finetuned jointly across weather tasks under a shared representation with task-specific decoding.The training objective balances tasks while accounting for target pixels or frames for generation and target text length for understanding.Source: §3.2 · p. 5

3.3 CHAIN-OF-THOUGHT CONSTRUCTION FOR WEATHER GENERATION REASONING

  • The CoT dataset structures weather-generation reasoning around causal factors and outcome indicators, with nowcasting requiring future-outcome inference and inversion using causal factors alone.Its construction uses attribute annotation, task-specific reasoning generation, and automated checks for structural consistency, causal alignment, and terminology.Source: §3.3 · pp. 5–6, Figure 4 · p. 6
  • CoT serves as auxiliary reasoning supervision during training and as an inference prompt, with the method reporting improved interpretability, perceptual fidelity, and qualitative consistency.During training, the model generates intermediate reasoning text alongside the final prediction.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate Omni-Weather across weather generation and understanding, then examine joint training, mixed scientific and general data, and reasoning trade-offs. Results report improvements across tasks alongside a trade-off between perceptual quality and pixel-level accuracy.

4.1 IMPLEMENTATION AND EVALUATION

  • Evaluation uses task-specific baselines and pixel-level and perceptual metrics for generation, with RadarQA-protocol evaluation for understanding.Images are capped at 256 × 256 resolution; the model is trained for 20k steps on 8× H200 GPUs.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

  • Figure 5 shows richer high-value inversion structures, coherent nowcasts with interpretable storm-evolution traces, and expert-like weather-understanding outputs.The figure includes two inversion examples, a nowcasting comparison, and examples of image and sequence understanding.Source: Figure 5 · p. 7, §4.2 · p. 9
  • Nowcasting reduces CRPS by over 15% and improves LPIPS by more than 25% over single-task baselines, while key image attributes exceed RadarQA by 20–25 points.Table 1 presents quantitative results across weather generation and understanding tasks.Source: Table 1 · p. 8, §4.2 · p. 8
  • Radar inversion CSI gains reach 20% at high-value levels; thinking inference reduces LPIPS by nearly 10% with minor reductions in CSI-Mean.The reported inversion gains are across thresholds; the reasoning comparison concerns radar nowcasting.Source: §4.2 · p. 8
  • For radar sequence understanding, Dynamic Consistency improves by over 10 points, with a 5% overall gain over RadarQA.Closed-source LLMs often achieve accuracies below 30% on this task.Source: §4.2 · p. 8

4.3 ABLATION STUDIES AND ANALYSIS

  • Joint U+G training improves understanding scores and consistency as well as generation accuracy and perceptual quality on 200 sampled validation examples per task.The comparison uses understanding-only, generation-only, or jointly trained Bagel-7B-MoT models.Source: §4.3 · p. 9, Table 2 · p. 9
  • Adding 20k general metaquery samples to SEVIR consistently improves deterministic metrics and perceptual quality relative to SEVIR alone.The authors describe scientific data as anchoring domain-specific fidelity and general data as adding coverage of diverse patterns.Source: §4.3 · p. 9
  • CoT reasoning yields more detailed generations and higher LPIPS, Radar-Score, and GPT4-Score, while pixel-level CSI moderately declines.Figure 7 illustrates sharper storm structures and more coherent temporal evolution despite lower CSI.Source: §4.3 · p. 9, Table 3 · p. 9

5 CONCLUSION

Omni-Weather unifies weather generation and understanding, and the conclusion presents it as a step toward more interpretable and auditable weather intelligence. The authors also identify limits in model adaptation, validation scope, and reasoning-trace faithfulness.

5 CONCLUSION

  • Omni-Weather uses a shared backbone for weather generation and understanding, surpasses task-specific baselines, and supports reasoning analysis.The conclusion frames the model as a generalist foundation model for future weather applications.Source: §5 · p. 9
  • Coupling prediction with diagnostic explanations is presented as a practical step toward event-level weather intelligence that is easier to audit and debug.The authors expect broader multi-task transfer and more scalable evaluation across heterogeneous meteorological modalities.Source: §5 · pp. 9–10
  • The model cannot yet adapt to general-domain VAEs, and broader validation remains necessary for medium-range forecasting and typhoon-track prediction.The authors also say reasoning traces may be imperfect and need stronger faithfulness guarantees, including tighter alignment with generated or predicted fields.Source: §5 · p. 10

ETHICS AND REPRODUCIBILITY STATEMENT

The statement describes the datasets, intended use, and materials provided or planned to support reproducibility.

ETHICS AND REPRODUCIBILITY STATEMENT

A.1 OVERVIEW

The appendix overview lists sections covering task and dataset details, evaluation metrics, CoT data and reasoning, experimental results, qualitative results, and LLM usage.

A.1 OVERVIEW

  • Section A.2 covers the task paradigm and dataset, and Section A.3 covers evaluation metrics.Source: §A.1 · p. 13
  • Sections A.4 and A.5 cover CoT dataset construction and reasoning details, respectively.Source: §A.1 · p. 13
  • Sections A.6–A.8 cover additional experimental results, qualitative results, and LLM usage.Source: §A.1 · p. 13

A.2.1 DATA DETAIL

The section describes SEVIR and RQA-70K as datasets supporting weather forecasting and forecast-quality assessment. SEVIR supplies temporally aligned observations, while RQA-70K combines human annotation and automated labeling across four assessment tasks.

A.2.1 DATA DETAIL

  • SEVIR contains over 20,000 storm events from 2017–2020, each spanning four hours at five-minute resolution across approximately 384 km × 384 km regions.The dataset integrates visible and infrared satellite imagery, lightning records, and radar-derived VIL; this work focuses on VIL.Source: §A.2.1 · p. 13
  • SEVIR forecasting sequences pair observed VIL frames with future frames, and evaluation uses threshold-based metrics such as CSI and HSS.The dataset is described as a widely adopted benchmark for data-driven weather prediction.Source: §A.2.1 · p. 13
  • RQA-70K covers frame rating, frame assessment, sequence rating, and sequence assessment using human annotation and automated labeling.It combines traditional forecasting metrics with expert knowledge for radar forecast assessment.Source: §A.2.1 · p. 13

A.2.2 TASK DETAIL

The task details specify radar nowcasting from observed VIL frames and language-based assessment of radar observations or forecasts. Understanding outputs are rated across dimensions and averaged into a Radar Score.

A.2.2 TASK DETAIL

  • Nowcasting uses 10 observed VIL frames to generate the subsequent 12 frames at 256 × 256 resolution.The setup is tailored to fine-scale storm structures and temporal continuity in convective development.Source: §A.2.2 · p. 13
  • Weather understanding takes a single VIL frame or frame sequence and produces a structured report on storm morphology, intensity, temporal evolution, and forecast quality.Forecast quality can include hits, misses, or false alarms.Source: §A.2.2 · p. 13
  • Understanding responses are rated on four ordinal levels mapped to 1–4, then averaged across dimensions into the Radar Score.The score is presented as an indicator of diagnostic quality.Source: §A.2.2 · p. 13

A.3 EVALUATION PROTOCOLS AND METRICS

This section covers evaluation measures, construction of the storm-dynamics CoT dataset, qualitative and ablation findings, additional examples, and LLM use in manuscript preparation. The supplied passages also identify two qualitative-result figures but provide no evaluation protocols or metrics.

A.3 EVALUATION PROTOCOLS AND METRICS

  • CSI uses thresholds 16, 74, 133, 160, 181, and 219; SSIM measures image similarity, and CRPS evaluates probabilistic forecast accuracy.The thresholds assess different VIL intensity ranges, following Cascast Gong et al. (2024); SSIM is a perceptual measure of contrast, luminance, and structure. Higher SSIM indicates stronger similarity between prediction and observation.Source: §A.3 · p. 14
  • Text and image-generation evaluation includes LPIPS, Rouge_L, BertScore, GPT4-Score, and Radar-Score.LPIPS uses deep features to assess image similarity aligned with human judgment (Zhang et al., 2018); Rouge_L measures longest-common-subsequence similarity, and BertScore compares text embeddings (Zhang et al., 2019). GPT4-Score assesses accuracy, content richness, and fidelity; Radar-Score averages four-level ratings across meteorological dimensions.Source: §A.3 · p. 14

A.4 DETAIL OF CHAIN-OF-THOUGHT CONSTRUCTION

  • The CoT pipeline preprocesses SEVIR events into three pairs of 10-frame inputs and 12-frame outputs, annotates storm attributes, and generates causal reasoning traces.After three-step quality control, the final dataset contains 4,000 annotations each for radar nowcasting and radar inversion.Source: §A.4 · pp. 14, 15, 17, 18

A.5 MORE DETAIL ABOUT REASONING

  • CoT finetuning produces more perceptually clear radar fields and storm-grounded explanations than non-CoT finetuning, whose “thinking” tends toward post-hoc evaluation.Figures 8–9 show representative inversion and nowcasting cases; additional examples across nowcasting, inversion, and understanding illustrate storm-dynamics capture and interpretable assessments.Source: §A.5 · p. 18, Figure 8 · p. 19, Figure 9 · p. 20

A.6 MORE EXPERIMENT RESULTS

  • CFG=2 yields better overall performance than CFG=1, especially on CSI metrics, SSIM, and PSNR; separately, a single general dataset at a 30% ratio achieves the best balance among mixing settings.The authors adopt CFG=2 as the default configuration.Source: §A.6 · p. 21

A.8 USAGE OF LLM

  • LLMs supported manuscript preparation through grammar checking, sentence refinement, and clarification of technical descriptions; authors reviewed and revised all suggested text.Source: §A.8 · p. 21
Loading 2512.21643v3…