Source-linked AI summary
OmniClimate-TC: Physics-Aware Visual Abstractions for Multimedia Reasoning over Tropical Cyclones
Luwei Xiao, Xin Wang, Keane Ong, Jiawen Wei, Chenyu Dong, Rui Mao, Erik Cambria, Gianmarco Mengaldo
TL;DR
VLMs struggle to reason over meteorological reanalysis because physically meaningful cyclone properties are encoded in continuous fields rather than object-centric visual structure, while existing approaches largely emphasize alignment or annotation. The paper introduces PAVA as a physics-aware representation and annotation interface, builds OmniClimate-TC with PAVA, and adapts VLMs for the benchmark. Across multiple tasks, ClimateTCX consistently outperforms strong baselines, supporting representation design for physically grounded hazard-field reasoning.
Problem
VLMs face a representational mismatch because reanalysis semantics arise from continuous spatial organization, extrema, gradients, asymmetry, and scale rather than discrete natural-image objects.
Method
PAVA maps continuous meteorological reanalysis fields to visually identifiable, semantically grounded abstractions for supervision, annotation, and evaluation.
Results
ClimateTCX consistently outperforms strong baselines across multiple tasks, supporting PAVA-aligned representation design for physics-grounded reasoning over hazard fields.
Takeaways & Limitations
OmniClimate-TC positions physics-aware representation design as a key component of multimedia reasoning over structured geophysical fields.
Abstract
from arXiv · showhide
Meteorological reanalysis encodes extreme weather through continuous, physically constrained fields, posing a fundamental challenge for vision-language models (VLMs) whose perceptual assumptions are shaped by natural images. Tropical cyclones exemplify this mismatch: critical properties such as intensity extrema, asymmetry, spatial extent, and physical impacts arise from field-level organization rather than object-centric visual cues. Existing approaches address this gap through text alignment or annotation, treating the problem as multimodal supervision rather than representation design. We introduce Physics-Aware Visual Abstraction (PAVA), a plug-and-play physics-aware representation and annotation interface that maps physical reanalysis fields to visually identifiable and semantically grounded perceptual abstractions for supervision and evaluation in vision-language reasoning. Building on PAVA, we construct OmniClimate-TC, a benchmark for tropical cyclone analysis spanning five classes of reasoning and nine tasks, with 243,890 physically grounded instruction-tuning pairs. Using PAVA-aligned supervision, we adapt VLMs and provide evidence that this representation design improves reasoning over tropical cyclone hazard fields. Our results position OmniClimate-TC as a benchmark for multimedia reasoning over structured geophysical fields, and highlight representation design as a key ingredient for physically grounded reasoning in scientific media.
1 Introduction
Tropical-cyclone reanalysis fields challenge VLMs because their physical meaning is organized through continuous spatial patterns rather than discrete objects. The paper addresses this representational mismatch with PAVA and builds a physics-grounded benchmark and supervision framework.
- Motivation: Reanalysis encodes cyclone hazards through continuous, physically constrained fields whose semantics differ from the object-centric structure of natural images.Cyclone evolution and hazard footprints depend on coupled variables and organization across scales.
- Motivation: Core meteorological concepts such as intensity extrema, asymmetry, and extent are defined over space and scale rather than directly readable as objects in raw heatmaps.Consequently, annotations are expensive and difficult to standardize.
- PAVA: PAVA reframes extreme-weather understanding as representation design by exposing physics-grounded perceptual structure for supervision, annotation, and evaluation.It serves as a plug-and-play interface rather than an auxiliary inference-time modality.
- PAVA: PAVA transforms wind, precipitation, and sea-level pressure fields into cyclonecentric abstractions encoding extremes, intensity organization, asymmetry, and spatial hazard extent.These abstractions provide a shared perceptual interface for language-addressable physical concepts.
- Benchmark and models: OmniClimate-TC spans five reasoning categories and nine tasks, comprising 243,890 physically grounded instruction-tuning pairs.The benchmark is paired with ClimateTCX models adapted for physics-grounded multimedia reasoning.
- Benchmark and models: Across multiple tasks, ClimateTCX consistently outperforms strong baselines, supporting PAVA-aligned representation design for reasoning over hazard fields.The reported outcome concerns physics-grounded reasoning across the benchmark’s tasks.
2 Related Work
Prior meteorological VLM benchmarks and climate-modeling systems have expanded multimodal and physically constrained forecasting capabilities, but raw field outputs still lack explicit hazard semantics. Existing multimodal works largely rely on models to infer meteorological meaning from heatmap textures, leaving interpretation dependent on implicit visual learning or expert effort.
- Vision-Language Models for Weather and Climate: Weather and climate multimodal benchmarks established text–field alignment and expanded toward Earth-system, contour-tracking, and spatiotemporal reasoning tasks.Examples include WeatherQA, CLLMate, OmniEarth-Bench, ClimateIQA, and MeteorPred.
- Vision-Language Models for Weather and Climate: These works predominantly use implicit visual learning, asking models to infer meteorological semantics directly from heatmap textures through end-to-end supervision.The approach leaves physically meaningful field semantics unstated in the visual interface.
- Extreme Weather Hazards and the Semantic Interpretation Gap: Forecasting and climate-modeling systems have progressed toward advanced architectures and physically constrained simulation frameworks.The passage cites Pangu-Weather, GraphCast, GenCast, Aurora, NNCAM, NeuralGCM, CondensNet, and ACE2.
- Extreme Weather Hazards and the Semantic Interpretation Gap: Raw outputs from weather and climate models lack explicit hazard semantics, making manual expert interpretation necessary and difficult to scale.The passage characterizes this dependency as unscalable and error-prone as data volumes grow.
3 Method
The method represents tropical-cyclone events as spatiotemporal physical fields from daily ERA5 reanalysis. Because these fields express meaning through spatial organization and constrained patterns, the framework motivates physics-aligned visual abstractions.
- Problem Setup and Motivation: A tropical-cyclone event is represented by spatiotemporal physical fields across K variables and times in the event duration T.The variables include wind, precipitation, and sea-level pressure.
- Problem Setup and Motivation: The fields are derived from daily ERA5 reanalysis over the event duration.ERA5 supplies the physical inputs used in the method’s event representation.
- Problem Setup and Motivation: Unlike object-centric inputs, reanalysis fields encode event semantics through spatial organization and physically constrained patterns rather than discrete entities.This mismatch with multimodal perceptual interfaces motivates physics-aligned visual abstractions.
- Benchmark comparison: The benchmark comparison identifies physics-grounded labels as labels or questions derived from continuous physical fields rather than post-hoc textual annotation.This distinction is stated in the Table 1 caption.
A Data curation
OmniClimate-TC aligns tropical cyclone records with ERA5 fields and uses PAVA to convert continuous hazards into interpretable labels, visualizations, and instruction-tuning data. The benchmark covers multiple perception and compositional reasoning tasks across globally distributed events.
- Dataset Collection & Curation: 2,489 tropical cyclone events are aligned with ERA5 daily fields, using wind components, precipitation, and mean sea-level pressure from 1979–2025.The corpus combines historical disaster records with globally available meteorological reanalysis.
- Physics-Aware Visual Abstraction: Temporal maxima, extrema locations, intensity categories, asymmetry, rainband structure, and area-weighted coverage provide physical labels for hazard-field reasoning.Localization uses event-aggregated precipitation and wind fields; intensity follows Beaufort and physics-informed rainfall categories, while extent uses latitude-corrected grid-cell areas and thresholds.
- Physics-Aware Visual Abstraction: PAVA maps continuous reanalysis fields to discrete, visually interpretable attributes grounded in established meteorological diagnostics.Its operators target attributes that are discriminable across severity or structural categories and suitable for language-based reasoning.
- OmniClimate-TC Construction: OmniClimate-TC represents wind and precipitation hazards as event-centric heatmaps paired with instructions constrained by PAVA-derived labels.The benchmark includes four direct perception categories and a fifth compositional category for mechanism-focused physical impact analysis.
- OmniClimate-TC Construction: 243,890 instruction-tuning examples are built from 4,978 heatmaps covering 2,489 events, approximately 130 countries, and 46 oceanic regions.Perceptual tasks each account for 24.5% of examples, while Physical Impact Analysis accounts for 2%; chronological splits follow approximately 7:1:2.
4 Experiment
Experiments show that general-purpose VLMs struggle with physically calibrated cyclone-field interpretation, while ClimateTCX performs most strongly and consistently across perception and impact reasoning. Ablations, expert review, and perception–reasoning analyses support PAVA-aligned representations as a reliable interface for physically grounded hazard understanding.
- Main results: General-purpose VLMs often fail on calibrated intensity and spatial extent, whereas ClimateTCX achieves the strongest and most balanced perception results.Closed-source models improve aggregate accuracy but retain weaknesses in intensity grounding and extent reasoning.
- Main results: ClimateTCX performs best across physical-impact metric groups, while combining perceptual and impact supervision gives the strongest overall balance.Lexical fluency alone does not establish physical correctness; the mixed strategy improves faithfulness, attribution, and semantic consistency.
- Ablation analysis: Removing any perceptual abstraction degrades its associated task and downstream impact reasoning, with extent and intensity removals causing especially broad losses.Extent removal most sharply harms footprint reasoning, while intensity removal weakens magnitude recognition, HSF, and HAA.
- Expert evaluation and Dataset Reliability: 40 cyclone cases yielded 360 expert-reviewed instances, with consistency, plausibility, and informativeness all scoring above 4.0 on 5-point scales.Weighted Cohen’s 𝜅 ranges from 0.69 to 0.75, and the overall value is 0.72.
- Perception–reasoning relation: Perceptual grounding generally correlates with downstream impact-analysis performance, but the descriptive relation is not explained by model family or scale alone.ClimateTCX occupies the upper-right regime, indicating jointly strong perception and impact reasoning.
5 Conclusion
The paper frames extreme-weather multimodal understanding as a representation problem and introduces PAVA as a physics-grounded interface for continuous geophysical fields. OmniClimate-TC and ClimateTCX show improved perception and impact reasoning, while the study remains limited to tropical cyclones and its current annotation pipeline.
- 5 Conclusion: PAVA makes meaningful structure in continuous geophysical fields accessible to VLMs as a supervision and evaluation interface.It is studied primarily as a representation interface rather than an auxiliary inference-time modality.
- 5 Conclusion: PAVA-aligned supervision improves perceptual prediction and impact reasoning over tropical-cyclone hazard fields.The conclusion highlights representation design beyond model scale as a key factor in scientific-field multimedia reasoning.
- 5 Conclusion: The study is limited to tropical cyclones and the current PAVA-aligned annotation pipeline, with matched non-PAVA controls needed to disentangle broader domain adaptation.Future work extends the interface-design recipe to other hazards and annotation schemes.