PaperlayerView exact revision on arXiv

Source-linked AI summary

EarthAlign: Learning Language-Aligned Representations of Earth System Data

Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma, Dat Tien Nguyen, Youssef Wehbe, Duncan Watson-Parris, Salman Khan

arXiv:2610.14624v1cs.CVcs.AIcs.LG

TL;DR

Earth system–language alignment remains underexplored, in part because diverse meteorologically grounded language supervision is limited. EarthAlign pairs weather descriptions with meteorological data, and MeteoLIP learns shared representations that improve over corresponding CLIP and SigLIP baselines across three evaluation tasks.

  • Problem

    General-purpose alignment between Earth system data and language remains underexplored, partly because diverse language supervision grounded in meteorological conditions and natural phenomena is limited.

    Source: §1 · p. 2

  • Method

    EarthAlign pairs event, disaster, and normal-weather records with ERA5 data and four grounded caption styles, while MeteoLIP learns aligned representations with a dual encoder and MetTok.

    Source: §1 · p. 2

  • Results

    MeteoLIP improves over corresponding CLIP and SigLIP baselines across retrieval, weather-condition classification, and physical-variable-estimation tasks.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    EarthAlign and MeteoLIP provide a foundation for learning and evaluating language-aligned representations of Earth system data.

    Source: §6 · p. 10

    Underlying source reporting and coverage patterns shape geographic, temporal, and event-distribution biases; MeteoLIP currently represents each sample with a single meteorological snapshot.

    Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 34

Abstract

from arXiv · show

Interpreting Earth system conditions is central to understanding environmental risks and informing decisions across weather- and climate-sensitive sectors. Existing multimodal weather systems primarily target task-specific prediction, generation, or question answering, while general-purpose alignment between Earth system data and language remains underexplored due to the limited availability of diverse language supervision grounded in meteorological conditions. To fill this gap, we introduce EarthAlign, a large-scale dataset with broad geographic coverage spanning 2000-2024 that combines 10 expert-curated natural event and disaster data sources with climatology-derived normal-weather samples. Each sample includes temporal and geospatial information and four grounded caption styles aligned with 63-variable ERA5 meteorological data. Building on EarthAlign, we introduce MeteoLIP, a dual-encoder framework for learning aligned meteorology-language representations. MeteoLIP incorporates MetTok, which uses a geolocation- and time-conditioned query to aggregate variable-specific patch embeddings into a compact sequence of meteorology tokens. To comprehensively assess Earth system multimodal representations, we introduce three downstream tasks: cross-modal retrieval, weather-condition classification, and physical variable estimation. Our experiments show that MeteoLIP improves upon the corresponding CLIP and SigLIP baselines across these tasks. Further analyses highlight the importance of meteorological input, temporal and geospatial conditioning, and multi-style supervision for meteorology-language alignment. Thus, EarthAlign and MeteoLIP provide a foundation for learning and evaluating language-aligned representations of Earth system data. Our code, dataset, and weights are publicly available at https://github.com/mbzuai-oryx/EarthAlign.

1 INTRODUCTION

Earth system models learn from physical-variable objectives, while general-purpose alignment with language remains underexplored because diverse, meteorologically grounded language supervision is limited. EarthAlign and MeteoLIP address this gap, and MeteoLIP improves over corresponding CLIP and SigLIP baselines across the proposed evaluation tasks.

1 INTRODUCTION

  • General-purpose Earth system–language alignment remains underexplored, partly because diverse language supervision grounded in meteorological conditions and phenomena is scarce.The paper distinguishes this gap from weather datasets designed mainly for task-specific prediction, generation, or question answering.Source: §1 · p. 2
  • EarthAlign pairs 553,738 samples from 2000–2024 with 63-variable ERA5 snapshots and four metadata-grounded caption styles.The dataset combines natural-event, disaster, and normal-weather records with broad geographic coverage.Source: §1 · p. 2
  • MeteoLIP uses MetTok to aggregate variable-specific patch embeddings into compact meteorology tokens conditioned on temporal and geospatial information.Training samples one of four caption styles for each meteorological input.Source: §1 · p. 2
  • Meteo-CLIP and Meteo-SigLIP improve over their corresponding CLIP and SigLIP baselines across retrieval, classification, and physical-variable-estimation tasks.The paper establishes these tasks to assess cross-modal alignment, semantic understanding, and quantitative inference.Source: §1 · p. 3

2 RELATED WORK

Prior vision-language methods extend beyond natural images, but Earth system alignment remains underexplored amid limited large-scale language supervision paired with geolocation and time. Existing weather-language datasets primarily target task-specific tasks, whereas EarthAlign supports representation alignment and evaluation.

2 RELATED WORK

  • Vision-language alignment has been adapted to domains including medical imaging and remote sensing, while Earth system alignment remains underexplored because large-scale language supervision paired with geolocation and time is scarce.EarthAlign addresses this gap with geocoded captions describing historical weather conditions and natural phenomena.Source: §2 · p. 3
  • WeatherQA, ClimateIQA, Aquilon, CLLMate, and MeteorPred primarily support instruction following or task-specific weather questions, detection, prediction, or warning generation.EarthAlign instead provides four grounded caption styles for vision-language alignment and introduces retrieval, classification, and physical-variable-estimation tasks.Source: §2 · p. 3

3 DATASET AND EVALUATION TASKS

EarthAlign combines expert-curated event and disaster records with climatology-based normal-weather samples, then pairs records with grounded captions and ERA5 data. The paper evaluates representations through retrieval, weather-condition classification, and physical-variable estimation.

3.1 DATASET CONSTRUCTION

  • EarthAlign contains 553,738 samples spanning 2000–2024, with temporal and geospatial information and four caption styles across natural phenomena and normal weather.The dataset combines event/disaster records and climatology-based normal-weather samples.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Ten expert-curated sources provide natural-event and disaster records, which are filtered, standardized, and geocoded when source records give only textual locations.Processing yields 489,185 records with phenomenon categories, metadata, and temporal and geospatial information.Source: §3.1 · pp. 3–4
  • Normal-weather samples are selected against local ERA5 climatology thresholds, retaining an ADM1-hour only when at least 80% of sampled cells are normal.The procedure yields 64,553 samples; only 1.6% and 11% of candidate hours pass the filter in California and Abu Dhabi, respectively.Source: §3.1 · p. 4
  • Four metadata-grounded captions offer natural-event, physical, impact, and atmospheric-pattern perspectives, with a 95.68% overall pass rate in an audit of 440 captions.The generation prompt excludes unsupported causes, impacts, forecasts, and descriptive modifiers.Source: §3.1 · p. 4
  • Each record is paired with a single multivariate ERA5 patch at its timestamp and spatial extent, using 28 single-level and 35 pressure-level inputs for 63 variables.ERA5 supplies globally gridded hourly data at 0.25° resolution.Source: §3.1 · p. 4

3.2 EVALUATION TASKS

  • The evaluation covers bidirectional text–meteorology retrieval, classification among 11 weather conditions, and estimation of cyclone maximum sustained wind speed and minimum central pressure.The tasks assess alignment, category-level semantics, and quantitative inference, respectively.Source: §3.2 · p. 5

4 METEOLIP: ALIGNING METEOROLOGICAL DATA WITH LANGUAGE

MeteoLIP maps meteorological inputs and text into a shared representation space, using MetTok to incorporate variable-specific, temporal, and geospatial information. It trains with sampled caption styles and applies shared representations to retrieval, classification, and value estimation.

4.1 MODEL ARCHITECTURE AND TRAINING OBJECTIVE

  • MeteoLIP uses separate transformer-based meteorology and text encoders to map both modalities into a shared representation space.Meteorological inputs are supplemented with year, day-of-year, hour, and latitude–longitude information.Source: §4 · p. 5, §4.1 · p. 5
  • MetTok patchifies variables separately, then uses time- and location-conditioned queries to cross-attend to each patch’s variable embeddings, retaining P tokens rather than V P tokens.The resulting tokens are projected to the meteorology encoder dimension, avoiding the additional quadratic self-attention cost of variable-wise tokenization.Source: §4.1 · pp. 5–6
  • Training samples one caption style per meteorological input and uses a cross-modal contrastive objective to increase paired similarity and decrease similarity for unmatched pairs.This exposes the model to diverse linguistic descriptions of the same Earth system condition.Source: §4.1 · p. 6

4.2 TASK-SPECIFIC INFERENCE

  • Retrieval ranks meteorological candidates against text queries or text candidates against meteorological queries using shared-representation cosine similarity.Classification builds text prototypes by ensembling prompts for each class and compares them with meteorological representations.Source: §4.2 · p. 6
  • Physical-variable estimation compares meteorological representations with text prototypes for candidate cyclone values and takes the median of the top-ρ candidates as the estimate.The value-conditioned templates cover maximum sustained wind speed and minimum central pressure.Source: §4.2 · p. 6

5 EXPERIMENTS AND RESULTS

MeteoLIP improves on corresponding CLIP and SigLIP baselines across retrieval, classification, and physical-variable estimation. Experiments also examine training choices and the effects of caption styles, meteorological input, and data scale.

5.1 EXPERIMENTAL SETUP

  • Training uses a year-based split that withholds 2011–2013, while CLIP and SigLIP baselines concatenate meteorological variables and train on natural-event-focused captions.The split trains on 2000–2010 and 2014–2024 to evaluate on unseen intermediate years.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • MeteoLIP improves on CLIP and SigLIP across retrieval, classification, and physical-variable estimation.Meteo-SigLIP reaches 66.07 M2T and 66.79 T2M R@1; under linear probing it achieves F1 80.75 and Top-1 accuracy 92.35, with R2 values of 71.85 for maximum sustained wind speed and 77.36 for minimum central pressure.Source: §5.2 · pp. 7–8, Table 1 · p. 8

5.3 ABLATION STUDIES

  • Natural-event-focused captions achieve the highest retrieval performance, and increasing training supervision from one to four caption styles improves both CLIP and Meteo-CLIP.With four styles, CLIP reaches 31.64/31.10 M2T/T2M R@1 from 20.25/20.03; Meteo-CLIP reaches 58.79/60.10 from 34.24/55.55.Source: §5.3 · p. 8, Table 2 · p. 9, Table 3 · p. 9
  • OpenAI CLIP initialization improves Meteo-CLIP retrieval over training from scratch.M2T R@1 rises from 34.46 to 58.79 and T2M R@1 from 41.43 to 60.10.Source: §5.3 · p. 8, Table 4 · p. 9
  • Temporal conditioning contributes substantially to retrieval, while metadata alone performs worse than full MetTok on classification and sustained-wind estimation.Metadata-only scores are 32.64 F1 and −155.14 R2, compared with 47.91 F1 and 62.27 R2 for full MetTok; removing time cues from captions widens the retrieval gap.Source: §5.3 · pp. 8–9, Table 5 · p. 9, Table 6 · p. 9
  • Performance generally improves with more training data, while source-held-out retrieval varies substantially across record sources.The full training set performs best across tasks; the authors identify local record sources as important and cross-source generalization as challenging.Source: §5.3 · p. 9

6 CONCLUSION

EarthAlign pairs meteorological data with multi-style descriptions of natural phenomena and normal weather, while MeteoLIP learns language-aligned representations. MeteoLIP improves over corresponding CLIP and SigLIP baselines across three downstream tasks; qualitative and appendix materials provide additional examples and evaluations.

6 CONCLUSION

  • EarthAlign pairs meteorological data with multi-style descriptions of natural phenomena and normal-weather conditions for learning language-aligned representations of meteorological data.The conclusion presents the dataset as a foundation for advancing language-aligned multimodal representation learning of Earth system data.Source: §6 · pp. 9–10
  • MeteoLIP improves over corresponding CLIP and SigLIP baselines across cross-modal retrieval, weather-condition classification, and physical variable estimation.The analyses also highlight meteorological input, temporal and geospatial conditioning, multi-style supervision, and diverse training data.Source: §6 · p. 10

APPENDIX

  • The appendix evaluates training-data scale across retrieval, classification, and physical variable estimation, and reports a source-held-out M2T R1 table averaged over three runs.A separate dataset-comparison table covers temporal and geographic coverage, evidence source, phenomena, size, and evaluation tasks.Source: Table 7 · p. 9, Figure 4 · p. 16, Table 8 · p. 16

A QUALITATIVE EXAMPLES

  • EarthAlign examples include geographically diverse natural-event records and detailed tornado, hail, and landslide records with temporal and spatial metadata and four caption styles.Source: Figure 5 · p. 17
  • Qualitative examples show cross-style retrieval, cold-wave classification, and physical-value similarity profiles, while full MetTok retrieves more meteorologically relevant events than geolocation and time alone.The examples include peaks around predicted minimum central pressure and maximum sustained wind speed.Source: Figure 6 · p. 18, Figure 7 · p. 19

B DATASET SAMPLES RECORD CONSTRUCTION

The dataset construction standardizes expert-curated natural-event and disaster records, adds normal-weather samples selected relative to local climatology, and generates four grounded caption styles from metadata.

B.1 EXPERT-CURATED NATURAL EVENT AND DISASTER RECORDS

  • Ten expert-curated sources undergo source-specific processing to produce standardized metadata records that ground multi-style caption generation.Source: §B.1 · p. 20

B.2 CLIMATOLOGY-BASED NORMAL-WEATHER SAMPLING

  • Climatology-based sampling identifies eligible normal-weather hours using grid-cell- and administrative-unit-level rules, yielding 64,553 samples from 2000–2024.Thresholds use the 1991–2020 reference period and a centered ±2-day window; the criteria are variable-specific.Source: §B.2 · pp. 22–23, Table 10 · p. 23

C.1 CAPTION GENERATION

  • Gemini 3.1 Pro generates four metadata-grounded caption styles, and the prompt requires supported details while prohibiting unsupported causes, impacts, forecasts, and modifiers.When metadata does not support a requested style, the caption is reduced to a minimal grounded restatement rather than inferred content.Source: §C.1 · pp. 23, 25

C.2 CAPTION AUDIT AND ERROR ANALYSIS

A manual audit checks generated captions across four styles and identifies weather/climate focus and location accuracy as the most frequent error categories.

C.2 CAPTION AUDIT AND ERROR ANALYSIS

  • A trained graduate annotator audited 440 captions from 11 sources across four styles, counting a caption as passing only when it satisfied every audit criterion.The checks included date/time, location, event/phenomenon, trigger/cause, unsupported details and modifiers, weather/climate focus, and standalone clarity.Source: §C.2 · pp. 26–27, Table 11 · p. 27
  • 7 erroneous captions failed the weather/climate-focus criterion, followed by 6 location-accuracy errors.Other reported failures involved unsupported modifiers (2), trigger/cause accuracy (2), unsupported added detail (1), and event/phenomenon accuracy (1).Source: §C.2 · p. 27

D ERA5 DATA ALIGNMENT AND PREPROCESSING

The pipeline retrieves ERA5 meteorology for each record by aligning it to a timestamp and spatial region, then standardizes and normalizes the extracted patches.

D ERA5 DATA ALIGNMENT AND PREPROCESSING

  • Each meteorological input contains 63 ERA5 variables: 28 single-level variables and 7 pressure-level variables at five pressure levels.Source: D ERA5 DATA ALIGNMENT AND PREPROCESSING · p. 27, Table 13 · p. 28
  • Records are aligned to one ERA5 hour using reported timestamps or uniformly sampled times, with timestamps standardized to UTC before retrieval.For date-only records, an hour is sampled within the local day and then converted to UTC.Source: D ERA5 DATA ALIGNMENT AND PREPROCESSING · p. 27
  • Records with point locations receive a 14°×14° region, while bounding boxes are expanded or adjusted to form square regions containing the original extent.Smaller boxes are expanded to a minimum extent where possible; larger extents determine the square region size.Source: D ERA5 DATA ALIGNMENT AND PREPROCESSING · p. 27
  • Extracted ERA5 patches are resampled to 56×56 grid cells and each variable is normalized using its across-sample mean and standard deviation.Mean wave direction uses nearest-neighbor interpolation; other variables use linear interpolation.Source: D ERA5 DATA ALIGNMENT AND PREPROCESSING · p. 27

E ADDITIONAL DATASET STATISTICS AND DISTRIBUTIONS

EarthAlign spans 19 regions, with North America most represented, while classification training and test sets have broadly similar class distributions except for drought.

E ADDITIONAL DATASET STATISTICS AND DISTRIBUTIONS

F.1 WEATHER CONDITION CLASSIFICATION ANALYSIS

MeteoLIP variants improve category-level classification over their corresponding baselines, while challenging temporal conditions remain difficult. Physical baselines, downstream fine-tuning, and WeatherQA transfer further assess the learned representations.

F.1 WEATHER CONDITION CLASSIFICATION ANALYSIS

  • Meteo-CLIP and Meteo-SigLIP improve category-level results over their respective baselines; linear probing yields macro F1 scores of 79.98 and 80.75, respectively.Atmospheric river, convective storm, high wind, and winter weather show notable text-prototype improvements; drought, heat wave, and cold wave remain challenging, while atmospheric river, cyclone, and convective storm exceed 94 F1 under linear probing.Source: §F.1 · p. 29, Table 14 · p. 30
  • A single meteorological snapshot limits MeteoLIP’s ability to capture persistence and accumulated anomalies relevant to drought, heat waves, and cold waves.The analysis motivates future extensions using multiple timestamps or temporal sequences.Source: §F.1 · pp. 29–30

F.2 PHYSICAL BASELINES FOR CYCLONE INTENSITY ESTIMATION

  • Linear probing of Meteo-CLIP achieves R2 of 76.36 and 71.34 for minimum central pressure and maximum sustained wind, outperforming direct ERA5 and regression baselines.Meteo-SigLIP achieves R2 of 77.36 and 71.85 for those respective targets.Source: §F.2 · p. 30, Table 15 · p. 31

F.3 DOWNSTREAM FINE-TUNING

  • MeteoLIP initialization achieves the best downstream fine-tuning results, including 81.31 F1 and 92.97 Top-1 accuracy for classification.For maximum sustained wind speed it achieves R2 of 76.15 with 7.56 MAE, and for minimum central pressure R2 of 81.24 with 5.24 MAE.Source: §F.3 · p. 31, Table 16 · p. 31

F.4 DOWNSTREAM TRANSFER TO WEATHERQA

  • Frozen Meteo-CLIP representations achieve 49.22 ± 4.45% accuracy on WeatherQA Area Affected and 51.73 ± 2.41% conditional accuracy on Concerning.Concerning is evaluated only when the affected area is correctly predicted.Source: §F.4 · p. 31

F.5 TRAINING DYNAMICS

Training dynamics differ across evaluation measures, and retrieval performance depends on positional encoding and model capacity. Scaling effects vary with initialization and model type.

F.5 TRAINING DYNAMICS

  • As Meteo-CLIP training progresses, test loss decreases and retrieval and physical-variable estimation generally improve, while classification F1 fluctuates.Frozen-encoder linear-probing F1 and Top-1 accuracy generally improve across checkpoints, while Top-3 accuracy remains approximately 98.8%.Source: §F.5 · p. 33

F.6 POSITIONAL ENCODING IN THE METEOROLOGICAL ENCODER

  • Learnable positional embeddings achieve the best retrieval performance across pretrained and scratch initialization, with a larger gap under scratch initialization.Combining learnable positional embeddings with 2D RoPE provides no additional gains.Source: §F.6 · p. 33, Table 17 · p. 33

F.7 MODEL CAPACITY

  • Scaling pretrained Meteo-CLIP from ViT-B to ViT-L lowers retrieval performance, whereas scratch-trained Meteo-CLIP and pretrained CLIP benefit from scaling.For pretrained Meteo-CLIP, M2T/T2M R@1 falls from 58.79/60.10 to 54.24/57.21; from-scratch Meteo-CLIP improves from 34.46/41.43 to 36.68/41.86.Source: §F.7 · p. 33

G LIMITATIONS AND FUTURE DIRECTIONS

EarthAlign’s source coverage shapes dataset biases, and MeteoLIP currently represents each sample with a single snapshot. The paper identifies broader data sources, temporal inputs, and higher-resolution products as future directions.

G LIMITATIONS AND FUTURE DIRECTIONS

  • Underlying source reporting and coverage patterns shape EarthAlign’s geographic, temporal, and event-distribution biases.The United States, Indonesia, and France have more records than many other countries, reflecting extensive coverage from NOAA Storm Events, IndDE, and FraDE.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 34
  • MeteoLIP represents each sample using a single meteorological snapshot, rather than multiple timestamps or temporally aggregated inputs.The paper identifies broader Earth-system products, operational forecasts, other reanalyses, and higher-resolution regional or downscaled products as possible extensions.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 34
  • Regional or downscaled products with higher spatial resolution may improve representation of localized phenomena.The paper also identifies alignment across temporal scales, spatial resolutions, and Earth-system data sources as a future direction.Source: G LIMITATIONS AND FUTURE DIRECTIONS · p. 34

Related papers

Loading 2610.14624v1…