AIMIP Phase 1: systematic evaluations of AI weather and climate models

Evaluation framework

• The project uses climate-science analysis methodologies and standardized output formats, including tools developed for CMIP, to evaluate AI models alongside established climate-model practices.This design aims to steer AI-based climate-model development and socialize these models within the broader climate science community.Source: §1 · p. 2

Protocol and scope

• AIMIP Phase 1 defines an intercomparison of AI weather and climate models trained on global reanalysis and forced by historical SST and SIC from 1979–2024.The protocol is designed in the style of AMIP and targets longer-term simulations than weather-focused evaluations.Source: §1 · p. 2

Motivation

• The protocol builds on recently documented AI models with stable and accurate historical AMIP simulation capabilities, including NeuralGCM, ACE2, DLESyM, and cBottle.These models provide the basis for applying the Phase 1 intercomparison design to AI-enabled climate simulation.Source: §1 · p. 2

Future development

• Phase 1 provides a first step toward AI weather and climate models that could eventually be interactively coupled to AI-based ocean and sea-ice components.Future AIMIP phases may evaluate coupled AI models once they are more mature.Source: §1 · p. 2

2 AIMIP Phase 1 goals and protocol

• AIMIP Phase 1 compares time-mean climate, trends, variability, weather phenomena, and extremes across AI-driven global-atmosphere models trained exclusively on ERA5 and forced by historical SST and SIC.The project also seeks common CMIP7-compatible outputs and greater visibility into AIWCM strengths and weaknesses.Source: §2 · p. 3

2.1 What types of AIWCMs are in scope for AIMIP Phase 1?

• AIMIP Phase 1 permits autoregressive replacements, hybrid physics–AI architectures, and conditional weather-state sampling, provided models represent long-term climate statistics and boundary-forced evolution.The protocol does not restrict AIWCM construction despite differences from conventional GCMs.Source: §2.1 · p. 3

2.2.1 ERA5 used for training and testing

• Models train exclusively on ERA5 from 1979–2014 and run continuously from late 1978 through 2024, including spinup, training, and 2015–2024 out-of-sample testing.ERA5 is the evaluation reference despite poorly constrained precipitation and some climate trends.Source: §2.2.1 · p. 3

2.2.2 Monthly SST and SIC forcing datasets

• Simulations use a custom monthly SST/SIC dataset extended through 2024, while excluding CO2 and other anthropogenic radiative forcings as input features to avoid proxy-based overfitting.SST trends can nevertheless correlate with learned radiative-forcing effects without establishing a physical cause.Source: §2.2.2 · p. 4, §2.2.3 · p. 4

2.2.4 Land surface simulation

• Each model supplies a 5-member ensemble to estimate unforced variability and model dispersiveness, although larger ensembles may be needed to robustly estimate forced responses.Land-surface treatment is unconstrained provided required near-surface atmospheric variables are supplied.Source: §2.2.5 · p. 4, §2.2.4 · p. 4

2.2.6 ∆SST = +2 K and +4 K simulations

• Uniform +2 K and +4 K SST ensembles test generalization to unseen conditions, where biases or instability may occur and no definitive ground truth exists.Physics-based models provide estimates of plausible climate changes, but performance there may not indicate near-historical predictive utility.Source: §2.2.6 · pp. 4–5

2.3.1 CMIP compatibility

• Submissions follow CF- and CMIP-style variable, unit, naming, grouping, and attribute conventions on native grids no coarser than 500 km, enabling CMIP evaluation tools.AIWCMs may provide fewer outputs, including no explicit cloud properties, requiring evaluation tools to interpret and harmonize coverage.Source: §2.3.1 · p. 5

2.3.2 Three-dimensional fields

• Required outputs include temperature, specific humidity, and eastward and northward winds at pressure levels 1000, 850, 700, 500, 250, 100, and 50 hPa, plus 500 hPa geopotential height.Models should report monthly means of all requested outputs for October 1978 through December 2024; surface fields should be reported if predicted.Source: §2.3.2 · p. 5, §2.3.3 · p. 5, §2.3.4 · p. 6

3 Participating Models

• AIMIP Phase 1 received eight AI weather and climate model submissions from six development groups, summarized in Table 1.The submissions are described individually in the remainder of this section.Source: §3 · p. 6, Table 1 · p. 7

3.1 ACE2.1-ERA5

• ACE2.1-ERA5 is an autoregressive SFNO-based emulator with a 6-hourly timestep that predicts required atmospheric diagnostics, including near-surface variables and fields at required pressure levels.AIMIP SST and SIC forcing were conservatively regridded to its native 1° Gaussian grid and linearly interpolated to the 6-hourly timestep; five ensemble members were generated for each experiment.Source: §3.1 · p. 8

3.2 ArchesWeather and ArchesWeatherGen

• ArchesWeather is a data-driven deterministic weather forecasting model, whereas ArchesWeatherGen is probabilistic, uses the same convolutional encoder-decoder and SwinTransformer architecture, and samples state residuals with a flow matching-based generative module.Source: §3.2 · p. 8

3.3 cBottle-1.3

• cBottle-1.3 is a diffusion model conditioned on SST and other inputs that independently samples atmospheric states, producing lower-variance monthly means than autoregressive models.Correlated-noise inference reintroduces temporal correlation, but the resulting smooth states are not expected to reproduce correct atmospheric dynamics because training remains non-autoregressive.Source: §3.3 · pp. 9–10

3.4 DLESyM

• DLESyM is a coupled convolutional Earth-system model with ConvNeXt blocks, gated recurrent units, a U-Net framework, and 6-hourly autoregressive atmospheric forecasts.For AIMIP, its ocean component is replaced by prescribed SST, but its parsimonious output set limits evaluation to near-surface temperature, 850 hPa temperature, 500 hPa geopotential height, and surface precipitation.Source: §3.4 · p. 10
• DLESyM does not conform fully to the AIMIP specification because its data begin in 2015 and do not include the 1979–1982 period.This difference may affect its relative skill in holdout evaluations.Source: §3.4 · p. 11

3.5 Monthly Diffusion at 1.5◦resolution (MD-1.5 v0.9)

• MD-1.5 v0.9 is an autoregressive latent diffusion model comprising an encoder, decoder, and denoising predictor, with all three networks trained jointly end-to-end.The encoder and decoder form a conditional variational autoencoder, while the predictor is a conditional latent diffusion model; all three networks use low-rank spectral operators based on SFNO.Source: §3.5 · p. 11

3.6 NeuralGCM and NeuralGCM-HRD

• NeuralGCM and NeuralGCM-HRD add machine-learned output heads that decode diagnostic variables from a prognostic atmospheric state on native 2.8° and, for HRD, downscaled 1° grids.The output heads are jointly optimized with the learned physics module using a CRPS loss; HRD combines upsampled prognostic fields with high-resolution auxiliary inputs and enforces coarse-grid consistency for temperature and precipitation.Source: §3.6 · p. 12

3.7 CMIP6 model: GFDL-CM4

• GFDL-CM4 supplies one physically based global atmospheric model for comparison, using AM4 output from CM4 AMIP and perturbed-SST simulations at approximately 1° resolution with 33 vertical levels.Unlike the five-member AIWCM ensembles, the submission provides one GFDL-CM4 AMIP ensemble member.Source: §3.7 · p. 13

4.1 Evaluation metrics and methods

• The evaluation comprises five metrics: E1, E2, E3, and E5 use 1979–2024 monthly-mean predictions, whereas E4 uses daily-mean predictions.Source: §4.1 · p. 13

4.2 E1: Biases

• AIWCMs typically have lower global RMSB than GFDL-CM4, but test-period biases are generally higher than training-period biases and vary across models.Temperature biases are usually larger over land and sea ice than over ocean, while precipitation bias patterns differ substantially near the equator; direct comparisons are limited when models submit different variables.Source: §4.2 · p. 14

4.3 E2: Trends

• Most AIWCMs correctly reproduce the sign of training-period trends but tend to underestimate trend magnitudes, including Arctic warming maxima and enhanced warming over land.Source: §4.3 · pp. 17–20

4.4 E3: ENSO response

• AIWCM temperature ENSO-coefficient errors are small over tropical oceans, while precipitation errors are small in training and increase modestly during testing.ERA5 coefficients are largest over the tropical Pacific, whereas extra-tropical test-period values may reflect insignificant correlations from the short period.Source: §4.4 · p. 20
• Most AIWCMs have similar ENSO coefficient-error magnitudes, with higher test-period errors and slightly smaller errors than CM4 relative to ERA5.MD-1.5 v0.9 tends to have somewhat higher errors, while the training–test difference is fairly consistent across models because the test period is short.Source: §4.4 · p. 20

4.5 E4: Daily variability

• Daily temperature anomaly errors are smaller over ocean than land, but the largest values over northern Eurasia and northeastern North America are typically underestimated.Specified, slowly varying sea-surface temperatures likely contribute to smaller ocean errors, while monthly-interpolated forcing may explain some ocean underestimation.Source: §4.5 · p. 22
• Across a larger set of variables, AIWCMs generally underestimate daily anomaly magnitude for most variables, except precipitation, whose magnitude they overestimate across most wet regions; DLESyM is the exception.Source: §4.5 · p. 22

4.6 E5: Perturbed SST response

• SST-perturbation responses vary far more across AIWCMs than historical-period comparisons, with some matching CM4’s warming pattern while others underestimate ocean warming or predict implausible land cooling.NeuralGCM-HRD and DLESyM closely match CM4, whereas ACE2.1-ERA5, cBottle1.3, and MD-1.5 v0.9 underestimate ocean warming and predict cooling over land.Source: §4.6 · pp. 22, 24, Figure 9 · p. 25

Overall performance

• Most AIWCMs produce lower climate biases than GFDL-CM4 and capture most ENSO-response and daily-variability patterns across diverse modeling frameworks.The models span autoregressive, hybrid physics/AI, and conditional-diffusion frameworks using U-Nets, Fourier operators, vision transformers, and other architectures.Source: §5 · p. 26

Historical trends

• Warming-trend fidelity varies significantly across models, with some AIWCMs underpredicting air-temperature and related trends driven by increasing SSTs.For ACE2.1-ERA5, the discrepancy from a similar AMIP experiment may reflect excluding CO2 forcing, whose historical variations strongly correlate with warming trends.Source: §5 · p. 26

Out-of-sample behavior

• Physically plausible responses to extreme perturbed-SST experiments vary widely, and differences are not explained solely by how models handle missing land and sea-ice SSTs.Models use constant filling, zonal interpolation, or merged ocean and land/sea-ice fields; architecture and sensitivity to large-scale input perturbations may also matter.Source: §5 · p. 26

Limitations

• Because unseen-scenario guidance is a primary climate-model goal, the divergent out-of-sample responses indicate that AIWCMs’ ability to generalize remains a work in progress.Physically based models are explicitly built for this task, whereas the AIMIP results show that AI-based models may still be challenged by it.Source: §5 · p. 26

Contribution and outlook

• AIMIP Phase 1 shows that common experimental specifications and output formats are feasible despite diverse training methods, supporting future intercomparisons and AIWCM development.Future phases could include coupled Earth-system experiments, more extensive outputs, and additional evaluation metrics.Source: §5 · pp. 26–27

• AIMIP Phase 1 defines an AMIP-modeled experiment in which eight AIWCM submissions from six groups are trained on ERA5 data from 1979 to 2014 and forced only with specified SST, SIC, and solar insolation.The five-member ensembles run from October 1, 1978 through December 31, 2024, including a 10-year out-of-sample period.Source: §6 · p. 27
• The project evaluates models using biases, trends, ENSO response, daily variability magnitudes, and out-of-sample perturbed-SST response under common CMIP conventions.Source: §6 · p. 27
• AIWCMs more faithfully replicate ERA5 climate patterns than a conventional CMIP6 model and strongly simulate the ENSO response to specified SSTs.The intercomparison also finds a wide range of behavior in both in-sample and out-of-sample warming trends.Source: §6 · p. 27
• The models show major disagreement in their responses to very much out-of-sample perturbed SST experiments.Source: §6 · p. 27
• The publicly available AIMIP Phase 1 dataset enables further evaluation and experimentation through a common experiment and data format intended to support more robust and efficient climate modeling tools.The submissions represent snapshots of ongoing model development and should not be understood as fixed references for the models.Source: §6 · p. 27

Dataset motivation

• The compact 1979–2024 monthly SST and SIC dataset replaces CMIP forcing that ended in 2022 and produced interpolation overshoots and annual-mean SIC biases.The CMIP algorithm could yield SIC values below zero and affect annual-mean temperature in seasonal ice zones.Source: Appendix A · p. 35

Forcing construction

• The ERA5-based dataset averages daily 0.25° outputs into beginning-of-month forcings, linearly interpolating between them and extending through Jan. 1, 2025 for December 2024.Intermediate SST and SIC forcings are obtained by linear interpolation.Source: Appendix A · p. 35
• The forcing-generation method produces no data overshoots and preserves the annual time-mean of each forcing field, although individual monthly means are not exactly preserved.The dataset is intended for AIMIP Phase 1 inference runs.Source: Appendix A · p. 35

Validation and implementation

• When used with the ACE2.1-ERA5 emulator, the monthly forcings produce a climate nearly identical to daily forcing, including in seasonal sea-ice zones.Modeling groups may spatially interpolate the monthly forcing to their native grids.Source: Appendix A · p. 35

Known limitations

• ERA5 SIC has small land-mask incompatibilities near polar coastlines and lake boundaries; limiting SIC to 1 - land fraction is the easiest workaround.Filling lost sea ice into adjacent coastal cells is possible but likely not worth the added effort.Source: Appendix A · p. 35

Architecture

• cBottle1.3 generates AIMIP Phase 1 physics realizations with an Ensemble-of-Experts architecture that assigns different denoising stages to different networks.Higher-noise stages use less-trained or early-stopped networks to avoid overfitting at large noise levels.Source: Appendix B · p. 35

Noise-dependent checkpoints

• The model uses three networks: the first is active at noise lower than 10, the second between 10 and 100, and the third over 100.Source: Appendix B · p. 35

Additional realizations

• The appendix also lists additional checkpoint triplets, including training-state-000640000, training-state-002048000, and training-state-010112000.checkpoint.For p5, the checkpoints match p4 while the latent space is uncorrelated in time, making every sample fully independent.Source: Appendix B · p. 36

C1 Biases

• The appendix presents 2-meter air-temperature biases for each model ensemble over both training and test periods.These results are shown in Figs. C1 and C2.Source: C1 Biases · p. 36
• It evaluates RMSB at 1° resolution across seven pressure levels for air temperature, specific humidity, and wind.The wind diagnostics cover both eastward and northward components.Source: C1 Biases · p. 36
• The wind-bias evaluation distinguishes eastward and northward wind components.These diagnostics are included alongside air temperature and specific humidity in Figs. C3–C6.Source: C1 Biases · p. 36

C2 Trends

• C2 Trends reports annual and global mean 2-meter air-temperature series for each AIWCM and its ensemble members.The series are shown in Fig. C7.Source: C2 Trends · p. 36
• C2 Trends evaluates air-temperature trends over pressure levels at 1° resolution.These trends are presented in Fig. C8.Source: C2 Trends · p. 36
• C2 Trends evaluates specific-humidity trends over pressure levels at 1° resolution.These trends are presented in Fig. C9.Source: C2 Trends · p. 36
• C2 Trends includes 2-meter air-temperature bias over the 1979–2014 training period for each model ensemble member.The comparison is shown in Figure C1.Source: Figure C1 · p. 37

C3 ENSO response

• C3 ENSO response evaluates globally averaged ENSO coefficient errors.These errors are shown in Figure C10.Source: C3 ENSO response · p. 37
• C3 ENSO response includes 2-meter air-temperature bias over the 2015–2024 test period for each model ensemble member.The comparison is shown in Figure C2.Source: Figure C2 · p. 38
• C3 ENSO response reports RSMB for air temperature over pressure levels during the training and test periods.The metric is shown in Figure C3.Source: Figure C3 · p. 39
• C3 ENSO response reports RSMB for specific humidity over pressure levels during the training and test periods.The metric is shown in Figure C4.Source: Figure C4 · p. 39

Daily variability

• Dry-day fraction errors are evaluated against ERA5 for models submitting daily surface data, using 1979 data at 1° resolution.A dry day is defined with a 0.1 mm precipitation cutoff.Source: C4 Daily variability · pp. 39, 44, Figure C11 · p. 44
• RSMB is shown for eastward and northward winds across pressure levels and training and test periods.Source: Figure C5 · p. 40, Figure C6 · p. 40
• Annual- and global-mean 2-meter air-temperature series are presented with models’ ensemble members, and global mean air-temperature trends are shown across pressure levels and periods.Source: Figure C7 · p. 41, Figure C8 · p. 41
• Global mean specific-humidity trends are evaluated at 1° resolution across pressure levels and training and test periods.Source: Figure C9 · p. 42
• ENSO-coefficient RMSE is evaluated against ERA5 coefficients using area-weighting over the globe on the 1° grid.The comparison covers the same variable set as Figure 2.Source: Figure C10 · p. 43, C4 Daily variability · p. 43

2.8°-resolution diagnostics

• At 2.8° resolution, the appendix evaluates biases and RMSB using NeuralGCM instead of NeuralGCM-HRD.These diagnostics correspond to the bias maps and RMSB analyses in Figs. 1 and 2.Source: Appendix D · p. 45, Figure D1 · p. 46, Figure D2 · p. 47
• The appendix also reports global-mean trend maps at 2.8° resolution, following the analysis in Fig. 5.The trend evaluation is presented in Fig. D3.Source: Appendix D · p. 45, Figure D3 · p. 48
• ENSO coefficient errors are evaluated at 2.8° resolution using the same comparison framework as Fig. 6.Figure D4 shows ERA5 ENSO coefficients and model coefficient errors.Source: Appendix D · p. 45, Figure D4 · p. 49
• Perturbation responses are additionally shown at 2.8° resolution using an evaluation analogous to Fig. 9.These results appear in Fig. D5.Source: Appendix D · p. 45, Figure D5 · p. 50