Source-linked AI summary
WeatherBench: A benchmark dataset for data-driven weather forecasting
Stephan Rasp, Peter D. Dueben, Sebastian Scher, Jonathan A. Weyn, Soukayna Mouatadid, Nils Thuerey
TL;DR
Data-driven methods may forecast global weather several days ahead, but prior studies lacked common data and evaluation metrics for direct comparison. The paper introduces WeatherBench, an ERA5-derived benchmark with clear metrics and baseline models, establishing a reproducible framework for medium-range forecasting while noting practical limitations of related hybrid approaches.
Problem
Global medium-range weather forecasting shows promise for data-driven methods, but the field lacks common datasets and evaluation metrics for direct comparison.
Method
The paper constructs an ERA5-derived, machine-learning-ready benchmark with standardized metrics, baseline models, reproducible code, and tutorials.
Results
The benchmark includes baseline scores from linear regression, deep learning, and purely physical forecasting models for comparison.
Takeaways & Limitations
WeatherBench provides a common framework for quantitatively comparing data-driven medium-range weather forecasting methods.
Takeaways & Limitations
Hybrid approaches remain constrained by poorly understood physical–machine-learning interactions that can produce unexpected instabilities and biases.
Abstract
from arXiv · showhide
Data-driven approaches, most prominently deep learning, have become powerful tools for prediction in many domains. A natural question to ask is whether data-driven methods could also be used to predict global weather patterns days in advance. First studies show promise but the lack of a common dataset and evaluation metrics make inter-comparison between studies difficult. Here we present a benchmark dataset for data-driven medium-range weather forecasting, a topic of high scientific interest for atmospheric and computer scientists alike. We provide data derived from the ERA5 archive that has been processed to facilitate the use in machine learning models. We propose simple and clear evaluation metrics which will enable a direct comparison between different methods. Further, we provide baseline scores from simple linear regression techniques, deep learning models, as well as purely physical forecasting models. The dataset is publicly available at https://github.com/pangeo-data/WeatherBench and the companion code is reproducible with tutorials for getting started. We hope that this dataset will accelerate research in data-driven weather forecasting.
1 Introduction
The paper frames global medium-range weather prediction as a challenging data-driven problem and proposes WeatherBench to make methods quantitatively comparable. It motivates purely data-driven models by their potential computational efficiency and scientific value while distinguishing them from post-processing, nowcasting, and hybrid approaches.
- Alternative approaches: Hybrid models replace selected uncertain or expensive physical components with machine-learning emulators but can introduce poorly understood interactions, instabilities, biases, and implementation difficulties.These drawbacks motivate studying a purely data-driven alternative rather than interfacing learning components with complex climate-model code.
- Research scope: The benchmark targets purely data-driven prediction of global atmospheric flow at 3- and 5-day lead times.These lead times combine substantial atmospheric nonlinearity with some remaining determinism and societal relevance for disaster preparation.
- Research scope: Medium-range forecasting requires modeling complex atmospheric dynamics and interactions among variables across multiple scales.This distinguishes it from location- or variable-specific post-processing, univariate short-range nowcasting, and approaches retaining physical large-scale dynamics.
- Motivation: Purely data-driven models could provide computationally cheaper forecasts and enable very large ensembles for estimating extreme-event probabilities.The motivation is to learn an efficient representation of underlying dynamical and physical equations, while the benchmark itself focuses on upper-level pressure and temperature fields.
- Research gap: Current research lacks a common benchmark challenge for data-driven global weather forecasting, making quantitative comparison difficult.Benchmark datasets can improve inter-comparability and foster competition in emerging research areas.
- Paper contribution: WeatherBench provides a ready-to-use dataset, comparison metrics, baseline models, and reproducible tutorials for data-driven weather forecasting.The framework can also be applied to post-processing using the same metrics.
2 Overview of previous work
Earlier studies demonstrated potential for data-driven global medium-range forecasting using neural networks and reanalysis or simplified-model data. Their results varied with forecast design and model complexity, reinforcing the need for a common benchmark.
- Scope of prior work: Previous work explored data-driven prediction of large-scale atmospheric states in the medium range using several datasets, architectures, and evaluation settings.The reviewed studies include simplified general circulation models, reanalysis data, and operational or reduced-resolution physical baselines.
- Dueben and Bauer (2018): A CNN predicting only geopotential performed best among the tested approaches but did not beat the low-resolution physical baseline.The result illustrates that strong performance in one data-driven configuration did not exceed the physical benchmark.
- Forecast design: Direct networks trained for a specific forecast time outperformed iterative networks in simplified-model experiments.Other studies generated longer forecasts by repeatedly feeding model outputs back as inputs.
- Scher and Messori: Simplified-GCM studies achieved high predictive skill and stable long climate runs, whereas more realistic GCMs retained short-term skill but failed to produce stable realistic climate runs.Architectures tuned on simplified GCMs also transferred to more complex GCMs and showed some skill on single-level reanalysis data.
- Weyn et al. (2019): Using two input time steps 6 hours apart and predicting two output steps performed better than a single-step setup.The best CNN outperformed climatology through 120 h and approached persistence at longer leads up to 14 days.
- Synthesis: Differences in data, methods, and evaluation across prior studies highlight the importance of a common benchmark case for comparing prediction skill.WeatherBench addresses this comparability problem with standardized data and evaluation.
3 Dataset
The benchmark uses ERA5 reanalysis data processed for machine learning, with multiple resolutions, atmospheric and surface variables, and downloadable workflows. Its design balances broader coverage and higher resolution against computational and data-quality constraints.
- Data source and structure: ERA5 reanalysis provides the benchmark’s training and testing data, with hourly observations from 1979 to 2018 on a 0.25° latitude-longitude grid and 37 vertical levels.The processed benchmark uses a subset and regridded form of this raw archive.
- Resolution and processing: The benchmark offers 5.625° (32×64), 2.8125° (64×128), and 1.40525° (128×256) grids produced by bilinear interpolation.Lower resolutions address GPU-memory and I/O constraints, and powers of two support common deep-learning architectures.
- Vertical coordinates: Pressure coordinates include standard levels used by climate models, with interpolation applied when surface pressure is below a requested level.The selected levels can support pretraining with CMIP6 data.
- Availability: The processed data are distributed in yearly NetCDF files, with the full 5.625° dataset occupying 191GB and higher resolutions requiring 4× and 16× more storage.A repository and template workflows support downloading and processing the data.
- Variables: The dataset includes prognostic atmospheric variables such as geopotential, temperature, humidity, and wind, alongside two-dimensional impact-related fields including 2-meter temperature, 10-meter wind, cloud cover, and precipitation.Potential vorticity and other derived atmospheric quantities are also described, while precipitation quality requires caution.
- Static fields and extensibility: Time-invariant fields include land-sea mask, seven-category soil type, orography, and latitude-longitude grids to represent surface and geographic information.The repository also provides scripts for downloading additional variables or regridding to other resolutions.
4 Evaluation
The benchmark separates validation from final testing, evaluates standard atmospheric fields at a common resolution, and uses latitude-weighted error and correlation metrics. It also highlights that metric choice matters especially for precipitation.
- Data splits: 2017 and 2018 serve as the final evaluation period, while 2016 is used for validation and 1979–2016 is recommended for hyperparameter tuning.The split avoids overlap between training and testing and discourages random validation splits because meteorological fields are temporally correlated.
- Resolution consistency: Higher-resolution predictions are downscaled to 5.625° for evaluation, and baseline scores differed by less than 1% across evaluated resolutions.This supports the conclusion that little information is lost by evaluating at the coarser resolution.
- Verification fields: Z500 is the primary verification field and T850 the secondary field, representing synoptic-scale pressure distribution and broader temperature trends, respectively.T850 is generally above the planetary boundary layer and is less affected by diurnal variations.
- Metrics: RMSE is the primary metric, defined as the mean latitude-weighted error between model forecasts and ERA5 truth.The benchmark also evaluates latitude-weighted anomaly correlation coefficient and mean absolute error.
- Metric sensitivity: Differences between metrics are small for smooth fields such as Z500 and T850 but matter substantially more for intermittent precipitation.Baseline scores are also provided for total 6-hourly accumulated precipitation and 2-meter temperature.
5 Baselines
The benchmark compares persistence, climatology, linear regression, CNN, and physical-model forecasts at multiple lead times and resolutions. Results show that weekly climatology is a necessary baseline, direct CNN forecasts outperform linear regression at longer leads, and coarse physical-model skill varies by field.
- Persistence and Climatology: Weekly climatology is significantly better than the single-mean climatology and approximately matches persistence between 1 and 2 days.A useful forecast system therefore needs to beat both weekly climatology and persistence.
- Physical NWP model run at coarser resolution: The coarse-resolution T42 and T63 physical-model runs have intermediate Z500 skill, while T42 performs significantly worse for T850.The poorer near-surface temperature performance is attributed to resolution and topography representation, and the model was not tuned for these resolutions.
- Simple linear regression: At 5 days, iterative linear regression performs just as well as direct forecasting and is about as good as weekly climatology.The iterative model predicts 6 hours ahead and is called 20 times to produce a 5-day forecast.
- Simple convolutional neural network: CNN forecasts beat linear regression at 3 and 5 days, but the 5-day forecasts are only marginally better than weekly climatology.The authors present this simple CNN as a starting point for more sophisticated data-driven methods.
- Simple convolutional neural network: Chained iterative CNN forecasts perform well to about 1.5 days, after which errors grow quickly and diverge.The paper notes that this weakness could reflect the overly simplistic network architecture; longer-lead direct training performed better than chaining short-term forecasts.
6 Discussion
The discussion identifies weather-specific obstacles for data-driven forecasting, including atmospheric structure, limited effective training data, large computational demands, and evaluation gaps. It also outlines extensions toward probabilistic, extreme-event, and climate prediction, alongside promising research directions.
- Weather-specific challenges: Atmospheric three-dimensionality, vertically varying dynamics, and latitude-dependent grid distortion challenge standard convolutional assumptions.The atmosphere is three-dimensional, vertical grid spacings change, and regular latitude-longitude grids stretch toward the poles.
- Weather-specific challenges: 40 years of hourly data yield around 350,000 samples, but temporal correlation may reduce this to around 15,000 independent weather situations.The passage notes that it is difficult to know whether this amount is sufficient for complex networks without overfitting.
- Weather-specific challenges: Random image augmentations such as rotations are questionable because the physical x and y directions are distinct, leaving suitable augmentation methods unresolved.The discussion identifies data augmentation for physical fields as an outstanding problem and mentions ensemble analyses and forecasts as a possible source of training diversity.
- Technical challenges: Large datasets and higher resolutions can exceed CPU or GPU memory, make data loading a bottleneck, and require multi-GPU training.A single variable with ten levels at 5.625° resolution uses around 30 GB, while higher-resolution competitive models may require multi-GPU scaling.
- Probabilistic forecasts and extremes: The benchmark does not currently address probabilistic forecasting or extreme weather, although uncertainty and rare socially important events motivate separate evaluation.Probabilistic extensions require probabilistic scores, while extremes were omitted because defining them and standardizing their evaluation is ambiguous.
- Climate simulations: Climate prediction poses a distinct challenge: prior iterative data-driven models produced stable runs with significant biases and poor seasonal cycles, rather than physically reasonable long-term predictions.The discussion distinguishes weather prediction at a specific future time from climate evaluation using long-term statistics and notes that climate change requires extrapolation beyond historical states.
- Promising research directions: Future work should increase training data and network complexity while exploring dataset variables and the benchmark’s higher horizontal resolution.The discussion presents these as promising directions for data-driven weather forecasting.
7 Code and data availability
The dataset and supporting code are publicly available, with instructions for downloading data and evaluating forecasts provided through the WeatherBench repository.
- Availability: The dataset is available through the stated data repository, while code and downloading and evaluation instructions are provided on GitHub.The repository supports access to the benchmark workflow.
8 Conclusions
The paper presents WeatherBench as a reproducible benchmark for global medium-range data-driven weather forecasting. It addresses a missing common basis for method comparison through defined metrics, baselines, accessible code, and a collaborative repository.
- Benchmark scope: WeatherBench targets global medium-range forecasting over roughly 2 days to 2 weeks, a challenging task with distinctive atmospheric and data-handling constraints.These include three-dimensional and anisotropic atmospheric structure, non-uniform grids, limited training data, and large data volumes.
- Scientific impact: The benchmark responds to the field’s lack of a common dataset enabling inter-comparison and is intended to accelerate research across atmospheric and data science.The paper frames this as a foundation for collaboration and further investigation of fully data-driven methods.
- Evaluation: The benchmark defines RMSE for 500 hPa geopotential and 850 hPa temperature as a simple measure of medium-range forecast success.These two fields are the primary verification targets.
- Reproducibility: The code repository includes a quick-start notebook, neural-network training and evaluation utilities, and periodic convolutions in Keras.The repository is designed to help users read data, train models, and compare predictions with targets.
- Reproducibility: All baselines and results are fully reproducible, with baseline predictions saved in the data repository and the data assigned a permanent DOI.This supports repeatable comparison and citation of benchmark results.
- Collaboration: The GitHub repository is intended as an evolving communication and collaboration hub where users can contribute useful code and discuss the project.GitHub issues are identified as the main platform for project questions.
Appendix A: Additional metrics
The appendix defines ACC and MAE and presents additional baseline evaluations using RMSE, ACC, and MAE across weather variables at 5.625° resolution.
- Metrics: ACC is defined using forecast and target anomalies relative to climatology.The prime denotes the difference from climatology.
- Metrics: MAE uses the same structure as MSE but replaces squared differences with absolute differences.
- Additional evaluations: RMSE results cover 2-meter temperature and 6-hourly accumulated precipitation across different baselines.
- Additional evaluations: ACC results cover 500 hPa geopotential, 850 hPa temperature, 2-meter temperature, and 6-hourly accumulated precipitation.
- Additional evaluations: MAE results cover the same four variables at 3-day and 5-day forecast times.
A VISION FOR THE DEVELOPMENT OF BENCHMARKS TO BRIDGE GEOSCIENCE AND DATA SCIENCE, in: 7th International
This section consists of references spanning climate modeling, weather forecasting, machine learning, deep learning, probabilistic prediction, and scientific computing.
- Climate and geoscience: The references include foundational and applied work on climate-model design and reanalysis data.
- Forecast evaluation: The bibliography also covers probabilistic forecasting, scoring rules, ensemble prediction, and quantitative precipitation forecasting.
- Machine learning: Several cited studies address machine learning and deep learning methods for forecasting, post-processing, and climate applications.
- Computational methods: Additional references concern neural architectures, optimization, regularization, spherical convolutions, and scalable scientific workflows.