PaperlayerView exact revision on arXiv

Source-linked AI summary

A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway

Siyan Chen, Lars Uebbing, Eirik Mikal Samuelsen, Georgios Leontidis, Arnt-Børre Salberg, Sébastien Lefèvre, Robert Jenssen, Kristoffer Wickstrøm

arXiv:2609.10564v1physics.ao-phcs.LG

TL;DR

Local wind forecasting skill remains uncertain for machine-learning models in Northern Norway’s complex terrain. This study evaluates FCN3, GraphCast, and HRES against station observations, finding that HRES performs slightly best overall while FCN3 performs best under high winds, which all models substantially underestimate.

  • Problem

    Evidence remains limited on how well machine-learning weather forecasts perform at station level in complex terrain and during high-wind conditions.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates gridded wind forecasts from FCN3, GraphCast, and HRES against interpolated station observations in Northern Norway.

    Source: §3.1 · p. 4

  • Results

    HRES achieves the lowest overall RMSE at 2.89 m s−1, while FCN3 and GraphCast remain comparable and all models substantially underestimate high wind speeds.

    Source: §1 · p. 2, §6 · pp. 8–9, §5 · p. 8

  • Takeaways & Limitations

    Machine-learning weather forecasting is competitive with HRES for local wind in Northern Norway, but complex geography and high-wind events remain important targets for improvement.

    Source: §5 · p. 8, §6 · p. 9

    The evaluation covers only 12 stations in Northern Norway, so its findings may not extend directly to regions with different climates, terrain, or observation networks.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent machine learning weather prediction (MLWP) models have demonstrated remarkable forecasting skill on global reanalysis-based benchmarks. However, their performance remains unclear in challenging environments such as Northern Norway, where narrow fjords and rapidly changing weather result in highly variable local wind conditions. In this case study, we evaluate FourCastNet3 (FCN3), GraphCast, and ECMWF High Resolution Forecast (HRES) for wind speed forecasting using multi-year station observations from Northern Norway, focusing on their relative performance, generalization beyond the training period, and performance under high-wind conditions. Our results show that HRES slightly outperforms FCN3 and GraphCast, with an overall RMSE of 2.89 $\mathrm{m\,s^{-1}}$, compared to 2.96 $\mathrm{m\,s^{-1}}$ for FCN3 and 2.94 $\mathrm{m\,s^{-1}}$ for GraphCast. Notably, the MLWP models maintain comparable performance beyond their respective training periods, with no clear evidence of noticeable degradation. FCN3 performs best under high-wind conditions, although all models substantially underestimate strong winds. Our findings suggest that MLWP has become competitive with NWP for local wind, but further refinements are still needed to capture complex terrain better.

1 Introduction

This study addresses whether globally benchmarked MLWP models extend to station-level 10 m wind forecasting in Northern Norway’s complex terrain. It compares FCN3 and GraphCast with HRES across lead times, stations, wind categories, training periods, and high-wind conditions.

Motivation

  • NWP remains operationally important but is computationally expensive and slow, motivating comparison with newer MLWP approaches.The introduction identifies wind forecasting as important for transportation, renewable energy, maritime operations, and disaster risk management.Source: §1 · p. 1

Research gap

  • Existing MLWP evaluations rely mainly on global gridded benchmarks, leaving their practical station-level performance in complex terrain insufficiently understood.Global weather models may fail to resolve fine-scale topography and resulting local wind variations.Source: §1 · pp. 1–2

Research questions

  • The study asks how accurately machine-learning and physics-based models predict 10 m wind, how performance changes between training and non-training years, and how it varies with lead time, location, and high-wind events.These questions reflect the practical importance of reliability at specific locations, decision-relevant lead times, and during high-wind situations.Source: §1 · p. 2

Study design

  • The evaluation covers FCN3, GraphCast, and HRES across forecast lead times, stations, and wind-force categories, with particular emphasis on high-wind conditions.The study evaluates two state-of-the-art MLWP models and HRES at 12 weather stations in Northern Norway.Source: §1 · p. 2

Key findings

  • HRES achieves the lowest overall RMSE, while FCN3 has the smallest bias and performs best under high-wind conditions; all models substantially underestimate strong winds.Source: §1 · p. 2

2 Related Work

Related work contrasts physics-based NWP with data-driven MLWP, tracing the development of global neural forecasting models and the shift from ERA5-based evaluation toward station observations. Existing studies demonstrate competitive global skill but raise concerns about temporal generalization, extreme events, and transfer to local point locations.

Traditional weather forecasting

  • NWP combines physical equations, numerical solvers, observations, and data assimilation, with operational global systems including IFS and GFS.These systems mainly differ in their dynamical cores and grid structures.Source: §2 · p. 2

Machine learning-based weather forecasting models

  • Large reanalysis datasets and deep learning enabled diverse global MLWP architectures, including FourCastNet [22], FCN2 [53], Pangu-Weather, Graphcast, FuXi [21], and AIFS.Earlier data-driven forecasting relied primarily on statistical time-series and conventional machine-learning methods [52].Source: §2 · pp. 2–3

Evaluation of global MLWP models

  • Global MLWP evaluations typically compare models against ERA5, with HRES as a common physics-based baseline, and WeatherBench 2 standardizes grid-based metrics across variables and lead times [28].These studies report competitive global skill but also identify limitations in temporal generalization and extreme-event prediction [36].Source: §2 · p. 3
  • ERA5-based gridded evaluation may not establish performance at out-of-grid point locations because ERA5 is a model-based gridded atmospheric reconstruction rather than independent point observations.Reported studies also found degraded performance during high-impact extreme events [69].Source: §2 · p. 3
  • Recent studies therefore compare forecasts with station observations, including Aurora and EPT-2 at more than 13,000 and 14,000 worldwide stations and Pangu-Weather at 183 Norwegian stations for up to 60-hour lead times [58] [60].These station-based evaluations targeted 10-m wind speed and 2-m temperature.Source: §2 · p. 3

3 Station-Based Evaluation of Weather Forecasting Models

The study evaluates FCN3, GraphCast, and HRES against observations from 12 Northern Norwegian stations using spatially and temporally matched wind-speed forecasts. It assesses overall accuracy, robustness across lead times, stations, years, and high-wind events with complementary continuous and categorical metrics.

Evaluation framework

  • Forecasts from FCN3, GraphCast, and HRES are bilinearly interpolated to 12 station locations and evaluated against matched in situ wind-speed observations.FCN3 and GraphCast are generated from ERA5 at 0.25° resolution, while HRES data come from WeatherBench 2; all products are aligned by valid time.Source: §3.1 · pp. 3–4, §3.2 · pp. 4–5

Forecast generation

  • Forecasts initialized at 00:00 UTC are evaluated every 6 h from +6 to +72 h, producing 12 lead times matched to observations at each valid time.The forecasts use a single multi-step autoregressive rollout, with station-level outputs stored after immediate interpolation from weekly processing batches.Source: §3.1 · p. 4

High-wind evaluation

  • High-wind performance is evaluated separately for observations satisfying w_obs ≥ 10.8 m s^-1, using recalculated metrics and forecast–observation scatter plots.The threshold follows the Yr online weather forecasting service, and the plots assess agreement and systematic over- or underestimation across the wind-speed range.Source: §3.1 · p. 4

Evaluation metrics

  • Forecast performance is summarized with RMSE, bias, normalized RMSE, normalized bias, and Pearson correlation across all stations and years, then separated by lead time, station, and year.Normalized metrics account for differences in characteristic wind-speed magnitude among stations, while correlation measures agreement between observed and predicted speeds.Source: §3.2 · p. 5

Categorical evaluation

  • Categorical skill across wind-speed ranges is assessed with the Equitable Threat Score, which adjusts for hits expected by random chance and ranges from negative values to 1.Higher ETS values indicate better categorical forecast skill; the analysis complements continuous error metrics across wind-speed categories.Source: §3.2 · p. 5

4 Results

Across 2016–2022 station observations, HRES achieved the lowest overall wind-speed RMSE, while FCN3 had the smallest bias and performed best under high-wind conditions. Performance differences varied by lead time, station, terrain setting, and year, but annual results showed no consistent temporal degradation.

Overall results

  • 2.89 m s−1 overall RMSE: HRES narrowly outperformed GraphCast (2.94 m s−1) and FCN3 (2.96 m s−1), with only the HRES–FCN3 difference statistically significant.HRES also had the highest correlation (r=0.685), while FCN3 had the smallest bias at -0.85 m s−1 and lowest absolute nBias of 15.63%.Source: §4.1 · p. 5, Table 1 · p. 6

Lead time analysis

  • HRES had the lowest RMSE at every lead time, with errors increasing from approximately 2.7–2.8 m s−1 at +6 h to 3.1–3.2 m s−1 at +72 h.FCN3 and GraphCast were similar at shorter lead times, but FCN3 increased notably after +48 h and had the highest RMSE at +72 h.Source: §4.2 · pp. 6–7

Station-based analysis

  • Station-wise errors were broadly similar, but Bardufoss had RMSE below 1.8 m s−1 for all models whereas Fakken was most challenging, reaching 4.58 m s−1 for HRES.Model rankings varied by station: FCN3 led at Tromsø–Langnes and Fakken, GraphCast at Bardufoss, Torsvåg Fyr, and Banak, and HRES at several sites.Source: §4.2 · p. 7

Terrain-based analysis

  • Marine-coast stations generally had lower RMSE than fjord and coastal-ridge stations, with Fakken showing the largest errors across all three models.The analysis classified stations by fjords, marine coasts, coastal ridges, and valleys.Source: §4.2 · p. 7

Evaluation years

  • Annual RMSE varied between years but showed no consistent increase over time, suggesting changing forecast difficulty rather than systematic temporal drift.The comparison covered periods within and outside the models’ respective training datasets.Source: §4.3 · p. 7

High-wind conditions

  • FCN3 performed best under high-wind conditions, achieving the lowest RMSE, nRMSE, and absolute bias, although all models substantially underestimated strong winds.High-wind normalized biases ranged from -43.42% to -45.59%, with RMSE values around twice those of the overall evaluation.Source: §4.4 · p. 7, Table 3 · p. 8

5 Discussion

At station level in Northern Norway, HRES slightly outperforms FCN3 and GraphCast, but their differences are small relative to station-to-station variability. The findings are limited by the study’s coverage of 12 stations and complex-terrain generalizability.

5 Discussion

  • Forecast performance varies markedly among stations, partly reflecting differences in local geography and exposure.Bardufoss’s sheltered valley setting, weak winds, and winter inversions contribute to low RMSE, while coarse grids may misrepresent local terrain.Source: §5 · p. 8
  • HRES slightly outperforms FCN3 and GraphCast at station level, but all three remain comparable over Northern Norway.The aggregate result indicates a modest numerical advantage rather than consistent HRES superiority.Source: §5 · p. 8
  • The differences between models are less than 0.07 m s−1, considerably smaller than the variation observed among stations.Source: §5 · p. 8

Limitations and future work

  • Because the evaluation uses only 12 stations in Northern Norway, its conclusions may not directly generalize to regions with different climates, terrain, or observation networks.Future work should evaluate additional regions with similarly complex coastal and mountainous terrain.Source: §5 · p. 8

6 Conclusion

HRES achieved the lowest station-level wind-forecast errors in Northern Norway, while FCN3 and GraphCast delivered comparable results. Errors increased with lead time and high-wind conditions, where all models underestimated wind speeds.

6 Conclusion

  • 2.89 m s−1 overall error for HRES was lower than 2.96 m s−1 for FCN3 and 2.98 m s−1 for GraphCast.The models were compared for station-level wind forecasts in Northern Norway.Source: §6 · p. 8
  • Errors increased with lead time, while spatial differences between stations were more pronounced.Source: §6 · p. 8
  • Under high-wind conditions, all models became less accurate and significantly underestimated wind speeds.This degradation was indicated by increased RMSE and more negative bias, motivating improved representation of complex geography and high-wind events.Source: §6 · pp. 8–9

A Computational Resources · B Beaufort Force · C Weather Stations

The study documents its computational setup, wind-intensity classification, and 12-station evaluation network. These details define the resources, environmental categories, and geographic context for interpreting forecast performance in Northern Norway.

A Computational Resources

  • FCN3 and GraphCast experiments used the Olivia GPU cluster, with seven annual SLURM tasks covering 2016–2022 and one GPU per task.Each task requested four CPU cores, 192 GB memory, and up to 40 hours; tasks could run concurrently subject to GPU availability.Source: A Computational Resources · p. 14
  • Table A.1 summarizes execution times and peak memory usage for individual annual FCN3 and GraphCast tasks.Source: Table A.1 · p. 14

B Beaufort Force

  • The Beaufort force scale categorizes wind intensity by wind speed for the study’s analysis.Table B.1 provides the corresponding Beaufort categories and wind-speed ranges.Source: B Beaufort Force · p. 14, Table B.1 · p. 14

C Weather Stations

  • The evaluation used 12 observation stations whose geographical characteristics are aggregated in Table C.1.Source: C Weather Stations · p. 14
  • Station coordinates, elevation, terrain category, and zoomed-in local surroundings provide context for interpreting station-level differences in forecast performance.The local surroundings are shown in Fig. G.5.Source: C Weather Stations · p. 14
  • The stations span coastal, fjord, and inland-valley environments across Northern Norway, with elevations from near sea level to approximately 76 m.Source: C Weather Stations · p. 14

D Evaluation against ERA5

Against ERA5, all three forecast models achieved much lower RMSE values, ranging from 1.14 m s−1 to 1.31 m s−1. GraphCast had the lowest RMSE and nRMSE, while FCN3 had the smallest systematic bias.

Evaluation against ERA5

E Comparison of CARRA1 and ERA5 with Station Observations

CARRA1 represents local near-surface winds at the stations better than ERA5, achieving lower RMSE and nRMSE and a smaller negative bias. Its regional 2.5 km resolution is substantially finer than ERA5’s approximately 31 km grid and is designed to better resolve complex Arctic terrain.

Dataset characteristics

  • CARRA1 uses an approximately 2.5 km grid, substantially finer than ERA5’s approximately 31 km resolution, to better represent regional geography and local atmospheric conditions.CARRA1 is a regional Arctic reanalysis based on HARMONIE-AROME.Source: E Comparison of CARRA1 and ERA5 with… · p. 14

Station-observation comparison

Prior evidence

F Station-level Heatmap

Figure G.4 compares RMSE, nRMSE, Bias, and nBias for FCN3, GraphCast, and HRES across 12 stations. nRMSE and bias vary substantially by station, with Bardufoss showing low absolute RMSE but among the highest nRMSE values.

F Station-level Heatmap

  • 33.52%–77.03%: nRMSE spans this range across stations and models in the heatmap.The heatmap reports RMSE, nRMSE, Bias, and nBias for FCN3, GraphCast, and HRES across 12 stations.Source: F Station-level Heatmap · p. 15
  • Highest nRMSE values occur at Bardufoss, Sørkjosen Lufthavn, and Tromsø-Langnes, while lower values occur at Fruholmen Fyr, Andøya, and Slettnes Fyr.The passage identifies station-level contrasts in normalized RMSE across the evaluated models.Source: F Station-level Heatmap · p. 15
  • Bias ranges from −3.50 to 1.02 m s−1 and is negative for most station-model combinations, while nBias ranges from −46.21% to 17.55%.The most negative bias values occur at Fakken, Banak, and Tromsø-Langnes; Bardufoss has the lowest absolute RMSE but among the highest nRMSE values.Source: F Station-level Heatmap · p. 15

G Station-Level Scatter Plots

Station-level scatter plots compare observed wind speeds with FCN3, GraphCast, and HRES predictions, providing a station-specific view of model performance across the remaining stations.

Station-level comparisons

Related papers

Loading 2609.10564v1…