Source-linked AI summary
SolarBench: A global solar energy nowcasting benchmark
Yuhao Nie, Stephen Campbell, Quentin Paletta, Liwenbo Zhang, Tao Jing, Samer Chaaraoui, Jonathan Giezendanner, Andea Scott, Tao Sun, Cong Feng, Max Aragon, Jacques Camier, Adam Jensen, Florian Kotthoff, Yuexing Yang, Yang Ming, Mengying Li, Stefanie Meilinger, Yupeng Wu, Adam Brandt, Sherrie Wang
TL;DR
Fragmented datasets and inconsistent evaluation hinder fair comparisons and leave generalization across climates, cloud regimes, and PV systems difficult to assess. SolarBench combines harmonized multimodal data from 11 sites with a Python toolbox for reproducible model development and evaluation. Image-based models achieve 10–20% median forecast skill relative to persistence but often have below 5% ramp recall; large-scale pre-training with lightweight local adaptation cuts local data needs by more than 50% while maintaining competitive forecast performance.
Problem
Fragmented datasets and inconsistent evaluation hinder fair comparisons and leave generalization across climates, cloud regimes, and PV systems difficult to assess.
Source: Main · p. 3Method
SolarBench combines harmonized multimodal data from 11 sites with a Python toolbox for reproducible model development and evaluation.
Source: Conclusion · p. 13Results
Image-based models achieve 10–20% median forecast skill relative to persistence but often have below 5% ramp recall; large-scale pre-training with lightweight local adaptation cuts local data needs by more than 50% while maintaining competitive forecast performance.
Source: Conclusion · pp. 13, 15Takeaways & Limitations
Solar nowcasting assessment should pair aggregate accuracy with ramp evaluation because accuracy gains do not ensure capture of rapid irradiance fluctuations.
Source: Conclusion · p. 13The benchmark is non-exhaustive, its multi-continental site coverage is nonuniform, and it does not yet assess forecast value using downstream data.
Source: Conclusion · p. 15
Abstract
from arXiv · showhide
As the share of solar power grows, nowcasting weather-driven solar variability becomes critical for reliable energy system operation. State-of-the-art approaches increasingly apply deep learning to sky camera and geostationary satellite observations, but fragmented datasets and inconsistent evaluation make it difficult to determine whether reported improvements generalize across climates, cloud regimes, and photovoltaic (PV) systems. Here we introduce SolarBench, an open global benchmark for image-based solar nowcasting. SolarBench harmonizes more than six million sky and satellite images from 11 diverse sites spanning a decade, together with irradiance or PV output and auxiliary atmospheric data. An accompanying toolbox supports reproducible data access, processing, model development, and evaluation. Using SolarBench, we benchmark representative models and reveal a gap between average forecasting accuracy and the ability to capture rapid solar fluctuations. We further quantify predictability across cloud regimes and demonstrate data-efficient adaptation to new PV systems. SolarBench provides an extensible foundation for fair comparison and methodological innovation in solar nowcasting.
Main
SolarBench addresses fragmented datasets and inconsistent evaluation with a harmonized, multimodal global benchmark and reproducible toolbox. Its benchmarks reveal that aggregate accuracy gains do not ensure rapid-ramp capture, while predictability varies across cloud regimes and geographically diverse pretraining can reduce local PV data needs.
Main
- Fragmented datasets and inconsistent evaluation hinder assessment of whether gains generalize across climates, cloud regimes, and PV systems; SolarBench combines broad coverage, harmonized data, and standardized evaluation.The paper identifies this combination as missing from existing multi-continental benchmarks.Source: Main · pp. 3–4
The SolarBench database and toolbox
- The database spans 11 sites across five continents, integrates ground observations, satellite imagery and atmospheric products, and contains over six million images across records from 2014–2024.The toolbox standardizes data access, processing, model development, and evaluation, with modular support for contributions and extensions.Source: The SolarBench database and toolbox · pp. 4, 6
Enabling consistent and comprehensive benchmarking
- Image-based models improve median forecast skill by 10–20% over persistence, but no model dominates across sites, lead times, and metrics; ViT median RMSE rises from 86 to 133 W m−2 between 10 and 120 min.The tested satellite–sky fusion model does not consistently outperform sky-image approaches.Source: Enabling consistent and comprehensive… · pp. 8–9, Figure 3 · p. 10
Revealing solar predictability across sites
- Persistence rRMSE correlates strongly with deep-learning forecasting error, while patchy cloud conditions generally produce the greatest forecasting difficulty.Variability alone is a weaker and less consistent indicator; overall site predictability closely aligns with patterns under patchy clouds.Source: Revealing solar predictability across… · pp. 9, 11
Lowering the local data barrier to deployment
- Multi-site irradiance pretraining plus local PV fine-tuning reduces local-data requirements by more than 50% while maintaining comparable forecast skill; local training typically needs about six months to break even with persistence.More stable skill generally emerges after about one year, and more predictable sites tend to require more local data.Source: Lowering the local data barrier to… · pp. 11, 13
Conclusion
- Image-based deep-learning models achieve median forecast skill of 10–20% relative to persistence, yet often have ramp recall below 5% across tested models and lead times, revealing an accuracy–reliability gap.Source: Conclusion · p. 13
- SolarBench is extensible but non-exhaustive: its multi-continental site coverage is nonuniform, and downstream forecast-value evaluation remains a proposed future extension.The paper also calls for improved multimodal input design and fusion strategies.Source: Conclusion · pp. 13, 15
Data collection, preprocessing, and standardization
SolarBench integrates site data from published and contributed ground datasets with public satellite and atmospheric archives. Its processing pipeline quality-controls and aligns records, applies consistent day-level splits, and makes the curated data accessible through the toolbox.
Data collection, preprocessing, and standardization
- Ground observations come from published open datasets and collaborating groups, while satellite imagery, ERA5 reanalysis, and GFS forecasts come from public archives.Satellite sources are GOES, Himawari, or MSG, according to site location.Source: Data collection, preprocessing, and… · p. 15
- Records are quality-controlled and timestamped; sky images are aligned with sensor data, down-sampled to 64 × 64 pixels, and matched to interpolated measurements when cadences differ.Nighttime, missing, or physically implausible records are excluded; satellite imagery is cropped to a 256 km × 256 km site-centered window and resampled to 128 × 128 pixels.Source: Data collection, preprocessing, and… · pp. 15–16
- Roughly 5–10% of each site’s days are held out for testing, sampled across the collection period, with identical day assignments across available modalities.Daily profiles are visually inspected, with variability index values used as an additional guide.Source: Data collection, preprocessing, and… · p. 16
- Curated data are uploaded to Hugging Face in a consistent structure for access through the SolarBench toolbox.The SI documents data sources, variables, preprocessing, and split dates.Source: Data collection, preprocessing, and… · p. 16
Standardized solar nowcasting tasks
SolarBench defines short-term forecasting tasks at representative operational lead times, using aligned historical inputs and a forward-averaged target. The toolbox handles the standardized data and evaluation workflow, while allowing extension to finer-resolution forecast trajectories.
Standardized solar nowcasting tasks
- Each task predicts irradiance or PV output at one of five lead times—10, 30, 60, 90, or 120 min—rather than a full trajectory.The framework can be extended to finer-resolution time-resolved trajectories by changing the task configuration.Source: Standardized solar nowcasting tasks · p. 16
- Inputs pair imagery and non-imagery data within a 15-min historical window at 5-min resolution, and the target is the forward average over the next 5-min interval at the specified lead time.Scalar inputs are backward averages, while imagery is aligned to input timestamps.Source: Standardized solar nowcasting tasks · p. 16
- For sky–satellite methods, satellite inputs match the number of ground-image steps, so coarser satellite cadence can produce a longer effective input span.The toolbox handles loading, temporal alignment, sample construction, splits, training, and metric computation.Source: Standardized solar nowcasting tasks · pp. 16–17
Benchmark models
The benchmark combines varied image and measurement inputs with standardized evaluation across representative forecasting models. It includes a persistence reference, measurement-only and sky-image models, multimodal sky–satellite processing, and a vision transformer.
Benchmark models
- Across analyses, models combine different histories of sky or satellite imagery with irradiance or PV output; atmospheric measurements, forecasts, and reanalysis are also included for future multimodal studies.All models use the same splits, input windows, lead times, targets, and evaluation metrics for fair comparison.Source: Benchmark models · p. 17
- Persistence is the common reference and assumes the clear sky index remains constant over the forecast horizon.The index normalizes the observed target against its corresponding clear-sky estimate; estimates account for solar geometry, and CSI values are clipped to [0.05, 1.5].Source: Benchmark models · p. 17
- The measurement-only MLP uses historical irradiance or PV output, while SUNSET combines those measurements with past sky-image sequences in a convolutional network.SUNSET concatenates extracted image features with measurement inputs before fully connected layers generate the forecast.Source: Benchmark models · p. 17
- OmniVision processes sky images, satellite images, and output history with separate spatial encoders and modality-specific temporal modules before regression.Its inputs use the first three available satellite spectral bands and stack the scalar measurement with each modality’s image channels.Source: Benchmark models · pp. 17–18
- The ViT model embeds image patches and measurement tokens across time, then uses transformer attention and a global token to predict the target.Learnable spatial and temporal embeddings encode token positions and input steps.Source: Benchmark models · p. 18
Evaluation metrics
The benchmark evaluates both forecast levels and rapid ramp events, using normalized measures for cross-site comparison and explicit criteria for event detection.
Evaluation metrics
- RMSE measures absolute level error, while forecast skill compares RMSE with persistence and rRMSE normalizes RMSE by the target's root-mean-square value.These measures support comparison across sites with different target scales; unlike normalization by the mean, rRMSE accounts for both target level and variability.Source: Evaluation metrics · pp. 18–19
- Ramp magnitudes are defined from consecutive target values separated by five minutes, with CSI-based ramps used to remove solar-geometry trends and isolate cloud-driven variability.The metric evaluates whether models capture short-term local variability.Source: Evaluation metrics · p. 19
- Ramp recall measures the fraction of observed ramp events correctly predicted, requiring a qualifying change in threshold, direction, and relative magnitude error.For this demonstration, the relative error tolerance is δ = 30%.Source: Evaluation metrics · p. 20
Predictability and cloud regime analysis
The analysis uses persistence-model rRMSE to characterize site predictability, then compares conditions across four cloud regimes using sequence-level labels.
Predictability and cloud regime analysis
- Site-level predictability is estimated from persistence rRMSE, with higher values indicating lower predictability; its consistency is assessed by correlating it with each model's forecasting error across sites.The comparison uses Pearson correlation.Source: Predictability and cloud regime analysis · p. 20
- The test data are stratified into clear, thin-cloud, thick-cloud, and patchy-cloud regimes, with patchy conditions spanning both stable broken-cloud motion and more dynamic cloud evolution.Regimes are distinguished by cloud visibility, opacity, spatial continuity, and temporal structure.Source: Predictability and cloud regime analysis · p. 20
- Researchers manually label 1,000 test samples per site when available, judging four sky images at five-minute intervals without access to targets, predictions, or forecast errors.Labels use the sequence's cloud structure and evolution; persistence rRMSE is then calculated separately by regime for each site and lead time.Source: Predictability and cloud regime analysis · pp. 20–21
Data scaling and transfer learning
The experiments vary local training-data volume and cross-site pre-training to test data scaling and transfer to new PV systems; the accompanying dataset and toolbox are intended for public release.
Data scaling and transfer learning
- ViT is trained at each site on nested chronological subsets equivalent to 1, 3, 6, 12, or 24 months of local data, with a fixed held-out test set.Durations represent sample-volume equivalents, assuming eight daytime observation hours per day and five-minute sampling, not necessarily contiguous calendar periods.Source: Data scaling and transfer learning · p. 21
- Transfer experiments pre-train ViT on GHI from one, three, or six geographically distributed source sites, then fine-tune for PV forecasting at Stanford and Gatton.The target systems are a 30 kW Stanford rooftop installation and a 3.275 MW hybrid system at the University of Queensland's Gatton facility.Source: Data scaling and transfer learning · p. 21
- Fine-tuned models use warm-start initialization and are compared with scratch-trained ViT models at the same local-data volumes and a full-local-data reference.All variants are evaluated on the same held-out local test set using forecast skill.Source: Data scaling and transfer learning · p. 21
Data availability and Code availability
- SolarBench's sample dataset and toolbox are slated for public release through Hugging Face and GitHub, respectively, with future dataset updates supported.The toolbox is to be released under the BSD-3-Clause license, with documentation provided online.Source: Data availability · pp. 21–22, Code availability · p. 22
1 Supplementary Notes
The supplementary notes describe newly released ground-based datasets and the ERA5 and GFS atmospheric products incorporated into SolarBench.
Supplementary Note 1. Newly Released Ground-based Datasets
- New ground-based datasets undergo contributor preprocessing and quality control, followed by additional standardized checks before integration into SolarBench.The newly released datasets described here include Akwatia, Hong Kong, Kologo, Nottingham, and Sankt Augustin.Source: Supplementary Note 1. Newly Released… · p. 28
- Akwatia and Kologo provide high-frequency sky images and tilted irradiance, while Sankt Augustin supplies all-sky images and multiple irradiance measurements.Akwatia and Kologo data were collected in Ghana during 2022; Sankt Augustin observations span November 2020 to November 2022.Source: Supplementary Note 1. Newly Released… · pp. 28–29
- Nottingham and Hong Kong datasets pair all-sky imagery with irradiance and surface meteorology, with observations recorded at one-minute resolution.Nottingham's record covers 2023; Hong Kong imagery was recorded throughout the full daily cycle, and no additional quality control was applied before its data were provided.Source: Supplementary Note 1. Newly Released… · p. 29
- Irradiance checks include rare-value and diffuse-ratio filters, and the solar-zenith cutoff was reduced to 75° for fisheye-camera coverage; tower-shadow periods remain unfiltered.At Nottingham, a southeast tower casts shadows at low solar elevations, particularly on winter afternoons.Source: Supplementary Note 1. Newly Released… · pp. 29–30
Supplementary Note 2. Atmospheric Model Products
- ERA5 supplies global reanalysis estimates combining numerical models and assimilated observations, at hourly resolution on a 0.25° grid.SolarBench provides ERA5 data for each site in nearest-grid-point and surrounding 2×2-grid forms.Source: Supplementary Note 2. Atmospheric… · pp. 30–31
- GFS provides location-specific forecast data, with forecasts initialized every six hours and multiple grid-point or interpolation forms.Available forecast horizons and temporal intervals vary over the record; the supplement lists date-specific configurations.Source: Supplementary Note 2. Atmospheric… · p. 31
Supplementary Methods
The supplementary methods document implementation choices, model inputs, and training configurations for the benchmark experiments. Unless stated otherwise, models use a 15-minute history sampled every five minutes and are trained separately for each forecast lead time.
Supplementary Methods
- The supplementary methods cover model architectures, input configurations, training hyperparameters, and toolbox evaluation metrics beyond those reported in the main paper.The models were implemented in PyTorch v2.3.1; the described configurations are benchmark experiment implementation details.Source: Supplementary Methods · p. 31, Model Implementation and Training… · p. 31
Model Implementation and Training Details
- The MLP uses only historical irradiance or PV measurements, with Adam optimization and mean squared error loss.Training starts with a 40-epoch schedule and can continue until validation loss fails to improve for five consecutive epochs; the lowest-validation-loss checkpoint is retained.Source: Model Implementation and Training… · p. 32
SUNSET
The supplementary section details adaptations and training setups for SUNSET, OmniVision, and ViT, then describes additional toolbox metrics for forecast errors, bias, and temporal misalignment.
SUNSET
- SUNSET uses a 15-minute input sequence sampled every five minutes, rather than the original one-minute sampling, and is reimplemented in PyTorch.It follows the publicly available TensorFlow implementation and is trained with Adam and mean squared error loss.Source: SUNSET · p. 32
- OmniVision adapts the Eclipse implementation and predicts irradiance or PV output directly at each lead time, without iterative future-state prediction or future-image segmentation.The implementation follows OmniVision’s described architecture while using an Eclipse backbone without a satellite image branch.Source: SUNSET · p. 32
- The ViT model combines historical sky-image sequences with irradiance or PV measurements using temporal encoding inspired by TimeSFormer.It retains only corresponding irradiance or PV output measurements as numerical inputs rather than the original model’s additional meteorological measurements.Source: SUNSET · pp. 32–33
Additional Evaluation Metrics
- MAE measures average forecast-error magnitude without direction and is less sensitive than RMSE to a small number of large errors.Lower MAE indicates better forecast accuracy.Source: Additional Evaluation Metrics · p. 33
- MBE measures systematic overprediction or underprediction, with positive and negative values indicating the respective directions.A value near zero indicates little overall bias, although positive and negative errors may cancel.Source: Additional Evaluation Metrics · p. 33
- The toolbox includes an experimental temporal distortion mix (TDM) implementation for assessing the direction of temporal misalignment, but it was not evaluated and has not been independently validated.Further implementation details and usage guidance are provided in the online documentation.Source: Additional Evaluation Metrics · p. 33
Supplementary Figures
The supplementary figures pair irradiance or cloud-regime information with images and measurement-device setups from benchmark sites. They also show how variability and cloud regimes differ across sites.
Supplementary Figures
- Supplementary Figure 1 pairs representative measured GHI conditions with corresponding ground-based sky and geostationary satellite images.The red circle marks the sky imaging system’s location in each satellite scene.Source: Figure 1 · p. 34
- At Akwatia, the setup includes a GeoVision all-sky camera and an RT1 radiometer measuring GTI; Kologo uses the same setup.The RT1 radiometer is shown in an inset close-up.Source: Figure 2 · p. 35
- The Sankt Augustin setup combines a GeoVision all-sky camera with a SOLYS2 sun tracker carrying instruments for GHI, DNI, and DHI measurements.Enlarged views show the camera and solar-radiation measurement system.Source: Figure 3 · p. 35
- The Hong Kong setup uses a network camera for sky images, a pyranometer for GHI and DHI, and a weather station for surface meteorology.DNI is calculated from the pyranometer measurements.Source: Figure 4 · p. 36
- The Nottingham rooftop setup shows two measurement bases with a fisheye camera, sun tracker, irradiance instruments, and a weather station.The instruments measure DHI and DNI, derive GHI, and cross-validate GHI.Source: Figure 5 · p. 36
- Supplementary Figure 6 plots model rRMSE against the variability index for three lead times, with each point representing a SolarBench site.Its reported weak and inconsistent correlations indicate that the fixed-interval aggregate index alone does not adequately characterize forecasting difficulty.Source: Figure 6 · p. 37
- Across sites, the cloud-regime figure reports sample fractions for Clear, Thin, Patchy, and Thick conditions, while example sequences show their varied visual appearance.The examples show 15-minute sky-image histories at five-minute intervals for Puy de Dôme, Akwatia, Golden, and Nottingham.Source: Figure 7 · p. 37, Figure 8 · p. 38
Supplementary Tables
The supplementary tables catalogue site and dataset characteristics, processed atmospheric inputs, and model benchmark performance across forecast lead times. The performance tables identify best-performing values by site and metric.
Supplementary Tables
- Supplementary Table 1 presents site characteristics, while Supplementary Table 2 details the ground-based dataset for each site.Source: Table 1 · p. 39, Table 2 · p. 39
- Supplementary Table 3 describes the geostationary satellite imagery data for each site.Source: Table 3 · p. 40
- Supplementary Tables 4 and 5 list processed ERA5 and GFS variables, respectively.Source: Table 4 · p. 41, Table 5 · p. 41
- Supplementary Tables 6–10 report benchmark performance at 10-, 30-, 60-, 90-, and 120-minute lead times, highlighting best-performing values for each site and evaluation metric.Ramp recall uses a 30% relative error threshold.Source: Table 6 · p. 42, Table 7 · p. 43, Table 8 · p. 44