Vision Transformers for Estimating Irradiance Using Data Scarce Sky Images

1. Introduction

• DHI measurements are scarce because they require specialised, costly sun-tracking instruments, while conventional systems also impose maintenance, calibration, and logistical burdens.These constraints are especially relevant to remote, developing, or under-instrumented regions.Source: §1 · p. 1
• ViTs show promise for replacing costly measurement equipment, but their large parameter counts and self-attention create extensive data demands and overfitting risk.The cited literature reports that insufficient training data limits transformer generalisation.Source: §1 · pp. 1–2
• The proposed hybrid input combines fisheye sky images, overlay-based segmentation, extraterrestrial irradiance, and cyclic time encodings for data-constrained settings.The stated aim is to support prediction without extensive meteorological datasets or dense image sequences.Source: §1 · p. 2
• The two proposed frameworks either feed predicted synthetic GHI into a second model for DHI or process raw and overlay-enhanced images in parallel.The study validates the frameworks using CFARR data, with frequent cloud variation at the temperate site.Source: §1 · p. 2
• The contributions are a sequential synthetic-GHI architecture and a parallel architecture that processes raw and augmented sky images for data-limited irradiance estimation.The paper reports experimental validation across varying data availability.Source: §1 · p. 2

2. Data Scarcity Relevance

• The experiment uses one month of 5-minute measurements, approximately 8% of a year, and tests on an untouched operational year without exogenous weather forecasts.The authors contrast this setup with earlier data-scarce precedents using more samples or synthetic and smoothed test data.Source: §2 · p. 2
• A one-month training split of about 2,000 samples meets the paper’s data-scarce benchmark criterion.The criterion is a corpus spanning no more than 10% of an annual cycle or containing fewer than 10,000 effective samples.Source: §2 · pp. 2–3

3. Related Work

• Instrumental and physics-based approaches face cost, maintenance, input-data, or partly cloudy-sky accuracy constraints, while simpler clearness-index methods lose accuracy under non-standard skies.Handcrafted-feature regression reported MAE percentages close to 21.9%.Source: §3 · p. 3
• CNNs learn hierarchical sky-image features, and multimodal conditioning improved forecast error by 18% over image-only CNNs in one cited study.CNNs nevertheless have fixed receptive fields that limit wide-field contextual dependencies in hemispherical images.Source: §3 · p. 3
• ViTs use self-attention to capture long-range image dependencies; prior all-sky work reported DHI RMSE of approximately 31 W/m² without auxiliary meteorological features.A separate temporal transformer study outperformed smart persistence by over 20% in skill score for 15-minute-ahead forecasts.Source: §3 · p. 3
• Transformer-based multimodal work has combined sky-image optical flow with irradiance time series, while other approaches use satellite observations or multispectral bands.The sky-image framework cited achieved a normalised RMSE of 4.28% for 10-minute-ahead global irradiance forecasting.Source: §3 · p. 3
• Unlike a cited raw-image-only ViT, this study combines synthetic GHI, overlay segmentation, and physics-informed auxiliary features for sparse-data estimation.The paper identifies dense, long-term datasets or high-resolution imagery as assumptions that constrain existing transformer applicability in remote or data-scarce regions.Source: §3 · p. 4

4.1. Incorporating Cyclic Time Encodings and Extra-Terrestrial Irradiance

• Calculated extraterrestrial irradiance and cyclic time encodings provide temporal and atmospheric context to the transformer without relying on direct ground-based measurements.ETI is calculated from the solar constant, Earth–Sun distance correction, and solar zenith angle.Source: §4.1 · p. 4

4.2. Sequential Stream Framework Utilising Synthetic GHI

• The sequential pipeline predicts synthetic GHI from an image, then combines that prediction with the image to estimate DHI under variable conditions.The synthetic input is intended to provide context when irradiance data are sparse or missing.Source: §4.2 · p. 4, Figure 1 · p. 4

4. Proposed Framework

• The shared preprocessing masks non-sky pixels, resizes and normalises images, divides them into 16 × 16 patches, and prepends a CLS token and cyclic-time and ETI scalars.The DeiT encoder maps each 224 × 224 image to a 768-D class token for a regression head that outputs DHI in W/m².Source: §4 · p. 4, §4.2 · p. 5

4.2. Sequential Stream Framework Utilising Synthetic GHI

• GHI-Appended adds the scalar to the tabular input without new parameters, while GHI-MLP embeds it for the regression head but does not expose it to self-attention.The GHI-Appended variant’s 1-month RMSE is 43.3 versus 42.2 W/m² for the ETI + Time baseline.Source: §4.2 · pp. 5–6
• GHI-Token yields the lowest reported errors across data regimes: 33.8 W/m² at 1 month, 33.1 W/m² at 3 months, and 24.6 W/m² with full-year training.It projects synthetic GHI to an embedding inserted at the encoder input, allowing self-attention layers to condition on irradiance context.Source: §4.2 · pp. 5–6

4.3.1. Overlay Generation Using Hybrid Thresholding and Super Pixel Segmentation

• When the sun is obscured, fixed nBRR thresholding generates the mask; when visible, SLIC super-pixels and local Otsu thresholds refine it.Solar position is mapped onto the fisheye image to determine whether the sun is visible or obscured.Source: §4.3.1 · p. 6

4.3.2. Threshold-Based Cloud Mask Generation

• R=0.78 for the greyscale mask’s CR–K% correlation, versus R=0.68 for a fixed nBRR threshold; the selected greyscale threshold was 0.3946.The threshold was chosen to align cloud masks with diffuse fraction and cloud ratio, rather than hand-labelled data.Source: §4.3.2 · p. 6

4.3.3. Super Pixel Segmentation and Adaptive Thresholding

• 100 SLIC regions balance spatial detail and runtime, while each region’s threshold averages its local Otsu threshold with the global image threshold.This refinement is intended to handle local brightness variations and partial occlusion.Source: §4.3.3 · p. 7

4.3.4. Overlay Generation and Cloud Ratio Calculation

• A semi-transparent colour layer overlays the binary cloud mask on the original image, making detected cloud regions visible.The overlay is presented as a visualisation and auxiliary input for irradiance estimation.Source: §4.3.4 · p. 7, Figure 2 · p. 7
• 6.8 W/m² lower DHI RMSE than the Normal-DeiT ablation; the heuristic cloud mask serves as an auxiliary signal, not an independent optimisation target.The approach is described as classifier-free and adapted for unlabelled datasets.Source: §4.3.4 · p. 7

• Two DeiTs process complementary inputs in parallel: one receives the overlay-processed image and the other the raw sky image.The two representations are provided to separate branches.Source: §4.4 · p. 8, Figure 3 · p. 8
• Each encoder’s embedding is concatenated with ETI and cyclical time encodings, then passed to a two-layer MLP regression head for DHI estimation.Source: §4.4 · p. 8, Figure 3 · p. 8
• Capacity-matched concatenation and single-stream variants were used to test whether gains came from dual-stream design rather than increased parameter count.The passage reports that these ablations confirm gains arise from architectural design rather than model scale.Source: §4.4 · p. 8

• CFARR images were matched to pyranometer GHI and DHI readings by timestamp, with missing-DHI intervals omitted and no imputation.Images were captured every five minutes; night frames and negative irradiance values were excluded.Source: §5 · p. 8
• Folsom provides three years of one-minute all-sky imagery paired with ground-truth GHI and DHI, and the same preprocessing was applied for distinct-site evaluation.The dataset also includes concurrent weather, satellite, and NWP variables.Source: §5 · p. 8
• Training subsets span one to twelve months, beginning in May 2016 and extending through April 2017.Source: §5 · p. 8
• Each subset uses a random 80/20 training-validation split and is tested on August 2017–August 2018, with no temporal overlap.The full-year test window is intended to support seasonally representative, cross-period comparisons.Source: §5 · p. 9

• Among three ways to integrate predicted GHI into DHI estimation, early-stage token integration achieved the highest accuracy.The alternatives appended GHI as a tabular feature or passed it through a small MLP; token integration placed it within the transformer encoder.Source: §6 · p. 9
• The parallel setup fuses embeddings from overlay-image and raw-image DeiTs with ETI and cyclic time encodings for DHI prediction.An Image-Only DeiT and an ETI + Time baseline precede these extensions.Source: §6 · p. 9
• All models trained for 10 epochs with batch size 16; a 50-trial Optuna search selected within-family hyperparameters that were then held constant.Experiments used an NVIDIA Quadro RTX 6000 GPU and Intel Core i7-7700K CPU.Source: §6 · p. 10

7.1. Performance

• The frameworks’ predictive advantages were greatest with severely limited data and decreased as data availability increased.Across five random seeds, average RMSE standard deviations were within ±1.76 W/m² for baseline, ±2.29 W/m² for GHI-Token, and ±1.45 W/m² for parallel dual.Source: §7.1 · p. 10, Figure 4 · p. 10, Figure 5 · p. 10

7.2. Model Benefits in Limited Dataset Scenarios

• With about 1/12 of the data, the sequential model reduced RMSE by approximately 20% versus the single-stage baseline, and the dual framework excelled in sparse-data conditions.The comparison concerns the sequential model’s two-stage approach and a single-stage baseline.Source: §7.2 · p. 11, Table 4 · p. 11, Table 5 · p. 11
• The dual-parallel model’s attention maps sometimes highlighted regions under-emphasized without overlays, while retaining baseline-like focus and adding complementary contextual cues.Figure 6 presents attention maps and input images for the baseline, GHI-token, and dual-parallel variants.Source: §7.2 · p. 12, Figure 6 · p. 12
• With one-month training and a disjoint full-year test, the sequential framework reached RMSE 33.8 and MAE 22.4, versus Mercier et al.’s RMSE 31 and MAE 19 using more data.Mercier et al. trained and tested within the same full dataset, while this study trained on one month and tested on a separate year.Source: Table 6 · p. 12, §7.2 · pp. 12–13

7.3. Cross-Season and Cross-Region Generalisation

• Seasonal testing favored GHI-Token in spring, while Dual-Stream had the lowest RMSE in summer, autumn, and winter; both frameworks also improved results at Folsom.GHI-Token reduced spring Baseline RMSE by 23%; Dual-Stream RMSE gains were 12–21% across summer, autumn, and winter. At Folsom, MAE fell ≈27% and RMSE ≈9% relative to baseline.Source: Table 7 · p. 13, Table 8 · p. 13, §7.3 · pp. 13–14

8. Discussion

• With only 1/12 of the data, the hybrid framework maintained RMSE within 10% of full-data baselines.The discussion attributes the approach to synthetic augmentation and reports results against prior CNN- and transformer-based models trained on complete datasets.Source: §8 · p. 14
• GHI-Token reduced RMSE most in clear conditions, while dual-stream gains were greatest in intermediate and clear regimes without requiring cloud labels.The test set was divided into overcast (CI < 0.3), intermediate (0.3 ≤ CI ≤ 0.78), and clear (CI > 0.78) conditions.Source: §8 · p. 14, Figure 7 · p. 14
• Transfer to other locations or imaging systems may require fine-tuning and minor segmentation-threshold adjustments for camera hardware or atmospheric differences.The stated limitation is based on results from CFARR; spatial-domain extension and temporal forecasting remain future research directions.Source: §8 · p. 14

9. Conclusion

• The sequential model feeds synthetic GHI back into the network, while the dual-stream model fuses raw and overlay-enhanced fisheye images using ETI, cyclic time, and lightweight segmentation.The conclusion characterizes these architectures as combining deterministic physical inputs with lightweight cloud cues.Source: §9 · pp. 14–15
• Under one-month training, GHI-Token reduced RMSE from 42.2 W/m² to 33.8 W/m², while dual-stream overlay reduced it to 36.9 W/m².GHI-Token improved R² from 0.77 to 0.84; dual-stream delivered the largest gains in partly cloudy conditions.Source: §9 · p. 15
• Using 1/12 of the full dataset, both frameworks achieved RMSE values within 2–6 W/m² of a state-of-the-art ViT, indicating reliable DHI extraction after only weeks of data.Source: §9 · p. 15

CRediT Authorship Contribution Statement

• Hamlyn handled original drafting, visualization, software, investigation, and conceptualization; Chaudhary contributed review, software, supervision, and conceptualization, while Rahman contributed review, supervision, and conceptualization.Source: CRediT Authorship Contribution Statement · p. 15