Source-linked AI summary
Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models
Ken Chen, Maneesha Perera, Wei Wang, Sachith Seneviratne, Hansani Weeratunge, Saman Halgamuge
TL;DR
Short-term solar forecasting needs visual information to anticipate cloud-driven changes, while site-specific data are scarce and existing LLM forecasters are mainly unimodal. SolCloudLLM bidirectionally fuses aligned sky-image and time-series patches before LLM mapping, outperforming baselines across horizons and remaining robust in few-shot settings.
Problem
Short-term solar forecasting must anticipate cloud-driven variability despite scarce site-specific data and predominantly unimodal LLM-based forecasting methods.
Method
SolCloudLLM aligns sky-image patches with time-series patches and applies token-level bidirectional multimodal fusion before mapping the unified representation into a frozen LLM.
Results
SolCloudLLM consistently outperforms the best baseline across H ∈{16, 32, 64}, achieving peak relative MSE reductions of 25.4% on SIRTA and 14.5% on SKIPP’D.
Takeaways & Limitations
Multimodal gains are strongest under cloudy conditions, and SolCloudLLM remains robust with only 5% to 10% of available training data.
Takeaways & Limitations
Ground-based sky cameras provide only localized and short-term cues about cloud dynamics.
Abstract
from arXiv · showhide
Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations. At these forecasting horizons, errors are predominantly driven by cloud induced ramps: relying solely on historical numerical data may struggle to anticipate an incoming cloud, making ground-based sky images a crucial complementary physical signal. Furthermore, forecast performance is highly sensitive to location and local observing conditions, creating a strong need for site-specific data that are often scarce. Recently, large language models (LLMs) have demonstrated competitive performance and high data efficiency in time-series forecasting. Despite their success, existing LLM-based forecasting methods remain predominantly unimodal, relying primarily on historical numerical time-series data. Effectively incorporating sky imagery into an LLM-based forecasting framework remains under-explored and an open challenge. In this paper, we propose SolCloudLLM, an LLM-based multimodal forecasting framework. SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM. Extensive experiments on the SIRTA and SKIPP'D datasets demonstrate that SolCloudLLM consistently outperforms the best baseline methods in MSE across all forecasting horizons, achieving a maximum relative MSE reduction of 25.4%. Stratified analysis further indicates that the benefits of multimodal fusion are concentrated primarily under cloudy conditions. Notably, SolCloudLLM achieves the best performance in nearly all few-shot settings, whereas other deep learning baselines experience substantial performance degradation and are frequently outperformed by the non-learning physical method.
1. Introduction
Short-term solar forecasting must anticipate cloud-driven variability despite site-specific domain shifts and scarce paired data. SolCloudLLM addresses this gap by bidirectionally fusing aligned sky-image and time-series representations before mapping them into an LLM.
- 1. Introduction: Short-term PV and GHI forecasting is crucial for dispatch, reserve scheduling, and stable market operations, but moving clouds cause rapid errors that historical data cannot anticipate.Purely time-series methods typically react only after an output drop has occurred.
- 1. Introduction: Site-specific camera, microclimate, and PV-system differences create domain shifts, while paired multimodal data remain costly and scarce.These conditions make data-efficient forecasting operationally necessary.
- 1. Introduction: The framework is motivated by LLM-based time-series forecasting, which transfers pretrained sequential priors through cross-modality alignment instead of training from scratch.The introduction positions multimodal sky-image integration as an under-explored extension of this paradigm.
- 1. Introduction: SolCloudLLM aligns sky-image and time-series patches, then uses token-level bidirectional fusion before mapping the unified representation into a frozen LLM backbone.Each modality generates scale-and-shift parameters for the other, enabling cross-modal interaction rather than plain concatenation.
- 1. Introduction: SolCloudLLM consistently outperforms baseline methods in MSE across H ∈{16, 32, 64}, with peak relative reductions of 25.4% on SIRTA and 14.5% on SKIPP’D.The evaluations use the SIRTA and SKIPP’D solar datasets.
2. Related work
Related work spans numerical, vision-based, and multimodal solar forecasting, alongside LLM and foundation-model approaches for time series. SolCloudLLM differs from recent multimodal foundation-model work by integrating visual and numerical streams before language-space alignment.
- 2. Related work: LLM-based forecasting methods transfer pretrained transformer attention to time series through alignment strategies such as text prototypes, decomposition prompts, and semantic anchors.These approaches generally require minimal task-specific adaptation.
- 2. Related work: Many multimodal LLM forecasters treat auxiliary information as text, while time-series foundation models target zero-shot forecasting.GPT4MTS is cited as an example of textual context used alongside numerical patches.
- 2. Related work: Existing solar forecasters combine numerical histories, sky imagery, or satellite observations, but their architectures vary from unimodal CNNs and temporal models to specialized multimodal fusion.Examples include CNN-LSTM, cloud-motion, concatenation, and broader multi-view systems.
- 2. Related work: Purely numerical histories cannot encode visual cloud kinematics, motivating ground-based cameras and satellites as complementary sources for solar forecasting.Public multimodal datasets now support cross-site evaluation of these visual pipelines.
- 2. Related work: PV-VLM encodes sky images and temporal data independently before late cross-modal attention, whereas SolCloudLLM uses a lightweight CNN and aligns the streams earlier.The contrast identifies the proposed framework’s architectural novelty within foundation-model approaches.
3. Method
SolCloudLLM forecasts PV power or GHI from synchronized historical time-series and sky images by aligning and fusing their patch representations before reprogramming them into a frozen LLM.
- Problem formulation: The forecasting objective uses a historical window and synchronized images to predict a future PV or GHI trajectory over horizons H ∈ {16, 32, 64}.The main formulation adopts T=H and standardizes targets using a scaler fitted only on the training split.
- LLM reprogramming: The fused representation is mapped into the frozen LLM language space, processed with prompt metadata, and projected from hidden states into the multi-step forecast.The LLM backbone remains frozen, while trainable components adapt the multimodal inputs and output head.
- Multimodal architecture: SolCloudLLM encodes synchronized solar time-series and sky-image sequences into temporally aligned patch embeddings, then mutually modulates the token streams before LLM reprogramming.The visual features are aggregated to match the time-series patch count and projected to the same token dimension.
- Bidirectional multimodal fusion: Bidirectional fusion lets sky-image tokens modulate time-series patches and time-series tokens modulate sky-image features through affine conditioning before concatenation and projection.The module uses parameter networks for both directions, nonlinear processing, scaled residual connections, and a projection back to dimension d.
- Training objective: Training minimizes mean squared error over the forecast horizon, using Adam for up to 10 epochs with early stopping and the best validation checkpoint for testing.Testing metrics are evaluated under the stated validation-checkpoint protocol.
4. Experiments
SolCloudLLM is evaluated on two synchronized solar datasets under full-data, regime-specific, few-shot, and fusion-ablation settings. It consistently improves MSE over baselines, with larger advantages under cloudy conditions and limited data.
- Main results: 25.4% relative MSE reduction is achieved against Time-LLM on SIRTA at one horizon, while SolCloudLLM outperforms all baselines across every horizon.The SIRTA reductions versus Time-LLM are 20.7%, 25.4%, and 24.8% across the three horizons.
- Main results: 14.5% MSE improvement over Time-LLM is achieved on SKIPP’D at H=64, with SolCloudLLM obtaining the lowest MSE at all prediction intervals.Improvements over Time-LLM are 7.0% at H=16, 12.0% at H=32, and 14.5% at H=64.
- Sunny/cloudy regime analysis: Cloudy-day MSE gains over Time-LLM increase from 3.5% at H=16 to 13.8% at H=64, while sunny-day errors remain low and SPM marginally wins at H=32.Cloudy conditions dominate overall MSE because transient cloud dynamics produce high residual variance.
- Few-shot data efficiency: SolCloudLLM retains the best performance in most 10% and 5% few-shot configurations, whereas conventional deep-learning baselines often underperform the physics-based SPM.The sole reported exception is SKIPP’D at H=16 with 5% data, where SPM scores 0.1476 versus SolCloudLLM’s 0.1559.
- Fusion ablation: Bidirectional fusion outperforms concatenation at all SIRTA horizons and at SKIPP’D H=16 and H=32, but concatenation is marginally better at SKIPP’D H=64.Table 6 compares the full fusion architecture with a baseline that replaces only the fusion operator.
- Image encoder ablation: The lightweight CNN is selected over ViT-small because it performs better in most scenarios and is expected to benefit few-shot data efficiency.The reported conclusion supports using CNN features in the primary experiments.
5. Conclusion
SolCloudLLM combines sky-image features with time-series forecasting through bidirectional multimodal fusion before processing by a frozen LLM. It outperforms baselines across tested horizons, remains effective in data-scarce cloudy settings, and is limited by the localized, short-term cues of ground-based cameras.
- 5. Conclusion: SolCloudLLM uses lightweight bidirectional multimodal fusion to mutually modulate visual and numerical tokens before they enter the frozen LLM backbone.The fusion uses token-wise affine modulation to integrate contemporaneous sky images with numerical history.
- 5. Conclusion: 25.4% and 14.5% peak relative MSE reductions are achieved on SIRTA and SKIPP’D, respectively, while outperforming the best baseline across all tested horizons.The reported peaks occur at SIRTA H=32 and SKIPP’D H=64.
- 5. Conclusion: In 5%–10% few-shot settings, SolCloudLLM remains resilient when conventional deep-learning baselines often underperform the physics-based SPM.The framework is also reported to perform strongly at longer horizons under cloudy conditions.
- 5. Conclusion: Ground-based sky cameras provide only localized and short-term cloud-dynamics cues, motivating future integration with satellite imagery and text-based weather forecasts.The proposed future direction targets broader coverage across short and extended prediction horizons.