Source-linked AI summary
Spatio-Temporal Wind Speed Forecasting using Graph Networks and Novel Transformer Architectures
Lars Ødegaard Bentsen, Narada Dilp Warakagoda, Roy Stenbro, Paal Engelstad
TL;DR
The paper studies multi-step spatio-temporal wind forecasting to produce informative local wind trajectories from measurements across the Norwegian continental shelf. It combines GNN-based spatial modelling with Transformer-derived temporal update functions and proposes the FFTransformer. Altered Transformer architectures outperform the spatio-temporal LSTM and MLP baselines, with FFTransformer strongest for the longest forecast horizon.
Problem
Short-term wind intermittency complicates forecasting, while existing wind studies provide limited higher-resolution multi-step trajectories and limited spatio-temporal formulations of altered Transformers.
Method
The study uses GNNs to extract spatial dependencies and neural update functions to model temporal correlations, comparing Transformer variants and introducing the two-stream FFTransformer.
Results
FFTransformer is on par with Autoformer for 1- and 6-step forecasts and significantly outperforms all other models for the longer 24-step forecasts.
Takeaways & Limitations
FFTransformer and Autoformer achieve superior results for 10-minute and 1-hour-ahead forecasts, while FFTransformer is strongest for 4-hour-ahead forecasting.
Takeaways & Limitations
The ST-FFTransformer is slower because its MDWD layer is not parallelised across input features, and FFT computation also increases forecast time.
Abstract
from arXiv · showhide
This study focuses on multi-step spatio-temporal wind speed forecasting for the Norwegian continental shelf. The study aims to leverage spatial dependencies through the relative physical location of different measurement stations to improve local wind forecasts. Our multi-step forecasting models produce either 10-minute, 1- or 4-hour forecasts, with 10-minute resolution, meaning that the models produce more informative time series for predicted future trends. A graph neural network (GNN) architecture was used to extract spatial dependencies, with different update functions to learn temporal correlations. These update functions were implemented using different neural network architectures. One such architecture, the Transformer, has become increasingly popular for sequence modelling in recent years. Various alterations have been proposed to better facilitate time series forecasting, of which this study focused on the Informer, LogSparse Transformer and Autoformer. This is the first time the LogSparse Transformer and Autoformer have been applied to wind forecasting and the first time any of these or the Informer have been formulated in a spatio-temporal setting for wind forecasting. By comparing against spatio-temporal Long Short-Term Memory (LSTM) and Multi-Layer Perceptron (MLP) models, the study showed that the models using the altered Transformer architectures as update functions in GNNs were able to outperform these. Furthermore, we propose the Fast Fourier Transformer (FFTransformer), which is a novel Transformer architecture based on signal decomposition and consists of two separate streams that analyse the trend and periodic components separately. The FFTransformer and Autoformer were found to achieve superior results for the 10-minute and 1-hour ahead forecasts, with the FFTransformer significantly outperforming all other models for the 4-hour ahead forecasts.
1 Introduction
The paper addresses the challenges of intermittent wind through multi-step, spatio-temporal forecasting, using GNNs and Transformer-based temporal update functions. It compares established architectures and proposes the FFTransformer for separately modelling trend and periodic components.
- Wind’s short-term intermittency creates challenges for operators and grid planning, motivating accurate forecasting.
- Multi-step forecasts at 10-minute resolution describe within-horizon wind development and support more precise wind-power estimation because P ∝ws^3.
- The framework uses GNNs to capture spatial correlations and optional neural update functions, including Transformers and LSTMs, to learn temporal dependencies.
- The study compares vanilla, LogSparse, Informer and Autoformer Transformer architectures for wind forecasting, including their formulation in a GNN setting.
- The proposed FFTransformer uses signal decomposition and two streams to analyse periodic frequency-domain components and trend components separately.
2 Related Works
Related work spans statistical, machine-learning, recurrent, convolutional and Transformer methods for wind forecasting. The paper builds on this literature by combining graph-based spatial modelling with Transformer-based temporal analysis and higher-resolution multi-step forecasts.
- ARIMA-based methods are robust and easy to implement, while decomposition techniques have shown potential for wind time-series forecasting.
- SVR can perform well but does not scale efficiently to larger datasets, whereas KNN can be fast during training and testing.
- MLPs, RNNs, CNNs and LSTMs are established neural architectures for modelling nonlinear relationships and sequences in wind forecasting.
- WaveNet uses dilated causal convolution to preserve sequence ordering while expanding the receptive field for time-series modelling.
- Transformers model long-term dependencies without recurrence or convolution, but their quadratic sequence-length complexity motivated architectural alternatives.
- Spatio-temporal forecasting uses physical distances among stations to jointly model non-stationary wind fields across space and time.
- The paper extends GNN-based spatio-temporal forecasting with recent Transformer architectures, the FFTransformer, and higher-resolution multi-step prediction.
3.1 Multi-Layer Perceptron
The MLP maps inputs to outputs through stacked weighted layers, with nonlinear activations and backpropagation used to improve learning and determine the weights.
- MLPs learn weights θ that map an input x to an output y through a feed-forward function.
- Stacking multiple layers gives the MLP depth, while ReLU or sigmoid nonlinearities are applied to neuron outputs.
- Backpropagation minimises a differentiable loss function to determine the model’s optimal weights.
3.2 LSTM
The LSTM modifies recurrent sequence modelling with gating mechanisms and skip connections to reduce information loss associated with exploding or vanishing gradients.
- Long sequences can cause exploding or vanishing gradients in standard recurrent neural networks, producing information loss.
- LSTM units introduce gating mechanisms and skip connections to alleviate these recurrent-network limitations.
3.3 Transformer Architectures
The Transformer uses attention in an encoder-decoder forecasting architecture, while several variants modify attention or the overall architecture to address time-series requirements and computational cost.
- Transformer architecture: The Transformer maps a multivariate look-back window to a univariate forecasting horizon using encoder-decoder sequence modelling.The encoder produces a hidden representation, while the decoder generates predictions for future positions.
- Transformer architecture: Inputs are projected into a hidden space and combined with positional encoding so sequence order is available without recurrence.The encoder uses multi-head self-attention, followed by residual connections, normalization, and position-wise MLP processing.
- Transformer architecture: The decoder adds cross-attention to encoder outputs and masks future positions to prevent subsequent information from flowing backward.Cross-attention uses encoder outputs as keys and values.
- Transformer limitations: Attention complexity scales quadratically with sequence length, motivating Transformer modifications for computational efficiency and time-series forecasting.The subsequent variants alter attention patterns or other architectural components to address these limitations.
- LogSparse Transformer: LogSparse attention restricts each position to itself and exponentially spaced positions, while causal 1D convolution supplies locally contextual keys and queries.These changes reduce space complexity and may better represent local high-frequency fluctuations or noise.
- Informer: Informer introduces ProbSparse attention by selecting dominant queries, rather than imposing a fixed heuristic sparsity pattern.The mechanism identifies queries using a surrogate for the KL-divergence between uniform and attention distributions.
- Autoformer: Autoformer replaces scaled dot-product attention with AutoCorrelation, selecting important time-delay similarities and aggregating time-shifted series.It also makes substantial changes to the overall Transformer architecture for time-series forecasting.
3.4 Graph Neural Networks
The GNN represents measurement stations as nodes and relationships such as inter-station distance as edges, updating and aggregating features through stacked graph blocks.
- Graph representation: A spatial forecasting graph contains node features for measurement-station attributes and edge features describing relationships between stations.Euclidean distance is an example of an edge feature.
- Graph updates: Stacked graph blocks perform per-edge and per-node updates to propagate information through the graph.The architecture applies updates before aggregating features across connected nodes.
- Graph updates: Update functions can use LSTM, MLP, or Transformer neural networks to learn temporal dependencies within the graph framework.Updated and aggregated features are distinguished during message processing.
- Feature aggregation: Edge features are aggregated to nodes using operations such as summation or averaging over the graph’s incoming-neighbour index set.The index set specifies which nodes send information to each receiving node.
3.5 Multilevel Wavelet Decomposition
Multilevel discrete wavelet decomposition separates wind-speed signals into trend and periodic components with different frequency characteristics.
- Single-level decomposition: Discrete wavelet decomposition separates a time series into approximate trend and detail periodic components using low- and high-pass filters.The input is convolved with a low-pass filter and a high-pass filter.
- Multilevel decomposition: Multilevel decomposition feeds each approximate component into the next layer, producing multiple detail signals and one final approximation.The detail components represent different frequency levels, while the final approximation retains the broad trend.
- Wind-speed decomposition: The Db4 Daubechies wavelet is used for the low- and high-pass coefficients because it has been shown suitable for wind forecasting.Inverse multilevel decomposition is applied independently to the decomposed outputs.
- Wind-speed decomposition: FFT analysis shows that the detail components have distinct periodic peaks at different frequencies, whereas the approximation captures trend information.Figure 5 visualizes both the component frequency characteristics and the approximation’s relationship to the input signal.
4 Methodology
The study builds a spatio-temporal GNN framework for multi-step offshore wind forecasting, using station graphs to model spatial dependencies and neural update functions to learn temporal patterns. The FFTransformer separately processes trend and periodic components, while forecasts are evaluated across 10-minute, 1-hour, and 4-hour horizons.
- Data and graph construction: Measurements from 14 Norwegian continental-shelf stations were represented as graph nodes with eight meteorological variables and latitude-longitude differences as edge features.Data covered June 23, 2015 to February 28, 2022; missing isolated time steps were linearly interpolated, while periods with consecutive gaps were excluded.
- Fast Fourier Transformer: The FFTransformer decomposed each input signal into periodic and trend components using a four-level MDWD and processed them in separate streams.Its periodic stream used FFT-Attention in the frequency domain, while the trend stream used Transformer-style processing; the streams were then added and linearly transformed into predictions.
- Spatio-temporal framework: The two-layer GNNs used encoder-only neural architectures as node update functions to combine spatial message passing with temporal sequence modelling.Compared update functions included Transformer variants, LSTM, and MLP architectures; predicted outputs were taken from the final P time steps.
- Input construction: Future inputs used persistence or mean placeholders for several models, while Autoformer seasonal inputs and FFTransformer detail components received zero placeholders.For FFTransformer inputs, placeholders were added after MDWD and before embedding and encoding.
- Experimental set-up: The experiments predicted every 10-minute interval over 1-, 6-, and 24-step horizons, corresponding to 10-minute, 1-hour, and 4-hour forecasts, using MSE training and Optuna tuning.Longer forecast horizons used longer historical contexts, with the context increased to 64 steps for 4-hour forecasts.
- Computational considerations: ST-FFTransformer was slower than the other models because MDWD was not parallelised across input features and both ST-Autoformer and ST-FFTransformer computed FFTs.Forecast computation remained below one second and below the 10-minute sampling interval in this study.
5 Results and Discussion
Across forecast horizons, altered Transformer architectures within GNNs generally improved multi-step wind forecasting, with ST-Autoformer and ST-FFTransformer strongest at shorter horizons and ST-FFTransformer superior at four hours. Results also indicate that spatial connectivity improves forecasts, especially for longer horizons, while station predictability varies geographically.
- Forecasting Error: MAE and MSE were evaluated across forecast horizons, with predictions and labels transformed back to meters-per-second units.Each model was trained five times, and percentage improvements were calculated relative to the persistence model.
- Forecasting Error: ST-LogSparse and ST-Informer consistently outperformed ST-Transformer across horizons in both MSE and MAE.Their attention mechanisms showed potential benefits for wind forecasting, although their single-step MAE was slightly worse than persistence.
- Forecasting Error: Transformer-based models generally outperformed ST-MLP and ST-LSTM for multi-step forecasts, while ST-Autoformer excelled at 1- and 6-step horizons but degraded at 24 steps.ST-Autoformer achieved more than three times the third-best model’s MSE improvement at 1 step and nearly doubled ST-MLP’s improvement at 6 steps.
- Forecasting Error: ST-FFTransformer achieved superior results for the 4-hour forecasts and continued to perform well across horizons with limited training variability.ST-Autoformer showed considerable variability at 1 step, whereas ST-FFTransformer remained consistently competitive.
- Forecasting Error: ST-FFTransformer reduced Interval MAE by 300 kWh for 4-hour forecasts, a 19% improvement over persistence and approximately 5% over most other models.Interval MAE estimates differences between predicted and true total energy over the forecast interval.
- Graph Connectivity: Test MAEs decreased sharply as graph connectivity increased before converging when more than around five neighbours were included.The larger percentage reductions at 6- and 24-step horizons suggest that long-range spatial connections benefit longer forecasts more than immediate forecasts.
6 Conclusion
The study finds that Transformer variants, particularly FFTransformer and Autoformer, strengthen spatio-temporal wind forecasting, while FFTransformer performs best for longer horizons. Further variations of its decomposition and attention mechanisms remain open for investigation.
- Transformer-based models were evaluated as predictors for spatio-temporal multi-step wind forecasting, including LogSparse Transformer, Informer, and Autoformer.
- FFTransformer achieved results on par with Autoformer for 1- and 6-step forecasts and significantly outperformed other models for 24-step forecasts.
- Convolutional attention in LogSparse Transformer and ProbSparse Attention in Informer slightly improved prediction performance, whereas vanilla Transformer generally did not significantly improve over MLP.
- FFTransformer showed an additional 5 % improvement over other models for 4-hour forecasting when associated prediction errors were estimated in kW and kWh.
- Alternative signal decomposition techniques and attention mechanisms could be relevant for other applications and warrant further research.
CRediT authorship contribution statement
The contribution statement assigns project administration, conceptualization, data, analysis, methodology, software, validation, visualization, and drafting to Lars Ø. Bentsen, with supervision and review contributions from the other authors.
- Lars Ø. Bentsen led project administration, conceptualization, data curation, analysis, methodology, software, validation, visualization, and original drafting.
- Narada Dilp Warakagoda, Roy Stenbro, and Paal Engelstad contributed supervision, with Warakagoda and Engelstad also contributing writing review and editing.