Source-linked AI summary
Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark
Jesus Lago, Grzegorz Marcjasz, Bart De Schutter, Rafał Weron
TL;DR
Electricity price forecasting lacks rigorous, reproducible evaluation because studies often rely on inconsistent datasets, short tests, weak baselines, and limited statistical testing. The paper surveys existing methods, compares statistical and deep-learning benchmarks across five markets, and releases datasets, forecasts, and a Python toolbox with best-practice guidelines. Deep-learning ensembles perform best overall, while LEAR remains nearly as accurate at up to 100 times lower computational cost.
Problem
Electricity price forecasting lacks rigorous evaluation because studies often use inconsistent datasets, short single-market tests, weak baselines, inadequate metrics, and limited significance testing.
Method
The paper surveys the literature, compares LEAR and DNN-based models across five markets, and provides open datasets, forecasts, a Python toolbox, and best-practice guidelines.
Results
Deep-learning models are more likely to outperform statistical models, with DNN ensembles statistically significantly better than the other models.
Takeaways & Limitations
LEAR remains the practical choice for decisions required within seconds because its performance is close to DNNs while computational costs are up to 100 times lower.
Takeaways & Limitations
The best hybrid model and the best individual hybrid components cannot be established because studies rarely compare hybrids with each other or with LEAR and DNN.
Abstract
from arXiv · showhide
While the field of electricity price forecasting has benefited from plenty of contributions in the last two decades, it arguably lacks a rigorous approach to evaluating new predictive algorithms. The latter are often compared using unique, not publicly available datasets and across too short and limited to one market test samples. The proposed new methods are rarely benchmarked against well established and well performing simpler models, the accuracy metrics are sometimes inadequate and testing the significance of differences in predictive performance is seldom conducted. Consequently, it is not clear which methods perform well nor what are the best practices when forecasting electricity prices. In this paper, we tackle these issues by performing a literature survey of state-of-the-art models, comparing state-of-the-art statistical and deep learning methods across multiple years and markets, and by putting forward a set of best practices. In addition, we make available the considered datasets, forecasts of the state-of-the-art models, and a specifically designed python toolbox, so that new algorithms can be rigorously evaluated in future studies.
1. Introduction
Electricity price forecasting research is difficult to evaluate because studies use inconsistent data, implementations, metrics, and statistical procedures. The paper responds with open-source benchmark models and datasets, plus best practices for fairer, more reproducible comparisons.
- Motivation: Comparisons between statistical and machine-learning methods produce contradictory conclusions because studies often use asymmetric baselines.Advanced statistical methods are commonly compared with simple machine-learning methods, while new machine-learning methods are often compared with simple statistical methods.
- Motivation: Testing periods are often too short, sometimes only one week, so they do not represent performance across special days and full years.The paper notes that meaningful conclusions require test datasets spanning at least a year.
- Motivation: Incomplete reporting of data splits, model inputs, and datasets prevents other researchers from validating results.These are identified as three common reproducibility problems in existing studies.
- Motivation: Deep-learning and hybrid studies frequently use limited or outdated comparisons, leaving relative model performance unclear.New deep-learning methods are often compared only with simpler machine-learning methods, while hybrid studies may omit established methods or use outdated methodologies.
- Contributions: The authors provide five six-year open-access benchmark datasets across established day-ahead markets, model forecasts, and guidelines for sounder, reproducible studies.The datasets contain hourly prices and two relevant exogenous variables for each market.
- Contributions: The paper selects LEAR and DNN as state-of-the-art statistical and deep-learning benchmarks and releases them through a documented Python library.The library is designed as a common research framework for electricity price forecasting.
2. Literature review
The review situates day-ahead electricity price forecasting within market-specific settings and surveys statistical, machine-learning, and hybrid approaches. It finds that inconsistent comparisons, short or narrow evaluations, and underreported experimental details make state-of-the-art performance difficult to establish.
- Day-ahead markets set 24 hourly delivery prices simultaneously from bids submitted before gate closure on the previous day.
- The review covers statistical, machine-learning, and hybrid point-forecasting methods for day-ahead markets since the last comprehensive review.
- Statistical methods: Regularized regression, variance-stabilizing transformations, long-term seasonal components, and mixed short- and long-window ensembles are prominent statistical developments.Ensembles combining calibration windows of 1–4 months with windows of approximately two years significantly outperformed the best ex-post selected window in cited studies.
- State-of-the-art models: LEAR is argued to be a highly accurate linear model, while there is no sound evidence that LSTMs are more accurate and hybrid models cannot be ranked reliably.The paper therefore selects LEAR and DNN as relatively simple, highly accurate benchmarks for evaluating new complex forecasting methods.
3. Open-access benchmark dataset
The paper introduces an open-access benchmark spanning five day-ahead electricity markets, each with six years of data and market-specific price dynamics. It uses long out-of-sample periods and includes relevant exogenous forecasts to support cross-market evaluation.
- Benchmark scope: The benchmark provides five open-access day-ahead electricity-market datasets, each spanning six years.The datasets represent markets with distinct frequencies of negative prices, zero prices, and price spikes.
- Data contents: Each dataset includes hourly day-ahead prices and two additional time series representing influential exogenous forecasts.The exogenous variables differ by market and are selected for their relevance to price forecasting.
- Access: The benchmark data can be freely accessed through the accompanying Python library.The authors note that they do not own the underlying data and provide an interface to access it from public sources.
- Market coverage: The benchmark covers Nord Pool, PJM, EPEX-BE, EPEX-FR, and EPEX-DE, whose price series exhibit different market dynamics.EPEX-BE and EPEX-FR are highly coupled, while EPEX-FR commonly contains negative prices, zero prices, and spikes.
- Training and testing periods: Testing uses the last 104 weeks, or two years, of each dataset, with exact testing dates reported in Table 1.The remaining observations are available for model estimation, with calibration windows and recalibration schedules treated as modeling choices.
4. Open-access benchmark models
The benchmark models combine a parameter-rich linear LEAR model and a deep neural network with automated feature selection and tuning. The LEAR implementation also introduces a computationally efficient hybrid procedure for selecting its LASSO penalty.
- Open-source models: The paper releases documented, automated implementations of the LEAR and DNN forecasting methods in an open-source Python toolbox.The methods are selected because they are both relatively simple and highly accurate, enabling researchers to evaluate new models without expert knowledge.
- Input features: Both models forecast 24 day-ahead hourly prices using historical prices, exogenous forecasts, their lags, and day-of-week information.The available feature set is shared across models, although each model applies a different feature-selection procedure.
- Feature selection: The benchmark supplies 247 available features for LEAR and 241 for DNN, with automated feature selection in both models.LEAR uses LASSO as embedded selection, whereas DNN features and hyperparameters are optimized using a tree Parzen estimator.
- DNN model: The DNN is a four-layer feedforward network with 24 outputs, trained using Adam and tuned through Bayesian optimization.The tree Parzen estimator jointly optimizes its hyperparameters and input features.
- LEAR model: LEAR is a parameter-rich ARX model estimated with LASSO after an asinh variance-stabilizing transformation.Its implementation uses daily recalibration across calibration windows of 8, 12, 156, and 208 weeks.
- LEAR estimation: The proposed hybrid LASSO procedure matches coordinate-descent cross-validation accuracy while reducing computational cost by an order of magnitude.It selects λ daily with LARS and in-sample AIC, then recalibrates LEAR using coordinate descent.
5. Guidelines and best practices in EPF
The paper recommends longer, representative test periods, comparisons against established methods, reproducible open-access resources, and evaluation metrics suited to electricity-price forecasting. These practices address limited comparability and reproducibility in EPF studies.
- Length of the test period: A minimum one-year testing period is recommended because short samples can be unrepresentative, enable cherry-picking, and omit rare extreme events.The proposed benchmark therefore uses two-year testing periods across several markets.
- Baseline comparisons: New forecasting methods should be compared with well-established methods rather than outdated or simplified alternatives.Without such comparisons, it is difficult to determine which algorithms perform best.
- Reproducibility: Public datasets and shared code are needed to reproduce research and compare methods under the same conditions.The paper addresses this through an open-access benchmark and toolbox containing five datasets, two state-of-the-art methods, and their day-ahead forecasts.
- Evaluation metrics: MAE and RMSE are not always informative across datasets because absolute and quadratic errors are difficult to interpret for electricity-price applications.The paper also notes that MAPE is dominated by periods with prices close to zero.
- Evaluation metrics: Percentage-error metrics such as sMAPE can have statistical distributions with undefined means and infinite variances.This limits their suitability for evaluating electricity-price forecasts despite sMAPE addressing some traditional MAPE problems.
- Evaluation metrics: The paper recommends always including relative MAE (rMAE) to obtain fairer evaluations and comparisons of new EPF methods.Scaled-error approaches compare performance with a naive forecast, but MASE can use different scaling factors across calibration settings.
5.5. Statistical testing
The paper recommends statistical testing to determine whether forecast-accuracy differences are genuine rather than random. It presents DM and GW tests, with day-ahead variants that either identify hourly differences or summarize them jointly.
- Statistical tests are needed to establish whether differences in forecast accuracy are statistically significant rather than random.
- The paper provides open-access forecasts and implements DM and GW tests in the epftoolbox library to support reproducible statistical comparisons.
- The Diebold-Mariano test: The Diebold-Mariano test evaluates forecasts through the mean loss differential, using prediction errors and a selected loss function.
- The Diebold-Mariano test: The DM statistic is asymptotically standard normal when the loss differential is covariance stationary, enabling one- or two-sided tail probabilities.
- Day-ahead variants: For day-ahead markets, the paper recommends univariate hourly tests or a multivariate test applied jointly across all 24 hours.The univariate variant reveals which hours differ, while the multivariate variant summarizes the comparison in a single p-value.
- The Giacomini-White test: The GW test extends DM testing to conditional predictive ability by accounting for parameter-estimation uncertainty through conditioning.
5.6. Recalibration
Forecasting models should be recalibrated as new market information arrives, because estimating them once and evaluating them over the entire test period does not represent real-life conditions. For a one-year day-ahead test period, realistic evaluation may require 365 estimations.
- Estimating a model once for the whole test dataset fails to represent real-life conditions, where models are retrained using new market information.
- A realistic day-ahead evaluation retrains the forecasting model daily as new information becomes available.
- 365 estimations are required over a one-year testing period when a day-ahead model is retrained daily.
5.7. Ex-ante hyperparameter optimization
The paper identifies unclear or ex-post hyperparameter selection as a reproducibility and evaluation problem. It recommends explicit ex-ante selection using validation data and supplies supporting toolbox modules and optimization files.
- Hyperparameter-selection procedures are often insufficiently explained or performed ex-post, including by examining configurations on the test dataset.
- Ex-post hyperparameter optimization overfits the test dataset and gives the model an unfair advantage over other models.
- Hyperparameters should be selected explicitly ex-ante using a validation dataset.
- The toolbox provides a hyperparameter-selection module and files containing the study’s optimization results.
5.8. Computation time
The paper argues that computation time is essential for judging whether a forecasting method is practical, especially when models must be recalibrated daily. A useful model should be compared with methods that have similar computational requirements, not only on accuracy.
- Computational requirements are rarely reported, even though marginal accuracy gains may not justify substantially greater deployment costs.
- Models requiring more than 30 minutes or 1 hour will unlikely be suitable for spot-market price forecasting because daily recalibration may not finish in time.
- A model with computation time in minutes should outperform all state-of-the-art models, whereas a seconds-scale model should outperform low-computation state-of-the-art methods.
- The paper reports computation times for its open-access models to facilitate comparisons by other researchers.
5.9. Reproducibility
EPF studies often omit essential dataset, input, and hyperparameter details, making replication difficult. Future papers should provide these details, with reviewers checking that they are complete.
- Four recurring omissions are undefined testing datasets, training datasets, model inputs, and hyperparameter selection.
- Future EPF papers should document enough information to allow replication, and reviewers should verify the necessary dataset details.
5.10. Data contamination
Reliable EPF evaluation requires strict temporal separation between training, validation, and testing data. Studies should also use current, freely accessible software, and compare ensemble methods fairly.
- Data contamination: Data contamination can cause models to overfit the test set, making reported forecasting accuracy too high.
- Data contamination: Test data should be the final segment of the time series and must not overlap with training or validation data.Random test sampling, unspecified splits, and ex-post hyperparameter optimization can create contamination and overestimate accuracy.
- Software toolboxes: Benchmark methods should be implemented with continuously updated, state-of-the-art toolboxes because outdated libraries produce unfair evaluations.
- Software toolboxes: Free-access libraries should be used whenever possible so that other researchers can replicate the study.
- Ensembles: Ensembles should be compared with other ensemble techniques rather than only against individual models.
- Ensembles: The benchmark provides arithmetic-average ensembles of four LEAR models using different calibration windows and four DNNs using different hyperparameters.
6. Evaluation of state-of-the-art methods
Across five markets, the benchmark compares statistical and deep-learning models using multiple accuracy metrics and statistical tests. DNN ensembles perform best overall, while LEAR offers much lower computational cost and MAPE is unreliable.
- Accuracy metrics: MAPE is unreliable because it can disagree with rMAE, MAE, sMAPE, and RMSE, while rMAE and sMAPE provide more consistent evaluations.
- Accuracy results: DNN models outperform LEAR models on reliable linear metrics across all five datasets, while DNNs are better on RMSE in four of five datasets.
- Ensembles: Ensembling improves the accuracy of both DNN and LEAR models relative to their best individual counterparts across markets and reliable metrics.
- Statistical testing: Across all five markets, the DNN ensemble is statistically significantly better than every other benchmark model.
- Computation time: LEAR computation is 30 to 100 times lower than DNN computation and up to 50 times faster at maximum estimation time.
- Practical comparison: New EPF methods should be compared with DNN or LEAR depending on available decision time, including both accuracy and computational requirements.
7. Checklist to ensure adequate EPF research
The checklist defines reproducible and meaningful EPF research through transparent data handling, broad benchmarking, appropriate metrics, significance testing, and computational evaluation.
- EPF studies should use at least one year of testing data, multiple markets, and open-access datasets.
- New models should be tested against state-of-the-art open-access models and their computational costs compared with existing methods.
- rMAE should be included among accuracy metrics, and statistical tests should assess whether performance differences are significant.
- Models should be recalibrated daily rather than estimated once for the entire out-of-sample period.
- Researchers should explicitly report dataset splits and dates, model inputs, and non-overlapping train, validation, and test sets.
- Benchmark models should use state-of-the-art, free toolboxes to support fair and reproducible comparisons.
8. Conclusion
The paper proposes a rigorous, reproducible framework for evaluating electricity price forecasting models through best practices, open-access datasets, and open-source benchmarks. Its evaluations show distinct roles for deep neural networks, LEAR, and ensemble methods, while identifying requirements for meaningful comparisons.
- 8. Conclusion: The paper derives best practices intended to make EPF research adequate, reproducible, and useful.These practices address factors such as dataset size and accuracy metrics.
- 8. Conclusion: A six-year benchmark spanning five markets provides common conditions for validating new forecasting methods and obtaining meaningful comparisons.The benchmark is designed to reduce reliance on previously untested datasets.
- 8. Conclusion: Open-source statistical and deep learning models are automatically tuned, require no expert knowledge, and can serve as benchmarks in future studies.The models are distributed through the epftoolbox Python library.
- 8. Conclusion: Deep neural networks are more likely to outperform LEAR, whereas LEAR is best for applications with short decision timeframes.The evaluation also found that ensemble methods often significantly outperform their individual counterparts.
- 8. Conclusion: MAPE is unreliable and should be avoided, statistical testing is mandatory, and test datasets should span at least one year.These guidelines are presented as the most notable conclusions from the benchmark results.