Short-term Load Forecasting with Deep Residual Networks

Kunjin Chen; Kunlong Chen; Qin Wang; Ziyu He; Jun Hu; Jinliang He

Source-linked AI summary

Short-term Load Forecasting with Deep Residual Networks

Kunjin Chen, Kunlong Chen, Qin Wang, Ziyu He, Jun Hu, Jinliang He

arXiv:1805.11956v1May 30, 2018stat.MLcs.LGstat.AP

TL;DR

Short-term load forecasting requires accurate, generalizable models, while deep neural networks provide flexible structures but can be difficult to train as they become deeper. This paper proposes a deep residual-network model with a modified residual structure, two-stage ensemble strategy, and MC dropout for probabilistic forecasting, reporting strong accuracy and generalization across public datasets.

Problem
Existing neural-network STLF models can be difficult to train as depth increases, alongside concerns about over-parameterization and overfitting.
Method
The paper develops an end-to-end deep residual STLF model, modifies its residual structure, ensembles multiple networks, and applies MC dropout for probabilistic forecasting.
Results
8.9% improvement in overall MAPE is achieved for 2011 on the ISO-NE dataset, while the proposed model outperforms existing models for the overall MAPE of 2010 and 2011.
Takeaways & Limitations
The proposed model provides accurate and robust forecasting, generalizes across datasets, and can be directly used for probabilistic forecasting with MC dropout.

Abstract

from arXiv · show

We present in this paper a model for forecasting short-term power loads based on deep residual networks. The proposed model is able to integrate domain knowledge and researchers' understanding of the task by virtue of different neural network building blocks. Specifically, a modified deep residual network is formulated to improve the forecast results. Further, a two-stage ensemble strategy is used to enhance the generalization capability of the proposed model. We also apply the proposed model to probabilistic load forecasting using Monte Carlo dropout. Three public datasets are used to prove the effectiveness of the proposed model. Multiple test cases and comparison with existing models show that the proposed model is able to provide accurate load forecasting results and has high generalization capability.

I. INTRODUCTION

Short-term load forecasting is important for power-system management, but conventional and neural-network approaches face concerns about over-parameterization, overfitting, and difficult training as depth increases. The paper addresses these issues with an end-to-end deep residual model, ensemble strategy, and Monte Carlo dropout extension.

Precise short-term load forecasts support power-system management, scheduling, dispatch, renewable-energy integration, and increasingly complex electricity markets.
Earlier STLF approaches include regression, support vector regression, autoregressive, fuzzy-logic, and several neural-network variants.
Neural-network STLF has faced over-parameterization, rapidly growing model size, overfitting concerns, and limited credibility of some reported results.
Deep neural networks offer flexible task-specific building blocks, but existing multilayer forecasting models become increasingly difficult to train as depth increases.
The proposed end-to-end model forecasts the next 24 hours from readily available raw load, temperature, and related information without external feature extraction or selection.
A modified residual network, two-stage ensemble, and MC dropout support improved forecasting performance, generalization across datasets, and probabilistic STLF.

A. Model Input and the Basic Structure for Load Forecasting of One Hour

The basic one-hour forecasting structure combines load, temperature, temporal, and calendar inputs to produce each next-day hourly forecast. Preliminary forecasts for all 24 hours are then associated and passed to a residual network, using SELU activations in the fully connected components.

Inputs combine recent loads, historical same-day information, temperatures, season, weekday or weekend status, and holiday status to capture short-term, long-term, periodic, and unusual temporal characteristics.
For each forecast hour, paired monthly and daily inputs are concatenated and processed through separate fully connected layers before combination with hourly and temperature features.
SELU is used in the fully connected layers, while the residual-network building block places SELU between two linear layers.
SELU is presented as a modification related to ReLU and PReLU that can help prevent vanishing and exploding gradients under specified parameter values.
Forecast values replace unavailable future entries and retain underlying neural-network connections so gradients can propagate backward across subsequent hourly forecasts.
The model concatenates the 24 hourly outputs into L, which becomes the basic model output before the deep residual network is added.

B. The Deep Residual Network Structure for Day-ahead Load Forecasting

The model uses residual blocks and shortcut connections to forecast 24-hour loads, then extends this structure with side blocks and dense connections in ResNetPlus. These additions are intended to improve representation capability and error back-propagation.

Residual blocks learn F(x, Θ) rather than directly learning the full mapping H(x).
Stacking K residual blocks forms a deep residual network that maps the residual-network input x0 to output xK.
Shortcut connections allow gradients to bypass residual blocks, making vanishing gradients much less likely during back-propagation.
ResNet: ResNet adds lower-level shortcuts across adjacent residual blocks and a higher-level shortcut from network input to output, averaging multiple arriving values.
ResNetPlus: ResNetPlus adds side residual blocks, averages their outputs with main-path outputs, and connects these averages to subsequent main residual blocks.
ResNetPlus: The additional side blocks and dense shortcuts are expected to improve representation capability and error back-propagation efficiency.

C. The Ensemble Strategy of Multiple Models

The ensemble strategy combines snapshot models from one training run with independently trained models. Their outputs are averaged to produce the final forecast, with model counts tuned on validation data.

The two-stage ensemble strategy combines multiple models to improve generalization and reduce performance variance.
Stage 1: snapshots: The first stage takes snapshots from a single model during training when validation losses are at similar levels.
Stage 2: independent models: The second stage trains additional models independently by re-initializing their parameters from the same initialization distribution.
The numbers of snapshots and retrained models are validation-tuned hyper-parameters.
The final forecast is obtained by averaging the outputs of all snapshot and independently trained models.

D. Probabilistic Forecasting Based on Monte Carlo Dropout

The paper uses Monte Carlo dropout to estimate predictive uncertainty for probabilistic load forecasting. It combines dropout-based model uncertainty with estimated data noise and derives forecast intervals from the resulting uncertainty.

Monte Carlo dropout approximates Bayesian forecasting uncertainty by stochastically dropping hidden units during M test-time passes.
The first uncertainty term is estimated from the variation among the M stochastic network outputs and their mean.
The second uncertainty term represents inherent noise in the data-generating process and is estimated using an independent validation dataset.
Ensemble uncertainty: For model ensembles, the first uncertainty term uses a dropout-trained model, while the noise scale β is selected using validation 90% and 95% interval forecasts.
Ensemble uncertainty: The data-noise estimate replaces the single-model prediction with the ensemble prediction and is specific to each hour of the day.
Probabilistic forecasts are produced by combining the point forecast with corresponding uncertainty quantiles to form α-level intervals.

E. Model Design and Implementation Details

The proposed model combines a one-hour basic load-forecasting structure with ResNet and ResNetPlus components for improving 24-hour forecasts.

The model consists of a basic one-hour neural-network structure, a ResNet for 24-hour forecast improvement, and the modified ResNetPlus.

1) The model with the basic structure:

The model combines a basic fully connected forecasting structure with residual networks and trains it using forecast-error and out-of-range penalty terms.

The basic structure uses fully connected layers to process inputs and produce preliminary forecasts for the next 24 hourly loads.
ResNet adds 30 residual blocks with 24-dimensional outputs, forming a 60-layer deep residual network with hierarchical shortcut connections.
ResNetPlus follows the Fig. 4 residual-block structure while retaining the ResNet blocks’ hyper-parameters.
The training loss combines a forecast-error term, LE, with an out-of-range penalty term, LR.LR penalizes forecasts outside the actual daily load-curve range and emphasizes overestimated peaks and valleys after accuracy improves.
Training uses Adam with default parameters; five individual models require less than eight hours in total.A three-year ResNetPlus training run with 700 epochs takes approximately 1.5 hours.

III. RESULTS AND DISCUSSION

The results section evaluates the proposed model on the North-American Utility and ISO-NE datasets, including tests with modified temperature inputs and probabilistic forecasting.

The experiments use the North-American Utility and ISO-NE datasets to evaluate the proposed model’s effectiveness.
The evaluation also includes modified-temperature tests and probabilistic forecasting on the North-American Utility and GEFCom2014 datasets.
Figure 6 compares test losses for the Basic, Basic + ResNet, and Basic + ResNetPlus models across repeated training runs.

A. Performance of the Proposed model on the North-American Utility Dataset

On the North-American Utility and ISO-NE datasets, ResNetPlus improves load-forecasting performance and generalizes across datasets and modified-temperature conditions.

A. Performance of the Proposed model on the North-American Utility Dataset: ResNet improves the basic model’s performance, while ResNetPlus achieves a further reduction in test loss.
A. Performance of the Proposed model on the North-American Utility Dataset: The proposed model outperforms existing North-American Utility models and shows a lower MAPE increase with modified temperature inputs.
B. Performance of the Proposed Model on the ISO-NE Dataset: The ResNetPlus model achieves the lowest overall MAPE for ISO-NE in 2006, although WT-ELM-MABC performs better in some months.
B. Performance of the Proposed Model on the ISO-NE Dataset: The model outperforms existing models on ISO-NE data from 2010 and 2011, with an 8.9% improvement reported for 2011 without dataset-specific tuning.
B. Performance of the Proposed Model on the ISO-NE Dataset: 0.0168% is the MAPE increase for case 1 under modified temperature, while the paper cautions that the comparison with 0.04% from prior work is not direct.
B. Performance of the Proposed Model on the ISO-NE Dataset: The ensemble strategy reduces the standard deviation across modified-temperature trials, indicating higher generalization capability than the single-model comparison.

C. Probabilistic Forecasting for the Ensemble Model

The proposed ensemble model is evaluated for probabilistic short-term load forecasting using Monte Carlo dropout on public load datasets. It achieves satisfactory interval coverage and improved probabilistic forecasting scores compared with existing methods.

The North-American Utility dataset is used to demonstrate probabilistic forecasting with Monte Carlo dropout, including 95% prediction intervals for winter and summer weeks.The last year is used for testing and the previous year for validation; dropout with p = 0.1 is added to the ensemble model.
The proposed model with Monte Carlo dropout provides satisfactory empirical coverages for different prediction intervals.Coverage results are reported for different z-scores in Table V, with illustrative 95% intervals shown for two weeks in 1992.
The probabilistic forecasting evaluation uses pinball loss and Winkler scores to assess coverage rate and interval width.Pinball loss is averaged across quantiles and forecast hours, while Winkler scores are averaged across all hours in the test year.
The proposed ResNetPlus model improves probabilistic forecasting results over existing methods on the 2011 GEFCom2014 dataset.The comparison covers the pinball loss and two Winkler scores against Lasso and quantile regression averaging methods.
The probabilistic forecasts are generated by sampling trained neural networks with Monte Carlo dropout, capturing uncertainty in short-term load forecasting.

IV. CONCLUSION AND FUTURE WORK

The paper concludes that deep residual networks and a two-stage ensemble strategy support accurate and robust short-term load forecasting. It also identifies additional deep-learning building blocks and further probabilistic comparisons as future work.

The proposed deep-residual-network model combines a basic neural network, ResNetPlus, and a two-stage ensemble strategy.The authors report that these components provide high accuracy and satisfactory generalization capability.
Comparisons across two public datasets and various test cases show superior forecasting accuracy and robustness to temperature variation.
Monte Carlo dropout allows the proposed model to be used directly for probabilistic forecasting.
Future work includes integrating additional deep-neural-network building blocks such as CNNs or LSTMs.The authors also plan to investigate deep-neural-network implementations for probabilistic forecasting and make further comparisons with existing methods.