PaperlayerView exact revision on arXiv

Source-linked AI summary

An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting

Filippo Maria Bianchi, Enrico Maiorino, Michael C. Kampffmeyer, Antonello Rizzi, Robert Jenssen

arXiv:1705.04378v2cs.NE

TL;DR

Accurate short-term load forecasting is important for reducing resource waste and service-related costs, yet modern recurrent architectures remain insufficiently compared for this task. The paper reviews and compares five RNN architectures on synthetic and real-world time series, providing training and configuration guidance. It finds that no single architecture dominates every problem, while ERNN and ESN offer particularly convenient combinations of performance and simplicity.

  • Problem

    Accurate real-valued load forecasting matters for resource management, but a comprehensive comparison of modern RNN architectures for STLF is lacking.

  • Method

    The paper reviews five RNN architectures, explains their mechanisms and training procedures, and compares them on synthetic benchmarks and real-world STLF datasets.

  • Results

    No specific RNN outperforms the others on every prediction problem; ESN is competitive in most tasks, while gated RNNs do not perform particularly better than ERNN.

  • Takeaways & Limitations

    ERNN and ESN may be convenient choices for time-series prediction in terms of performance and implementation and training simplicity.

  • Takeaways & Limitations

    RNN training lacks well-established methodologies, and architecture-specific strategies may require considerable user expertise.

Abstract

from arXiv · show

The key component in forecasting demand and consumption of resources in a supply network is an accurate prediction of real-valued time series. Indeed, both service interruptions and resource waste can be reduced with the implementation of an effective forecasting system. Significant research has thus been devoted to the design and development of methodologies for short term load forecasting over the past decades. A class of mathematical models, called Recurrent Neural Networks, are nowadays gaining renewed interest among researchers and they are replacing many practical implementation of the forecasting systems, previously based on static methods. Despite the undeniable expressive power of these architectures, their recurrent nature complicates their understanding and poses challenges in the training procedures. Recently, new important families of recurrent architectures have emerged and their applicability in the context of load forecasting has not been investigated completely yet. In this paper we perform a comparative study on the problem of Short-Term Load Forecast, by using different classes of state-of-the-art Recurrent Neural Networks. We test the reviewed models first on controlled synthetic tasks and then on different real datasets, covering important practical cases of study. We provide a general overview of the most important architectures and we define guidelines for configuring the recurrent networks to predict real-valued time series.

1. Introduction

Short-term load forecasting matters for managing limited resources, but changing and nonlinear temporal dynamics make accurate prediction difficult. The paper motivates recurrent neural networks as a framework for modeling temporal dependencies and compares several architectures for this task.

  • Motivation: Accurate short-term load forecasts help manage limited distribution-network resources and reduce costs from over- and under-contracting.Overestimation can create excess supply and curtailments, while underestimation can require costly supplementary services.
  • Motivation: Load profiles exhibit cyclic and seasonal patterns, while their generating dynamics can vary across the observation period because of latent and external influences.Consequently, forecasting accuracy may differ substantially across samples even when the same model is used.
  • Prior approaches: Traditional autoregressive and exponential-smoothing models require lag selection and rely on assumptions about the underlying system.These requirements limit their use when suitable model orders or prior system knowledge are unavailable.
  • Prior approaches: Fixed-window function-approximation methods can fail when important temporal dependencies exceed the chosen window because they cannot learn an internal state representation.Takens-style embedding reformulates prediction as phase-space interpolation using sampled past values.
  • Recurrent neural networks: RNNs use recurrent self-connections and hidden states to process sequences while retaining information from previous inputs and computations.Their theoretical ability to represent nonlinear dynamical systems makes them suitable for data with nonlinear interactions and potentially long temporal dependencies.
  • Paper scope: The paper addresses the lack of a comprehensive comparison of modern RNN architectures for STLF by reviewing architectures, training procedures, and configuration guidelines.It evaluates five RNN architectures on controlled synthetic datasets and real-world load time series from energy and telecommunication networks.

2. Properties and Training in Recurrent Neural Networks

RNNs model temporal data through recurrent state updates and can approximate complex nonlinear dynamical systems, but their flexibility makes training and architecture design difficult.

  • Architecture: RNNs recursively map input sequences to output sequences through nonlinear transformations of previous states and inputs.They can approximate arbitrary nonlinear dynamical systems under loose regularity conditions.
  • Architecture: A general RNN is a weighted, directed, cyclic graph containing input, hidden, and output nodes connected across time.Time-shift operators represent delays, including skip connections for longer delays.
  • Training challenges: Although RNNs can theoretically model arbitrary dynamical systems, training methods and design choices lack a fully established general methodology.Architectures may use gradient-based, derivative-free, convex, filtering, evolutionary, or fixed-weight approaches.
  • Training: Unfolding an RNN converts its recurrent graph into a feedforward network while sharing the same weights across time steps.This representation enables standard feedforward learning algorithms such as Back Propagation Through Time.
  • Training: BPTT truncates the unfolded network after finitely many time steps, limiting depth while approximating gradient propagation through the recurrent sequence.BPTT(2n, n) is described as a trade-off between accuracy and computational performance relative to BPTT(n, 1).
  • Optimization: Gradient descent alternates a forward loss evaluation on inputs with backpropagation of the loss gradient to update all network parameters.The loss compares predicted outputs with ground-truth values and may include regularization.
  • Loss and regularization: The paper considers MSE error and four regularization types, including L1, L2, elastic net, and dropout.Dropout uses a random hidden-unit mask during training, with the same units dropped across each gradient-descent epoch in the adopted formulation.

3. Recurrent Neural Networks Architectures

The paper reviews Elman, LSTM, and GRU recurrent architectures for real-valued time-series forecasting, emphasizing their state updates, nonlinear transformations, and gating mechanisms. These architectures extend basic recurrent processing with mechanisms intended to manage temporal information and memory.

  • Elman RNN: Elman RNNs are the basic recurrent architecture, and LSTM and GRU can be interpreted as extensions or variations of them.ERNNs use recurrent hidden-layer connections, while the more complex architectures introduce additional mechanisms for controlling information flow.
  • Elman RNN: At each time step, an ERNN combines the current input and previous hidden state, applies a nonlinear activation, and transforms the resulting state into an output.The hidden state carries memory from previous inputs and network states; the output transformation is usually linear.
  • Long Short-Term Memory: LSTM is widely used for modeling short- and long-term dependencies and has been applied successfully to sequence-learning tasks, including handwriting, speech, music, grammar, and image tagging.The reviewed literature contains fewer applications of LSTM to real-valued time-series prediction, although one cited traffic-speed study found LSTM nearly always best among the compared approaches.
  • Long Short-Term Memory: LSTM modifies recurrent processing through three gates that control forgetting, updating, and output of cell information.Its cell state is modified through linear interactions, allowing information to backpropagate more smoothly across time and enhancing memory capacity.
  • Long Short-Term Memory: In LSTM, the forget gate discards selected previous cell-state information, the update gate controls incorporation of a candidate state, and the output gate filters the current state.Each gate depends on the current input and the previous cell output.
  • Long Short-Term Memory: The LSTM cell can preserve its state exactly only when the forget gate is 1 and the update gate is 0; sigmoid gates do not reach these values at finite inputs.Consequently, LSTM improves long-term memory relative to ERNN but cannot keep cell content completely unchanged in practice.

4. Other Recurrent Neural Networks Models

This section introduces NARX and ESN architectures, emphasizing their distinct recurrence structures and training procedures for time-series prediction.

  • NARX networks: NARX networks use output feedback for recurrence rather than feedback from the whole internal state.
  • NARX networks: A NARX network regresses the next output on finite windows of current and delayed exogenous inputs and previous outputs.Its MLP mapping uses input delay dx and output delay dy.
  • NARX networks: During NARX training, the desired output replaces feedback, making the network feed-forward and enabling standard backpropagation techniques.After optimization, the teacher signal is removed and predicted outputs are fed back.
  • NARX networks: NARX networks can still suffer vanishing gradients because stability constraints require the expanded state-transition Jacobian to have eigenvalues inside the unit circle.The limiting Jacobian is described as tending to zero for long expansions.
  • Echo State Networks: ESNs combine a large, sparsely connected untrained nonlinear recurrent layer with a trained linear, memory-less readout.ESN training is characterized by a fast procedure that usually solves a convex optimization problem.
  • Echo State Networks: In ESNs, the reservoir state is driven by the input and produces the output through readout weights, with nonlinear reservoir activation and typically linear readout activation.The state has Nh components, while Ni and No denote input and output dimensionalities.

5. Synthetic time series

The study evaluates recurrent architectures on three synthetic forecasting tasks designed to vary in signal structure, memory requirements, and difficulty.

  • Benchmark design: The synthetic benchmarks are selected as controlled and replicable tasks with different levels of difficulty.The forecast horizon is chosen so observations become decorrelated, using the first zero of the autocorrelation function.
  • Mackey-Glass: The Mackey-Glass system provides a benchmark for forecasting chaotic time series.The experiments use τMG = 17 and forecast horizon tf = 12.
  • NARMA: The NARMA task models an r-order nonlinear autoregressive moving-average system driven by uniform random noise.The output depends on inputs and outputs from the last r time-steps; experiments use r = 10 and tf = 1.
  • Multiple Superimposed Oscillator: The Multiple Superimposed Oscillator combines four sine waves with incommensurable frequencies, producing extremely long periodicity.Accurate prediction requires substantial memory to implement multiple decoupled internal dynamics, with tf = 10.

6. Real-world load time series

The real-world evaluation covers electricity and telecommunication load series, using dataset-specific exploratory analysis and preprocessing to configure the forecasting inputs.

  • Dataset overview: The real-world study uses three datasets containing electricity and telephonic load measurements, with exogenous variables in two datasets.Each dataset undergoes pre-analysis to identify suitable preprocessing.
  • Orange dataset: The Orange forecasting task predicts the next day’s incoming-call volume using a 24-step-ahead horizon, while remaining series provide exogenous information.The data contain 3336 hourly measurements split into 70% training, 15% validation, and 15% test sets.
  • Orange dataset: Orange load series show strong daily seasonality at lag 24, which is removed through seasonal differencing so recurrent networks focus on remaining dynamics.Mutual-information analysis found strong load relationships with hour-of-day and low dependency on weekday; ts6 and ts5 were discarded after preprocessing.
  • Orange dataset: The Orange preprocessing sequence is log-transform, seasonal differencing at lag 24, and standardization, with transformations reversed for forecast evaluation.The data show heteroscedasticity but not overdispersion, motivating the log transformation.
  • ACEA dataset: The ACEA electricity series spans almost three years and varies across periods because consumption is related to temperature, daylight saving time, holidays, and other seasonal events.January and June exhibit different load profiles, illustrating time-varying seasonal behavior.
  • GEFCom2012 dataset: For GEFCom2012 electricity forecasting, temperature is used as an exogenous variable because consumption rises at both low and high temperatures.The joint distribution has a V-shape around a mean temperature of about 22°C, despite an estimated linear correlation of 0.2.

7. Experiments

Across synthetic and real-world forecasting experiments, the strongest architecture varies by task and dataset, while several configuration patterns recur. NARX often performs poorly, and model complexity, memory windows, optimization, and regularization require task-specific tuning.

  • Synthetic datasets: The best-performing RNN differs by synthetic task: ESN leads MG and MSO, whereas LSTM is most accurate on NARMA.ESN performs poorly on NARMA, while ERNN follows ESN on MSO.
  • Synthetic datasets: NARX struggles across synthetic tests, reaching NRMSE 0.53 on NARMA and 1.99 on MSO.Because NRMSE values above 1 are worse than a constant mean predictor, the MSO result is especially poor.
  • Synthetic datasets: ERNN outperforms GRU and LSTM on MSO despite the task’s long-memory requirement, likely because learning the constituent frequencies does not require high modeling capacity.The optimal MSO network uses fewer hidden nodes, reducing overfitting risk while retaining sufficient modeling ability.
  • Synthetic datasets: LSTM and GRU perform similarly across tasks without a consistent winner, making advance selection of the better gated architecture difficult.The result agrees with earlier empirical studies cited by the authors.
  • Configuration analysis: Adam is often selected as the optimal optimizer, while L2 regularization is consistently preferred and L1 and dropout are unused across synthetic tasks.SGD is selected only for GRU on MG; the authors attribute its rarity partly to slower convergence within 400 validation epochs.
  • Configuration analysis: Higher-complexity models often obtain the best results, but their small performance gains require substantially longer training times.For real-world datasets, configurations vary: Orange and ACEA favor short NARX delays and low complexity, while ESN settings use small spectral radii and short memory.

8. Conclusions

The study finds that no single RNN architecture dominates across prediction problems, while ERNN and ESN often offer attractive combinations of performance and simplicity. Model choice depends on the task, training strategy, and hyperparameter configuration.

  • The study compares five RNN architectures across synthetic benchmarks and real-world short-term load-forecasting problems.The architectures are ERNN, LSTM, GRU, NARX, and ESN.
  • No specific RNN model outperforms the others in every prediction problem, so architecture choice depends on the task and configuration.The authors emphasize evaluating multiple training strategies and configurations for each RNN.
  • NARX achieved the lowest average performance, especially on NARMA, MSO, and GEFCom.
  • Gradient-based ERNN, LSTM, and GRU training is slower and more complex because it requires unfolding and backpropagation through time.Satisfactory results can nevertheless be obtained with minimal fine-tuning and default hyperparameters.
  • ESN results are competitive in most tasks, but its faster and simpler training comes with greater sensitivity to hyperparameter selection.The authors state that ESN requires a more accurate hyperparameter search during validation than gradient-based models.
  • LSTM and GRU did not perform particularly better than the simpler ERNN, suggesting that gated mechanisms may be unnecessary for many load time-series tasks.The paper contrasts irregular sequence dependencies with dynamical systems whose dependencies decay smoothly over time.
  • The paper concludes that ERNN and ESN may be the most convenient choices when both prediction performance and implementation and training simplicity matter.
Loading 1705.04378v2…