PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning with Long Short-Term Memory for Time Series Prediction

Yuxiu Hua, Zhifeng Zhao, Rongpeng Li, Xianfu Chen, Zhiming Liu, Honggang Zhang

arXiv:1810.10161v1cs.NEcs.LG

TL;DR

Long-range dependencies challenge time-series prediction, while LSTM addresses them at substantial parameter-related computing cost. The paper introduces sparsely connected RCLSTM and evaluates it on telecommunications traffic and user-mobility prediction, reporting comparable accuracy with reduced computation.

  • Problem

    Time-series algorithms often struggle to learn long-range dependencies, while telecommunications systems need predictions under changing traffic and user movements.

  • Method

    The paper introduces RCLSTM, which forms LSTM architectures through stochastic sparse connectivity, and evaluates it on traffic and user-location forecasting.

  • Results

    RCLSTM achieves prediction accuracy comparable to conventional LSTM while using fewer connections; traffic experiments report 30% lower computing time at 1% connectivity.

  • Takeaways & Limitations

    Sparse connectivity makes RCLSTM more applicable to latency-stringent telecommunications scenarios while retaining comparable prediction accuracy.

Abstract

from arXiv · show

Time series prediction can be generalized as a process that extracts useful information from historical records and then determines future values. Learning long-range dependencies that are embedded in time series is often an obstacle for most algorithms, whereas Long Short-Term Memory (LSTM) solutions, as a specific kind of scheme in deep learning, promise to effectively overcome the problem. In this article, we first give a brief introduction to the structure and forward propagation mechanism of the LSTM model. Then, aiming at reducing the considerable computing cost of LSTM, we put forward the Random Connectivity LSTM (RCLSTM) model and test it by predicting traffic and user mobility in telecommunication networks. Compared to LSTM, RCLSTM is formed via stochastic connectivity between neurons, which achieves a significant breakthrough in the architecture formation of neural networks. In this way, the RCLSTM model exhibits a certain level of sparsity, which leads to an appealing decrease in the computational complexity and makes the RCLSTM model become more applicable in latency-stringent application scenarios. In the field of telecommunication networks, the prediction of traffic series and mobility traces could directly benefit from this improvement as we further demonstrate that the prediction accuracy of RCLSTM is comparable to that of the conventional LSTM no matter how we change the number of training samples or the length of input sequences.

I. INTRODUCTION

Time-series prediction estimates future values from historical observations, an important capability for managing dynamic telecommunications networks. The paper introduces RCLSTM to reduce LSTM’s parameter-related computing cost while maintaining comparable prediction accuracy.

  • Telecommunications motivation: Dynamic traffic and user movements make maintaining telecommunications quality of service and experience challenging.The paper motivates predicting data traffic and likely user-demand locations in mobile communication networks.
  • Prediction objective: Time-series prediction estimates y_i from preceding observations by learning a function that approximates the ground truth.The input is a historical window x={y_i−k,...,y_i−1}, and the prediction is ŷ_i=f(x).
  • Existing approaches: RNNs support temporal modeling through recurrent connections, but standard architectures struggle to learn long-range dependencies in practice.Earlier statistical and machine-learning approaches also have limitations in capturing rapidly changing processes or selecting model parameters.
  • Contribution: RCLSTM introduces random sparse connectivity into LSTM to reduce the number of trainable parameters and computational cost.The approach is motivated by LSTM computing time being proportional to its parameter count.
  • Evaluation design: Two datasets—GÉANT traffic and realistic user trajectories—are used because neural-network performance can vary across datasets and both tasks matter for telecommunications.The simulations use a three-layer RCLSTM network with 300 memory cells per layer.
  • Reported results: 30% lower computing time is reported for RCLSTM than conventional LSTM in traffic prediction, while 1% connectivity outperforms ARIMA, SVR, and FFNN.Across traffic and user-location forecasting, the paper reports comparable capability with less than half the neural connections.

II. AN OVERVIEW OF ARTIFICIAL NEURAL NETWORKS AND LSTM

The paper reviews FFNNs, RNNs, and LSTMs, emphasizing how LSTM memory blocks and gates support learning long-term dependencies. It then situates the proposed work against prior LSTM variants and conventional fully connected architectures.

  • FFNNs and RNNs: FFNNs use layered neurons without within-layer connections, whereas RNNs add recurrent connections that carry information through hidden states.RNN outputs depend on the current input x_t and previous hidden state h_t−1.
  • LSTM architecture: LSTMs are specialized RNNs designed for learning long-term dependencies through recurrently connected memory blocks.Each block contains memory cells and gates that regulate information flow.
  • LSTM architecture: Input, forget, and output gates respectively regulate new information, retained cell information, and information used to produce the block output.The memory cell stores the network’s temporal state, while multiplicative gates control information flow.
  • Forward propagation: The LSTM forward process uses sigmoid layers for gate decisions and a tanh layer to generate candidate cell-state values.The forget and input decisions precede updating the cell state, after which the output gate controls the activation passed onward.
  • Prior variants: Conventional LSTM and eight investigated variants performed well on three benchmark problems, with no variant significantly improving performance.The evaluated tasks were TIMIT, IAM Online, and JSB Chorales.

III. RANDOM CONNECTIVITY FOR LSTM

RCLSTM replaces conventional LSTM’s full connectivity with stochastic, potentially sparse neural connections. The design targets lower computational cost while preserving the recurrent memory-block framework.

  • Motivation: Random neural connectivity is proposed as a possible way to improve LSTM performance and efficiency, based on random topology formation in neural microcircuits.This motivates constructing the RCLSTM model.
  • Architecture: RCLSTM neurons are randomly connected instead of fully connected, with omitted or added connections determining the architecture.Full connectivity recovers the standard LSTM, whereas rule-based random connectivity produces RCLSTM.
  • Architecture: The RCLSTM design changes connectivity among the input and recurrent-output components and the gate and input-update components where LSTM parameters reside.The sparse structure is illustrated as a randomly connected neural graph.
  • Connectivity generation: Each potential connection receives a probability value, and a threshold determines which connections remain active.The simulations use a uniform distribution for these probabilities and assign the same connection probability to all neurons.

IV. NUMERICAL SIMULATIONS FOR TRAFFIC AND MOBILITY PREDICTION

The simulations evaluate a three-layer RCLSTM network for next-moment traffic and user-mobility prediction, varying training-sample counts and input-sequence lengths.

  • The three-layer RCLSTM network predicts the next value from historical inputs y1 through yT.The predicted next value is denoted ŷT+1.
  • The experiments compare RCLSTM prediction accuracy with other algorithms or models for traffic and user mobility.
  • The study varies training-data sample counts and input-sequence lengths to assess their influence on prediction accuracy.

A. Data Description and Processing

The evaluation uses real GÉANT traffic data and mobile-user location histories, with preprocessing applied to traffic values before modeling.

  • The traffic dataset contains 10,772 GÉANT link measurements sampled every 15 minutes over four months.The observations span 2005-01-01 through 2005-04-30 and are measured in Kbps.
  • Traffic values are logarithmically transformed and normalized because the raw data span about three orders of magnitude.
  • The processed traffic data are constrained to [0, 1] to support faster convergence and avoid bad local optima during ANN training.
  • The mobility dataset contains mobile-user location histories and manually marked important places represented as map polygons.

B. Evaluation Metrics

Traffic prediction is evaluated with RMSE, while mobility prediction uses accuracy defined as the percentage of correct location predictions.

  • RMSE quantifies traffic-prediction error as the square root of the mean squared deviations between predicted and actual values.
  • Mobility-prediction accuracy measures the percentage of correct location predictions.
  • The accuracy formula compares each actual location yi with its prediction ŷi across N predicted locations.

1) Traffic Prediction

Traffic experiments examine connectivity, data volume, sequence length, and model comparisons. Sparse RCLSTM reduces computing time while retaining competitive prediction performance, though fully connected LSTM performs best in the reported comparison.

  • At 1% neural connectivity, RCLSTM reduces computing time by around 30% compared with baseline LSTM, with slightly higher RMSE.The 100% connectivity case represents the fully connected LSTM model.
  • Computing time nearly stops increasing above 20% connectivity because the acceleration method is effective mainly for extremely sparse matrices.
  • RCLSTM models are more sensitive than LSTM to the number of training samples but less sensitive to input-sequence length.
  • The comparison includes SVR, ARIMA, FFNN, and LSTM baselines with specified input and model settings.The comparison also includes an LSTM with memory cell size 30, which has a parameter count similar to 1%-connected RCLSTM with size 300.
  • LSTM with memory cell size 300 performs much better than the other compared models, followed by RCLSTM with size 300 and 1% connectivity.

2) Human Mobility Prediction

RCLSTM predicts human mobility across users and input-sequence lengths while using sparse neural connectivity. Its accuracy can approach LSTM, with user-specific differences linked to mobility regularity.

  • User Differences: RCLSTM prediction accuracy differs only slightly across users, with more regular schedules yielding higher expected predictability.Users A and D show similar behavioral patterns, whereas User E travels more with an unfixed schedule.
  • Figure Overview: Figure 5 compares RCLSTM mobility-prediction accuracy across users and across input-sequence lengths for Users A and E.Its panels cover different users, User A sequence lengths, and User E sequence lengths.
  • Efficiency: RCLSTM computes faster than LSTM because its high neural-connection sparsity reduces the computational burden.The mobility-prediction comparison reports lower accuracy than LSTM alongside faster computation.
  • Input Sequence Length: RCLSTM models can catch up with LSTM in prediction accuracy across different input-sequence lengths.This pattern is reported for both Users A and E.
  • Input Sequence Length: For User A, increasing input-sequence length scarcely changes accuracy, while User E’s accuracy improves as sequence length increases.Both models need more input data to identify User E’s movement patterns because that mobility data is more irregular.

V. CONCLUSION

The conclusion presents RCLSTM as a sparse redesign of LSTM for traffic and user-mobility forecasting. With substantially fewer connections, it reduces computational complexity while retaining competitive prediction accuracy across tested conditions.

  • Conclusion: RCLSTM redesigns conventional LSTM by constructing a random sparse neural graph for traffic and user-mobility forecasting.The model was evaluated across various temporal scales.
  • Conclusion: 30% lower computing complexity is achieved by RCLSTM with 1% neural connections in traffic prediction.The conclusion also reports an approximately 25% prediction-accuracy degradation associated with sparse connections.
  • Conclusion: RCLSTM accuracy fluctuates in line with LSTM as input-sequence length and training-sample count change.It also outperforms SVR, ARIMA, FFNN, and equal-parameter LSTM in the reported traffic-prediction comparison.
  • Conclusion: The paper concludes that RCLSTM combines lower computing costs with satisfactory performance for time series prediction.This conclusion follows the reported traffic and user-mobility experiments.
Loading 1810.10161v1…