PaperlayerView exact revision on arXiv

Source-linked AI summary

Do RNN and LSTM have Long Memory?

Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, Guangjian Tian

arXiv:2006.03860v2stat.MLcs.LG

TL;DR

The paper asks whether RNN and LSTM genuinely have long memory, given the difficulty of modeling long-term dependence and the limits of heuristic evaluation. It proves that their Markovian dynamics lack long memory statistically, defines long-memory network processes, and adds a polynomially weighted memory filter; the resulting models show advantages on long-term-dependence tasks.

  • Problem

    Long-range dependence remains difficult to model, and it is unclear whether RNN and LSTM possess long memory when assessed from a statistical perspective.

  • Method

    The paper proves a limitation of Markovian recurrent dynamics, introduces a long-memory network definition, and augments RNN and LSTM with a learnable historical-memory filter.

  • Results

    RNN and LSTM are shown not to have long memory statistically, while the modified MRNN and MLSTM models are reported to better handle long-range dependence.

  • Takeaways & Limitations

    Adding a memory filter converts RNN and LSTM into models capable of approximating fractional differencing effects while retaining neural-network flexibility.

  • Takeaways & Limitations

    MRNN and MLSTM with dynamic d are time-consuming, and the paper leaves model simplification and faster optimization for future work.

Abstract

from arXiv · show

The LSTM network was proposed to overcome the difficulty in learning long-term dependence, and has made significant advancements in applications. With its success and drawbacks in mind, this paper raises the question - do RNN and LSTM have long memory? We answer it partially by proving that RNN and LSTM do not have long memory from a statistical perspective. A new definition for long memory networks is further introduced, and it requires the model weights to decay at a polynomial rate. To verify our theory, we convert RNN and LSTM into long memory networks by making a minimal modification, and their superiority is illustrated in modeling long-term dependence of various datasets.

1. Introduction

The paper argues that recurrent networks face a fundamental difficulty with long-range dependence: statistically, Markovian RNN and LSTM dynamics lack long memory. It introduces a statistical definition and a learnable memory filter that modifies these networks to model long-memory effects.

  • Motivation: Long-range dependence remains difficult for recurrent networks despite advances such as LSTM and GRU designed to combat vanishing gradients.The paper situates its question in the broader challenge of learning dependencies over long time horizons.
  • Statistical perspective: Statistical analysis shows that recurrent network processes with Markovian updates, including RNN and LSTM, do not have long memory under mild conditions.The paper contrasts this formal conclusion with heuristic judgments based on task performance.
  • Statistical perspective: A new definition of long-memory network processes characterizes whether a network can extract long-term dependence from non-i.i.d. inputs.The definition addresses the difficulty of directly transferring classical statistical long-memory concepts to neural networks.
  • Implications: The theory implies that hidden states in Markovian recurrent networks cannot be assumed to store information stably, motivating access to more historical observations.The paper connects this implication to avoiding vanishing-gradient difficulties.
  • Network design: A learnable memory filter uses parameter d to assign dynamic weights to historical observations and feeds past information into hidden units.Integrated into RNN and LSTM, the filter yields MRNN and MLSTM, which can approximate fractional differencing while retaining neural-network flexibility.
  • Empirical verification: Numerical studies on several datasets illustrate advantages of the modified models in forecasting and modeling sequences with long-term dependence.The introduction presents experiments as empirical verification of the theoretical proposal.

2. Memory Property of Recurrent Networks

The paper formalizes long memory for recurrent network processes and analyzes when recurrent dynamics produce geometric ergodicity and short memory. Under these results, standard RNNs and LSTMs generally cannot model stable long-range dependence, motivating a coefficient-based network definition that extends to nonlinear networks.

  • Memory definitions: Statistical long memory is defined for stationary processes through autocovariance or spectral-density behavior, while geometric ergodicity implies exponentially decaying dependence and short memory.The paper uses geometric ergodicity as a route to establish short memory for recurrent processes.
  • Recurrent network processes: A recurrent network can be represented as a homogeneous Markov chain by augmenting its state with hidden variables and specifying a transition function.This representation supports the paper’s ergodicity-based analysis of RNN and LSTM processes.
  • RNN results: Under Assumption 1, linear recurrent processes are geometrically ergodic if and only if the transition matrix has spectral radius ρ(W) < 1, and therefore have short memory.The result is both sufficient and necessary for the linear recurrent model.
  • RNN results: Under the stated assumptions, RNNs commonly have short memory: weights bounded away from one suffice for linear or ReLU activations, while tanh or sigmoid activations always have short memory.The broader corollary covers bounded and continuous output and activation functions.
  • LSTM results: Under Assumption 1, the LSTM process is geometrically ergodic and has short memory, with the forget gate mainly affecting this memory property.The stated conditions include bounded output behavior and a contraction condition involving the forget-gate parameters.
  • Network long memory: Because exogenous inputs can themselves contain long-range dependence, the paper defines network long memory through slowly polynomially decaying input-output coefficients and extends the definition to nonlinear networks.The linearized network process is used for nonlinear networks, while coefficient decay characterizes extracted long-term dependence.

3. Long Memory Recurrent Networks

The paper adds a long-memory filter to RNN and LSTM architectures, producing MRNN and MLSTM models that use historical observations to model long-range dependence. Theorem 3 establishes the capability of MRNNF to handle long-range dependence, unlike RNN.

  • Long memory filter: A long-memory filter based on fractional integration is added to neural networks to model long-term dependence.The filter is described as a special attention mechanism with few memory parameters and a guaranteed memory-elongation effect when active.
  • 3.1. Memory-augmented RNN (MRNN): MRNN adds a parallel hidden unit m(t) that receives filtered inputs and models long-memory patterns alongside the traditional hidden unit h(t).The output combines the traditional and memory hidden units through learned weights.
  • Long memory filter: The memory filter uses parameter d to assign dynamic weights to historical observations of arbitrary lengths before feeding past information to hidden units.The memory parameter may vary over time, d(t), and the infinite filter summation is truncated at lag K in implementation.
  • 3.1. Memory-augmented RNN (MRNN): Theorem 3 states that MRNNF can handle long-range dependence data, whereas RNN cannot under the paper’s long-memory definition.MRNNF is the constant-memory-parameter version implemented by fixing Wd = 0.
  • 3.2. Memory-augmented LSTM (MLSTM): MLSTM revises LSTM cell states with a long-memory filter, treating past cell states as soft attention controlled by a few memory parameters.The modified cell renames the forget gate as the memory gate and uses the filtered cell-state update with parameter d.
  • 3.2. Memory-augmented LSTM (MLSTM): The cell state c(t) has long memory under Definition 3, while constant-gates-LSTM does not and constant-gates-MLSTM does.The paper notes that gating prevents both LSTM and MLSTM from being reasonably simplified to linear networks.

4. Experiments

Experiments evaluate the proposed models on forecasting and sentiment-analysis tasks using repeated random initializations and standard prediction metrics. MRNN variants generally outperform recurrent baselines on long-memory forecasting, while performance depends on model type and truncation lag K.

  • Experimental design: The experiments cover one-step forecasting on four long-memory datasets and one short-memory dataset, followed by two sentiment-analysis tasks.Forecasting uses RMSE, MAE, and MAPE, while the sentiment experiments include CMU-MOSI and Paper Reviews.
  • Experimental design: 100 random initializations are used for each model, with overall performance reported as the distribution of locally optimized results and best performance as its strongest result.This procedure addresses optimization non-convexity and distinguishes typical from best-case outcomes.
  • 4.1. Long Memory Datasets: MRNN and MRNNF have smaller average RMSE and smaller quantiles than other models in the reported overall forecasting comparison.MLSTM and MLSTMF show no obvious RMSE advantage over LSTM in this comparison, possibly because of training difficulty.
  • 4.1. Long Memory Datasets: MRNN is significantly better than RNN at the 5% level on all datasets and significantly better than LSTM on all datasets except DJI.The comparison uses two-sample t-tests on mean RMSE.
  • 4.1. Long Memory Datasets: MRNNF and MRNN achieve better best RMSE performance than the other models on ARFIMA, traffic, and tree datasets, while remaining competent on DJI.The statement concerns the best performance comparison rather than the overall distribution.
  • 4.2. Short Memory Dataset: On a synthetic short-memory RNN dataset, MRNN(F) and MLSTM(F) perform comparably to the true RNN model, with somewhat larger error variation.This supports the reported claim that the proposed models do not deteriorate on data without long-memory effects.
  • 4.3. Model Parameter K: MRNN and MRNNF generally improve with larger K and retain smaller average RMSE than baselines across the tested K values.By contrast, MLSTM and MLSTMF improve as K decreases and can outperform LSTM on ARFIMA and traffic when K = 25.
  • 4.4. Sentiment Analysis: On Paper Reviews, MLSTMF significantly improves all reported measures over LSTM, while MRNNF achieves the highest accuracy and MLSTMF leads the other metrics.The experiment combines the proposed component with a fixed LSTM second layer.

5. Conclusion

The paper concludes that standard RNN and LSTM processes lack long memory from a time-series perspective, while minimally modified versions better model long-term dependence. It also identifies computational cost and broader applicability as open issues.

  • RNN and LSTM are shown not to have long memory from a time-series perspective.
  • MRNN and MRNNF have advantages in forecasting time series with long-term dependency.
  • A combination of MLSTMF50 and LSTM layers significantly improves over a pure LSTM network on a paper reviews dataset.
  • Dynamic-d MRNN and MLSTM models are time-consuming compared with other models, motivating work on simplification and faster optimization.
  • Future work includes testing the memory filter on other recurrent or feed-forward sequence models and exploring alternative slow-decaying filters or filter banks.

A.1. Proof of Theorem 1

Theorem 1 establishes that recurrent processes with Markovian updates have short memory under suitable regularity conditions. The proof uses Markov-chain ergodicity to obtain geometrically decaying autocovariances.

  • The model is represented as a homogeneous Markov chain on a Borel state space with a transition probability.
  • Positive transition density makes the Markov chain irreducible under the stated noise assumption.
  • Under continuity and drift conditions, the process is geometrically ergodic and has a unique strictly stationary solution.
  • The theorem therefore concludes that the recurrent network process has short memory under mild conditions on its Markovian dynamics.
  • For a univariate stationary process, autocovariances satisfy γk ≤ γ0a^k for k ∈ N and some 0 < a < 1.This geometric bound makes the autocovariance function summable, implying short memory.

A.2. Proof of Theorem 2

Theorem 2 connects geometric ergodicity of the linear recurrent model with stability of its state-transition matrix. The proof applies irreducibility and drift arguments to establish the equivalence conditions.

  • Under the linear setting, the recurrent model is written as a Markov chain with transition matrix W and a corresponding transition probability.
  • When ρ(W) < 1, an integer s exists such that the s-step chain satisfies Tweedie’s drift criterion.
  • The s-step chain is Feller and irreducible, making the relevant bounded set small and yielding geometric ergodicity.
  • Geometric ergodicity provides a strictly stationary, nonanticipative solution to the model.
  • Under the converse argument, irreducibility and the Lyapunov result imply a strictly negative top Lyapunov exponent and spectral radius ρ(W) < 1.

A.3. Proof of Corollary 1

The corollary applies Theorem 1 to MRNN and LSTM-style processes under bounded activations and outputs. These conditions establish geometric ergodicity and short memory for the examined models.

  • Bounded activation and output functions let MRNN satisfy a linear drift bound with constants a ∈ (0, 1) and b.
  • Theorem 1 then implies that the MRNN model is geometrically ergodic and has short memory.
  • The LSTM example considers p = q = 1, l1 norms, sigmoid, ReLU, or tanh gates, and linear, sigmoid, or softmax outputs.
  • For the LSTM proof, the forget-gate norm is bounded by a0 ∈ (0, 1), supporting the required contraction inequality.
  • Under these conditions, the LSTM model is geometrically ergodic and has short memory.

A.6. Proof of Theorem 3

The proof shows exponential decay for ordinary RNN and constant-gates-LSTM coefficients, whereas modified recurrent models obtain polynomial decay and therefore long-memory capability under Definition 3.

  • Theorem 3 proof: RNN response coefficients decay exponentially across lags.The proof represents the RNN process through lag coefficients whose matrix entries decay exponentially.
  • Theorem 3 proof: MRNNF coefficients decay polynomially because the fractional differencing term dominates the exponentially decaying factors.The resulting entries of A_k decay at rate k^(-d_j-1).
  • Theorem 4: Theorem 4 concludes that constant-gates-MLSTM can handle long-range dependence, while constant-gates-LSTM cannot.This conclusion follows the paper’s Definition 3 of long-memory networks.
  • Constant-gates-LSTM: Constant-gates-LSTM coefficients decay exponentially, so this model cannot represent long-range dependence under the paper’s criterion.The proof explicitly derives exponential decay for the coefficients A_k.
  • Constant-gates-MLSTM: Constant-gates-MLSTM coefficients are governed by polynomially decaying W_k terms.The proof states that the coefficients Θ_k decay at rate k^(-d_j-1).

B.2. Overall Performance of the Models

The section reports one-step forecasting performance using RMSE, MAE, and MAPE, with averages, standard deviations, best values, and initialization boxplots across the datasets.

  • Overall metrics: One-step forecasting performance is reported using average RMSE and standard deviation, with additional MAE and MAPE results.The section also provides figures for the error distributions.
  • RMSE: RMSE boxplots summarize 100 initializations across the ARFIMA, DJI, traffic, and tree datasets.The corresponding figures report the distributions for these four datasets.
  • MAPE: MAPE boxplots summarize 100 initializations across the four datasets.Average MAPE and standard deviation are reported separately in Table 9.
  • Best performance: Best model performance in MAE and MAPE is reported in Tables 10 and 11.These tables complement the overall average-and-variation summaries.

B.4. Performance on a Dataset without Long Memory

On a synthetic dataset generated from a model without long memory, the proposed models perform comparably to the true RNN model, with somewhat greater error variation.

  • Synthetic dataset: The synthetic RNN dataset has length 4001 and is generated from a model identified as lacking long memory.Its construction uses segments of lengths 2000, 1200, and 800.
  • Results: The proposed models have performance comparable to the true RNN model on this dataset.The comparison is based on boxplots of error measures.
  • Results: The proposed models show somewhat larger variation in the error measures than the true RNN model.The passage qualifies this as a difference in variation rather than average performance.

B.5. Experiment on Parameter K

The parameter-K experiment compares 20 settings using error boxplots and finds opposite K trends for MRNN-family and MLSTM-family models.

  • Experimental setup: Twenty model settings are sorted by average RMSE from left to right in the parameter-K comparisons.K values are appended to proposed-model abbreviations, such as MRNN25 for K = 25.
  • MRNN and MRNNF: Larger K generally improves MRNN and MRNNF predictions, and both have lower average RMSE than every baseline for all tested K values.This pattern is reported across the parameter settings examined.
  • MLSTM and MLSTMF: Smaller K improves MLSTM and MLSTMF performance, with K = 25 outperforming LSTM on ARFIMA and traffic.The passage recommends larger K for MRNN-family models but further investigation for MLSTM-family models.
Loading 2006.03860v2…