PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks

Pantelis R. Vlachas, Wonmin Byeon, Zhong Y. Wan, Themistoklis P. Sapsis, Petros Koumoutsakos

arXiv:1802.07486v5physics.comp-phcs.LGnlin.CD

TL;DR

Forecasting high-dimensional chaotic systems is hindered by their dimensionality and chaotic behavior. The paper develops data-driven LSTM models in reduced space, finding better short-term predictions than GPR and slightly superior hybrid LSTM-MSM performance over GPR-MSM.

  • Problem

    Forecasting complex natural systems remains limited by their high dimensionality and chaotic behavior.

  • Method

    The paper trains data-driven LSTM networks on reduced-order time series and combines them with a mean stochastic model to support long-term forecasting.

  • Results

    Across Lorenz 96, Kuramoto-Sivashinsky, and climate-model cases, LSTM improves short-term predictions, while hybrid LSTM-MSM slightly outperforms GPR-MSM.

  • Takeaways & Limitations

    LSTM-MSM captures steady-state statistical behavior while trading off some short-term forecasting accuracy.

  • Takeaways & Limitations

    LSTM prediction errors accumulate, and in more chaotic cases its forecasts diverge faster than GPR.

Abstract

from arXiv · show

We introduce a data-driven forecasting method for high-dimensional chaotic systems using long short-term memory (LSTM) recurrent neural networks. The proposed LSTM neural networks perform inference of high-dimensional dynamical systems in their reduced order space and are shown to be an effective set of nonlinear approximators of their attractor. We demonstrate the forecasting performance of the LSTM and compare it with Gaussian processes (GPs) in time series obtained from the Lorenz 96 system, the Kuramoto-Sivashinsky equation and a prototype climate model. The LSTM networks outperform the GPs in short-term forecasting accuracy in all applications considered. A hybrid architecture, extending the LSTM with a mean stochastic model (MSM-LSTM), is proposed to ensure convergence to the invariant measure. This novel hybrid method is fully data-driven and extends the forecasting capabilities of LSTM networks.

1 Introduction

The paper addresses forecasting challenges in high-dimensional chaotic systems by introducing data-driven LSTM methods that operate in reduced order space without requiring governing equations. It positions these methods against prior analogue, Gaussian-process, and neural-network approaches and evaluates them across complex chaotic systems.

  • Motivation: High-dimensional, multiscale dynamics in natural systems hinder understanding and forecasting despite advances in simulation, dimensionality reduction, and prediction methods.Examples include climate, ocean circulation, organisms, and cells.
  • Prior approaches: Analogue forecasting uses historical observations and nearest-state evolution, but sparse attractor regions create uncertainty that deterministic models cannot represent.Related approaches also face computational expense when they require solving an eigen-system proportional to training-set size.
  • Prior approaches: Earlier LSTM and other recurrent-neural-network approaches had been applied to chaotic systems, but LSTM methods had been used only on low-dimensional systems.The introduction also identifies MLPs, ESNs, and radial basis functions as related machine-learning approaches.
  • Contribution: The paper proposes LSTM methods that use recent histories of reduced-order states to forecast high-dimensional dynamics from time-series data without knowledge of governing equations.The method constructs an embedding space from time-delayed reduced-order variables and seeks a globally approximate forecasting rule.
  • Evaluation: The proposed LSTM and LSTM-MSM methods are benchmarked against GPR and GPR-MSM using comparison metrics across three complex chaotic systems.The paper introduces a blended LSTM-MSM technique and presents GPR, MSM, and hybrid GPR-MSM methods for comparison.

2 Long Short-Term Memory (LSTM) Recurrent Neural Networks

LSTM networks regularize recurrent neural-network training and address vanishing or exploding gradients through gated memory, enabling reduced-order forecasting from short histories. The method truncates temporal dependencies, tunes hidden-state dimension, and can be combined with an MSM to improve long-term attractor behavior.

  • 2 Long Short-Term Memory (LSTM) Recurrent Neural Networks: LSTM networks were introduced to regularize recurrent neural-network training and use gates to address vanishing or exploding gradients and capture long-term dependencies.RNNs pass information between consecutive temporal steps through recurrent loops and hidden states.
  • 2 Long Short-Term Memory (LSTM) Recurrent Neural Networks: The LSTM maps the current input and previous hidden and cell states through forget, input, and output gates to compute outputs and updated LSTM states.Its weight matrices and biases parameterize the cell, while sigmoid activation functions define the gates.
  • 2 Long Short-Term Memory (LSTM) Recurrent Neural Networks: In reduced-order space, the LSTM is trained on system time-series data to predict the state derivative from d consecutive reduced-order states.The derivative is approximated with first-order forward differences, producing T − d + 1 training samples from T states.
  • 2 Long Short-Term Memory (LSTM) Recurrent Neural Networks: The truncation layer d limits explicit loss-function dependencies, while hidden-state dimension h is grid-searched because small h reduces expressiveness and large h increases overfitting and training cost.Training uses truncated back-propagation through time, mini-batch Adam optimization, Xavier initialization, validation monitoring, and a maximum of 1000 epochs.
  • Mean Stochastic Model Hybrid: The MSM-LSTM hybrid uses LSTM inference near the training distribution and MSM forecasting elsewhere, combining local dynamical accuracy with forecasts that remain on the attractor.MSM models each state component with an Ornstein-Uhlenbeck process and is designed to capture global attractor statistics at low computational complexity.

3 Benchmark and Performance Measures

The LSTM prediction mechanism is benchmarked against MSM and GPR using forecasts averaged across independently sampled attractor initial conditions. Performance is assessed with ensemble-mean RMSE, attractor standard deviation, and mean ACC.

  • The proposed LSTM prediction mechanism is benchmarked against the Mean Stochastic Model (MSM) and Gaussian Process Regression (GPR).
  • For every application and performance measure, reported values average results from multiple initial conditions sampled independently and uniformly from the attractor.Ground-truth trajectories are generated by integrating the discretized reference equation from each initial condition and projecting states into reduced order space.
  • Ensembles contain Nen = 50 perturbed states, and forecasts track their mean while LSTM members evolve deterministically and GPR members evolve stochastically.The ensemble represents uncertainty in the initial system state through small-variance empirical Gaussian perturbations.
  • RMSE compares each predicted ensemble mean with the ground-truth trajectory at every time and reduced-order-state component, producing error curves over time.Attractor standard deviation provides a relative scale: when prediction error exceeds it, a trivial mean predictor performs better.
  • Mean ACC over initial conditions quantifies pattern correlation between predicted and ground-truth trajectories, ranging from −1.0 to 1.0.A perfect forecast has ACC = 1.0; the score uses energy-based mode weights and is widely used in meteorological forecasting.

4 Applications

The proposed forecasting method is evaluated on the Lorenz 96 system, Kuramoto-Sivashinsky equation, and a prototypical barotropic climate model. Across these applications, LSTM models provide strong short-term prediction, while hybrid LSTM-MSM schemes improve long-term stability.

  • The method is demonstrated on three chaotic systems spanning weakly chaotic to fully turbulent behavior: Lorenz 96, Kuramoto-Sivashinsky, and a prototypical barotropic climate model.
  • Lorenz 96: As Lorenz 96 forcing increases, predictability decreases, but LSTM models extend the 10%σ predictability horizon for all tested forcing regimes compared with GPR.The comparison uses F ∈ {6, 8, 16} over the first N = 10 time steps, with T = 0.1.
  • Lorenz 96: At T = 2 in Lorenz 96, GPR and LSTM diverge, whereas MSM and blended schemes remain close to the attractor over the long term.
  • Kuramoto-Sivashinsky equation: In the Kuramoto-Sivashinsky equation, LSTM remains above the ACC predictability threshold of 0.6 for approximately 2.5 at ν = 1/10 and 0.6 at ν = 1/16.Higher chaoticity makes accurate prediction more challenging, and blending with MSM is less imperative because LSTM diverges more slowly.
  • Barotropic climate model: For the barotropic climate model, LSTM reaches the attractor in 400–500 h, compared with 300–400 h for GPR-based methods and 1 h for MSM.The result concerns the four most energetic reduced-order variables and indicates that LSTM better captures nonlinear dynamics than GPR.

5 A Comment on Computational Cost of Prediction

The section contrasts prediction costs for GPR and LSTM models. GPR inference scales with training-set size, whereas LSTM inference is independent of that size and has lower complexity, enabling faster prediction.

  • Computational cost of prediction: GPR prediction costs O(N^2), where N is the number of training samples; in the previous section, N ≈2500.GPR carries the training dataset at each time step to identify input–dataset relations and infer outputs.
  • Computational cost of prediction: LSTM inference complexity is independent of training-set size because it adjusts parameters to reproduce local dynamics.This contrasts with GPR’s global-dynamics modeling through uniformly sampled attractor data.
  • Computational cost of prediction: LSTM inference complexity is roughly O(d_i · d · h + d · h^2), significantly smaller than GPR and therefore enabling faster prediction.Here, d_i is the dimension of each input, d is the number of inputs, and h is the number of hidden units.

6 Conclusions

The paper proposes a data-driven LSTM method for forecasting chaotic dynamical systems in reduced order space and evaluates it against GPR and MSM on three benchmark applications. LSTM improves short-term prediction, while hybrid LSTM-MSM restores invariant-measure behavior with a short-term accuracy tradeoff.

  • Method: The proposed LSTM models and predicts chaotic dynamical systems in reduced space using short-term history, without prior knowledge of governing equations.It predicts the reduced-order state derivative for one-step prediction and generates long-term forecasts iteratively from time-series training data.
  • Evaluation: The study compares LSTM with GPR and MSM on Lorenz 96, the Kuramoto-Sivashinsky equation, and a barotropic climate model using RMSE and ACC.The applications span weakly chaotic to fully turbulent regimes.
  • Results: In all cases, LSTM outperforms GPR for short-term prediction by capturing local dynamics and complex modal interactions more efficiently.Its prediction error accumulates during iterative forecasting, and divergence occurs faster than GPR as chaoticity increases.
  • Hybrid forecasting: Blending LSTM or GPR with MSM worsens short-term prediction but captures steady-state statistical behavior, with LSTM-MSM slightly outperforming GPR-MSM across the systems.The hybrid approach addresses the failure of standalone forecasts to converge to the invariant measure.
  • Application-specific findings: In the Kuramoto-Sivashinsky equation, lower intrinsic attractor dimensionality improves LSTM’s local-dynamics capture, while LSTM-MSM restores invariant-measure convergence in the chaotic regime 1/ν = 16.This restoration requires compromising short-term forecasting accuracy.
  • Future directions: Future work targets stochastic modeling of lower-energy modes and interpolation errors, plus mixtures of LSTM models for attractor regions with different dynamics.The current LSTM models the attractor globally, which may not represent all regions simultaneously.

7 Data Accessibility

The authors provide online access to the code and data used in the work, along with implementation requirements for reproducing the LSTM and Gaussian Process codes.

  • 7 Data Accessibility: The code and data used in the work are available through an online link.The provided address is https://polybox.ethz.ch/index.php/s/keH7PftvLmbkYU1, with password “rspa paper.”
  • 7 Data Accessibility: The LSTM architectures were implemented using the TensorFlow library and python3, while Gaussian Processes were implemented in Matlab.These packages must be installed to run the codes.

8 Authors’ Contributions

The authors divided responsibilities across conception, implementation, supervision, interpretation, consultation, and manuscript preparation. Contributions covered the blended LSTM-MSM scheme, LSTM and GPR implementations, simulations, and analysis.

  • PRV conceived the blended LSTM-MSM scheme, implemented the neural network architectures and simulations, interpreted computational results, and wrote the manuscript.
  • WB supervised the work and contributed to implementing the LSTM, while ZYW implemented the GPR and contributed to the manuscript.
  • PK originated the LSTM scheme and contributed to the manuscript; PK and TPS interpreted results and provided consultation.

10 Funding · A Long short-term memory (LSTM)

The paper acknowledges support for TPS and ZYW from three U.S. research agencies and for PK and PRV from an ERC Advanced Investigator Award.

  • 10 Funding: TPS and ZYW were supported by an Air Force Office of Scientific Research grant FA9550-16-1-0231.
  • 10 Funding: TPS and ZYW were supported by an Office of Naval Research grant N00014-15-1-2381.
  • 10 Funding: TPS and ZYW were supported by an Army Research Office grant 66710-EG-YIP.
  • 10 Funding: PK and PRV acknowledge support from the European Research Council Advanced Investigator Award.
  • 10 Funding: The European Research Council support is identified as an Advanced Investigator Award.
  • 10 Funding: The European Research Council award number is 341117.

A.1 Training and inference … C.1 Dimensionality reduction

The paper trains LSTM networks on split dynamical-system time series using mini-batch Adam optimization and validation-based stopping, with weighted losses for reduced-order outputs. It compares stateless and stateful recurrent architectures, constructs reduced spaces from energetic modes, and examines how Kuramoto–Sivashinsky chaoticity affects dimensionality reduction.

  • A.1 Training and inference: Time-series data are divided into training and validation sets with N_train/N = 0.8 before forming LSTM input-output samples.
  • A.1 Training and inference: Mini-batches reduce noisy gradient estimates and slow convergence associated with using a single sample for loss-gradient estimation.
  • A.1 Training and inference: Adam optimization updates network weights using previously computed gradients and moments, while validation loss monitors generalization and halts training when validation error stops decreasing.
  • A.2 Weighting the loss function: Loss weighting assigns different importance to multidimensional derivative outputs according to the significance of each reduced-order component, such as its physical-system energy.
  • A.3 LSTM architecture: Stateless RNNs initialize hidden states at the truncation layer to zero, so predictions use only the short-term history of d previous inputs.
  • A.3 LSTM architecture: Stateful RNNs initialize hidden states through teacher forcing from longer history, but this introduces initialization error and did not improve prediction accuracy in the considered chaotic systems.
  • B Lorenz 96: The Lorenz 96 reduced-order phase space is constructed using the most energetic Fourier modes for forcing regimes F ∈ {4, 6, 8, 16}.
  • C.1 Dimensionality reduction: As ν declines in the Kuramoto–Sivashinsky equation, chaoticity rises and more modes are needed to capture most of the system’s energy.

C.2 Results · D Barotropic model

The merged sections report hybrid LSTM-MSM behavior for the Kuramoto-Sivashinsky system and describe dimensionality reduction of the 231-dimensional Barotropic climate model. The Barotropic reduction uses generalized multidimensional scaling, kinetic-energy-preserving EOFs, and projection onto the most energetic modes.

  • C.2 Results: For hybrid LSTM-MSM, the LSTM-member ratio drops more slowly in the long run for 1/ν = 10 because the attractor has smaller intrinsic dimensionality and slower trajectory divergence.Initially, the ratio is higher for 1/ν = 16 because stronger MSM damping drives boundary-near initial conditions toward the attractor faster.
  • C.2 Results: For the Kuramoto-Sivashinsky system at T = 1.5, Figure 14 compares the LSTM-MSM and GPR-MSM ensemble-member ratios over time for 1/ν = 10 and 1/ν = 16.The plotted means are based on 1000 initial conditions.
  • D Barotropic model: The Barotropic climate model is reduced from its original problem dimension of 231 using generalized classical multidimensional scaling, followed by EOF construction and projection of the dynamics.The EOFs form an orthogonal basis of the reduced order space.
  • D Barotropic model: Generalized multidimensional scaling seeks a lower-dimensional embedding that preserves pairwise inner products between the dataset points and their reduced representations.Its optimization minimizes total squared error between pairwise products.
  • D Barotropic model: When both inner products are scalar products, the multidimensional-scaling solution is equivalent to PCA; when only the embedding product is scalar, it also has an analytic solution.The construction uses the Gram-matrix coefficients and eigenvalues sorted by magnitude.
  • D Barotropic model: The optimization is equivalent to finding the best rank-d approximation in the Frobenius norm, with optimality established by the Eckart-Young-Mirsky theorem.The Barotropic reduction uses the standard kinetic energy product to preserve nonlinear symmetries of the system dynamics.
  • D Barotropic model: Because the multidimensional-scaling solution is optimal only for the training points, EOFs provide a practical orthogonal basis for embedding new states.The EOFs are sorted in descending order by energy, and the first four are shown in Figure 15.
  • D Barotropic model: The first EOF represents an Arctic center of action surrounded by a zonally symmetric mid-latitude structure, resembling the Arctic Oscillation or Northern Hemisphere Annular Mode.EOF analysis identifies individual realistic climatic modes such as teleconnections.

E Sensitivity to noise in the data … E.3 Barotropic model

The section evaluates robustness to Gaussian noise added to training data, with noise scaled by each system’s attractor standard deviation. Across Lorenz 96, Kuramoto–Sivashinsky, and Barotropic-model experiments, low noise is generally tolerated, while the highest tested noise can degrade performance.

  • E Sensitivity to noise in the data: Training data are perturbed with Gaussian noise whose standard deviation is σnoise = k σattractor, with σattractor computed from reduced-order training samples.Different noise levels are used to assess robustness.
  • E.1 Lorenz 96: For Lorenz 96, noise sensitivity is evaluated using short-term RMSE, long-term RMSE at T = 2, and ACC across forcing regimes F = 4, 8, 16.Figure 16 reports averages over 1000 initial conditions.
  • E.2 Kuramoto-Sivashinsky equation: For the Kuramoto–Sivashinsky equation at 1/ν = 10, predictability is robust against k ∈ {0.001, 0.01}, with error evolution changing only slightly.The experiment also considers the more chaotic regime 1/ν = 16.
  • E.3 Barotropic model: For the Barotropic model, prediction performance deteriorates only at the highest tested noise level, k = 0.2.RMSE evolution is evaluated for the four most energetic EOFs, averaging over 500 initial conditions.
  • E.3 Barotropic model: At low noise, k = 0.001 can improve Barotropic-model prediction performance, possibly because noise regularizes back-propagation during stochastic training.The passage presents adding input noise as a practical neural-network technique.

F Trajectory examples

This section presents LSTM-predicted KS trajectories against ground truth in reduced-order and reconstructed original spaces. It also examines forecasting without dimensionality reduction for a lower-Lyapunov-exponent chaotic system with complete state information.

  • Trajectory comparisons: The trajectory examples use the Kuramoto-Sivashinsky equation as the reference system and compare predicted reduced-order states with ground truth.Equivalent original-space trajectories are obtained by setting unmodeled PCA modes to zero before projection.
  • Reduced-order forecasting: For ν = 1/10 and ν = 1/16, the LSTM models use h = 100 hidden units and back-propagation horizon d = 50.Figure 20 illustrates ν = 1/10 predictions, true reduced-order evolution, RMSE, and corresponding original-space errors.
  • Full-state forecasting: Without dimensionality reduction, the method is tested on a chaotic system with a lower Lyapunov exponent, complete state information, and less prominent chaotic effects.The passage states that this setting is easier to capture than the main-paper applications because no state information is missing.
  • Reduced-order forecasting: Figure 21 presents predicted and true reduced-order KS evolution for ν = 1/16, with RMSE and original-space trajectories obtained by projection with unmodeled modes set to zero.The figure uses the same reduced-order-to-original-space comparison described for the trajectory examples.
  • Full-state forecasting: For the KS equation with ν = 1, Figure 22 shows true and predicted state evolution together with RMSE for three different initial conditions.The figure summarizes trajectories and associated errors across the three initial conditions.
Loading 1802.07486v5…