PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-free prediction of spatiotemporal dynamical systems with recurrent neural networks: Role of network spectral radius

Junjie Jiang, Ying-Cheng Lai

arXiv:1910.04426v1cs.LGnlin.CDphysics.data-anstat.ML

TL;DR

Choosing fixed neural-network parameters is a persistent challenge in machine learning, including reservoir computing for nonlinear-system prediction. The paper examines spectral radius in reservoir networks and finds an interval where prediction error is minimized across tested systems. The interval’s location and size depend on the target system, while the study varies edge weights with topology fixed.

  • Problem

    General guidance for choosing fixed neural-network parameters is lacking, leaving selection commonly to trial and error.

    Source: §I · p. 1

  • Method

    The study examines whether varying reservoir-network spectral radius can identify a minimum-error interval while holding topology fixed and varying edge weights.

    Source: §VI · p. 11

  • Results

    ρ ∈ [0.8, 1.6] gives the minimum-error interval in the NLSE random-topology case, while the valley phenomenon also holds for the 1D CGLE.

    Source: A. Nonlinear Schr¨odinger equation · p. 5, C. Predicting spatiotemporal chaotic… · p. 9

  • Takeaways & Limitations

    The location and size of the optimal spectral-radius interval depend on the target system being predicted.

    Source: §VI · p. 11

    The exploration fixes the network connection topology and varies only link weights.

    Source: §VI · p. 11

Abstract

from arXiv · show

A common difficulty in applications of machine learning is the lack of any general principle for guiding the choices of key parameters of the underlying neural network. Focusing on a class of recurrent neural networks - reservoir computing systems that have recently been exploited for model-free prediction of nonlinear dynamical systems, we uncover a surprising phenomenon: the emergence of an interval in the spectral radius of the neural network in which the prediction error is minimized. In a three-dimensional representation of the error versus time and spectral radius, the interval corresponds to the bottom region of a "valley." Such a valley arises for a variety of spatiotemporal dynamical systems described by nonlinear partial differential equations, regardless of the structure and the edge-weight distribution of the underlying reservoir network. We also find that, while the particular location and size of the valley would depend on the details of the target system to be predicted, the interval tends to be larger for undirected than for directed networks. The valley phenomenon can be beneficial to the design of optimal reservoir computing, representing a small step forward in understanding these machine-learning systems.

I. INTRODUCTION

Reservoir computing offers a model-free, data-based approach to predicting nonlinear dynamical systems, but choosing its fixed parameters lacks general guidance. This paper identifies spectral radius as a useful parameter for prediction, with an interval where error is minimized across several spatiotemporal systems.

I. INTRODUCTION

  • Reservoir computing uses a fixed high-dimensional neural reservoir and trains its output layer to predict nonlinear dynamical systems from data.The trained system predicts by evolving without external input.Source: §I · p. 1
  • Choosing fixed, pre-defined machine-learning parameters remains an outstanding problem, commonly addressed through random trial and error.The paper distinguishes these fixed parameters from tunable learning parameters.Source: §I · p. 1
  • The study fixes reservoir topology and varies edge weights to make the search over network properties feasible.The authors note that topology, average degree, network size, and edge weights otherwise create a vast parameter space.Source: §I · p. 1
  • A finite spectral-radius interval minimizes prediction error across NLSE, KSE, and CGLE systems, forming a valley with a near-zero-error bottom.The authors present the interval as a potential guide to selecting reservoir parameters.Source: §I · p. 2

II. RESERVOIR COMPUTING

The paper describes echo state networks as a reservoir-computing approach in which a fixed, high-dimensional neural network transforms input data and a trained output layer produces predictions. Training fits that readout to system data; prediction then feeds the output back into the reservoir without external input.

II. RESERVOIR COMPUTING

  • Echo state networks use a driven neural reservoir whose nonlinear responses are linearly combined by trainable parameters; liquid state machines additionally receive signals from other neurons.The authors focus on ESNs for simplicity.Source: §II · p. 2
  • An ESN maps low-dimensional input through WIR into an N-neuron reservoir characterized by Wres, then maps its state through WRO to the output.The input and output dimensions are both much smaller than the reservoir dimension.Source: §II · p. 2, Figure 1 · p. 2
  • For a chosen spectral radius ρ, reservoir edge weights are sampled and rescaled so the largest absolute eigenvalue of Wres equals ρ.The reservoir is typically a large, sparse, directed or undirected random network.Source: §II · p. 2
  • The reservoir state updates from its previous state and input; after transient steps, the authors square even elements to form the normalized state used for training.The activation function f is typically tanh, and the normalized state is used to determine WRO.Source: §II · p. 3
  • Training fits WRO to map reservoir states to next-step outputs using a regularized regression loss that penalizes large output weights.The output matrix is calculated by the regression scheme, with Γ as a positive regularization constant to prevent over-fitting.Source: §II · p. 3
  • Prediction can begin from the final training state or a zero state driven briefly by true system data, then feeds each output back as the next input.This feedback allows the reservoir to generate a continuous predicted time series.Source: §II · p. 3
  • Earlier work demonstrated that properly parameterized ESNs can predict the spatially extended chaotic KSE across its domain for several Lyapunov times.The cited prior result concerns the KSE in its chaotic regime.Source: §II · p. 3

III. FUNDAMENTAL ROLE OF SPECTRAL RADIUS OF RESERVOIR NETWORK IN PREDICTING SPATIOTEMPORAL DYNAMICAL SYSTEMS

The paper identifies reservoir-network spectral radius as fundamental to successful prediction and supports this finding with three spatiotemporal systems from physics.

III. FUNDAMENTAL ROLE OF SPECTRAL RADIUS OF RESERVOIR NETWORK IN PREDICTING SPATIOTEMPORAL DYNAMICAL SYSTEMS

  • The reservoir network’s spectral radius plays a fundamental role in successful prediction.The section frames spectral radius as a property relevant to prediction performance.Source: §III · p. 3
  • The authors substantiate the finding using the NLSE, KSE, and CGLE.These are spatiotemporal dynamical systems arising from physics.Source: §III · p. 3
  • The study connects spectral-radius choice to achieving successful prediction in the tested physical systems.Source: §III · p. 3

A. Nonlinear Schr¨odinger equation

For several NLSE wave patterns, reservoir-computing prediction error is minimized within a spectral-radius interval, but the interval and reliability vary by target. Akhmediev breathers and Kuznetsov-Ma solitons are predictable across valley realizations, whereas collisions are less reliable.

A. Nonlinear Schr¨odinger equation

  • The tested NLSE targets include analytic Akhmediev breathers, Kuznetsov-Ma solitons, and collisions formed by nonlinear superposition of two first-order breathers.The paper investigates reservoir-computing prediction of finite-background soliton solutions.Source: A. Nonlinear Schr¨odinger equation · pp. 3–4
  • A reservoir predicted 11 Akhmediev breathers over 1600 time steps, with approximately five solitons occurring in time predicted at relatively small error.The training phase contained approximately 56 solitons, and prediction used a warm start.Source: A. Nonlinear Schr¨odinger equation · p. 5
  • For Akhmediev breathers, ⟨RMSE⟩ is minimized across ρ ∈ [0.8, 1.6], with near-zero error and standard deviation at ρ = 1.4.Outside the valley, error is substantially larger; it rises approximately linearly above ρ ≈ 1.6.Source: Figure 2 · p. 4, A. Nonlinear Schr¨odinger equation · p. 5
  • For Kuznetsov-Ma solitons, the valley is ρ ∈ [1.98, 2.34], wider than the Akhmediev interval and at a different location.All tested reservoir realizations have near-zero errors when ρ is chosen from this valley.Source: A. Nonlinear Schr¨odinger equation · pp. 5–6, Figure 3 · p. 5
  • For soliton collisions, only about half of the realizations predict acceptably within the valley; outside it, nearly all have large errors.Bouncing solitons preserve the dynamical-state climate, while merging changes it after training on two solitons.Source: Figure 3 · p. 5, A. Nonlinear Schr¨odinger equation · p. 6

B. Predicting spatiotemporal chaotic solutions of Kuramoto-Sivashinsky equation

Reservoir computing also exhibits a spectral-radius valley for chaotic Kuramoto-Sivashinsky solutions. The valley coexists with a short prediction horizon characteristic of the tested chaotic solution.

B. Predicting spatiotemporal chaotic solutions of Kuramoto-Sivashinsky equation

C. Predicting spatiotemporal chaotic solutions of complex Ginzburg-Landau equation

For the one-dimensional CGLE, the reservoir predicts chaotic evolution over about four Lyapunov times at ρ = 0.1, and prediction error is minimized across a valley interval of spectral radii. The interval persists despite differing error behavior across tested ρ values.

C. Predicting spatiotemporal chaotic solutions of complex Ginzburg-Landau equation

  • The CGLE is used to model phenomena including nonlinear waves, chemical reactions, superconductivity, superfluidity, Bose-Einstein condensation, and liquid crystals.The studied parameter region is defect chaos, with α = 2 and β = −2.Source: C. Predicting spatiotemporal chaotic… · pp. 6–7
  • The reservoir uses separate input streams for the real and imaginary parts of the CGLE field; using only its magnitude yields no meaningful prediction.The field is sampled from numerical CGLE solutions.Source: C. Predicting spatiotemporal chaotic… · p. 7
  • The valley also appears when predicting the complex solution’s magnitude, with ensemble RMSE evaluated over 100 random reservoirs for each ρ.The CGLE has maximum Lyapunov exponent Λm ≈ 0.22; Figure 5 displays real and imaginary parts as well as the magnitude.Source: Figure 5 · p. 8
  • 0.1 ≲ ρ ≲ 2.5 is the CGLE valley interval; outside it, prediction errors rise or useful prediction time falls.At ρ = 0.1, prediction succeeds for about four Lyapunov times; errors are large at ρ = 3.9811×10−4 and ρ = 3.8.Source: C. Predicting spatiotemporal chaotic… · pp. 8–9

IV. EFFECT OF RESERVOIR NETWORK STRUCTURE ON PREDICTION

The optimal spectral-radius valley persists across directed and undirected reservoir networks and several tested dynamical systems. Network structure affects the valley’s location and width, with undirected networks tending to produce wider intervals.

IV. EFFECT OF RESERVOIR NETWORK STRUCTURE ON PREDICTION

  • The optimal spectral-radius region persists for directed and undirected networks across tested NLSE, KSE, and CGLE patterns.Tests of undirected networks included random and small-world topologies for Akhmediev breathers in the NLSE.Source: §IV · p. 9
  • Undirected networks yield a wider valley interval than directed networks, while random networks have a slightly wider interval than small-world networks.Source: §IV · p. 9
  • Network structure affects both the valley interval’s size and the spectral radius that achieves the absolute minimum prediction error.Source: §IV · p. 9

V. ERROR IN TRAINING OUTPUT DATA

The authors examine time-averaged training error as a possible clue to prediction error. Small training errors occur over an interval, but the error-growth behavior on either side is not analytically understood.

V. ERROR IN TRAINING OUTPUT DATA

  • A small-error interval appears in training for both directed and undirected networks, resembling the prediction valley.Figure 7 varies spectral radius by adjusting link weights while keeping network structure fixed.Source: Figure 7 · p. 10
  • For the directed KSE network, time-averaged training error varies non-monotonically with spectral radius, and its minimum lies within the prediction valley.As ρ increases away from the minimum, training error rises alongside a shorter prediction horizon.Source: §V · pp. 9–10
  • The training-error behavior as ρ decreases from its optimum does not explain the sharp rise in prediction RMSE, and error growth on either side remains analytically unexplained.Source: §V · p. 10
  • The smallest average single-step prediction error for KSE is about 6 × 10−5, and the error grows to about 0.5 in five Lyapunov times.The authors state that prediction time is mainly determined by single-step reservoir prediction error; parameter fine-tuning can reduce it incrementally, with no general method then available to reduce it drastically.Source: §V · p. 10

VI. DISCUSSION

The discussion identifies the spectral radius as a useful design parameter for model-free reservoir prediction, where general rules for preset network parameters are lacking. Numerical results show a target-dependent low-error interval across several systems and network structures, but its mechanism remains analytically unexplained.

VI. DISCUSSION

  • No general rules guide preset machine-learning parameters, leaving their selection largely to random trial and error.The paper frames systematic theory for choosing these free parameters as a major challenge.Source: §VI · pp. 10–11
  • Across NLSE, KSE, and CGLE systems, a spectral-radius interval yields minimum prediction error across tested reservoir structures, with wider intervals tending to occur for undirected networks.The interval’s location and size depend on the target system; the finding is numerical rather than analytic.Source: §VI · p. 11
  • The authors propose a heuristic balance: too-small spectral radius may leave reservoir dynamics incoherent, while too-large values may make them overly coherent.They suggest an optimal match to the target’s dynamical complexity, but state that the mechanism is not analytically understood.Source: §VI · p. 11
  • The interval’s location and size depend on the target system, and results for NLSE and CGLE do not support treating spectral radius as a simple proxy for input-memory length.The discussion notes that NLSE patterns of similar regularity can have different intervals, while the CGLE interval spans values near zero to well above one.Source: §VI · p. 11

Appendix A: Standard deviation of prediction error

Across several tested dynamical states, prediction-error variability is smaller within the spectral-radius valley than outside it, though soliton-collision cases can retain large variability. The appendix also gives an example of accurate long-term Akhmediev-breather prediction.

Appendix A: Standard deviation of prediction error

  • Standard deviation is small in the valley for Akhmediev breathers and Kuznetsov-Ma solitons, but is large at every tested spectral radius for two soliton-collision cases.The two collision cases differ in their NLSE parameters; the reported all-values result is stated for the cases in Fig. 8(c,d).Source: Appendix A · pp. 11–12
  • Standard deviation of ensemble-averaged error is markedly smaller in the valley than outside it for spatiotemporally chaotic KSE and 1D CGLE states.The results are shown in Fig. 9.Source: Appendix A · p. 12

Appendix B: Example of long term prediction of Akhmediev breathers in NLSE

  • For a properly chosen spectral radius, reservoir computing can accurately predict the NLSE Akhmediev-breather state over the long term.An example is shown in Fig. 10.Source: Appendix B · p. 12

Appendix C: Origin of standard deviation in the ensemble-averaged prediction error

The valley is defined using mean prediction error over 100 network realizations, but individual realizations can vary substantially. Error spread tends to be lowest near the optimal spectral radius, and valley boundaries are approximate rather than sharp.

Appendix C: Origin of standard deviation in the ensemble-averaged prediction error

  • The valley interval is defined from the ensemble-averaged error across 100 reservoir-network realizations at each fixed spectral radius.Individual prediction errors can still vary substantially even when the spectral radius lies inside the valley.Source: Appendix C · p. 12
  • At ρ = 1.4, nearly all Akhmediev-breather realizations have small errors, while at ρ = 0.8 some already show large errors as the value departs from the optimum.Outside the valley, almost all realizations have large errors; the standard deviation is minimized at the optimal value and tends to increase away from it.Source: Appendix C · p. 12
  • Valley boundaries are approximate: neither ensemble-averaged error nor its standard deviation changes abruptly enough to define sharp edges.The appendix also shows individual-realization variations for NLSE soliton collisions and a chaotic 1D CGLE state.Source: Appendix C · p. 12

Appendix D: Solution of soliton collision in NLSE

Appendix D supplies the complete NLSE soliton-collision solution, alongside supplementary figures describing prediction-error variability and long-term prediction examples. The figures cover multiple NLSE states and spatiotemporal systems.

Appendix D: Solution of soliton collision in NLSE

  • The complete NLSE soliton-collision solution is given, with definitions for parameters including l1, l2, κ1, χ1, χ2, and the complex-conjugation notation.The cited solution is attributed to.Source: Appendix D · p. 12
  • For four NLSE dynamical states, Fig. 8 plots standard deviation of ensemble-averaged prediction error against time and spectral radius using 100 random network realizations per fixed ρ.The states include Akhmediev breathers, Kuznetsov-Ma solitons, and two soliton-collision cases.Source: Figure 8 · p. 13
  • Fig. 9 shows standard-deviation results for the KSE and 1D CGLE, while Figs. 11–13 show realization-level RMSE evolution for Akhmediev breathers, soliton collisions, and chaotic 1D CGLE states.Fig. 11 identifies low errors for most or all realizations inside the valley and large errors for almost all realizations outside it.Source: Figure 9 · p. 13, Figure 11 · p. 13, Figure 12 · p. 14
  • Fig. 10 compares the true Akhmediev-breather evolution, the reservoir prediction, and their instantaneous-state difference.The accompanying text describes the example as an accurate long-term prediction for a properly chosen spectral radius.Source: Figure 10 · p. 13
Loading 1910.04426v1…