Source-linked AI summary
Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model
Jaideep Pathak, Alexander Wikner, Rebeckah Fussell, Sarthak Chandra, Brian Hunt, Michelle Girvan, Edward Ott
TL;DR
Chaotic-system forecasts face a trade-off between imperfect mechanistic models and data-hungry machine-learning methods. The paper combines both through a hybrid reservoir-computing framework, which produced accurate predictions substantially longer than either component alone across low- and high-dimensional chaotic systems.
Problem
Knowledge-based models can be inaccurate, while machine-learning forecasts may require extensive data and computational resources.
Method
The paper combines a knowledge-based model with reservoir computing to form a hybrid forecasting scheme.
Results
The hybrid technique consistently outperformed both reservoir-only and knowledge-based prediction in forecast duration for the Lorenz system and spatiotemporal Kuramoto-Sivashinsky equations.
Takeaways & Limitations
The hybrid approach can improve forecasts even when both component predictors are individually inaccurate and can reduce computational demands relative to larger reservoir-only models.
Abstract
from arXiv · showhide
A model-based approach to forecasting chaotic dynamical systems utilizes knowledge of the physical processes governing the dynamics to build an approximate mathematical model of the system. In contrast, machine learning techniques have demonstrated promising results for forecasting chaotic systems purely from past time series measurements of system state variables (training data), without prior knowledge of the system dynamics. The motivation for this paper is the potential of machine learning for filling in the gaps in our underlying mechanistic knowledge that cause widely-used knowledge-based models to be inaccurate. Thus we here propose a general method that leverages the advantages of these two approaches by combining a knowledge-based model and a machine learning technique to build a hybrid forecasting scheme. Potential applications for such an approach are numerous (e.g., improving weather forecasting). We demonstrate and test the utility of this approach using a particular illustrative version of a machine learning known as reservoir computing, and we apply the resulting hybrid forecaster to a low-dimensional chaotic system, as well as to a high-dimensional spatiotemporal chaotic system. These tests yield extremely promising results in that our hybrid technique is able to accurately predict for a much longer period of time than either its machine-learning component or its model-based component alone.
I. INTRODUCTION
Knowledge-based and machine-learning predictors each address chaotic-system forecasting but face different limitations. The paper proposes combining them in a hybrid framework and illustrates its potential with reservoir computing.
- Knowledge-based prediction uses mechanistic understanding to construct a mathematical model, whereas machine learning learns from extensive past state measurements.
- Model-based forecasts can be inaccurate because of model errors, while data-driven methods may require prohibitive data and computational resources.
- The proposed framework combines physical-model information with measurement data to improve forecasting performance and implementation feasibility.
- The paper tests the general hybrid approach using reservoir computing, while noting relevance to other machine-learning implementations and physical reservoir computers.
- When both component predictors performed accurately for only a fraction of a Lyapunov time, the hybrid produced good predictions for about 4 Lyapunov times.
A. Knowledge-Based Model
The knowledge-based predictor advances an approximate mathematical model from estimated system states. Its accuracy is constrained by imperfections in that model, even when individual time steps are small.
- Knowledge-based prediction integrates an approximate model forward from an initial condition and possibly recent system-state values.
- The one-step-ahead predictor K is explicitly imperfect and may introduce substantial unwanted error.
- For continuous-time systems, the prediction step ∆t is small relative to the system’s change timescale, but forecasts require many such steps.
B. Reservoir-Only Model
The reservoir-only model learns a forecasting map from past state measurements using a high-dimensional dynamical reservoir. After training, it autonomously generates predictions, whose accuracy is time-limited for chaotic systems.
- The reservoir computer trains on system-state data from t = −T to t = 0 before making predictions for t > 0.
- Its reservoir has state dimension D_r much larger than the measured-state dimension M, with forecasting accuracy ideally increasing with D_r but computational cost limiting reservoir size.
- During training, the reservoir is driven by measured inputs and adjustable output parameters are selected by minimizing prediction error.
- During prediction, the reservoir evolves autonomously through a feedback loop, using its output as the forecast.
- Because chaotic trajectories diverge exponentially, reservoir-only forecasts can remain accurate only for a limited time.
C. Hybrid Scheme
The hybrid scheme feeds training data into both the knowledge-based predictor and reservoir, then combines their outputs through a trained output layer. In prediction, the combined system evolves autonomously.
- C. Hybrid Scheme: The hybrid approach combines the knowledge-based model with the reservoir-only model.
- C. Hybrid Scheme: During training, measured states are supplied to both predictors, while the reservoir also receives the knowledge-based one-step prediction.
- C. Hybrid Scheme: The output layer is trained using both model and reservoir inputs, effectively learning how to weight their information.
- C. Hybrid Scheme: During prediction, feedback is closed and the hybrid system evolves autonomously using its own previous hybrid predictions.
III. IMPLEMENTATION
The implementation uses reservoir computing without requiring knowledge of the reservoir dynamics, while allowing multiple physical or hardware realizations of the reservoir component.
- Reservoir training uses only the system’s time-series training data, not knowledge of the dynamics or details of the reservoir system.
- The illustrative reservoir implementation uses an artificial neural network, but nonlinear optical devices and Field Programmable Gate Arrays are also possible implementations.
A. Reservoir-Only and Hybrid Implementations
The reservoir-only and hybrid schemes update a nonlinear reservoir, map its state to predictions, and train output parameters by regularized regression; the hybrid additionally incorporates model-based information during training and prediction.
- Reservoir architecture: A large, low-degree Erdős-Rényi network of nonlinear units forms the reservoir, with adjacency matrix A scaled to spectral radius ρ.
- Reservoir architecture: The reservoir state is updated with a hyperbolic tangent of recurrent and input terms, while vector hyperbolic tangent acts componentwise.
- Output mapping: The output is W_out r⋆, where r⋆ retains r_j for odd j and uses r_j^2 for even j.
- Training: Training minimizes squared prediction error over the training interval and determines W_out through Tikhonov-regularized linear regression.
- Hybrid implementation: In the hybrid scheme, the reservoir input matrix has 2M columns, connecting reservoir nodes to raw inputs and the model-related input according to fraction γ.
- Hybrid implementation: During prediction, the hybrid replaces u(t) with the previous output and produces the predicted vector time series ũ_H(t).
B. Training Reusability
Prediction errors grow roughly exponentially because chaos, initial-condition error, and imperfect reproduction of the true dynamics jointly limit forecast duration.
- Prediction error grows roughly exponentially with time when chaos, small initial-condition error, and imperfect dynamics reproduction are present.
C. Assessments of Prediction Methods
The study compares knowledge-based, reservoir-only, and hybrid predictors using valid time, the duration before normalized error crosses a specified threshold, across repeated trials.
- Valid time t_v measures the elapsed time until normalized error E(t) first exceeds threshold f, with E(t_v) = f.
- The prediction symbol ũ(t) denotes the output of the knowledge-based, reservoir-based, or hybrid method being evaluated.
- The experiments use f = 0.4 and 20 disjoint intervals of length τ from a long true-system run.
- Reservoir-only and hybrid evaluations use 32 random realizations of A and W_in, with separately determined W_out parameters for each realization.
IV. LORENZ SYSTEM
The Lorenz-system experiments test hybrid prediction with an imperfect knowledge-based model and reservoir computing across reservoir sizes and model errors. Hybrid predictions remain accurate longer than either component alone and can retain this advantage with smaller reservoirs and varied model error.
- Experimental setup: The imperfect knowledge-based Lorenz model changes parameter b to b(1 + ϵ), with ϵ quantifying discrepancy from the true system.The true system uses a = 10, b = 28, and c = 8/3; the prediction trajectory uses ∆t = 0.1.
- Illustrative prediction: About 10 Lyapunov times of low prediction error are obtained in an illustrative hybrid trial with ϵ = 0.05 and Dr = 500.The valid time is marked when normalized error E(t) first reaches f = 0.4.
- Reservoir-size dependence: The hybrid scheme yields large prediction improvements over reservoir-only prediction for all tested reservoir sizes, despite the knowledge-based model performing very poorly alone.With Dr = 50, hybrid valid time is comparable to reservoir-only prediction with Dr = 500.
- Model-error dependence: The hybrid method is much better than both the imperfect knowledge-based and reservoir-only components across the tested range of model errors at Dr = 50.Figures 5 and 6 together suggest potential robustness of the hybrid approach.
- Reservoir-size dependence: Figure 5 compares median valid time versus reservoir size for hybrid, reservoir-only, and knowledge-based prediction at fixed ϵ = 0.05.The knowledge-based curve is horizontal because it does not depend on Dr, and error bars span the first-to-third quartile range.
V. KURAMOTO-SIVASHINSKY EQUATIONS
The hybrid scheme combines an inaccurate knowledge-based model with reservoir computing to forecast high-dimensional spatiotemporal chaos. In the Kuramoto–Sivashinsky system, it outperforms either component alone, including when both components perform poorly.
- The experiment tests hybrid forecasting on simulated data from the one-dimensional Kuramoto–Sivashinsky equation.
- The KS simulation uses periodic spatial boundaries, 64 grid points, and a positive maximum Lyapunov exponent, indicating chaotic dynamics.
- For ϵ = 0.01 and Dr = 8000, both standalone predictors are fairly accurate, while the hybrid prediction is evaluated alongside them.
- For ϵ = 0.1 and Dr = 500, both standalone predictors are relatively inaccurate, providing a test of hybridization under poor component performance.
- The hybrid scheme significantly outperforms both components when their valid prediction times are similar.
- ϵ = 1: even a totally ineffective model combined with a relatively small reservoir predicts for a significant duration, while smaller errors also benefit from hybridization.
VI. CONCLUSIONS
The paper concludes that hybridizing reservoir computing with knowledge-based prediction improves forecasting duration across low-dimensional and spatiotemporal chaotic systems. The approach remains useful when component models are seriously flawed and can reduce reservoir-size requirements.
- The proposed method hybridizes reservoir computing and knowledge-based prediction for chaotic dynamical systems.
- The hybrid technique consistently outperforms reservoir-only and knowledge-based prediction in forecast duration for both the Lorenz and Kuramoto–Sivashinsky systems.
- The hybrid technique robustly improves performance even when both standalone predictors are too flawed to make accurate predictions.
- With a significantly flawed knowledge-based model, small reservoirs can produce predictions comparable to much larger reservoir-only models, potentially saving computational resources.
- Both hybrid and reservoir-only models support training reusability, allowing repeated predictions after a short resynchronization run without retraining each time.