Source-linked AI summary
Stabilized Neural Ordinary Differential Equations for Long-Time Forecasting of Dynamical Systems
Alec J. Linot, Joshua W. Burby, Qi Tang, Prasanna Balaprakash, Michael D. Graham, Romit Maulik
TL;DR
The paper addresses high-wavenumber amplification that can push unreduced neural ODE trajectories off the Kuramoto–Sivashinsky attractor. It adds learned linear and nonlinear terms to the ODE right-hand side; tests on Burgers and Kuramoto–Sivashinsky equations report improved shock and spectrum capture, noise robustness, and attractor tracking.
Problem
Without dimension reduction, high wavenumbers can be amplified in neural ODEs, causing long-time Kuramoto–Sivashinsky trajectories to leave the attractor.
Source: §I · p. 5Method
The approach trains linear and nonlinear neural networks together to approximate an ODE’s right-hand side, adding an explicit linear term to the neural ODE.
Source: §I · p. 3, §II · p. 7Results
Stabilized neural ODEs capture Burgers shock dynamics and energy spectra, keep Kuramoto–Sivashinsky trajectories on the attractor, and are more robust to noisy initial conditions.
Source: §I · p. 6, A. Viscous Burgers Equation · p. 13, §IV · p. 22Takeaways & Limitations
The learned linear term provides a natural basis for reduced-order modeling.
Source: §I · p. 5, §IV · p. 22
Abstract
from arXiv · showhide
In data-driven modeling of spatiotemporal phenomena careful consideration often needs to be made in capturing the dynamics of the high wavenumbers. This problem becomes especially challenging when the system of interest exhibits shocks or chaotic dynamics. We present a data-driven modeling method that accurately captures shocks and chaotic dynamics by proposing a novel architecture, stabilized neural ordinary differential equation (ODE). In our proposed architecture, we learn the right-hand-side (RHS) of an ODE by adding the outputs of two NN together where one learns a linear term and the other a nonlinear term. Specifically, we implement this by training a sparse linear convolutional NN to learn the linear term and a dense fully-connected nonlinear NN to learn the nonlinear term. This is in contrast with the standard neural ODE which involves training only a single NN for learning the RHS. We apply this setup to the viscous Burgers equation, which exhibits shocked behavior, and show better short-time tracking and prediction of the energy spectrum at high wavenumbers than a standard neural ODE. We also find that the stabilized neural ODE models are much more robust to noisy initial conditions than the standard neural ODE approach. We also apply this method to chaotic trajectories of the Kuramoto-Sivashinsky equation. In this case, stabilized neural ODEs keep long-time trajectories on the attractor, and are highly robust to noisy initial conditions, while standard neural ODEs fail at achieving either of these results. We conclude by demonstrating how stabilizing neural ODEs provide a natural extension for use in reduced-order modeling by projecting the dynamics onto the eigenvectors of the learned linear term.
I. INTRODUCTION
Data-driven dynamical models can be vulnerable to perturbations and may generalize poorly beyond training data, while repeated forecasting can compound errors. This work adds an explicit linear term to neural ODEs to address high-wavenumber growth and improve long-time prediction, noise robustness, and reduced-order modeling.
I. INTRODUCTION
- Repeatedly feeding predictions back into a time-series model can compound errors as states move farther from the training data.Data-driven models may also lack robustness to perturbations and generalize poorly beyond their training data.Source: §I · pp. 2–3
- The proposed approach jointly learns linear and nonlinear terms, with reported gains in long-time prediction and noise robustness and a basis for reduced-order modeling.The paper evaluates shock behavior in the viscous Burgers equation and chaotic attractor dynamics in the Kuramoto-Sivashinsky equation.Source: §I · pp. 3, 5, 6
- Neural ODEs can use widely spaced data, unlike derivative-based approaches that need closely spaced data when time derivatives are unavailable.Other approaches discussed include discrete-time models, DMD, reservoir networks, and recurrent neural networks, each with distinct constraints or drawbacks.Source: §I · pp. 4–5
- Without dimension reduction, high wavenumbers can grow in neural ODE models of the Kuramoto-Sivashinsky equation, driving trajectories off the attractor.The paper seeks to address this by explicitly adding a linear term rather than using dimension reduction.Source: §I · p. 5
II. STABILIZED NEURAL ODE FRAMEWORK
The framework extends neural ODEs by learning a linear component alongside a nonlinear component. The learned linear operator also supplies eigenvectors for reduced-order modeling.
II. STABILIZED NEURAL ODE FRAMEWORK
- The stabilized neural ODE adds a linear term to the conventional neural ODE, motivated by the stabilizing effect of linear terms in many dynamical systems.The framework uses snapshots from autonomous ODEs, with the paper focusing on systems whose long-time solutions lie on attractors and inertial manifolds.Source: §II · p. 6
- The linear operator is constrained to be symmetric for the viscous Burgers equation but is unconstrained for the Kuramoto-Sivashinsky equation.The authors choose limited constraints to test whether the model can learn inertial-manifold behavior from data; explicitly enforcing a spectral-gap condition is described as future work.Source: §II · p. 7
- The RHS can be modeled with a nonlinear network alone, a nonlinear network plus a fixed linear term, or a nonlinear network plus a learned linear CNN.The learned linear term is represented by a CNN, while the nonlinear term is represented by a fully connected network.Source: §II · pp. 6–7
- The learned linear operator supplies eigenvectors that define resolved and unresolved projections for a reduced-order model.The paper applies Galerkin and nonlinear Galerkin approaches to the learned dynamics, without requiring the governing ODE to be known a priori.Source: §II · p. 8
- Galerkin modeling sets the unresolved component to zero, whereas nonlinear Galerkin assumes its time derivative is zero and approximates that component iteratively.The authors use one iteration; they report nonlinear Galerkin as more accurate but more computationally expensive than Galerkin.Source: §II · pp. 8–9
III. RESULTS
The results section evaluates the proposed models on the viscous Burgers and Kuramoto-Sivashinsky equations, focusing on shock behavior, chaotic attractor prediction, and reduced-order modeling.
III. RESULTS
- The viscous Burgers equation tests shock capture and energy-spectrum accuracy, while the Kuramoto-Sivashinsky equation tests whether long-time chaotic trajectories remain on the attractor.The section also demonstrates Galerkin and nonlinear Galerkin reduced-order modeling using eigenvectors of the learned linear term.Source: §III · p. 9
- All reported figures compare model reconstructions with test data not used to train the models.The neural ODE implementations use PyTorch with code modified from Chen et al. [5].Source: §III · p. 9
A. Viscous Burgers Equation
For the viscous Burgers equation, stabilized neural ODEs track the true trajectories and energy spectrum better than the nonlinear-only model, which develops persistent high-wavenumber behavior. The stabilized models also retain their performance with noisy initial conditions.
A. Viscous Burgers Equation
- The models are compared on VBE data with ν = 8 · 10−4, using 1000 random initial conditions for training and 100 for testing over [0, 5].The three approaches are nonlinear-only, fixed-linear plus nonlinear, and learned linear-CNN plus nonlinear; the CNN learns a sparse tridiagonal linear operator.Source: A. Viscous Burgers Equation · pp. 9–12
- The fixed-linear and CNN stabilized models agree well with true trajectories, while the nonlinear-only model develops persistent high-wavenumber behavior and poor predictions.Across initial conditions, the stabilized models’ relative errors level off or slightly decline at long times; the nonlinear model’s relative error continues increasing.Source: Figure 1 · p. 10, A. Viscous Burgers Equation · p. 12, Figure 2 · p. 12
- At t ≳ 2, both stabilized methods match the true energy spectrum across wavenumbers, while the nonlinear model increasingly deviates at high wavenumbers.At t = 1, all methods capture most of the spectrum, with errors at high wavenumbers.Source: Figure 3 · p. 13, A. Viscous Burgers Equation · p. 13
- With noisy initial conditions, stabilized models retain behavior similar to the no-noise case and capture the spectrum, whereas the nonlinear model’s error rapidly increases.The comparison adds Gaussian noise N(0, ϵ²) at each grid location for 100 initial conditions.Source: A. Viscous Burgers Equation · pp. 13–14, Figure 4 · p. 14
B. Kuramoto-Sivashinsky Equation
For the chaotic KSE, stabilized neural ODEs keep long-time trajectories on the attractor while the nonlinear model departs; the learned linear term also supports reduced-order modeling.
B. Kuramoto-Sivashinsky Equation
- The KSE test uses one trajectory sampled every 0.25 time units over [0, 10^5], with the first 80% for training and the remaining 20% for testing.The learned linear term is a width-5 convolutional filter; the true and learned filter parameters differ greatly in magnitude.Source: B. Kuramoto-Sivashinsky Equation · pp. 15–16
- Stabilized neural ODE trajectories stay on the KSE attractor at long times, while the nonlinear model develops erroneous high-wavenumber behavior and moves away.At short times, all models track well, with the fixed linear model tracking slightly better.Source: B. Kuramoto-Sivashinsky Equation · pp. 16, 18, Figure 6 · p. 17
- Over 10^4 time units, stabilized models match the true joint PDF of u_x and u_xx, while the nonlinear prediction’s probability of correct derivatives is nearly zero.The figure compares the true PDF with nonlinear and stabilized model predictions.Source: B. Kuramoto-Sivashinsky Equation · p. 18, Figure 8 · p. 18
- Even extreme Gaussian noise on Fourier modes 20–31 decays back to the attractor in the CNN model, which rapidly damps high wavenumbers.The CNN’s linear term is much smaller in magnitude than the true linear term.Source: B. Kuramoto-Sivashinsky Equation · p. 18, Figure 9 · p. 19
- Sorting eigenvectors by eigenvalue gives poor CNN projection matrices, whereas sorting by variance of each projected derivative makes CNN and true projections nearly identical.The learned ODE makes these projected derivatives straightforward to compute from data.Source: B. Kuramoto-Sivashinsky Equation · pp. 20–21, Figure 10 · p. 21
- At dp ∼18, all ROMs agree well with the full system; CNN Galerkin outperforms true Galerkin, while CNN nonlinear Galerkin is less accurate than true nonlinear Galerkin but better than standard Galerkin.The comparison uses KL divergence between true and ROM joint PDFs; above dp ∼18, further dimension increases make little difference.Source: B. Kuramoto-Sivashinsky Equation · pp. 20–22, Figure 11 · p. 22
IV. CONCLUSION
The stabilized neural ODE improves predictive capabilities across the tested Burgers and Kuramoto–Sivashinsky cases, and its linear-term eigenvectors support reduced-order modeling. Applying the method to more complex problems and determining whether additional stability constraints are needed remain future directions.
IV. CONCLUSION
- The stabilized neural ODE captures Burgers shock dynamics and energy spectra, generalizes to noisy initial conditions, and keeps Kuramoto–Sivashinsky trajectories on the attractor while remaining robust to noise.For the Kuramoto–Sivashinsky equation, properly sorted eigenvectors of the learned linear term also recreate the joint PDF of u_x and u_xx with far fewer dimensions than the full simulation.Source: §IV · p. 22
- With properly sorted eigenvectors of the learned linear term, the method recreates the joint PDF of u_x and u_xx using far fewer dimensions than the full simulation.This reduced-order modeling demonstration uses the Kuramoto–Sivashinsky equation.Source: §IV · p. 22
- Future work includes applying the method to more complex problems where an inertial manifold may be unknown and investigating whether explicit neural-network stability constraints are needed.For increasing complexity, additional constraints may be needed, such as prescribing the linear term’s eigenvalue spectrum or tuning the nonlinearity’s Lipschitz constant.Source: §IV · pp. 22–23