Source-linked AI summary
Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
Samuel H. Rudy, J. Nathan Kutz, Steven L. Brunton
TL;DR
Data-driven dynamical-system modeling must handle limited, noisy measurements without relying on prior denoising or vast datasets. The paper jointly learns measurement noise and dynamics using a neural-network vector field embedded in a Runge-Kutta time-stepper, producing predictive models on increasingly complex test problems while highlighting generalization risks.
Problem
Automated dynamical-system modeling is limited by measurement noise and scarce data, while complex systems can make first-principles derivations intractable.
Method
The method jointly estimates observation-level measurement noise and dynamics by embedding a neural-network vector field in a Runge-Kutta discrete map.
Results
The framework forms predictive models from limited, noisy measurements across dynamical systems of increasing complexity, including substantial measurement error.
Takeaways & Limitations
Explicit noise modeling and constrained time-stepping separate measurement error from the underlying state while enabling learning with unevenly spaced data.
Takeaways & Limitations
Models learned near bounded attractors may not generalize to new initial conditions or accurately represent dynamics away from the attractor.
Abstract
from arXiv · showhide
A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples.
1 Introduction
Data-driven dynamical-system modeling is challenged by complex governing equations and limited or noisy measurements. The proposed framework jointly learns dynamics and measurement noise while using structured time-stepping to improve robustness.
- Motivation: Limited or noisy measurements compromise automated methods for developing dynamical-system models when first-principles derivations are difficult.These difficulties arise when mechanisms are poorly understood, systems are complex, or measurements are corrupted by noise.
- Related work: Interpretable nonlinear methods can identify governing equations but depend on candidate-function libraries that struggle with complex dynamics.Their flexibility is limited by the available library of functions and the number of modeled degrees of freedom.
- Related work: Deep learning offers flexible function approximation for increasingly complex systems, given sufficient training data of adequate variety and quality.Neural networks are therefore attractive for modeling dynamics beyond the functional forms easily represented by sparse regression.
- Contribution: Earlier structured neural autoregressive models interpolate vector fields with multistep or Runge-Kutta methods but do not explicitly account for measurement noise.The proposed work extends this framework by learning measurement noise jointly with the dynamics rather than through sequential or alternating optimization.
- Contribution: The paper introduces a paradigm that accounts for measurement noise explicitly in the map between successive observations.This treats measurement noise and governing dynamics as unknowns to identify together.
Nomenclature
The nomenclature defines the state, observations, measurement errors, vector field, neural-network approximations, error measures, and figure color codes used throughout the paper. It also situates the paper's computational framework and application sections.
- Nomenclature: At timestep t_j, x_j denotes the state, ν_j the measurement error, and y_j the measurement.The state dimension is n, the number of timesteps is m, and X is the state matrix in R^n×m.
- Nomenclature: f denotes the true underlying vector field, while ˆf_θ denotes its data-driven neural-network approximation.θ parameterizes the neural network, and ˆF_θ denotes the data-driven approximation of the discrete dynamics map.
- Computational framework: The method simultaneously estimates measurement noise ˆN and discrete dynamics ˆF_θ by optimizing a prediction-fidelity cost function.The discrete map is modeled by a Runge-Kutta time-stepper whose continuous vector field is represented by a feed-forward network.
- Computational framework: Runge-Kutta schemes embed the neural vector-field approximation into a discrete flow map and support learning with variable timesteps.The paper applies the method to test problems and then discusses neural-network overfitting and generalization risks.
2 Methods
The method jointly estimates measurement noise and system dynamics from noisy observations, rather than denoising first and then fitting a model. It constrains a neural-network vector field within a Runge-Kutta time-stepper and regularizes the optimization to improve robustness.
- Problem formulation: The framework treats measurement error and dynamics as unknown components of the map between successive observations.It estimates pointwise measurement errors while learning the dynamical model that propagates the underlying state.
- Problem formulation: Measurements are decomposed into learned error and state, propagated forward or backward with learned dynamics, and recombined to evaluate predictions.The framework uses observations at multiple offsets, including negative steps, so the learned map is tested across local forward and backward pairs.
- Optimization: The optimization jointly refines neural-network parameters and timestep-specific noise estimates by minimizing prediction discrepancies against observed data.The learned map propagates a denoised state and adds the estimated measurement error back before comparison with observations.
- Constrained dynamics model: A feed-forward neural network models the continuous vector field, while an explicit Runge-Kutta scheme produces the discrete flow map.Embedding the network in the time-stepper avoids numerical differentiation of noisy data and focuses learning on the vector field rather than the discrete map.
- Optimization: Local, exponentially weighted observation pairs limit the loss calculation, while penalties on noise estimates and network weights suppress trivial solutions and overfitting.Without regularization, the objective admits a zero-vector-field solution that explains the observations through estimated measurement errors.
- Error measurement: Model accuracy is assessed with prediction, state, noise, and vector-field error metrics, but some evaluations rely on known equations and noiseless training data.The stringent forward-orbit error is especially sensitive to numerical differences and is considered only for the damped-oscillator example.
3 Results
Across canonical systems, the method separates measurement noise from learned dynamics and remains effective under substantial corruption. Experiments also expose a practical boundary: vector-field accuracy deteriorates away from the training data, and trajectory-error evaluation can fail for unstable or chaotic dynamics.
- Cubic oscillator: The cubic oscillator used 2,500 snapshots with 0–20% artificial noise and a neural vector field embedded in a four-step Runge-Kutta scheme.Random-timestep experiments produced similar vector-field and noise errors to constant-timestep data, while forward-orbit error was significantly lower.
- Cubic oscillator: Across 50 cubic-oscillator trials, Figure 5 reports mean measurement-noise and vector-field errors plus median normalized L2 forward-orbit error across noise levels.The 18% noise forward-orbit distribution was omitted because one trajectory diverged and produced non-numeric values; its median was retained.
- Cubic oscillator: The learned model was highly robust to significant measurement noise, recovering inferred states, forward trajectories, and vector fields across increasing noise levels.Figure 3 compares observations with inferred states, true states with learned forward orbits, and learned with true vector fields.
- Cubic oscillator: For 10% Gaussian noise, measurement-noise estimates were compared with the true noise, while vector-field error stayed small near training data and grew near domain edges.The heat-map comparison indicates that extrapolation away from the training trajectory is less accurate.
- Lorenz system: For the Lorenz system, a single 2,500-timestep trajectory was modeled with a three-hidden-layer neural network embedded in a four-step Runge-Kutta scheme.Tests included clean data, 1% Gaussian noise, and 10% Student’s T noise, with predictions extended to five times the training horizon.
- Lorenz system: The learned Lorenz measurement-noise distributions faithfully captured the underlying distributions for both 10% Gaussian and 10% Student’s T noise.The comparison included analytic, empirical, and learned distributions, with moments reported for the x-coordinate.
- Navier–Stokes simulation: The Navier–Stokes test represented x, y, and z as time series obtained by projecting the full flow field onto POD mode 1, POD mode 2, and the shift mode.The figure identifies the reduced coordinates used for this simulation.
- Double pendulum: For the double pendulum, a five-hidden-layer neural network in a four-step Runge-Kutta scheme produced accurate predictions with noise magnitudes up to 10%.The reported results state that error was effectively separated from training data across tested noise levels.
4 Cautionary remarks on neural networks and overfitting
The paper demonstrates accurate learned dynamics from limited, noisy data, but emphasizes that neural-network models can behave unreliably away from densely sampled training attractors. Training trajectories and parameterization strongly affect extrapolation and generalizability.
- The method accurately represents dynamics from limited and noisy time-series data in Lorenz, fluid-flow, and double-pendulum examples.
- The double-pendulum figure compares clean, 5% Gaussian-noise, and 10% Gaussian-noise cases across learned states, forward orbits, and estimated measurement noise.
- Trajectories integrated with the learned dynamics remain on the attractor for long times when the attractor is densely sampled during training.
- A single long trajectory does not accurately interpolate the Lorenz vector field off the attractor, whereas many short transient trajectories produce a more accurate model.
- Different neural-network parameterizations accurately reproduce the training orbit and limit cycle but fail to identify the problem’s fundamental radial symmetry.
5 Discussion
The proposed framework learns continuous vector fields from limited, noisy measurements by jointly estimating measurement noise and dynamics within a constrained Runge-Kutta time-stepper. Its flexibility extends beyond fixed functional libraries but sacrifices interpretability, while future work targets broader noise, latent-variable, and PDE settings.
- The method jointly separates measurement error from the underlying state and learns a neural representation of the governing vector field.
- Embedding the neural vector field in a Runge-Kutta scheme enables learning from unevenly spaced data while focusing discovery on continuous dynamics.
- Neural-network vector fields flexibly represent complex dynamics without predetermined basis functions, but this flexibility forfeits interpretability.
- Future extensions include process noise, rigorous analysis of vector-field interpolation, time-delay coordinates for latent variables, and partial differential equations.
Appendix A: Expected error and structure of loss function
The appendix analyzes how prediction errors accumulate across repeated flow-map applications and uses this structure to motivate exponentially decreasing weights for multistep loss terms. The resulting error bound is exponential in the number of timesteps under stated assumptions.
- The loss evaluates predictions over pairs separated by i forward or backward timesteps, with larger i expected to accumulate more error.
- The analysis defines measurement-noise and flow-map approximation errors and assumes both are sufficiently small on a restricted domain.
- Repeated composition of the approximate flow map produces error terms involving derivatives of the true flow and higher-order approximation terms.
- The upper bound for prediction error grows exponentially with the number of forward and backward timesteps under the stated assumptions.
- Chaotic systems and longer timesteps call for larger ρ values and therefore more aggressive discounting of distant predictions.
- The loss uses exponentially decreasing weights, ω_i = ω_0ρ^-i, while acknowledging that its Gaussian-error assumption is naive.