Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability

Shaowu Pan; Karthik Duraisamy

Source-linked AI summary

Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability

Shaowu Pan, Karthik Duraisamy

arXiv:1906.03663v5Jun 9, 2019math.DSstat.ML

TL;DR

The paper addresses the challenge of learning useful continuous-time Koopman representations while handling finite-dimensional approximation, stability, and uncertainty. It develops a measure-theoretic probabilistic deep-learning framework with differential and recurrent models, stable parameterization, residual DMD learning, and MFVI, then evaluates the framework on polynomial, Duffing, and cylinder-wake systems. The conclusions describe evaluation across three continuous nonlinear systems, including noisy unstable wake flow.

Problem
The paper addresses limitations in practical Koopman analysis, including infinite-dimensional representations, global reconstruction difficulties, and the lack of a unified continuous framework for stabilization, physics information, and uncertainty quantification.
Method
The framework learns continuous-time Koopman decompositions with differential or recurrent neural models, structurally parameterizes the Koopman operator for guaranteed stability, embeds residual SVD-DMD learning, and uses mean-field variational inference for uncertainty quantification.
Results
The framework was evaluated on three continuous nonlinear systems: a toy polynomial problem, the Duffing oscillator, and an unstable cylinder wake flow with noisy measurements.
Takeaways & Limitations
The paper presents one framework spanning continuous-time Koopman learning, stabilization, uncertainty quantification, and evaluation from toy dynamics to unstable wake flow.
Takeaways & Limitations
The learned representation can use a spurious nonlinear embedding with incorrect eigenvalues, while global finite-dimensional Koopman reconstruction is limited for systems with multiple fixed points.

Abstract

from arXiv · show

The Koopman operator has emerged as a powerful tool for the analysis of nonlinear dynamical systems as it provides coordinate transformations to globally linearize the dynamics. While recent deep learning approaches have been useful in extracting the Koopman operator from a data-driven perspective, several challenges remain. In this work, we formalize the problem of learning the continuous-time Koopman operator with deep neural networks in a measure-theoretic framework. Our approach induces two types of models: differential and recurrent form, the choice of which depends on the availability of the governing equations and data. We then enforce a structural parameterization that renders the realization of the Koopman operator provably stable. A new autoencoder architecture is constructed, such that only the residual of the dynamic mode decomposition is learned. Finally, we employ mean-field variational inference (MFVI) on the aforementioned framework in a hierarchical Bayesian setting to quantify uncertainties in the characterization and prediction of the dynamics of observables. The framework is evaluated on a simple polynomial system, the Duffing oscillator, and an unstable cylinder wake flow with noisy measurements.

1. Introduction.

The Koopman operator offers an operator-theoretic route to representing nonlinear dynamics through linear evolution of observables, but practical use requires finite-dimensional approximations and learned reconstructions. This work addresses continuous-time learning while combining stabilization, physics information, and uncertainty quantification.

1. Introduction.: The Koopman operator is linear but infinite-dimensional, motivating finite-dimensional invariant subspaces for practical analysis of nonlinear dynamics.It acts on observables in a Hilbert space and governs their temporal evolution.
1. Introduction.: Finite-dimensional Koopman analysis seeks eigenfunctions spanning an invariant subspace that can represent observables of interest, especially the state.The identity observable corresponds to a linear representation of the dynamics.
1. Introduction.: Global finite-dimensional linear reconstruction is limited for systems with multiple fixed points, motivating nonlinear reconstruction and additional modes for different attractor basins.The passage links this limitation to the impossibility of a global topological conjugacy with a linear system.
1. Introduction.: Existing EDMD and KDMD approaches can overfit and be difficult to interpret because their approximated eigenfunction counts scale with dictionary features or training snapshots.Deep-learning approaches have therefore been explored for extracting Koopman decompositions.
1. Introduction.: The paper proposes a continuous-time probabilistic deep-learning framework that combines uncertainty quantification, structural stability enforcement, and physics information.The continuous formulation is also described as better suited to desired constraints and non-uniform sampling.

2. Data-driven framework to learn continuous-time Koopman decompositions.

The framework seeks a finite-dimensional Koopman-invariant representation whose observations evolve linearly while an inverse map reconstructs the original state. It formulates this representation through neural-network-learned observation functions and distinguishes several established approximation strategies.

2. Data-driven framework to learn continuous-time Koopman decompositions.: The framework seeks a finite-dimensional Koopman invariant subspace with linearly independent smooth observation functions.The observation vector is constructed from basis functions spanning the desired subspace.
2. Data-driven framework to learn continuous-time Koopman decompositions.: The observation vector is required to evolve linearly according to a finite-dimensional Koopman matrix K.This condition is expressed through the evolution equation dΦ/dt = ΦK and its chain-rule form F · ∇xΦ = ΦK.
2. Data-driven framework to learn continuous-time Koopman decompositions.: An inverse reconstruction map Ψ is required so that Ψ ◦ Φ equals the identity, allowing recovery of the state from the learned observations.This gives the learned coordinates a reconstruction role beyond merely predicting observable evolution.
2. Data-driven framework to learn continuous-time Koopman decompositions.: The practical goal is to find a minimal basis spanning a Koopman-invariant subspace, using neural networks as the selected approximation mechanism.The framework contrasts neural-network observations with linear DMD and prescribed or kernel-based extended DMD observations.

2.1. Neural network.

The neural-network component is formulated as a feedforward mapping and embedded in a constrained variational learning problem. The formulation incorporates assumptions progressively and adapts to differing data or governing-equation requirements.

2.1. Neural network.: The feedforward neural network uses hidden layers with activation functions and a linear final layer.The network maps an input of dimension n to an output of dimension m through learned weights and biases.
2.1. Neural network.: The neural-network learning problem is defined first as a constrained variational problem and then incorporates data through an empirical measure.Two formulations are proposed based on differing requirements.

2.2. Problem formulation.

The paper formulates continuous-time Koopman learning in differential and recurrent forms, depending on whether governing equations or trajectory data are available. Both forms optimize finite-dimensional neural-network embeddings using Koopman and reconstruction or prediction errors.

2.2. Problem formulation.: The differential form uses known governing equations without requiring trajectory data, whereas the recurrent form uses discrete trajectory data for high-dimensional systems.The recurrent formulation accesses the action of K_t over discrete time intervals.
2.2. Problem formulation.: The target is a finite-dimensional Koopman-invariant observation space whose matrix realization approximates the operator while enabling state reconstruction.A zero Koopman error would yield a perfect linear embedding in the L2(µ) sense.
2.2.1. Leveraging known physics: differential form.: The differential formulation minimizes a constrained variational problem over neural-network observation and decoder parameters, approximated with penalty-based gradient descent.The neural network searches only a fixed-architecture finite-dimensional subset of the invariant-function space.
2.2.1. Leveraging known physics: differential form.: The neural-network architecture can create an approximation gap because the search is restricted to the chosen parameterized subset of observation functions.The formulation also relaxes linear independence while bounding the resulting subspace dimension by D.
2.2.1. Leveraging known physics: differential form.: The differential form is the natural continuous-time approximation, while trajectory information can improve accuracy by minimizing errors over multiple time steps.This motivates the recurrent formulation’s use of trajectory data.
2.2.2. Unknown physics with only the trajectory data: recurrent form.: The recurrent form defines Koopman error over predicted versus ground-truth trajectories and jointly optimizes reconstruction or prediction error with the learned operator.The authors directly minimize prediction error because they do not control the decoder’s Lipschitz constant.

2.3. Measures.

The measure construction turns sampled phase-space points and trajectory observations into empirical risks for Koopman learning. It also represents trajectory sampling through a Markov kernel, while Hankelization improves efficiency at some prediction-accuracy cost.

2.3.1. Measure for data generated by sampling in phase space.: Independent phase-space samples induce an empirical measure that converges uniformly to the underlying measure as the sample count grows.This permits rewriting the differential formulation as empirical risk minimization.
2.3. Measures.: Trajectory sampling depends on the initial condition, so the sampling pattern is modeled with a Markov kernel over states and observation times.Sampling rates can vary across regions such as sharp phase-space changes and fixed-point attractors.
2.3.2. Measure for trajectory data generated by solving the initial value problem.: For multiple trajectories, independently sampled initial conditions and time points define an empirical trajectory measure that converges uniformly as the number of trajectories and maximum horizon increase.The recurrent objective is then expressed using this empirical measure.
2.3.2. Measure for trajectory data generated by solving the initial value problem.: The recurrent empirical formulation generalizes the LRAN model with a simpler loss function and a prior discrete-spectrum model.The comparison is stated at the level of the resulting data-driven formulation.
2.3.2. Measure for trajectory data generated by solving the initial value problem.: The trajectory-data setup generalizes to Hankelized segments from one long trajectory, improving computational efficiency relative to the original data.The shorter time horizon can reduce prediction accuracy.
2.3. Measures.: For measure-preserving or attractor dynamics, the continuous-time Koopman eigenspectrum lies on the imaginary axis even when the underlying system is chaotic.The passage distinguishes this setting from Koopman operators that may also admit unstable modes.

2.4. Guaranteed stabilization of the Koopman operator.

The paper uses a structured parameterization of the Koopman matrix to guarantee non-positive eigenvalue real parts. This parameterization preserves expressivity for diagonalizable stable operators through transformations absorbed by the encoder and decoder.

2.4. Guaranteed stabilization of the Koopman operator.: Non-uniqueness of ideal observation functions allows encoder and decoder layers to absorb the linear transformations needed to impose the stable parameterization.The guarantee applies where unstable modes are absent.
2.4. Guaranteed stabilization of the Koopman operator.: The proposed realization is motivated by stable neural-network structures using skew-symmetric weights inside nonlinear activations.The paper applies this idea to a Koopman-operator parameterization.
2.4. Guaranteed stabilization of the Koopman operator.: The parameterization guarantees stability because every allowed matrix has eigenvalues with non-positive real parts.The construction makes the matrix negative semi-definite, which implies the eigenvalue condition.
2.4. Guaranteed stabilization of the Koopman operator.: The structured form is rich enough to represent stable diagonalizable Koopman operators without loss of expressivity.This is the expressivity result stated alongside the stability guarantee.
2.4. Guaranteed stabilization of the Koopman operator.: Any real diagonalizable matrix with non-positive eigenvalue real parts can be represented up to real similarity by the structured parameterization.The proof uses an eigendecomposition and real similarity between matrices sharing the same complex eigenvalues.
2.4. Guaranteed stabilization of the Koopman operator.: The construction explicitly assembles real blocks for complex and real eigenvalues before establishing similarity to the stable parameterization.These blocks preserve the target eigenvalues while enabling the real-valued realization.
2.4. Guaranteed stabilization of the Koopman operator.: The tridiagonal parameterization reduces the number of parameters from O(D^2) to O(D), although a 2 × 2 block alternative would require more implementation effort.The authors therefore use the tridiagonal form throughout.

2.5. Design of neural network architecture with SVD-DMD embedding.

The architecture embeds SVD-DMD into neural networks so the model learns a residual around dominant modes, while supporting differential and recurrent Koopman formulations.

2.5.1. Input normalization: Input normalization uses standard deviations for neural-network training but can be modified for POD coefficients when component significance should be preserved.
2.5.2. Embedding with SVD-DMD: The model embeds SVD-DMD into an autoencoder and learns the residual after projecting normalized states onto dominant modes.The embedding uses q = zΛV_D and separates nonlinear reconstruction from the SVD-DMD component.
2.5.2. Embedding with SVD-DMD: SVD-DMD is used because dominant POD modes can reduce noise and approximate Koopman-invariant subspaces, including in high-dimensional nonlinear systems.
2.5.2. Embedding with SVD-DMD: The residual architecture can initialize near the desired DMD solution when the dynamics are exactly representable by DMD, because nonlinear weights start small.Nonlinear reconstruction may attenuate spurious modes for essentially linear dynamics.
2.5.2. Embedding with SVD-DMD: The differential and recurrent forms are presented as separate network architectures for learning continuous-time Koopman dynamics.The recurrent model reduces to an over-parameterized SVD-DMD, related to a simplified optimized DMD, when its nonlinear network is deactivated.
2.5.2. Embedding with SVD-DMD: The architecture is optimized with Adam, weight decay, truncated-normal network initialization, and DMD-based initialization of the Koopman operator.

3. Probabilistic formulation.

The probabilistic formulation uses Bayesian neural networks and mean-field variational inference to represent parameter and predictive uncertainty. Stability constraints permit finite-time uncertainty propagation, while the framework is evaluated through three numerical examples.

3. Probabilistic formulation: The framework uses Bayesian inference to quantify uncertainty in neural-network parameters and model predictions, avoiding impractical high-dimensional MCMC.Variational inference provides a compromise between computationally expensive MCMC and less descriptive local approximations.
3.2. Mean-field variational inference.: Variational inference optimizes an approximating posterior by maximizing the ELBO, with ADVI using Gaussian distributions and a mean-field factorization.
3.2. Mean-field variational inference.: The mean-field Gaussian approximation is scalable and effective for deep neural networks but cannot generally capture multimodal posteriors.Robustness and scalability of high-dimensional inference remain active research problems.
3.3. Bayesian hierarchical model setup.: The hierarchical Bayesian model assigns priors and variational posteriors to encoder and decoder weights, biases, stabilized Koopman parameters, and noise-scale parameters.
3.3. Bayesian hierarchical model setup.: Separate likelihood constructions are specified for differential and recurrent forms, with aleatoric covariance parameters included in the probabilistic formulation.The likelihood assumes independence between the data and the structure of aleatoric uncertainty.
3.4. Propagation of uncertainties.: For differential-form forecasting, unknown process uncertainty is approximated using Gaussian white noise, yielding a multivariate Ornstein–Uhlenbeck process.
3.4. Propagation of uncertainties.: Because the stabilized Koopman operator has no eigenvalues with positive real part, the propagated variance does not diverge in finite time and can be sampled by Monte Carlo.
3.4. Propagation of uncertainties.: The framework is demonstrated through three numerical examples, while recurrent-form posterior prediction is obtained directly from the inferred model and initial condition.

4. Numerical Results & discussion.

The framework is evaluated on fixed-point, Duffing, and cylinder-wake systems, showing uncertainty characterization, stable eigenvalue learning, and robustness to noisy data.

4. Numerical Results & discussion.: The experiments use differential-form models when governing equations are available and a recurrent-form model with SVD-DMD for the noisy cylinder-wake data.The cylinder-wake trajectory contains 1245 snapshots, with the first 600 used for training.
4.1. 2D fixed point attractor: For the fixed-point attractor, the inferred mean Koopman eigenvalues are λ1 = −0.99656 and λ2 = −0.05049, and the learned eigenfunctions resemble the analytic solution.Figure 5 reports posterior means of eigenfunction amplitudes and phases for both eigenvalues.
4.1. 2D fixed point attractor: Prediction uncertainty for the fixed-point system decreases as more data are collected near the region of interest, while halving the data still yields a small eigenvalue standard deviation.Figure 6 compares Monte Carlo predictions using 800, 1600, and 10000 data points.
4.2. 2D unforced Duffing oscillator: For the Duffing oscillator, λ1,2 = −0.535 ± 0.750i and λ3 = −2 × 10−5; λ3 indicates the basin of attraction, with uncertainty increasing outside the training region.The reported normalized standard deviation is uniform within [−2, 2]×[−2, 2] and rises sharply outside it.
4.3. Flow wake behind a cylinder: The cylinder-wake model keeps all learned Koopman eigenvalues stable under noise, while preserving the dominant vortex-shedding frequency at λ = ±0.528j.The frequency corresponds to Strouhal number St = 0.168.
4.3. Flow wake behind a cylinder: The posterior prediction mean closely matches the clean cylinder-wake trajectory, while projected uncertainty follows vortex shedding and remains relatively small near the wake.Monte Carlo samples characterize uncertainty from Gaussian measurement noise across five noise ratios.

5. Conclusions.

The paper presents a probabilistic, stabilized framework for continuous nonlinear Koopman decomposition and evaluates it across systems, data densities, unseen states, and noise.

5. Conclusions.: The framework combines continuous-time Koopman learning, stable operator parameterization, differential and recurrent formulations, an SVD-DMD-informed architecture, and mean-field variational inference.The formulations target different data and governing-equation availability conditions.
5. Conclusions.: Across three nonlinear systems, the model captures uncertainty from data density, unseen data, and measurement noise, while remaining robust to noise.The evaluation spans polynomial problems, the Duffing oscillator, and an unstable cylinder wake.

Appendix A. Real part of eigenvalues of a negative semi-definite real square matrix.

The appendix establishes that a negative semi-definite real square matrix has eigenvalues with non-positive real parts.

Appendix A. Real part of eigenvalues of a negative semi-definite real square matrix.: A real matrix is negative semi-definite when x⊤Ax ≤ 0 for every non-zero real vector x.The definition permits non-symmetric matrices.
Appendix A. Real part of eigenvalues of a negative semi-definite real square matrix.: Every eigenvalue of a negative semi-definite real square matrix has a non-positive real part.The proof writes λ = α + jβ and derives α ≤ 0 from the negative semi-definite condition.