Source-linked AI summary
Linearly-Recurrent Autoencoder Networks for Learning Dynamics
Samuel E. Otto, Clarence W. Rowley
TL;DR
EDMD and KDMD face dictionary-selection, computational, and overfitting challenges as representational capacity grows. The paper introduces autoencoder-based linear recurrent dynamics, balanced feature-space reduction, and nonlinear reconstruction, demonstrating Koopman learning and short-time prediction across several nonlinear systems.
Problem
EDMD and KDMD depend on dictionary or kernel choices, while large dictionaries increase computational cost and can over-fit insufficient data.
Method
An autoencoder with learned linear recurrent dynamics learns a small Koopman-invariant observable subspace, complemented by balanced feature-space reduction and nonlinear state reconstruction.
Results
The techniques identify Koopman eigenfunctions, produce accurate low-dimensional cylinder-wake models, and make short-time predictions for the chaotic Kuramoto-Sivashinsky equation.
Takeaways & Limitations
Finite training horizons help select features relevant over chosen time spans, while nonlinear reconstruction can improve accuracy when data is intrinsically low-dimensional.
Takeaways & Limitations
Kernel-based encoding and decoding require inner products with all training data, making predictions costly as training sets grow.
Abstract
from arXiv · showhide
This paper describes a method for learning low-dimensional approximations of nonlinear dynamical systems, based on neural-network approximations of the underlying Koopman operator. Extended Dynamic Mode Decomposition (EDMD) provides a useful data-driven approximation of the Koopman operator for analyzing dynamical systems. This paper addresses a fundamental problem associated with EDMD: a trade-off between representational capacity of the dictionary and over-fitting due to insufficient data. A new neural network architecture combining an autoencoder with linear recurrent dynamics in the encoded state is used to learn a low-dimensional and highly informative Koopman-invariant subspace of observables. A method is also presented for balanced model reduction of over-specified EDMD systems in feature space. Nonlinear reconstruction using partially linear multi-kernel regression aims to improve reconstruction accuracy from the low-dimensional state when the data has complex but intrinsically low-dimensional structure. The techniques demonstrate the ability to identify Koopman eigenfunctions of the unforced Duffing equation, create accurate low-dimensional models of an unstable cylinder wake flow, and make short-time predictions of the chaotic Kuramoto-Sivashinsky equation.
1. Introduction.
EDMD and KDMD offer data-driven Koopman approximations but face dictionary-selection, overfitting, and scalability challenges. The paper introduces autoencoder-based linear recurrent dynamics, balanced feature-space reduction, and nonlinear reconstruction, then tests them across three dynamical systems.
- Motivation: EDMD approximates the Koopman operator using a data-driven dictionary, but selecting a sufficiently rich dictionary is difficult and very large dictionaries become impractical.Dictionary size can grow combinatorially with state dimension.
- Motivation: KDMD reduces the computational cost of large dictionaries, yet kernel choice and excessive representational capacity can still cause overfitting and produce too many modes.Reduced-order modeling requires identifying a small salient Koopman-invariant subspace.
- Proposed approach: The proposed architecture learns a small collection of informative observables with an autoencoder, while a learned linear transformation evolves encoded features in time.A nonlinear decoder reconstructs the full state from the learned features.
- Proposed approach: Nonlinear reconstruction relaxes the assumption that the full-state observable belongs to the learned Koopman-invariant subspace, allowing fewer observables to recover more state information.This distinguishes the approach from methods using linear reconstruction through Koopman modes.
- Proposed approach: Balanced Proper Orthogonal Decomposition constructs reduced models from over-specified KDMD systems by retaining features most excited and observable over a finite horizon.The resulting model operates in nonlinear feature space with a separately learned nonlinear decoder.
- Evaluation: The methods are evaluated on Duffing eigenfunctions, unstable cylinder-wake prediction, and short-time Kuramoto–Sivashinsky dynamics.These examples span basin identification, high-dimensional flow modeling, and chaotic reduced-order prediction.
2. Extended Dynamic Mode Decomposition.
EDMD fits a finite-dimensional matrix approximation of the Koopman operator on a chosen observable dictionary using snapshot pairs. KDMD extends this construction through kernel inner products, but its effective feature space remains limited by data rank and can overfit.
- Koopman formulation: The Koopman operator evolves observables by composition with the system dynamics and is linear, although it generally acts on an infinite-dimensional function space.Koopman eigenfunctions can represent intrinsic dynamical features whose values evolve linearly.
- EDMD: EDMD uses snapshot pairs and a finite dictionary to approximate the Koopman operator by minimizing empirical residuals for updated observables.The approximation updates coefficients in linear combinations of dictionary functions.
- EDMD: The EDMD matrix is obtained from data-dependent matrices and a Moore–Penrose pseudoinverse, with rank truncation available as regularization.The pseudoinverse solves the least-squares approximation, while truncation removes low-singular-value components.
- KDMD: KDMD avoids explicit high-dimensional feature matrices by expressing the approximation through kernel inner products between snapshot pairs.Its kernel matrices scale with the number of snapshot pairs rather than the squared number of explicit features.
- Limitations: The effective number of retained features cannot exceed the training-data rank, so increasing dictionary resolution eventually reaches a data-limited ceiling.When rank reaches the number of distinct data points, the representation can memorize the data rather than capture reliable dynamics.
- Limitations: A toy example shows severe overfitting despite a learned matrix residual below 6 * 10^-15, with none of the learned eigenfunctions or eigenvalues correct.The true eigenfunctions lie outside the span of the feature-space data.
3. Recent approach for dictionary learning.
The dictionary-learning approach adapts EDMD observables with a neural network while alternating optimization of the Koopman matrix and dictionary parameters. It keeps the dictionary small while exploring a broader function space, addressing overfitting from oversized dictionaries.
- Optimization: EDMD dictionary learning alternates least-squares optimization of K with gradient descent on neural-network parameters θ.The procedure uses Tikhonov-regularized EDMD for K and gradient descent for θ.
- Optimization: The approach fixes selected dictionary elements to the state observable, preventing the trivial zero solution and encouraging a Koopman-invariant subspace containing the full state.The remaining dictionary elements are learned through neural-network parameterization.
- Dictionary learning: The learned dictionary D(θ) is selected from a larger parameterized dictionary space S, unlike methods whose feature complexity requires enlarging the dictionary itself.This allows a relatively small dictionary to explore a much larger space of candidate functions.
- Dictionary learning: Deep neural networks are used because they can represent complex features through compositions of many simple operations.This supplies expressive candidate observables without requiring a proportionally larger dictionary.
4. New approach: deep feature learning using the LRAN.
The LRAN combines neural-network encoding and decoding with learned linear latent dynamics to identify compact Koopman-invariant observables. It trains on sequential snapshots, supports several extensions, and can represent nonlinear normal modes and continuous-time dynamics.
- LRAN architecture: LRAN encodes the state into low-dimensional features, evolves them with a learned linear matrix K, and decodes the result into an approximation of the full state.The encoder provides dictionary functions, while the decoder replaces Koopman-mode reconstruction.
- LRAN architecture: LRAN sacrifices linear Koopman-mode reconstruction to learn ultra-low-dimensional Koopman-invariant subspaces with nonlinear decoding.The decoder neural network reconstructs the full state from the learned features.
- Training: Longer snapshot sequences help LRAN capture low-amplitude features in highly non-normal systems that influence larger-amplitude dynamics several steps later.Snapshot-pair training may neglect such features because they contribute little to immediate reconstruction accuracy.
- LRAN architecture: The latent prediction pipeline computes zt+τ, applies [K(θK)^τ]T, and decodes the predicted latent state into x̂t+τ.These steps define encoding, latent evolution, and decoding for multi-step prediction.
- Training: Training jointly optimizes reconstruction error and hidden-state time-evolution error, using minibatches of randomly drawn snapshot sequences.The relative weighting of the two errors is controlled by β, and ADAM is used for optimization.
- Extensions: The architecture generalizes to non-uniformly sampled continuous-time systems using a Koopman operator semigroup generated by the infinitesimal Koopman operator.The formulation uses sequential snapshots whose sampling times need not be evenly spaced.
- Extensions: The LRAN can parameterize nonlinear normal modes through complex-conjugate eigenvalues and associated Koopman eigenfunctions.The real and imaginary parts of the eigenfunction provide coordinates on the invariant manifold, whose angular frequency is tied to the eigenvalue and sampling interval.
5. EDMD-based model reduction as a shallow autoencoder.
The EDMD-based reduction treats an over-specified feature-space model as a state-space system and uses balanced reduction to retain informative nonlinear features. This approach is limited by EDMD’s fixed dictionary and by spurious modes that can corrupt reduced dynamics.
- The EDMD-based procedure can retain overfitting because its dictionary remains fixed and overspecified, and inaccurate EDMD dynamics cannot yield accurate reduced models.
- EDMD-based reduction first applies EDMD with a large dictionary, then projects the identified linear dynamics onto a low-dimensional subspace.
- The reduced model reconstructs the full state from features using a linear output map obtained by regression analogous to Koopman modes.
- Balanced truncation retains feature subspaces that are simultaneously maximally observable and controllable.The balancing transformation simultaneously diagonalizes the observability and controllability Gramians.
- Hankel singular values quantify the observability and controllability of the retained features, while balancing and adjoint modes define the reduced coordinates and projected coefficients.
- BPOD estimates finite-horizon Gramians empirically by evolving the dynamics from impulsive initial conditions over a finite time horizon.
- Pre-selecting nearly Koopman-invariant EDMD modes is proposed to reduce corruption of long-time dynamics by spurious modes.
- Kernel-based encoding and reconstruction require evaluating each new example against all training data, increasing prediction cost as datasets grow.
6. Numerical examples.
The numerical examples evaluate LRAN and balanced KDMD models on Duffing dynamics, cylinder wake flow, and chaotic Kuramoto–Sivashinsky dynamics. Across these problems, the methods learn informative low-dimensional representations, with performance depending on reconstruction strategy and regularization.
- 6.1. Duffing equation.: The unforced Duffing experiment compares constrained LRAN, free LRAN, and a fifth-order balanced KDMD reduced-order model using simulated trajectories.The constrained LRAN fixes known discrete-time eigenvalues, whereas the free LRAN learns its transition matrix.
- 6.1. Duffing equation.: 0.9650 was the KDMD balanced reduced-order model’s basin-classification accuracy, compared with 0.9488 for free LRAN and 0.9274 for constrained LRAN.The classification uses the median eigenfunction value as a threshold between basins of attraction.
- 6.1. Duffing equation.: Free LRAN had the lowest Duffing prediction error, while constrained LRAN best matched the true fixed-point locations.The reported explanation is that unconstrained functions gave free LRAN greater flexibility.
- 6.2. Cylinder wake.: In cylinder wake flow, nonlinear LRAN reconstruction produced a mean relative error roughly an order of magnitude smaller than nonlinear KDMD-ROM reconstruction over many time steps.Both models’ linear reconstructions had comparable performance, with errors growing slowly over time.
- 6.2. Cylinder wake.: The nonlinear LRAN decoder maintained a constant low prediction error across the training interval from the unstable equilibrium to the stable limit cycle.The reported training interval was T∆t = 8.0D/U∞, and error began growing outside it.
- 6.3. Kuramoto-Sivashinsky equation.: For chaotic Kuramoto–Sivashinsky dynamics, LRAN and KDMD-ROM models made quantitatively accurate short-term predictions, but accuracy declined after t ≈ 5 while predictions remained qualitatively plausible.The result followed a reduction from a 256-dimensional time-delay state to a 16-dimensional encoded state with linear evolution.
- 6.3. Kuramoto-Sivashinsky equation.: The examples motivate adaptive low-dimensional dictionaries and nonlinear decoders to avoid EDMD over-fitting while retaining dynamically informative features.The LRAN combines an autoencoder-like architecture with linear recurrence so learned features are dynamically important rather than merely energetic.
7. Conclusions.
The paper combines low-dimensional feature discovery, linear encoded dynamics, balanced reduction, and nonlinear reconstruction to model complex dynamical systems. Its conclusions emphasize finite-horizon accuracy, reconstruction trade-offs, and differing computational costs between LRAN and KDMD models.
- Conclusions: Balanced model reduction and nonlinear reconstruction target accurate KDMD reduced-order models from very low-dimensional spaces.Nonlinear reconstruction can substantially improve accuracy for intrinsically low-dimensional data embedded in many dimensions.
- Conclusions: Nonlinear reconstruction is not uniformly beneficial: it may be insensitive to improvement, parameter-sensitive, or less accurate because of over-fitting.
- Conclusions: Unfolding LRAN recurrences improves prediction accuracy, particularly within the training time horizon, while errors grow beyond that horizon.For more complicated systems such as Kuramoto–Sivashinsky, additional dimensions may be needed to preserve linear-model accuracy over time.
- Conclusions: Carefully chosen finite time horizons select features that remain relevant over the time span of interest.
- Conclusions: LRANs have high training costs but negligible evaluation costs, whereas each new KDMD prediction requires inner products with every training data point.This makes LRAN evaluation efficient for producing many predictions, while KDMD evaluation retains a data-dependent computational burden.
- Conclusions: Future work extends LRANs toward controlled, stochastic, and generative dynamical systems through encoded-state control, uncertainty modeling, and richer reconstruction.The paper frames transformations into and out of simple-dynamics spaces as a broader direction for understanding complicated systems.