PaperlayerView exact revision on arXiv

Source-linked AI summary

Bilevel optimization for data-driven learning of Koopman embeddings using kernel-based autoencoders

Joel-Pascal Ntwali N'konzi, Feliks Nüske, Stefan Klus

arXiv:2610.12370v1cs.LGmath.DSstat.ML

TL;DR

Handcrafted EDMD dictionaries are challenging, especially in high-dimensional state spaces, and whether kernel methods can learn effective dictionaries remains an open question. EDMD-kDL learns a kernel dictionary and Koopman approximation through collocation and bilevel optimization; across four prediction settings it performs comparably to or better than ANN-based approaches.

  • Problem

    Handcrafted EDMD dictionaries are challenging, especially in high-dimensional state spaces, and whether kernel methods can learn effective dictionaries remains an open question.

    Source: §I · p. 2, C. Reproducing kernel Hilbert spaces… · p. 4

  • Method

    EDMD-kDL learns a finite-dimensional Koopman embedding by optimizing an RKHS dictionary and associated Koopman matrix through a collocation-based bilevel formulation.

    Source: §III · p. 5, A. Derivation of EDMD-kDL · p. 6, §V · p. 13

  • Results

    Across four settings, EDMD-kDL performs comparably to or better than ANN-based approaches; its maximum relative error over the 318-week period is 6.7% for global SST forecasting.

    Source: D. Global sea-surface temperature… · p. 11, §V · p. 13

  • Takeaways & Limitations

    The tested video example shows EDMD-kDL can predict pendulum dynamics directly from video data.

    Source: §V · p. 13

    For high-dimensional systems, the current approach first uses principal component analysis and then applies EDMD-kDL in the reduced space.

    Source: §V · p. 13

Abstract

from arXiv · show

Koopman operator theory provides a linear framework for analyzing nonlinear dynamical systems and has become a major tool for data-driven modeling. A central challenge, however, is that finite-dimensional approximations computed by methods such as extended dynamic mode decomposition (EDMD) require the dictionary to be specified a priori. Recent machine-learning approaches address this limitation by learning the dictionary from data, predominantly using artificial neural network (ANN) autoencoder architectures. Although kernel methods offer an alternative with greater interpretability and tractability for theoretical analysis, they have received little attention in this setting. We introduce extended dynamic mode decomposition with kernel-based dictionary learning (EDMD-kDL), a kernel-based method for learning finite-dimensional Koopman embeddings directly from data. The method combines ideas from collocation methods and bilevel optimization to simultaneously learn a kernel dictionary and the corresponding Koopman approximation. We evaluate EDMD-kDL against state-of-the-art ANN-based approaches on a range of numerical experiments, including global sea-surface-temperature forecasting and learning directly from video data. Across all tested settings, EDMD-kDL achieves performance comparable to or better than the ANN-based methods. Moreover, in contrast to standard kernel methods, the proposed approach is scalable to large datasets by design since the size of the required kernel matrices depends on the number of collocation points rather than the size of the training dataset.

I. INTRODUCTION

The Koopman perspective makes nonlinear dynamics amenable to linear analysis, but finite-dimensional data-driven approximations depend on choosing informative dictionaries. Prior approaches include kernel methods and ANN-based dictionary learning; this paper motivates learning Koopman embeddings with kernels.

I. INTRODUCTION

  • EDMD and VAC produce finite-dimensional approximations or spectral information from simulation or observational data, supporting applications such as system identification, model reduction, and forecasting.EDMD extends DMD, originally introduced to decompose fluid flows into coherent patterns; VAC generalizes TICA.Source: §I · p. 2
  • EDMD estimates a finite-dimensional Koopman model from data, but its dictionary is difficult to design and becomes especially challenging in high-dimensional settings.A suitable dictionary should span an approximately Koopman-invariant subspace; polynomial dictionaries can grow combinatorially with state-space dimension.Source: §I · p. 2
  • ANN-based dictionary learning addresses dictionary selection by training an encoder for the Koopman embedding alongside a linear embedded-dynamics objective and reconstruction loss.Variants use hybrid dictionaries, stability promotion, multi-step prediction, forward-backward consistency, or stochastic-system objectives.Source: §I · p. 2
  • Kernel EDMD can use an implicitly defined, potentially infinite-dimensional embedding, but its practical subspace is spanned by kernels centered at training data points.The approach has been derived through a kernel trick and through empirical estimation on an RKHS.Source: §I · p. 2

A. Koopman operator theory

  • The Koopman operator is linear even for nonlinear state dynamics, enabling spectral analysis of dominant timescales, coherent patterns, and metastability.Its infinite dimensionality can include a continuous spectrum, motivating finite-dimensional approximations from trajectory data.Source: A. Koopman operator theory · p. 3
  • Koopman eigenfunctions span invariant subspaces, while a full-state observable may require infinitely many eigenfunctions for its Koopman mode decomposition.The resulting decomposition can express the state evolution through the eigenvalues and modes.Source: A. Koopman operator theory · p. 3

B. Extended dynamic mode decomposition (EDMD)

EDMD uses trajectory data and a dictionary of observables to approximate the Koopman operator on a finite-dimensional subspace. The resulting matrix supports approximate spectral analysis.

B. Extended dynamic mode decomposition (EDMD)

  • EDMD uses paired data matrices X and Y, where Y contains the one-step-forward data corresponding to X.The data may come from one trajectory or multiple trajectories.Source: B. Extended dynamic mode decomposition… · p. 3
  • A dictionary of observables defines an embedding, and when its span contains the full-state observable and is Koopman-invariant, embedded dynamics are linear.The finite-dimensional matrix K represents the Koopman operator restricted to that subspace.Source: B. Extended dynamic mode decomposition… · p. 3
  • EDMD estimates the Koopman matrix by solving a linear regression problem and provides approximate eigenpairs from its spectral decomposition.In the infinite-data limit, the EDMD matrix converges to the Galerkin projection onto the dictionary span.Source: B. Extended dynamic mode decomposition… · pp. 3–4

C. Reproducing kernel Hilbert spaces and kernel EDMD

Kernel EDMD uses a positive semi-definite kernel to represent an RKHS and estimate Koopman dynamics. Its practical embedding is constrained to the span of kernel functions centered at training points.

C. Reproducing kernel Hilbert spaces and kernel EDMD

  • Handcrafting dictionaries that span an approximately Koopman-invariant subspace is challenging, motivating kernel EDMD with an implicitly defined, potentially infinite-dimensional dictionary.Kernel EDMD uses a kernel trick or empirical estimation of the Koopman operator on an RKHS.Source: C. Reproducing kernel Hilbert spaces… · p. 4
  • A positive semi-definite kernel is symmetric and yields positive semi-definite Gram matrices for finite collections of points.Such a kernel uniquely defines an RKHS, and conversely every reproducing kernel is positive semi-definite.Source: C. Reproducing kernel Hilbert spaces… · p. 4
  • An RKHS is a Hilbert space of functions with a reproducing kernel that reproduces function evaluation and spans the space.The mapping x to k(x, ·) is the canonical feature map.Source: C. Reproducing kernel Hilbert spaces… · p. 4
  • Assuming the RKHS is Koopman-invariant, kernel EDMD estimates a finite-rank operator from data and obtains approximate eigenfunctions through an eigenvalue problem.The operator formulation includes regularization and covariance operators at the current and next time steps.Source: C. Reproducing kernel Hilbert spaces… · p. 4

D. Learning Koopman dictionaries with neural networks

ANN-based methods learn Koopman dictionaries with an embedded-dynamics objective, sometimes combined with reconstruction or multi-step prediction losses. EDMD-kDL instead learns dictionaries over an RKHS.

D. Learning Koopman dictionaries with neural networks

  • ANN-based dictionary learning jointly optimizes a learnable embedding component and the EDMD matrix, sometimes using alternating exact matrix updates and stochastic gradient descent.Hybrid dictionaries retain a fixed component, while autoencoders pair an encoder with a decoder and reconstruction loss.Source: D. Learning Koopman dictionaries with… · pp. 4–5
  • Koopman autoencoders use encoder-decoder reconstruction with linear prediction, and consistent variants learn forward and backward dynamics with a consistency constraint.Multi-step prediction has been observed to improve performance over single-step training.Source: D. Learning Koopman dictionaries with… · p. 5

III. EDMD WITH KERNEL-BASED DICTIONARY LEARNING

  • EDMD-kDL learns finite-dimensional Koopman embeddings by optimizing EDMD dictionaries over an RKHS associated with a user-specified kernel.It is presented as an alternative to ANN-based dictionary learning.Source: §III · p. 5
  • EDMD-kDL's learned dictionary dimension is independent of training-set size, and it supports multi-step prediction training unlike standard kernel EDMD.The method is introduced as a kernel-based approach to learning finite-dimensional Koopman embeddings.Source: §III · p. 5

A. Derivation of EDMD-kDL

EDMD-kDL casts dictionary learning as bilevel optimization over pseudo-values at collocation points, then jointly selects those values and the EDMD matrix. The inner dictionary problem has an explicit solution in the kernel span of the collocation points.

A. Derivation of EDMD-kDL

  • Pseudo-values at collocation points make the dictionary-learning problem finite-dimensional, with the outer problem selecting those values and the corresponding EDMD matrix.The inner problem fixes the pseudo-values and optimizes the complementary dictionary.Source: A. Derivation of EDMD-kDL · pp. 5–6

1. Solving the inner optimization problem

  • The representer theorem reduces each inner-problem solution to a linear combination of kernel functions evaluated at the collocation points.The resulting coefficient problem uses the Gram matrix evaluated at those points.Source: §1 · p. 6
  • Even when the Gram matrix is singular, the inner problem has a unique solution, although the coefficient representation need not be unique.The passage distinguishes uniqueness of the function solution from non-uniqueness of the associated coefficient solution.Source: §1 · p. 6

2. Solving the outer optimization problem

The outer optimization uses the kernel Gram matrix to express its objective compactly and can eliminate the EDMD matrix by variable projection. The resulting problem over pseudo-values is solved with L-BFGS and automatic differentiation.

2. Solving the outer optimization problem

  • An SVD of the symmetric positive-semidefinite Gram matrix rewrites the objective compactly, with its squared RKHS term expressed using the Frobenius norm.The factorization uses coincident left and right singular vectors.Source: §2 · pp. 6–7
  • For fixed pseudo-values, the EDMD matrix can be solved exactly as in standard EDMD, allowing variable projection to reduce the outer problem to optimization over pseudo-values.The authors also state that the variables can instead be optimized jointly.Source: §2 · p. 7
  • The reduced optimization over pseudo-values is solved using L-BFGS with automatic differentiation.The text also notes that an analytic gradient and standard numerical optimization routines are alternatives.Source: §2 · p. 7
  • The algorithm takes training data and collocation points as inputs and returns a learned dictionary, an EDMD matrix, and approximated Koopman spectral information.The algorithm description lists the kernel, data matrices, collocation points, and number of learnable dictionary elements as inputs.Source: §2 · p. 7

B. EDMD-kDL with kernel-based autoencoders

The kernel-based autoencoder learns the full dictionary together with a decoder trained to reconstruct the full-state observable. Its bilevel objective combines EDMD prediction, reconstruction, and RKHS regularization losses.

B. EDMD-kDL with kernel-based autoencoders

  • Kernel-based autoencoders jointly learn the full dictionary and a decoder map trained to reconstruct the full-state observable.Source: B. EDMD-kDL with kernel-based… · p. 7
  • The decoder is constrained to be linear because the full-state observable must lie in the dictionary span for its decomposition into approximate Koopman eigenfunctions.Source: B. EDMD-kDL with kernel-based… · p. 7
  • The outer objective combines the standard EDMD loss, an autoencoder reconstruction loss, and RKHS regularization.The reconstruction loss imposes a soft encoder-decoder constraint at the collocation points.Source: B. EDMD-kDL with kernel-based… · pp. 7–8
  • For fixed pseudo-values, the Koopman matrix and decoder can be solved separately, so variable projection can reduce the joint optimization.Source: B. EDMD-kDL with kernel-based… · p. 8

C. Learning with a multi-step prediction objective

The framework extends kernel-based autoencoder training to multi-step prediction, using data that provide forward predictions for T steps. In this setting, variable projection can be used for the decoder matrix but not the Koopman matrix.

C. Learning with a multi-step prediction objective

  • The authors extend the kernel-based autoencoder formulation to multi-step prediction, motivated by reported accuracy and robustness gains from multi-step ANN-based Koopman autoencoders.They state that they expect a similar pattern for their framework.Source: C. Learning with a multi-step… · p. 8
  • The multi-step formulation uses training data with forward predictions for T steps, where T is at least one.Source: C. Learning with a multi-step… · p. 8
  • Variable projection is available for the decoder matrix in the multi-step formulation, but not for the Koopman matrix.Source: C. Learning with a multi-step… · p. 8

IV. NUMERICAL EXPERIMENTS

The experiments compare EDMD-kDL with ANN-based approaches and EDMD-RFF across synthetic and real-world prediction tasks. They use Sobol-sequence collocation points and a Gaussian RBF kernel for EDMD-kDL.

IV. NUMERICAL EXPERIMENTS

  • The benchmarks include a nonlinear pendulum, cylinder-flow vortex shedding, global sea-surface-temperature forecasting, and next-frame prediction from pendulum video.The first two use synthetic data; the latter two are real-world datasets.Source: §IV · p. 8
  • EDMD-kDL is compared with state-of-the-art ANN-based methods and EDMD-RFF, an approximation of standard kernel EDMD.The ANN implementations are the official implementations proposed in Ref. 30.Source: §IV · p. 8
  • EDMD-kDL uses Sobol-sequence collocation points and a Gaussian RBF kernel with bandwidth σ across the experiments.Source: §IV · p. 8

A. Undamped nonlinear pendulum

The pendulum benchmark uses a frictionless, unforced two-dimensional Hamiltonian model, with initial angular position determining the degree of nonlinearity. A single trajectory supplies training and test data for almost-linear and truly nonlinear regimes.

A. Undamped nonlinear pendulum

  • The frictionless, unforced pendulum is modeled as a two-dimensional Hamiltonian system, with x2(0) = 0 and nonlinearity set by x1(0).Source: A. Undamped nonlinear pendulum · p. 8
  • The dataset contains 1600 points from one trajectory simulated at Δt = 0.1 using the exact solution in terms of elliptic functions.Source: A. Undamped nonlinear pendulum · p. 8
  • The almost-linear and truly nonlinear regimes use x1(0) = 0.8 and 2.4, respectively; the first 600 time steps train models and the remaining 1000 test them.Source: A. Undamped nonlinear pendulum · p. 9
  • EDMD-kDL and the Koopman autoencoders use embedding dimension N = 6 and T = 8 forward steps; consistent KAE also uses eight backward steps.The kernel bandwidth is σ = 0.75 for EDMD-kDL and EDMD-RFF; noise robustness is also investigated.Source: A. Undamped nonlinear pendulum · p. 9

1. Clean data setting

Across the pendulum, cylinder-flow, and video tests, EDMD-kDL outperforms neural approaches in several settings, while its relative performance against EDMD-RFF varies with the task and noise. The reported results also identify phase drift and irregular neural predictions as relevant behavior in noisy tests.

1. Clean data setting

  • On clean pendulum data, EDMD-RFF has the lowest prediction error, while EDMD-kDL consistently outperforms the neural dictionary-learning methods.In the truly nonlinear regime, KAE and cKAE fail to capture the phase portrait properly.Source: §1 · p. 9, Figure 1 · p. 10

2. Noisy data setting

  • In the noisy almost-linear regime, EDMD-kDL is marginally worse than EDMD-RFF at 1% noise, but its errors grow more slowly as noise increases.EDMD-RFF's phase drift at high noise explains its higher errors; EDMD-kDL outperforms KAE and cKAE across noise levels in error and phase-portrait preservation.Source: §2 · p. 9
  • In the noisy truly nonlinear pendulum regime, EDMD-kDL outperforms the other methods at substantial noise by balancing phase and amplitude preservation.EDMD-RFF preserves amplitude but incurs phase drift and high prediction errors; ANN-based predictions are highly irregular even with forward-backward consistency.Source: §2 · p. 9

B. Kármán vortex shedding

  • EDMD-kDL has the lowest long-term prediction error for cylinder-flow vortex shedding; all methods except EDMD-RFF predict the dynamics reasonably well.Source: B. Kármán vortex shedding · p. 10

C. Learning from video data

  • EDMD-kDL produces sensible pendulum-video predictions across the test horizon, while KAE and cKAE fail to learn the dynamics properly.The SVD reconstruction shows dimensionality-reduction error independently of the temporal model; increasing the autoencoder embedding dimension did not improve their performance.Source: C. Learning from video data · p. 10

D. Global sea-surface temperature forecasting

For global SST forecasting, EDMD-kDL maintains low error across the forecast horizon and produces forecasts similar to SINDy-SHRED. The discussion reports comparable or better performance than ANN-based alternatives across four examples and describes the method’s joint optimization and remaining high-dimensional-data work.

D. Global sea-surface temperature forecasting

V. DISCUSSION

  • EDMD-kDL jointly optimizes an RKHS embedding map and Koopman matrix, using collocation pseudo-values and bilevel optimization to obtain a finite-dimensional problem that also learns a linear decoder.Source: §V · p. 13
  • Across four examples, EDMD-kDL performed comparably or better than ANN-based alternatives, including SINDy-SHRED for global SST, and predicted pendulum video where ANN approaches drastically failed.Source: §V · p. 13
  • The current high-dimensional workflow first reduces dimensionality with PCA; extending EDMD-kDL to work directly with high-dimensional data remains future work.Source: §V · p. 13

Related papers

Loading 2610.12370v1…