Source-linked AI summary
Forecasting Sequential Data using Consistent Koopman Autoencoders
Omri Azencot, N. Benjamin Erichson, Vanessa Lin, Michael W. Mahoney
TL;DR
The paper addresses the challenge of forecasting high-dimensional nonlinear time series while incorporating physical structure beyond conventional RNNs. It proposes a Consistent Koopman Autoencoder that learns linear latent dynamics in both forward and backward directions and regularizes their consistency. Across challenging clean and noisy datasets, the method reports accurate long-horizon predictions, robustness to noise, and improved performance over other models.
Problem
RNN-based forecasting can struggle with long-term training and physical constraints, while practical Koopman methods rely on finite-dimensional approximations whose assumptions and forward-backward consistency require analysis.
Method
The model learns approximate forward and backward Koopman systems in a transformed latent space and trains them with penalties promoting accurate reconstruction, prediction, and dynamical consistency.
Results
The method notably outperforms other models on noisy data and long time predictions across several challenging high-dimensional datasets.
Takeaways & Limitations
Forward-backward modeling and consistency regularization provide a physics-constrained forecasting framework with accurate predictions, time reversibility, and stable long-horizon behavior.
Takeaways & Limitations
The practical Koopman approximation assumes a transformation whose finite-dimensional representation captures most of the dynamics, and one discussion penalty approximates a fuller term using κ indices rather than m.
Abstract
from arXiv · showhide
Recurrent neural networks are widely used on time series data, yet such models often ignore the underlying physical structures in such sequences. A new class of physics-based methods related to Koopman theory has been introduced, offering an alternative for processing nonlinear dynamical systems. In this work, we propose a novel Consistent Koopman Autoencoder model which, unlike the majority of existing work, leverages the forward and backward dynamics. Key to our approach is a new analysis which explores the interplay between consistent dynamics and their associated Koopman operators. Our network is directly related to the derived analysis, and its computational requirements are comparable to other baselines. We evaluate our method on a wide range of high-dimensional and short-term dependent problems, and it achieves accurate estimates for significant prediction horizons, while also being robust to noise.
1. Introduction
The paper develops a physics-constrained Koopman approach for forecasting high-dimensional nonlinear time series by representing dynamics linearly in a learned latent space. It incorporates forward and backward evolution with consistency constraints, targeting accurate, reversible, and stable long-horizon prediction.
- Motivation: RNNs face exploding and vanishing gradients over long horizons and limited expressivity for short-term dependencies.These challenges complicate long-term memory and sequential forecasting.
- Koopman framework: Koopman theory represents nonlinear dynamics through a linear operator acting on scalar functions, enabling linear-algebraic and spectral analysis.Practical models approximate this infinite-dimensional operator in a finite-dimensional latent space.
- Proposed approach: Consistency between forward and backward dynamics is formalized through their composition and incorporated as a training constraint.In the Koopman setting, consistency is related to the matrix condition DC = Iκ.
- Proposed approach: The proposed model learns a transformation under which nonlinear states evolve along a linear path, while incorporating physics priors linking forward and backward maps.The latent forward and backward systems are represented by approximate matrices C and D.
- Proposed approach: The model directly predicts both future and past observations using separate forward and backward Koopman operators.This setup supports time-reversible prediction without retraining a separate reverse-direction network.
- Evaluation: The Physics Constrained Learning framework is designed for accurate predictions, time reversibility, and stable behavior over long horizons on clean and noisy high-dimensional systems.Evaluations include pendulum, cylinder flow, vortex flow on a curved domain, and climate data.
2. Related Work
Related work spans Koopman-based dynamical-system modeling, recurrent architectures, and physics-based neural networks. Prior methods approximate Koopman operators, enforce stability or conservation properties, or incorporate forward and backward dynamics without the paper’s full analysis and consistency design.
- Koopman-based models: Koopman-based methods model dynamical systems through operator approximations, including Dynamic Mode Decomposition and neural approaches.Related variants minimize residual errors, promote stability, use variational methods, or apply graph convolutions.
- Positioning: Prior work considered forward and backward dynamics or similar network designs, but without the paper’s combined analysis, back-prediction, and consistency terms.The paper positions its contribution relative to non-neural forward-backward optimization and neural Koopman architectures.
- Recurrent models: RNNs process sequences by maintaining hidden states that combine current inputs with previous internal states.LSTM and GRU variants have achieved strong results in language modeling and machine translation.
- Physics-based models: Physics-based neural networks encode mechanics, conservation laws, or Hamiltonian structure to improve physical plausibility and handle stiff problems.Examples include Euler–Lagrange networks and methods preserving energy.
3. Method
The PCL model combines autoencoding with linear forward and backward latent dynamics, while training enforces accurate reconstruction, prediction, consistency, and stability-related behavior.
- 3.1. Autoencoding Observations: The model encodes observations into a low-dimensional latent space and decodes latent variables back to the spatial domain.The encoder χe performs the embedding, while χd reconstructs observations; the specific architectures are problem dependent.
- 3.2. Backward Dynamics: Two bias-free linear layers represent approximate Koopman operators for forward estimates and backward forecasts.C and D propagate latent dynamics forward and backward, respectively.
- 3.2. Backward Dynamics: Multistep forecasting is used because it improves prediction and generalization compared with one-step forward and backward computation.Training uses supplied states up to λs steps away and powers of C and D to obtain forecasts.
- 3.2. Backward Dynamics: Unlike forward-only models, PCL supports direct backward prediction through the independently modeled D operator.Forward-trained models typically produce poor backward predictions, whereas PCL supports back prediction by construction.
- 3.3. Consistent Dynamics: Consistency training links the forward and backward operators by promoting DC = Iκ, with related but different penalties for continuous and discrete settings.The consistency penalty connects the maps’ composition to the identity operator.
- 3.3. Consistent Dynamics: A spectral regularization term promotes eigenvalues near the unit circle, while the full loss balances reconstruction, prediction, and consistency.The model permits quasi-stable behavior; unregularized unstable modes can produce rapidly diverging forecasts.
4. Consistent Dynamics via Koopman
The paper characterizes consistency between forward and backward dynamics through Koopman operators in continuous and discrete settings. It derives computational conditions and uses them to formulate a practical consistency penalty.
- Continuous-space consistency: Invertibility in continuous space is equivalent to the backward and forward Koopman operators composing to the identity in an orthogonal basis.The condition is expressed as ⟨ξi, Uψ Kϕ ξj⟩M = δij for every basis-element pair.
- Discrete-space construction: The discrete construction represents maps with matrices Pϕ and Qψ, which act as Koopman operators in the canonical basis.Scalar functions are stored as vertex values, while Pϕδz = hϕ(z) encodes interpolated vertex coefficients.
- Discrete-space construction: In an orthogonal basis, the discrete operators are represented by C = B^T Pϕ B and D = B^T Qψ B.The basis matrix B contains orthogonal basis elements as columns.
- Discrete-space consistency: A discrete map is consistent when ψ ◦ ϕ(z) = z for every z, corresponding to matrix conditions involving the forward and backward operators.The proof establishes QP = I by showing the condition on basis elements and using that the basis spans scalar functions.
- Practical penalty: The practical consistency penalty approximates the full condition over the first κ basis elements, while the proposition’s condition extends to all m sampled dimensions.The penalty is closer to the full term than the simpler identity loss for κ ≤ m, including when κ ≪ m.
5. Experiments
Experiments evaluate the proposed model on nonlinear pendulum, fluid-flow, and climate datasets using clean and noisy high-dimensional sequences. Across these settings, the model generally improves long-horizon forecasting and robustness relative to the compared baselines.
- Experimental setup: The evaluation compares the proposed model with Dynamic AE, feed-forward, and recurrent neural-network baselines across several datasets.The training uses forward and backward prediction steps with reconstruction, prediction, and consistency loss weights.
- Nonlinear pendulum: The pendulum experiments use amplitudes θ0 = 0.8 and θ0 = 2.4, high-dimensional transformed snapshots, and a 1000-step forecasting horizon.The sequence contains 1700 points sampled at Δt = 0.1, with 600 points used for training.
- Nonlinear pendulum: The proposed model achieves the best pendulum results across clean and noisy settings, while RNN performance deteriorates for noise and long horizons.DAE struggles as pendulum nonlinearity increases, and the proposed model also outperforms the Hamiltonian NN in all settings.
- Fluid flows: For cylinder flow, the models perform similarly on clean data, but the proposed model outperforms DAE on noisy data and long prediction horizons.The authors attribute this pattern to the regularizing effect of the additional loss terms.
- Fluid flows and climate data: For sphere flow and sea-surface-temperature data, the proposed model generally produces better predictions and greater robustness than DAE.Sphere-flow results worsen as diffusion increases and makes the dynamics non-invertible; climate forecasts at days 120 and 175 are closer to ground truth.
6. Ablation Study
The ablation study varies the weights of the backward-prediction and consistency penalties on a noisy pendulum task. The proposed model generally remains stronger than the other baselines across the tested settings.
- Ablation design: The ablation varies λbwd and λcon, which control the backward-prediction and consistency penalties, respectively.The study revisits the noisy pendulum with initial condition θ0 = 0.8.
- Ablation results: Across the tested parameter settings, the model generally outperforms other baselines on both average error and most distant prediction error.The comparison is summarized in Table 5, with the experimental setting marked by a star.
7. Discussion
The paper proposes a physically constrained Koopman learning model that incorporates backward dynamics and consistency between forward and backward systems. Across challenging datasets, it outperforms other models on noisy data and long-term predictions.
- The model approximates high-dimensional dynamical systems with linear evolution matrices derived from Koopman theory.
- It incorporates backward dynamics during prediction and promotes consistency between the forward and backward systems.
- These modifications relax strict reversibility and stability constraints while still regularizing the parameter space.
- The method is evaluated against a state-of-the-art Koopman-based network and other baselines on several challenging datasets.
- It notably outperforms the other models on noisy data and for long time predictions.
A. Network architecture
The evaluation uses a shallow autoencoding architecture with three-layer encoders and decoders, enabling focused comparison between DAE and the proposed model. The architecture is specified in Table 6, with width controlled by α.
- The encoder and decoder each contain only three layers in the evaluation architecture.
- The shallow design focuses comparisons on differences between the DAE model and the proposed model.
- Table 6 lists the network structure, including layer sizes and activation functions.
- The architecture uses p = 16 · α, where α controls the width of encoder and decoder layers.
B. Computational requirements
The proposed model has modestly greater parameterization and training cost than DAE because of additional consistency-related terms, while inference remains about 1 ms for both models.
- DAE has κ^2 fewer parameters than the proposed model.
- Training the proposed model requires approximately 1.8 times DAE’s average time per epoch across the tested cases.
- The additional training time is attributed to the model’s extra penalty terms.
- The forward-prediction asymptotics match the backward component at O(λsnm), while the consistency term has a non-tight κ^4 bound.
- Inference takes approximately 1 ms and is the same for DAE and the proposed model.
- Figure 9 compares average epoch runtimes across cylinder-flow, pendulum, and SST test cases, including noisy variants.
C. Backward prediction of dynamical systems
The model supports direct backward prediction through a learned backward map, while Koopman linearity also permits back forecasting through the inverse forward operator. On cylinder-flow data, it substantially outperforms DAE in backward prediction.
- Given an observation f_t, the network produces forward and backward estimates using the Koopman matrices C and D.
- Unlike typical neural networks, the model predicts backward dynamics without retraining in the reverse direction.
- Koopman linearity enables back forecasting through the inverse forward operator C^-1.
- On cylinder-flow data, the proposed model clearly outperforms DAE by an order of magnitude in backward prediction error.The comparison includes clean and noisy versions of the data, and also reports estimates obtained from the learned matrix D.
- Figure 10 compares backward prediction for the proposed model and DAE on cylinder-flow data, with red and blue curves respectively.