PaperlayerView exact revision on arXiv

Source-linked AI summary

Principled Koopman Representations with Kalman Inference for Efficient Time-Series Prediction

Ruiquan Li, Yuheng Bu

arXiv:2609.17815v1cs.LGcs.AI

TL;DR

Prior neural approaches may learn Koopman spaces that lack a consistent operator interpretation or fail to capture low-rank dynamics. K2SVD learns leading singular functions with a Hilbert-Schmidt objective and applies Kalman filtering in a linear Gaussian state-space model. Across benchmarks, it retains competitive forecasting accuracy while reducing parameter use and inference time.

  • Problem

    Neural forecasting approaches may use Koopman spaces that are mathematically inconsistent with the operator framework and fail to capture intrinsic low-rank dynamics.

    Source: §1 · p. 1

  • Method

    K2SVD learns leading Koopman singular functions through a Hilbert-Schmidt objective and performs inference with a linear Gaussian state-space model and Kalman filtering.

    Source: §1 · pp. 1–2

  • Results

    Across multiple dynamic systems and real-world benchmarks, K2SVD achieves competitive forecasting performance with fewer parameters and shorter inference time than existing methods.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    The results support modeling compact transition structure in Koopman space to reduce latent dynamic dimension while retaining competitive prediction accuracy and achieving strong inference efficiency.

    Source: §6 · p. 7

    Kalman inference treats residual components as noise under the premise that the learned Koopman space is sufficiently representative and residuals carry limited predictive information.

    Source: Kalman Filter for State Space Evolution · p. 4

Abstract

from arXiv · show

The Koopman operator has been widely used for time-series prediction in dynamical systems. However, prior work that learns latent ``Koopman spaces'' using neural networks often did not construct a valid Koopman space for forecasting, as these representations may be mathematically inconsistent with the operator-theoretic formulation and fail to capture the intrinsic low-rank structure of system dynamics. To address this issue, we introduce K$^2$SVD, a method that explicitly learns the leading singular functions of the Koopman operator by optimizing a Hilbert-Schmidt objective. This yields a well-defined low-rank approximation of the Koopman operator with an interpretable linear combination, featuring a compact latent space with less than $10\%$ of the dimensions used in previous work. In the learned Koopman space, K$^2$SVD further captures temporal evolution with a linear Gaussian state-space model and performs inference via Kalman filtering, mitigating noise accumulation during multi-step prediction. Empirical results show that K$^2$SVD outperforms state-of-the-art methods across multiple datasets, with significantly faster prediction speeds and lower computational cost than previous efficiency-focused models. This highlights the benefits of principled low-rank Koopman representations and opens up broader potential for applications.

1 Introduction

The introduction motivates principled Koopman representations as an alternative to black-box forecasting when interpretable latent spaces matter. K2SVD learns a low-rank Koopman space and uses Kalman inference for temporal prediction.

1 Introduction

  • Existing probabilistic forecasting approaches may use heuristic or loosely defined Koopman spaces that do not preserve a consistent operator interpretation or low-rank structure.The introduction also notes that forecasting must handle nonlinear transformations under noisy perturbations, while explainable and expressive latent spaces are important in domains such as economics, weather, and energy.Source: §1 · p. 1
  • On Electricity, K2SVD outperforms competing methods in prediction performance, time efficiency, and space efficiency.Circle size indicates peak allocated GPU memory during training.Source: Figure 1 · p. 1
  • K2SVD learns leading Koopman singular functions with a Hilbert-Schmidt objective, yielding a low-rank representation with well-defined linear dynamics.The objective targets the best rank-k approximation of the operator.Source: §1 · pp. 1–2
  • It models evolution in the learned Koopman space with a linear Gaussian state-space model and Kalman filtering to mitigate noise accumulation during multi-step prediction.The introduction describes a two-stage training scheme that first establishes the Koopman subspace, then refines dynamical and observation components end-to-end.Source: §1 · p. 2

2 Related Works

Related work spans deep forecasting architectures, Koopman operator learning, and probabilistic models that combine Koopman representations with structured dynamics. The cited literature includes methods for extracting low-rank Koopman structure and for handling uncertainty in temporal prediction.

Probabilistic time series forecasting

  • Deep forecasting approaches include DeepAR (Salinas et al. 2020), N-BEATS (Oreshkin et al. 2020), and transformer-based models (Lim et al. 2021; Zhou et al. 2021; Rasul et al. 2021).The passage describes these as learning flexible sequence representations for multi-horizon forecasting.Source: §2 · p. 2

Koopman operator learning

  • The Koopman operator framework offers a principled way to analyze nonlinear dynamical systems by lifting them into a linear space, with classical work establishing spectral foundations and links to ergodic theory and modal decomposition.Source: §2 · p. 2
  • DMD and related data-driven methods approximate Koopman operators from data, while SVD-based and neural approaches recover dominant modes or Koopman subspaces.The cited examples include Schmid (2010), Tu (2013), Williams et al. (2015a,b), Wu and Noé (2020), Mardt et al. (2018), and Kostic et al. (2024).Source: §2 · p. 2

Koopman-Based Probabilistic Forecasting

  • Koopman-based probabilistic models include distributional forecasting, hierarchical predictors, generative latent dynamics, and hybrid models with Kalman filtering.Examples include KNF (Wang et al. 2022), Koopa (Liu et al. 2023), Koopman VAEs (Naiman et al. 2023), K-ESKF (Huang et al. 2024), and K2VAE (Wu et al. 2025).Source: §2 · p. 2

3 Preliminaries

The preliminaries formulate forecasting as prediction under a stochastic transition process and define the Koopman operator through its action on observables. They also introduce its SVD structure, low-rank property, and empirical second-moment quantities.

3 Preliminaries

  • A possibly nonlinear map with time-invariant dynamics and independent noise induces a time-homogeneous Markov process with transition density p(x′ | x).The stochastic dynamics are analyzed through this transition density.Source: §3 · p. 2
  • The Koopman operator acts on observables through conditional expectation, and repeated applications give multi-step posterior means under the paper's compact-operator assumption.The problem setting predicts future sequences from historical data and conditional distributions.Source: §3 · pp. 2–3
  • The operator's SVD expresses it as rank-1 terms with orthonormal input and output singular functions; the paper assumes singular values decline rapidly in real-world low-rank dynamics.The singular values are ordered σ1 ≥ σ2 ≥ · · · ≥ 0.Source: §3 · p. 3
  • Second-moment matrices and joint second moments provide quantities estimated from trajectories, with empirical estimates denoted by hats.The passage gives identities relating joint moments to the Koopman operator and its adjoint.Source: §3 · p. 3

4 Proposed Method

K2SVD learns a low-rank Koopman space, then models its temporal evolution with Kalman inference for forecasting. Its two-stage training separates learning the Koopman basis from training the inference and observation components.

4 Proposed Method

  • K2SVD combines a low-rank Koopman representation with linear dynamics perturbed by Gaussian noise and Kalman inference for time-series forecasting.It trains in two stages, then maps predicted latent states back to the observation space through a linear projection.Source: §4 · p. 3

4.1 First Stage: Learning the Koopman Space

  • Multivariate series are divided into non-overlapping patches that serve as tokens to model cross-variable interaction during state transitions.The context and prediction horizon use the same patch size.Source: Encoder · p. 3
  • The Hilbert–Schmidt objective targets the best rank-d Koopman approximation, and its loss can train the Koopman space separately without ground-truth predictions.This separate stage is intended to stabilize and well-structure the learned space.Source: LoRA Loss Function · p. 3

4.2 Second Stage: Forecast with Kalman Inference

  • Kalman inference addresses misalignment between actual latent base vectors and their linear expansion, which can arise from residuals including dimension reduction and non-Markovian dynamics.The method treats residual components as noise under the assumption that they carry limited predictive information, and uses iterative prediction and updates to refine the latent state.Source: Kalman Filter for State Space Evolution · p. 4

4.3 Overall Algorithm

  • After learning and freezing the Koopman functions, K2SVD initializes its transition matrix from linear regression and trains the finite-dimensional Koopman matrix with the forecasting objective.The trainable matrix adapts downstream dynamics while preserving the learned operator-consistent latent space, with only small computational and parameter overhead.Source: §4.3 · p. 5

5 Experiments

The experiments evaluate K2SVD on real-world and dynamical forecasting datasets, examining low-rank structure, prediction performance, efficiency, and component ablations.

5.1 Experiment Setup

  • Nine datasets span high-dimensional real-world and image-like observations and low-dimensional time-series and financial data; evaluation compares seven baselines using NRMSE, complexity, and inference time.The high-dimensional group contains Electricity-L, Weather-L, and Ordered-MNIST; the low-dimensional group contains six datasets.Source: §5.1 · p. 5

5.2 Main Results

  • 10–30 dimensions capture the singular-value decay in most datasets, while ILI-L requires about 60; K2SVD uses dimension 16 for most datasets rather than the baselines’ commonly used 128.Increasing latent dimension yields only minor precision effects while substantially increasing computational cost.Source: §5.2 · p. 5, Figure 3 · p. 6
  • K2SVD performs strongly on Electricity-L, Weather-L, Ordered-MNIST, and Exchange-L, and remains competitive on low-dimensional datasets, supporting performance across observation scales.Table 2 reports NRMSE results for high-dimensional datasets and Exchange-L; full low-dimensional results are in Appendix D.1.Source: §5.2 · p. 6, Table 2 · p. 7
  • K2SVD is fastest on all three evaluated datasets, roughly 3x faster than every baseline on Electricity and MNIST, and uses under one-third the peak GPU memory of the most efficient baselines.The comparison used the same GPU and software environment; the efficiency advantage was particularly pronounced on high-dimensional datasets.Source: §5.2 · p. 6, Table 3 · p. 7

5.3 Ablation Study

  • Increasing Koopman dimension from 16 to 64 does not significantly improve prediction precision and sometimes reduces it, while FLOPs rise substantially.Inference time does not visibly increase; the authors infer it is limited by memory bandwidth rather than compute capacity.Source: Ablation on the Dimension of Dynamic… · p. 6, Table 4 · p. 7
  • The LoRA-objective training scheme performs better than end-to-end training on MNIST, Electricity, and Exchange, with comparable results on Weather at the same hidden-space dimension.The comparison removes stage 1 and trains the encoder, Koopman matrix, and Kalman parameters end to end using only Le.Source: Table 5 · p. 7, Ablation on LoRA Objective · p. 7
  • Directly decoding linear rollouts without Kalman inference causes performance declines, supporting the use of filtering to handle noise in linear prediction.The reported declines vary across datasets.Source: Table 6 · p. 7, Ablation on Kalman inference · p. 7

6 Conclusion

K2SVD builds forecasting around compact Koopman structure, reducing latent dynamic dimension while retaining competitive accuracy and achieving SOTA inference efficiency. The results also distinguish datasets through their learned spectra.

6 Conclusion

  • K2SVD reduces latent dynamic dimension while retaining competitive prediction accuracy and achieving SOTA inference efficiency.The conclusion frames forecasting around operator structure rather than predictive capacity alone.Source: §6 · p. 7
  • Experiments distinguish intrinsically different datasets through the learned spectrum, providing a visible basis for judging them.Source: §6 · p. 7
  • Future work could adapt Koopman rank and transition operators over time and separate random perturbations from genuine dynamic changes.Source: §6 · p. 7

A Proof to the LoRA Loss

The appendix justifies the LoRA loss and outlines a two-stage K2SVD training algorithm that applies it to learn a low-rank Koopman space before forecasting.

A Proof to the LoRA Loss

B Algorithm

  • Algorithm 1 first learns a low-rank Koopman space with LoRA, then freezes f and g while training the Kalman forecasting module.The static variant keeps the regression-induced Koopman matrix fixed; the dynamic variant updates it during downstream training.Source: B Algorithm · p. 10

C Experimental Settings

The experimental appendix specifies training and model settings, then reports additional NRMSE results across low- and high-dimensional datasets. K2SVD performs strongly on several datasets, but is not uniformly best.

C Experimental Settings

  • Training uses 15 warm-up epochs followed by 35 end-to-end epochs, with up to 50 epochs and 100 batches per epoch on a single GPU.The settings also specify a learning rate of 1 × 10^-3 and gradient clipping at 0.5.Source: C Experimental Settings · p. 10
  • The model uses 24-step patches and a dynamic dimension of 16, while each encoder and decoder has three hidden layers of width 256.Inference draws 100 samples per input and summarizes the predictive distribution over 20 quantiles.Source: C Experimental Settings · p. 10
  • All input features are standard-scaled, with training and evaluation batch sizes of 64 and 32, respectively.The data pipeline uses eight parallel data-loading workers and a held-out validation split.Source: C Experimental Settings · p. 10

D.1 Additional Results on Long-Term Time Series Prediction

  • K2SVD is best on Exchange-L at both horizons, ETTh1-L at 96, and ETTh1-L, ETTh2-L, and ETTm2-L at 720.K2VAE and Koopa remain strong competitors on several shorter-horizon or lower-variance cases; the reported metric is NRMSE.Source: §D.1 · p. 10
  • Algorithm 1 learns the low-rank Koopman space with LoRA, then freezes f and g and trains the Kalman forecasting module for latent-state rollout.The rollout predicts and updates latent state and covariance using the Kalman equations.Source: §D.1 · p. 11
  • K2SVD leads Electricity-L and Ordered-MNIST at prediction length 96 and remains competitive at 720, while K2VAE leads Weather-L.The high-dimensional datasets include coupled-variable and image-like states; the reported metric is NRMSE.Source: §D.1 · p. 11

D.2 Additional Results on Long Prediction Efficiency

For long-horizon inference, K2SVD is fastest across three datasets and uses the least GPU memory. Its efficiency is associated with forecasting through low-rank linear state evolution and Kalman inference.

D.2 Additional Results on Long Prediction Efficiency

  • The long-horizon comparison includes FITS, iTransformer, Koopa, and K2VAE, covering general-purpose and structurally closest efficiency-competitive baselines.Source: §D.2 · p. 11
  • Inference time falls by about 2.4×–5.5× versus the strongest non-K2SVD baseline, with K2SVD fastest on all three datasets and consistently lowest in GPU usage.Table 7 compares inference time, GPU usage, and meta-FLOPs at T = 480 for MNIST and T = 720 for the other datasets.Source: §D.2 · p. 11, Table 7 · p. 12
  • After encoding observations into the low-rank Koopman space, long-range prediction uses lightweight linear state evolution and Kalman inference instead of repeated high-dimensional nonlinear transformations.The FLOPs comparison is presented as highlighting this source of efficiency.Source: §D.2 · p. 12

D.3 Discussion of High-rank Dataset

The discussion presents ILI-L as a high-rank, distribution-shifted case where K2SVD’s structural advantage is weaker. K2SVD remains comparable but does not dominate on this dataset.

D.3 Discussion of High-rank Dataset

  • On ILI-L, K2SVD remains comparable but does not dominate the NRMSE results at prediction lengths 96 and 720.Table 10 reports results for these settings.Source: §D.3 · p. 12, Table 10 · p. 13
  • Figure 4 shows ILI-L’s Koopman spectrum decays more slowly than those of other datasets, making it a high-rank case despite low-dimensional observations.Source: §D.3 · p. 12, Figure 4 · p. 13
  • In a high-rank regime, a compact latent space discards more non-negligible singular directions, limiting how much predictive dynamics its finite-dimensional transition preserves.Source: §D.3 · p. 12
  • Abrupt epidemic-phase changes and shifts between context and forecast windows make a single low-dimensional transition harder to estimate reliably, while recursive long-horizon rollout can amplify mismatch.Source: §D.3 · p. 12
  • K2SVD is designed for dominant low-rank Koopman structure, so its structural advantage is weaker on ILI-L’s high-rank, distribution-shifted regime.Source: §D.3 · p. 13

D.4 Additional Ablation study

The ablation compares a dynamically updated Koopman matrix with a fixed regression-induced matrix. K2SVD consistently improves over the static variant on the evaluated datasets.

D.4 Additional Ablation study

Related papers

Loading 2609.17815v1…