Source-linked AI summary
Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors
Yong Liu, Chenyu Li, Jianmin Wang, Mingsheng Long
TL;DR
Non-stationary temporal distributions challenge deep forecasting models. Koopa uses Fourier-based disentanglement and hierarchical Koopman Predictors with global and localized operators, achieving competitive forecasting while reducing training time and memory.
Problem
Non-stationary time series vary in statistics and temporal dependencies, creating distribution gaps that make deep forecasting models difficult to generalize.
Method
Koopa disentangles time-invariant and time-variant dynamics with Fourier filtering, then models them using hierarchical Koopman Predictors in a residual forecasting architecture.
Results
Koopa achieves competitive forecasting performance while saving 77.3% training time and 76.0% memory averaged across six real-world benchmarks.
Takeaways & Limitations
Koopman Predictors provide a forecasting framework that models global and localized non-stationary dynamics and can adapt operators to scale the forecast horizon.
Takeaways & Limitations
Experiments use a lookback window fixed at T = 2H, and time-variant processing assumes segment lengths are compatible with the lookback and forecast horizons.
Abstract
from arXiv · showhide
Real-world time series are characterized by intrinsic non-stationarity that poses a principal challenge for deep forecasting models. While previous models suffer from complicated series variations induced by changing temporal distribution, we tackle non-stationary time series with modern Koopman theory that fundamentally considers the underlying time-variant dynamics. Inspired by Koopman theory of portraying complex dynamical systems, we disentangle time-variant and time-invariant components from intricate non-stationary series by Fourier Filter and design Koopman Predictor to advance respective dynamics forward. Technically, we propose Koopa as a novel Koopman forecaster composed of stackable blocks that learn hierarchical dynamics. Koopa seeks measurement functions for Koopman embedding and utilizes Koopman operators as linear portraits of implicit transition. To cope with time-variant dynamics that exhibits strong locality, Koopa calculates context-aware operators in the temporal neighborhood and is able to utilize incoming ground truth to scale up forecast horizon. Besides, by integrating Koopman Predictors into deep residual structure, we ravel out the binding reconstruction loss in previous Koopman forecasters and achieve end-to-end forecasting objective optimization. Compared with the state-of-the-art model, Koopa achieves competitive performance while saving 77.3% training time and 76.0% memory.
1 Introduction
Non-stationary distributions make deep forecasting difficult, while Koopman theory offers a way to represent time-varying dynamics through localized linear operators. Koopa combines Fourier disentanglement, modular predictors, and residual learning for efficient forecasting.
- Non-stationary time series have time-variant statistics and temporal dependencies, creating distribution gaps across training, inference, and lookback windows.
- Koopman theory represents complex nonlinear dynamics in measurement-function space using linear operators, including localized operators for time-variant systems.
- Koopa disentangles time-invariant and time-variant dynamics with Fourier analysis and advances them using modular Koopman Predictors.
- 77.3% training time and 76.0% memory are saved on average across six real-world benchmarks while performance remains competitive with state-of-the-art methods.
2 Related Work
Time-series forecasting has progressed from convolutional, recurrent, MLP, and Transformer architectures, but varying temporal distributions remain a central generalization challenge. Koopa applies Koopman-based predictors to model non-stationary dynamics while removing the reconstruction constraint of earlier Koopman forecasters.
- TCN, RNN, MLP, and Transformer models capture temporal patterns through convolution, recurrence, point-wise weighting, or attention-based sequence modeling.
- Transformers and other deep networks can struggle to generalize across varying temporal distributions, while PatchTST may incur high computational cost for many variates.
- Koopman research combines dynamical-systems analysis with data-driven learning, including autoencoders that learn measurement functions and operators.
- Koopa uses modular predictors with hierarchical operators for time-variant and time-invariant components and removes reconstruction loss for fully predictive training.
3 Background
Koopman theory reframes nonlinear dynamics through linear evolution in a measurement-function space, while localized operators address non-stationary behavior. Koopa uses this perspective to separate and model global and local time-series dynamics.
- 3.1 Koopman Theory: Koopman theory projects nonlinear system states into a measurement-function space governed by a linear operator that advances the representation forward in time.
- 3.2 Dynamic Mode Decomposition: Dynamic Mode Decomposition fits finite-dimensional matrices to approximate Koopman operators, while Koopman Autoencoders learn measurement functions and operators from data.
- 3.3 Time Series as Dynamics: Localized time series may exhibit weak stationarity, motivating neighborhood-wise linear operators for distinct regions of a larger nonlinear dynamical system.
- 3.3 Time Series as Dynamics: Koopa’s stackable blocks learn hierarchical dynamics by fitting each residual after the preceding block’s dynamics have been removed.
- 3.3 Time Series as Dynamics: The time-series decomposition separates deterministic and stochastic components, which are modeled using globally learned and localized linear Koopman operators.
4 Koopa
Koopa uses Fourier filtering and stackable Koopman Predictors to disentangle and advance globally shared and localized dynamics in non-stationary time series.
- Koopa Block: Koopa Blocks hierarchically learn input dynamics by feeding each block the residual of previously fitted dynamics.The final forecast sums predicted components from all blocks.
- Fourier Filter: Fourier Filter separates time-invariant and time-variant components using globally shared and localized frequency spectrums.The filter ranks averaged FFT amplitudes across training windows and selects dominant spectrums.
- Time-invariant KP: Time-invariant KP learns a shared Koopman embedding and parameterized linear operator from lookback-forecast windows.It maps the time-invariant lookback representation to the forecast representation through a globally learned operator Kinv.
- Time-variant KP: Time-variant KP divides inputs into segments and calculates a window-specific operator from localized embedding snapshots.The resulting Kvar varies across windows and represents local temporal variation as a linear system.
- Time-variant KP: Time-variant KP iterates operator forwarding to generate forecasts and decodes predicted embeddings back into time segments.The procedure arranges decoded segments into fitted inputs and forecast outputs.
- Forecasting Objective: Koopa optimizes its forecasting objective directly with MSE against ground truth while calculating Kvar on the fly.Encoder and Decoder parameters are shared across the two predictor types to maintain embedding consistency.
5 Experiments
Koopa is evaluated across multivariate and univariate forecasting, efficiency, dynamics disentanglement, ablations, reconstruction design, and forecast-horizon scaling. It delivers competitive forecasting while substantially reducing training time and memory relative to PatchTST.
- Forecasting results: Koopa achieves state-of-the-art performance in more than 70% of multivariate settings and consistently outperforms other deep models in univariate settings.
- Forecasting results: Koopa surpasses the Koopman-based forecaster KNF by a large margin and can match or exceed PatchTST while addressing non-stationary properties.
- Model efficiency: 62.3% and 96.5% training time are saved on ETTh2 and Traffic, respectively, with only 26.8% and 2.9% memory footprint.
- Dynamics disentanglement: The time-variant component exhibits larger Degree of Variation, while localized Koopman operators show changing temporal patterns across periods.
- Ablation study: Time-invariant and time-variant Koopman Predictors are complementary, and removing either reduces performance; Fourier Filter provides effective disentanglement.
- Avoiding rigorous reconstruction: Forecasting-only optimization outperforms adding reconstruction loss, reducing the optimization gap between training and inference.
- Scaling up forecast horizon: Operator adaptation improves scaling to longer forecast horizons, particularly on non-stationary datasets, by fitting time-variant dynamics with incoming ground truth.
6 Conclusion
The conclusion frames Koopa as a dynamical-systems approach that hierarchically models disentangled time-variant and time-invariant components. It reports competitive performance, efficiency, and potential for longer-horizon forecasting through operator adaptation.
- Koopa treats time series as dynamical systems and hierarchically learns Koopman embeddings and operators for disentangled dynamics.
- Experiments show competitive performance, remarkable efficiency, and potential for longer forecast horizons through operator adaptation.
- Koopa can scale forecast length by adapting localized time-variant operators with incoming ground truth while keeping model parameters frozen.
A.1 Implementation of Operator Adaptation
Koopa adapts its time-variant Koopman operator as incoming ground-truth embeddings arrive, improving local-dynamics fitting while reducing repeated-update cost.
- Operator adaptation: The time-variant predictor forms adjacent embedding collections and computes Kvar by fitting transitions from Zback to Zfore.Zback contains embeddings z1 through zF−1, while Zfore contains z2 through zF.
- Operator adaptation: Incoming ground-truth embeddings are appended to incremental collections, producing Kvar+ for more accurate local-dynamics description.The procedure repeats across the forecast horizon, using each incoming embedding successively.
- Computational acceleration: The naïve repeated update has O(LD3) complexity, whereas the accelerated algorithm requires O((L + D)D2).Because L is proportional to Hte, the corresponding complexities are O(HteD3) and O((Hte + D)D2).
- Computational acceleration: Algorithm 2 gives the same Kvar as Algorithm 1 while reducing each iteration’s complexity to O(D2).The theorem states equivalence, and the complexity reduction follows from the incremental update derivation.
B Implementation Details
Experiments use repeated PyTorch training with standardized optimization and baseline configurations, including Series Stationarization for fair comparison.
- Training setup: Koopa uses L2 loss and ADAM with learning rate 0.001, batch size 32, and early stopping within 10 epochs.Each experiment is repeated three times with different random seeds on NVIDIA TITAN RTX 24GB GPUs.
- Baseline setup: Reproduced baselines use TimesNet Repository configurations and are all equipped with Series Stationarization for fair comparison.This standardizes comparisons between models that originally did and did not adopt the method.
C Hyperparameter Sensitivity
Koopa fixes core settings for efficient search, evaluates operator-adaptation procedures, and finds robustness to several hyperparameters but sensitivity to segment length in some datasets.
- Experimental settings: Koopa fixes segment length at S = T/2 and uses B = 3 blocks in all experiments.The study separately verifies robustness to other hyperparameters.
- Operator adaptation: Operator adaptation receives observed embeddings and successive incoming ground-truth embeddings, then returns predicted embeddings across the horizon.Both algorithms use segment embeddings in RD and process incoming embeddings successively.
- Hyperparameter sensitivity: Koopa is insensitive to encoder-decoder and embedding hyperparameter choices in the reported Figure 7 sensitivity study.The study considers Koopman embedding dimension, hidden layer number, and hidden dimension.
- Hyperparameter sensitivity: Larger block numbers generally lower error, while embedding dimension becomes sensitive when the model is shallow, especially at B = 1, 2.Stacking blocks can compensate for smaller embedding dimensions when the model is sufficiently deep.
- Hyperparameter sensitivity: Segment length is dataset-dependent: performance is sensitive on ECL and Traffic, with differences of about 10%, because one shared S may not suit all variables.The paper therefore uses S = T/2 to limit performance fluctuation across most situations.
D.1 Full Forecasting Results
Additional ETT multivariate and M4 univariate benchmarks show that Koopa remains competitive against state-of-the-art deep forecasting and specialized univariate models.
- Additional benchmarks: Koopa achieves competitive performance on additional ETT multivariate benchmarks and full M4 univariate results.The ETT data include hourly ETTh2 and 15-minute ETTm1/ETTm2; M4 includes yearly, quarterly, and monthly marketing data.
D.2 Full Ablation Results
Koopa’s ablations and efficiency evaluations support complementary Koopman predictors for non-stationary dynamics, with competitive forecasting and substantially lower training and memory costs. Stability measures and smooth training curves further support the model’s optimization behavior.
- Ablation results: Koopa’s complementary Time-invariant KP, Time-variant KP, and Fourier Filter achieve the best forecasting performance in the ablation study.The ablation also evaluates alternative disentanglement choices and branch ordering.
- Model efficiency: 77.3% training time and 76.0% memory footprint are saved on average across six real-world datasets versus PatchTST.Koopa uses 15.2% training time and 3.6% memory in ECL, with savings also reported for ETTh2, Exchange, ILI, Traffic, and Weather.
- Forecasting performance: 38.0% MSE is reduced in ILI and 13.6% MSE in Weather compared with DLinear, while average MSE reduction versus the previous state-of-the-art MLP model reaches 12.2%.These results show competitive forecasting performance despite Koopa being efficiently built with MLP networks.
- Training stability: Koopman operators can face convergence problems on complicated non-stationary variations, so Koopa applies initialization, residual hierarchical disentanglement, and explosion checking.Explosion checking replaces an operator producing NaN during multi-step multiplication with the identity matrix.
- Training stability: Koopa’s training curves show consistent and smooth convergence on the ETT and Weather datasets.The curves are used to check convergence relative to other forecasting models.
- Overall evaluation: Koopa achieves state-of-the-art performance on six real-world forecasting tasks spanning energy, economics, disease, traffic, and weather.The paper presents the model as a promising option for forecasting under limited computational budgets.
F Limitation
The model does not separately model dynamics in different variates, leaving multivariate forecasting improvements involving varied evolution patterns and series relationships as future work.
- Scope boundary: Koopa does not separately consider dynamics across variates, limiting its treatment of varied evolution patterns and series relationships in multivariate forecasting.The authors identify this as an opportunity for improvement.