PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Latent-Dimension Scaling in Dynamical-System Learning through Spectral Reliability

Itsushi Sakata, Yuta Miyauchi, Yoshinobu Kawahara

arXiv:2610.11866v1cs.LGmath.DSnlin.CD

TL;DR

The paper asks whether larger representations improve autoregressive dynamics prediction and compares Koopman models trained with latent-prediction or spectral-residual objectives. Across six chaotic systems, the spectral-residual model achieved lower median rollout errors than latent prediction at every evaluated dimension, alongside eigenvalues concentrated in low-residual regions.

  • Problem

    The paper asks how autoregressive prediction error changes with representation size and whether that dependence varies with the loss used to learn latent dynamics.

    Source: §1 · p. 2

  • Method

    The paper analyzes Koopman eigenstructure using relative residuals and establishes convergence of dictionary-space minimal residuals under boundedness and observable-space approximation.

    Source: §1 · p. 3, §2.2 · p. 4, §6 · p. 17

  • Results

    Across six chaotic systems, spectral-residual training yielded lower median windowed rollout errors and longer mean VPTs than latent-prediction training at all five evaluated dimensions, with larger median-error reductions in every system.

    Source: §6 · p. 17

  • Takeaways & Limitations

    For bounded Koopman operators, approximation of the observable space makes minimal residuals converge to their full-space values, supporting the expectation of more reliable residual-based spectral information at larger dimensions.

    Source: §6 · p. 17

    The baseline comparison uses finite-data estimates at a fixed bandwidth.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

In deep learning, approximation theory motivates increasing representation size. We ask whether this benefit extends to dynamics learning through autoregressive prediction. We analyze the learned time evolution through the eigenstructure of Koopman operators, using relative residuals to detect spurious eigenpairs arising even as one-step error falls. For bounded Koopman operators, we show that minimal residuals over learned dictionary spaces converge pointwise to their full-space counterparts as these spaces approximate the observable space in $L^2$. Our hypothesis is that Koopman spectral reliability helps explain how consistently rollout error decreases with increasing dimension. We compare two models of a shared Koopman autoencoder trained alternately for reconstruction and latent evolution, using latent-prediction loss (one-step prediction errors in latent coordinates) or spectral-residual loss (relative residuals of candidate eigenpairs). Across six chaotic systems, both models reduced median windowed rollout error from smallest to largest dimension. The spectral-residual model achieved lower medians than the latent-prediction model for all systems and dimensions, and its median fell by a larger factor in every system. Its median decreased monotonically with dimension in four systems, against one for latent prediction. Against four baseline families, its mean valid prediction times were nearly always longer. At the largest dimension under two-stage training, we compared eigenvalue positions with each learned dictionary's residual contours. Spectral-residual eigenvalues concentrated in low-residual regions, whereas latent-prediction eigenvalues also appeared in high-residual regions, consistent with the hypothesis.

1 Introduction

The introduction asks whether increasing representation size improves autoregressive dynamics prediction, and whether the training objective shapes the consistency of that improvement. It motivates a comparison of latent-prediction and spectral-residual Koopman autoencoders and frames spectral reliability as a possible explanation.

1 Introduction

  • Prior scaling results span domains, but chaotic-system studies report both decreasing and nonmonotonic multi-step error, and had not tested whether scaling depends on the dynamics loss.The cited studies include decreasing field-prediction error with latent dimension and nonmonotonic multi-step prediction error.Source: §1 · pp. 1–2
  • The paper asks how the size-related improvement in autoregressive prediction depends on the loss used to learn latent dynamics.Scaling denotes prediction error's dependence on representation size N, which sets latent dimension and hidden-layer width in the two models.Source: §1 · p. 2
  • The study compares two stages of a shared Koopman autoencoder: latent-prediction loss measures one-step latent error, while spectral-residual loss measures candidate eigenpair residuals.The authors hypothesize that eigenvalues in low-residual regions help explain how consistently added dimensions reduce rollout error.Source: §1 · p. 3
  • Under two-stage training, spectral-residual training reduced median rollout error more consistently and by a larger smallest-to-largest-dimension factor than latent-prediction training.Both models reduced median error across the six chaotic systems; spectral-residual eigenvalues also concentrated in low-residual regions, unlike some latent-prediction eigenvalues.Source: §1 · p. 3
  • For bounded Koopman operators, the minimal residual over learned spaces converges to the full-space infimum as those spaces approximate every observable in L2.The introduction also reports that spectral-residual eigenvalues cluster in low-residual regions for their learned dictionaries, whereas latent-prediction eigenvalues also occur in high-residual regions.Source: §1 · p. 3

2 Problem formulation

The formulation represents nonlinear state evolution through a Koopman operator on observables and evaluates finite learned dictionaries using relative residuals. Under boundedness, approximation, and sampling assumptions, dictionary-space minimal residuals converge toward their full-space counterparts.

2.1 Koopman operator and observables

  • The Koopman operator K acts by composing observables with the state map F and is assumed bounded on the L2 space of observables.The underlying state evolves in discrete time, and the operator remains linear even when F is nonlinear.Source: §2.1 · p. 3
  • At dimension N, the learned dictionary comprises encoder-coordinate observables spanning VN, while the autoencoder advances latent coordinates with a learned matrix and decodes them to states.The encoder and decoder map between the measured state and N-dimensional latent coordinates.Source: §2.1 · p. 3

2.2 Relative residuals and pseudospectra

  • A small τN(λ) means some observable in VN has a small residual, but it does not certify a particular computed eigenpair.A large τN(λ), by contrast, implies large residuals for every candidate in VN.Source: §2.2 · p. 4
  • The empirical relative residual uses M one-step state pairs to assess candidate eigenpairs, and τM,N minimizes that residual over nonzero observables in VN.The residual is formed from sample evaluations, with the empirical Gram and cross-correlation matrices used in its computation.Source: §2.2 · p. 4

2.3 Convergence assumptions

  • The dimension-limit assumption requires approximation of every observable, and then any λ whose full-space minimal residual is below ε enters the dictionary-space ε-sublevel set for sufficiently large N.This implication follows from the stated convergence of dictionary-space minimal residuals to their full-space values.Source: §2.3 · pp. 4–5
  • For a bounded Koopman operator, if learned dictionary spaces approximate every observable in L2, their minimal residual converges pointwise to the full-space relative-residual infimum.The spaces need not be nested; boundedness makes the L2 approximation sufficient for convergence of residuals.Source: §2.3 · pp. 4–5
  • For each fixed dictionary dimension, convergence of empirical Gram and cross-correlation matrices to population matrices yields uniform residual convergence on compact sets under A4.The ordered limits in sample size, dictionary dimension, and threshold characterize the approximate point spectrum under A1–A4.Source: §2.3 · pp. 4–5

3 Methods

The methods compare latent-prediction and spectral-residual losses within a shared Koopman autoencoder framework, alongside auxiliary training and baseline comparisons. They evaluate prediction and spectral behavior across six chaotic systems and increasing representation sizes.

3.1 Model and training losses

  • Both models use a shared Koopman autoencoder, with reconstruction training followed by encoder updates using either latent-prediction or spectral-residual loss.The decoder is fixed during the second stage; joint training is also included in comparisons.Source: §3.1 · p. 5, §3 · p. 5, §3.1.5 · p. 7

3.1.1 Operator estimation

  • The latent evolution matrix is estimated by regularized least squares from encoded snapshot pairs, with regularization stabilizing nearly singular Gram matrices.A separate held-out minibatch evaluates the time-evolution loss without estimating the operator at that update.Source: §3.1.1 · pp. 5–6

3.1.2 Loss functions

  • Latent-prediction loss measures one-step latent-coordinate errors, while spectral-residual loss penalizes normalized empirical residuals of candidate eigenpairs.The spectral-residual candidates come from eigenpairs of the estimated operator transpose, and normalization makes residuals scale-independent.Source: §3.1.2 · p. 6

3.1.3 Training

  • Two-stage training uses reconstruction pretraining before time-evolution updates; comparisons also include operator-norm penalties and joint training.The two-stage procedure runs for 200 epochs with at most 250 updates per stage per epoch.Source: §3.1.3 · p. 6, §3.1.4 · pp. 6–7, §3.1.5 · p. 7

3.2 Benchmark systems

  • The six benchmarks combine four delay-coordinate systems, including Duffing and Mackey–Glass, with Lorenz-96 and Kuramoto–Sivashinsky using measured states directly.The benchmark systems use chronological train/validation and test intervals, with delay vectors constructed separately within each interval.Source: §3.2 · pp. 7–8

3.3.1 Prediction metrics

  • Evaluation summarizes rollout error using VRMSE over (0.5, 0.7] LT and valid prediction time (VPT), which counts consecutive prediction steps below normalized-error threshold 0.5 within a five-LT horizon.Source: §3.3.1 · p. 8

4 Prediction error across representation sizes

Across six systems and five latent dimensions, the spectral-residual model had lower median windowed VRMSE than latent prediction in every main two-stage comparison. Its error reductions were larger in every system, and its median decreased monotonically in more systems.

4.1 Median error and variability

  • In all 30 system–dimension comparisons, the spectral-residual model had lower median windowed VRMSE than latent prediction.Fig. 1 reports the two-stage comparison across six systems and five dimensions.Source: §4.1 · p. 9, Figure 1 · p. 10
  • From N0 to 16N0, spectral-residual median VRMSE fell 10%–76%, versus 2%–52% for latent prediction, with a larger reduction in every system.The spectral-residual median declined at every interval in four systems; in Rössler and Lorenz-96 it first rose by less than 1%.Source: §4.1 · pp. 9–10
  • Spectral-residual median VRMSE decreased at every consecutive dimension in four systems, compared with only Duffing for latent prediction.In Rössler and Lorenz-96, spectral-residual medians first increased by less than 1%; latent-prediction medians increased over 10 of 20 intervals in the other five systems, sometimes by up to 20%.Source: §4.1 · p. 10

4.2 Training variants and valid prediction time

  • Mean VPT was longer for spectral residual than latent prediction at every dimension in all six systems, by factors of 1.06–5.21.Under two-stage training, spectral-residual mean VPT increased from N0 to 16N0 by factors of 1.39–5.07 in all six systems.Source: §4.2 · p. 11, Table 2 · p. 12
  • At 16N0, two-stage spectral-residual mean VPT exceeded joint-training VPT in all six systems, and exceeded penalized joint-training VPT in five.Adding the operator-norm penalty increased spectral-residual mean VPT in five systems and decreased it in Lorenz-96; with the penalty, spectral residual exceeded latent prediction at all dimensions and systems.Source: §4.2 · p. 11

4.3 Comparison with baselines

  • Against four baseline families, spectral-residual median VRMSE was lower at nearly every comparison; at 16N0 it beat the lowest baseline median by 1.08–1.76× in five systems.Exceptions were kernel DMD on Rössler at 16N0 and Consistent KAE on Lorenz-96 at N0, 2N0, and 4N0; no baseline family decreased monotonically in more than three systems.Source: §4.3 · p. 11
  • Baseline mean VPTs were shorter than spectral-residual VPTs in all but eight comparisons; seven exceptions were at N ≤ 4N0.In five exceptions, the mean difference was smaller than the baseline’s across-seed sample standard deviation.Source: §4.3 · p. 11, Table 2 · p. 12
  • The baseline comparison uses finite-data estimates at fixed bandwidth, whereas the cited EDMD convergence result is population-level.Source: §4.3 · p. 12

5 Spectral properties and predictions

At the largest evaluated dimension under two-stage training, spectral-residual eigenvalues fell in low-residual regions, a pattern consistent with better prediction performance. Qualitative predictions show how the two model objectives compare across horizons and systems.

5.1 Spectral analysis of the learned operators

  • Under two-stage training, spectral-residual eigenvalues clustered in low-residual regions across all six systems, while latent-prediction eigenvalues also entered high-residual regions; under joint training, spectral-residual eigenvalues extended farther toward the origin.In Lorenz-96, most Prediction eigenvalues clustered near λ = 1 in low-residual regions under both training methods.Source: §5.1 · pp. 13–14

5.2 Qualitative comparison of predictions

  • For Rössler, the two-stage spectral-residual model showed the reference fold at 0.6 LT and the main loop at 1.0 LT, while latent-prediction loops were distorted at 0.2 LT.Adding the operator-norm penalty retained the overall shape for the two-stage spectral-residual model; joint training showed only the planar loop at 1.0 LT.Source: §5.2 · p. 14, Figure 4 · p. 16
  • For Lorenz-63, Duffing, and Mackey–Glass, two-stage spectral-residual predictions retained overall attractor shapes up to 1.0 LT, whereas joint-training shapes were smaller or distorted and latent-prediction distortions appeared by 0.2 LT.Both models distorted Lorenz-96 and Kuramoto–Sivashinsky field structure at 0.6 LT.Source: §5.2 · p. 14

5.1 Spectral analysis of the learned operators

  • At 16N0, the two-stage spectral-residual model had eigenvalues concentrated in low-residual regions, longer mean VPT than the two-stage latent-prediction model and joint spectral-residual model, and lower median windowed VRMSE than the former.The authors interpret this correspondence as consistent with spectral reliability.Source: §5.2 · p. 14

6 Conclusions

The conclusions report more consistent dimension scaling for the spectral-residual model and a convergence result for minimal residuals as learned spaces approximate the observable space. They also identify unresolved questions about how rollout-error scaling depends on its contributing factors.

6 Conclusions

  • Across six chaotic systems, both models reduced median windowed rollout error from smallest to largest dimension, while the spectral-residual model had lower medians and a larger reduction factor in every system.Its median decreased monotonically with dimension in four systems, compared with one for latent prediction; it also achieved favorable VRMSE and VPT results against baseline families, with exceptions.Source: §6 · p. 17
  • For bounded Koopman operators, the minimal residual over learned dictionary spaces converges to the full-space infimum as the spaces approximate the observable space.The paper presents this as a mathematical basis for expecting residual-based spectral information to become more reliable as latent dimension increases.Source: §6 · p. 17
  • The authors leave open how residuals, latent-coordinate prediction-error propagation, and decoding jointly determine rollout-error scaling with latent dimension.Source: §6 · p. 17

Convergence of the minimal residual

Setting

  • The setting fixes a basis of each learned space independently of evaluation samples, with each element represented by a unique coefficient vector.The evaluation matrices, Gram matrix, and cross-correlation matrix are defined for that fixed basis.Source: Setting · p. 18

Assumptions for the data limit

The appendix establishes data-limit and latent-dimension convergence results under stated sampling and space-approximation conditions. It also records a pointwise reconstruction bound for exact latent collisions.

Assumptions for the data limit

  • Under iid sampling or the stated ergodic single-trajectory condition, the empirical Gram matrix is eventually positive definite on a common full-measure set.The trajectory condition requires convergent averages of the integrable Gram observables.Source: Assumptions for the data limit · p. 18
  • For fixed N, the empirical minimal residual converges uniformly on compact subsets of C to the dictionary-space residual as the number of sample pairs grows.The convergence of Gram and cross-correlation matrices yields uniform convergence of the minimized quotients and their square roots.Source: Assumptions for the data limit · p. 18

Assumption for the latent-dimension limit

  • As dictionary spaces approximate the observable space, τN(λ) converges to τ(λ); consequently, if τ(λ) < ϵ, then λ belongs to the dictionary-space ϵ-approximate spectrum for sufficiently large N.Source: Assumption for the latent-dimension… · p. 19

Approximate point pseudospectra

  • Under assumptions A1–A4, the successive limits M → ∞ and N → ∞ recover τ(λ) for every λ on a common full-measure set.The appendix uses this zero-residual characterization to relate the limits to approximate point pseudospectra.Source: Approximate point pseudospectra · p. 19

Reconstruction and exact latent collisions

Appendix B Training and baseline settings

The appendix specifies how initial conditions were generated for the six systems and reports the hardware used for the experiments.

Initial conditions

  • Initial conditions use system-specific reference states, perturbations, or Fourier-mode draws; random seeds vary initial-condition perturbations and minibatch sampling.Burn-in steps are included in the step counts but discarded before the data split.Source: Initial conditions · pp. 19–20

Computational environment

Training protocol

The section specifies training, rollout-operator construction, residual-contour evaluation, and settings for the Koopman autoencoder and baseline models.

Training protocol

  • Training uses reconstruction-pretrained autoencoders for 200 epochs, with 20 warm-up epochs, 180 cosine-decay epochs, and two-stage stopping based on a plateau rule.Two-stage training allows up to 250 updates per stage per epoch; the protocol draws disjoint pools of 2,048 pairs for operator estimation and evolution-loss evaluation.Source: Training protocol · p. 20

Operators for rollouts and validation

  • For rollouts, the spectral-residual model uses the operator estimated from all encoded training pairs, while latent prediction uses a rank-constrained operator after the full operator diverged.The rank retains at least 99.9% of squared singular-value mass; validation uses rank-constrained operators for both models.Source: Operators for rollouts and validation · p. 20

Residual contours

  • Residual contours are evaluated on a 150×150 grid over [−1.45, 1.45]², with color ranges set separately using the combined values of each figure’s eight panels.The SVD retains singular values greater than 10⁻¹⁰ times the largest singular value.Source: Residual contours · p. 20

Baseline settings

  • The Neural ODE and Consistent KAE baselines are selected using their lowest validation-interval VRMSE.Source: Baseline settings · p. 20

Latent-state Neural ODE

  • The latent-state Neural ODE uses one-hidden-layer encoder and decoder networks, a two-hidden-layer latent vector field, and dopri5 integration.Its training objective is mean squared observation-reconstruction error over autonomous evolution for n = q steps.Source: Latent-state Neural ODE · pp. 20–21

Kernel DMD with random Fourier features

  • Kernel DMD uses random Fourier features with γ = 1.0 and a linear readout to reconstruct states after repeated application of the EDMD operator.For reported rollouts, the EDMD operator is estimated at the smallest rank retaining 99.9% of the training feature matrix’s squared singular-value mass.Source: Kernel DMD with random Fourier features · p. 21

Consistent KAE

  • Consistent KAE combines an MLP encoder and decoder with forward and backward evolution matrices, trained using forward-evolution, reconstruction, backward-evolution, and consistency losses.Forward and backward evolution each use four steps.Source: Consistent KAE · p. 21

Echo state network

  • The echo state network uses a tanh reservoir and a ridge-regression readout trained to predict the one-step-ahead input, then advances autoregressively.For prediction, the reservoir is reset to zero and receives up to q preceding observations followed by the starting observation.Source: Echo state network · p. 21

Data Availability

The paper reports that its code and configuration files for generating the data and numerical results are available online; the supplement provides additional figures for two analyses in the main text.

Data Availability

  • The code and configuration files used to generate the data and numerical results are available at the project repository.Source: Data Availability · p. 27
  • The supplement adds figures for two main-text analyses, including spectral plots and predictions for five systems, with time expressed in Lyapunov time.Lyapunov time is specified as the inverse of each system’s largest Lyapunov exponent.Source: Journal: Nonlinear Dynamics · p. 28

S1 Lorenz-63 system

The Lorenz-63 supplement presents spectral plots and predictions; the two-stage spectral-residual model captures the reference attractor’s overall shape through the displayed 1.0 LT horizon.

S1 Lorenz-63 system

  • Through 1.0 LT, the two-stage spectral-residual model captures the reference Lorenz-63 attractor’s overall shape, with two lobes smaller than the reference distinguishable.Source: S1 Lorenz-63 system · p. 29, Figure S2 · p. 30
  • The spectral plots show minimal-residual contours and estimated Koopman eigenvalues for eight model variants, with two-stage training above joint training.Contour colors use a logarithmic scale; white curves mark the unit circle and red points mark eigenvalues.Source: S1 Lorenz-63 system · p. 29
  • The Lorenz-63 predictions are plotted in (x, y, z) coordinates across horizons of 0.2, 0.6, and 1.0 LT, with shared axis limits following the reference trajectory.Source: Figure S2 · p. 30

S2 Duffing system

The supplement reports prediction and spectral-plot observations for Duffing and Mackey–Glass, including differences by training regime and residual structure.

S2 Duffing system

  • For Duffing, the two-stage spectral-residual model captures the reference attractor’s overall shape through 1.0 LT, while its joint-training predictions have noticeably smaller extent from 0.6 LT onward.Source: S2 Duffing system · p. 31, Figure S4 · p. 32
  • Duffing spectral plots compare minimal-residual contours and estimated Koopman eigenvalues across eight variants, with two-stage training above joint training.Prediction panels use the (x, ẋ) plane and horizons of 0.2, 0.6, and 1.0 LT.Source: S2 Duffing system · p. 31, Figure S4 · p. 32

S3 Mackey–Glass system

  • For Mackey–Glass, the two-stage spectral-residual panels show low-residual regions forming a band along the unit circle.Source: S3 Mackey–Glass system · p. 33, Figure S5 · p. 33
  • Mackey–Glass predictions are plotted in the delay plane (x(t), x(t − τ)) with τ = 17, across 0.2, 0.6, and 1.0 LT horizons.Rows distinguish horizons and columns compare the reference with latent-prediction and spectral-residual variants, with or without the operator-norm penalty.Source: Figure S6 · p. 34

S4 Lorenz-96 system

The Lorenz-96 and Kuramoto–Sivashinsky supplements show spectral plots and prediction fields for eight model variants. Lorenz-96 results include eigenvalue clustering patterns and a dimension comparison, while both systems show distorted field structure at 0.6 LT.

S4 Lorenz-96 system

  • Lorenz-96 spectral-residual mean VPT rose from 0.198 to 0.211 LT as dimension increased from N0 to 2N0, while median VRMSE rose from 0.9959 to 0.9980.This comparison is for two-stage training.Source: S4 Lorenz-96 system · p. 35
  • Under two-stage and joint training, most Lorenz-96 latent-prediction eigenvalues clustered in low-residual regions near λ = 1, with more scattered into high-residual regions under two-stage training.The plots show minimal-residual contours and estimated eigenvalues for eight variants; +L2 denotes an operator-norm penalty.Source: S4 Lorenz-96 system · p. 35

S4 Lorenz-96 system; S5 Kuramoto–Sivashinsky system

S5 Kuramoto–Sivashinsky system

  • The Kuramoto–Sivashinsky supplement presents spectral-residual contours and estimated eigenvalues for eight variants, alongside prediction fields across horizons of 0.2, 0.6, and 1.0 LT.The spectral plots use a logarithmic residual-contour scale; the prediction panels plot spatial samples on a domain of length 22, with color showing field value.Source: S5 Kuramoto–Sivashinsky system · p. 37, Figure S10 · p. 38
Loading 2610.11866v1…