PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-driven discovery of intrinsic dynamics

Daniel Floryan, Michael D. Graham

arXiv:2108.05928v2cs.LG

TL;DR

High-dimensional state descriptions can obscure system behavior. This paper learns intrinsic-dimensional representations and dynamics from data using multiple charts, and examples show accurate models whose dimension matches the system’s intrinsic dimensionality.

  • Problem

    High-dimensional dynamical models can obscure the underlying behavior of a system.

    Source: §I · p. 2

  • Method

    The method learns minimal-dimensional dynamical models from high-dimensional time-series data by finding and parameterizing representations of minimal dimensionality.

    Source: §V · pp. 19–20

  • Results

    Examples show accurate dynamical models with dimension equal to the system’s intrinsic dimensionality, while single-chart methods cannot produce accurate models.

    Source: §V · p. 19

  • Takeaways & Limitations

    Multiple charts support models that capture the system’s dynamics at its intrinsic dimensionality.

    Source: §V · p. 19

    The global manifold-learning examples can mix points of dissimilar colors in Cartesian coordinates.

    Source: S1. LIMITATIONS OF GLOBAL MANIFOLD… · p. 28

Abstract

from arXiv · show

Dynamical models underpin our ability to understand and predict the behavior of natural systems. Whether dynamical models are developed from first-principles derivations or from observational data, they are predicated on our choice of state variables. The choice of state variables is driven by convenience and intuition, and in the data-driven case the observed variables are often chosen to be the state variables. The dimensionality of these variables (and consequently the dynamical models) can be arbitrarily large, obscuring the underlying behavior of the system. In truth, these variables are often highly redundant and the system is driven by a much smaller set of latent intrinsic variables. In this study, we combine the mathematical theory of manifolds with the representational capacity of neural networks to develop a method that learns a system's intrinsic state variables directly from time series data, and also learns predictive models for their dynamics. What distinguishes our method is its ability to reduce data to the intrinsic dimensionality of the nonlinear manifold they live on. This ability is enabled by the concepts of charts and atlases from the theory of manifolds, whereby a manifold is represented by a collection of patches that are sewn together -- a necessary representation to attain intrinsic dimensionality. We demonstrate this approach on several high-dimensional systems with low-dimensional behavior. The resulting framework provides the ability to develop dynamical models of the lowest possible dimension, capturing the essence of a system.

I. INTRODUCTION

The introduction and manifold primer explain how CANDyMan uses multiple local charts to learn minimal-dimensional dynamical models from data. They motivate this atlas-based approach by showing that a single-chart representation may require a higher-dimensional representation.

I. INTRODUCTION

  • High-dimensional dynamical models can obscure system behavior, making analysis difficult and predictions prohibitively expensive.The introduction motivates learning low-dimensional models directly from time-series data.Source: §I · p. 2
  • CANDyMan aims to learn minimal-dimensional, lossless latent representations and dynamical models directly from data, reducing observations to their nonlinear manifold’s intrinsic dimensionality.The paper states this ability in principle.Source: §I · p. 2
  • It represents a manifold with overlapping patches, uses autoencoders to reduce each patch and neural networks to model its dynamics, then stitches the patches into a global model.This avoids requiring a global parameterization of the manifold.Source: §I · pp. 2–3

II. A PRIMER ON MANIFOLDS

  • A circle needs at least two overlapping charts for a continuous one-dimensional representation, because a single global coordinate would be discontinuous or non-one-to-one.An atlas is a collection of charts whose coordinate domains cover the manifold.Source: §II · p. 4
  • A one-dimensional autoencoder can reconstruct circle training data well yet produce a gap on the full circle and fail to generalize, regardless of training-data amount or network architecture.The failure follows from the circle’s geometry and illustrates the limitation of global manifold-learning methods.Source: §II · pp. 4–5
  • A multi-chart method can reduce an n-dimensional manifold to n dimensions, while a single-chart method may require 2n dimensions in the worst case.For example, a single-chart autoencoder can represent a circle in three-dimensional space in two dimensions, but not at its intrinsic dimension of one.Source: §II · p. 5
  • An example in Section IV C suggests that the practical gap between multi-chart and single-chart methods is greater than the theoretical gap.Source: §II · p. 5
  • Prior chart-based approaches either globalize manifold parameterization or use coordinate maps that can cause discontinuities or non-smoothness in dynamics.Brand uses charts for a global parameterization, while Pitelis et al. use linear coordinate maps; Schonsheck et al. [34] would also lead to non-smooth charts.Source: §II · pp. 5–6
  • A single-chart autoencoder's circle representation develops a gap on the full circle because its encoder cannot be one-to-one.Source: §II · p. 6

III. LEARNING AN ATLAS AND DYNAMICS ON A MANIFOLD

The method learns a data manifold and its dynamics from time-series observations, representing the manifold with an atlas of local charts and modeling dynamics in each chart’s coordinates.

III. LEARNING AN ATLAS AND DYNAMICS ON A MANIFOLD

  • The data are pairs of states and their successors, sampled at intervals Δt; for continuous systems, the paired states are assumed to come from the continuous trajectory.Source: §III · p. 7
  • The motivating setting is dynamics on a submanifold of dimension n ≪ m, including dissipative systems whose long-time behavior can reside on a finite-dimensional submanifold.Source: §III · p. 7
  • The goal is to learn the data manifold and the dynamics on it directly from observations.Source: §III · p. 7
  • The method first learns an atlas of charts, then learns the dynamics in each chart’s local coordinates and patches those descriptions into a global one.Source: §III · p. 7
  • A particle moving at constant speed around the unit circle illustrates the method on a one-dimensional manifold embedded in R2.Source: §III · p. 7

A. Learning an atlas

The atlas is built by creating overlapping data patches, fitting a coordinate map and approximate inverse for each patch, and learning dynamics in the resulting local coordinates.

A. Learning an atlas

  • K-means partitions the data, then a nearest-neighbor graph expands clusters so neighboring coordinate domains overlap.Source: A. Learning an atlas · pp. 7–8
  • Figure 3D reports that the first and 25th motion cycles are well predicted in the circle example.Source: A. Learning an atlas · p. 8
  • For each cluster, a neural autoencoder learns a dimension-reducing coordinate map and approximate inverse; latent dimension is estimated from reconstruction error or covariance rank.Source: A. Learning an atlas · p. 8
  • Transition maps between charts are approximated by composing one chart’s encoder with another chart’s decoder.Source: A. Learning an atlas · p. 8
  • Because local coordinates may differ and distort data distributions, normalizing them can help when learning dynamics.Source: A. Learning an atlas · p. 9

B. Learning the dynamics

The method learns low-dimensional dynamics in each chart, then links local models through chart transitions. It encourages smooth transitions by keeping overlapping charts’ reconstructions close.

B. Learning the dynamics

  • Each chart’s local dynamics map is approximated by a neural network, yielding dynamics directly in low-dimensional coordinates.The map takes a point’s local coordinates through the manifold dynamics and back into local coordinates.Source: B. Learning the dynamics · p. 9
  • The training loss compares each chart’s predicted next local coordinate with the encoded next training point.The index set Iα tracks which training data belong to chart α.Source: B. Learning the dynamics · p. 9
  • To evolve a state globally, the method selects a chart by its nearest centroid, advances with its local model, and switches charts when the state reaches a border point.The transition uses the new chart’s coordinate map; subsequent evolution follows that chart’s local dynamics.Source: B. Learning the dynamics · pp. 9–10
  • Chart transitions appear seamless in the reported example.Source: B. Learning the dynamics · p. 10

C. Smooth transitions

  • Overlapping charts can have discontinuous dynamics if their reconstructions of the same point differ significantly.The difference between reconstructions is bounded by the sum of the two charts’ reconstruction errors, motivating separate autoencoder training with greater weight on shared-domain data.Source: C. Smooth transitions · pp. 10–11
  • The authors did not weight shared-domain data in the reported results; appreciable effects appeared only at weights that made single-domain reconstructions poor.They note this weighting may matter in future work.Source: C. Smooth transitions · p. 11

D. Time complexity

The method’s computational cost is dominated by nearest-neighbor searches, whose efficiency degrades in higher dimensions. Examples include a torus model that reduces the state to two dimensions, with accurate but imperfect learned dynamics.

D. Time complexity

  • A forward step or chart transition requires a neural-network pass, while the major expense is nearest-neighbor search.With a k-d tree, search is O(log2(N)) at constant search-space dimension, but approaches brute-force O(dN) when d ≳10.Source: D. Time complexity · p. 11

IV. EXAMPLES

  • The examples progress from a planar periodic orbit to complex bursting dynamics of the Kuramoto-Sivashinsky equation.Source: §IV · p. 12

A. Quasiperiodic dynamics on a torus

  • The method reduces the torus particle’s state from three observed coordinates to the two-dimensional submanifold’s intrinsic dimension.The authors state that charts and atlases are needed for this reduction.Source: A. Quasiperiodic dynamics on a torus · p. 12
  • Chart transitions in the torus example are apparently smooth, though higher-order differences reveal hiccups because decoders only approximate encoder inverses.The figure’s longer-time predictions show a small amount of phase drift.Source: A. Quasiperiodic dynamics on a torus · pp. 12–13

B. Reaction-diffusion system

This section uses multi-chart atlases to learn low-dimensional dynamics from reaction-diffusion and Kuramoto–Sivashinsky PDE data, including long-term predictions and bursting behavior. It also examines how latent dimension and chart choice relate to capturing the qualitative dynamics of Kuramoto–Sivashinsky bursting.

B. Reaction-diffusion system

  • For five models, three charts yielded an average mean squared error of 2.82 × 10−6; the data form a one-dimensional submanifold of R20402, with local one-dimensional dynamics for each chart.Source: B. Reaction-diffusion system · p. 14
  • The reaction-diffusion model predicts the state 250 time units ahead after training on 10 time units, with a roughly 0.4% phase-speed error.The prediction is nearly indistinguishable from ground truth; the phase-speed error can be improved with further training.Source: B. Reaction-diffusion system · pp. 14–15

C. Kuramoto-Sivashinsky bursting dynamics

  • For K-S bursting dynamics, six charts correspond to two saddle points and four connecting heteroclinic orbits, yielding a three-dimensional latent model.The data are discretized at 64 grid points and clustered automatically with k-means.Source: C. Kuramoto-Sivashinsky bursting… · pp. 15–16
  • The six-chart atlas reconstructs the three-dimensional K-S submanifold with an order-of-magnitude lower error than a six-dimensional one-chart model with the same trainable parameters.Reconstruction error plateaus at latent dimension three.Source: C. Kuramoto-Sivashinsky bursting… · p. 16, Figure 6 · p. 17
  • Learning K-S dynamics required normalized local coordinates and off-attractor training data because the attractor is thin, yet many learned models still became periodic.Periodic outcomes included one or two stable limit cycles or a longer cycle joining all four heteroclinic orbits; the authors attribute these outcomes to a gluing bifurcation.Source: C. Kuramoto-Sivashinsky bursting… · pp. 16, 18
  • The learned K-S model reproduces qualitatively correct non-periodic bursting and quasi-steady states, but differs in how long it remains in those states.The dwell-time discrepancy reflects sensitivity near saddle points, where time spent scales logarithmically with distance from the stable manifold.Source: C. Kuramoto-Sivashinsky bursting… · p. 18
  • The six-dimensional latent model settles to a fixed point off the true attractor, so it does not reflect the true dynamics.Source: C. Kuramoto-Sivashinsky bursting… · p. 18
  • Although higher-dimensional one-chart models can theoretically capture the attractor by the strong Whitney embedding theorem, in practice they fail to capture this complex system’s qualitative dynamics.Source: C. Kuramoto-Sivashinsky bursting… · p. 18

V. DISCUSSION

CANDyMan learns minimal-dimensional dynamical models directly from high-dimensional time series using multiple charts. The discussion highlights their advantages over single-chart methods, practical choices still to make, and possible applications.

V. DISCUSSION

  • The approach parameterizes the low-dimensional manifold from high-dimensional time series, learns dynamics on it, and sews local models into a global model.The charts are mutually independent in the implementation, making training embarrassingly parallelizable.Source: §V · p. 19
  • CANDyMan learns accurate dynamical models at the system’s intrinsic dimension, while single-chart methods cannot achieve accurate models of intrinsic dimensionality.The discussion says this limitation holds irrespective of training-data amount or network architecture.Source: §V · p. 19
  • The multi-chart method outperforms an otherwise equal single-chart method, even where the strong Whitney embedding theorem suggests equal performance.The authors speculate this may be because local geometry is easier to learn than global geometry.Source: §V · p. 19
  • Independent local descriptions avoid the dilution problem that can cause single-chart models to approximate rare but dynamically important excursions poorly.With constant-interval sampling, brief excursions contribute little to a single-chart training loss when more frequently sampled regions dominate.Source: §V · pp. 19–20
  • The number of charts and the minimal dimension remain open choices; the authors suggest empirical sampling over both, with dimension tuning potentially preceding chart-count tuning.They note that a single chart limits achievable dimension reduction and that graph theory or topological data analysis may guide chart selection.Source: §V · p. 20
  • Future work could impose coordinate-map structure, such as isometry or interpretability, if it makes learning the correct dynamics easier.The theory is agnostic to coordinate-map form as long as the maps are homeomorphic.Source: §V · p. 20
  • CANDyMan may reveal hidden system properties such as conserved quantities, though translating intrinsic variables into interpretable physics remains future work.Source: §V · p. 21
  • Because it decomposes state space into patches, CANDyMan may suit systems with distinct regional behaviors, including turbulent flows, neuronal activity, and solar flares.Source: §V · p. 21

S1. LIMITATIONS OF GLOBAL MANIFOLD LEARNING METHODS

Global, single-chart manifold learning can fail to give continuous, one-to-one coordinates for a circle when forcing its two-dimensional data into one dimension. The resulting overlaps follow from the circle’s geometry, not the chosen hyperparameters.

S1. LIMITATIONS OF GLOBAL MANIFOLD LEARNING METHODS

  • The issue is that global manifold-learning methods seek a single latent representation, which can yield discontinuities or coordinate maps that are not one-to-one.The circle example demonstrates the effect across several methods, alongside the autoencoder example discussed in the main text.Source: S1. LIMITATIONS OF GLOBAL MANIFOLD… · p. 28
  • Kernel PCA, Isomap, Laplacian eigenmaps, Hessian eigenmaps, locally linear embedding, and t-SNE all produce one-dimensional representations with this mixing.The comparison applies these methods to 20 evenly spaced points on a unit circle represented in two-dimensional Cartesian coordinates.Source: S1. LIMITATIONS OF GLOBAL MANIFOLD… · p. 28
  • One-dimensional global coordinates collapse distinct parts of the circle onto the same latent point, mixing points that are far apart on the circle.The effect is attributed to the circle’s closed geometry and was verified to be independent of the methods’ hyperparameter values.Source: S1. LIMITATIONS OF GLOBAL MANIFOLD… · p. 28

S2. COMPARING CANDYMAN TO SINGLE-CHART METHODS

In the reaction-diffusion comparison, a multi-chart one-dimensional model closely matches the ground truth, whereas the single-chart one-dimensional model produces inaccurate trajectories and sharp jumps. This illustrates how multiple charts can support accurate models at intrinsic dimensionality.

S2. COMPARING CANDYMAN TO SINGLE-CHART METHODS

  • The comparison uses the same reaction-diffusion data and, for the prior method, the same architecture, training procedure, and training data, changing latent dimension for the one- versus two-dimensional models.The data consist of 128 × 128 image frames.Source: S2. COMPARING CANDYMAN TO SINGLE-CHART… · p. 29
  • The multi-chart one-dimensional model produces trajectories nearly indistinguishable from ground truth, while the single-chart one-dimensional model is inaccurate and shows sharp jumps.The single-chart two-dimensional model is also nearly indistinguishable from ground truth.Source: S2. COMPARING CANDYMAN TO SINGLE-CHART… · pp. 29–30
  • The authors suggest combining CANDyMan with the methods of and [52] to pursue more accurate and interpretable models at intrinsic dimensionality.They note that the earlier intrinsic-dimensionality method [52] is also single-chart.Source: S2. COMPARING CANDYMAN TO SINGLE-CHART… · pp. 29–30

S3. ADDITIONAL EXAMPLES

A. Periodic dynamics on a torus

Across the additional systems, CANDyMan learns low-dimensional dynamics from high-dimensional data using multiple local charts. The reported models track the reference dynamics, including a traveling-wave case with periods spanning nearly 200-fold different timescales.

A. Periodic dynamics on a torus

  • The torus example’s learned global dynamics match the true dynamics closely, with seamless transitions between its three charts.The particle’s periodic orbit lies on a one-dimensional submanifold of R3, represented using three clusters and local dynamical models.Source: A. Periodic dynamics on a torus · p. 30

B. Kuramoto-Sivashinsky beating dynamics

  • For the Kuramoto-Sivashinsky beating standing wave, the method learns a minimal one-dimensional dynamical model for nominally 64-dimensional data, with a trajectory described as indistinguishable from the data.The reconstruction from local coordinates is also reported as excellent.Source: B. Kuramoto-Sivashinsky beating dynamics · p. 32

C. Kuramoto-Sivashinsky beating travelling dynamics

  • For the travelling-wave case, the method separates shape and phase and predicts phase change from the shape variable for each chart.The learned model has good results and seamless chart transitions; the travelling-period discrepancy improves with increased neural-network training time.Source: C. Kuramoto-Sivashinsky beating… · pp. 33–34
  • 94.88 ± 0.10 time units for the model’s travelling period, compared with 90.46 ± 0.10 time units for the data; its beating period is 0.456 ± 0.001 time units, identical to the data.The two periods differ by a factor of almost 200, and the model captures the long timescale despite training data spanning a small fraction of it.Source: C. Kuramoto-Sivashinsky beating… · pp. 34–35

S4. DETAILS OF DATASETS AND MODELS

The supplementary experiments use neural networks with a common training setup and choose latent dimensions and chart counts using known structure or empirical checks. Architectures and hyperparameters were not systematically optimized.

S4. DETAILS OF DATASETS AND MODELS

  • Neural networks used fully connected layers, mean squared error, Adam, Glorot uniform initialization, and full-chart batches unless otherwise indicated.The authors did not attempt to optimize architectures or hyperparameters, using settings that produced accurate results.Source: S4. DETAILS OF DATASETS AND MODELS · p. 35
  • Latent dimension was chosen from known submanifold dimension or reconstruction-error behavior, with the bursting example assigned dimension 3 when error plateaued there.Dimension 2 was ruled out for bursting because it could not produce the observed dynamics.Source: S4. DETAILS OF DATASETS AND MODELS · p. 35
  • Chart counts were often chosen near a known minimum; the bursting example used six charts after exploratory analysis identified two saddles connected by four heteroclinic connections.Source: S4. DETAILS OF DATASETS AND MODELS · p. 35

1. Periodic dynamics on a circle

The examples specify sampled data and chart construction for periodic, quasiperiodic, reaction-diffusion, and bursting dynamics. The bursting example also normalizes chart-local coordinates before learning dynamics.

1. Periodic dynamics on a circle

  • The circle example samples a particle moving clockwise at constant speed at 40 points, spaced π/20 radians apart on the unit circle.Source: §1 · p. 35

2. Quasiperiodic dynamics on a torus

  • In the torus example, motion is constant in both directions, with poloidal motion three times faster; 1,000 data points span 15 full periods, generating a quasiperiodic orbit.Source: §1 · p. 36

3. Reaction-diffusion system

  • For the reaction-diffusion example, 200 samples of the u and v fields capture a spiral wave over slightly more than one period on a 101 × 101 grid.The model uses three charts, four-nearest-neighbour graph connections, and one cluster expansion.Source: §1 · pp. 35, 37

4. Kuramoto-Sivashinsky bursting dynamics

  • The Kuramoto–Sivashinsky bursting dataset contains 6,565 field-value data points sampled at 64 equally spaced spatial locations, 0.05 time units apart across 16 bursts; its dynamics dataset has 2,189 slightly-off-attractor trajectories.Each trajectory spans one time unit and contains 21 points spaced 0.05 time units apart. The trajectories came from perturbing every third autoencoder-dataset point.Source: §1 · pp. 38–39
  • Before learning bursting dynamics, each chart's local coordinates are centered, transformed to a PCA basis without dimension reduction, and scaled by coordinate standard deviation.The authors report normalization helped learn accurate dynamics and hypothesize it softens sharp cusps near saddle points.Source: §1 · p. 39

B. Additional examples from Supplementary Information

The supplementary examples extend the demonstrations to periodic torus motion and Kuramoto-Sivashinsky beating waves. Their descriptions specify the sampled trajectories and chart-graph settings.

1. Periodic dynamics on a torus

  • The periodic torus example samples 100 particle coordinates across one full period, with toroidal samples spaced π/50 radians apart.It uses three charts, connects points to two nearest neighbours, and expands clusters twice along the graph.Source: B. Additional examples from… · p. 40

2. Kuramoto-Sivashinsky beating dynamics

  • The Kuramoto-Sivashinsky beating standing-wave example uses 100 snapshots of a 64-point field, covering slightly more than two beating periods.It uses three charts, four-nearest-neighbour connections, and one cluster expansion.Source: B. Additional examples from… · p. 41

3. Kuramoto-Sivashinsky beating travelling dynamics

  • After transients decay, the beating travelling-wave dataset contains 100 field snapshots at 64 equally spaced spatial points, sampled uniformly 0.01 time units apart, covering slightly more than two beating periods and 1% of the travelling period.Source: B. Additional examples from… · p. 41

Related papers

Loading 2108.05928v2…