Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems

Ryan Lopez; Paul J. Atzberger

Source-linked AI summary

Variational Autoencoders for Learning Nonlinear Dynamics of Physical Systems

Ryan Lopez, Paul J. Atzberger

arXiv:2012.03448v2Dec 7, 2020cs.LGcs.AIeess.SYmath.DGmath.DS

TL;DR

The paper addresses how to learn parsimonious, robust representations of nonlinear physical dynamics while incorporating geometric and topological information. It develops VAE-based nonlinear state-space models with general manifold latent spaces and evaluates them on Burgers equations and constrained mechanics. The authors conclude that these geometric and topological structures support robust learning of nonlinear PDE and mechanical dynamics.

Problem
Learning nonlinear physical dynamics requires representations that capture structure beyond observed outputs and support stable longer-term extrapolation.
Method
The paper develops VAE-based nonlinear state-space models with probabilistic encoders and decoders, latent evolution maps, and prescribed geometric and topological manifold spaces.
Results
The methods are demonstrated for nonlinear Burgers equation dynamics and constrained mechanical systems, with geometric and topological structures used for robust learning.
Takeaways & Limitations
Geometric and topological priors provide a framework for learning reduced nonlinear dynamical models from physical observations.

Abstract

from arXiv · show

We develop data-driven methods for incorporating physical information for priors to learn parsimonious representations of nonlinear systems arising from parameterized PDEs and mechanics. Our approach is based on Variational Autoencoders (VAEs) for learning from observations nonlinear state space models. We develop ways to incorporate geometric and topological priors through general manifold latent space representations. We investigate the performance of our methods for learning low dimensional representations for the nonlinear Burgers equation and constrained mechanical systems.

Introduction

Learning nonlinear dynamics remains challenging because nonlinear systems exhibit rich behaviors and lack unified identification approaches. This work targets robust, structured representations that support longer-term extrapolation.

Nonlinear system identification has fewer unified approaches because nonlinear dynamics exhibit rich behaviors.
A central challenge is inferring representations that support stable multi-step extrapolation across future steps and input states.
The proposed methods incorporate smoothness, periodicity, topology, and other constraints from underlying physical systems.
Probabilistic Autoencoders use noise-based regularization and priors to learn lower-dimensional representations from observations.

Variational Autoencoders (VAEs)

The paper uses VAEs to learn probabilistic encoders, latent evolution maps, and decoders for nonlinear dynamical predictions. ELBO-based regularization and latent priors promote reconstruction, organization, and parsimonious representations.

VAEs learn lower-dimensional representations from observation data for predicting dynamical evolution, including experimental measurements and computational simulations.
Noise in probabilistic encoders and decoders promotes smoother, more connected latent encodings and can support multi-step consistency.
The ELBO-based loss combines reconstruction and KL terms, with the negative ELBO providing a regularized maximum-likelihood objective.
A latent-space map fθℓ evolves encoded representations between observed input and output state functions.
The KL prior encourages feature disentanglement, reference scale, and parsimonious codes using fewer dimensions when possible.
The model encodes states into z(t), evolves z(t) to z(t+τ), and decodes the evolved representation to predict the next state.

Topology

The method prescribes non-Euclidean manifold latent spaces to encode geometric and topological structure. This enables controllable topology and geometry while supporting more parsimonious, robust, interpretable representations.

Encoders map observations through an intermediate Euclidean representation onto a prescribed manifold M with specified topology.
The manifold projection can use either an analytic map or a quantized nearest-point map based on a point-cloud representation and local gradients.
Sphere, torus, and Klein bottle latent spaces provide topologies intrinsically different from R^n and allow priors such as uniform compact-manifold distributions.
Additional latent structure can reduce sensitivity to noise, simplify decoder learning, and aid identifiability and interpretability.

Related Work

Prior dynamical autoencoder work largely uses Euclidean latent spaces, while this paper extends representations to general manifolds and applies them to PDEs and constrained mechanics.

RNN-based autoencoders offer rich sequential approximation classes but present interpretability and multi-step robustness challenges for dynamical systems.
Earlier VAE approaches for dynamical systems primarily use Euclidean latent spaces and include human-motion and ODE applications.
This work develops non-orientable and point-cloud manifold representations with controllable topology and geometry, including shape merging, subtraction, stretching, and contraction.
The methods are applied to nonlinear Burgers equations and constrained mechanical systems, with additional regularizations for multi-step prediction and interpretability.

Results

The VAE methods learn reduced representations for nonlinear Burgers dynamics and constrained mechanical systems by combining latent-space dynamics with geometric and topological priors. Results show accurate low-dimensional prediction, structured latent representations, extrapolation beyond training inputs, and improved reconstruction under compatible manifold constraints and noise.

Burgers’ Equation: The viscous Burgers equation models advective transport and shocks under periodic boundary conditions and serves as a benchmark for computational methods.
Burgers’ Equation: The learned Burgers models extract latent variables from u(x, t) to predict u(x, t + τ), using linear latent evolution with exponential decay.The discrete latent update is z_n+1 = exp(−λ_0τ) · z_n.
Burgers’ Equation: Errors below 1% are achieved with 2 latent dimensions for Burgers dynamics, while DMD with 3 modes encounters challenges approximating the nonlinear evolution.The linear VAE with 2 modes provides some improvement by using different linear spaces for input and output functions, at additional computational cost.
Burgers’ Equation: The reconstruction regularization aligns successive latent representations, facilitating multi-step prediction and supporting some extrapolation to inputs outside the training set.Varying β improved multiple-step accuracy at larger values, whereas smaller values improved single-step accuracy.
Constrained Mechanics: Manifold latent spaces represent constrained mechanics on torus and Klein bottle geometries, learning segment-length constraints and nearly decoupled coordinates for torus motions.For the Klein bottle dataset, the model learns two segment motions that generate configurations.
Constrained Mechanics: Compatible manifold latent spaces improve reconstruction, whereas incompatible R2 topology significantly deteriorates accuracy and larger Euclidean spaces can decrease accuracy through noise overfitting.Under noisy inputs, manifold restrictions improve reconstruction by filtering representations; the reported errors are L2-relative reconstruction errors.

Conclusions

The paper develops VAEs for robust nonlinear dynamics learning using latent representations with general geometric and topological structures, demonstrated on PDEs and constrained mechanical systems.

The developed VAEs learn robust nonlinear dynamics of physical systems through latent representations with general geometric and topological structures.
The methods are demonstrated for learning nonlinear dynamics in PDEs and constrained mechanical systems.
The authors expect the methods to leverage prior geometric and topological knowledge in other physics-related nonlinear-system tasks.

Appendix A: Backpropogation of Encoders for Non-Euclidean Latent Spaces given by General Manifolds

The appendix constructs learnable encoder maps from Euclidean inputs to general manifold latent spaces by projecting neural-network outputs onto embedded manifolds and differentiating the projection for backpropagation.

Encoder maps Eθ = Λ(˜Eθ(x)) combine a neural network from R^d to R^2m with a manifold mapping to support general latent geometries and topologies.The manifold is represented as a point cloud embedded in R^2m, with local gradient information used computationally.
The resulting gradients make the manifold encoder maps learnable within general backpropagation frameworks.The approach evaluates derivatives of the projection and its local parameterization after determining the closest point.
For an m-dimensional manifold, Λ maps w ∈ R^2m to the nearest manifold location z ∈ M by solving a squared-distance optimization.The nearest point is represented through a local chart z* = σk*(u*).
The closest-point solution is characterized using local coordinate charts and their gradients, with the input w and chart coordinates u* determined by the optimization.
The Implicit Function Theorem provides gradients ∇wΛ(w) needed to backpropagate through variations of the manifold projection.These gradients are assembled with local derivatives and the numerically determined closest point for the manifold encoder.