PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiscale Simulations of Complex Systems by Learning their Effective Dynamics

Pantelis R. Vlachas, Georgios Arampatzis, Caroline Uhler, Petros Koumoutsakos

arXiv:2006.13431v3physics.comp-phcs.LGnlin.CD

TL;DR

Predictive simulation of multiscale complex systems requires capturing effective dynamics, but full-resolution simulations are costly and reduced-order models can be limited. LED combines autoencoders, recurrent networks, and Equation-Free modeling to learn and propagate latent dynamics, achieving accurate forecasts with substantial computational savings.

  • Problem

    Accurately forecasting multiscale complex dynamical systems while reducing the cost of full-resolution simulation remains a central challenge.

  • Method

    LED combines autoencoders for fine-to-coarse mappings with recurrent neural networks that propagate latent dynamics, optionally alternating with microscale simulations.

  • Results

    LED achieved order-of-magnitude lower approximation error than competing Equation-Free approaches and was up to two orders of magnitude faster than full solvers across benchmark systems.

  • Takeaways & Limitations

    LED provides a systematic speed–accuracy tradeoff for forecasting effective dynamics in multiscale systems.

  • Takeaways & Limitations

    LED lacks a mechanism to determine when to upscale latent dynamics and may not generalize to dynamical regions unlike those in its training data.

Abstract

from arXiv · show

Predictive simulations of complex systems are essential for applications ranging from weather forecasting to drug design. The veracity of these predictions hinges on their capacity to capture the effective system dynamics. Massively parallel simulations predict the system dynamics by resolving all spatiotemporal scales, often at a cost that prevents experimentation while their findings may not allow for generalisation. On the other hand reduced order models are fast but limited by the frequently adopted linearization of the system dynamics and/or the utilization of heuristic closures. Here we present a novel systematic framework that bridges large scale simulations and reduced order models to Learn the Effective Dynamics (LED) of diverse complex systems. The framework forms algorithmic alloys between non-linear machine learning algorithms and the Equation-Free approach for modeling complex systems. LED deploys autoencoders to formulate a mapping between fine and coarse-grained representations and evolves the latent space dynamics using recurrent neural networks. The algorithm is validated on benchmark problems and we find that it outperforms state of the art reduced order models in terms of predictability and large scale simulations in terms of cost. LED is applicable to systems ranging from chemistry to fluid mechanics and reduces the computational effort by up to two orders of magnitude while maintaining the prediction accuracy of the full system dynamics. We argue that LED provides a novel potent modality for the accurate prediction of complex systems.

I. LEARNING THE EFFECTIVE DYNAMICS (LED)

LED learns effective dynamics by encoding high-dimensional system states into a low-dimensional latent manifold, propagating that representation with a recurrent neural network, and decoding it back to the original state space. Its multiscale forecasting scheme transfers information between coarse and fine scales while reducing repeated high-dimensional evolution and mitigating iterative prediction errors.

  • Representation learning: LED encodes high-dimensional states st ∈ R^ds into low-dimensional latent states zt ∈ Z ⊂ R^dz, with dz ≪ ds, and decodes them back to reconstructed states.The encoder and decoder are trained by minimizing mean squared reconstruction error.
  • Latent dynamics: An RNN propagates the latent dynamics while maintaining an internal hidden state to capture non-Markovian memory effects.The hidden-to-hidden and hidden-to-output mappings can be implemented with an LSTM.
  • Latent dynamics: The RNN updates its hidden memory from current and previous latent states, forecasts future latent states, and is trained by backpropagation through time to minimize forecasting loss.The forecasting loss is ||z̃t+∆t − zt+∆t||2.
  • Training: The autoencoder and LSTM are trained on fully resolved simulations either sequentially, as AE-LSTM, or jointly by minimizing reconstruction and prediction losses.Sequential training pretrains the autoencoder before latent-space prediction training; joint training combines both losses.
  • Multiscale forecasting: LED transfers information between coarse and fine scales through the autoencoder and propagates latent dynamics without lifting to the high-dimensional state at every timestep.Its multiscale forecasting scheme is inspired by Equation-Free modeling and addresses the propagation of errors from iterative latent-space integration.

II. RESULTS

The results demonstrate LED across benchmark problems and compare it with existing state-of-the-art algorithms. Additional results extend the evaluation to alanine dipeptide in water using stochastic molecular dynamics components.

  • LED is applied to several benchmark problems and compared with existing state-of-the-art algorithms.
  • Additional results evaluate LED on alanine dipeptide in water.These results are provided in SI Section 3D.
  • The molecular system’s stochastic dynamics are handled with an MD decoder and an MD-LSTM in latent space.

A. FitzHugh-Nagumo Model (FHN)

The study applies LED to the FitzHugh–Nagumo equations, modeling periodic activator–inhibitor dynamics with distinct fast and slow timescales. It evaluates LED against CSPDE-based EFF variants using MNAD forecasts of inhibitor density.

  • A. FitzHugh-Nagumo Model (FHN): LED captures the FitzHugh–Nagumo equations, which describe activator and inhibitor density evolution on a spatial domain.The activator is the fast variable and the inhibitor is the slow variable.
  • A. FitzHugh-Nagumo Model (FHN): The FHN system uses ϵ = 0.006 to control the timescale difference, with Du = 1, Dv = 4, L = 20, α0 = −0.03 and α1 = 2.The system evolves periodically under two timescales.
  • A. FitzHugh-Nagumo Model (FHN): The equations are discretized with N = 101 grid points and solved by the Lattice Boltzmann method using δt = 0.005.Data from 6 initial conditions forms the fine-grained solution, then every 200th point is retained to obtain ∆t = 1.
  • A. FitzHugh-Nagumo Model (FHN): Accuracy is measured by MNAD on inhibitor density, comparing model forecasts with Lattice Boltzmann ground truth across 32 initial conditions.All propagators use a warm-up period of Twarm = 60.
  • A. FitzHugh-Nagumo Model (FHN): LED is benchmarked against EFF variants based on coarse-level PDE identification, including CSPDE models using Gaussian processes or neural networks, through final time Tf = 451.The comparison uses MNAD consistently with prior work.

B. Kuramoto-Sivashinsky

For the Kuramoto–Sivashinsky equation, LED learns a low-dimensional inertial-manifold representation and uses latent dynamics to forecast chaotic behavior. A CNN-LSTM latent model reproduces long-term climate dynamics while achieving over two orders of magnitude speed-up versus the microsolver.

  • B. Kuramoto-Sivashinsky: The KS equation has a low-dimensional inertial manifold under high dissipation and small domain size, and LED is used to learn its effective dynamics.The inertial manifold attracts neighboring states exponentially after a transient period.
  • B. Kuramoto-Sivashinsky: L = 22 yields a structurally stable chaotic attractor, simulated on 64 grid points with data subsampled to Δt = 0.25 for LED.Training and validation each use 15 · 10^3 samples.
  • B. Kuramoto-Sivashinsky: dz = 8 marks the reconstruction-MSE plateau, with CNNs outperforming autoencoders and PCA; LED therefore uses a CNN latent representation.The selected latent dimension agrees with prior estimates of the attractor dimensionality.
  • B. Kuramoto-Sivashinsky: LSTM, end-to-end LSTM, and SINDy propagators show comparable accuracy, whereas MLP and reservoir-computer propagators have large errors; LED therefore uses an LSTM.Forecasts are evaluated from 100 test initial conditions up to Tf = 800 using MNAD.
  • B. Kuramoto-Sivashinsky: More than two orders of magnitude faster than the micro solver, Latent-LED remains on the attractor and qualitatively reproduces the simulated density of values.As ρ increases, error decreases and correlation increases, while speed-up decreases.
  • B. Kuramoto-Sivashinsky: Latent-LED matches the forecasting performance of a full-state LSTM but has inferior short-term accuracy to a full-state reservoir computer.Using the reservoir computer on LED’s latent space makes the error grow significantly relative to CNN-LSTM.

C. Viscous Flow Behind a Cylinder

The cylinder-flow benchmark evaluates LED’s autoencoder–LSTM framework for predicting viscous flows at Re = 100 and Re = 1000. LED captures the flow dynamics and remains close to the data attractor over long horizons, although errors near the cylinder challenge drag prediction.

  • Method: LED represents the cylinder-flow state with pressure, two velocity components, and vorticity, using CNN autoencoders and a tuned latent dimension of dz = 4 for Re = 100 and dz = 10 for Re = 1000.The state is st ≡ {p, ux, uy, ω} ∈ R4×512×1024, with CNNs exploiting spatial correlations.
  • Evaluation: The benchmark compares LED’s LSTM propagator with SINDy and RC using 10 test initial conditions over prediction horizon T = 20.Hyperparameters are tuned using MNAD on validation data.
  • Prediction: Latent-LED captures the flow for both Re ∈ {100, 1000}, with predictions remaining close to the attractor even at horizon T = 20.At lead time T = 4, prediction error concentrates mostly around the cylinder.
  • Prediction: For Re = 1000, prediction error concentrates around the cylinder and leaves the drag-coefficient error on the order of 0.15, despite Multiscale-LED reducing it.This localized error makes accurate drag computation challenging.
  • Prediction: For Re = 100, LED recovers a periodic nonlinear latent-space mode using dz = 4, whereas Galerkin and DMD ROMs use six to eight degrees of freedom.The LED latent representation forecasts the dynamics accurately in this regime.

III. DISCUSSION

LED combines Equation-Free modeling with machine learning to learn and propagate effective dynamics for multiscale complex systems. It provides computational savings and an adjustable speed–accuracy tradeoff, while remaining limited by training-domain coverage and the absence of automatic upscaling decisions.

  • Framework: LED augments the Equation-Free formalism with state-of-the-art machine-learning methods to accelerate simulations of stochastic or deterministic multiscale complex dynamical systems.The framework learns effective dynamics rather than directly resolving all system scales.
  • Framework: LED accelerates expensive high-dimensional simulations by propagating latent-space dynamics and decoding them into high-dimensional states, with network training performed once offline.It uses probabilistic generative-mixture-density or deterministic convolutional decoders, and trained models can forecast dynamics.
  • Evaluation: An order of magnitude lower approximation error and two orders of magnitude faster simulation were achieved for FitzHugh–Nagumo dynamics versus Equation-Free approaches and the Lattice Boltzmann solver, respectively.The framework was evaluated on benchmark problems and also demonstrated for Kuramoto dynamics and stochastic molecular dynamics.
  • Trade-offs: LED trades speed-up against accuracy by switching between latent-dynamics propagation and original-equation evolution, iteratively correcting statistical error at reduced speed-up.This provides a systematic control over computational efficiency and prediction accuracy in multiscale systems.
  • Limitations: LED lacks a mechanism to determine when to upscale latent dynamics and is not expected to generalize to dynamical regions drastically different from its training data.Adapting the training procedure is identified as a direction for future research.
  • Scope: The methodology applies to first-principles and data-only problems, creating algorithmic combinations of data-driven and first-principles models for efficient multiscale prediction.The authors note applicability to problems where Equation-Free, HMM, and FLAVOR methodologies have been used.

Supplementary Information I: Methods … D. L1-Norm Histogram Distance

The LED framework combines neural-network representations, latent-space dynamics, and specialized decoders with metrics for evaluating prediction accuracy and distributional agreement. Its methods include autoencoders, recurrent networks, invariance-aware models, mixture-density decoding, and complementary comparison measures.

  • Supplementary Information I: Methods: LED comprises models for learning and propagating effective dynamics in complex systems.The framework is introduced as a collection of models described in the methods.
  • A. Autoencoders (AE): Feed-forward autoencoders identify coarse representations of the FitzHugh-Nagumo and Kuramoto-Sivashinsky equations.Autoencoders map inputs to low-dimensional latent spaces and reconstruct the original dimension.
  • B. Variational Autoencoders (VAE): Variational autoencoders showed no significant advantages over feed-forward autoencoders for the FitzHugh-Nagumo and Kuramoto-Sivashinsky cases studied.VAEs remain part of LED and may be useful in other applications.
  • C. Convolutional Neural Networks: Autoencoding convolutional neural networks incorporate spatial structure and were applied to three systems, including cylinder flow at Re ∈{100, 1000}.CNNs were compared with feed-forward autoencoders for identifying coarse representations.
  • D. Permutation Invariance: Permutation-invariant layers represent systems of same-kind particles while preserving configurations under particle reordering.The approach was used for collective particle dynamics governed by one- and three-dimensional advection-diffusion equations.
  • E. Mixture Density Decoder: Mixture-density decoders model bounded, potentially non-Gaussian and multimodal state distributions using mixture components with covariance structure.They are employed for stochastic systems such as molecular dynamics, with decoder outputs parameterizing the distribution.
  • F. Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs): LSTM recurrent neural networks capture nonlinear, non-Markovian latent-space dynamics and are trained by minimizing latent forecasting error with backpropagation through time.The LSTM uses cell and internal memory states, while the output mapping is linear.
  • Supplementary Information II: Comparison Measures: Comparison measures include MNAD for deterministic prediction, Pearson correlation for Kuramoto-Sivashinsky prediction, and WD and L1-NHD for distributional differences.WD and L1-NHD compare spatial particle distributions in the Advection-Diffusion model; L1-NHD uses normalized histograms on a shared grid with Rice-rule binning.

Supplementary Information III: Results

Figure 4 evaluates LED using histogram and Wasserstein distances over time and initial conditions, with self-similar error providing a reference for statistical significance. It also reports LED’s speed-up relative to micro-scale simulations.

  • Error metrics: Figure 4A–B assess the L1-Norm Histogram distance, averaged across time and initial conditions and tracked over time.Errors below the self-similar error level are identified as statistically insignificant.
  • Error metrics: Figure 4C–D assess the Wasserstein distance, averaged across time and initial conditions and tracked over time.
  • Computational efficiency: Figure 4E reports LED’s speed-up compared to the micro scale solver.

A. LED for Advection-Diffusion Equation · a. Hyper-parameter Tuning · b. Generalization to Different Number of Particles

LED is applied to a three-dimensional advection-diffusion system of 1,000 particles, learning an 8-dimensional latent representation with a permutation-invariant autoencoder and recurrent dynamics. It provides efficient multiscale forecasting, captures metastable collective behavior, and is tested for robustness across diffusion regimes and particle counts.

  • A. LED for Advection-Diffusion Equation: LED models a three-dimensional advection-diffusion process with N = 1000 particles and evaluates forecasts against particle-distribution distances.The system uses diffusion, cosine advection, explicit Euler integration, reflective boundaries, and separate training, validation, and test datasets.
  • A. LED for Advection-Diffusion Equation: The permutation-invariant autoencoder identifies a low-dimensional representation of particle configurations whose collective dynamics are not straightforward to determine a priori.The particle system concentrates in metastable states and transitions between them, motivating latent-variable modeling.
  • a. Hyper-parameter Tuning: dz = 8 is selected from validation log-likelihood, while the encoder uses 3 × 50 tanh layers and a permutation-invariant space of dimension M = 100.Further reducing dz worsens validation loss, whereas increasing it produces no significant improvement.
  • A. LED for Advection-Diffusion Equation: LED forecasts 30 test trajectories to Tf = 2000 and compares predicted particle distributions with ground truth using L1-NHD and Wasserstein distance.The setup includes an initial warm-up period of Twarm = 100.
  • A. LED for Advection-Diffusion Equation: ρ = 20 corresponds to Tm = 100 and Tµ = 5, and multiscale propagation alleviates iterative statistical error propagation by alternating latent and microscale evolution.The alternative Tm = 50 gives ρ = 10; increasing the multiscale ratio spends more time in latent propagation, while evolving the high-dimensional state adds computational cost.
  • A. LED for Advection-Diffusion Equation: Spectral clustering of the dz = 8 latent dynamics identifies six high-density regions whose centers can be decoded into particle configurations.This maps frequently visited latent metastable states back to realizations of high-dimensional simulation states.
  • b. Generalization to Different Number of Particles: A model trained with N = 1000 particles is evaluated on N = 400 particles to test generalization enabled by the permutation-invariant encoder.The comparison includes iterative latent propagation with Tµ = 0 and multiscale forecasting with Tµ = 100 and ρ = 20, using Twarm = 100.

B. FitzHugh-Nagumo Model (FHN)

The FHN model uses a two-channel CNN autoencoder with scaled inputs and outputs, while experiments compare latent forecasting methods and equation-free variants. Increasing the macro-to-micro ratio raises LED speed-up but also increases MNAD error.

  • Forecasting comparisons: The FHN experiments compare Latent-LED with dz = 2 against equation-free variants and compare five macrodynamics propagators in iterative latent forecasting.The propagators are AE-LSTMend2end, AE-LSTM, AE-MLP, AE-RC, and AE-SINDy.
  • Efficiency and accuracy trade-off: Increasing the macro-to-micro ratio ρ = Tm/Tµ increases LED speed-up relative to the LB solver but also increases MNAD error.Latent-LED denotes AE-LSTMend2end with dz = 2, while Latent corresponds to Tµ = 0.
  • Prediction visualization: The FHN results include activator-density evolution from a test initial condition, Latent-LED predictions, and their absolute difference.These visualizations are presented as the evolution, prediction, and error panels.

a. Hyper-parameters and Training Time · C. Kuramoto-Sivashinsky · a. Hyper-parameters and Training Times

The paper reports model hyper-parameters and training times for LED and comparator methods across FHN and KS, while demonstrating that Latent-LED reproduces the long-term KS climate using an 8-dimensional latent state.

  • a. Hyper-parameters and Training Time: FHN hyper-parameter tuning and training times are reported for the LED autoencoder, LSTM, CNN autoencoder, reservoir computer, and SINDy.PCA and Diffusion maps each require approximately one minute of fitting; the MLP uses three CELU-activated layers and trains for 100 minutes.
  • a. Hyper-parameters and Training Time: For FHN, the best-performing validation model is identified by red-colored parameters in the reported hyper-parameter tables.The tables cover the autoencoder, LED-RNN, CNN autoencoder, reservoir computer, and SINDy configurations.
  • a. Hyper-parameters and Training Time: The FHN CNN autoencoder pads inputs to the closest power of two before applying four convolutional, pooling, activation, and dropout stages.An MLP then projects to the low-order latent space, with output activation 1 + 0.5 tanh(·).
  • C. Kuramoto-Sivashinsky: Latent-LED reproduces the long-term climate of a Kuramoto–Sivashinsky trajectory while propagating an 8-dimensional latent state.The predicted field is generated after a warm-up period Twarm = 60 with Tµ = 0.
  • a. Hyper-parameters and Training Times: KS hyper-parameters and training times are reported for the autoencoder, CNN, LSTM-RNN LED, reservoir computer, and SINDy models.The KS CNN autoencoder architecture is documented in Table 10 and Figure 12, while PCA fitting takes approximately one minute.
  • a. Hyper-parameters and Training Times: KS reservoir-computer and SINDy configurations, including their training times, are provided in Tables 12 and 13.The reservoir computer is identified as operating in CNN-RC, and SINDy as operating in CNN-SINDy.

D. Viscous Flow Behind a Cylinder

The cylinder-wake simulations solve incompressible Navier–Stokes flow with Brinkman penalization and pressure projection, using finite-difference discretization. Experiments cover Re = 100 and Re = 1000, with vortex-shedding metrics and LED states defined on high-resolution flow fields.

  • Numerical method: The flow solver uses Brinkman penalization to enforce no-slip cylinder boundaries and pressure projection to impose incompressibility.Advection and diffusion are applied before pressure projection, followed by penalization of the cylinder motion.
  • Numerical method: The discretization uses second-order central finite differences for diffusion and a third-order upwind scheme for advection.The computational domain is Ω = [0, 1] × [0, 0.5], with a cylinder of diameter D = 0.075.
  • Simulation setup: Re = 100 and Re = 1000 are simulated for a cylinder moving at v = 0.15 in the −x-direction, using ν = 0.0001125 and ν = 0.00001125, respectively.Both cases use 1024 × 512 gridpoints.
  • Simulation setup: The vortex-shedding Strouhal numbers are St = 0.175 for Re = 100 and St = 0.225 for Re = 1000.The Strouhal number characterizes periodic vortex shedding in the cylinder wake.
  • LED representation: LED represents each simulation state with ux, uy, pressure p, and vorticity ω, saved every Δt = 0.2 for 1000 coarse time-steps.The state has shape R4×512×1024; LED uses latent dimension 4 at Re = 100 and 10 at Re = 1000.

E. Alanine Dipeptide Dynamics

LED is evaluated on alanine dipeptide dynamics in water using molecular-dynamics data and a latent-space recurrent model. It captures the molecule’s three mostly visited metastable states and maps low-energy latent regions to known configurations.

  • E. Alanine Dipeptide Dynamics: 38.4ns of molecular-dynamics data were used for training, 38.4ns for validation, and 100ns for testing, with δt = 1fs.LED used ∆t = 0.1ps for its time step.
  • E. Alanine Dipeptide Dynamics: LED uses an MD decoder and an MD-LSTM in latent space for alanine dipeptide dynamics.
  • E. Alanine Dipeptide Dynamics: LED captures the three mostly visited metastable states {C5, PII, αR} in predicted state evolution.
  • E. Alanine Dipeptide Dynamics: Low-energy, high-probability regions in LED’s one-dimensional latent space map to known metastable alanine dipeptide configurations.The latent free energy is defined as F/κBT = −log p(zt).
Loading 2006.13431v3…