Source-linked AI summary
Constructing Neural Network-Based Models for Simulating Dynamical Systems
Christian Møldrup Legaard, Thomas Schranz, Gerald Schweiger, Ján Drgoňa, Basak Falay, Cláudio Gomes, Alexandros Iosifidis, Mahdi Abkar, Peter Gorm Larsen
TL;DR
Complex dynamical systems are difficult to model analytically, motivating neural-network approaches that learn system dynamics from observations. This survey organizes neural-network simulation models, reviews their training and validation, and examines their numerical challenges. It concludes that model variants are closely related but differ in architecture, physics incorporation, and numerical integration, making application-specific selection difficult and motivating standardized benchmarks.
Problem
Deriving accurate and efficient models for complex dynamical systems is difficult, while direct-solution neural networks can require substantial data and may fit collocation points without reproducing underlying trends.
Method
The paper surveys direct-solution and time-stepper neural-network models, explaining their simulation mechanisms, training procedures, validation properties, challenges, and extensions.
Results
The survey finds that many model variants are closely related, with differences mainly involving neural-network architecture, physics-based regularization, automatic differentiation, and numerical integration schemes.
Takeaways & Limitations
Model selection should consider the capabilities required by the application and how physical knowledge can be incorporated, rather than relying on fixed general guidelines.
Takeaways & Limitations
The surveyed approaches include concrete limitations such as direct-solution models failing to preserve derivative relationships and time-stepper simulations diverging from true dynamics over time.
Abstract
from arXiv · showhide
Dynamical systems see widespread use in natural sciences like physics, biology, chemistry, as well as engineering disciplines such as circuit analysis, computational fluid dynamics, and control. For simple systems, the differential equations governing the dynamics can be derived by applying fundamental physical laws. However, for more complex systems, this approach becomes exceedingly difficult. Data-driven modeling is an alternative paradigm that seeks to learn an approximation of the dynamics of a system using observations of the true system. In recent years, there has been an increased interest in data-driven modeling techniques, in particular neural networks have proven to provide an effective framework for solving a wide range of tasks. This paper provides a survey of the different ways to construct models of dynamical systems using neural networks. In addition to the basic overview, we review the related literature and outline the most significant challenges from numerical simulations that this modeling paradigm must overcome. Based on the reviewed literature and identified challenges, we provide a discussion on promising research areas.
1 INTRODUCTION
The survey addresses the difficulty of deriving accurate, efficient dynamical-system models for complex physical phenomena by organizing neural-network simulation approaches and their training and validation practices. It also connects numerical-simulation challenges with deep-learning challenges, including stability, solver convergence, and long-term prediction.
- Motivation: Complex physical phenomena make it difficult to derive models that balance accuracy and speed, motivating data-driven modeling approaches.
- Scope and contribution: The survey provides a practical guide to constructing dynamical-system models with neural networks, emphasizing trajectory splitting, simulation-specific optimization, and empirical validation.
- Challenges and research directions: The survey relates numerical-simulation challenges to deep-learning simulation challenges, including energy conservation, numerical stability, and solver convergence in long-term prediction.
- Organization: For each model category, the survey explains simulation mechanisms, parameter training, and challenges or extensions designed to address them.
2 BACKGROUND
This section introduces first-principles and data-driven dynamical-system models, then explains how differential equations, numerical solvers, and neural networks support simulation. It culminates in a taxonomy distinguishing direct-solution models from time-stepper models.
- First-principles models derive system behavior from physical laws, whereas data-driven models construct models from experimental data.
- Differential Equations: An ideal pendulum represents its state with angle θ and angular velocity ω, whose evolution is described by ordinary differential equations.
- Differential Equations: An initial value problem combines the derivative function with an initial state to define the system trajectory over time.
- Numerical Solvers: Numerical solvers approximate continuous initial value problems as discrete-time dynamical systems by iteratively advancing the state.
- Neural Networks: Fully-connected neural networks learn predictions by tuning weights against a loss function, defined here as mean squared error between predicted and true trajectories.
- Model Taxonomy: The taxonomy separates direct-solution models, which estimate states at particular times without explicit integration, from time-stepper models, which advance states using numerical-simulation techniques.
3 DIRECT-SOLUTION MODELS
Direct-solution models learn a continuous mapping from time to system state, enabling predictions at arbitrary times but typically for fixed initial conditions. Their accuracy and generalization depend strongly on training-data quality, while automatic differentiation, physics-informed constraints, and hidden-variable methods can improve physical consistency or recover difficult-to-measure quantities.
- Model formulation: Direct-solution models map time t_k to the system state x_k and can evaluate the learned continuous function at arbitrary times.A neural network represents the mapping from time to the solution at that time.
- Model formulation: Predictions for multiple time instances can be evaluated in parallel because the estimates have no dependencies on one another.For the pendulum example, a feed-forward network takes time as input and predicts θ and ω.
- Limitations: A trained direct-solution model is tied to the initial conditions encoded during training and cannot be modified for different initial conditions at inference.The model is trained on collocation points from a trajectory associated with a specific initial condition.
- Limitations: Sparse or poorly chosen collocation points can produce excellent fits at training points but poor generalization between them.The issue can persist under sufficiently sparse sampling, while acquiring denser data may be expensive or impractical.
- Physical consistency: Naive models may violate derivative relationships between predicted state variables, even when both variables match the collocation points.For the pendulum, predicted ω may be neither the derivative of predicted θ nor close to the true trajectory.
- Physical consistency: Automatic differentiation can obtain one state variable by differentiating another, improving trajectory generalization and enforcing derivative consistency.The pendulum model predicts θ and derives ω from its time derivative; physics-informed methods similarly use equations encoding prior knowledge.
- Hidden variables: Hidden physics neural networks infer difficult-to-measure variables from observed data by using governing equations to extract otherwise unavailable features.The surveyed example extracts pressure and velocity fields from measured dye concentrations, including sparse space-time observations.
4 TIME-STEPPER MODELS
Time-stepper models learn a system’s derivative function and repeatedly integrate it to simulate future states, allowing established numerical solvers to be incorporated. Their behavior depends on both how derivatives are produced and which integration scheme is used, and the surveyed combinations are not exhaustive.
- Model formulation: Time-stepper models approximate the system’s derivative function and use repeated time steps to generate simulations.This replaces a hand-derived derivative function while retaining the structure of numerical simulation.
- Model formulation: Integrating established numerical solvers into time-stepper models is a stated advantage of this approach.The derivative network supplies the learned dynamics, while the solver advances the simulation.
- Model design: The main differences between time-stepper models concern how derivatives are produced and which integration scheme is applied.For example, direct models use no integration scheme, whereas Euler time-steppers resemble Forward Euler and can differ substantially in predictive ability.
- Model design: The neural-network choice and numerical-solver choice are independent, so the described models are examples rather than an exhaustive combination list.The survey aims to describe and compare combinations commonly encountered in the literature.
4.1 Methodology
Training time-stepper models requires generalization across initial conditions and control of errors that accumulate because future predictions depend on earlier predictions. The methodology trains on sampled trajectories, validates on new initial conditions, and compares simulations against the original ODE.
- Training considerations: Time-stepper training must support accurate simulations from different initial conditions while limiting errors that accumulate over multiple prediction steps.Future predictions depend on past predictions, making error accumulation a central training consideration.
- Training data: Training data can consist of several short trajectories or a few long trajectories, but it should represent conditions expected in the intended application.The survey emphasizes matching the training distribution to the deployment setting.
- Training objective: A simple training strategy minimizes single-step prediction error using the learned derivative and an integration rule.For a Forward Euler solver, the next prediction is ˜x_k+1 = ˜x_k + h_k*N(˜x_k), starting from ˜x_0 = x_0.
- Experimental setup: The described experiments train on 100 two-sample trajectories with initial states sampled over θ: (−1, 1) and ω: (−1, 1) using Latin hypercube sampling.The models use fully connected networks with 8 hidden layers and 32 neurons per layer.
- Model variants: The surveyed time-stepper examples use single-step training, with derivative production varying for architectures such as Lagrangian networks.Lagrangian networks obtain derivatives through automatic differentiation rather than explicit network outputs.
- Validation: Validation samples 100 new initial conditions on a grid and simulates each system for 4π seconds using the original ODE for comparison.The paper displays only one corresponding trajectory for simplicity.
4.2 Integration Schemes
Integration schemes determine how neural-network outputs become simulated states, and solver choice also affects training. The survey contrasts direct, residual, Euler, and NODE-based approaches, emphasizing compounding error, stability, and computational trade-offs.
- Solver choice determines both how a model produces simulations and how it must be trained when objectives depend on integrated states.
- Residual time-stepper: Residual time-steppers predict a state change that is added to the current state, motivated by the possibility that changes are easier to learn than direct mappings.
- Direct time-stepper: Direct time-steppers repeatedly predict the next state, but their simulations quickly diverge because errors compound over multiple steps.
- Euler time-stepper: Euler time-steppers scale the predicted derivative by the step size before updating the state, but matching training and plotting step sizes can yield minimal improvement over residual models.
- Numerical solvers: Forward Euler is simple but accumulates more error than advanced methods such as midpoint and linear multistep schemes for a given step size.
- Neural ordinary differential equations: NODEs combine neural derivative approximations with numerical solvers, while stability, convergence, inference-step sensitivity, and training overhead remain important concerns.
4.3 External Input
Time-stepper models can incorporate external inputs at every derivative evaluation through several architectures. The choice between concatenated and separated input pathways depends on how inputs interact with the system state.
- External inputs such as forces or control signals can be introduced into time-stepper models at every derivative evaluation.
- Neural state-space models: One approach combines states and inputs within a network, allowing the network to learn their joint influence without imposing aggregation structure.
- Neural state-space models: Separate networks can model autonomous and forced dynamics and sum their contributions when input effects are known to be state-independent.
- Neural state-space models: These neural nonlinear state-space models extend state-space modeling by using neural networks to represent nonlinear dynamics.
- Neural ordinary differential equations: NODEs with adaptive solvers require interpolation when solver evaluation times do not coincide with sampled external-input times.
- Controlled differential equations: Neural controlled differential equations treat time and external inputs as a driving signal, covering both autonomous systems and systems driven purely by inputs.
4.4 Network Architecture
Specialized neural architectures encode physical structure or relational organization into dynamical-system models. The survey covers energy-based, potential-energy, and graph-based designs, including their computational benefits and modeling roles.
- Domain-specific architectures can be integrated into time-steppers to enforce energy conservation or represent systems naturally structured as graphs.
- Hamiltonian and Lagrangian networks: Hamiltonian and Lagrangian networks learn scalar energy functions and obtain state derivatives by differentiating those outputs with respect to state variables.
- Hamiltonian and Lagrangian networks: These energy-based networks naturally incorporate energy preservation into the network structure, while symplectic networks target energy-conserving dynamics.
- Deep potential-energy networks: Deep potential-energy models learn scalar potentials whose gradients provide interaction forces, with kinetic terms handled explicitly through Newtonian dynamics.
- Deep potential-energy networks: Learned potentials replace expensive quantum-chemistry calculations in molecular dynamics and support simulations of up to 100M atoms, compared with a 300M-dimensional naive time-stepper mapping.
- Graph neural networks: Graph neural time-steppers encode the current state as a graph, process it into a state update, and decode that update back to the original state space.
4.5 Uncertainty
Uncertainty-aware models address noisy observations and stochastic dynamics through latent-variable, Bayesian, and stochastic differential-equation formulations. These approaches provide richer uncertainty representations but introduce inference, computational, or solver complexity.
- Measurement noise and intrinsically random dynamics motivate probabilistic and stochastic neural models.
- Deep Markov models: Deep Markov models represent observations probabilistically through latent variables whose evolution is not fully deterministic.
- Latent neural ordinary differential equations: Latent NODEs encode several observations into an initial latent state, simulate the trajectory in latent space, and decode it into observable space.
- Deep Markov models: Deep Markov models lack supervised targets for latent variables, so variational inference is commonly used for training.
- Latent neural ordinary differential equations: Latent formulations separate measurements from dynamics but create an inference problem for estimating hidden initial conditions.
- Bayesian neural ordinary differential equations: Bayesian NODEs represent network parameters as distributions and generate multiple trajectories to estimate confidence bounds and mean predictions.
- Bayesian neural ordinary differential equations: Bayesian neural extensions require specialized training algorithms that generally do not scale well to large network architectures.
- Neural stochastic differential equations: Neural stochastic differential equations add stochastic diffusion to deterministic drift and require numerical solvers suited to stochastic dynamics.
5 DISCUSSION
The survey distinguishes direct-solution and time-stepper models, showing that many variants differ through modest changes in physics regularization, network architecture, or numerical integration. Because model capabilities and physical knowledge vary by application, concrete selection guidelines remain difficult, motivating benchmark datasets and standardized evaluation procedures.
- Two fundamentally different simulation strategies use NNs to approximate either the problem’s solution or the system’s dynamics.
- Many direct-solution and time-stepper variants are closely related, differing mainly in physics-based regularization, automatic differentiation, NN architecture, or numerical integration.
- Model selection lacks concrete guidelines because applications require different capabilities and incorporate physical knowledge in different ways.
- Future research: A benchmark dataset spanning diverse dynamical systems would support fair comparisons, general trends, and heuristics for selecting NN-based models.
- Future research: Evaluation procedures should test whether models produce accurate simulations from the initial conditions encountered in use, including systems where small errors accumulate rapidly.
- Future research: Training guidelines should clarify the data required for target accuracy and how trajectories should be divided into training and validation sets.
6 SUMMARY
The survey addresses fragmented literature on neural-network simulation of dynamical systems by organizing models into direct-solution and time-stepper types. It provides construction and training guidance, implementation references, and identifies broad evaluation as an open research question.
- The literature’s varied terminology and notation make neural-network approaches to simulating physical phenomena difficult to digest outside specialized fields.
- The survey categorizes NN simulation models into direct-solution and time-stepper types and guides their construction, training, and use.
- Source code for many models provides reference implementations for their detailed construction.
- How well these methods work across a broad set of representative real-world problems remains an open research question.