PaperlayerView exact revision on arXiv

Source-linked AI summary

Respecting causality is all you need for training physics-informed neural networks

Sifan Wang, Shyam Sankaran, Paris Perdikaris

arXiv:2203.07404v1cs.LGmath.NAnlin.CDphysics.flu-dynstat.ML

TL;DR

PINNs have struggled with multi-scale, chaotic, and turbulent dynamics, motivating an explanation centered on violations of physical causality. The paper re-formulates PINN losses to account for causality during training and reports successful simulations across challenging benchmarks, alongside a quantitative convergence criterion. The authors also note that PINN forward solving remains more time-consuming than traditional numerical solvers.

  • Problem

    Existing PINN formulations have not successfully simulated dynamical systems with multi-scale, chaotic, or turbulent solutions, despite their relevance to realistic scientific and engineering scenarios.

  • Method

    The paper re-formulates PINN loss functions and introduces causal training to explicitly respect the spatio-temporal causal structure of physical systems.

  • Results

    The causal modification yields significant accuracy improvements and enables successful PINN simulations of the chaotic Lorenz, Kuramoto–Sivashinsky, and turbulent Navier–Stokes systems.

  • Takeaways & Limitations

    The results support applying causal training to challenging dynamical systems and provide a practical quantitative mechanism for assessing PINN convergence.

  • Takeaways & Limitations

    PINNs used as forward PDE solvers remain significantly more time-consuming than traditional numerical solvers, motivating further work on accelerating training.

Abstract

from arXiv · show

While the popularity of physics-informed neural networks (PINNs) is steadily rising, to this date PINNs have not been successful in simulating dynamical systems whose solution exhibits multi-scale, chaotic or turbulent behavior. In this work we attribute this shortcoming to the inability of existing PINNs formulations to respect the spatio-temporal causal structure that is inherent to the evolution of physical systems. We argue that this is a fundamental limitation and a key source of error that can ultimately steer PINN models to converge towards erroneous solutions. We address this pathology by proposing a simple re-formulation of PINNs loss functions that can explicitly account for physical causality during model training. We demonstrate that this simple modification alone is enough to introduce significant accuracy improvements, as well as a practical quantitative mechanism for assessing the convergence of a PINNs model. We provide state-of-the-art numerical results across a series of benchmarks for which existing PINNs formulations fail, including the chaotic Lorenz system, the Kuramoto-Sivashinsky equation in the chaotic regime, and the Navier-Stokes equations in the turbulent regime. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such systems, introducing new opportunities for their applicability to problems of industrial complexity.

1 Introduction

Existing PINNs often fail on nonlinear, multi-scale, chaotic, and turbulent systems because their training can violate the spatio-temporal causal structure of physical evolution. The paper proposes causal loss re-weighting, reports improved accuracy and convergence assessment, and demonstrates success on challenging benchmarks.

  • Motivation: PINNs remain unreliable for strongly nonlinear, multi-scale, chaotic, and turbulent systems relevant to science and engineering.The paper identifies these cases as a pressing challenge rather than pathological corner cases.
  • Physical causality: Physical causality constrains how local changes in initial or boundary data affect states at later times, including temporal precedence and covariation.The paper uses this causal structure as the relevant organizing principle for time-dependent physical systems.
  • Problem diagnosis: Continuous-time PINNs can violate causality and converge toward erroneous solutions because their formulation does not explicitly enforce the causal structure of the solution.The paper characterizes this as an implicit bias of existing training formulations.
  • Proposed approach: The proposed approach re-formulates PINN loss functions to explicitly respect physical causality during model training.The paper presents this as a general causal training strategy for nonlinear PDEs.
  • Contributions: The modification produces significant accuracy improvements and a practical quantitative criterion for assessing PINN training convergence.The paper also reports state-of-the-art results on challenging benchmarks where existing formulations fail.
  • Benchmarks: The method successfully simulates the chaotic Lorenz system, chaotic-regime Kuramoto–Sivashinsky equation, and turbulent-regime Navier–Stokes equations.The authors describe this as the first reported success of PINNs on these systems.

2 Physics-informed neural networks (PINNs)

This section introduces PINNs for PDE solution inference, then shows how conventional training can violate temporal causality and converge to erroneous solutions. It motivates a causality-aware loss reformulation and a quantitative convergence criterion.

  • Problem setup: PINNs represent the unknown PDE solution u(t, x) with a neural network uθ(t, x) and train it by minimizing a composite loss.The PDE may involve linear or nonlinear differential operators and Dirichlet, Neumann, Robin, or periodic boundary conditions.
  • An illustrative example: The Allen-Cahn equation is a difficult benchmark for conventional continuous-time PINNs, despite adaptive resampling, weighting, and exact periodic boundary enforcement.The setup uses a fully connected tanh network with four hidden layers and 128 neurons per layer.
  • Conventional PINN failure: 49.87% relative L2 error: the conventional PINN prediction remains inaccurate after 2 × 105 Adam iterations and appears trapped in an erroneous local minimum.Its loss decreases rapidly initially and then barely changes, even though periodic boundary conditions are enforced exactly.
  • Temporal causality: The original residual loss can become small at later times while predictions at current and previous times remain inaccurate, violating temporal causality.For Allen-Cahn, the temporal residual is large near the initial state and decays to nearly zero after t = 0.5; such a small residual is meaningful only with accurate earlier predictions.
  • Implicit bias: The temporal convergence rate increases with time, revealing an implicit gradient-descent bias toward minimizing residuals at larger t.The paper defines a practical temporal convergence rate C(t) and uses it to assess training convergence.

3 Causal training for physics-informed neural networks

The paper reformulates PINN losses to enforce temporal causality, making later-time residual minimization depend on earlier residuals. Causal training also supplies a weight-based convergence criterion and improves accuracy on the Allen-Cahn example.

  • A simple re-formulation: The weighted residual loss explicitly accounts for the causal structure of nonlinear PDE solutions.Temporal residuals are weighted using cumulative residual information from earlier time steps.
  • A simple re-formulation: Later-time residuals are not minimized unless preceding temporal residuals decrease sufficiently to activate their weights.The weights are inversely exponentially proportional to cumulative residual loss from previous time steps.
  • Allen-Cahn example: 1.43e −03 relative L2 error is achieved for the Allen-Cahn prediction using the causal training algorithm.The experiment uses the weighted residual loss with ϵ = 100 for 3 × 10^5 gradient-descent iterations.
  • Allen-Cahn example: 1.39e −04 relative L2 error is achieved when causal training is combined with the modified MLP architecture.This result is more accurate than the same-network Allen-Cahn experiment.
  • A stopping criterion for assessing training convergence: Monitoring temporal weights provides a quantitative stopping criterion for assessing PINN training convergence.Training can terminate when min_i wi > δ, with δ ∈ (0, 1); the criterion also reduces training time and improves predictive accuracy.
  • Sensitivity on the causality parameter ϵ: The causality parameter ϵ is sensitive, so the method uses annealing to avoid tedious hyper-parameter tuning.Very small ϵ can impede later residual minimization, while large ϵ can make optimization difficult.
  • Causal training for PINNs: Algorithm 1 incorporates weighted residual and initial-condition losses, ϵ annealing, and temporal-weight convergence stopping.The algorithm can be adapted to boundary constraints and used within several physics-informed machine-learning pipelines.
  • Connection to existing approaches: Causal training complements time-marching methods because causality violations may still occur within individual time windows.Its weighting strategy has an effect similar to changing collocation-point sampling density.

4 Practical considerations

The paper describes practical enhancements for challenging PINN problems, including modified MLPs, exact periodic embeddings, and Taylor-mode automatic differentiation. These techniques target accuracy, boundary enforcement, derivative computation, and computational efficiency.

  • Motivation: Higher-order accuracy is needed for problems with sensitivity to initial data and strong spatio-temporal correlations.The paper reports that the practical enhancements can further improve accuracy and computational efficiency on its considered problems.
  • Modified multi-layer perceptrons: Modified MLPs use two encoders and point-wise feature merging to improve PDE-residual minimization and sharp-gradient capture.The architecture retains a standard MLP while modifying its forward pass through encoder-derived features.
  • Exact periodic boundary conditions: Fourier-feature embeddings can impose periodic boundary conditions exactly within the neural-network representation.For one-dimensional periodic constraints, any network representation using the embedding is guaranteed to satisfy periodicity.
  • Exact periodic boundary conditions: Two-dimensional Fourier embeddings extend exact periodic enforcement to periodic cells in the x and y directions.The construction combines sine and cosine features across both coordinates.
  • Exact periodic boundary conditions: For time-dependent problems, time coordinates are concatenated with the Fourier-feature embedding.The same construction is described for one- and two-dimensional spatial domains.
  • Taylor-mode automatic differentiation for high-order derivatives: Taylor-mode automatic differentiation offers better scaling than conventional differentiation as derivative order increases.It addresses memory and computation bottlenecks when PINNs require derivatives above second order.

5 Results

The causal training algorithm is evaluated on chaotic ODEs, chaotic PDEs, and turbulent Navier–Stokes flow. Across these benchmarks, predictions agree well with reference solutions, although long-time chaotic integration eventually loses accuracy.

  • Lorenz system: The chaotic Lorenz trajectory achieves relative L2 errors of 1.139e −02, 1.656e −02, and 7.038e −03 for x, y, and z, respectively.The stopping criterion is satisfied for every training time window.
  • Kuramoto–Sivashinsky equation: The regular Kuramoto–Sivashinsky case reaches a relative L2 error of 3.49e −04 through time t = 1.Predictions are compared with reference solutions at t = 0, 0.5, and 1.0.
  • Computational efficiency: Taylor-mode automatic differentiation provides a computational-efficiency advantage over conventional reverse-mode differentiation for larger architectures and batch sizes.The comparison varies network depth and batch size while keeping the remaining hyper-parameters fixed.
  • Kuramoto–Sivashinsky equation: The chaotic Kuramoto–Sivashinsky case is solved to T = 0.5, with predictions showing good agreement with the reference solution.The case uses parameters α = 100/16, β = 100/162, and γ = 100/164 on [0, 2π].
  • Long-time behavior: In long-time chaotic Kuramoto–Sivashinsky integration, the model captures the transition to chaos near t = 0.4 but loses accuracy after t = 0.8.The final state has relative L2 error above 10%, attributed to numerical error accumulation and chaotic dynamics.
  • Navier–Stokes turbulence: For two-dimensional decaying turbulence, relative errors are 3.90e−02, 2.61e−02, and 3.53e−02 for u, v, and w over the full spatio-temporal domain.Predicted velocity and vorticity fields agree with reference solutions at T = 1.

6 Discussion

The discussion attributes PINN failures partly to violations of physical causality and proposes causality-aware training as a remedy. It also identifies scope boundaries in forward simulations, problem-dependent causal structures, computational cost, and architecture design.

  • Causal training: PINNs can prefer minimizing later-time PDE residuals before fitting initial data, making them susceptible to erroneous solutions.This implicit bias is identified through analysis of the limiting neural tangent kernel.
  • Causal training: Re-weighting PDE residual losses during gradient descent restores physical causality and provides a practical stopping criterion for total-loss convergence.The proposed causal training algorithm performs this re-weighting at each iteration.
  • Reported outcomes: 10-100x improvements in accuracy were achieved against competing approaches, enabling simulations of chaotic and turbulent benchmark systems previously inaccessible to PINNs.Reported examples include the Lorenz and Kuramoto-Sivashinsky equations and incompressible turbulent Navier-Stokes equations.
  • Scope and future work: The study focuses solely on forward simulation, while applying causality to inverse problems is proposed through point-source observational data and remains for future exploration.The authors suggest minimizing residuals at observation locations before propagating information outward.
  • Scope and future work: Causal structure can differ across problems, as state and adjoint simulations in optimal control may require forward and backward temporal evolution, respectively.The authors retain the general message that training should respect information propagation in the underlying system.
  • Scope and future work: PINN forward solvers remain more time-consuming than traditional numerical solvers, and future work should improve training speed and architecture design.The discussion mentions distributed or parallel implementations and more specialized architectures as possible directions.

A Nomenclature

This appendix summarizes the notation used throughout the work.

  • Nomenclature: Table 2 provides the main symbols and notations used in the work.The appendix presents the nomenclature as a reference for the paper’s equations and discussion.

B Hyper-Parameters

This appendix documents the network architectures and training-cost conventions used across the numerical experiments.

  • Hyper-parameters: Table 3 summarizes the network hyper-parameters and architectures used for the numerical experiments.The hyper-parameters were tuned manually rather than optimized for an absolute best setting.
  • Computational cost: Training costs in Table 4 are reported for networks trained on NVIDIA RTX A6000 graphics cards.The table also lists model sizes and architectures from Table 3.
  • Computational cost: The default tolerance list is [10^-2, 10^-1, 10^0, 10^1, 10^2], with maximum iterations defined separately for each tolerance and time window.Total iterations can vary across examples because of the stopping criterion in Algorithm 1.

D Allen-Cahn equation

The Allen-Cahn benchmark uses a spectral reference solution and evaluates a causality-aware PINN against it across the full time interval and selected snapshots.

  • Validation: The reference Allen-Cahn solution starts from u0(x) = x2 cos(πx), uses periodic boundary conditions, and is integrated to T = 1.Validation data use Chebfun, 512 Fourier modes, ETDRK4 time stepping, and time-step size 10^-5.
  • Prediction accuracy: 2.46e −04 relative L2 error is obtained by the trained PINN using Algorithm 1 and a modified MLP.The figure compares the predicted and exact solutions over the benchmark domain.
  • Prediction accuracy: Figure 15 compares exact and predicted solutions at temporal snapshots t = 0.0, 0.5, and 1.0.Figure 16 additionally shows loss convergence, temporal residual loss, and temporal weights across training iterations.

E Lorentz system

The Lorenz-system experiment uses time-windowed PINN training and evaluates loss convergence and predicted solutions across windows.

  • The reference Lorenz solution is obtained with scipy.integrate.odeint using default settings.
  • The domain [0, 20] is split into 40 disjoint time windows of size Δt = 0.5.
  • The latent variables are represented by a 5-layer fully connected network with 512 neurons per hidden layer.
  • Because the Lorenz system is highly sensitive to initial conditions, the initial condition is imposed exactly.
  • The residual loss is used after reducing the loss function, with temporal weights updated by the proposed algorithm.
  • Figures 17–19 report per-window loss convergence and comparisons between reference and predicted solutions using Algorithm 1.

F Kuramoto–Sivashinsky equation

The Kuramoto–Sivashinsky experiments apply time-windowed PINNs to regular and chaotic regimes, with regime-specific architectures and loss settings.

  • Regular regime: Regular-regime validation uses spectral Fourier discretization with 512 modes and ETDRK4 time stepping at size 10^-5.
  • Regular regime: For the regular case, the temporal domain [0, 1] is split into 10 windows of size Δt = 0.1.
  • Regular regime: The regular-case solution uses a 5-layer modified MLP with 256 neurons per hidden layer and encoded periodicity.
  • Regular regime: The regular-case loss combines the PDE residual formulation with an initial-condition weight λic = 10^3 and mini-batch gradient descent.
  • Chaotic regime: For the chaotic case, [0, 0.5] is split into 5 windows of size Δt = 0.1, using a 10-layer MLP with 128 neurons per hidden layer and λic = 10^4.
  • Figures 20 and 21 show loss convergence for the regular and chaotic cases, respectively.

G Navier-Stokes equation

The Navier–Stokes experiment evaluates causality-aware PINN training on decaying turbulence, comparing predicted velocity and vorticity fields with high-resolution ground truth.

  • Validation data simulate two-dimensional decaying turbulence in a periodic box using a 1024 × 1024 Fourier spectral solver with dt = 10^-4.
  • The time domain [0, 1] is decomposed into 10 windows of size Δt = 0.1, with velocity represented by a 6-layer modified MLP.
  • Vorticity is approximated as wθ = ∂xvθ − ∂yuθ using automatic differentiation, then used to define the PDE residual.
  • The experiment uses Nt = 64, Nx = 512, λw = 1, λc = 10^2, and λic = 10^4, with random temporal and spatial collocation points.
  • At T = 2, the relative L2 error exceeds 10%, while the predicted velocity and vorticity fields remain qualitatively correct against ground truth.
  • Figures 22–25 compare predicted velocity components with ground truth and show loss convergence across time windows.
Loading 2203.07404v1…