Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems

Kevin Linka; Amelie Schafer; Xuhui Meng; Zongren Zou; George Em Karniadakis; Ellen Kuhl

Source-linked AI summary

Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems

Kevin Linka, Amelie Schafer, Xuhui Meng, Zongren Zou, George Em Karniadakis, Ellen Kuhl

arXiv:2205.08304v2May 12, 2022cs.LGmath.DSnlin.CD

TL;DR

Nonlinear dynamical systems are difficult to model from real-world data because neural networks can ignore physical laws and data are noisy and incomplete. The paper embeds a damped harmonic oscillator in a neural network and combines physics-informed modeling with Bayesian inference, finding improved extrapolation and uncertainty quantification, while noting sensitivity to scaling and limited uncertainty quantification in some methods.

Problem
Neural networks can fit data while ignoring physical laws, and real-world dynamical data are noisy and incomplete, creating a need for physically informed models with uncertainty quantification.
Method
The study combines a fully connected neural network, a damped harmonic oscillator, Physics-Informed Neural Networks, and Bayesian Inference using real-world COVID-19 case data.
Results
Physics-Informed Neural Networks fit both observed data and behavior outside the training window, while Bayesian Inference additionally provides credible intervals for uncertainty quantification.
Takeaways & Limitations
The comparison reveals distinct advantages and disadvantages among Neural Networks, Bayesian Inference, and their combination, supporting model selection for nonlinear dynamical systems.
Takeaways & Limitations
The method is sensitive to scaling and can fail to converge when scaled inappropriately.

Abstract

from arXiv · show

Understanding real-world dynamical phenomena remains a challenging task. Across various scientific disciplines, machine learning has advanced as the go-to technology to analyze nonlinear dynamical systems, identify patterns in big data, and make decision around them. Neural networks are now consistently used as universal function approximators for data with underlying mechanisms that are incompletely understood or exceedingly complex. However, neural networks alone ignore the fundamental laws of physics and often fail to make plausible predictions. Here we integrate data, physics, and uncertainties by combining neural networks, physics-informed modeling, and Bayesian inference to improve the predictive potential of traditional neural network models. We embed the physical model of a damped harmonic oscillator into a fully-connected feed-forward neural network to explore a simple and illustrative model system, the outbreak dynamics of COVID-19. Our Physics-Informed Neural Networks can seamlessly integrate data and physics, robustly solve forward and inverse problems, and perform well for both interpolation and extrapolation, even for a small amount of noisy and incomplete data. At only minor additional cost, they can self-adaptively learn the weighting between data and physics. Combined with Bayesian Neural Networks, they can serve as priors in a Bayesian Inference, and provide credible intervals for uncertainty quantification. Our study reveals the inherent advantages and disadvantages of Neural Networks, Bayesian Inference, and a combination of both and provides valuable guidelines for model selection. While we have only demonstrated these approaches for the simple model problem of a seasonal endemic infectious disease, we anticipate that the underlying concepts and trends generalize to more complex disease conditions and, more broadly, to a wide variety of nonlinear dynamical systems.

1 Motivation

The study addresses the difficulty of predicting nonlinear dynamical systems when data are abundant but noisy, incomplete, and insufficiently interpretable. It investigates whether combining physical knowledge with neural networks and Bayesian uncertainty quantification improves modeling of real-world dynamics.

Traditional machine learning can fit observations but often ignores physical laws and produces inconsistent or implausible predictions.
Physics-informed machine learning incorporates known physical parameters, boundary conditions, and constraints into training.This can make training more efficient, accurate, and robust, particularly for data-limited settings.
Real-world data are stochastic, noisy, and incomplete, motivating models that quantify uncertainty rather than relying on data alone.
The study evaluates Bayesian Physics-Informed Neural Networks on real-world COVID-19 data where the underlying nonlinear dynamics are not fully known.
A damped harmonic oscillator provides a simple, interpretable model for comparing Neural Networks, Bayesian Inference, and their combination.The authors explicitly do not seek a rigorous physical explanation of every feature of global pandemic dynamics.

2 Model problem

The paper models worldwide daily COVID-19 cases with a fully connected neural network and a damped harmonic oscillator, then combines data, physics, and Bayesian inference. The resulting comparisons show distinct trade-offs between fitting, extrapolation, and uncertainty quantification.

Data: The dataset is worldwide daily COVID-19 case counts from January 1 through December 31, 2021, considered both as raw reports and seven-day averages.The raw series is fluctuating, whereas the seven-day moving average is smoother.
Neural Network modeling: The classical neural network takes time t as input and approximates daily cases x(t) with a fully connected feed-forward architecture.The illustrative network uses two hidden layers with 32 nodes each and one output.
Physics-informed modeling: The physics model is a damped harmonic oscillator governed by a second-order differential equation with damping, stiffness, and offset parameters.The parameter vector is ϑ = {c, k, x0}, with mass fixed at m = 1.
Physics-informed modeling: The oscillator’s damping ratio determines whether the system is overdamped, critically damped, or underdamped.
Integrating data and physics: Physics-Informed Neural Networks combine data fitting with physics constraints, while Bayesian Inference quantifies uncertainty in the learned parameters.
Results: Neural Networks fit the first 225 training days but fail to predict the remaining 140 days outside the training window.This behavior occurs for both raw daily cases and seven-day averaged data.
Results: Physics-Informed Neural Networks approximate both the training data and the subsequent 140 days with good accuracy.Their fit is virtually identical when trained on daily case data or seven-day averages.
Results: Bayesian Inference provides credible intervals while approximating both the training data and behavior outside the training window.Its training-data fit is worse than the Physics-Informed Neural Network because the solution is biased toward the physics equations, and raw-data training produces wider intervals than averaged-data training.

3 Neural Network modeling

Neural Network models are trained by minimizing data and, for Physics Informed Neural Networks, physics losses to learn network and physical parameters. Varying the weighting between these losses trades descriptive data fit against predictive physics fit, while self-adaptive models learn this weighting over time.

3.1 Neural Networks: Classical Neural Networks minimize data loss to learn network weights and biases without using prior physical knowledge.The data loss is defined as the mean square error between model output and observations and is optimized with ADAM.
3.2 Physics Informed Neural Networks: Physics Informed Neural Networks jointly learn network parameters and physical parameters by minimizing a weighted sum of data loss and physics residual.Automatic differentiation evaluates the model time derivatives needed for the physics residual.
3.3 Self Adaptive Physics Informed Neural Networks: Self-adaptive Physics Informed Neural Networks learn the time-dependent weighting ε(t) alongside network and physical parameters.The learned weighting reveals how the relative importance of data and physics losses changes over time.
3.4 Comparison of Neural Network models: With ε = 0, the Physics Informed Neural Network reduces to a classical Neural Network; increasing ε shifts emphasis from data fit toward physics fit.The weighting range shown is ε = [10^-∞, 10^-5, 10^-4, 10^-3, 10^-2, 10^0], while ε = 1 reduces the objective to the physical model and yields a zero-amplitude solution.
3.4 Comparison of Neural Network models: The classical Neural Network fits the first 225 training days but fails beyond the training window, whereas weighting selection determines the balance between descriptive and predictive performance.Smaller ε improves data fit during the descriptive regime, while larger ε improves physics fit during the predictive regime.
3.4 Comparison of Neural Network models: A weighting coefficient near ε ≈ 10^-3 is reasonable, matching the self-adaptive model’s learned mean ε = 0.007 ± 0.003.For ε = 10^-3, the physics parameters converge after 60K epochs to c = 1.251, k = 374.6, and x0 = 0.558.

4 Bayesian Inference modeling

The section formulates Bayesian inference for data-fitting and physics satisfaction, then compares classical, neural-network, and physics-informed variants on COVID-19 case data. The Bayesian Physics-Informed Neural Network jointly estimates network and physical parameters while producing posterior uncertainty.

Bayesian formulation: Bayesian inference estimates parameter posteriors so model statistics agree with observed data and, when included, satisfy the physics equation.The likelihood combines data fit and physics satisfaction, while priors weight parameter plausibility.
Bayesian formulation: The likelihood models daily observations with normal distributions centered on model outputs, with σ representing observation error.Multiplying the pointwise likelihoods over 365 days defines the overall data likelihood.
Bayesian formulation: The physics likelihood evaluates how accurately the simulated case trajectory satisfies r(t) = 0 at each day, then aggregates these daily evaluations.This adds a probabilistic measure of physical consistency to the data-fitting objective.
Classical Bayesian Inference: Classical Bayesian inference estimates posterior distributions for damping c, stiffness k, and offset x0 under a damped-harmonic-oscillator model.The model assumes an underdamped system with m ≡ 1 and c^2 < 4k.
Bayesian neural-network variants: Bayesian neural networks infer posterior distributions over network parameters, whereas Bayesian Physics-Informed Neural Networks infer both network and physics parameters.The latter combines the physics-informed network with Bayesian inference so outputs fit data and satisfy the physics equation.
Comparison of methods: For 225 training days, the Bayesian Physics-Informed Neural Network produced wider parameter uncertainties and was more expensive and less robust than classical Bayesian inference.Its standard deviations were larger by factors of 3 for damping, 23 for stiffness, and 10 for offset.

5 Discussion

The discussion compares neural, physics-informed, and Bayesian approaches for modeling nonlinear dynamics, emphasizing predictive behavior, uncertainty, data requirements, and scaling sensitivity.

Training-data convergence: Predictive data loss decreases as the training set grows for all three Physics Informed Neural Network variants, while Bayesian Physics Informed Neural Networks improve only marginally.The Self Adaptive model performs well even with small training sets, whereas the classical model converges as data increases.
Neural Networks: Classical Neural Networks fit training data well but extrapolate poorly because they lack physical constraints and credible intervals.Their predictions can continue a linear trend beyond the training window even when the underlying dynamics are oscillatory.
Self Adaptive Physics Informed Neural Networks: Self Adaptive Physics Informed Neural Networks learn a time-varying weighting term, addressing bias between data and physics and improving performance with smaller training sets.The adaptive weighting allows the model to perform well in regions with steep gradients using fewer training epochs.
Physics Informed Neural Networks: Physics Informed Neural Networks integrate data with physics, improving interpolation and extrapolation while remaining sensitive to the data–physics weighting coefficient.They can be trained effectively with limited data, but the example required larger training sets for strong convergence.
Bayesian Physics Informed Neural Networks: Bayesian Physics Informed Neural Networks provide good fits, predictions, and narrower credible intervals than Bayesian Neural Networks, but their performance depends on appropriate scaling.Scaling the physics equation can substantially improve performance by narrowing credible intervals.

Self Adaptive Physics Informed Neural Networks provide adaptive and robust fits

Self Adaptive Physics Informed Neural Networks adapt the balance between data and physics, supporting robust fits with smaller training sets while retaining sensitivity to scaling.

Adaptive weighting: Self Adaptive Physics Informed Neural Networks introduce a time-varying weighting coefficient to reduce bias between data and physics.This adaptive term allows good performance in regions with steep gradients and can reduce the number of training epochs.
Robust fits: They perform well even with small training sets while inheriting the predictive advantages of Physics Informed Neural Networks.Their adaptive weighting addresses the fixed-balance limitation of standard Physics Informed Neural Networks.
Scaling limitation: The method is sensitive to scaling and could fail to converge when the physics equation is scaled inappropriately.Scaling is therefore a practical condition for reliable optimization.

Bayesian Inference fits a physics based model to data and provides credible

Bayesian approaches combine physics-informed or neural models with parameter distributions and credible intervals. The Bayesian Physics Informed Neural Network extends this framework to fit and prediction while requiring careful scaling and substantial computation.

Bayesian Inference: Classical Bayesian Inference fits a physics-based model to data by inferring distributions for damping, stiffness, and offset parameters.It provides credible intervals when the hypothesized physics-based model is reasonable.
Bayesian Neural Networks: Bayesian Neural Networks fit data without underlying physics and provide credible intervals through distributions over network parameters.Their wide credible intervals indicate unreliable predictions beyond the training window in the example.
Bayesian Physics Informed Neural Networks: Bayesian Physics Informed Neural Networks learn distributions for both network and physics parameters, combining data fitting, physical structure, and uncertainty quantification.The narrower credible intervals relative to Bayesian Neural Networks indicate improved prediction when physics is included.
Scope and cost: Bayesian Physics Informed Neural Networks are sensitive to scaling and have a larger computational cost because they include distributions over network and physics parameters.Their scope was demonstrated on a simple seasonal endemic infectious-disease model.
Implications: Embedding physical principles into neural architectures can produce effective models that train with small amounts of data and support forward, inverse, interpolation, and extrapolation tasks.Combining Physics Informed Neural Networks with Bayesian Neural Networks also provides posterior estimators for uncertainty quantification.