Variational Physics-Informed Neural Networks For Solving Partial Differential Equations

E. Kharazmi; Z. Zhang; G. E. Karniadakis

Source-linked AI summary

Variational Physics-Informed Neural Networks For Solving Partial Differential Equations

E. Kharazmi, Z. Zhang, G. E. Karniadakis

arXiv:1912.00873v1Nov 27, 2019cs.NEcs.LGmath.NAphysics.comp-phstat.ML

TL;DR

The paper addresses the cost and accuracy limitations of strong-form PINNs for PDEs. It develops a Petrov–Galerkin VPINN using neural networks as trial functions and Legendre-polynomial test functions, with variational losses and integration by parts. Across the reported examples, VPINNs show advantages over PINNs in accuracy and speed, while initialization and loss sensitivity remain important limitations.

Problem
PINNs enforce PDEs at random points using strong-form residuals, motivating a formulation that can reduce derivative and penalization costs while retaining accurate PDE solutions.
Method
VPINN combines DNN nonlinear trial functions with linear test functions in a Petrov–Galerkin variational loss, using integration by parts to lower differential-operator order.
Results
VPINNs demonstrate clear advantages over PINNs in accuracy and speed across several examples.
Takeaways & Limitations
Variational residuals can replace many PINN penalizing points with fewer quadrature points and support domain decomposition with locally chosen test functions.
Takeaways & Limitations
Sine-activation VPINNs require careful initialization, and increasing neurons or test functions can make optimization sensitive to weights and biases.

Abstract

from arXiv · show

Physics-informed neural networks (PINNs) [31] use automatic differentiation to solve partial differential equations (PDEs) by penalizing the PDE in the loss function at a random set of points in the domain of interest. Here, we develop a Petrov-Galerkin version of PINNs based on the nonlinear approximation of deep neural networks (DNNs) by selecting the {\em trial space} to be the space of neural networks and the {\em test space} to be the space of Legendre polynomials. We formulate the \textit{variational residual} of the PDE using the DNN approximation by incorporating the variational form of the problem into the loss function of the network and construct a \textit{variational physics-informed neural network} (VPINN). By integrating by parts the integrand in the variational form, we lower the order of the differential operators represented by the neural networks, hence effectively reducing the training cost in VPINNs while increasing their accuracy compared to PINNs that essentially employ delta test functions. For shallow networks with one hidden layer, we analytically obtain explicit forms of the \textit{variational residual}. We demonstrate the performance of the new formulation for several examples that show clear advantages of VPINNs over PINNs in terms of both accuracy and speed.

1. Introduction.

The paper motivates replacing strong-form PINNs with a Petrov–Galerkin VPINN that combines neural-network trial functions with linear test functions. Its variational formulation reduces derivative and sampling costs while supporting flexible local learning.

Nonlinear approximation extends beyond linear spaces, aiming for robust estimates through sparser representations and cheaper computation.
PINNs incorporate physical models into neural-network losses, but their strong-form formulations enforce conservation laws at random space-time points.
VPINNs use DNNs as nonlinear trial functions and linear-space test functions within a Petrov–Galerkin variational formulation.
Integration by parts lowers differential-operator order, reduces required solution regularity, and makes training less computationally expensive.
A relatively small number of quadrature points replaces the large number of penalizing points used by PINNs.
Domain decomposition allows local test-function counts to follow solution regularity, yielding a more flexible elemental learning approach.

2. Nonlinear Function Approximation.

DNNs provide nonlinear function approximations through compositions of affine mappings and nonlinear activations. Their practical accuracy reflects approximation, generalization, and optimization errors, with optimal approximation remaining unresolved.

A DNN represents a target function through a composition of nonlinear approximators and a final linear mapping.
For fixed depth, width, and activation, training minimizes a norm measuring discrepancy between the target and network output.
DNN accuracy decomposes into approximation, generalization, and optimization errors.
High-dimensional inputs and deep structures can create numerical issues such as local-minimum traps, so practical training obtains an approximate optimizer.
Achieving an optimal DNN approximation depends on multiple factors and remains an open problem.

3. Variational Physics-Informed Neural Networks (VPINNs).

The section contrasts PINNs, which minimize strong-form residuals at interior and boundary points, with VPINNs, which minimize variational residuals using neural-network trial functions and selected test functions. VPINNs replace pointwise residual evaluation with test-function integration, while introducing integral-computation challenges.

PINNs: PINNs approximate the solution with a DNN and penalize strong-form residuals at interior points together with boundary-condition violations at boundary points.The strong-form loss uses interior residuals L_quNN(x)-f(x) and boundary residuals uNN(x)-h(x), weighted through a penalty parameter.
PINNs: The strong-form PINN formulation is a penalty-like method whose penalty parameter is selected for the problem, often through numerical experiments or as a hyperparameter.The formulation adds boundary conditions to the interior equation through a penalty term, but no general analysis of PINNs as penalty methods is provided here.
VPINNs: VPINNs represent the solution with a neural network, multiply the governing equation by admissible test functions, integrate over the domain, and optimize a variational loss.A finite-dimensional test space VK is formed from selected test functions, and the neural-network weights and biases minimize the resulting variational loss.
VPINNs: In VPINNs, trial functions are neural networks while test functions may come from separate neural networks or spaces such as polynomials and trigonometric functions.The Petrov-Galerkin construction keeps the trial and test spaces distinct.
Residual formulation: Choosing delta test functions recovers the strong-form residual at penalizing points, whereas general test functions produce variational residuals without such pointwise penalizing points.The variational residual is enforced against a set of test functions rather than evaluated only at selected points.
Computational challenge: Computing VPINN loss integrals is a major challenge: shallow one-hidden-layer networks permit analytical derivations in special cases, while deep networks require numerical integration.The compositional structure of deeper networks makes analytic integral expressions difficult and quadrature analysis incomplete.

4. Shallow VPINNs.

The paper constructs shallow VPINNs by combining one-hidden-layer neural-network trial functions with sine or polynomial test functions in a variational Petrov-Galerkin formulation. For one-dimensional steady Burgers’ and related problems, the method derives analytical residuals and examines accuracy, convergence, initialization, and penalty effects.

Formulation: A shallow VPINN represents the solution with a one-hidden-layer neural network and enforces the PDE through variational residuals against compactly supported test functions.The construction uses N neurons and K test functions, with boundary residuals included in the loss.
Formulation: Integration by parts yields three variational residual forms, with boundary treatment differing across formulations.For one formulation, compact test-function support removes a boundary term, while a penalty coefficient controls boundary accuracy.
Analytical residuals: For shallow networks with sine activation and sine test functions, the variational residuals and loss expressions can be obtained analytically.The paper presents final expressions after trigonometric simplification and gives detailed derivations in the appendix.
Numerical examples: In the Burgers’ example, the best successful initialization achieved an L8-norm error of order 10^-9, while sine-activation PINN errors were O(1).A comparable tanh PINN reached O(10^-5), but the reported VPINN result used a simpler network.
Numerical examples: For periodic Burgers’ problems, increasing neurons and test functions reduced whole-domain error until optimization error caused saturation.The best L8 projection error was O(10^-4) with N = K = 5, while other combinations were less accurate.
Polynomial test functions: Polynomial test functions performed much better than sine test functions for non-sinusoidal exact solutions, but their residuals became more initialization-sensitive as test-function counts increased.Higher network expressivity could improve approximation if optimization succeeded, while also making the loss harder to train.

5. Shallow to Deep VPINNs.

The paper extends VPINNs from shallow to deep networks and evaluates them on one- and two-dimensional Poisson problems. Deep networks reduce error substantially but make training less stable, while VPINNs accurately resolve steep and boundary-layer solutions, with multidimensional quadrature adding computational cost.

5. Shallow to Deep VPINNs.: Deep VPINNs require numerical quadrature because deep network compositions generally prevent analytic evaluation of the variational integrals.The formulation evaluates network outputs or derivatives at quadrature points, allowing different activation functions, test functions, and quadrature rules.
5. Shallow to Deep VPINNs.: The experiments use fabricated solutions for one- and two-dimensional Poisson equations, obtaining forcing terms by substituting the exact solutions into the equations.The one-dimensional results are shown in Figs. 9 and 10, while the two-dimensional results are shown in Figs. 11 and 12.
5. Shallow to Deep VPINNs.: Increasing the number of hidden layers drops the error by orders of magnitude, demonstrating the greater expressivity of deep networks.With similar optimizers, deeper networks also exhibit less stable training and more spikes in the loss over iterations.
5. Shallow to Deep VPINNs.: VPINNs are more accurate than PINNs for boundary-layer solutions, whereas their results are relatively similar for steep solutions.PINNs require many penalizing points clustered near the boundary layer to capture its sharp variation; VPINN point-wise errors are oscillatory because residuals are projected onto hierarchical polynomial test functions.
5. Shallow to Deep VPINNs.: For the two-dimensional Poisson problem, VPINNs accurately capture the exact solution, with maximum point-wise error near the steep change and smaller error elsewhere.The two-dimensional test functions are constructed from tensor products of one-dimensional subspaces, and the network and VPINN parameters are given in Table 2.
5. Shallow to Deep VPINNs.: Higher-dimensional VPINNs incur additional computational cost from multidimensional integration; the reported example evaluates integrands at 70×70 quadrature points.More complicated solutions, higher dimensions, and larger test-function sets require further increases in quadrature points.

6. Summary.

The paper concludes that VPINNs combine neural-network trial functions with polynomial or trigonometric test functions in a Petrov-Galerkin formulation. Integration by parts lowers differential-operator order, while shallow networks permit analytic residuals and deep networks require numerical integration.

6. Summary.: VPINNs use DNNs as nonlinear trial functions and polynomial or trigonometric functions as test functions within a Petrov-Galerkin method.The method incorporates the variational form of the governing model rather than penalizing the strong-form residual directly at random points.
6. Summary.: Integration by parts reduces the differential-operator order and can lower the regularity required of the neural-network output.This is the principal computational and analytical advantage identified for the variational formulation.
6. Summary.: For shallow one-hidden-layer networks, the paper analytically derives distinct variational residual forms, supporting further numerical analysis of VPINNs.The appendices derive residual expressions for sine activation and sine or Legendre test functions, under assumptions such as w_j ≠ kπ.
6. Summary.: Deep networks can approximate more accurately, but their variational formulations require numerical integration.The paper presents quadrature-based residual expressions and notes that the choice of activation, test functions, and quadrature rules can vary.
6. Summary.: The shallow-network derivations use sine activation functions and sine or Legendre test functions, with recursive integral formulas obtained through integration by parts.The appendix develops recursion formulas for the relevant integrals and auxiliary quantities.