Source-linked AI summary
A comparison study of deep Galerkin method and deep Ritz method for elliptic problems with different boundary conditions
Jingrun Chen, Rui Du, Keke Wu
TL;DR
Enforcing boundary conditions in DNN-based PDE solvers is nontrivial, and evidence comparing methods across boundary conditions is useful for understanding their relative performance. This study compares DGM and DRM on elliptic problems with four boundary-condition types. Its examples show that method performance depends on dimensionality and solution regularity, while exact boundary implementation can improve approximation and training in some cases.
Problem
Imposing boundary conditions on DNN representations is difficult, and the study examines how DGM and DRM compare across four boundary-condition types.
Source: §1 · p. 2Method
Results
Examples show DRM can outperform DGM for smooth solutions depending on dimension, DGM can outperform DRM for low-regularity solutions, and exact boundary implementation can improve approximation and training in some cases.
Source: §4 · p. 21Takeaways & Limitations
Method performance does not follow the expected smoothness-based division in all examples, and exact boundary implementation is beneficial in some cases.
Source: §4 · p. 21Choosing penalty parameters remains an important issue, and why DGM outperforms DRM on low-regularity problems remains to be understood.
Source: §4 · p. 21
Abstract
from arXiv · showhide
Recent years have witnessed growing interests in solving partial differential equations by deep neural networks, especially in the high-dimensional case. Unlike classical numerical methods, such as finite difference method and finite element method, the enforcement of boundary conditions in deep neural networks is highly nontrivial. One general strategy is to use the penalty method. In the work, we conduct a comparison study for elliptic problems with four different boundary conditions, i.e., Dirichlet, Neumann, Robin, and periodic boundary conditions, using two representative methods: deep Galerkin method and deep Ritz method. In the former, the PDE residual is minimized in the least-squares sense while the corresponding variational problem is minimized in the latter. Therefore, it is reasonably expected that deep Galerkin method works better for smooth solutions while deep Ritz method works better for low-regularity solutions. However, by a number of examples, we observe that deep Ritz method can outperform deep Galerkin method with a clear dependence of dimensionality even for smooth solutions and deep Galerkin method can also outperform deep Ritz method for low-regularity solutions. Besides, in some cases, when the boundary condition can be implemented in an exact manner, we find that such a strategy not only provides a better approximate solution but also facilitates the training process.
1 INTRODUCTION
The introduction frames DNNs as a way to address high-dimensional PDEs and focuses on comparing deep Galerkin and deep Ritz methods across four boundary conditions. It challenges expected method advantages based on solution smoothness and notes benefits from exact boundary enforcement in some cases.
1 Introduction
- DNNs avoid mesh points, and their degrees of freedom grow at most polynomially with PDE dimension, unlike the exponential growth in classical methods.The paper identifies this as an advantage for high-dimensional PDEs.Source: §1 · p. 2
- DNN boundary conditions are difficult to impose exactly, so losses often add a penalty for boundary mismatch; the study compares Dirichlet, Neumann, Robin, and periodic conditions.The introduction notes a distinct two-network construction for exact Dirichlet enforcement.Source: §1 · p. 2
- Although DGM is expected to favor smooth solutions and DRM low-regularity solutions, examples show DRM can win for smooth solutions depending on dimension, while DGM can win for low-regularity solutions.In some cases, exact boundary implementation improves approximation and facilitates training.Source: §1 · p. 2
- The paper introduces DGM and DRM, treats boundary conditions using a penalty method, compares examples, and concludes with a discussion.Source: §1 · p. 3
2 Methodology
The methodology treats neural PDE solving as a combination of loss design, network structure, and parameter optimization. It introduces DGM and DRM, then addresses boundary conditions, network design, and stochastic gradient descent.
2 Methodology
- The methodology organizes DNN-based PDE solving around the loss function, neural-network structure, and optimization over network parameters.Source: §2 · p. 3
- It introduces DGM and DRM, then discusses penalty-based treatment of different boundary conditions and the network structure used to approximate PDE solutions.Source: §2 · p. 3
- The methodology describes stochastic gradient descent as an often-used approach for optimizing the loss functions.Source: §2 · p. 3
2.1 Deep Ritz method and deep Galerkin method
DGM and DRM use a DNN approximation but differ in their loss functions: DGM minimizes the PDE residual in least squares, while DRM minimizes a variational objective. Boundary conditions enter through penalty terms, and the resulting optimization searches over admissible functions.
2.1 Deep Ritz method and deep Galerkin method
- Both methods represent the PDE solution with u(x; θ), but DGM minimizes the squared PDE residual while DRM minimizes a variational objective whose variation yields the PDE.The PDE setup assumes well-posedness; d is the dimension, f and g are given functions, L is a differential operator, and Γ represents the boundary condition.Source: §2.1 · pp. 3–4
- Boundary conditions are included in both methods by adding a penalty term weighted by the penalty parameter λ.Source: §2.1 · p. 4
- The approximation is obtained by optimizing the total loss over the admissible-function set H(Ω).Source: §2.1 · p. 4
2.2 Boundary conditions
The boundary-condition examples use a smooth cosine exact solution on the default domain and formulate losses for Dirichlet, Neumann, Robin, and periodic conditions. Each condition contributes a corresponding penalty term to the DGM and DRM losses.
2.2 Boundary conditions
- The examples default to Ω=(0,1)^d and use the smooth exact solution u(x)=Σ cos(πx_k), with f and g specified later.Source: §2.2 · p. 4
2.2.1 Dirichlet boundary condition
- For Dirichlet conditions, the paper specifies the boundary condition and its penalty term, then forms total DGM and DRM losses.Source: §2.2.1 · pp. 4–5
2.2.2 Neumann boundary condition
- For Neumann conditions, the boundary penalty is based on the outward normal derivative, and the corresponding total losses are given for both methods.Source: §2.2.2 · p. 5
2.2.3 Robin boundary condition
- For Robin conditions, the paper specifies a boundary penalty and incorporates it into the total losses for DGM and DRM.Source: §2.2.3 · p. 5
2.2.4 Periodic boundary condition
- For periodic conditions, the penalty comprises two terms, which are incorporated into the corresponding DGM and DRM loss functions.Source: §2.2.4 · p. 6
2.3 Network structure
The paper uses residual blocks with a skip connection to build its neural-network approximation of PDE solutions. The architecture has an affine input and output layer, and its parameter count depends on the input dimension, width, and number of blocks.
2.3 Network structure
- Each residual block combines two weight layers and nonlinear transformations with an identity skip connection.The block maps its input to an output while adding the input through the skip connection.Source: §2.3 · p. 6
- The network applies an affine input layer, stacks residual blocks, then uses an affine output layer.The input layer maps d-dimensional inputs to width m; the output layer maps the final block to a scalar.Source: §2.3 · p. 6
- The final activation is adaptive swish, whose additional parameter a is optimized during training.Source: §2.3 · p. 7
- The total parameter count is m(d + 1) + (2mn + 1)(m + 1) for width m, input dimension d, and n residual blocks.Source: §2.3 · pp. 6–7
2.4 Stochastic gradient descent algorithm
The authors use stochastic gradient descent with Monte Carlo mini-batches to optimize the neural-network loss. They use Adam in implementation and approximate loss derivatives with finite differences.
2.4 Stochastic gradient descent algorithm
- SGD uses mini-batch sample gradients, incorporating Monte Carlo sampling to avoid the curse of dimensionality in high-dimensional integrals.At each iteration, it evaluates the loss gradient only at a batch of samples.Source: §2.4 · p. 8
- Uniformly sampled points from the domain and boundary provide the single-sample loss approximations used in each SGD update.Source: §2.4 · p. 8
- Adam is used to accelerate training, and finite differences approximate derivatives in the loss function.The global learning rate is 0.001, with moment decay rates 0.9 and 0.999 and numerical-stability constant 10^-8.Source: §2.4 · p. 8
3 Numerical results
Approximation error is assessed using relative L2 error, comparing the DGM or DRM neural-network approximation with the exact solution.
3 Numerical results
- Relative L2 error measures the approximation error by comparing the DGM or DRM network solution with the exact solution.Source: §3 · p. 8
3.1 Training process and dimensional dependence for four boundary conditions
Across the tested dimensions, relative-error comparisons show a shift in which method performs better, while training-speed comparisons also depend on dimensionality. The study additionally tests penalty-free constructions for Dirichlet and periodic boundary conditions.
3.1 Training process and dimensional dependence for four boundary conditions
- The authors report penalty-free results for Dirichlet and periodic conditions; exact enforcement is practical when a suitable boundary extension and distance function are available.For Neumann and Robin conditions, finding a natural extension satisfying the boundary condition is unclear in practice.Source: §3.1 · p. 9
- DGM converges faster in low dimensions, while DRM is faster in high dimensions; DGM requires substantial effort to converge in 16D.The penalty parameter is tuned for approximation, and larger damping implies better agreement with the exact solution on the boundary.Source: §3.1 · p. 9
- The network widens across the training-process comparisons, from three residual blocks with four units per layer in 2D to three blocks with 32 units in 16D.The supplied 4D and 8D figure captions specify three blocks with eight and 16 units per layer, respectively.Source: Figure 2 · p. 10, Figure 3 · p. 11, Figure 4 · p. 12
- DGM generally has better relative-error accuracy at 2D and 4D, whereas DRM outperforms at 8D and 16D, despite the smooth exact solution.For problem (5), DGM uses second-order derivatives while DRM uses first-order derivatives, so the observed pattern is contrary to the authors’ expectation.Source: §3.1 · p. 13
3.2 Dependence on network structures
The section tests how approximation accuracy depends on penalty parameters, sampling, activation functions, and network structure, including periodic-boundary experiments.
3.2.1 Penalty parameter
- For 4D Dirichlet problems, larger penalty parameters improve approximation until extremes can make the penalty term negligible or dominant, producing solutions that satisfy only the PDE or boundary condition.Choosing a finite penalty parameter that gives optimal accuracy remains an open consideration for a given network.Source: §3.2.1 · p. 14, Table 2 · p. 14
3.2.2 Mini-batch size
- For 4D Robin problems, the authors expect balancing domain and boundary mini-batch sizes to yield optimal approximation accuracy.Tables 3 and 4 vary domain and boundary batch sizes separately.Source: §3.2.2 · p. 15, Table 3 · p. 15, Table 4 · p. 15
3.2.3 Activation function
- For 4D Neumann problems, ReLU fails in DGM because its low regularity interacts with higher derivatives in DGM’s loss; the expected DRM advantage on low-regularity solutions also does not hold in Section 3.3.The authors also note that DGM’s expected advantage for smooth solutions is not universal, based on Section 3.1.Source: §3.2.3 · p. 15, Table 5 · p. 16
3.2.4 Neural depth and neural width
- For 4D Dirichlet problems, increasing neural depth or width is expected to reduce error to some extent, but DNN errors do not decrease systematically.Tables 6 and 7 vary depth and width, respectively.Source: §3.2.4 · p. 16, Table 6 · p. 16, Table 7 · p. 16
3.2.5 Periodic boundary condition
- For periodic problems, the chosen exact solution cannot be explicitly represented by the input transform, and Table 8 reports no-penalty relative L2 errors with k = 3.The preceding exact solution could be exactly expressed by the transform, so the authors choose a different one.Source: §3.2.5 · p. 17, Table 8 · p. 17
3.3 A nonlinear problem with low-regularity solution
For a nonlinear problem with a C1 solution, the experiments compare DGM and DRM across dimensions and training choices. DGM outperforms DRM for this low-regularity solution, while exact boundary enforcement improves training and approximation accuracy.
3.3 A nonlinear problem with low-regularity solution
- The low-regularity test is a nonlinear PDE on the unit sphere with homogeneous Dirichlet boundary conditions.The problem is −∆u + u^3 = f(x) in the domain, with u(x) = 0 on its boundary.Source: §3.3 · p. 17
3.3.1 Dimensional dependence
- Across d=2, 4, and 8, DGM outperforms DRM by over one order of magnitude for the C1 solution.The authors report that this result persists across penalty parameters, mini-batch sizes, activation functions, neural depths, and widths.Source: Table 9 · p. 18, §3.3.1 · p. 18
3.3.2 Penalty parameter
- In 2D, Table 10 reports relative L2 errors for DGM and DRM across penalty-parameter values.The table uses swish activation, three residual blocks with eight units per layer, and domain and boundary mini-batches of 2000 and 400.Source: Table 10 · p. 19, §3.3.2 · p. 19
3.3.3 Activation function
- In 4D, swish is the best tested activation function, while ReLU still has a problem.Table 11 compares relative L2 errors across activation functions.Source: §3.3.3 · p. 19, Table 11 · p. 19
3.3.4 With versus without penalty
- Without a penalty, both methods converge better, and exact boundary enforcement improves relative L2 errors by over one order of magnitude for both methods.DGM remains more accurate than DRM regardless of penalty; the comparison uses the trial function u(x; θ) = (1 − |x|)DNN(x; θ).Source: §3.3.4 · pp. 19–20
4 CONCLUSIONS
The study compares DGM and DRM for Dirichlet, Neumann, Robin, and periodic boundary conditions. Its examples show that either method can outperform the other depending on dimensionality and solution regularity, and exact boundary enforcement can improve approximation and training.
4 Conclusions
- Across the four boundary conditions, DRM can outperform DGM for smooth solutions depending on dimensionality, while DGM can outperform DRM for low-regularity solutions.This contrasts with the expected pattern that DGM would work better for smooth solutions and DRM for low-regularity solutions.Source: §4 · p. 21
- In some cases, exact boundary enforcement provides a better approximation and facilitates training.The authors identify this as a benefit of implementing the boundary condition exactly.Source: §4 · p. 21
- The choice of penalty parameter remains an open issue because it affects training and approximation for a fixed network structure.The authors also identify understanding why DGM outperforms DRM on low-regularity problems as a question for further study.Source: §4 · p. 21