PaperlayerView exact revision on arXiv

Source-linked AI summary

A unified deep artificial neural network approach to partial differential equations in complex geometries

Jens Berg, Kaj Nyström

arXiv:1711.06464v2stat.MLcs.LG

TL;DR

The paper addresses PDE approximation in complex geometries, where classical mesh-based techniques can become infeasible. It develops a unified deep feedforward ANN method with modified backpropagation and reports complex-geometry examples plus benefits from increased depth and convergence techniques.

  • Problem

    Classical mesh-based PDE methods require discretization and can become infeasible for high-dimensional problems and complex domains.

  • Method

    The method uses a solution ansatz, deep feedforward ANNs, modified backpropagation for spatial derivatives, and unconstrained gradient-based optimization.

  • Results

    The method solves advection and diffusion PDE examples in 1D and 2D, including a highly complex polygon where traditional mesh-based methods are infeasible.

  • Takeaways & Limitations

    Increasing hidden-layer depth reduces the training iterations needed to reach a specified accuracy, with further gains expected from techniques addressing vanishing gradients.

  • Takeaways & Limitations

    The ANN method is not competitive with classical methods for low-dimensional problems on simple polygonal geometries with high-quality meshes.

Abstract

from arXiv · show

In this paper we use deep feedforward artificial neural networks to approximate solutions to partial differential equations in complex geometries. We show how to modify the backpropagation algorithm to compute the partial derivatives of the network output with respect to the space variables which is needed to approximate the differential operator. The method is based on an ansatz for the solution which requires nothing but feedforward neural networks and an unconstrained gradient based optimization method such as gradient descent or a quasi-Newton method. We show an example where classical mesh based methods cannot be used and neural networks can be seen as an attractive alternative. Finally, we highlight the benefits of deep compared to shallow neural networks and device some other convergence enhancing techniques.

1 Introduction

The paper motivates deep feedforward ANNs as mesh-free approximators for PDEs in complex geometries, while targeting a unified method with minimal user intervention. It also investigates pre-training and network depth as ways to reduce training iterations.

  • Motivation: Mesh-based PDE methods are efficient for low-dimensional regular geometries but become difficult to use for complicated domains requiring suitable discretizations.Traditional approaches include FEM, FDM, and FVM.
  • Motivation: Mesh-free collocation methods simplify point generation inside domains but can suffer from numerical instability or inefficiency.The passage identifies RBF and Monte Carlo methods as examples.
  • Motivation: Deep ANNs provide smooth analytical approximations that can be evaluated at points inside or outside the domain without reconstruction.The paper situates this advantage within an emerging literature on deep ANNs for PDEs.
  • Contribution: The proposed unified method uses only deep feedforward ANNs, a cost function, and an unconstrained numerical optimization method, with close to no user intervention.It avoids auxiliary user-defined functions, radial basis function ANNs, and thin-plate-spline linear-system computations used in earlier approaches.
  • Convergence considerations: With fixed network capacity, increasing hidden-layer depth can dramatically reduce iterations needed to reach a desired accuracy, and boundary-data pre-training is another convergence aid.The authors report this as evidence that depth adds value for ANN-based PDE solving.

2 Network architecture

The paper uses fully connected feedforward ANNs to map inputs to outputs through weighted, biased layers and activation functions. Their parameters are calibrated by minimizing a cost function with gradients computed through backpropagation.

  • Architecture: A deep ANN has an input layer, an output layer, and hidden layers whose activation functions may include sigmoids, rectified linear units, or hyperbolic tangents.Unless otherwise stated, the paper uses sigmoid hidden activations and a linear output activation.
  • Architecture: Each non-input neuron has a bias, while weight matrices represent connections between neurons in successive layers.The paper denotes neuron biases by b^l_j and connection weights by w^l_jk.
  • Forward computation: The network computes layer outputs recursively by applying activation functions elementwise to weighted inputs formed from previous-layer outputs.The weighted input is expressed through the previous layer and is used to define the deep recursion.
  • Parameter calibration: Given input x and target y, the weights and biases are chosen so the network output y^L(x; w, b) approximates y by minimizing a cost function C.The cost function maps the target and network output to a scalar objective.
  • Backpropagation: The paper focuses on gradient-based optimization and uses backpropagation to compute the cost gradients required for parameter calibration.It also introduces input derivatives needed for PDE applications.

3 Unified ANN approximations to PDEs

The paper formulates PDE solution approximations with a unified ansatz built only from feedforward ANNs, while modifying gradient algorithms to handle spatial and parameter derivatives.

  • PDE formulation: The PDE approximation uses collocation points and minimizes a quadratic residual over discrete domain and boundary constraints.The constrained problem can instead be transformed into unconstrained optimization by designing the ansatz to satisfy boundary conditions automatically.
  • Ansatz: For Dirichlet conditions, the ansatz combines a smooth boundary-data extension G(x) with a smooth distance function D(x) and the network output.Because D vanishes on the boundary, the construction ensures that the approximation attains the prescribed boundary values.
  • Ansatz: G and D are pre-computed with low-capacity feedforward ANNs, so the unified method uses neural networks and gradient-based unconstrained optimization throughout.G is trained on boundary collocation points, while D is fitted from sampled distances and boundary points.
  • Distance function: The smooth distance function can be computed with nearest-neighbor searches whose typical complexity is O(Nd log Nb) instead of the naive O(NdNb).The efficient search options include k-d trees and ball trees for very high-dimensional inputs.
  • Distance function: Simulations indicate that one distance-function construction can negatively affect convergence and solution quality.The paper therefore treats the exact form of the distance approximation as a design consideration beyond smoothness requirements.
  • Gradient computations: Deep ANNs require modified feedforward and backpropagation algorithms to compute derivatives with respect to spatial variables and network parameters.These derivatives are needed both for applying the differential operator and for optimizing the quadratic residual.

4 Numerical examples

The numerical examples apply the modified gradient algorithms to one-dimensional advection and diffusion problems, using BFGS and small deep networks. The authors report BFGS as the strongest tested optimizer and identify pre-training and depth as convergence-related factors.

  • Optimization: The examples use modified gradient schemes with SciPy's default BFGS optimizer, which outperformed the other tested gradient-based, gradient-free, and descent methods.The comparisons included online, batch, and stochastic gradient descent methods.
  • Optimization: BFGS line searches can fail when the Hessian is very ill-conditioned, so the implementation switches temporarily to stochastic gradient descent with a learning rate of about 10^-9.The fallback runs for 1000 iterations before returning to BFGS and repeats until convergence or the iteration limit.
  • Network architectures: The one-dimensional examples use ANNs with two hidden layers containing 10 neurons each, while the two-dimensional examples use five hidden layers with 10 neurons each.This reflects the paper's stated increase in network depth for the more complicated two-dimensional cases.
  • Linear advection in 1D: The advection example computes an approximate solution and error using 100 equidistant collocation points and a two-hidden-layer ANN with 10 neurons per layer.The reported construction uses the modified feedforward and backpropagation algorithms to evaluate the required terms.
  • Linear diffusion in 1D: The diffusion example likewise uses 100 equidistant collocation points and an ANN with two hidden layers of 10 neurons each.The solution is presented together with its error in Figure 4.

4.3 A remark on 1D problems

In one-dimensional problems, boundary data can sometimes be incorporated through post-processing, allowing solutions for different boundary data without retraining. The 2D advection example extends the method to star-shaped domains using collocation points, smoothed distance functions, and deep neural networks.

  • A remark on 1D problems: 1D first- and second-order problems can incorporate boundary data in a pure post-processing step.First-order problems use one boundary condition, while second-order problems use data at both boundary points.
  • A remark on 1D problems: Solutions for different boundary data can be evaluated without retraining the neural network.This is useful when experimenting with different boundary data for complicated problems or large networks.
  • Linear advection in 2D: The 2D advection experiment uses a star-shaped domain with uniformly distributed interior and boundary points.After training, the neural networks can be evaluated at any number of points.
  • Linear advection in 2D: The smoothed distance function is computed by evaluating a neural-network approximation trained on a coarser grid over a finer grid.Boundary points that do not satisfy the inflow condition are moved to the collocation set before this computation.
  • Linear advection in 2D: The advection solution uses five hidden layers with 10 neurons each, while a single-hidden-layer network with 20 neurons computes the boundary-data extension.The reported errors follow streamlines, a behavior also recognized as a problem in FEM.

4.5 Linear diffusion in 2D

The paper applies its neural-network PDE method to a linear diffusion equation in a two-dimensional domain. Boundary data are imposed through a smoothed distance function and a neural-network extension, and the resulting solution and error are shown for a deep network.

  • Linear diffusion in 2D: The 2D experiment studies a linear, scalar diffusion equation with boundary conditions imposed on the entire boundary.The smoothed distance function therefore uses all boundary points.
  • Linear diffusion in 2D: An analytic solution is selected to compute the forcing and boundary data for comparison.The chosen solution is used to construct f and g in the diffusion equation.
  • Linear diffusion in 2D: A single-hidden-layer network with 20 neurons computes the boundary-data extension from all fine-mesh boundary points.The diffusion operator is then applied to the solution ansatz, with residual gradients computed using the paper’s backpropagation algorithms.
  • Linear diffusion in 2D: The reported diffusion solution uses five hidden layers with 10 neurons each.The resulting solution is presented with its error in Figure 9.

4.6 Linear diffusion in a complex 2D geometry

The method is tested on Sweden’s highly irregular polygonal domain, where mesh generation is difficult. The neural-network simulation produces a solution and error for the complex geometry, while higher-dimensional extensions use collocation sequences to address sampling challenges.

  • Linear diffusion in a complex 2D geometry: Sweden is represented as a polygon with 160876 vertices and a highly irregular boundary containing very small details.The boundary data use 500 uniformly distributed collocation points along the boundary.
  • A remark on mesh based methods: FEniCS mesh generation did not complete within 16 hours across different settings and resolutions, whereas the ANN simulation took about 10 minutes.The reported ANN time includes computing the smoothed distance function and boundary-data extension.
  • Higher-dimensional problems: For sufficiently deep networks, the parameter increase associated with adding one first-layer neuron per spatial dimension is negligible.The computational complexity of the diffusion operator increases linearly with the number of space dimensions.
  • Higher-dimensional problems: The number of collocation points is the main cause of increased computational time in higher dimensions.Regular grids grow exponentially, while uniformly random points can cluster on hyperplanes and provide poor coverage.
  • Higher-dimensional problems: The paper uses Sobol low-discrepancy sequences for high-dimensional collocation points and lower-dimensional boundary points.The remaining procedure follows the same method described for lower-dimensional problems.

5 Convergence considerations

The paper examines boundary-data pre-training and network depth as ways to reduce iterations and training cost. Pre-training helps especially for less efficient optimizers and ill-conditioned diffusion problems, while deeper networks can converge faster until vanishing gradients appear.

  • 5.1 Boundary data pre-training: Boundary-data pre-training uses the available boundary fit to provide a good starting point for solution-network optimization.The boundary ANN is trained only on boundary points, while the solution ANN is trained on boundary and collocation points.
  • 5.1 Boundary data pre-training: Pre-training has limited effect with BFGS but can substantially improve less efficient methods such as gradient descent.For diffusion, pre-training greatly reduces the time until BFGS’s first line-search failure, a major bottleneck of the method.
  • 5.2 Number of hidden layers: With comparable network capacity, increasing hidden layers produces a dramatic decrease in iterations needed to reach a desired accuracy.The study compares networks with one through five hidden layers and approximately 477–517 trainable parameters.
  • 5.2 Number of hidden layers: Single-hidden-layer training failed to reach the required tolerance before the maximum iteration limit, whereas deeper networks eventually showed deteriorating convergence.At five hidden layers, the authors began to observe the vanishing-gradient problem.

6 Summary and conclusion

The paper presents deep feedforward ANNs for advection and diffusion PDEs in complex geometries, supported by modified backpropagation. Its examples include a highly complex polygon where traditional mesh-based methods are infeasible, and deeper networks reduce training iterations.

  • 6 Summary and conclusion: The method solves advection and diffusion type PDEs in complex geometries using deep feedforward artificial neural networks.The derivations can be extended to nonlinear systems of PDEs of arbitrary order.
  • 6 Summary and conclusion: Examples cover advection and diffusion problems in one and two dimensions, including a highly complex polygon infeasible for traditional mesh-based methods.This example illustrates the intended setting for the proposed approach.
  • 6 Summary and conclusion: Increasing the number of hidden layers reduces the training iterations required to reach a specified accuracy.The paper identifies techniques addressing vanishing gradients and extensions using convolutional layers, dropout, and batch normalization as future work.

A Advection problems

For advection PDEs, the paper modifies feedforward and backpropagation calculations to obtain input derivatives and residual-cost gradients. The resulting algorithms support first-order linear and nonlinear PDEs while reusing standard pass data with additional memory.

  • A Advection problems: Advection operators require gradients of the ANN output with respect to the input variables.These gradients are obtained by taking partial derivatives of the feedforward computation.
  • A Advection problems: The modified backpropagation computes gradients of the residual cost function with respect to network parameters.The computation uses componentwise backpropagation with the identity as the cost function and then assembles vector and matrix forms.
  • A Advection problems: The feedforward and modified backpropagation algorithms are sufficient for solving any first-order linear or nonlinear PDE.The input derivatives are organized through Jacobian matrices whose rows contain the ANN’s partial derivatives with respect to the inputs.
  • A Advection problems: The derivative computation reuses data from standard feedforward and backpropagation passes, adding memory overhead for activations and weighted inputs.Its computational complexity is stated to be the same as the corresponding standard passes apart from that additional storage.

B Diffusion problems

For diffusion PDEs, the paper extends the derivative-aware algorithms to second-order and higher-order terms. The resulting procedures support linear and nonlinear second-order PDEs and can be repeatedly differentiated for higher orders.

  • B Diffusion problems: Diffusion operators require second-order derivatives that are computed by differentiating the advection-oriented derivative algorithm.The paper defines matrix forms for non-mixed second partial derivatives and assembles the resulting expressions.
  • B Diffusion problems: The residual-cost gradient for diffusion problems requires computation of third-order derivative terms.These terms are obtained by taking another partial derivative of the modified backpropagation expression.
  • B Diffusion problems: The derived algorithms can solve any second-order linear or nonlinear PDE, while mixed derivatives require corresponding modifications.Higher-order PDEs can be handled by repeatedly differentiating the feedforward and backpropagation algorithms.
Loading 1711.06464v2…