Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations

Suchuan Dong; Zongwei Li

Source-linked AI summary

Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations

Suchuan Dong, Zongwei Li

arXiv:2012.02895v1Dec 4, 2020math.NAcs.LGphysics.comp-phphysics.flu-dyn

TL;DR

DNN-based PDE solvers face accuracy, convergence, and computational-cost limitations. The paper presents locELM, which combines local neural networks, domain decomposition, fixed randomized hidden layers, least-squares training, and block time marching; it reports strong performance against DGM, PINN, and FEM.

Problem
Existing DNN-based PDE solvers generally show limited accuracy, inconsistent convergence with increasing degrees of freedom, and high training cost compared with classical methods.
Method
locELM combines ELM-style fixed randomized hidden layers, local neural networks on decomposed domains with C^k interface continuity, least-squares training, and block time marching.
Results
locELM errors typically decrease exponentially or nearly exponentially with degrees of freedom, while errors and training times are typically orders of magnitude smaller than DGM and PINN.
Takeaways & Limitations
The reported computational performance of locELM is on par with, and often exceeds, FEM for the paper’s commonly encountered low-dimensional test problems.
Takeaways & Limitations
The basic method is generally not advisable for large temporal-domain dimensions because accuracy tends to deteriorate at later times.

Abstract

from arXiv · show

We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and $C^k$ continuity is imposed on the sub-domain boundaries. Each local neural network consists of a small number of hidden layers, while its last hidden layer can be wide. The weight/bias coefficients in all hidden layers of the local neural networks are pre-set to random values and are fixed, and only the weight coefficients in the output layers are training parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation type algorithms. We introduce a block time-marching scheme together with the presented method for long-time dynamic simulations. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of degrees of freedom increases. Extensive numerical experiments have been performed to demonstrate the computational performance of the presented method. We compare the current method with the deep Galerkin method (DGM) and the physics-informed neural network (PINN) in terms of the accuracy and computational cost. The current method exhibits a clear superiority, with its numerical errors and network training time considerably smaller (typically by orders of magnitude) than those of DGM and PINN. We also compare the current method with the classical finite element method (FEM). The computational performance of the current method is on par with, and oftentimes exceeds, the FEM performance.

1 Introduction

The paper introduces locELM, combining local neural networks, domain decomposition, and ELM-style least-squares training to solve linear and nonlinear PDEs. It targets limitations of DNN-based PDE solvers through fixed randomized hidden layers, coupled local representations, and block time marching.

DNN-based PDE solvers parameterize PDE solutions with deep neural networks and optimize equation, boundary, and initial-condition residuals, typically using gradient descent.
Network architecture and training parameters: Hidden-layer weights and biases are fixed random values, while output-layer weights are the training parameters.The last hidden layer may be wide, despite the network having only a small number of hidden layers.
Training method: Linear PDEs use linear least squares, whereas nonlinear PDEs use nonlinear least squares rather than back-propagation training.
Domain decomposition and local neural networks: locELM partitions the domain into sub-domains, represents each local solution with a shallow network, and enforces C^k continuity across interfaces.The local networks are trained together as a coupled model.
For nonlinear problems, locELM provides NLSQ-perturb and Newton-LLSQ solvers; perturbations help prevent trapping at high-cost local minima in under-resolved and long-time simulations.
Block time marching: Block time marching divides long temporal domains into successive time blocks, making long-time PDE simulations easier to train than solving the entire domain at once.The basic method becomes less accurate at later times when the temporal domain dimension is large.
The method supports up to three local hidden layers and can produce accurate simulation results with one or a small number of hidden layers.
Experiments cover linear and nonlinear, stationary and time-dependent PDEs, including Helmholtz, advection, diffusion, nonlinear spring, and viscous Burgers’ equations.

2 Domain Decomposition and Local Extreme Learning Machines

The method represents functions locally with shallow neural networks over decomposed sub-domains, enforcing C^k continuity and training only output weights by least squares. Fixed random hidden-layer coefficients, local coupling, and block time marching support efficient PDE simulation, including long-time problems.

Local representation: The domain is partitioned into sub-domains, each represented by a shallow feed-forward neural network with C^k continuity enforced across shared boundaries.The continuity order is user-defined and can reflect the differential-equation order in each coordinate direction.
Network architecture and training: Hidden-layer weights and biases are randomly initialized and fixed, while output-layer weights are the adjustable training parameters.The last hidden layer may be wide, allowing the network to retain many fixed features while training only the output coefficients.
Network architecture and training: The coupled local networks are trained through linear or nonlinear least squares rather than back-propagation-type algorithms.When the requirements are imposed on a single global network for a linear problem, the resulting model is termed an extreme learning machine; the local version is called locELM.
Local representation: The local representation uses fixed last-hidden-layer functions whose derivatives can be computed by automatic differentiation for enforcing differential-equation and continuity constraints.These functions are pre-computed by forward evaluation and remain fixed throughout computation.
Collocation and performance: Uniform, quadrature, and random collocation distributions were compared; with equal point counts, quadrature was generally most accurate, followed by uniform and then random sampling.Uniform points were used predominantly because the implementation could not accurately compute Gaussian quadrature points beyond 100 points.
Collocation and performance: The method is reported as more accurate and faster than gradient-descent-trained DNN PDE solvers, while its performance is comparable to or often better than classical FEM.Its training cost is essentially the linear least-squares computation, and its errors typically decrease exponentially or nearly exponentially with increasing degrees of freedom.
Block time marching: For long-time simulations, the temporal domain is divided into moderate-sized blocks solved successively, using the previous block’s final solution as the next initial condition.The basic method becomes difficult to train on large temporal domains, with pronounced errors tending to occur at later times.

21 end

The method solves nonlinear PDE discretizations through nonlinear least squares, with perturbation-based retries or Newton iterations paired with linear least squares. For long-time problems, block time marching controls temporal resolution and supports successive nonlinear solves.

Nonlinear least-squares solvers: NLSQ-perturb retries nonlinear least squares from randomly perturbed initial guesses when the converged residual exceeds a threshold.The perturbation range is controlled by δ; δ = 0.5 is reported as a typical effective value, and numerical tests use a 10^-3 threshold.
Nonlinear least-squares solvers: NLSQ-perturb typically reaches a final converged cost on the order of 10^-13 after only around 4 or 5 triggered sub-iterations.This behavior is reported when the computation uses an appropriate resolution.
Nonlinear least-squares solvers: Newton-LLSQ applies Newton-like iterations while solving the resulting non-square Jacobian system for each increment by linear least squares.The increment is computed with the LAPACK linear least squares routine.
Solver trade-offs: Newton-LLSQ generally costs considerably less than NLSQ-perturb but produces locELM solutions that are markedly less accurate.The comparison is reported specifically for training locELM neural networks.
Block time marching: For long-time nonlinear simulations, block time marching divides the temporal interval into uniform blocks and solves them successively.The solution at the end of one block supplies the initial condition for the next; reducing Γ can improve nonlinear convergence, whereas overly large Γ can hinder it.
Numerical comparisons: For the tested nonlinear PDEs, locELM is considerably superior to DGM and PINN in accuracy and computational cost, while matching or often exceeding FEM performance.The comparison concerns both accuracy and computational performance.

3 Numerical Examples

Numerical experiments show that locELM accuracy improves exponentially or nearly exponentially with increasing degrees of freedom, while domain decomposition reduces training cost. Across PDE tests, locELM is competitive with or superior to PINN, DGM, and FEM in accuracy and computational cost.

Convergence with degrees of freedom: Errors decrease exponentially or nearly exponentially as sub-domains, collocation points, or training parameters increase.This establishes a clear convergence trend with respect to the method’s degrees of freedom.
Collocation strategies: Quadrature collocation points are most accurate, random points least accurate, and uniform points intermediate at equal point counts.Quadrature-point errors can be orders of magnitude smaller than errors from random points, although implementation difficulties arise at large point counts.
Network depth: Increasing collocation points with two or three hidden layers initially decreases errors exponentially before saturation.The multilayer local networks retain accurate behavior beyond the single-hidden-layer configuration.
Domain decomposition: With fixed total degrees of freedom, more sub-domains significantly reduce computation and training time without seriously compromising accuracy.Domain decomposition reduces coupling among sub-domain degrees of freedom and lowers least-squares cost relative to global ELM.
Comparison with PINN: locELM is much more accurate and cheaper than PINN, requiring about a second versus over 500 seconds with Adam and over 1000 seconds with L-BFGS.The comparison uses maximum and rms errors, training time, and training iterations.
Comparison with FEM: locELM performance is competitive with FEM and in some tests surpasses FEM in accuracy and computational cost.The FEM comparison includes maximum and rms errors across several numerical resolutions and corresponding computation times.
Random coefficients: Moderate random hidden-layer coefficient magnitudes generally provide better accuracy than very large or very small values.The suitable range of Rm tends to shift leftward as the number of sub-domains increases.
Time-dependent simulations: Block time marching enables accurate long-time simulations, including advection over a temporal domain of 100, approximately 40 wave-propagation periods.The method also produces considerably more accurate advection results than DGM.

4 Concluding Remarks

The locELM method combines domain decomposition, local neural networks, and ELM training for linear and nonlinear PDEs, with block time marching for long-time simulations. Experiments show convergence with increasing degrees of freedom and performance competitive with or better than DGM, PINN, global ELM, and FEM.

Method: locELM represents PDE solutions with local neural networks on sub-domains and imposes C^k continuity across sub-domain boundaries.The continuity order k is determined by the PDE order.
Method: Block time marching solves successive spatial-temporal time blocks separately, supporting long-time simulations of time-dependent PDEs.The scheme is especially important for nonlinear time-dependent PDEs.
Accuracy and convergence: Errors typically decrease exponentially or nearly exponentially as sub-domains, collocation points per sub-domain, or training parameters per sub-domain increase.This provides a clear convergence trend with respect to neural-network degrees of freedom.
Accuracy and convergence: Moderate random hidden-layer weight and bias magnitudes generally yield higher accuracy, whereas very large or very small Rm values reduce it.The optimal Rm range depends on the problem and tends to expand with increasing local degrees of freedom.
Computational performance: With fixed total degrees of freedom, increasing sub-domains generally preserves comparable accuracy while dramatically reducing training time relative to global ELM.Training time generally increases linearly or super-linearly with the number of sub-domains when other local settings are considered.
Computational performance: locELM is typically orders of magnitude more accurate and faster to train than DGM and PINN, while matching or exceeding FEM performance.Compared with FEM, locELM errors and computation times are comparable or often smaller.
Scope and limitations: With fixed total degrees of freedom, excessive domain subdivision can eventually deteriorate accuracy because each sub-domain receives too few degrees of freedom.The reported comparable-accuracy observation applies only below that subdivision threshold.

5 Appendix. Additional Numerical Tests

Additional 2D Helmholtz tests examine locELM accuracy and cost as discretization and subdivision parameters vary, then compare it with PINN and FEM. The tests report exponential-like locELM convergence, substantially better PINN accuracy and training time, and performance comparable to or better than FEM.

Problem setup: The 2D Helmholtz test uses a rectangular domain with prescribed source and boundary data chosen so an analytic solution is available.The domain parameters are λ = 10, a1 = a2 = 0, and b1 = b2 = 3.6.
locELM setup: locELM partitions the domain into uniform sub-domains, imposes C1 continuity, and uses QxQy collocation points with a local tanh network containing M hidden nodes.The input is normalized to [−1, 1] × [−1, 1], and hidden-layer coefficients are randomized with magnitude controlled by Rm.
Degrees-of-freedom study: Figure 45 varies sub-domains, collocation points per sub-domain, and training parameters per sub-domain to assess maximum and rms error behavior.The collocation-point and training-parameter studies use four sub-domains.
PINN comparison: PINN comparisons use Adam and L-BFGS optimizers on 50×50 uniform points, with 107,000 Adam epochs and 24,500 L-BFGS iterations.The L-BFGS result is generally more accurate than the Adam result.
PINN comparison: locELM is two to three orders of magnitude more accurate than PINN and trains over two orders of magnitude faster on the 2D Helmholtz problem.Figure 47 compares solution and error profiles along horizontal lines at y = 1.0 and y = 2.6.
FEM comparison: FEM exhibits second-order convergence, while a 500×500 FEM mesh and locELM with Q = 20×20, M = 300 have comparable cost and accuracy.At comparable cost, the 590×590 FEM mesh has errors over an order of magnitude larger than locELM with Q = 25×25 and M = 400.