PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations

Suchuan Dong, Zongwei Li

arXiv:2012.02895v1math.NAcs.LGphysics.comp-phphysics.flu-dyn

TL;DR

Existing neural-network PDE solvers can be inaccurate and computationally uncompetitive, motivating a more efficient approach. This paper combines local ELM networks, domain decomposition, continuity constraints, least-squares training, and block time marching, achieving convergent accuracy and performance on par with or exceeding FEM.

  • Problem

    Existing DNN-based PDE solvers generally exhibit absolute errors around 10^-3–10^-4 and can be computationally uncompetitive for scientific computing.

    Source: §1 · p. 2

  • Method

    The method combines ELM-like local neural networks, domain decomposition with C^k interface continuity, least-squares training, and block time marching for long-time simulations.

    Source: §1 · pp. 3–4

  • Results

    The method shows typically exponential or near-exponential error reduction with increasing degrees of freedom, outperforms commonly used DNN-based solvers, and matches or exceeds FEM performance.

    Source: §1 · p. 3, §4 · p. 55

  • Takeaways & Limitations

    The results indicate that local ELM domain decomposition can make neural-network PDE solvers competitive with established numerical methods in the studied low-dimensional problems.

    Source: §4 · p. 55

    For large temporal dimensions, the basic method becomes harder to train and can produce pronounced errors at later time instants, making block time marching advisable.

    Source: §2.2.2 · p. 13

Abstract

from arXiv · show

We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and $C^k$ continuity is imposed on the sub-domain boundaries. Each local neural network consists of a small number of hidden layers, while its last hidden layer can be wide. The weight/bias coefficients in all hidden layers of the local neural networks are pre-set to random values and are fixed, and only the weight coefficients in the output layers are training parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation type algorithms. We introduce a block time-marching scheme together with the presented method for long-time dynamic simulations. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of degrees of freedom increases. Extensive numerical experiments have been performed to demonstrate the computational performance of the presented method. We compare the current method with the deep Galerkin method (DGM) and the physics-informed neural network (PINN) in terms of the accuracy and computational cost. The current method exhibits a clear superiority, with its numerical errors and network training time considerably smaller (typically by orders of magnitude) than those of DGM and PINN. We also compare the current method with the classical finite element method (FEM). The computational performance of the current method is on par with, and oftentimes exceeds, the FEM performance.

1 Introduction

The paper introduces locELM, an ELM-like solver combining domain decomposition, local neural networks, and least-squares training for linear and nonlinear PDEs. It targets the accuracy and computational-cost limitations of DNN-based PDE solvers through convergent, efficient computation and block time marching for long-time simulations.

1 Introduction

  • The paper motivates locELM by noting that current DNN-based PDE solvers commonly show errors around 10^-3–10^-4, weak convergence, and training costs far exceeding classical methods.DNN solvers typically rely on gradient-descent training of residual-based losses, whereas locELM uses least-squares computation.Source: §1 · pp. 2–3
  • locELM typically achieves exponentially or nearly exponentially decreasing errors, outperforms DGM and PINN by orders of magnitude, and matches or exceeds FEM in accuracy and computational cost.At equal accuracy, locELM training time is comparable to or smaller than FEM computation time; at equal cost, its errors are comparable to or markedly smaller than FEM.Source: §1 · pp. 3–5
  • The method represents each sub-domain solution with a shallow local feed-forward network, fixes hidden-layer weights and biases randomly, and trains only output weights by linear or nonlinear least squares.The last hidden layer may be wide, and local networks are coupled through enforced C^k continuity across sub-domain boundaries.Source: §1 · p. 3
  • Block time marching divides long temporal domains into successive time windows, making training easier and enabling accurate long-time solutions, especially for nonlinear time-dependent PDEs.Solving the entire temporal domain at once can produce pronounced errors at later times, whereas moderate block sizes improve trainability.Source: §1 · pp. 3–4
  • For nonlinear PDEs, locELM uses NLSQ-perturb or Newton-LLSQ; random perturbations prevent local-minimum trapping, especially in under-resolved cases and long-time simulations.This extends existing ELM-based differential-equation methods beyond linear problems and single-hidden-layer networks.Source: §1 · pp. 3–4
  • Unlike single-hidden-layer ELM implementations, locELM supports local networks with up to three hidden layers while retaining accurate simulation results with one or a few hidden layers.The method is named locELM because it combines ELM, domain decomposition, and local neural networks.Source: §1 · p. 4

2 Domain Decomposition and Local Extreme Learning Machines

This section develops the locELM domain-decomposition framework for steady and time-dependent linear and nonlinear PDEs, including local network coupling, collocation-based formulations, least-squares solution procedures, and block time-marching. It also specifies threshold-based stopping criteria for the nonlinear differential-equation procedure.

2.1 Local Extreme Learning Machines (locELM) for Representing Functions

  • LocELM represents functions locally on sub-domains using shallow feed-forward networks whose outputs satisfy user-defined C^k continuity across shared boundaries.The continuity order can reflect the differential equation, typically using C^{m−1} where the highest derivative in a coordinate direction is m, coupling otherwise separate local networks.Source: §2.1 · p. 5
  • Hidden-layer weights and biases are randomly preset and fixed, while a potentially wide last hidden layer feeds a linear output layer whose weights are fitted by least squares.The adjustable parameters are the output-layer weights, with M training parameters per local network; back-propagation is not used.Source: §2.1 · pp. 5–6

2.2.1 Time-Independent Linear Differential Equations

  • For time-independent linear Dirichlet PDEs, collocation equations, boundary conditions, and C^1 interface constraints form a linear algebraic system solved by minimum-norm linear least squares.For second-order equations in x and y, the system has NxNy(QxQy + 2Qx + 2Qy) equations and NxNyM unknown output-layer weights; quadrature points are generally more accurate than uniform points, which outperform random points at equal count.Source: §2.2.1 · pp. 7–9
  • LocELM is typically orders of magnitude more accurate and faster to train than gradient-trained DNN PDE solvers, while matching or exceeding classical FEM performance.Its training cost is essentially the linear least-squares computation; at equal time, accuracy is comparable or superior to FEM, and equal accuracy often requires less time.Source: §2.2.1 · p. 10

2.2.2 Time-Dependent Linear Differential Equations

  • The method is next extended to time-dependent linear initial-boundary value problems in a two-dimensional spatial domain plus time, assuming second spatial order and Dirichlet conditions.This subsection introduces the corresponding basic method for a generic time-dependent second-order linear PDE with initial and boundary conditions.Source: §2.2.2 · p. 10
  • For time-dependent linear PDEs, local networks use x, y, and t inputs with fixed hidden-layer coefficients, while output weights are determined by a linear least-squares system.The formulation enforces PDE residuals, spatial boundary and initial conditions, C1 continuity in space, and C0 continuity across temporal interfaces.Source: §2.2.2 · pp. 11–12
  • Block time-marching divides long temporal domains into moderate-size windows, successively solving each with the previous block’s final solution as the next initial condition.The basic method becomes less accurate at later times when the temporal dimension is large, motivating this strategy.Source: §2.2.2 · p. 13

2.3 Nonlinear Differential Equations

  • For nonlinear differential equations, domain decomposition enforces residual, Dirichlet boundary, and C1 interface conditions, producing nonlinear algebraic equations in the local output weights.The highest-order spatial terms are assumed linear, while nonlinearities may involve the field and lower-order derivatives; auto-differentiation evaluates required derivatives.Source: §2.3 · p. 13, §2.3.1 · p. 14

2.3.1 Time-Independent Nonlinear Differential Equations

  • The nonlinear training system is solved either as a nonlinear least-squares problem or by Newton’s method combined with linear least-squares computations.Both approaches determine the output-layer training parameters while keeping the local hidden-layer basis functions known and differentiable.Source: §2.3.1 · p. 14
  • The procedure stops or triggers a condition when the returned cost falls below a specified threshold or is less than c.The supplied implementation excerpts show both forms of the cost-based criterion.Source: §2.3.1 · p. 15

21 end

The section develops robust nonlinear solvers for locELM training and extends the method to time-dependent nonlinear PDEs through local space-time networks and block time marching. It identifies resolution-dependent convergence issues and reports locELM advantages over DGM, PINN, and FEM in accuracy and computational performance.

21 end

  • NLSQ-perturb addresses local-minimum failures in nonlinear least squares by retrying from randomly perturbed initial guesses when the converged residual exceeds a threshold.For Section 3 tests, the threshold is 10^-3, and the final converged cost is typically on the order of 10^-13 after around 4 or 5 sub-iterations.Source: §21 · pp. 15–16
  • Newton-LLSQ combines Newton iterations with linear least squares, costing considerably less than NLSQ-perturb but generally producing markedly less accurate locELM solutions.Its convergence behavior is less regular, although it appears less likely to become trapped in local minima.Source: §21 · p. 16

2.3.2 Time-Dependent Nonlinear Differential Equations

  • For time-dependent nonlinear PDEs, local space-time networks enforce the PDE, boundary and initial conditions, C1 spatial continuity, and C0 temporal continuity through a nonlinear algebraic system.The system has NxNyNt[QxQyQt + 2(Qx + Qy)Qt + QxQy] equations and NxNyNtM unknowns, solved using NLSQ-perturb or Newton-LLSQ.Source: §2.3.2 · pp. 17–18
  • Block time marching solves long-time simulations successively on moderate time blocks, carrying each block’s endpoint solution forward as the next initial condition.Reducing the block size Γ generally improves nonlinear-iteration convergence, whereas an excessively large Γ can make convergence difficult.Source: §2.3.2 · p. 18
  • locELM is considerably superior to DGM and PINN in accuracy and computational cost, while matching or often exceeding FEM performance.These comparisons are reported for the nonlinear PDE experiments described in Section 3.Source: §2.3.2 · p. 18

3 Numerical Examples

Numerical examples assess locELM on stationary and time-dependent linear and nonlinear PDEs, including diffusion, advection, Helmholtz, nonlinear spring, and viscous Burgers problems. They examine accuracy, convergence with increasing degrees of freedom, training or computational cost, and long-time integration using block time marching, with comparisons to PINN, DGM, global ELM, and FEM.

3.1 One-Dimensional Helmholtz Equation

  • locELM errors decrease exponentially or nearly exponentially as the numbers of sub-domains, collocation points per sub-domain, or training parameters per sub-domain increase, with eventual stagnation under fixed parameters.With Q = 50 and M = 50, maximum error fell from around 10^1 for one sub-domain to about 10^-7 for 8 sub-domains, while training time rose from about 0.1 to 0.8 seconds.Source: §3.1 · pp. 20–21
  • Quadrature collocation was most accurate, uniform points intermediate, and random points least accurate; however, quadrature computation became unreliable above 100 points, so uniform points dominated subsequent tests.At equal collocation counts, quadrature errors could be orders of magnitude smaller than random-point errors.Source: Figure 3 · p. 21, §3.1 · p. 22
  • Domain decomposition preserved comparable accuracy while dramatically reducing training time because only neighboring sub-domains couple, producing a sparse rather than dense least-squares matrix.This comparison fixed total degrees of freedom at NeQ = 200 and NeM = 400, contrasting one global sub-domain with multiple local sub-domains.Source: §3.1 · p. 24, Figure 7 · p. 24
  • locELM was considerably more accurate than PINN, with errors nearly five orders of magnitude smaller and training taking about a second versus over 500 or 1000 seconds.The comparisons used PINN with Adam and L-BFGS optimizers; locELM was clearly superior in both accuracy and computational cost.Source: Table 1 · p. 24, §3.1 · p. 25
  • Against FEM, locELM achieved comparable wall times with comparable or smaller errors, including substantially smaller errors than FEM at similar 100,000-element wall times.For M = 75, locELM matched the performance of FEM with 50,000 elements; M = 100 and M = 125 had wall times comparable to FEM with 100,000 elements.Source: Table 2 · p. 25, §3.1 · p. 26

3.2 Advection Equation

  • The advection-equation experiment combines locELM with block time-marching to test long-time simulation over a one-dimensional space-time domain with periodic spatial boundaries.The supplied setup includes a temporal domain and reports a configuration with tf = 10 and 10 time blocks.Source: §3.2 · pp. 26–27, Figure 10 · p. 27
  • Advection errors decrease approximately exponentially as sub-domains, collocation points, or training parameters increase, then stagnate when the complementary fixed resolution limits further improvement.The convergence trend is reported for maximum and rms errors over the full spatial-temporal domain.Source: §3.2 · p. 28, Figure 11 · p. 28
  • The advection solution remains accurate over tf = 100, spanning approximately 40 wave-propagation periods, with 100 time blocks and about 892 seconds of total network training.The computed solution overlaps the exact midpoint history, and the final-time profiles and blockwise errors remain accurate throughout the long-time simulation.Source: §3.2 · p. 29, Figure 13 · p. 29, Figure 14 · p. 30
  • locELM is about an order of magnitude more accurate and nearly two orders of magnitude faster to train than DGM, with block time marching preserving or improving accuracy while reducing training time.Table 3 compares maximum and rms errors, training iterations, and training time for DGM and locELM with and without block time marching.Source: Table 3 · p. 31, §3.2 · pp. 31–32
  • With fixed total degrees of freedom, four local sub-domains reduce advection training time from around 105 seconds to around 40 seconds while maintaining comparable accuracy to global ELM.The two-sub-domain configuration is reported as more accurate than the other tested configurations.Source: Figure 15 · p. 30, §3.2 · p. 31

3.3 Diffusion Equation

  • For 1D diffusion, locELM captures the solution with absolute errors of 10^-9–10^-8, while increasing sub-domains reduces errors from around 10^-1 to around 10^-8.The tests use block time marching and five sub-domains per time block for the high-accuracy result.Source: §3.3 · p. 33, Figure 17 · p. 33, Figure 18 · p. 34
  • Diffusion errors initially decrease exponentially with collocation points below about 20 and training parameters below about 250, then stagnate or slow as resolution increases.Moderate random-coefficient magnitudes give better accuracy, whereas very small or very large values are less accurate; two-hidden-layer networks also produce accurate results.Source: §3.3 · pp. 34–35, Figure 19 · p. 34
  • In the diffusion equation, locELM achieves comparable or better accuracy than global ELM while reducing training time from 277 seconds with one sub-domain to 79 seconds with two sub-domains.With two or more sub-domains, errors are comparable to or smaller than those with one sub-domain.Source: Figure 21 · p. 36, §3.3 · p. 36
  • For the 1D diffusion equation, locELM errors are five orders of magnitude smaller than DGM errors and training time is over two orders of magnitude shorter.The comparison is supported by maximum and rms errors, training iterations, and training times in Table 4.Source: Table 4 · p. 37, §3.3 · p. 37

3.4.1 Nonlinear Helmholtz Equation

  • For nonlinear Helmholtz, NLSQ-perturb is more accurate than Newton-LLSQ, reaching errors around 10^-11 or 10^-10 as collocation points or training parameters increase, while Newton-LLSQ is generally faster.NLSQ-perturb errors decrease initially exponentially with collocation points, whereas Newton-LLSQ errors stagnate around 10^-6; with more than 200 training parameters, their error levels are around 10^-10 and 10^-5, respectively.Source: Figure 26 · p. 40, §3.4.1 · pp. 40–41, Figure 27 · p. 41
  • LocELM is more accurate and cheaper than PINN for the nonlinear Helmholtz equation, with NLSQ-perturb errors about six orders smaller and training time about 8 versus 806 seconds for PINN/L-BFGS.Newton-LLSQ errors are about two orders smaller than PINN/L-BFGS errors, while its training time is also substantially lower.Source: §3.4.1 · pp. 43–44, Table 6 · p. 44
  • For nonlinear Helmholtz, locELM generally matches or exceeds FEM, achieving lower cost and errors roughly an order of magnitude smaller than an 800,000-element FEM case.With (Q, M) = (125, 200), locELM costs approximately 10 seconds versus approximately 18 seconds for FEM.Source: §3.4.1 · p. 44, Table 7 · p. 44

3.4.2 Nonlinear Spring Equation

  • The nonlinear spring example evaluates locELM together with block time marching for long-time dynamic simulations.The tested temporal domains include tf = 100, 15, and 2.5.Source: Figure 33 · p. 45, §3.4.2 · p. 45
  • For the nonlinear spring equation, locELM captured the long-time solution over tf = 100 with maximum absolute error on the order of 10^-8.The simulation used Nb = 40, Q = 60, M = 100, and Rm = 5.0.Source: §3.4.2 · p. 46
  • For the nonlinear spring equation, errors decreased approximately exponentially with collocation points and training parameters before stagnating, with parameter convergence less regular.Collocation-point errors stagnated beyond about 60 points, while training-parameter errors stagnated beyond about 50 parameters.Source: §3.4.2 · p. 46, Figure 34 · p. 46
  • Against PINN, locELM produced errors six orders of magnitude smaller and training time nearly two orders of magnitude shorter.Both methods captured the solution accurately, but locELM used NLSQ-perturb while PINN used Adam.Source: §3.4.2 · pp. 47–48, Table 8 · p. 47

3.4.3 Viscous Burger’s Equation

  • For viscous Burgers equation, locELM accurately resolved the full spatial-temporal solution, with absolute errors of 10^-8 ∼10^-7 overall and 10^-10 ∼10^-8 at x = 2.75.At t = 8.75, the solution profile overlapped the exact solution and errors were around 10^-10 ∼10^-7.Source: §3.4.3 · p. 49, Figure 37 · p. 49, Figure 38 · p. 49
  • For Burgers equation, errors decreased essentially exponentially with collocation points and substantially with training parameters, while collocation-point errors stagnated beyond Qx = Qt = 15.The tests used local networks on uniform space-time sub-domains with C1 spatial and C0 temporal continuity.Source: §3.4.3 · pp. 48, 50, Figure 39 · p. 50
  • Against DGM, locELM was more accurate: NLSQ-perturb errors were 10^-9 ∼10^-8 versus 10^-3 ∼10^-2 for DGM Adam and 10^-3 for DGM L-BFGS.locELM/Newton-LLSQ errors were 10^-6 ∼10^-5, and locELM/NLSQ-perturb training took around 28 seconds versus around 1800 seconds for DGM/L-BFGS.Source: Figure 42 · p. 51, §3.4.3 · p. 52, Table 9 · p. 52
  • Against FEM, locELM achieved comparable or smaller errors at lower cost, with one (Q, M) = (20 × 20, 200) case nearly three orders more accurate at comparable computational cost.The locELM/Newton-LLSQ cases had errors comparable to FEM cases while costing notably less; FEM showed second-order temporal convergence before saturation.Source: §3.4.3 · pp. 53–54, Table 10 · p. 53, Figure 43 · p. 53

4 Concluding Remarks

The paper develops locELM, a domain-decomposed local extreme learning method for linear and nonlinear PDEs, trained through least-squares output-layer optimization with fixed random hidden-layer coefficients. Experiments show rapid convergence and computational performance that is competitive with or often better than DGM, PINN, global ELM, and FEM.

Method

  • locELM represents each sub-domain with a local feed-forward network, imposes C^k interface continuity, fixes random hidden-layer weights and biases, and trains only output weights.PDE, boundary, initial, and continuity constraints produce linear or nonlinear algebraic systems solved by least-squares methods rather than back-propagation.Source: §4 · p. 54

Long-time simulations

  • Block time marching makes long-time PDE simulations more manageable by solving successive moderate temporal windows, enabled by locELM’s high accuracy and low computational cost.The approach was demonstrated for long-time advection simulations, although it requires retraining on each time block.Source: §4 · p. 54

Numerical results

  • locELM errors typically decrease exponentially or nearly exponentially as sub-domains, collocation points per sub-domain, or training parameters per sub-domain increase.This establishes a clear convergence trend with increasing neural-network degrees of freedom.Source: §4 · p. 54

Parameter selection

  • Accuracy is highest for moderate ranges of the hidden-layer random-coefficient magnitude R_m, which can be estimated through preliminary manufactured-solution experiments.Very large or very small R_m values reduce accuracy, while the favorable range expands as collocation points or training parameters per sub-domain increase.Source: §4 · pp. 54–55

Numerical results

  • With fixed total degrees of freedom, increasing sub-domains generally preserves comparable accuracy while dramatically reducing training time relative to global ELM.The accuracy advantage holds only before sub-domain degrees of freedom become too small; beyond a threshold, accuracy deteriorates.Source: §4 · p. 55
  • locELM is considerably more accurate and faster than DGM and PINN, typically reducing numerical errors and network training time by orders of magnitude.Its performance is also comparable to, and often better than, FEM: at equal cost errors are often smaller, while equal-accuracy training time is often markedly lower.Source: §4 · p. 55

5 Appendix. Additional Numerical Tests

Additional 2D Helmholtz tests show that locELM converges rapidly as its degrees of freedom increase and outperforms PINN while matching or exceeding FEM performance. The method remains accurate with domain decomposition and comparable total degrees of freedom.

5.1 Two-Dimensional Helmholtz Equation

  • The locELM formulation partitions the domain into uniform sub-domains, imposes C1 continuity, and uses local tanh networks with fixed random hidden-layer coefficients and trained output weights.Inputs are normalized to [−1, 1] × [−1, 1], and simulations use a fixed TensorFlow random seed of 1.Source: §5.1 · p. 56
  • Maximum and rms errors decrease essentially exponentially as the number of sub-domains increases, while increasing collocation points or training parameters initially yields near-exponential reductions.Errors stagnate after collocation points reach a certain level, whereas increasing training parameters eventually produces slower reductions, demonstrating convergence with increasing degrees of freedom.Source: Figure 45 · p. 57, §5.1 · p. 57
  • The 4-sub-domain locELM solution is slightly more accurate than the one-sub-domain global ELM solution when both use the same total collocation points and training parameters.The one-subdomain case uses Q = 50 × 50 and M = 1600, while the four-subdomain case uses Q = 25 × 25 per sub-domain and M = 400 per sub-domain.Source: §5.1 · p. 57
  • locELM is two to three orders of magnitude more accurate than PINN, over two orders faster, and outperforms FEM at comparable computational cost.With 4 sub-domains, locELM requires around 33 seconds versus around 410 seconds for global ELM; a 590 × 590 FEM mesh has errors over an order of magnitude larger than locELM with Q = 25 × 25 and M = 400 at comparable cost.Source: Table 11 · p. 58, §5.1 · p. 59, Table 12 · p. 59
  • FEM exhibits second-order convergence, while locELM has comparable accuracy and cost to a 500 × 500 FEM mesh under Q = 20 × 20 and M = 300.The 590 × 590 FEM mesh has computational cost comparable to locELM with Q = 25 × 25 and M = 400, but larger errors.Source: §5.1 · p. 59, Figure 48 · p. 59, Table 12 · p. 59

Related papers

Loading 2012.02895v1…