Source-linked AI summary
A Fully Dual Domain Decomposition Method for Extreme Learning Machines Using GenEO Spectral Coarse Spaces
Chang-Ock Lee, Byungeun Ryoo
TL;DR
Large least-squares solves limit the use of ELMs on large PDE problems, while existing ELM domain decomposition approaches have had difficulty scaling. FD3 treats interface continuity and flux conditions equally and uses an abstract Schwarz framework to construct an additive Schwarz preconditioner and GenEO coarse space. Experiments confirm condition-number bounds and weak scalability with GenEO, and report stabler iteration counts on non-elliptic problems than DDELM.
Problem
The cost of least-squares algorithms constrains ELM use on large systems, and existing ELM domain decomposition methods have struggled with scalability.
Source: §1 · pp. 1–2Method
FD3 is a nonoverlapping ELM domain decomposition method that enforces continuity and flux conditions equally, with an abstract Schwarz additive preconditioner and GenEO coarse space.
Source: §1 · p. 2Results
Experiments confirm condition-number bounds and weak scalability with GenEO coarse spaces, while FD3 has stabler iteration counts than DDELM on non-elliptic problems.
Source: §1 · pp. 2–3Takeaways & Limitations
FD3's interface reduced system avoids singular-value inversion and can provide more accuracy than other ELM domain decomposition methods.
Source: §1 · p. 2The abstract Schwarz analysis assumes an SPD system and a geometric subdomain decomposition satisfying condition (C3).
Source: §2.2 · p. 5, §3.1 · p. 10
Abstract
from arXiv · showhide
Extreme learning machines (ELMs) are a class of machine learning algorithms that randomly intialize and subsequently fix the hidden layer parameters, and solve for the last layer coefficients using a least squares method. ELMs have been successfully applied to the solution of various partial differential equations, but the cost of solving large least squares problems limits their use in large scale problems or when high accuracy is desired. To address this limitation, domain decomposition methods (DDMs) have been developed for ELMs, reducing training times via parallel computation. Yet these methods, even those incorporating a coarse space, have struggled with scalability. This paper introduces a novel nonoverlapping DDM for ELMs that treats and enforces continuity and flux conditions equally. Additive Schwarz preconditioners and GenEO (Generalized Eigenvalues problems in the Overlaps) coarse spaces are constructed to accelerate the method. Numerical experiments show the new method outperforms previous DDMs for ELMs in both iteration count and accuracy, especially for the Stokes and wave equations. Furthermore, weak scalability has been observed in up to 4,096 subdomains when employing GenEO coarse spaces.
1 Introduction
FD3 is a nonoverlapping domain decomposition method for ELMs that treats interface continuity and flux conditions equally. The paper analyzes it in an abstract Schwarz framework and reports weak scalability with GenEO coarse spaces.
1 Introduction
- ELMs fix randomly initialized hidden-layer parameters and solve for the last-layer coefficients by least squares, whose cost limits large-system applications.ELMs have been applied to PDEs and are typically faster and more accurate than PINNs, but least-squares complexity remains a constraint.Source: §1 · pp. 1–2
- Earlier localized ELM DDMs required one large least-squares solve across subdomain problems, whereas DDELM introduced interface variables and enabled parallel computation.FD3 extends the nonoverlapping substructuring approach by enforcing Dirichlet and Neumann interface conditions equally.Source: §1 · p. 2
- FD3 has stabler iteration counts than DDELM on non-elliptic problems and can provide more accuracy than other ELM DDMs.The paper attributes the accuracy advantage to an interface reduced system that is free of singular-value inversion.Source: §1 · pp. 2–3
- Experiments confirm the abstract Schwarz condition-number bounds and weak scalability when GenEO coarse spaces are used.The authors also report this as the first weak-scalability result in scientific machine learning, to the best of their knowledge.Source: §1 · pp. 2–3
- The paper also discusses reweighting Physics Informed Loss terms and handling small singular values in local operators to improve FD3 performance.It provides a strategy for assigning loss weights in a weak-scaling scenario and considers heuristic handling of small singular values.Source: §1 · p. 3
2 Preliminaries
The preliminaries frame PDE domain decomposition and GenEO coarse spaces, then establish how ELMs discretize PDEs and how DDELM reduces their computation to an interface system. They also describe the initialization approach and note that DDELM has not shown weak scalability.
2 Preliminaries
- The PDE setup assumes a bounded domain and a well-posed problem with a unique solution.The notation defines L as a differential operator and B as the boundary-condition operator.Source: §2 · p. 3
2.1 Nonoverlapping domain decomposition methods
- Nonoverlapping domain decomposition recasts a PDE as coupled subdomain problems with continuity and flux conditions; FETI enforces flux, while Neumann-Neumann methods share continuity.Coarse spaces address problematic modes, with spectral spaces used for more difficult problems.Source: §2.1 · pp. 3–4
2.2 GenEO coarse spaces
- The abstract Schwarz framework defines a one-level additive Schwarz preconditioner and a GenEO coarse space selected through local generalized eigenvalue problems.Under assumptions (C1)–(C4), the theorem bounds the nonzero eigenvalues of HAP below by 1/τ when the coarse space contains the specified eigenvectors.Source: §2.2 · pp. 5–6
2.3 ELMs for solving PDEs
- ELMs discretize PDEs by collocation and solve for network coefficients in the least-squares sense; with M ≤ n and randomly sampled weights and biases from a continuous distribution, the matrix has full column rank with probability 1.The paper notes that full rank is relevant to least-squares methods and to ensuring later matrices are SPD.Source: §2.3 · p. 6
2.3.1 Initialization method
- The initialization scheme uses weights and geometry-informed biases to produce basis functions whose linear combinations approximate PDE solutions without excessive redundancy.The initialization is qualitatively unchanged under translated and uniformly scaled domains.Source: §2.3.1 · p. 7
2.4 DDELM
- DDELM reduces the problem to an interface variable and enables parallel computation, but its coarse space and Neumann-Neumann acceleration have not yielded weak scalability.The paper identifies DDELM as a nonoverlapping method inspired by substructuring methods [22].Source: §2.4 · p. 7
3 Fully dual domain decomposition (FD3) methods for ELMs
FD3 enforces continuity and flux conditions through auxiliary variables, reducing the coupled subdomain problem to an interface system. The section then develops an abstract Schwarz preconditioner and GenEO coarse space, and proposes domain-scale-based loss weighting.
3 Fully dual domain decomposition (FD3) methods for ELMs
- FD3 replaces both continuity and flux conditions with equivalent conditions involving auxiliary variables, unlike FETI and DDELM, which use a dual variable for only one condition.After solving for the interface variables, the local network coefficients can be obtained in parallel.Source: §3 · pp. 7–9
- FD3 enforces continuity and flux along each edge or face to which an interface point belongs, rather than enforcing conditions between subdomains that share only a corner.Figure 1 contrasts this edge-wise choice with fully redundant enforcement.Source: §3 · p. 8, Figure 1 · p. 9
3.1 Abstract Schwarz formulation and GenEO coarse space for FD3
- The abstract Schwarz formulation yields a one-level preconditioner and a GenEO coarse space built from local generalized eigenvalue problems with threshold τ.The stated condition-number bound is χτ.Source: §3.1 · p. 10
3.2 Weighting loss terms
- Because convergence and accuracy can depend strongly on relative loss weights, the paper proposes estimating weights on small problems and adapting them to larger domains without further experimentation.The weighting strategy is based on how loss-term magnitudes change when the domain is geometrically scaled.Source: §3.2 · p. 10
- When a loss term represents multiple operators, the paper says to consider and weight those operators separately.For the Stokes equations, L represents two operators.Source: §3.2 · p. 11
4 Numerical experiments
The experiments test FD3 variants against DDELM across Poisson, Stokes, and wave equations, tracking accuracy, iterations, condition numbers, and coarse-space sizes. Increasing problem complexity with subdomain count probes weak scaling relevant to large, high-frequency problems.
4.1 Singular value thresholding
- With four collocation points per hidden-layer neuron, FD3 remains stable without singular-value thresholding, trading a small convergence cost for improved accuracy.All FD3 experiments therefore use a threshold of zero; DDELM uses an empirically selected threshold to maximize its accuracy for comparison.Source: §4.1 · p. 12
4.2 A red-black strategy for GenEO coarse spaces
- FD3-RB uses only red-subdomain generalized eigenvectors, halving the coarse-space size with little observed performance loss, although the strategy has no supporting theory.The full FD3 GenEO coarse space otherwise grows large, even at large eigenvalue thresholds.Source: §4.2 · p. 12
4.3 Poisson equation
- For Poisson, FD3-RB runs at 64 × 64 subdomains with bounded condition number and iteration count, while FD3 cannot run there because of coarse-space memory demands.FD3 condition numbers do not exceed 16 despite a theoretical bound of 2e+12; FD3-RB has a very small condition-number penalty while using half the FD3 coarse-space size.Source: §4.3 · p. 13, Table 1 · p. 14
4.4 Stokes equation
- For Stokes, adding the full mean-pressure space is important for weak scalability; FD3-RB maintains bounded iterations, though pressure errors reach 10^-3 near subdomain boundaries.The pressure error is large in absolute value at interfaces, while the velocity errors are smooth and relative L2 errors are largely consistent.Source: §4.4 · p. 16, Table 2 · p. 17
4.5 Wave equation
- For wave, FD3-RB condition numbers stay within the theoretical bound of 200 and iteration counts rise only slightly, but errors become larger near t = 1 as subdomain count and propagation speed increase.Without GenEO, FD3 also struggles to converge quickly as the number of subdomains grows.Source: §4.5 · pp. 19–20, Table 3 · p. 19
5 Conclusions
FD3 is a nonoverlapping DDM for ELMs that enforces continuity and flux conditions equally. The paper reports weak scalability with GenEO coarse spaces and strong performance on non-elliptic problems.
5 Conclusions
- FD3 enforces interface continuity and flux conditions equally by introducing dual variables for both.The method is a nonoverlapping domain decomposition method for ELMs.Source: §5 · p. 20
- An abstract Schwarz analysis yields a one-level additive Schwarz preconditioner and a GenEO coarse space, with experiments validating the theory and weak scalability.Source: §5 · p. 20
- Compared with DDELM, FD3 works well on non-elliptic problems including the Stokes and wave equations.The paper identifies reducing the large GenEO coarse spaces as a direction for future research.Source: §5 · p. 20
A Counterexample
The appendix presents a counterexample to a proposed full-column-rank claim, using a smooth activation function whose derivatives are linearly independent.
A Counterexample
- Proposition A.1 is marked false despite claiming full column rank with probability 1 under its stated conditions.Source: A Counterexample · p. 24
- The counterexample uses the activation function e^-1/x for x > 0 and 0 for x ≤ 0, which is C∞ and has linearly independent derivatives.Source: A Counterexample · p. 24
- For a given x_i, the event that w_j^T x_i + b_j ≤ 0 for every j has nonzero probability.Source: A Counterexample · p. 24
B Proof of Theorem 2.2
The proof establishes Theorem 2.2 by reducing the random full-rank claim to the existence of weights and biases yielding a nonzero determinant. It constructs that point through linear independence of functions under the theorem’s activation and operator assumptions.
B Proof of Theorem 2.2
- Analytic dependence makes the determinant either identically zero or zero only on a null set, so the proof seeks weights and biases for which it is nonzero.Source: B Proof of Theorem 2.2 · p. 25
- Linear independence of the functions ψ_i implies that there are weights and biases making the matrix L_n nonsingular; the proof establishes this implication by induction.Source: B Proof of Theorem 2.2 · p. 25
- Under its assumptions, Theorem 2.2 claims full column rank with probability 1 for M ≤ n distinct points and iid weights and biases drawn from an absolutely continuous distribution.Source: B Proof of Theorem 2.2 · p. 26
- To establish function independence, the proof uses Taylor expansions and selects v with distinct nonzero |v^T x_i| and nonzero operator-weighted coefficients, contradicting a dependence assumption.The argument uses that σ is non-polynomial.Source: B Proof of Theorem 2.2 · pp. 27–28