Source-linked AI summary
A simple shallow neural network for emulating the solution to singularly perturbed problems
Christos Xenophontos, Aayushman Raina
TL;DR
Singularly perturbed boundary value problems contain smooth and rapidly varying layers that challenge standard neural approximations. The paper proposes an exponentially enriched shallow tanh network based on known asymptotic structure and reports robust emulation over the tested perturbation ranges, while limiting its claims to those experiments and architectures.
Problem
Standard neural networks must learn both smooth solution components and layers whose widths vary substantially with small perturbation parameters, making robust approximation challenging.
Method
The paper augments a shallow tanh network with explicit exponential features representing known boundary-layer structure and couples the enriched space with residual or energy minimization.
Results
The numerical experiments report nearly parameter-independent L∞, L2, and energy errors over the tested ranges, including one- and two-dimensional examples.
Takeaways & Limitations
When dominant layer locations and scales are reliably known, incorporating them into the approximation space can support accurate emulation across a wide range of tested perturbation parameters.
Takeaways & Limitations
The results provide evidence over tested parameter ranges and do not establish accuracy for all perturbation parameters or optimization algorithms.
Abstract
from arXiv · showhide
We consider (feed-forward) Neural Networks (NNs) for the emulation of the solution to singularly perturbed second order boundary value problems, with two small parameters. We describe a shallow NN which exploits available asymptotic expansions for the solution. These additive decompositions into smooth and layer components, allow for derivative estimates which are explicit in the order of differentiation as well as the singular perturbation parameter(s) \cite{melenk, Irene, SX}. Utilizing such decompositions, we propose a simple NN for emulating the solution to such problems using the $\tanh$ activation function together with different training objectives, such as residual or energy minimization. The key idea is to augment the approximation space with suitable exponential functions, similar to enriched spaces in finite element methods, e.g.~\cite{Kellogg}. Numerical examples in one and two dimensions, including a smooth non-tensor-product domain, illustrate the resulting parameter-robust behavior over the tested perturbation ranges.
1 Introduction
Singular perturbations create solutions with smooth and rapidly varying layer components, making standard neural approximation difficult in small-parameter regimes. The paper addresses this by enriching a shallow tanh network with explicit layer features guided by asymptotic structure.
- Motivation: Singularly perturbed solutions decompose into a regular component and one or more layers whose derivatives can grow as the perturbation parameter decreases.The regular component has parameter-uniform derivative estimates, whereas the k-th layer derivative typically grows like O(ε^-k).
- Motivation: Layer-adapted finite-element meshes resolve these components and can achieve parameter-uniform errors with exponential convergence as polynomial degree increases.The SBL mesh uses a fixed number of elements designed around the asymptotic solution components.
- Motivation: Neural networks face difficulty because boundary and interior layer widths may vary by several orders of magnitude as perturbation parameters change.Standard architectures must learn both smooth and rapidly varying components simultaneously, which can reduce robustness for very small parameters.
- Proposed approach: The proposed Boundary Layer Neural Network incorporates known layer behavior through exponential enrichments instead of relying on optimization to discover the layers.The construction is an enriched approximation space analogous in spirit to enriched finite-element spaces.
- Proposed approach: The one-dimensional method is an exponentially enriched shallow tanh network that can use residual minimization or energy minimization as training objectives.The architecture contains one hidden tanh layer augmented by two explicit exponential layer features.
- Results: Numerical examples in one and two dimensions report robust and accurate emulations across a wide range of tested singular perturbation regimes.The paper presents the architecture and numerical demonstrations as evidence for incorporating reliable asymptotic information directly into neural-network design.
2 Second order SPPs and the regularity of their solution
The paper studies a second-order boundary value problem with two perturbation parameters and analyzes how its solution changes across parameter regimes. Under the focused regime, the solution separates into smooth, localized exponential layers and a negligible remainder.
- Model problem: The model problem is a second-order boundary value problem with diffusion and convection scales ε1, ε2 ∈ (0,1] and analytic coefficient and forcing functions.The differential equation is posed on I=(0,1), with b(x), c(x), and f(x) assumed analytic.
- Layer structure: The solution structure is determined by characteristic-equation roots, whose associated parameters control the strength of the endpoint boundary layers.Because |λ0(x)|<|λ1(x)|, the layer at x=1 is stronger than the layer at x=0.
- Regularity and decomposition: In the focused regime, the solution admits an additive decomposition into a smooth part, endpoint boundary layers, and a remainder.The regular component is analytic for analytic data, while the boundary layers are exponentially localized and have different widths.
- Regularity and decomposition: The regularity estimates use constants independent of ε1 and ε2 and quantify derivative behavior across the decomposed solution components.The paper states estimates for all derivative orders n∈N0, with parameter-independent constants.
3 Neural Networks
The paper defines neural networks through affine transformations, biases, and componentwise nonlinear activations, distinguishing the network representation from the function it realizes. It motivates shallow tanh networks because they can approximate smooth functions while exponential layer profiles can be represented efficiently.
- Neural-network formalism: A neural network is specified by a finite sequence of weight matrices, bias vectors, and a nonlinear activation function.The input and output dimensions are determined by the first and last network layers.
- Neural-network formalism: The realization of a network is the function obtained by composing parameter-dependent affine transformations with nonlinear activations.The activation is applied componentwise in hidden layers, while the final layer has no activation.
- Neural-network formalism: A shallow network uses a fixed number of layers, whereas a deep network increases its number of layers to achieve the desired accuracy.The paper treats depth as the number of layers in the network architecture.
- Tanh activation: The paper uses the tanh activation function and notes that it can emulate e^-x within a prescribed tolerance.This provides a basis for representing exponential profiles associated with boundary layers.
- Tanh activation: Tanh networks can approximate smooth functions with a fixed number of layers, making them suitable for the analytic solution component.The proposed computational architecture nevertheless inserts explicit exponential layer features rather than relying only on neural representation.
4 Emulation of the solution to SPPs
The proposed BL-NN uses asymptotic decompositions to enrich a shallow tanh approximation with explicit exponential layer features. The same framework supports strong or weak boundary enforcement and residual- or energy-based training, while separating approximation-space design from sampling and optimization.
- The expressivity construction decomposes the solution into separate neural approximations for smooth and layer components, leaving the exponentially small remainder untreated.
- The practical BL-NN augments a one-hidden-layer tanh network with two trainable exponential features whose widths represent boundary-layer scales.The layer locations and widths are motivated by the asymptotic expansion, while the exponential decay rates are trainable to accommodate imperfectly known scales.
- The essential approximation-theoretic construction is the enriched space V_NN = span{tanh(w_jx + b_j)} + span{E_i}, rather than a particular loss or optimizer.This formulation distinguishes the approximation space from sampling strategy and optimization algorithm.
- Boundary conditions may be imposed strongly through parametrization or weakly with Nitsche’s method, with the latter producing almost identical numerical results.
- Residual minimization uses collocation points, while energy minimization uses an integrating-factor functional and layer-adapted Shishkin meshes.The Ritz formulation is most attractive for self-adjoint reaction-diffusion problems because integrating factors can become poorly conditioned in strongly convection-dominated regimes.
5 Numerical results
The numerical experiments evaluate BL-NN emulation in one and two dimensions using exact or FEM references, residual or energy training, and decreasing perturbation parameters. Across the tested cases, the enriched architecture shows parameter-robust errors, while thinner layers increase unweighted H1 error.
- Experimental setup: The experiments compare emulations with exact solutions when available and with FEM approximations otherwise.The study covers second-order singularly perturbed problems in one and two dimensions.
- Example 1: O(10^-3) accuracy is reported across all tested parameter combinations in the two-parameter Example 1.The longest run for the displayed Example 1 case took about 1 minute.
- Example 1: Using both exponential enrichments yields the best results for ε1 = 10^-5 and ε2 = 10^-2, compared with no or partial enrichment.The no-enrichment case corresponds to a standard one-layer PINN, and other parameter values showed similar behavior.
- Example 2: For Example 2, the pointwise absolute error relative to hp-FEM is O(10^-3) at ε = 10^-8.The comparison uses an hp-FEM approximation on a Spectral Boundary Layer mesh; computational efficiency is not claimed.
- Two-dimensional results: The maximum-norm error remains below 10^-3 over the tested two-dimensional parameter range, while L2 and energy errors remain nearly parameter-independent.The unweighted H1 error grows as perturbation parameters decrease because derivatives increase in thinner layers, whereas the energy norm weights derivatives by diffusion parameters.
- Two-dimensional results: On a smooth domain, errors remain essentially unchanged over 10^-3 ≥ ε ≥ 10^-11, providing empirical evidence of parameter-robust behavior.The two-dimensional extension combines edge- and corner-layer features and uses a boundary projector for strong boundary enforcement.
6 Conclusions
The paper proposes a simple shallow tanh neural network enhanced with exponential features for emulating second-order singularly perturbed problems in one and two dimensions. Numerical experiments support accuracy across wide tested parameter ranges, while the authors limit the conclusion to those ranges and note dependencies on problem-specific layer information.
- The proposed BL-NN combines a shallow tanh network with exponential enrichments for emulating solutions to second-order singularly perturbed problems.The study provides numerical results in one and two dimensions.
- When dominant layer locations and scales are reliably known, incorporating them into the approximation space can preserve accuracy across a wide range of perturbation parameters.
- The design combines shallow tanh features for regular behavior, explicit asymptotic layer features, and residual or energy minimization trained with Adam.
- The experiments provide evidence over tested parameter ranges rather than proof of accuracy for all perturbation parameters or optimization algorithms.
- Applying the principle to other singularly perturbed problems requires deriving appropriate enrichments and boundary treatments for each problem.Future work also includes more challenging plate and fluid-flow problems and separating enrichment, sampling, and optimizer effects.