Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains

Ziqi Liu; Wei Cai; Zhi-Qin John Xu

Source-linked AI summary

Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains

Ziqi Liu, Wei Cai, Zhi-Qin John Xu

arXiv:2007.11207v3Jul 22, 2020physics.comp-phcs.LGmath.NA

TL;DR

The paper addresses DNN difficulty with high-frequency PDE solutions and complex or singular domains. It introduces MscaleDNNs using radial Fourier-domain scaling and compactly supported activations. The reported experiments show faster, lower-error elliptic PDE solutions than similarly sized fully connected networks, including complex molecular geometries.

Problem
DNNs learn low-frequency content quickly but struggle with high-frequency PDE solutions, while Poisson-Boltzmann problems involve complex molecular surfaces that challenge mesh-based methods.
Method
MscaleDNNs combine radial scaling in the Fourier domain with compactly supported activation functions to represent multiple frequency scales.
Results
MscaleDNNs solve tested elliptic PDEs faster and with smaller generalization errors than similarly sized fully connected networks across complex and singular domains.
Takeaways & Limitations
MscaleDNNs provide an efficient, easy-to-implement mesh-less approach for elliptic PDEs, including Poisson-Boltzmann problems on complex and singular domains.
Takeaways & Limitations
Direct frequency-scaled approximation requires computationally expensive convolution for scattered data, especially in higher dimensions.

Abstract

from arXiv · show

In this paper, we propose multi-scale deep neural networks (MscaleDNNs) using the idea of radial scaling in frequency domain and activation functions with compact support. The radial scaling converts the problem of approximation of high frequency contents of PDEs' solutions to a problem of learning about lower frequency functions, and the compact support activation functions facilitate the separation of frequency contents of the target function to be approximated by corresponding DNNs. As a result, the MscaleDNNs achieve fast uniform convergence over multiple scales. The proposed MscaleDNNs are shown to be superior to traditional fully connected DNNs and be an effective mesh-less numerical method for Poisson-Boltzmann equations with ample frequency contents over complex and singular domains.

1 Introduction

The paper targets DNN limitations on high-frequency PDE solutions and difficult molecular geometries, proposing MscaleDNNs as mesh-less solvers for complex Poisson-Boltzmann problems.

Motivation: DNNs learn low-frequency content quickly but are inadequate when high-frequency data are involved.This limitation motivates improving convergence for PDE solutions with fine structures.
Motivation: Poisson-Boltzmann equations model electrostatic interactions important to drug design and disease studies, but molecular surfaces are complex and singular.Van der Waals and solvent-accessible surfaces create challenging geometries for numerical methods.
Motivation: Traditional finite element and finite difference methods face costly mesh generation and expensive solution of discretized linear systems for these domains.
Approach: MscaleDNN converts high-frequency approximation into lower-frequency learning by exploiting differences in frequency-dependent learning behavior.The work addresses response frequencies of input-output mappings rather than spatial frequencies within images.
Approach: Radial partitioning of Fourier space is proposed to reduce high-frequency content without the many phase-shifted subnetworks used by PhaseDNN.The passage identifies PhaseDNN's computational cost as substantial even for three-dimensional problems.
Evaluation: Experiments compare two MscaleDNN architectures with similarly sized fully connected networks across variable-coefficient elliptic equations and complex domains.Tests include ring-shaped domains, cubic domains with holes, and molecular surfaces with cusps and self-intersections.
Results: MscaleDNNs solve the tested elliptic PDEs faster, achieve smaller generalization errors, and operate as mesh-less solvers in complex domains.

2 Frequency scaled DNNs and compact activation functions

The frequency-scaled construction decomposes Fourier content into radial bands, down-scales high-frequency bands for DNN learning, and exposes a computational limitation of direct convolution.

Frequency scaling: The initial frequency-scaling idea aims to convert high-frequency function learning into low-frequency DNN learning.The paper presents this as a naive construction before introducing the practical multiscale architecture.
Frequency decomposition: A band-limited function is partitioned in Fourier space into concentric annuli with uniform or non-uniform widths.
Frequency decomposition: The Fourier decomposition induces a corresponding decomposition in physical space using frequency-selection kernels computed analytically with Bessel functions.
Frequency scaling: Radial down-scaling maps each high-frequency Fourier region to a low-frequency region, producing a scaled function in physical space.
Frequency scaling: When iK0/αi is small, a DNN can learn the scaled function and immediately approximate the corresponding frequency component and the full function.
Practical limitation: Direct application requires computing a convolution that is computationally expensive for scattered data, especially in higher dimensions.This difficulty motivates the subsequent multiscale DNN framework.

3 MscaleDNN structures

MscaleDNN structures combine radial frequency scaling with compact-support activations to separate frequency contents and learn solutions across a wide frequency range. Two architectures distribute scaled inputs either across first-layer neuron groups or across summed subnetworks.

3.1 Activation function with compact support: Compact-support activation functions are introduced to improve scale separation and frequency identification in MscaleDNNs compared with ReLU.The paper defines sReLU and a quadratic B-spline activation with compact support; their spatial and frequency-domain behavior is illustrated in Figures 1 and 2.
3.2 Two MscaleDNN structures: The procedure leading to Eq. (2.15) is not practical for numerical approximation in high dimension.
3.2 Two MscaleDNN structures: Radial frequency scaling converts higher-frequency components into lower-frequency functions for approximation across a wide range of frequencies.The proposed scale coefficients can range from 1 to a large number, supporting uniform accuracy across frequencies.
3.2 Two MscaleDNN structures: MscaleDNN-1 partitions first-hidden-layer neurons into groups, with each group receiving a differently scaled input aix.The complete network uses weight, input, bias, scalar activation, entry-wise, and Hadamard operations in its multiscale representation.
3.2 Two MscaleDNN structures: MscaleDNN-2 sums N subnetworks, assigning each scale input to a separate subnetwork.The paper illustrates the two structures in Figure 3 and considers scale coefficients such as ai=i or ai=2i−1.
3.2 Two MscaleDNN structures: A normal network is defined as a fully connected DNN without multiscale features, providing the comparison baseline for numerical experiments.All models are trained with Adam using learning rate 0.001.

4 MscaleDNN for approximations and elliptic PDE’s solutions

The paper applies DNNs to function fitting and elliptic Poisson-Boltzmann equations, training them with mean squared error or Ritz variational minimization. The framework addresses discontinuous dielectric interfaces and uses sampled integrals and boundary penalties in practice.

4 MscaleDNN for approximations and elliptic PDE’s solutions: The section studies fitting functions and solving PDEs, including Poisson-Boltzmann equations, to evaluate MscaleDNN effectiveness.
4.1 Mean squared error training for fitting functions: Mean squared error trains fθ(x) to fit a target function f∗(x) using samples drawn from the target at each epoch.The sample size is denoted by n.
4.1 Mean squared error training for fitting functions: The fitting study examines whether DNNs can learn high-frequency functions when sufficient target-function information is available.
4.2 A Ritz variational method for Poisson-Boltzmann equations: The elliptic Poisson-Boltzmann equation uses dielectric constant ϵ(x) and inverse Debye-Huckel length κ(x), with discontinuous ϵ(x) across the solute-solvent interface.The study imposes a transmission condition at the interface and uses an approximate homogeneous boundary condition on ∂Ω.
4.2 A Ritz variational method for Poisson-Boltzmann equations: The deep Ritz method finds a variational solution by minimizing an energy functional, with MscaleDNN uθ(x) serving as the trial function.The minimizer θ∗ is found using stochastic gradient descent.
4.2 A Ritz variational method for Poisson-Boltzmann equations: The Ritz integral is estimated from random points sampled at each training step, and boundary conditions can be enforced by adding a penalty term.The numerical Ritz loss uses samples from Ω and ∂Ω, with β=1000 in all experiments.
4.2 A Ritz variational method for Poisson-Boltzmann equations: Learning accuracy is evaluated with the L2 error between uθ(x) and utrue(x) on test data points.

5 Effectiveness of various MscaleDNN settings

Experiments compare activation functions, network structures, and scale selections for fitting functions and solving elliptic and PB equations. MscaleDNNs generally converge faster and achieve lower errors, with MscaleDNN-2 using compactly supported φ(x) selected for later experiments.

Overall effectiveness: MscaleDNN losses decay faster and reach smaller values than normal fully connected networks, producing smaller solution errors.This comparison is reported across the section’s fitting and PDE experiments.
Experimental protocol: Several thousand training epochs are sufficient for comparing network performance in the activation-function experiments.Increasing the total number of epochs to 50000 produced similar results.
Activation functions: Compactly supported sReLU and φ outperform ReLU in both MscaleDNN structures, while activation performance is unstable in normal fully connected networks.For the normal structure, φ performs best in fitting but worst for the PDE test.
Network structures: Both MscaleDNN structures outperform the normal structure in both test problems and have similar performance overall.MscaleDNN-2 performs better than MscaleDNN-1 with much fewer connections and supports adaptive scale addition and removal.
Scale selection: Larger-range scales solve elliptic PDEs faster, whereas using all scales equal to 1 performs much worse than using multiscales.The experiments report that proper scale choices improve efficiency, while selection is not highly sensitive.
Selection for later experiments: The selected architecture for subsequent complex-domain experiments is MscaleDNN-2 with compact support function φ(x).The paper identifies this combination as the preferred network for solving Poisson and PB equations in complex or singular domains.

6 MscaleDNNs for Poisson and Poisson-Boltzmann equations in complex and singular domains

The paper applies MscaleDNNs to elliptic equations with broad frequency ranges, variable coefficients, complex domains, and geometric singularities. These experiments include ring-shaped and multiply perforated domains as well as bead-model-inspired PB geometries.

Scope of applications: MscaleDNNs are applied to elliptic equations with broad frequency ranges, variable coefficients, ring-shaped domains, and cubic domains with multiple holes.The applications culminate in PB equations with cusps and self-intersecting surfaces from bead-model bio-molecular geometries.

6.1 Poisson equation in complex domains

MscaleDNNs are tested on Poisson equations in oscillatory, ring-shaped, and multiply perforated domains. Across these settings, they capture multiscale oscillations and reach lower errors faster than normal fully connected networks.

6.1.1 Broad range of frequencies: For broad-frequency solutions, MscaleDNN captures different-scale oscillations that normal DNNs miss, including oscillations in the marked region and corners.The normal network’s expected oscillation nearly disappears in the marked area, whereas MscaleDNN resolves it.
6.1.1 Broad range of frequencies: MscaleDNNs solve both two-dimensional and three-dimensional broad-frequency problems faster to lower errors than normal networks.The corresponding error-versus-epoch comparisons are shown in Figure 13.
6.1.2 A ring-shaped domain: In ring-shaped domains with µ=5 and µ=10, MscaleDNNs faithfully capture the largest-amplitude oscillations that normal networks completely miss.The comparisons use exact and numerical solutions shown in Figures 14 and 15, with error histories in Figure 16.
6.1.3 Square domains with few holes: For square domains with several holes, MscaleDNNs solve both problems faster to lower errors and accurately capture each oscillation of the exact solutions.Normal DNNs fail to resolve the magnitudes of many oscillations in both domains.
6.1.4 A square domain with many holes: The many-hole three-dimensional problem removes 125 holes from a cube, creating complex geometry with oscillatory exact solutions at µ=7π.The experiments compare normal and MscaleDNN-2 structures on three cases.
6.1.4 A square domain with many holes: Across all three many-hole cases, normal fully connected networks do not converge, while MscaleDNNs solve the problems with much smaller errors.The comparison is reported in Figure 21.

6.2 Poisson-Boltzmann equations with domain and source singularities

The paper evaluates MscaleDNNs on Poisson-Boltzmann equations with geometric and source singularities, comparing them with fully connected DNNs. Across these tests, MscaleDNNs converge faster and achieve more accurate solutions, including near singular interfaces and sharp source-induced peaks.

Geometric singularities: The unbounded physical domain is truncated to a ball or cube, with the approximate boundary condition u2 = 0 imposed on the truncation boundary.The paper explicitly notes that this crude condition introduces error into the PDE solution.
Geometric singularities: MscaleDNNs converge faster to smaller training losses and relative L2 errors than normal fully connected DNNs in both geometric-singularity examples.Training losses are used as a practical stopping criterion when the exact solution is unavailable.
Geometric singularities: After 5000 epochs, the normal network gives a wrong interior solution, whereas MscaleDNN provides a satisfactory approximation to the finite-difference reference on a cross-section.The comparison is made on the line x1 = x3 = 0 for the second geometric-singularity example.
Source and geometric singularities: For equations with source and geometric singularities, MscaleDNN errors decay much faster and reach much smaller final values than the normal DNN in both examples.The source terms represent point charges modeled by Dirac delta functions, while geometric singularities arise from the domain construction.
Source and geometric singularities: In the source-singularity experiments, the normal network cannot capture the exact solution’s peaks well on the evaluated line.The comparison uses finite differences, a normal DNN, and MscaleDNN after 10000 epochs.

7 Conclusion and future work

The conclusion presents MscaleDNNs as multi-scale, mesh-less solvers for elliptic PDEs with rich frequency content on complex and singular domains. It identifies frequency-localized activation functions, engineering applications, and high-dimensional PDEs as future directions.

Conclusion: MscaleDNNs combine radial Fourier-domain scaling with compactly supported activation functions to solve elliptic PDEs containing rich frequency content.The method is described as generating multi-scale capability for PDE solution.
Conclusion: MscaleDNN is presented as an efficient, easy-to-implement, mesh-less method for PDEs on complex and singular domains.The paper contrasts this setting with mesh generation and large linear-system costs associated with finite element and finite difference solvers.
Future work: Future work includes wavelet-like activation functions for stronger frequency localization and separation in MscaleDNNs.The proposed direction is motivated by mother wavelet properties.
Future work: Future studies will apply MscaleDNNs to large-scale engineering problems and high-dimensional PDEs, including comparisons with finite element and finite difference methods.These applications are identified as areas for further exploration rather than results established in this paper.