Source-linked AI summary
Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
Wei Cai, Zhi-Qin John Xu
TL;DR
Common DNNs can struggle with high-frequency content, despite the high-dimensional scientific problems that motivate their use. The paper proposes MscaleDNN, combining radial frequency scaling with compact-supported activations. The reported numerical results show faster learning for high-dimensional function fitting and elliptic problems, as well as accurate capture of highly oscillatory functions.
Problem
Common DNNs learn low-frequency content quickly but can be inadequate for high-frequency data relevant to high-dimensional physical problems.
Source: §1 · pp. 1–2Method
MscaleDNN combines radial Fourier-space scaling with compact-supported activations to support multi-scale approximation of high-frequency, high-dimensional functions and PDE solutions.
Source: §1 · p. 2, §2 · pp. 3–4, §3 · p. 4Results
The authors report faster learning for high-dimensional function fitting and elliptic problems, and MscaleDNN accurately captures highly oscillatory test functions where a one-scale DNN does not.
Source: §4.3 · p. 8, §5 · p. 13Takeaways & Limitations
Numerical experiments support using MscaleDNN for high-dimensional function fitting and elliptic problems with either Ritz energy or least-squares residual minimization.
Source: §3 · p. 5, §5 · p. 13The current implementation uses scaled compact-supported activations, with redundant or overlapping scales; the authors propose wavelet-like neurons as a more sophisticated direction.
Source: §5 · p. 13
Abstract
from arXiv · showhide
In this paper, we propose the idea of radial scaling in frequency domain and activation functions with compact support to produce a multi-scale DNN (MscaleDNN), which will have the multi-scale capability in approximating high frequency and high dimensional functions and speeding up the solution of high dimensional PDEs. Numerical results on high dimensional function fitting and solutions of high dimensional PDEs, using loss functions with either Ritz energy or least squared PDE residuals, have validated the increased power of multi-scale resolution and high frequency capturing of the proposed MscaleDNN.
1 Introduction
The introduction identifies high-frequency learning as a key challenge for DNNs used in high-dimensional scientific problems. It presents MscaleDNN as a radial Fourier-space scaling approach intended to address that challenge without PhaseDNN’s curse of dimensionality.
1 Introduction
- Common DNNs learn low-frequency content quickly but can be inadequate for high-frequency physical phenomena, including wave propagation in random media and excited quantum states.The introduction contrasts this behavior with multigrid methods, which converge first in the solution’s high-frequency spectrum.Source: §1 · pp. 1–2
- MscaleDNN uses radial partitioning and scaling in Fourier space to convert high-frequency learning into low-frequency learning for high-dimensional functions and PDEs.The introduction contrasts this with PhaseDNN, whose coordinate-wise phase shifts can incur a curse of dimensionality (Cai et al. 2019).Source: §1 · p. 2
2 The idea of frequency scaled DNN and activation function with compact support
This section develops frequency scaling for high-dimensional function and PDE approximation, alongside compact-supported activations for multi-scale resolution. It explains how radial frequency bands can be downscaled for DNN learning and why scaled compact support helps neurons target different frequencies.
2 The idea of frequency scaled DNN and activation function with compact support
- The Fourier domain is partitioned into concentric annuli, yielding a corresponding decomposition in physical space.The annuli may have uniform or nonuniform widths.Source: §2 · pp. 2–3
- Downscaling a high-frequency annulus can place its spectrum in a low-frequency region, where a DNN is trained to learn the scaled function and approximate the original.The section invokes the F-Principle to motivate learning when the scaled frequency is small.Source: §2 · p. 3
- The direct procedure requires a d-dimensional convolution, making it impractical for high-dimensional approximation.Source: §2 · p. 4
- Scaling a compact-supported activation scales its frequency spectrum, helping the corresponding neuron learn the associated solution frequency.Source: §2 · p. 4
3 MscaleDNN with compact supported activation function
MscaleDNN combines multiple input scales to represent functions with many frequencies, aiming to speed convergence. The section describes using this network with Ritz and least-squares residual losses for PDE solutions.
3 MscaleDNN with compact supported activation function
- A series of scales from 1 to a large value forms MscaleDNN’s function space to speed convergence for solutions with many frequencies.The procedure that motivates this function space is described as impractical for high-dimensional approximation.Source: §3 · p. 4
- MscaleDNN partitions first-hidden-layer neurons into groups that receive differently scaled inputs; this is its only stated difference from a normal fully connected network.The section gives the group input as i x and illustrates A = 3 groups.Source: §3 · p. 5
3.1 A Ritz variational method for PDE
- For the Ritz approach, MscaleDNN output parameterizes the variational solution, with the energy integral sampled at random points during training.The Ritz loss also includes a boundary-condition penalty.Source: §3.1 · p. 6
3.2 MscaleDNN least square error method for PDEs
- The alternative least-squares method minimizes the mean-square PDE residual, while solution accuracy is assessed by the distance from the true solution.Source: §3 · p. 5, §3.2 · p. 6
4 Numerical experiments
Across the experiments, MscaleDNNs learn high-frequency functions and high-dimensional PDE solutions faster than one-scale networks. Results also depend on activation: sReLU benefits from multiple scales, while ReLU does not show that advantage in the 60-dimensional tests.
4 Numerical experiments
- Across most cases, training-loss curves overlapped well with test-loss curves, indicating good generalization.Source: §4 · p. 6
4.1 Fitting 3-dimensional functions
- In 3D function fitting, sReLU learned faster than ReLU, and 100-scale sReLU-DNNs were much faster than their one-scale counterparts.The comparisons cover one-hidden-layer and multiple-hidden-layer networks.Source: §4.1 · p. 7, Figure 3 · p. 7, Figure 4 · p. 7
4.2 Fitting 60-dimensional functions with intrinsic low dimensional structure
- For 60-dimensional data with intrinsic dimension 3, sReLU MscaleDNNs learned both linear and nonlinear embeddings faster; ReLU showed no advantage from more scales, and D2 results were similar.Source: §4.2 · p. 8, Figure 5 · p. 8
4.3 Fitting high-frequency function
- On high-frequency 1D and 2D fitting tasks, MscaleDNN loss decreased faster and captured target oscillations better than one-scale DNNs.Source: §4.3 · pp. 8–9, Figure 7 · p. 9, Figure 9 · p. 10
4.4 Solving high dimensional PDE
- With Ritz loss, 100-scale MscaleDNNs were much faster than one-scale networks for d = 3, 10, and 25.The comparisons use initialization D1 in Fig. 11 and D2 in Fig. 12.Source: §4.4 · p. 11, Figure 11 · p. 11, Figure 12 · p. 12
- With LSE loss, 100-scale MscaleDNNs were much faster than one-scale networks for d = 3 and 10, although LSE computation is costlier because it requires a Laplacian.The comparisons use initialization D1 in Fig. 13 and D2 in Fig. 14.Source: §4.4 · p. 12, Figure 13 · p. 12, Figure 14 · p. 12
5 Conclusion and future work
MscaleDNN combines frequency-domain scaling with compactly supported activations for high-frequency, high-dimensional function approximation and PDE solving. The conclusion highlights faster learning and a possible wavelet-based refinement, while distinguishing the costs and applicability of two minimization approaches.
5 Conclusion and future work
- Radial scaling and compactly supported activations let MscaleDNN learn high-dimensional function fits and elliptic PDE solutions faster with either least-squares residuals or Ritz energy.The two minimization approaches have different derivative costs and applicability.Source: §5 · p. 13
- Ritz minimization requires only first derivatives, whereas least-squares residuals require the full Laplacian but can handle non-self-adjoint operators such as high-dimensional Fokker–Planck equations.The higher derivative cost is the stated trade-off for the least-squares approach.Source: §5 · p. 13
- The current scaled compact-support neurons create overlapping scales; mother-wavelet-like neurons with vanishing moments are proposed as a route to a Wavelet-DNN multi-resolution framework.The current implementation corresponds to scaling spaces in wavelet theory.Source: §5 · p. 13