PaperlayerView exact revision on arXiv

Source-linked AI summary

DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations

Kejun Tang, Xiaoliang Wan, Chao Yang

arXiv:2112.14038v2math.NAstat.ML

TL;DR

Uniform sampling can be inefficient for low-regularity and high-dimensional PDEs because it affects the statistical approximation of residual-based losses. The paper proposes DAS, which uses KRnet to generate residual-focused collocation points, and reports significantly improved accuracy in these settings. The authors also identify open questions about optimally coordinating the two deep models.

  • Problem

    Uniform sampling is inefficient for some low-regularity and high-dimensional PDE problems, where it can produce ineffective collocation sets and large Monte Carlo variance.

  • Method

    DAS couples residual-minimizing neural networks with KRnet, which models the residual-induced density and generates new collocation points for training-set refinement.

  • Results

    DAS significantly improves approximation accuracy for low-regularity problems, especially when the dimensionality is relatively large.

  • Takeaways & Limitations

    Residual-guided generative sampling provides an adaptive strategy for improving neural-network approximations of PDEs beyond uniform collocation sampling.

  • Takeaways & Limitations

    The optimal way to coordinate the two deep models remains an open question, and the ideal residual-based sampling density is impractical because it depends on an unknown quantity.

Abstract

from arXiv · show

In this work we propose a deep adaptive sampling (DAS) method for solving partial differential equations (PDEs), where deep neural networks are utilized to approximate the solutions of PDEs and deep generative models are employed to generate new collocation points that refine the training set. The overall procedure of DAS consists of two components: solving the PDEs by minimizing the residual loss on the collocation points in the training set and generating a new training set to further improve the accuracy of current approximate solution. In particular, we treat the residual as a probability density function and approximate it with a deep generative model, called KRnet. The new samples from KRnet are consistent with the distribution induced by the residual, i.e., more samples are located in the region of large residual and less samples are located in the region of small residual. Analogous to classical adaptive methods such as the adaptive finite element, KRnet acts as an error indicator that guides the refinement of the training set. Compared to the neural network approximation obtained with uniformly distributed collocation points, the developed algorithms can significantly improve the accuracy, especially for low regularity and high-dimensional problems. We demonstrate the effectiveness of the proposed DAS method with numerical experiments.

1. Introduction

Deep-learning PDE solvers rely on collocation samples to approximate residual-based loss functionals, but uniform sampling can be inefficient for low-regularity and high-dimensional problems. DAS addresses this by using KRnet to adaptively generate residual-focused training points for iterative improvement.

  • Motivation: The training set affects neural-network generalization because it determines the random samples used to discretize the loss functional.The approximate solution depends on minimizing the discrete loss, whose accuracy is tied to the training samples.
  • Motivation: Uniform collocation sampling can be inefficient when PDE solutions have low regularity or high dimensionality.Low regularity increases Monte Carlo variance, while high-dimensional geometry further reduces sampling effectiveness.
  • Proposed method: DAS uses a deep generative model, KRnet, to guide sample generation for residual-based neural-network PDE approximation.The framework constructs one neural network for the solution and another for refining the training set.
  • Proposed method: DAS treats the residual as a probability density function and approximates it using KRnet-based density modeling.The approximation is obtained by minimizing the KL divergence between the KRnet-induced density and the residual-induced distribution.
  • Adaptive refinement: KRnet generates more collocation points where residuals are large and fewer where residuals are small, thereby guiding training-set refinement.The updated set is used to further improve the current approximate solution, analogous to residual-based adaptive finite elements.
  • Contributions: The work presents DAS sampling procedures and training algorithms for adaptively improving neural-network approximations of PDEs.The contributions emphasize deep generative models for residual-error correspondence, efficient density approximation, and sample generation.

2. Deep learning for PDEs

The paper formulates PDE approximation with neural networks by minimizing a loss combining interior residuals and boundary-condition violations. Its statistical-error discussion distinguishes sampling error from neural-network approximation error and motivates reducing the former.

  • PDE formulation: The PDE setting uses a bounded, connected spatial domain Ω⊂R^d with polygonal boundary ∂Ω.The unknown solution is a function u(x) defined on the spatial domain.
  • PDE formulation: PINNs approximate the PDE solution with a neural network by minimizing a loss functional based on equation residuals and boundary-condition errors.The residual and boundary operators measure how well the network satisfies the PDE and boundary conditions.
  • Optimization: The empirical loss discretizes the objective numerically using sampled points and can be minimized with stochastic gradient-based methods.The formulation focuses on integrating the residual while assuming the boundary integral is adequately approximated by a prescribed sampling strategy.
  • Error decomposition: Neural-network PDE error contains statistical error from Monte Carlo loss discretization and approximation error from the hypothesis space.The statistical component depends on the training-set sampling, whereas approximation error depends on neural-network capability.
  • Scope: The paper focuses on reducing statistical error for the stated formulation while noting that the algorithm can generalize to other neural-network PDE formulations.Related work includes prior error estimates for neural-network-based PDE methods.

3. Illustration of the statistical error

The section interprets machine-learning approximation error through the choice of hypothesis space and random samples. Its analysis separates statistical and approximation errors and explains why sampling matters differently across dimensions.

  • Statistical error: The statistical error arises from replacing a continuous loss or integral with a Monte Carlo approximation based on sampled data.The regression formulation treats the empirical loss as a Monte Carlo approximation of the population loss.
  • Analysis framework: The analysis uses weighted L2 norms and probability-density-weighted samples to characterize approximation and sampling behavior.The continuous least-squares formulation models samples from a density ρ(x), with polynomial spaces providing one analyzed example.
  • Statistical error: For sufficiently large N, the statistical error decreases toward zero, leaving the approximation error determined by the hypothesis space.The stated bound holds with probability at least 1−2δ under the lemma’s conditions.
  • Error sources: Machine-learning function approximation depends on both the hypothesis space and the random training samples.These choices determine the trade-off between statistical error and approximation error.
  • Dimensionality: In low dimensions, classical quadrature can avoid statistical error, whereas high-dimensional classical methods face the curse of dimensionality.Neural networks may instead perform better in high dimensions when paired with a capable hypothesis space and affordable sample size.

4. Deep adaptive sampling method

DAS reduces Monte Carlo error in neural PDE solvers by sampling collocation points from a residual-induced distribution approximated with KRnet. The method targets localized residuals and high-dimensional settings where uniform sampling and simple density models are less effective.

  • Variance reduction: DAS formulates adaptive sampling as variance reduction for the Monte Carlo approximation of the residual loss.The method seeks more effective random samples instead of uniform samples when residuals are localized or irregular.
  • Variance reduction: When the residual is strongly localized, uniform sampling requires O(1/ζ) samples to obtain relative error O(1).This rare-event-like behavior makes increasing the sample size quickly less effective.
  • Variance reduction: Importance sampling improves Monte Carlo accuracy at fixed sample size when the variance of r^2(X)p^-1(X) is smaller than under uniform sampling.The optimal density is unavailable in practice because it depends on the quantity being computed, but a density close to the residual-induced distribution can reduce variance.
  • PDF approximation and sample generation: KRnet addresses the difficulty of approximating residual-induced PDFs in high-dimensional spaces, where exponential-family and Gaussian-mixture models are generally insufficient.The generated samples can add more points in regions of large residual, while the method also handles the compact PDE domain versus whole-space generative-model domain issue.
  • PDF approximation and sample generation: KRnet approximates the residual-induced PDF and generates samples for both probability approximation and training-set refinement.It is a normalizing flow with an explicit likelihood and an invertible transport map from a prior distribution to the target distribution.

5. Analysis of DAS

The analysis establishes error estimates for DAS under stability and residual-related assumptions and describes adaptive-stage refinement procedures. It also identifies unresolved difficulties in quantifying error decay across the DAS variants.

  • Analysis of DAS: DAS refines the collocation training set to reduce discretization error and improve the accuracy of the approximate PDE solution.The method is designed around variance reduction for the residual-loss discretization.
  • Adaptive sampling procedure: The DAS-G procedure samples batches from the interior and boundary training sets, updates the neural PDE solution, and then refines the training set.Its implementation includes KRnet density updates and a dedicated refinement stage.
  • Error analysis: Corollary 1 characterizes the error behavior of approximate solutions at adjacent adaptivity stages when its additional assumptions and boundary condition hold.The analysis states that under certain conditions the expectation of the error bound becomes smaller with adaptive sampling.
  • Limitations of the analysis: Quantifying error decay under adaptive refinement remains unresolved because DAS-R depends on a tail-probability bound and DAS-G changes the loss at each stage.The paper leaves these issues for future study.

6. Numerical experiments

Numerical experiments show that deep adaptive sampling improves neural-network PDE approximation, particularly for localized low-regularity solutions and high-dimensional problems. DAS methods refine collocation sets using residual-informed distributions and outperform uniform sampling, with method-specific differences between DAS-G and DAS-R.

  • Experimental setup: The experiments cover two-dimensional low-regularity problems with one or two solution peaks and high-dimensional linear and nonlinear PDEs.The study also compares DAS methods with uniform sampling and, for high-dimensional problems, residual-based adaptive refinement.
  • Adaptive sampling: DAS treats residual-informed probability distributions as sampling guides, generating collocation points where approximation errors are concentrated.KRnet is used for probability approximation and sample generation, while DAS-G selects new samples through an optimization problem.
  • Two-dimensional peak problem: For the one-peak problem, DAS-G and DAS-R outperform uniform sampling, with DAS-R performing better than DAS-G at the same sample size.Their errors also decay more consistently with training epochs than those from uniform sampling.
  • Two-dimensional two-peak problem: For the two-peak problem, DAS-G and DAS-R outperform uniform sampling, DAS-G performs better than DAS-R, and error decreases as adaptivity iterations increase.Adaptive training sets concentrate around the two peaks and progressively flatten the error profile.
  • High-dimensional problems: In high-dimensional problems, uniform sampling produces relative errors growing quickly to O(1) as dimension increases despite losses close to zero.Uniform points concentrate near the boundary, leaving too few samples to resolve the peak at the origin; DAS methods improve accuracy significantly compared with uniform sampling and RAR.
  • High-dimensional problems: DAS-G and DAS-R reduce residual variance significantly compared with RAR, while DAS-G decreases error slightly faster and appears more robust than DAS-R for one test problem.With relatively few nonuniform samples, DAS methods achieve good approximations where uniform sampling requires many more samples for comparable accuracy.

7. Conclusion

The conclusion presents DAS as a flexible adaptive framework that couples PDE approximation with residual-guided collocation-point generation. Numerical experiments report improved accuracy, particularly for low-regularity and high-dimensional problems, while identifying open design questions.

  • Framework: DAS couples a neural PDE approximator with a deep generative model that generates collocation points according to an error-indicator distribution.The paper uses a regular DNN for PDE approximation and KRnet for density approximation and sample generation.
  • Framework: Residual-induced sampling refines the training set by placing more collocation points where residuals are large and fewer where they are small.This follows the adaptive-refinement principle associated with residual-based indicators.
  • Numerical results: The numerical experiments show significant accuracy improvements for low-regularity problems, especially when dimensionality is relatively large.The conclusion identifies this regime as a central strength of the proposed approach.
  • Numerical results: DAS-G and DAS-R improve accuracy significantly compared with uniform sampling in the reported high-dimensional nonlinear experiments.The figures examine error versus sample size, sampling strategy, adaptivity, and residual variance for a ten-dimensional nonlinear test problem.
  • Future scope: The framework remains flexible because the approximator, error-indicator model, and underlying training-set distribution can be selected according to the problem.The paper uses a regular DNN, KRnet, and a residual-induced distribution, while also discussing gradient-based indicator distributions and space-time domains.
  • Future scope: Open questions include how the two deep models should communicate, what sample size is optimal for Sg_Ω,k, and how the training process can be improved.These questions are presented as directions for future work rather than resolved conclusions.

A. Proof of Lemma 1

The proof develops approximation-error bounds by representing the target and learned functions in an orthonormal basis and controlling perturbations of the associated linear system.

  • Perturbation bound: A perturbed identity matrix I + δA is treated under the condition ∥δA∥2 < 1, enabling the perturbation bound used in the argument.The proof further assumes ∥δA∥2 ≤ 1 − r with 0 < r < 1 when N is sufficiently large.
  • Basis representation: The proof represents V(x) and its approximation using basis functions q_i(x) with coefficient vectors v* and ˆv*.The basis representation reduces the approximation comparison to coefficient estimation.
  • Linear-system perturbation: The coefficient vectors satisfy corresponding linear systems, whose matrix and right-hand-side perturbations are denoted δA and δb.The proof rewrites the learned system in terms of perturbations from the exact system.
  • Assumptions: Under compact support and bounded products of basis functions, the proof controls the relevant quantities using a finite bound M.The assumption is |m1(x)m2(x)| ≤ M for basis elements and x in the domain.
  • Conclusion: The error terms are combined through the stated bounds, orthonormality, and the Pythagorean theorem to reach the lemma’s conclusion.The probability statement is obtained after applying the Hoeffding bound.

B. Proof of Lemma 2

The proof bounds deviations of residual-related quantities by separately controlling three terms and using concentration, variance, and probability-distance inequalities.

  • Termwise bounds: The proof decomposes the target bound into three terms I1, I2, and I3, then bounds each term separately.I3 is controlled in the same way as I1.
  • Termwise bounds: The first term is bounded using the Cauchy–Schwarz inequality.The inequality is explicitly identified as the final step in the bound for I1.
  • Termwise bounds: The second term is controlled using the variance bound Var(Y) ≤ (α−β)^2/4 for variables constrained between α and β.The proof applies this bounded-variable variance estimate to I2.
  • Probability control: The tail probability is estimated through total-variation and L1-norm relations together with Pinsker’s inequality.This step concerns P(|r2/p−µ| > a; p).
  • Conclusion: Combining the bounds for I1, I2, and I3 with the stated equation yields the lemma’s conclusion.The proof presents this combination as the final step.

C. Proof of Theorem 1

The theorem proof combines two preceding inequalities under the stated assumption and obtains a probability guarantee with exponential dependence on the sample size and problem parameters.

  • Proof structure: The proof begins from Assumption 1 and combines inequalities (C.1) and (C.2).The combination is the direct route used to establish the theorem.
  • Probability guarantee: The resulting statement holds with probability at least 1 − exp(−2Nrε2/(τ2 − τ1)2).This is the probability guarantee stated at the end of the proof.

D. Proof of Corollary 1

The proof identifies N as the optimal solution at stage k + 1, takes an expectation involving Rk+1, and concludes the argument.

  • N is identified as the optimal solution at the (k + 1)-th stage.
  • Substituting N into (D.1) yields the stated intermediate result.
  • Because Rk+1 is a random variable, taking its expectation produces the required conclusion.
  • The resulting statement completes the proof.
Loading 2112.14038v2…