PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimally weighted loss functions for solving PDEs with Neural Networks

Remco van der Meer, Cornelis Oosterlee, Anastasia Borovykh

arXiv:2002.06269v4math.NA

TL;DR

Neural-network PDE solvers must balance multiple interior and boundary constraints, but finite network capacity can cause unequal objectives to be optimized unevenly. The paper introduces loss weighting, derives an analytically optimal choice and a solution-based heuristic, and reports better accuracy for most considered problems, including singularly perturbed cases. The method remains sensitive to initialization and can become unstable on difficult problems.

  • Problem

    Finite-capacity neural networks may sacrifice one PDE constraint when multiple objectives have different difficulty, limiting the effectiveness of equally weighted losses.

  • Method

    The paper introduces a loss weight, derives its optimum using ϵ-closeness, approximates it through magnitude normalization, and adaptively updates collocation-point counts.

  • Results

    Much better accuracy was obtained for most considered model problems, and magnitude normalization showed useful properties for singularly perturbed problems.

  • Takeaways & Limitations

    Neural-network PDE accuracy depends on both the PDE and the complexity of its solution, making loss weighting useful for imbalanced objectives.

  • Takeaways & Limitations

    Magnitude normalization can be sensitive to network initialization and unstable for more difficult problems; constant loss weights may also be unsuitable when solution behavior changes.

Abstract

from arXiv · show

Recent works have shown that deep neural networks can be employed to solve partial differential equations, giving rise to the framework of physics informed neural networks. We introduce a generalization for these methods that manifests as a scaling parameter which balances the relative importance of the different constraints imposed by partial differential equations. A mathematical motivation of these generalized methods is provided, which shows that for linear and well-posed partial differential equations, the functional form is convex. We then derive a choice for the scaling parameter that is optimal with respect to a measure of relative error. Because this optimal choice relies on having full knowledge of analytical solutions, we also propose a heuristic method to approximate this optimal choice. The proposed methods are compared numerically to the original methods on a variety of model partial differential equations, with the number of data points being updated adaptively. For several problems, including high-dimensional PDEs the proposed methods are shown to significantly enhance accuracy.

I. INTRODUCTION

Neural networks solve PDEs by representing the solution and minimizing residuals for interior and boundary constraints. The paper motivates this formulation mathematically for linear, well-posed PDEs while noting that low training loss and accurate approximation are not guaranteed in practice.

  • Neural-network PDE solvers: Deep neural networks approximate PDE solutions by mapping d-dimensional inputs to a scalar output representing the solution.The network is trained to satisfy interior and boundary constraints without explicit solution information.
  • Neural-network PDE solvers: PDE constraints can be enforced as hard boundary conditions with soft interior residuals, or as a single loss containing all soft constraints.The paper focuses on the latter formulation used by several prior studies.
  • Scope and limitations: The paper shows that well-posedness and large-network analysis motivate the method, but accurate solutions cannot be inferred solely from reaching a low loss.The introduction also reports inaccurate approximations for some well-posed PDE instances and notes that reaching sufficiently small loss values is not formally established.
  • Mathematical motivation: Minimizing the sampled loss enforces the PDE only at finitely many collocation points, unlike the continuous functional whose global minimizer coincides with the PDE solution.The continuous functional is positive and vanishes only when the interior and boundary residuals vanish everywhere.
  • Mathematical motivation: For linear PDEs, the generalized loss functional is convex, so its local minima in function space are absent.Local minima encountered during neural-network training therefore arise from the network parameterization rather than the functional itself.

III. LOSS FUNCTIONAL MODIFICATION

The paper reframes PDE solving as loss minimization and generalizes the loss by adding a hyperparameter that can be tailored to the PDE. This modification is intended to make the method applicable to a wider range of problems.

  • III. LOSS FUNCTIONAL MODIFICATION: The modification extends the PDE-solving framework by treating the choice of loss weighting as an explicit design decision.This reframes the earlier minimization approach rather than changing the neural-network training engine.
  • III. LOSS FUNCTIONAL MODIFICATION: The proposed modification adds a hyperparameter to the loss functional, creating a more general formulation tailored to the PDE being solved.The parameter choice is investigated later as either a constant or a function.

A. Multi-Objective Optimization

The original loss merges multiple PDE constraints into one equally weighted objective, which can let finite-capacity networks neglect harder constraints. The paper therefore introduces weighted scalarization using a loss weight λ ∈(0, 1).

  • A. Multi-Objective Optimization: The original formulation scalarizes multiple PDE constraints by merging them into a single objective function.This approach is adequate when both individual losses can become very small.
  • A. Multi-Objective Optimization: Finite network capacity can cause unequal constraint difficulty to produce imbalanced optimization, with one objective sacrificed to improve another.The paper identifies this as a reason to treat the multi-objective problem more carefully.
  • A. Multi-Objective Optimization: The generalized method applies weighted scalarization with a scaling parameter λ ∈(0, 1).The parameter is called the loss weight and determines the relative weighting of the objectives.
  • A. Multi-Objective Optimization: The weighted loss family contains the original formulation as a special scaled case, while allowing λ to be selected for broader problem coverage.The paper investigates constant and function-valued choices of λ.

B. Optimized Loss Weights

The paper derives a loss-weight choice that minimizes an upper bound on relative error for a fixed total loss, then identifies assumptions limiting that criterion. The resulting weight makes interior and boundary-loss importance invariant to PDE rescaling.

  • Optimization criterion: The criterion compares loss-function values with solution error because training minimizes loss while the desired outcome is an accurate PDE solution.Holding the optimal loss value constant enables optimization over λ despite uncertainty about which functions the network can represent.
  • Limitations: The derived inequalities are necessary but not sufficient for ϵ-closeness, and their bounds can be extremely lenient.The bounds depend on assumptions about derivative distributions and on the true PDE solution, whereas sufficient absolute-error conditions would depend only on the PDE.
  • Optimal choice: The optimal λ minimizes the smallest admissible ϵ for a fixed total loss value ˆL(ˆu).The derivation equalizes the relevant upper-bound terms, producing the stated closed-form choice.
  • Scale invariance: The resulting λ is scale-invariant: rescaling interior and boundary operators leaves their relative importance unchanged.This makes inherent PDE scale factors almost irrelevant, unlike the default choice c1 = c2 = 1.
  • Limitations: ϵ-closeness is unreliable for functions that vanish within the domain and for homogeneous boundary conditions, where it can imply zero boundary error and λ = 0.The definition is best suited to solutions whose boundary behavior is comparable to their interior behavior and, according to the paper, to zero-mean solutions.

IV. MAGNITUDE NORMALIZATION

The heuristic method replaces unavailable true-solution magnitudes with quantities computed from the candidate solution and normalizes each loss by its corresponding magnitude. This restores the true solution as the unique global minimizer, but local minima and stability issues remain possible.

  • Heuristic weighting: The heuristic approximates the optimal loss weight by making λ a functional of the current approximation rather than keeping it constant.It is designed for settings where the true solution needed by the optimal formula is unavailable.
  • Limitations: When the approximation differs from the true solution, candidate-dependent magnitude bounds need not remain constant, so the heuristic may diverge from the optimal loss.The normalized formulation behaves like the optimal-weight loss mainly when ˆu ≈ u.
  • Magnitude normalization: Magnitude normalization divides each interior and boundary loss by the magnitude of the terms comprising it.A common scale factor preserves the relative sizes of the normalized terms while preventing the zero-loss pathology of the unscaled formulation.
  • Properties: The normalized loss has the true PDE solution as its unique global minimizer, although additional local minima may compromise convergence.The paper identifies local minima as a stability threat because they can obstruct convergence to the true solution.
  • Stability: Magnitude normalization can favor functions with large derivatives, causing instabilities but potentially aiding more difficult problems.Using known boundary magnitudes for Dirichlet or Neumann conditions can significantly improve stability.

V. METHOD SETUP

The experiments use small fully connected feedforward networks with a fixed architecture and train them primarily with L-BFGS. L-BFGS outperformed the tested first-order methods here but requires the full dataset at once.

  • Network configuration: The default network has four hidden layers with twenty neurons each, yielding 1301 + 20d degrees of freedom for PDE dimension d.The architecture is held constant throughout the experiments, with hyperbolic tangent activations unless otherwise specified.
  • Optimization: L-BFGS is the training method of choice because it achieved more accurate results than Adam and stochastic gradient descent in these experiments.Its principal limitation is incompatibility with batching, requiring a dataset representative of the full solution.
  • Optimization: Adam is recommended when datasets are too large to process simultaneously.This recommendation follows the full-dataset requirement of L-BFGS rather than a reported accuracy advantage for Adam.

A. Loss Functions

The experiments compare the original loss, the analytically optimal loss weight, and its heuristic approximation using Monte Carlo estimates of the loss functionals. The comparison uses p = 2 and treats these as three methods of interest.

  • Implementation: The implementation optimizes neural-network parameters θ with L-BFGS while approximating loss-function integrals by Monte Carlo integration.The setup is instantiated for linear PDEs using interior and boundary collocation points.
  • Compared methods: The three experimental methods are the original loss, the proposed optimal loss weight, and the heuristic approximation of that weight.The methods are compared using the corresponding Monte Carlo loss approximations.
  • Loss specification: The experiments use norm p = 2 for the loss functions.The paper notes that minimizing a monotonically transformed objective is equivalent to minimizing the original objective.

B. Adaptive Collocation Points

The method adaptively increases interior or boundary collocation points when validation losses substantially exceed corresponding training losses, improving loss estimation during training.

  • Training uses separate interior and boundary training and validation collocation sets to monitor whether each loss is adequately approximated.The initial counts nI and nB determine both sets.
  • When a validation loss exceeds its corresponding training loss by a factor q, the respective collocation-point count is doubled.The rule is applied independently to interior and boundary losses.
  • Adaptive counts address the difficulty of choosing nI and nB because loss variance depends on the neural network state.

C. GPU Acceleration

The experiments emphasize problem formulation rather than neural-network or optimizer tuning, while exploiting parallel neural-network computations on available hardware. The implementation used a Tesla K80 GPU for the numerical experiments.

  • Neural-network computations can be distributed across many cores because many training-iteration operations are independent.
  • The numerical experiments were computed on a Tesla K80 GPU provided through Google’s Colaboratory project.
  • The experiments use small neural networks and powerful training algorithms to focus on improving the PDE problem formulation rather than hyperparameter tuning.Limiting network capacity requires using that capacity efficiently.
  • The study compares the original, optimal-loss-weight, and magnitude-normalization methods across increasingly difficult model PDEs.The test problems include Laplace, Poisson, and convection-diffusion equations.

A. Laplace Equation

For two-dimensional Laplace problems, increasing frequency makes the neural-network approximation harder, while the proposed loss-weighting methods substantially improve accuracy over the original method. Adaptive training behavior reveals different trade-offs between boundary fitting, interior loss, and point usage.

  • Problem setup: The experiments vary the Laplace eigenfrequency from ω = π to ω = 10π, with higher frequencies expected to be harder to learn.The study also extends the Laplace experiments to dimensions up to six.
  • Problem setup: The optimal loss weight decreases from approximately 1.58e-2 at ω = π to 1.61e-5 at ω = 10π.The paper gives the approximation λ ≈ 1/(1+2ω^3).
  • Accuracy comparison: Problems with frequencies higher than 4π were not solved accurately by the original method, whereas magnitude normalization and optimal loss weights produced significantly more accurate approximations.
  • Accuracy comparison: In most cases, the proposed methods were at least one order of magnitude more accurate than the original method, with the difference increasing at higher frequencies.
  • Accuracy comparison: At ω = 10π, the original method overemphasized the PDE relative to the boundary conditions, yielding a very small Laplacian but unsatisfied boundary conditions.
  • Training behavior: Magnitude normalization and optimal loss weights show similar training behavior, but magnitude normalization initially adapts more slowly and exhibits larger loss spikes.Magnitude normalization first estimates the solution’s overall shape before approximating the optimal weight.
  • Adaptive collocation: Magnitude normalization tends to use more collocation points because of its tendency to overfit, while the original method severely underfits the boundary at ω = 10π.
  • Loss-error relation: Both the loss and absolute error decay as the solution becomes flat, indicating that ε-closeness can predict the overall loss behavior despite not holding exactly.

2. Higher-Dimensional Problems

The proposed loss-weighting methods remain effective for higher-dimensional and high-frequency PDEs, where the original method loses accuracy more rapidly. Adaptive collocation generally required no more than 16,384 points per category, even in six dimensions.

  • The original method had trouble solving even the three-dimensional problem, while the proposed methods became gradually less accurate as dimensionality increased.
  • 16,384 adaptive interior or boundary collocation points were never exceeded, even for six-dimensional problems.The results suggest that required data depends more on solution complexity than dimensionality alone.
  • For frequencies of 4π or higher, the original method failed, whereas the proposed methods solved all considered frequency problems.Accuracy declined at higher frequencies, likely because the solutions became more complex.
  • In Poisson problems, the original method’s accuracy dropped faster than that of the proposed methods as frequency increased.The original method mainly failed because boundary conditions contributed too little to its loss, while both proposed methods fit them even at the highest frequency.
  • Source functions appeared to increase the collocation points required by adaptive counting, and they also affected magnitude normalization’s stability at very high frequencies.At sufficiently high frequencies, convergence depended on neural-network initialization and could be restored with pre-training.
  • Both optimally weighted methods and magnitude normalization handled Poisson problems with much greater derivatives, supporting ε-closeness even with source functions.

2. Source Functions with Peaks

The experiments examine peaked source functions and boundary-layer solutions, showing when magnitude normalization and other proposed methods improve accuracy and when instability or extreme solution variation limits them.

  • The peaked-source problem was designed to test increasingly complex solutions, including a case previously studied without a convincing solution.
  • Source Functions with Peaks: The proposed methods outperformed the original method on the peak problem, and all three methods improved substantially over the results reported in.The difference between methods remained fairly small because early collocation points likely captured the low-frequency solution behavior and established boundary fit.
  • Source Functions with Peaks: Magnitude normalization produced a smooth error distribution near the solution peak without visible neural-network artifacts.
  • For hyperbolic PDEs, theoretically optimal loss weights improved accuracy, while magnitude normalization generally produced similar gains but could become unstable.The instability acts as a driving force toward larger derivatives.
  • For diffusivity α ≥ 10^-2, all three convection-diffusion methods were very accurate, but none obtained accurate results for smaller α.A single global loss weight may be insufficient for the sharply different behaviors inside and outside the boundary layer.
  • For α = 10^-4, starting with 256 interior collocation points enabled magnitude normalization to reach relative L2 error 1.41e-4 and relative L∞ error 8.78e-4.Increasing the initial point count prevented spikes between collocation points in this case.
  • Increasing initial collocation points did not enable the other two methods to solve small-α problems and only slightly improved large-α accuracy at substantial computational cost.

VII. DISCUSSION AND CONCLUSION

The paper generalizes neural-network PDE solvers with weighted objectives, adaptive collocation, and theoretically motivated error control. Experiments report improved accuracy for imbalanced and singularly perturbed problems, while stability, convergence, and practical scope remain open concerns.

  • Contributions: The method introduces a loss weight to compensate for imbalanced boundary-condition and PDE objectives, plus heuristic magnitude normalization and adaptive collocation updates.The optimal weight depends on the true PDE solution, while magnitude normalization approximates it from the learned solution.
  • Results: Much better accuracy was obtained for most considered model problems with epsilon-closeness and optimal loss weights.The experiments specifically included problems constructed with imbalanced objective functions.
  • Results: Magnitude normalization showed useful properties for solving singularly perturbed problems.The paper also relates solver accuracy to the complexity of the solution, not only to the PDE itself.
  • Limitations: Magnitude normalization can be sensitive to initialization and unstable for more difficult problems.The authors suggest that local loss weights may help when solutions exhibit changing behavior.
  • Limitations: Error bounds and convergence guarantees are currently unavailable, and neural-network solvers remain computationally costly for single PDE instances where traditional methods exist.The authors instead identify parametrized PDE families as a likely application area.
  • Future directions: The theoretical framework may extend beyond PDE solving to simultaneous optimization of more than two objectives.The paper identifies solving multiple PDEs at once as a natural next step.
Loading 2002.06269v4…