PaperlayerView exact revision on arXiv

Source-linked AI summary

Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks

Suryanarayana Maddu, Dominik Sturm, Christian L. Müller, Ivo F. Sbalzarini

arXiv:2107.00940v1cs.LGcs.NAmath.NAphysics.comp-phq-bio.QM

TL;DR

PINNs optimize weighted data-fidelity and physics-derived objectives, but scale imbalances can cause training pathologies. The paper connects these pathologies to Sobolev training and proposes inverse-Dirichlet weighting; it reports that the method outperforms other weighting approaches and best protects against catastrophic forgetting in sequential training.

  • Problem

    PINN objectives can have dominant gradient statistics from high derivatives or high-frequency components, biasing optimization against objectives with smaller gradients.

    Source: B. Vanishing task-specific gradients… · p. 3

  • Method

    The paper connects PINN training pathologies to Sobolev training and proposes dynamically adapting regularization weights using inverse-Dirichlet weighting.

    Source: §I · p. 1, §V · p. 8

  • Results

    Inverse-Dirichlet weighting outperformed other weighting heuristics and pinn-MGDA, and performed best at protecting against catastrophic forgetting in sequential training.

    Source: §V · p. 8

  • Takeaways & Limitations

    The method can be included in SGD-type optimizers at almost no additional computational cost and may improve PINN accuracy and convergence by orders of magnitude.

    Source: §V · p. 8

Abstract

from arXiv · show

We characterize and remedy a failure mode that may arise from multi-scale dynamics with scale imbalances during training of deep neural networks, such as Physics Informed Neural Networks (PINNs). PINNs are popular machine-learning templates that allow for seamless integration of physical equation models with data. Their training amounts to solving an optimization problem over a weighted sum of data-fidelity and equation-fidelity objectives. Conflicts between objectives can arise from scale imbalances, heteroscedasticity in the data, stiffness of the physical equation, or from catastrophic interference during sequential training. We explain the training pathology arising from this and propose a simple yet effective inverse-Dirichlet weighting strategy to alleviate the issue. We compare with Sobolev training of neural networks, providing the baseline of analytically $\boldsymbol{\epsilon}$-optimal training. We demonstrate the effectiveness of inverse-Dirichlet weighting in various applications, including a multi-scale model of active turbulence, where we show orders of magnitude improvement in accuracy and convergence over conventional PINN training. For inverse modeling using sequential training, we find that inverse-Dirichlet weighting protects a PINN against catastrophic forgetting.

I. INTRODUCTION

The introduction identifies unresolved challenges in balancing PINN objectives and proposes inverse-Dirichlet weighting, evaluated against Sobolev training and other weighting strategies. It frames PINNs as weighted multi-objective optimization over data and physics constraints.

I. INTRODUCTION

  • PINN training must balance conflicting objectives, yet the field lacks consensus on when to use weighting strategies and optimal benchmarks for differential-equation objectives.Source: §I · p. 1
  • The paper proposes inverse-Dirichlet weighting to address PINN training pathologies and evaluates it against Sobolev training and other weighting approaches.The comparison includes a case with provably optimal weights and state-of-the-art PINN weighting methods.Source: §I · p. 1

II. A PHYSICS INFORMED NEURAL NETWORK IS A MULTI-OBJECTIVE OPTIMIZATION PROBLEM

  • Weighted-sum scalarization assigns relative weights to objectives, but manual tuning and grid search become prohibitively expensive for large neural networks.Source: §II · p. 2
  • A PINN loss is a scalarized multi-objective loss whose terms represent data fidelity or constraints derived from the physical model.Source: §II · p. 2

A. Sobolev training is a special case of PINN training

Sobolev training casts derivative information as additional objectives, connecting it to PINN training. The paper links high derivatives or dominant frequencies to imbalanced task gradients, which can produce stiff learning dynamics and a likely PINN failure mode.

A. Sobolev training is a special case of PINN training

B. Vanishing task-specific gradients are a failure mode of PINNs

III. STRATEGIES FOR BALANCED PINN TRAINING

PINN objectives can require conflicting parameter updates, making balanced optimization difficult. The section contrasts gradient-magnitude weighting with inverse-Dirichlet weighting, which balances gradient variance across objectives.

III. STRATEGIES FOR BALANCED PINN TRAINING

  • The earlier inverse-gradient-magnitude strategy can improve PINN performance but does not prevent vanishing task-specific gradients.Its weights use moving averages of inverse gradient-magnitude statistics; α is user-defined and is set to 0.5 here.Source: §III · p. 3

A. Inverse-Dirichlet weighting

  • Inverse-Dirichlet weighting sets weights so weighted-gradient component variances match across objectives, directly preventing vanishing task-specific gradients.It uses the update rule in Eq. 4 and is based on gradient variance rather than inverse average gradient magnitude.Source: A. Inverse-Dirichlet weighting · p. 3
  • With Adam, inverse-Dirichlet weights can be computed from averaged squared task-specific gradients, proportional to their Dirichlet energy.The energy is expressed as an integral of squared shared-parameter gradients over parameter space.Source: A. Inverse-Dirichlet weighting · p. 4
  • Unlike inverse-average-magnitude weighting, inverse-Dirichlet weighting measures uncertainty from loss-gradient noise, not observational noise.The former uses average gradient magnitude as its measure; the latter uses gradient variance.Source: A. Inverse-Dirichlet weighting · p. 4

B. Gradient-based multi-objective optimization

The paper compares gradient-based multi-objective optimization with weighting strategies, then tests those approaches in Sobolev training. In that test, inverse-Dirichlet weighting avoids vanishing task-specific gradients and reaches accuracy comparable to analytically ε-optimal weights.

B. Gradient-based multi-objective optimization

  • pinn-MGDA finds weights satisfying KKT conditions for shared-parameter gradients, producing a descent direction that improves all objectives and converges to a Pareto-stationary point.The paper adapts MGDA for PINNs and calls the resulting algorithm pinn-MGDA.Source: B. Gradient-based multi-objective… · p. 4

IV. RESULTS

  • Sobolev training tests function approximation and unknown coefficient inference, with analytically derived ε-optimal weights serving as an unbiased-optimization baseline.The test uses derivatives up to fourth order and coefficients whose true values are all 1.Source: §IV · p. 4

A. Inverse-Dirichlet weighting avoids vanishing task-specific gradients

  • Uniform weights bias training toward the fourth-derivative objective, while pinn-MGDA favors the zeroth-derivative objective and max/avg weighting leaves gradients unbalanced.Inverse-Dirichlet weighting produces balanced gradient histograms closest to those from ε-optimal analytical weights.Source: A. Inverse-Dirichlet weighting avoids… · pp. 4–5
  • Inverse-Dirichlet weighting achieves orders-of-magnitude better function and coefficient accuracy than the compared strategies, matching ε-optimal weights and approximating high frequencies.Among the tested approaches, only inverse-Dirichlet weighting avoids vanishing task-specific gradients; pinn-MGDA appears to escape the stiffness instead.Source: A. Inverse-Dirichlet weighting avoids… · p. 5
  • In this test, pinn-MGDA can escape stiffness through rapid attenuation of the residual Fourier spectrum, despite retaining unbalanced gradients.The authors note this behavior may be problem-specific.Source: A. Inverse-Dirichlet weighting avoids… · p. 5

B. Inverse-Dirichlet weighting enables multi-scale modeling using PINNs

For multi-scale PINN modeling, inverse-Dirichlet weighting outperforms the compared weighting schemes in active-turbulence approximation and convergence. It also performs better in two supplementary multi-scale examples, while the uniformly weighted PINN fails in the tested active-turbulence geometries.

B. Inverse-Dirichlet weighting enables multi-scale modeling using PINNs

  • After 7000 epochs, inverse-Dirichlet weighting gives the best vorticity accuracy; uniform weighting fails in both square and annular domains by neglecting initial, boundary, and incompressibility constraints.pinn-MGDA ranks next, followed by max/avg weighting.Source: B. Inverse-Dirichlet weighting enables… · p. 6
  • Inverse-Dirichlet weighting converges at order r = 4, versus linear convergence (r = 1) for pinn-MGDA and max/avg weighting; uniform weighting does not converge.The comparison concerns convergence as spatial resolution h changes.Source: B. Inverse-Dirichlet weighting enables… · p. 6
  • Inverse-Dirichlet weighting recovers the active-turbulence power spectrum and captures fine structures that other weighting methods miss.The reported residual Fourier spectra also show rapid solution convergence with inverse-Dirichlet weighting.Source: B. Inverse-Dirichlet weighting enables… · p. 6
  • In two supplementary multi-scale examples—2D Poisson and curvature-driven level-set flow—inverse-Dirichlet weighting also outperforms the other approaches in accuracy or convergence rate.The authors conclude that avoiding vanishing task-specific gradients may enable PINNs for multi-scale problems previously not amenable to neural-network modeling.Source: B. Inverse-Dirichlet weighting enables… · p. 6

C. Inverse-Dirichlet weighting protects against catastrophic forgetting

Inverse-Dirichlet weighting performed best across the compared PINN weighting schemes and gave the strongest protection against catastrophic forgetting in the tested sequential-training setting. The paper connects PINN training pathologies to high-frequency components or high derivatives and identifies first-order SGD as the scope for its analysis.

C. Inverse-Dirichlet weighting protects against catastrophic forgetting

  • Adding divergence-freeness and then the PDE residual to velocity-data training caused uniform-weight PINNs to lose their earlier parameterization and develop increasing velocity test error.The objectives were introduced at epochs 1000 and 2000, respectively.Source: C. Inverse-Dirichlet weighting… · p. 7
  • When objectives were introduced sequentially, inverse-Dirichlet and pinn-MGDA were minimally affected by forgetting, while max/avg weighting protected the network less well.Inverse-Dirichlet weighting also outperformed the alternatives on inferred model coefficients and latent pressure, for which no training data was given.Source: C. Inverse-Dirichlet weighting… · p. 7

V. CONCLUSION AND DISCUSSION

  • The analysis explains PINN pathologies through numerical stiffness in Sobolev training and attributes them to dominant high-frequency components or high derivatives in the physics prior.The proposed dynamic adaptation strategy is termed inverse-Dirichlet weighting.Source: §V · p. 8
  • Across 2D Poisson, multi-scale active turbulence, and curvature-driven level-set flow, inverse-Dirichlet weighting empirically performed best in the reported comparisons.The baselines included max/avg weighting, pinn-MGDA, and analytically derived ǫ-optimal static weights in a linear test problem.Source: §V · p. 8
  • Inverse-Dirichlet weighting offered the best protection against catastrophic forgetting among the compared schemes in sequential PINN training.The conclusion links this protection to making sequential PINN training a viable practical option.Source: §V · p. 8
  • The analysis applies to PINNs, Sobolev training, and other multi-objective losses when training uses an explicit first-order SGD optimizer.The authors identify generalization to other first-order optimizers as an open question.Source: §V · p. 8

Appendix A: Material and Methods

This appendix derives static weights for Sobolev training using ǫ-closeness bounds. It formulates the choice as a maximization-minimization problem and states that an analytical solution exists.

1. Determining analytical optimal weights

  • For elliptic-PDE Sobolev losses, ǫ-closeness analysis can determine weights that minimize all objectives in an unbiased way.A candidate solution is defined as ǫ-close when it meets the stated condition for every multi-index and point in the domain.Source: §1 · p. 10
  • The appendix uses the ǫ-closeness inequality to bound total loss and seeks weights that give the tightest upper bound.The weights are selected by a maximization-minimization problem.Source: §1 · p. 10
  • The maximization-minimization problem becomes a piecewise linear convex optimization problem with an auxiliary variable, subject to weights summing to one and upper-bound constraints.The text states that an analytical solution exists, but the supplied passage does not give its full expression.Source: §1 · p. 11
  • For the typical Sobolev-training loss, the appendix applies ǫ-closeness analysis to compute weights leading to ǫ-optimal solutions.The supplied passage introduces this computation but does not provide the complete formula.Source: §1 · pp. 10–11

2. PINN Multiple Gradient Descent Algorithm (pinn-MGDA)

The appendix adapts multiple gradient descent to PINNs by expressing Pareto-stationarity conditions as a quadratic program. It also outlines a Fourier-based argument for the training-rate proposition under a dominant-frequency assumption.

2. PINN Multiple Gradient Descent Algorithm (pinn-MGDA)

  • The PINN version of MGDA formulates Pareto-stationarity conditions as a quadratic program and solves it with Frank-Wolfe.The program uses Q = UᵀU, with U defined in the appendix.Source: §2 · p. 11
  • The MGDA schematic depicts two objectives, a Pareto-optimal front, the feasible objective space, and a descent direction that seeks to reduce both objectives.The depicted direction is closer to the Pareto-optimal front.Source: Figure A.1 · p. 11

Proof of Proposition 1

  • The Proposition 1 proof assumes a delta-function Fourier spectrum with a dominant wavenumber and uses Parseval’s theorem to compute objective-gradient magnitudes.The derivation also uses the triangle inequality and the delta-spectrum assumption for the Sobolev objective.Source: Proof of Proposition 1 · p. 12
  • The proof combines the derived gradient expressions and assumes the network residual’s spectral rate is inversely proportional to wavenumber.These steps support the stated first-order parameter-update analysis of the dominant Fourier mode.Source: Proof of Proposition 1 · p. 12

Sobolev training set-up

The appendix specifies a synthetic Sobolev-training experiment and active-turbulence simulation settings. It then describes the PINN architecture, sampling, and optimization setup for forward solutions in square and annular domains.

Sobolev training set-up

  • The Sobolev experiment trains a five-layer, 64-neuron-per-layer sine network on a 128 × 128 grid of a two-dimensional sinusoidal target, using 8192 training points and Adam for 20000 epochs.Training uses two mini-batch updates per epoch with batch size 4096, and the learning rate decreases after epochs 10000 and 15000.Source: Sobolev training set-up · p. 13

Appendix C: Forward and inverse modeling in active turbulence

  • The active-fluid ground truth uses a pseudo-spectral solver on a 256 × 256 grid with time step 0.0001, enforcing incompressibility by projection.The square-domain simulation uses periodic boundaries and the listed parameter values ξ0 = 3.5, α = −1, β = 0.5, Γ0 = −0.045, and Γ2 = |Γ|3.Source: Appendix C · p. 15

Forward problem set-up

  • For forward PINN solutions, a seven-layer, 100-neuron-per-layer sine network maps space and time to velocity, with vorticity computed by automatic differentiation.The square setup uses 50 time steps and trains for 7000 epochs; the annular setup uses inner and outer radii of approximately 0.245 and 0.85 units and also trains for 7000 epochs.Source: Forward problem set-up · p. 15
  • The square and annular forward problems use distinct training domains, and the listed loss weights correspond to boundary, equation-residual, divergence, and initial-condition objectives.The square setup has 4096 initial, 12800 boundary, and 192200 interior residual points; the annular setup has 3536 initial, 14400 boundary, and 162400 interior points.Source: Figure C.1 · p. 15, Forward problem set-up · p. 15

Inverse problem set-up

The inverse problem uses velocity measurements to infer effective pressure and active-turbulence parameters with a neural network. Training introduces data, divergence, and equation-residual objectives in sequence, with figures documenting weighting behavior and inferred quantities.

Inverse problem set-up

  • The five-layer, 100-neuron network infers effective pressure and parameters ξ0, α, β, Γ0, and Γ2 from 204,800 velocity samples, using 143,360 training points.The model maps (x, y, t) to (u, v, p*), and trains for 7,000 epochs with 35 mini-batch updates per epoch.Source: Inverse problem set-up · p. 16
  • Training proceeds from velocity-data fitting to adding the divergence constraint at epoch 1,000 and the equation residual at epoch 2,000.The learning rate starts at 10^-3 and decreases tenfold after epochs 3,000 and 6,000; max/avg weights are reset when a task is introduced.Source: Inverse problem set-up · p. 16
  • The appendix tracks weighting trajectories across square and annular forward problems and compares Fourier residual spectra for horizontal and vertical velocity across methods.The residual spectrum plots encode magnitude |er(k)|; the weighting-trajectory panels are grouped by geometry and weighting strategy.Source: Figure C.2 · p. 17, Figure C.3 · p. 17
  • At epoch 5,000, gradient histograms compare task-specific objective gradients under different λ weighting schemes for the squared-domain forward problem.Source: Figure C.4 · p. 18
  • The sequential-training evaluation tracks velocity test error, inferred-coefficient errors, and latent pressure-gradient error under noisy velocity data with additive Gaussian noise.The figure marks introduction of divergence and PDE residual constraints at epochs 1,000 and 2,000, respectively.Source: Figure C.5 · p. 18
  • The appendix also visualizes effective-pressure-gradient predictions and pointwise errors for clean and noise-corrupted velocity inputs, and profiles per-epoch training times for two active-turbulence formulations.Source: Figure C.6 · p. 19, Figure C.7 · p. 19, Figure C.8 · p. 20

Appendix D: 2D Poisson problem

The Poisson example compares weighting strategies across oscillation frequencies, while the level-set appendix describes a curvature-driven spiral problem and its training setup.

Appendix D: 2D Poisson problem

  • The Poisson PINN uses 400 boundary points, 2,500 interior points, and a five-layer, 50-node tanh network trained for 30,000 epochs.The equation's oscillatory solution is evaluated on a 100 × 100 grid; the frequency parameter ω controls its oscillation.Source: Appendix D · p. 21
  • Inverse-Dirichlet weighting performs on par with analytically ε-optimal weighting across increasing Poisson wavenumbers, followed by max/avg and PINN-MGDA.The figure includes the solution at ω = 6 and relative L2 error comparisons as wavenumber increases.Source: Appendix D · p. 21, Figure D.1 · p. 21

Appendix E: Curvature-driven level-set

  • The level-set example evolves a wound spiral under local curvature, representing the interface through a zero level set initialized from a distance function.The simulation uses a 128 × 128 grid and second-order Runge-Kutta time integration with step size 0.001.Source: Appendix E · p. 22

Training set-up

  • The level-set PINN maps (x, y, t) to φ and trains for 2,500 epochs on interior, boundary, and initial-condition points; figures compare weighting strategies and solution errors.Training uses a five-layer, 64-node network with ELU activation and 187 mini-batch updates per epoch.Source: Training set-up · p. 22, Figure E.1 · p. 23, Figure E.2 · p. 23

Appendix F: Network initialization and parameterization

The appendix specifies the network's batch-based parameterization and initialization, then describes how dynamic task weights are updated during training.

Appendix F: Network initialization and parameterization

  • The network maps batches of input coordinates through an n-layer composite function to outputs, with each layer applying weights, biases, and a nonlinear activation.The output dimension corresponds to the modeled variable's dimension.Source: Appendix F · p. 24
  • Weights use normal Xavier initialization, biases are zeroed, and inputs are normalized using training-data column means and standard deviations.The stated gain is 1 for all activations except tanh, for which the recommended gain is 5/3.Source: Appendix F · p. 24

Weights computation

  • Dynamic weights start at 1 and are updated every fifth epoch using a moving average with α = 0.5, based on a newly randomized interior-point batch.Initial and boundary points remain fixed because they are much fewer than interior points.Source: Weights computation · p. 24

Related papers

Loading 2107.00940v1…