PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Rafael Bischof, Michael Kraus

arXiv:2110.09813v2cs.LGcs.AI

TL;DR

PINN training must balance multiple competing loss terms whose gradients can become pathological. The paper compares existing balancing schemes and proposes ReLoBRaLo, evaluating them on forward and inverse PDE benchmarks. ReLoBRaLo consistently outperforms baseline scaling methods in accuracy while adding less computational overhead.

  • Problem

    Imbalanced loss terms can produce gradient pathologies that impede PINN training and cause imbalanced solutions.

  • Method

    The paper compares Learning Rate Annealing, GradNorm, and SoftAdapt, then proposes ReLoBRaLo, a self-adaptive loss-balancing scheme using relative improvements and random lookbacks.

  • Results

    ReLoBRaLo consistently outperforms baseline scaling methods in accuracy while inducing significantly less computational overhead across the evaluated PDE problems.

  • Takeaways & Limitations

    Adaptive balancing of PINN loss terms improves approximation capabilities across forward and inverse benchmark problems.

Abstract

from arXiv · show

Physics-Informed Neural Networks (PINN) are algorithms from deep learning leveraging physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms into their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINNs loss function and their gradients. After reviewing of three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named \emph{ReLoBRaLo} (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation and Helmholtz's equation. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy, while also inducing significantly less computational overhead.

I. Introduction

PINNs address low-data PDE problems by embedding physical laws into differentiable neural-network training, but their multiple loss terms can produce gradient pathologies. This paper compares adaptive balancing methods and introduces ReLoBRaLo, evaluated on forward and inverse benchmark problems.

  • PINNs incorporate physical laws as implicit priors and can approximate PDEs with little to no data.
  • Imbalanced loss terms can create gradient pathologies, impede training, and produce imbalanced solutions.Different physical terms may have substantially different measurement units and magnitudes.
  • Manual tuning of term-specific scaling factors requires laborious grid search and becomes intractable as the number of terms grows.
  • The paper compares Learning Rate Annealing, GradNorm, and SoftAdapt, and proposes ReLoBRaLo as a self-adaptive loss-balancing scheme.
  • The methods are evaluated on Burgers, Kirchhoff, and Helmholtz PDEs in both forward and inverse settings.

III. Physics-Informed Neural Networks (PINNs)

PINNs use a fully connected neural network to approximate a PDE solution, compute derivatives by automatic differentiation, and jointly optimize losses enforcing equations, conditions, and measurements.

  • A fully connected feed-forward network takes space and time coordinates as inputs and approximates the PDE solution.
  • Automatic differentiation computes input derivatives that enter residuals of the governing equations.
  • The governing-equation loss penalizes PDE residuals in both forward and inverse problems.
  • Boundary-condition and initial-condition losses are included only in the forward problem.
  • Measurement loss makes the network approximate observed measurements and is included in both forward and inverse problems.

IV. Multi-Objective Optimisation

Multi-objective optimization jointly handles potentially conflicting objectives, while linear scalarization combines them using loss weights. In neural-network PINNs, dynamic scaling is needed because non-convex optimization and manual weight selection complicate training.

  • Multi-objective optimization simultaneously optimizes k > 1 potentially conflicting objectives.
  • Linear scalarization turns multiple objectives into a single objective through weighted combination.
  • Multi-objective problems generally require trade-offs to satisfy all objectives to a certain degree.
  • Neural-network loss spaces are highly non-convex, so gradient-based optimization is not guaranteed to find the globally optimal solution.
  • Dynamic automated selection of loss scalings is motivated by the difficulty of manually finding optimal weights as the number of objectives increases.

A. Learning Rate Annealing

Learning Rate Annealing adaptively rescales PINN losses using gradient statistics and exponential smoothing to reduce gradient imbalance. Its unbounded scaling and extra tuning or computation can introduce instability and cost.

  • Learning Rate Annealing uses gradient statistics to adaptively scale loss terms and reduce gradient imbalance.The method addresses gradient stiffness caused by imbalance among different loss terms.
  • When PDE gradients greatly exceed boundary or initial-condition gradients, the method adjusts scalings so gradient magnitudes become similar.
  • Exponential decay smooths balancing and avoids drastic changes in loss space between optimization steps.
  • Unbounded scaling can change terms by several orders of magnitude, and up-scaling may repeatedly overshoot the objective.
  • Scaling every term to equal magnitude can favor objectives that are easiest to optimize.
  • GradNorm requires a separate backward pass for each task and two optimization rounds per step, becoming prohibitively expensive as k grows.

C. SoftAdapt

The paper situates ReLoBRaLo among loss-balancing methods that use relative loss progress, exponential history, softmax normalization, and random lookbacks to adaptively scale PINN objectives.

  • SoftAdapt: SoftAdapt balances loss terms using relative progress between adjacent iterations, differences rather than divisions, and softmax-normalized scalings.
  • ReLoBRaLo: ReLoBRaLo uses loss statistics rather than gradient statistics, with a softmax that keeps the sum of scalings bounded.
  • ReLoBRaLo: Its relative progress divides each current loss Li(t) by the corresponding previous loss Li(t −1).
  • ReLoBRaLo: Exponential decay incorporates loss statistics from multiple past steps instead of only one previous step.
  • ReLoBRaLo: Random lookback ρ either retains previous scaling history or compares progress with the initial loss L(0).
  • ReLoBRaLo: The method combines Pareto-oriented updates, gradient-statistics avoidance, smoothed changes, and tunable memory through α.

VII. Hyperparameter Tuning and Meta Learning

Hyperparameters for network architecture and training are tuned with grid search combined with Bayesian Optimisation, followed by local fine-grained search.

  • Hyperparameter search: The study tunes hidden-layer depth dK, layer width wK, learning rate lr, exponential decay rate α, and saudade ρ.
  • Bayesian Optimisation: Bayesian Optimisation fits prior Gaussian Processes to evaluated hyperparameter configurations and samples 80 further points using Expected Improvement.
  • Hyperparameter search: Fine-grained grid search around the Bayesian-Optimisation output provides the final hyperparameter refinement.
  • Hyperparameter search: Table I lists the architecture and training hyperparameters together with their ranges for Bayesian Optimisation.
  • Hyperparameter search: The same Bayesian Optimisation configuration is used for all examples, with corresponding grid-search and Bayesian-Optimisation results reported in section VIII.

VIII. Results

The evaluation compares adaptive loss-balancing schemes on Burgers, Kirchhoff, and Helmholtz problems in both forward and inverse settings, using networks of varying depth and width.

  • Evaluation setup: The study evaluates balancing schemes on Burgers equation, Kirchhoff plate bending, and Helmholtz equation problems.
  • Evaluation setup: The comparison covers both forward and inverse problem settings and considers performance alongside computational efficiency.
  • Evaluation setup: Training uses networks of varying depth and width and is limited to 105 gradient-descent steps with Adam optimization.

A. Burgers’ Equation

Burgers’ equation is evaluated in both forward and inverse PINN settings, with ReLoBRaLo balancing loss contributions during training. The method achieves low forward error, adapts scaling according to relative loss progress, and reduces computational overhead compared with gradient-based alternatives.

  • Burgers’ equation models phenomena including shock waves, gas dynamics, and traffic flow.
  • Forward problem: In the forward problem, the PDE parameter is fixed and a neural network is trained to approximate the latent solution using a multi-term MSE loss.The training loss weakly enforces approximation of the PDE solution.
  • Forward problem: Below 5% relative maximum error was obtained for the ReLoBRaLo PINN solution relative to a finite-element reference.The comparison uses squared differences between the PINN and FEM solutions because no analytical solution is available.
  • Inverse problem: In the inverse problem, viscosity ν is treated as an unknown trainable PDE parameter estimated from noiseless FEM-derived observations.The additional PDE variable is updated through gradient descent alongside network weights and biases.
  • ReLoBRaLo behavior: ReLoBRaLo assigns higher scaling to relatively smaller loss contributions and lower scaling to larger contributions such as the PDE and initial-condition terms.For Burgers’ equation, boundary-condition terms receive higher scaling while PDE and initial-condition terms receive lower scaling.
  • Efficiency and comparison: ReLoBRaLo adapts scalings every iteration with one backward pass, making Burgers’ training about 40% faster than Learning Rate Annealing and 70% faster than GradNorm.Updating alternative methods less frequently can improve efficiency but introduces a sensitive flexibility–efficiency trade-off.

B. Kirchhoff Plate Bending Equation

The Kirchhoff plate bending benchmark models a fourth-order plate PDE with eight boundary conditions, creating nine competing PINN loss terms. ReLoBRaLo achieves high accuracy and reveals where training remains constrained.

  • Problem setup: The Kirchhoff–Love model represents thin-plate stresses and deformations through a two-dimensional fourth-order PDE.It uses a midsurface representation and linear elastic material assumptions.
  • Problem setup: Eight boundary conditions produce nine terms in the PINNs loss, making balanced optimization challenging.The conditions constrain both displacements and bending moments.
  • Results: The PINN solution has a negligible maximum squared error relative to the analytical reference solution.The reported visualization compares the analytical solution, prediction, and squared error.
  • Training behavior: Increasing moment-condition weights while displacement-condition weights decrease indicates that the moments limit training progress.The paper suggests using an activation function with better second-derivative behavior than tanh.
  • Results: ReLoBRaLo outperforms the baseline and other algorithms by almost an order of magnitude in accuracy on Kirchhoff’s equation.The comparison includes a challenging nine-term loss and reports consistency across training runs.
  • Results: ReLoBRaLo provides up to sixfold speedup because it balances losses without gradient statistics or separate backward passes for each term.The efficiency benefit grows as the number of loss terms increases.

C. Helmholtz equation

The Helmholtz benchmark evaluates PINNs on a time-independent wave equation with four boundary-condition terms. ReLoBRaLo improves the approximation while exposing a limitation of basic loss balancing and motivating random lookback.

  • Problem setup: The Helmholtz equation is a time-independent wave equation used in physical and engineering problems including acoustics and electromagnetism.The benchmark uses a wave number k and an analytically available solution with Dirichlet boundaries.
  • Problem setup: Four boundary conditions yield a five-term physics-informed loss for the forward Helmholtz problem.The input variables are bounded on [−1, 1]^2.
  • Results: The PINN prediction has negligible maximum error relative to the analytical Helmholtz solution after successful convergence.The comparison reports the analytical solution, prediction, and squared difference in u.
  • Balancing behavior: GradNorm and Learning Rate Annealing outperform ReLoBRaLo with E[ρ] = 1 for boundary-condition L2 accuracy, while ReLoBRaLo better reduces the governing-equation error.The differing behavior is associated with the initial magnitude disparity between governing and boundary terms.
  • Balancing behavior: The Helmholtz case reveals a limitation of basic loss balancing and motivates introducing random lookback.The basic approach eventually forgets earlier deterioration and focuses on the dominant governing equation.

IX. Ablation and Sensitivity Study

The ablation study examines temperature, exponential decay, and expected random lookback in ReLoBRaLo. Results show that aggressive balancing and task-dependent lookback frequencies can materially affect accuracy.

  • Study design: The study varies temperature T, exponential decay rate α, and expected saudade E[ρ] to assess ReLoBRaLo sensitivity.The analysis uses ablation and sensitivity experiments alongside benchmark validation losses.
  • Temperature and decay: Larger α makes the network remember longer, while T controls how strongly the scaling factors separate.Small T values produce more aggressive balancing for Helmholtz’s equation.
  • Temperature and decay: Setting α = 1 always causes deterioration of model performance in the reported sensitivity study.Exponential relaxation therefore creates a trade-off between memory and rapid adaptation.
  • Random lookback: Helmholtz performs best at E[ρ] = 0.99, whereas Burgers and Kirchhoff require only an expected lookback every 10’000 optimization steps.The preferred lookback frequency is therefore problem-dependent.
  • Random lookback: Adding random lookback improves Helmholtz accuracy by more than an order of magnitude and has a lesser but significant effect on Burgers and Kirchhoff.The stochastic scaling factors become less interpretable while increasing boundary-condition weights.

X. Synopsis and Outlook

The paper addresses competing PINN loss terms by proposing ReLoBRaLo, which combines adaptive scaling with occasional historical lookback. Across forward and inverse PDE benchmarks, it reports improved accuracy, lower computational cost, and interpretable scaling diagnostics.

  • Synopsis: Competing physics-loss terms can spoil PINN training because their objectives may conflict.The synopsis frames adaptive loss balancing as a multi-objective optimization problem.
  • Synopsis: ReLoBRaLo combines attributes of existing approaches and adds a saudade parameter ρ for occasional incorporation of historical loss contributions.The method selects bespoke weights for multiple PINN loss terms.
  • Results: ReLoBRaLo consistently outperforms the baseline methods in accuracy across forward and inverse Burgers, Kirchhoff, and Helmholtz problems.The compared methods are GradNorm, Learning Rate Annealing, and SoftAdapt.
  • Results: ReLoBRaLo is up to six times more computationally efficient than the existing scaling methods.The comparison is expressed in training epochs or wall-clock time.
  • Implications: Inspecting adaptive scalings can reveal weak points in PINN training and support informed framework improvements.The paper identifies this diagnostic use as an additional merit of the method.
  • Outlook: Future work will test robustness and scalability on further PDE classes and address the high costs of estimating neural-network PDE solutions.Suggested extensions include Sobolev Training and Mixture-of-Experts.
Loading 2110.09813v2…