Source-linked AI summary
Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
Ameya D. Jagtap, George Em Karniadakis
TL;DR
Neural networks and PINNs need effective activation choices for approximating nonlinear functions and PDE solutions across smooth, discontinuous, and high-gradient settings. The paper introduces a trainable, scalable activation hyper-parameter and evaluates it on forward and inverse problems. Adaptive activations generally capture solution frequencies faster and improve convergence and accuracy, although large scaling factors can cause oscillations or divergence.
Problem
Activation-function choice affects neural-network training, but no obvious choice works for every approximation or physics-informed problem.
Method
The paper adds a trainable, scalable hyper-parameter to activation functions and applies the resulting networks to deep-learning and PINN forward and inverse problems.
Results
Adaptive activations improve convergence and accuracy over fixed activations across tested function and PDE problems, with especially faster early training and faster frequency capture.
Takeaways & Limitations
Adaptive activation functions offer a simple approach for improving neural-network approximation efficiency, robustness, and accuracy, especially for forward problems.
Takeaways & Limitations
Large scaling factors accelerate convergence but can increase hyper-parameter sensitivity and cause loss oscillations or eventual divergence.
Abstract
from arXiv · showhide
We employ adaptive activation functions for regression in deep and physics-informed neural networks (PINNs) to approximate smooth and discontinuous functions as well as solutions of linear and nonlinear partial differential equations. In particular, we solve the nonlinear Klein-Gordon equation, which has smooth solutions, the nonlinear Burgers equation, which can admit high gradient solutions, and the Helmholtz equation. We introduce a scalable hyper-parameter in the activation function, which can be optimized to achieve best performance of the network as it changes dynamically the topology of the loss function involved in the optimization process. The adaptive activation function has better learning capabilities than the traditional one (fixed activation) as it improves greatly the convergence rate, especially at early training, as well as the solution accuracy. To better understand the learning process, we plot the neural network solution in the frequency domain to examine how the network captures successively different frequency bands present in the solution. We consider both forward problems, where the approximate solutions are obtained, as well as inverse problems, where parameters involved in the governing equation are identified. Our simulation results show that the proposed method is a very simple and effective approach to increase the efficiency, robustness and accuracy of the neural network approximation of nonlinear functions as well as solutions of partial differential equations, especially for forward problems.
1. Introduction
The paper motivates adaptive activation functions for neural-network and physics-informed approximations of smooth, discontinuous, and PDE solutions. It proposes tuning an activation hyper-parameter and reports faster convergence and improved accuracy, especially early in training.
- Neural networks provide approximation spaces for numerical solutions of PDEs, integro-differential equations, and dynamical systems.
- PINNs incorporate structured prior information about governing equations into neural-network solution inference and system identification.Prior work extended physics-informed learning from linear to nonlinear problems.
- The activation function affects training because loss gradients depend on activation-function derivatives, yet no single activation choice is suitable for every problem.
- The paper tunes activation functions with an adaptable hyper-parameter across hidden layers and evaluates smooth, steep-gradient, forward, and inverse problems.The studied equations include Burgers, Klein-Gordon, and Helmholtz equations.
- The proposed adaptive activation function improves accuracy and convergence rate over fixed activations, particularly during early training.
2. Methodology
The methodology combines neural-network representations with physics-informed residual and boundary or initial-condition losses, optimized using gradient-based methods. An additional scalable hyper-parameter adapts activation-function slopes during training.
- 2. Methodology: A deep network composes affine layer transformations with nonlinear activations, while the output layer uses the identity function.Trainable parameters include layer weights and biases.
- 2. Methodology: Training data can come from exact or high-resolution numerical solutions and can be sampled to reflect small spatial or temporal scales.
- 2.2. Loss function and optimization algorithm: PINN optimization minimizes a loss combining governing-equation residuals at domain points with known boundary or initial conditions.The residual enforces the governing equation, while the second term enforces boundary or initial data.
- 2.2. Loss function and optimization algorithm: Gradient-descent methods iteratively minimize the loss, and the study uses the ADAM optimizer, a stochastic-gradient-descent variant.
- 2.3. Adaptive activation functions: The adaptive method adds a trainable hyper-parameter a to the activation function alongside weights and biases.The parameter is optimized jointly with the network parameters.
- 2.3. Adaptive activation functions: The hyper-parameter changes activation-function slope, while a scaling factor is introduced to accelerate convergence without changing the loss-function structure.Sigmoid, tanh, ReLU, and Leaky-ReLU are shown for varying a values.
- 2.3. Adaptive activation functions: The adaptive PINN procedure constructs a network and governing-equation residual, forms the loss, and optimizes the resulting parameters.
3. Results and discussion
The study compares fixed and adaptive activation functions for neural-network approximation of smooth, discontinuous, and Burgers-equation solutions. Adaptive activations generally accelerate convergence, improve accuracy, and capture frequency content more effectively, although large scaling factors can increase sensitivity.
- Adaptive activations are evaluated for smooth and discontinuous function approximation and PINN forward and inverse problems involving Burgers, Klein-Gordon, and Helmholtz equations.
- Neural network approximation of nonlinear smooth and discontinuous functions: Adaptive activation captures all frequencies in 22000 iterations for the smooth function, while fixed activation does not.The adaptive loss decreases faster, with an optimal a around 4.5.
- Neural network approximation of nonlinear smooth and discontinuous functions: Adaptive activation captures Burgers-solution frequencies faster than fixed activation, although neither captures all frequencies after 80000 iterations.The loss decreases faster with adaptive activation, and the optimal a is around 1.56.
- Neural network approximation of nonlinear smooth and discontinuous functions: Adaptive activation captures all frequencies in 28000 iterations for the discontinuous function, whereas fixed activation fails to capture them.The network captures the discontinuity, then low-frequency components for x < 0, and finally high-frequency components for x > 0.
- Burgers equation: For Burgers equation, introducing the adjustable activation parameter improves solution accuracy and makes loss convergence faster as the scaling factor increases.The relative L2 error decreases with increasing scaling factor.
- Burgers equation: Adaptive tanh activation captures correct Burgers-equation frequencies faster than fixed tanh at t = 0.5 and t = 0.75, while ReLU captures neither case's frequencies.Dominant frequencies are captured in 1000 iterations for one adaptive tanh comparison.
- Burgers equation: Increasing the scaling factor accelerates convergence but makes the parameter a more sensitive, producing oscillations in the loss and parameter values.The authors attribute this sensitivity to the SGD optimization algorithm.
F B utt −∆uNN −N(uNN)
Adaptive activation functions are evaluated on Klein-Gordon, Helmholtz, and inverse sine-Gordon PINN problems, with comparisons against fixed activations. The reported results show faster convergence, improved accuracy, and faster frequency capture in the adaptive models.
- Klein-Gordon equation: For the Klein-Gordon equation, increasing the adaptive scaling factor improves the solution, accelerates convergence, and drives a toward approximately 1.548.No difference was observed between n = 5 and n = 10, so n = 5 was plotted.
- Klein-Gordon equation: Adaptive activation captures all frequencies in the Klein-Gordon solution in approximately 400 iterations, whereas fixed activation requires more than 1500.The comparison evaluates solutions at x = −0.5 and x = 0.5.
- Helmholtz equation: For the Helmholtz equation, adaptive activation gives relative L2 error 7.1945e-2 versus 1.0591e-1 for fixed activation after 3600 iterations.The adaptive model also captures frequencies faster at x = −0.5 and x = 0.5, with an optimized a close to 3.
- Inverse problem: The inverse formulation reconstructs the solution while identifying unknown parameters λ through a physics-informed loss combining solution data and governing-equation residuals.The training data include boundary and interior points, and the network identifies parameterized linear or nonlinear terms in the governing equation.
- Inverse problem: After 3500 iterations, maximum parameter error is almost 9% with fixed activation and 1.24% with adaptive activation on clean data.The adaptive activation also produces more accurate identified PDEs than the fixed counterpart under clean data and noise.
4. Conclusions
The paper introduces a scalable hyper-parameter for adaptive activation functions in deep and physics-informed neural networks. Across smooth, discontinuous, forward, and inverse problems, the adaptive approach is reported to converge faster and achieve smaller solution errors than fixed activations.
- 4. Conclusions: A scalable activation hyper-parameter improves convergence and accuracy across deep-network and PINN approximations of smooth, discontinuous, and high-gradient problems.The evaluation includes Burgers, Klein-Gordon, and Helmholtz equations in forward and inverse settings.
- 4. Conclusions: Frequency-domain analysis shows that adaptive activation captures solution frequencies faster than fixed activation during training.The paper uses this analysis to examine the successive learning of frequency components.