Source-linked AI summary
CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method
Pao-Hsiung Chiu, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, Yew-Soon Ong
TL;DR
PINNs can require many collocation points and still produce inaccurate solutions when automatic-differentiation losses are under-constrained. This paper introduces can-PINNs, coupling automatic and numerical differentiation, and reports consistently accurate, efficient performance on challenging fluid-flow problems where conventional AD-based PINNs fail.
Problem
Automatic-differentiation PINNs can require dense collocation-point sampling and remain under-constrained, making training accuracy unreliable in sparse regimes.
Method
Can-PINNs couple automatic differentiation with local-support-point numerical differentiation in the loss, using upwind and central-difference instantiations for incompressible Navier–Stokes problems.
Results
Can-PINNs consistently outperformed conventional AD-based formulations across all evaluated test settings while efficiently producing accurate solutions from sparse samples.
Takeaways & Limitations
Can-PINNs provide a sample-efficient and accurate PINN formulation for the challenging fluid-dynamics problems evaluated in this study.
Takeaways & Limitations
The proposed methods have a more natural implementation for equidistant collocation points, and their effectiveness on highly irregular or statistical sampling requires further study.
Abstract
from arXiv · showhide
In this study, novel physics-informed neural network (PINN) methods for coupling neighboring support points and their derivative terms which are obtained by automatic differentiation (AD), are proposed to allow efficient training with improved accuracy. The computation of differential operators required for PINNs loss evaluation at collocation points are conventionally obtained via AD. Although AD has the advantage of being able to compute the exact gradients at any point, such PINNs can only achieve high accuracies with large numbers of collocation points, otherwise they are prone to optimizing towards unphysical solution. To make PINN training fast, the dual ideas of using numerical differentiation (ND)-inspired method and coupling it with AD are employed to define the loss function. The ND-based formulation for training loss can strongly link neighboring collocation points to enable efficient training in sparse sample regimes, but its accuracy is restricted by the interpolation scheme. The proposed coupled-automatic-numerical differentiation framework, labeled as can-PINN, unifies the advantages of AD and ND, providing more robust and efficient training than AD-based PINNs, while further improving accuracy by up to 1-2 orders of magnitude relative to ND-based PINNs. For a proof-of-concept demonstration of this can-scheme to fluid dynamic problems, two numerical-inspired instantiations of can-PINN schemes for the convection and pressure gradient terms were derived to solve the incompressible Navier-Stokes (N-S) equations. The superior performance of can-PINNs is demonstrated on several challenging problems, including the flow mixing phenomena, lid driven flow in a cavity, and channel flow over a backward facing step. The results reveal that for challenging problems like these, can-PINNs can consistently achieve very good accuracy whereas conventional AD-based PINNs fail.
1. Introduction
PINNs incorporate governing physical laws into neural-network training but remain difficult and computationally demanding to train accurately, particularly with sparse collocation points. The study proposes can-PINNs, which couple automatic and numerical differentiation to combine their respective advantages and evaluates the method on challenging forward and inverse problems.
- PINN foundations: PINNs model physical-system dynamics with deep neural networks while incorporating governing laws into the training objective.The network represents u(x, t; w) over space and time, with parameters optimized under physical constraints.
- Motivation: Accurate PINN training remains challenging and computationally demanding, despite efforts involving architectures, loss calibration, sinusoidal spaces, and importance sampling.The introduction specifically notes that training accurate PINN models remains a challenge and generally requires many collocation points.
- Limitations of existing methods: AD-based PINNs can achieve accuracy only with huge numbers of collocation points, while low optimized training losses may still accompany solutions far from the true solution.The study refers to these models as a-PINNs and reports that insufficient collocation points can decouple training-loss optimization from accuracy.
- Proposed method and evaluation: The proposed can-PINNs couple automatic and numerical differentiation to inherit the advantages of a-PINNs and n-PINNs and target accurate, efficient training with minimal collocation points.The method is tested on flow mixing, lid-driven cavity flow, backward-facing-step channel flow, and an inverse problem inferring Reynolds number from sparse observations.
2. Methodology
The methodology formulates PINNs by minimizing weighted data, PDE, initial-condition, and boundary-condition losses evaluated at collocation points. It addresses sparse-sample failures of automatic differentiation with numerical-differentiation coupling, then augments numerical derivatives using AD-derived gradients in can-PINNs.
- PINN formulation: A fully connected DNN represents the solution u(x,t; w), whose parameters are optimized to satisfy the governing differential equation and associated conditions.The formulation includes temporal and spatial differential operators, initial conditions, and boundary conditions.
- PINN formulation: The training objective combines data, PDE, initial-condition, and boundary-condition losses with relative weights λ that control their trade-off.The data-loss term is omitted for forward differential-equation problems, while appropriate scaling can significantly speed convergence.
- Limitations of AD-based PINNs: In sparse collocation regimes, over-parameterized AD-based PINNs can achieve near-zero training loss while producing inaccurate or unphysical solutions.The issue arises because AD exactly enforces differential constraints only at the sampled collocation points, leaving the optimization under-constrained.
- Numerical-differentiation PINNs: ND-PINNs replace AD-computed differential operators with derivative approximations built from local support points, coupling neighboring evaluations during training.The framework uses conventional schemes including first-order upwind, second-order upwind, and second-order central difference formulations.
- Coupled AD-ND PINNs: Can-PINNs combine network outputs with AD-derived first-order gradients to improve the gradient-matching behavior of ND-based approximations.The framework can also extend to other Taylor-series-based schemes with varying forms and accuracy.
3. Experimental study
The experimental study compares a-PINNs, n-PINNs, and can-PINNs under identical network architectures and training settings, while also reporting training cost. The PINN configurations use specified hidden-layer node counts, sinusoidal input mapping, and sine activations in subsequent hidden layers.
- Experimental design: Each test problem compares a-PINNs, n-PINNs, and can-PINNs using identical network architectures and training settings, with training cost also reported.These comparisons are summarized in the study’s experimental configuration table.
- PINN architecture: The architecture notation lists the number of nodes in each hidden layer between the input and output variables.For example, (𝑥)–64–20–20–20–(𝑢ො) denotes four hidden layers with 64, 20, 20, and 20 nodes.
- PINN architecture: The first hidden layer incorporates sinusoidal mapping, with weights initialized from 𝒩(0, σ²) using σ=1, while subsequent hidden layers use sine activation.The passage specifies the sinusoidal mapping and initialization scheme as part of the PINN configuration.
3.1. Validation on ODE
On two 1D ODEs with prescribed cosine source terms and zero boundary conditions, can-PINN was evaluated against automatic- and numerical-differentiation PINNs under converged training. With 41 collocation points, a-PINN failed despite a very low training loss, whereas n-PINNs and can-PINNs achieved substantially more accurate solutions, with can-PINN solutions closely matching ground truth.
- Experimental setup: The study solved two ODEs on x=(0,2π), using f₁=cos(x) and f₂=cos(x)+2cos(2x), whose ground truths are u=sin(x) and u=sin(x)+sin(2x).Both problems used u(0)=u(2π)=0, batch size 6, and 1e5 training iterations.
- Accuracy comparison: n-PINNs and can-PINNs approximated the true solutions with accuracy 1 to 3 orders better than a-PINN.This result was reported for the 41-collocation-point setting.
- Interpretation: The proposed schemes were attributed to better dispersion behavior and smaller leading error coefficients.These properties were presented as explanations for their improved ODE performance.
3.2. Flow mixing
The flow-mixing study evaluates a-PINN, n-PINN, and two can-PINN schemes on a transient two-dimensional mixing problem. Using identical settings across 50 runs, it compares percentile solutions and error distributions against the ground truth.
- Flow mixing: The models use the governing equation’s conservative form, with spatial derivatives approximated by n-PINN and can-PINN schemes and temporal derivatives obtained by AD.The spatial approximations include a second-order upwind n-PINN, can(uw2)-PINN, and can(cd)-PINN.
- Flow mixing: Figure 6 compares ground-truth and predicted solutions at the 10th, 50th, and 90th percentiles of ascending MSE from 50 independent runs.The visual comparison includes a-PINN, n-PINN, and can(uw2)-PINN.
- Flow mixing: Figure 7 reports boxplots of training loss and MSE between the three PINN families and the ground-truth solution for the flow-mixing problem.The results summarize 50 independent runs and include a-PINNs, n-PINNs, and can-PINNs.
- Flow mixing: The comparison trains a-PINN, n-PINN, can(uw2)-PINN, and can(cd)-PINN with 65,025 collocation points under identical architecture and training settings.Each model is evaluated over 50 independent runs.
3.3. Lid-driven cavity
For the lid-driven cavity at Re=400, can-PINN combines upwind-based can(uw2) convection with can(cd) pressure-gradient treatment and outperforms a-PINN and n-PINN, especially with sparse collocation points. Increasing sampling improves all models but requires longer, more difficult training, while can-PINN achieves MSE below 1e-5 under the densest setting.
- Problem setup: The benchmark is a unit-square cavity with top-wall velocity u_lid=1, no-slip remaining walls, and Re=400.The solution domain is x∈[0,1], y∈[-1,0], and ground truth comes from an in-house IDFC numerical solver.
- Method configuration: can-PINN uses upwind-based can(uw2) for convection and can(cd) for pressure-gradient terms, whereas n-PINN uses second-order upwind and central differences.All models share the same PINN architecture and training setting in the reported comparison.
- Sparse sampling: With 2,601 collocation points, can-PINN agrees best with ground truth, n-PINN also performs well, and a-PINN struggles to produce a reasonable solution.The comparison uses 50 independent runs and reports median-MSE solutions for the velocity-magnitude contours and absolute errors.
- Sparse sampling: a-PINN solutions have MSEs >1e-2, more than 1 order of magnitude above n-PINN and about 2 orders above can-PINN.Despite the lowest training loss, a-PINN produces consistently poor solutions in this sparse-collocation experiment.
- Velocity-profile validation: The can-PINN velocity profiles show good agreement with both the in-house simulation and Ghia et al. benchmark, while n-PINN deviates slightly more and a-PINN performs poorly.Profiles are evaluated along u(0.5,y) and v(x,-0.5) using the median solution.
- Sampling and efficiency: With 40,401 collocation points, can-PINN solutions improve to an MSE below 1e-5, while higher sampling resolution generally requires longer training and more hyper-parameter optimization.Random uniform sampling also uses Δx=0.01 as a satisfactory spacing for this problem.
3.4. Backward-facing step
The backward-facing step experiments evaluate a-PINN, n-PINN, and can-PINNs on increasingly challenging forward and inverse incompressible-flow problems. At Re=200, low training loss did not ensure an accurate a-PINN solution, while the can-PINN framework was further tested at Re=400 and for Reynolds-number inference from sparse observations.
- Problem setup: The backward-facing step models flow through a 20-by-1 channel with a fully developed parabolic inlet profile and a primary vortex downstream of the step.A secondary eddy appears near the upper mid-region as Re increases.
- Comparison between a-PINN, n-PINN, and can-PINNs results: At Re=200, a-PINN, n-PINN, and two can-PINN variants were compared using 16,000 collocation points and 25 independent runs.The comparison examined velocity-magnitude contours and absolute deviations from the simulated ground truth.
- Comparison between a-PINN, n-PINN, and can-PINNs results: At Re=200, all models had median training losses below 1e-6, but a-PINN’s lowest training-loss distribution did not correspond to an accurate solution.The passage states that a-PINN solutions were poorly solved and differed substantially from the ground truth despite their low training loss.
- Higher-Reynolds-number forward problem: At Re=400, can(uw2-conv, cd-p)-PINN was trained with 32,000 collocation points for 2 million iterations using 1000 mini-batch samples, then evaluated on a 1600×80 grid.The solution and its error were compared with IDFC simulation ground truth through velocity contours and streamlines.
- Inverse modelling: The inverse experiment applied can(uw2-conv, cd-p)-PINN to infer unknown Re values of 200, 400, and 800 while solving the full domain from separate observation sets.The models used 16,000 collocation points and tested limited observations, including n=30.
3.5. Complex channel
The study evaluates can-PINN on a 2D incompressible Navier–Stokes flow through an irregular channel geometry at Re=400. With sparse geometric sampling, a-PINN produces implausible solutions despite low training loss, whereas can-PINN achieves good solution MSE.
- 3.5. Complex channel: The test problem is steady-state, two-dimensional incompressible Navier–Stokes flow through a 2-unit-long channel whose middle section has a Singapore-island shape, at Re=400.The inlet and outlet widths are 0.3 unit, with no-slip top and bottom walls and a freestream outlet condition.
- 3.5. Complex channel: The velocity contours and absolute errors are compared with IDFC simulation for can(uw2-conv, cd-p)-PINN at Re=400, using the medium solution from 25 independent runs.The ground truth is generated with the same in-house numerical solver described earlier.
- 3.5. Complex channel: The a-PINN, n-PINN, and can-PINN models use 674 boundary points and 2,763 equidistant interior collocation points to represent and enforce the irregular geometry.An additional routine identifies whether sampled points lie inside, on, or outside the complex domain.
- 3.5. Complex channel: At this training-sample density, a-PINN fails to produce a plausible solution despite attaining the lowest training loss.The failure appears as a large discrepancy from the expected solution.
- 3.5. Complex channel: can-PINN achieves good solution MSE despite the complex, irregular domain.The comparison reports distributions of training losses and solution MSEs for a-PINN, n-PINN, and can-PINN.
4. Conclusions · Reference
The study finds that conventional AD-based PINNs can become under-constrained without sufficient sampling, whereas ND-based formulations improve sample efficiency. The proposed can-PINN combines AD with local-support values, achieving robust accuracy and efficiency across fluid-dynamics tests while motivating extensions to broader numerical schemes and sampling settings.
- 4. Conclusions: AD-formulated PINN losses can become under-constrained, making training accuracy unrelated to solution accuracy when sampling is insufficient.The study contrasts this behavior with its ND-based approach.
- 4. Conclusions: ND-formulated PINNs are more sample efficient and provide fairly good solution predictions across collocation-point regimes.This finding motivates coupling numerical differentiation with automatic differentiation.
- 4. Conclusions: CAN-PINN combines automatic differentiation with local-support-point values to approximate derivative terms, unifying advantages of AD- and ND-based formulations.The framework is built on the idea of multi-moment schemes in computational physics.
- 4. Conclusions: The resulting can-PINN improves sample efficiency and accuracy, with fluid-dynamics applications using two derived numerical-inspired schemes.The supplied conclusion identifies both efficiency and accuracy as benefits of the combined formulation.
- 4. Conclusions: Although demonstrated with upwind and central-difference schemes, can-PINN is presented as a generic framework extensible to coupled schemes with varied forms and accuracy.This extension could leverage numerical schemes developed in computational physics and scientific computing.
- 4. Conclusions: Both n-PINNs and can-PINNs apply to non-uniform sampling, but equidistant collocation points provide a more natural implementation and choice of ∆x.Further work is needed for highly irregular or statistically distributed collocation points.
- 4. Conclusions: CAN-PINN consistently delivered high efficiency and efficacy across the experimental test problems, outperforming conventional AD-based formulations in every evaluated setting.This conclusion summarizes the comparative experimental outcome.
- 4. Conclusions: By training efficiently on sparse samples while producing accurate solutions robustly, can-PINN may extend PINN methodology to more challenging problems across multiple domains.The passage frames this broader applicability as a potential enabled by the formulation.