Source-linked AI summary
PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
Zichao Long, Yiping Lu, Bin Dong
TL;DR
PDE-Net 2.0 addresses data-driven discovery of hidden PDEs when governing mechanisms are unclear and prior knowledge is limited. It combines learnable convolutional differential operators with a symbolic neural network, recovering PDE forms and supporting long-term prediction, including with noisy data. The paper reports recovery of Burgers’ equation and accurate prediction without retraining for new initial conditions.
Problem
Many complex systems lack clear governing mechanisms, motivating discovery of PDEs directly from observed dynamic data with limited prior knowledge.
Method
PDE-Net 2.0 jointly learns differential operators through constrained convolutions and nonlinear response functions through a symbolic neural network.
Results
PDE-Net 2.0 recovers Burgers’ equation with good confidence and performs accurate long-term prediction without retraining for new initial conditions.
Takeaways & Limitations
The approach has potential to uncover previously unknown PDEs from observed data while predicting dynamics in noisy environments.
Takeaways & Limitations
The current SymNet uses only addition and multiplication, while temporal discretization uses forward Euler.
Abstract
from arXiv · showhide
Partial differential equations (PDEs) are commonly derived based on empirical observations. However, recent advances of technology enable us to collect and store massive amount of data, which offers new opportunities for data-driven discovery of PDEs. In this paper, we propose a new deep neural network, called PDE-Net 2.0, to discover (time-dependent) PDEs from observed dynamic data with minor prior knowledge on the underlying mechanism that drives the dynamics. The design of PDE-Net 2.0 is based on our earlier work \cite{Long2018PDE} where the original version of PDE-Net was proposed. PDE-Net 2.0 is a combination of numerical approximation of differential operators by convolutions and a symbolic multi-layer neural network for model recovery. Comparing with existing approaches, PDE-Net 2.0 has the most flexibility and expressive power by learning both differential operators and the nonlinear response function of the underlying PDE model. Numerical experiments show that the PDE-Net 2.0 has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.
1 Introduction
PDE-Net 2.0 targets transparent recovery and prediction of hidden PDEs from dynamic data with limited prior mechanism knowledge. Its design combines learnable differential-operator approximations with symbolic modeling of nonlinear responses.
- Complex systems often lack clear governing mechanisms, so their equations are commonly obtained from empirical formulas.
- Existing SINDy-based methods may incur high memory and computation costs because they require sufficiently large dictionaries.
- The approach jointly addresses analytic model recovery and accurate prediction, including reuse under altered initial conditions.
- PDE-Net 2.0 uses a symbolic neural network to approximate nonlinear response functions, reducing reliance on prior knowledge of PDE type.
- Learnable filters and symbolic-network constraints support recovery of analytic response forms while improving flexibility and long-term prediction.
2 PDE-Net 2.0: Architecture, Constraints and Training
PDE-Net 2.0 seeks to recover unknown PDE response functions and differential operators directly from solution samples. The target network also supports long-term prediction for arbitrary initial conditions.
- The framework learns from observed solution samples rather than requiring the governing PDE to be specified explicitly.
- The observed dynamics are assumed to satisfy a PDE whose temporal derivative is an unknown function of spatial derivatives and physical quantities.
- The network is designed to reveal the analytic response function and differential operators while predicting dynamical behavior over long time horizons.
2.1 Architecture of PDE-Net 2.0
PDE-Net 2.0 discretizes PDE evolution into reusable δt-blocks that combine convolutional differential operators with a symbolic approximation of the nonlinear response. Multiple shared-parameter blocks enable long-term prediction.
- 2.1 Architecture of PDE-Net 2.0: Forward Euler temporal discretization makes the network ResNet-like, while trainable convolution filters approximate spatial finite differences.
- 2.1 Architecture of PDE-Net 2.0: A δt-block combines convolution operators for differential quantities with a symbolic network that approximates the response function F.
- 2.1 Architecture of PDE-Net 2.0: The averaging operator D00 increases expressive power and helps the network capture more complex dynamics.
- 2.1 Architecture of PDE-Net 2.0: SymNet approximates the response function as a multivariate symbolic network, with separate networks used for vector-valued components.
- 2.1 Architecture of PDE-Net 2.0: Multiple δt-blocks are stacked with shared parameters to account for accumulated error during long-term prediction.
2.2 Convolutions and Differentiations
PDE-Net 2.0 uses constrained convolution filters to approximate differential operators while retaining learnability. Moment and sum-rule conditions determine operator correspondence and approximation accuracy.
- The implementation uses deep-learning correlation conventions for convolution and periodic boundary conditions for finite-sized inputs.
- The order of a filter’s sum rules links its convolution behavior to the order of the represented differential operator.
- A filter with sum rules of order (1, 0) and total sum rules of order 3 corresponds to a second-order-accurate discretization of ∂/∂x.
- Convolution filters are constrained through moments so they can approximate prescribed differential operators with specified accuracy.
- PDE-Net 2.0 learns filters under partial moment constraints while maintaining at least second-order accuracy.
2.3 Design of SymNet: a Symbolic Neural Network
SymNet is a transparent symbolic network that approximates the PDE’s nonlinear response function while retaining polynomial expressive power and compact computation. Its multiplicative hidden layers support interpretable model recovery and can reduce complexity relative to SINDy for compact representations.
- Motivation: SymNet approximates the multivariate nonlinear response function F while allowing its analytic form to be inferred after training.The design prioritizes both expressive power and transparency.
- Architecture: Each SymNet hidden layer directly uses preceding outputs and adds one multiplicative variable, distinguishing its construction from EQL/EQL÷.The paper focuses on multiplication as the dyadic operation.
- Example: With zero biases, SymNet^2_6 represents −u u_x − v u_y, the right-hand side of the inviscid Burgers equation.This example illustrates direct recovery of a PDE expression from the network.
- Expressive power: SymNet^k_m can represent all polynomials in m variables whose total number of multiplications does not exceed k.Its actual capacity is larger than the corresponding polynomial class P_k[x_1, ..., x_m].
- Complexity: SymNet has memory load O(m + k) and evaluation cost O(k(m + k)), while compact system identification favors smaller k.The paper notes that SymNet can be worse than SINDy when k is large and l is small.
- Complexity: For k = l = 3 with 12 inputs, SINDy uses a dictionary of 455 elements and 909 flops, whereas SymNet^3_12 uses 15 memory units and 180 flops.The example approximates P = −u u_x − v u_y + u_xx.
2.4 Loss Function and Regularization
PDE-Net 2.0 trains data fidelity, filter regularization, and symbolic sparsity jointly, while pseudo-upwind selects filter orientations to improve inference stability without requiring the PDE type in advance.
- Loss function: The loss-function hyperparameters are λ1 = 0.001 and λ2 = 0.005.These values are specified for the proposed PDE-Net 2.0 training loss.
- Data approximation: Training uses data pairs whose initial state U_j(t_0, ·) is the input and later state U_j(t_i, ·) is the label.Long solution paths can be split into multiple shorter trajectories and δt-blocks.
- Regularization: The moment loss regularizes learnable convolution filters to reduce overfitting, with s = 0.01 in the numerical experiments.The filters are represented through q_ij and their moment matrix M(q).
- Regularization: The SymNet regularization term enforces parameter sparsity, helping suppress unnecessary terms and enabling more stable prediction.The experiments set s = 0.001 for this sparsity-related penalty.
- Pseudo-upwind: Pseudo-upwind uses SymNet derivatives with respect to spatial derivative inputs to decide whether each convolution filter should be applied directly or flipped.For vector PDEs, corresponding response derivatives determine approximations for u_x and u_y.
2.5 Initialization and training
PDE-Net 2.0 organizes parameters into filter, SymNet, and hyperparameter groups, then trains progressively across δt-blocks. Learnable filters are retained because fixed finite-difference approximations cannot generally be selected without knowing the PDE form.
- Parameter initialization: PDE-Net 2.0 parameters comprise moment matrices for convolution kernels, SymNet parameters, and hyperparameters such as filter counts and regularization weights.SymNet parameters are shared across the computational domain and initialized from a Gaussian distribution.
- Parameter initialization: Filters are initialized with a second-order pseudo-upwind scheme for D_01 and D_10, and central differences for the remaining filters.This initialization is part of the training setup before filters are learned.
- Training: Layer-wise training starts with one δt-block and progressively adds blocks, reusing the previous results as initialization while sharing parameters across layers.A warm-up stage first fixes filters and sets λ1 = λ2 = 0 to obtain an initial SymNet estimate.
- Learnable filters: Frozen-PDE-Net 2.0 fixes filters at their initial values, whereas regular PDE-Net 2.0 learns them jointly with the PDE model.The paper compares the two variants to assess the role of learnable filters.
3 Numerical Studies: Burgers’ Equation
The Burgers’ equation experiments test PDE-Net 2.0 on noisy simulated data, assessing PDE recovery, long-term prediction, learnable filters, sparsity, and pseudo-upwind.
- PDE model identification: PDE-Net 2.0 recovers the terms of Burgers’ equation with good accuracy, while nonphysical terms receive relatively small SymNet weights.Learnable filters improve identification of the PDE model.
- Prediction: PDE-Net 2.0 performs long-term prediction despite noisy training data, and stacking multiple δt-blocks improves prediction accuracy.The evaluation uses 1000 randomly generated initial guesses and compares predicted dynamics with high-precision numerical solutions.
- Prediction: PDE-Net 2.0 performs significantly better than Frozen-PDE-Net 2.0, indicating that unknown PDEs require learning their discretization rather than freezing finite-difference filters.The comparison uses 5 × 5 filters and prediction times through (0, 4].
- Sparsity and pseudo-upwind: Sparsity constraints suppress weights on absent Burgers’ terms, while sparsity and pseudo-upwind both significantly reduce prediction errors.The effects are examined through remainder coefficients and prediction-error plots.
4 Numerical Studies: Diffusion Equation
The diffusion-equation experiments assess PDE identification and prediction on simulated heat-equation data, including comparisons between learnable and frozen filters.
- Experimental setup: The heat-equation data use periodic boundaries, coefficient c = 0.1, second-order Runge-Kutta time integration, central spatial differences, and restriction from a 128 × 128 to 32 × 32 mesh.The time step is δt = 1/1600, and initial values are generated from the stated random-field construction.
- PDE model identification: PDE-Net 2.0 recovers the terms of the heat equation with good accuracy, while excluded terms have much smaller SymNet weights.The identification results are reported in Table 2.
- Prediction: PDE-Net 2.0 shows an advantage over Frozen-PDE-Net 2.0 in diffusion prediction-error comparisons, demonstrating the value of learning the filters.Predicted dynamics are also visualized for both models using 9 δt-blocks.
5 Numerical Studies: Convection Diffusion Equation with A Reactive Source
In a reactive convection-diffusion experiment, PDE-Net 2.0 recovered the governing terms with good accuracy and predicted dynamics more accurately than its frozen-filter counterpart.
- The experiment models a reactive convection-diffusion system describing material transfer under diffusion and convection with chemically induced production.
- Training used noisy data generated with second-order Runge-Kutta time integration, second-order upwinding for convection, central differences for diffusion, and mesh restriction from 128 × 128 to 32 × 32.
- Results and discussions: PDE-Net 2.0 recovered the reaction convection-diffusion equation’s terms with good accuracy, while excluding irrelevant terms through relatively small SymNet weights.
- Results and discussions: PDE-Net 2.0 produced more accurate predictions than Frozen-PDE-Net 2.0 because learning discretizations improved nonlinear convection coefficient estimates.
- Results and discussions: Figure 12 reports relative prediction errors over times (0, 2] across 1000 test samples, with banded curves spanning the 25%-100% percentiles.
6 Conclusions and Future Work
PDE-Net 2.0 recovers analytic PDE forms from observed dynamics with minor assumptions and supports long-term prediction for new initial conditions. Reaction convection diffusion figures compare true, predicted, and error-map dynamics for Frozen-PDE-Net 2.0 and PDE-Net 2.0.
- PDE-Net 2.0 recovers the analytic form of PDE models from observed dynamic data with minor assumptions about the underlying mechanisms.
- Figure 13 compares true reaction convection diffusion dynamics with predictions from Frozen-PDE-Net 2.0 and PDE-Net 2.0 using 24 δt-blocks at δt = 0.01.
- Figure 14 presents reaction convection diffusion error maps for Frozen-PDE-Net 2.0 and PDE-Net 2.0 with 24 δt-blocks and δt = 0.01.
- Burgers’ equation was recovered with good confidence without prior knowledge of the equation type.
- After training, PDE-Net 2.0 can make accurate long-term predictions for new initial conditions without retraining.