PaperlayerView exact revision on arXiv

Source-linked AI summary

PDE-Net: Learning PDEs from Data

Zichao Long, Yiping Lu, Xianzhong Ma, Bin Dong

arXiv:1710.09668v2math.NAcs.LGcs.NEstat.ML

TL;DR

Many systems have unknown or only partially known governing PDEs, creating a need for data-driven models that support both discovery and prediction. PDE-Net jointly learns convolutional differential operators and nonlinear responses under filter constraints. Numerical experiments report hidden-equation recovery and relatively long-time prediction, including noisy settings, while the method’s filter size creates an accuracy–cost trade-off.

  • Problem

    Unknown or partially known governing equations in complex systems motivate methods that can discover PDEs from data rather than assume their explicit form or fixed differential approximations.

  • Method

    PDE-Net jointly learns differential operators through convolutional filters and nonlinear responses through neural networks or other machine-learning methods, using constrained filters.

  • Results

    Numerical experiments show that PDE-Net can uncover hidden equations and predict dynamical behavior for relatively long times, even with noisy data.

  • Takeaways & Limitations

    The method combines PDE identification with predictive modeling while retaining expressive and predictive power through partially constrained learnable filters.

  • Takeaways & Limitations

    Filter size limits approximation capability; larger filters improve representation but increase memory and computation costs.

Abstract

from arXiv · show

In this paper, we present an initial attempt to learn evolution PDEs from data. Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulfill two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models. The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (filters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses. Comparing with existing approaches, which either assume the form of the nonlinear response is known or fix certain finite difference approximations of differential operators, our approach has the most flexibility by learning both differential operators and the nonlinear responses. A special feature of the proposed PDE-Net is that all filters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network. These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of filters (an important concept originated from wavelet theory). We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet). Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

1 INTRODUCTION

The paper targets PDE learning from data when governing equations are unknown, seeking both accurate prediction and interpretable equation discovery with minimal prior knowledge. PDE-Net learns differential operators and nonlinear responses jointly while constraining filters for identifiability.

  • Motivation: Complex systems often have only partially known governing equations, motivating data-driven discovery of PDE models.The paper frames this as a question of learning PDEs from measurements and using the learned models for prediction.
  • Motivation: Existing approaches are limited by expensive symbolic regression, fixed differentiation approximations, or assumed PDE forms.These restrictions make extracting governing equations in a less restrictive setting challenging.
  • Objective: PDE-Net aims to predict complex-system dynamics and uncover hidden PDE models simultaneously, with minimal prior knowledge.The objective combines predictive accuracy with discovery of the governing equation.
  • Approach: The network learns nonlinear response functions with neural networks or other machine-learning methods and differential operators with convolutional filters.Both components are learned jointly rather than fixing the differential approximations in advance.
  • Results: Numerical experiments on linear and nonlinear convection-diffusion equations show equation discovery and relatively long-time prediction under Gaussian noise.The experiments use high-precision numerical data with added noise to mimic real situations.
  • Approach: Constrained learnable filters support governing-equation identification while retaining expressive and predictive power.The constraints exploit relations between differential-operator orders and filter sum-rule orders.

2 PDE-NET: A FLEXIBLE DEEP ARCHTECTURE TO LEARN PDES FROM DATA

PDE-Net represents PDE evolution using learned convolutional approximations to differential operators and a learned nonlinear response. Its filter constraints are derived from sum-rule and moment conditions, balancing approximation capability against computational cost.

  • Problem formulation: The target PDE has the form u_t(t,x,y) = F(x,y,u,u_x,u_y,u_xx,u_xy,u_yy, . . .) on a spatial domain and time interval.The framework assumes observed data are associated with a PDE of this general form.
  • PDE-Net design: PDE-Net combines automatic determination of differential operators and their discrete approximations with approximation of the nonlinear response F.These are the two main components jointly represented in the network.
  • Convolutions and differentiations: Wavelet-frame filter relations show that convolutional filters with suitable sum rules can approximate differential operators.For example, the Haar filter h10 corresponds, after scaling, to a first-order discretization of ∂/∂x.
  • Filter constraints: Moment-matrix constraints let filters approximate selected differential operators at specified approximation orders.Partial constraints preserve learnability, whereas fully constrained filters become uniquely determined frozen filters.
  • Filter constraints: Larger filters increase representation capability and can support higher-order or more accurate differential approximations, but increase memory and computation costs.The paper gives the example that 3-element filters cannot approximate a fifth-order operator whereas 7-element filters can.

2.2 ARCHITECTURE OF PDE-NET

For temporal discretization, the paper uses forward Euler, with more sophisticated schemes noted as possible alternatives that would lead to different network architectures.

  • Temporal discretization: PDE-Net uses forward Euler for temporal discretization, while other temporal schemes could produce different architectures.The paper adopts forward Euler for simplicity.

ONE δt-BLOCK:

A δt-block advances the state by combining an averaged current quantity with a time-step-scaled nonlinear response computed from convolutional operator approximations.

  • δt-block update: The predicted next state is formed as ũ(t_i+1,·) = D0u(t_i,·) + Δt · F(x,y,D00u,D10u,D01u,D20u,D11u,D02u, . . .).This is the one-step update implemented by a δt-block.
  • δt-block update: D0 and D_ij are convolution operators whose filters approximate averaging and spatial differential operators.D10, D01, D11, and related operators approximate derivatives, while D0 and D00 are averaging operators.
  • δt-block design: The averaging operators replace the identity to improve network stability and capture more complex dynamics.The paper distinguishes these averaging paths from the differential-operator approximations.

PDE-NET (MULTIPLE δt-BLOCKS):

PDE-Net stacks multiple δt-blocks with shared parameters to improve stability and support longer-term dynamical prediction.

  • A single δt-block guarantees only one-step accuracy and may accumulate errors, causing severe prediction instability.
  • Stacking multiple δt-blocks forms the deep PDE-Net and improves stability for long-term prediction.
  • The network is built by repeatedly stacking δt-blocks while sharing parameters across all blocks.
  • Larger n gives the PDE-Net longer time stability while parameter sharing reduces memory usage.

LOSS FUNCTION AND CONSTRAINTS:

The PDE-Net trains on multi-step input–label pairs with an ℓ2 loss and constrains filter moments to preserve differential-operator identifiability.

  • Training samples pair u^j(t_i, ·) with u^j(t_i+n, ·), using the latter as the target for the PDE-Net output.
  • The model is trained by minimizing the accumulated ℓ2 prediction error after n δt-blocks.
  • Filters q_0 and q_ij are constrained through their associated moment matrices.
  • For i+j>0, moments up to the required order are zero except the designated moment, which equals one.
  • Multiple filters can increase expressive power, but this paper uses one filter per differential operator for simplicity and identifiability.

NOVELTY OF THE PDE-NET:

PDE-Net jointly learns differential operators and nonlinear responses while constraining filters so the underlying PDE can remain identifiable.

  • Learnable filters avoid fixing spatial-differentiation approximations in advance, increasing flexibility and supporting robust approximation and longer-time prediction.
  • The response function F is learned from data rather than assumed known in advance.
  • Moment-matrix constraints identify included differential operators and help identify the nonlinear response function F.

2.3 INITIALIZATION AND TRAINING

PDE-Net parameters comprise filters, point-wise-network parameters, and hyper-parameters; training initializes filters conventionally before releasing their constraints and proceeds layer-wise.

  • The parameter groups are differential-operator filters, point-wise neural-network parameters for F, and architectural hyper-parameters.
  • Point-wise neural-network parameters are shared across the computation domain and initialized by Gaussian random sampling.
  • Filters are initialized by freezing moments to match their corresponding differential operators, then released under the training constraints.
  • Layer-wise training starts with one δt-block and progressively adds blocks, reusing earlier results for initialization while sharing block parameters.

2.4 RELATIONS TO SOME EXISTING NETWORKS

PDE-Net shares structural ideas with NIN and ResNet: its blocks combine convolutional and point-wise nonlinear processing, while its two-path blocks resemble residual blocks.

  • PDE-Net’s structure is similar to the Network-In-Network and deep Residual Neural Network architectures.
  • Each δt-block resembles an NIN mlpconv layer by combining convolutions with small point-wise neural networks.Unlike NIN, PDE-Net does not use pooling or ReLU operations.
  • Multiple δt-blocks resemble the NIN structure, extending the block-level analogy across the network.
  • Each δt-block has separate paths for the averaged quantity u and the increment F, matching the two-path structure of a ResNet residual block.

3 NUMERICAL STUDIES: CONVECTION-DIFFUSION EQUATIONS

Numerical studies on linear and nonlinear convection-diffusion equations evaluate PDE-Net’s prediction, equation-identification, and hyperparameter behavior under noisy data. The experiments report stable long-time prediction, improved reliability with deeper networks and larger filters, and coefficients close to the true PDE coefficients.

  • Experimental setup: The experiments generate convection-diffusion data with high-precision numerical methods and add Gaussian noise to mimic real situations.The study includes a 2-dimensional linear variable-coefficient equation and a nonlinear convection-diffusion equation with a nonlinear source.
  • Prediction experiments: PDE-Net is evaluated on 560 randomly generated initial guesses using normalized prediction errors against high-precision numerical solutions.The prediction experiments vary the number of δt-blocks and use 5 × 5 or 7 × 7 learnable filters.
  • Prediction experiments: Multiple δt-blocks improve stability, while noisy-data training still supports long-term prediction.The paper attributes the longer-horizon behavior to stacking multiple δt-blocks and illustrates it with long-time predictions.
  • Prediction experiments: PDE-Net performs significantly better than Frozen-PDE-Net, especially with 7 × 7 filters.The comparison uses prediction-error curves over time for both filter sizes.
  • Equation identification: Learned coefficients are close to the true coefficients, with noise-related oscillations; multiple δt-blocks improve estimation, but larger filters do not appear to improve coefficient learning.The learned coefficients corresponding to absent PDE terms are reported as close to zero, while larger filters mainly improve prediction duration.

4 NUMERICAL STUDIES: DIFFUSION EQUATIONS WITH NONLINEAR SOURCE

The experiments evaluate PDE-Net on a two-dimensional diffusion equation with a nonlinear source, testing both long-time prediction and recovery of the hidden equation. Results show strong prediction and coefficient/source-term identification under noisy data.

  • 4 NUMERICAL STUDIES: DIFFUSION EQUATIONS WITH NONLINEAR SOURCE: The study uses a 2D diffusion equation with nonlinear source fs(u)=15 sin(u) on a 50 × 50 mesh, with Gaussian noise added to generated sequences.Data are generated from a higher-resolution numerical solution and restricted to the computational mesh.
  • PREDICTING LONG-TIME DYNAMICS: PDE-Net prediction is compared with Frozen-PDE-Net, and additional long-time tests assess generalization to noisy initial values with maximum frequency 10 versus 6 during training.The paper reports that the learned PDE-Net performs well in these prediction experiments.
  • DISCOVERING THE HIDDEN EQUATION: The learned first-order coefficients are close to zero, matching their absence from the true PDE.This provides a direct check on recovery of the equation's differential structure.
  • DISCOVERING THE HIDDEN EQUATION: The learned source response approximates fs more accurately near the interval center, where the observed values of u are concentrated, than near the boundary.The paper attributes the difference to the distribution of training data.

5 CONCLUSION AND DISCUSSION

The paper concludes that PDE-Net jointly learns differential operations and nonlinear responses to discover hidden PDEs and predict dynamics. Its deep structure and larger learnable filters improve stability and extend reliable prediction in noisy settings.

  • 5 CONCLUSION AND DISCUSSION: PDE-Net jointly approximates differential operations with properly constrained convolution filters and nonlinear responses with neural networks or other machine-learning methods.The architecture can also incorporate prior knowledge about the response function.
  • 5 CONCLUSION AND DISCUSSION: Experiments show that PDE-Net uncovers hidden equations and predicts dynamical behavior for relatively long times, even with noisy observations.The conclusion reports this outcome for a linear variable-coefficient convection-diffusion example.
  • 5 CONCLUSION AND DISCUSSION: Multiple δt-blocks and larger learnable filters improve stability and can prolong reliable predictions.
Loading 1710.09668v2…