PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-driven discovery of PDEs in complex datasets

Jens Berg, Kaj Nyström

arXiv:1808.10788v1stat.MLcs.LGmath.NA

TL;DR

The paper asks how PDEs governing complex physical processes can be recovered when first-principles derivations are difficult, using measurements instead. It develops deep-neural-network methods for PDE discovery and applies them to model and weather-station data. The discovered ODE operators accurately reproduce dynamics within and slightly beyond training regions, while remaining computationally cheaper than PDE models.

  • Problem

    The paper addresses discovering PDEs from measured data when first-principles derivations are intractable or physical laws are unknown.

  • Method

    Deep neural networks approximate measured functions and discover differential operators from model-problem and weather-station datasets, including coordinate transformations induced by feature scaling.

  • Results

    The automatically discovered ODE operators accurately approximate nonlinear PDE dynamics in training regions and slightly beyond, including the Swedish temperature-distribution model.

  • Takeaways & Limitations

    The discovered ODE models can reproduce the studied dynamics while being solvable in fractions of a second on a laptop compared with PDE models requiring substantial computational resources.

  • Takeaways & Limitations

    The ODE operators are unable to extrapolate far beyond the training data, and linear interpolation can introduce non-physical relations in neural-network approximations.

Abstract

from arXiv · show

Many processes in science and engineering can be described by partial differential equations (PDEs). Traditionally, PDEs are derived by considering first principles of physics to derive the relations between the involved physical quantities of interest. A different approach is to measure the quantities of interest and use deep learning to reverse engineer the PDEs which are describing the physical process. In this paper we use machine learning, and deep learning in particular, to discover PDEs hidden in complex data sets from measurement data. We include examples of data from a known model problem, and real data from weather station measurements. We show how necessary transformations of the input data amounts to coordinate transformations in the discovered PDE, and we elaborate on feature and model selection. It is shown that the dynamics of a non-linear, second order PDE can be accurately described by an ordinary differential equation which is automatically discovered by our deep learning algorithm. Even more interestingly, we show that similar results apply in the context of more complex simulations of the Swedish temperature distribution.

1. Introduction

The paper addresses data-driven PDE discovery when first-principles derivations are difficult or impossible. It uses deep neural networks to identify governing PDEs from complex measurement datasets and incorporates coordinate transformations arising from input-data transformations.

  • Scientific and industrial applications need predictive models that reveal data-generating processes from abundant measurements.
  • PDE identification from data is less explored than time-dependent dynamical-system identification, with recent machine-learning progress enabling its development.
  • When physical laws are unknown or derivations become intractable, the paper aims to automatically identify PDEs from spatio-temporal data.
  • The proposed deep-neural-network approach unifies sparse regression and general differential-operator identification.
  • Input transformations for complex datasets appear as coordinate transformations and produce metric coefficients in the identified PDE.

2. Method

The method fits neural networks to unordered space-time measurements, differentiates the learned function, and learns a parameterized differential operator. Feature scaling is treated as a coordinate transformation, while model choices trade expressive power against interpretability and well-posedness.

  • Method overview: The method assumes unordered space-time coordinates and function values with an unknown governing equation, then fits a function whose derivatives can be computed.Derivative estimation is a separate preprocessing step that can use neural networks or other approximation methods.
  • Function approximation: Deep neural networks approximate the measured function as smooth maps whose derivatives are analytically available through backpropagation or automatic differentiation.
  • Operator discovery: The second stage learns a parameterized differential operator from the neural-network approximation and its partial derivatives.The operator can be represented by a feedforward neural network and optimized with gradient-based methods.
  • Model parameterization: Sparse regression is recovered with a hidden-layer-free neural network, while richer neural networks represent more general differential operators.
  • Model selection: L1 regularization favors sparse PDEs, but model complexity creates a trade-off between expressive power, interpretability, and well-posedness.The paper notes that general neural-network operators may be difficult to read or understand and may not produce well-posed PDEs.
  • Feature scaling: Feature scaling is formulated as an invertible coordinate transformation, commonly shifting and scaling inputs to zero mean and unit variance.Derivatives must be transformed back to physical coordinates using the Jacobian or, for separable transformations, scalar factors.
  • Feature scaling: Under coordinate transformations, the learned neural network approximates a transformed PDE rather than the original equation unless derivatives are mapped back to physical coordinates.

3. Examples

The examples show how deep-learning PDE discovery behaves on a known viscous Burgers’ model and on complex weather-related temperature data. Feature scaling appears as a coordinate transformation in the identified PDE, while model choices trade interpretability against generality.

  • 3.1. The viscous Burger’s equation in 1D: The viscous Burgers’ solution forms a decaying stationary viscous shock after finite time.
  • 3.1. The viscous Burger’s equation in 1D: 255000 entries sampled from the numerical solution provide the dataset for reconstructing the differential operator.The preprocessing network uses 5 hidden layers with 10 neurons per layer and hyperbolic-tangent activation.
  • 3.1. The viscous Burger’s equation in 1D: Three parametrizations recover PDE models ranging from a known-form linear model to a general neural-network operator.The models use either selected Burgers terms, a broader library of derivative and nonlinear terms, or a neural-network representation.
  • 3.1. The viscous Burger’s equation in 1D: The models trade off resemblance to the true PDE, suppression of spurious coefficients, and human interpretability.The general model can nevertheless serve as input to automatic-discretization or physics-informed-neural-network software.
  • 3.1. The viscous Burger’s equation in 1D: Feature scaling changes the identified equation because the neural network approximates the transformed PDE rather than the original physical-coordinate equation.Inverting the coordinate transformation and recomputing derivatives recovers the desired PDE in physical coordinates.
  • 3.1. The viscous Burger’s equation in 1D: The simple Burgers model does not require coordinate transformations because its domain is [0, 1] × [0, 1].
  • 3.2. Temperature distribution in 2D: The weather application uses irregular station measurements in a complicated geometry and linearly interpolates them onto a regular space-time grid.The temperature data are used as a starting point for applying the method to weather and climate modeling.
  • 3.2. Temperature distribution in 2D: Linear interpolation can introduce non-physical linear relations into the neural-network approximation, motivating more advanced reconstruction methods for real cases.

4. Feature selection

Feature selection balances interpretability with robustness: sparse regularization removes negligible terms, while variance alone is misleading because derivative order inflates variance independently of importance.

  • Regularization and sparsity: L1 regularization produces a sparser polynomial PDE model by shrinking negligible coefficients that can be removed with a predefined cutoff.The passage gives small coefficients and describes their removal by thresholding.
  • Regularization and sparsity: For the linear polynomial PDE model, traditional least squares with Lasso provides an even sparser alternative.This approach is efficient for the convex polynomial model but does not extend to arbitrary-complexity operators or very large datasets.
  • Feature ranking: Variance-based filtering is unsuitable because higher-order derivatives generally have higher variance regardless of feature importance.The dataset analysis identifies u, u_x, and u_xx as independent terms, while derivative variance increases with order.
  • Feature ranking: Combining randomized Lasso with recursive feature elimination provides a reasonable basis for understanding feature importance and ranking.Table 1 reports variance, feature importance, and feature ranking for the viscous Burger’s dataset.

5. Model selection

Model selection uses cost-function searches across polynomial orders and neural-network architectures. The resulting models capture viscous Burger’s dynamics and identify short-time ODE approximations for both Burger’s and Swedish temperature data, while extrapolation remains limited.

  • Polynomial model selection: The viscous Burger’s equation has true derivative and nonlinear orders m/k = 2/1, identified by a sharp cost reduction when a sufficient model is selected.The cost function is evaluated across derivative order m and nonlinear order k.
  • Neural-network model selection: Neural-network model selection varies network depth, width, and derivative order to locate architectures with low cost.The search compares different network designs and partial derivative orders as inputs.
  • Viscous Burger’s equation: A network with 2 hidden layers and 50 neurons per layer yields low cost without partial derivatives, effectively producing an ODE for the viscous Burger’s equation.This corresponds to m = 0, where the learned operator uses no partial derivatives as input.
  • Viscous Burger’s equation: The learned Burger’s ODE is accurate for 0 ≤t ≤1 and slightly beyond, but cannot extrapolate far beyond the training interval.Runge-Kutta 4(5) integration reproduces the PDE solution over the trained range, whereas performance deteriorates for t >> 1.
  • Temperature models: No sufficient polynomial temperature model is found across the tested derivative and nonlinear orders.The temperature-model cost-function search therefore invalidates that polynomial model class over the tested parameter range.
  • Temperature models: Neural-network searches find sufficient temperature models, including ODE configurations that appear to capture the dynamics.The search varies 1, 2, 4, 6, or 8 hidden layers, widths of 5, 10, 20, or 50 neurons, and derivative orders m = 0, 1, 2, 4.
  • Temperature models: The temperature ODE remains accurate through t = 1.25, equivalent to one quarter of a week, but cannot extrapolate far beyond the training data.It is trained on the first week of July 2016 and evaluated over two weeks using mean squared error in time.
  • Temperature models: A two-week Swedish temperature simulation using the ODE operator takes only a fraction of a second on a laptop.The authors suggest that incorporating more measured quantities could enable fast, accurate short-time predictions with a discovered ODE system.

6. Summary and conclusions

The paper uses deep neural networks to discover PDEs from both model-generated and physical measurement data. It shows that data transformations alter the discovered PDE through coordinate transformations, while automatically discovered ODEs can efficiently approximate selected PDE dynamics.

  • Deep neural networks discover PDEs from both known model data and measurements of physical quantities.
  • Data transformations correspond to coordinate transformations in the discovered PDE and must be recorded to recover the physical-space equation.
  • The discovered PDE operator is not unique, so parameter searches identify a range of operators describing the data-generating PDE.
  • The nonlinear, second-order viscous Burgers equation can be well approximated by an automatically discovered ODE.
  • ODE models for the studied dynamics are accurate near training data, extrapolate slightly beyond it, and solve much faster than PDE models.The ODEs solve in fractions of a second on a laptop, whereas PDE models require substantial computational resources.
Loading 1808.10788v1…