Source-linked AI summary
DeepMoD: Deep learning for Model Discovery in noisy data
Gert-Jan Both, Subham Choudhury, Pierre Sens, Remy Kusters
TL;DR
DeepMoD addresses model discovery from noisy, small spatio-temporal datasets, where numerical differentiation can make existing sparse-regression methods impractical. It uses neural networks with automatic differentiation and sparse regression to identify governing PDEs, recovering equations with substantially greater noise tolerance and applying the method to noisy electrophoresis data.
Problem
Existing model-discovery methods are sensitive to noise and require many samples because inaccurate numerical derivatives corrupt their function libraries.
Method
DeepMoD uses a neural network to approximate data, construct derivative-based function libraries, and perform L1-regularized sparse regression jointly.
Results
DeepMoD recovers governing PDEs across artificial benchmarks with as few as O(10^3) samples at more than 50% Gaussian noise and identifies advection-diffusion in noisy electrophoresis data.
Takeaways & Limitations
The approach enables automated PDE selection from noisy experimental time-series data while also denoising observations and approximating derivatives.
Takeaways & Limitations
Correct discovery depends on the candidate library containing the underlying functions, and threshold selection can affect the resulting sparse solution.
Abstract
from arXiv · showhide
We introduce DeepMoD, a Deep learning based Model Discovery algorithm. DeepMoD discovers the partial differential equation underlying a spatio-temporal data set using sparse regression on a library of possible functions and their derivatives. A neural network approximates the data and constructs the function library, but it also performs the sparse regression. This construction makes it extremely robust to noise, applicable to small data sets, and, contrary to other deep learning methods, does not require a training set. We benchmark our approach on several physical problems such as the Burgers', Korteweg-de Vries and Keller-Segel equations, and find that it requires as few as $\mathcal{O}(10^2)$ samples and works at noise levels up to $75\%$. Motivated by these results, we apply DeepMoD directly on noisy experimental time-series data from a gel electrophoresis experiment and find that it discovers the advection-diffusion equation describing this system.
1 Introduction
DeepMoD addresses noisy PDE discovery by combining neural-network approximation, automatic differentiation, and sparse regression within one training objective. The approach is designed to improve robustness over existing model-discovery methods and is demonstrated on artificial and experimental data.
- Problem and prior approaches: PDE discovery can be formulated as sparse regression over a library of candidate functions and spatial derivatives.The library may include terms such as u, u_x, and uu_x.
- Problem and prior approaches: Existing sparse-regression methods are sensitive to noise and require many samples because numerical differentiation produces inaccurate derivatives.These limitations make them unsuitable for noisy experimental data.
- DeepMoD approach: DeepMoD embeds sparse regression in a neural-network cost function, jointly optimizing network parameters and the sparse coefficient vector ξ.An L1 penalty promotes sparsity, while training also denoises the dataset and yields the underlying PDE.
- Results and scope: DeepMoD combines automatic differentiation with regression-based regularization to improve model discovery for noisy, small-data settings.The paper applies the approach to Burgers’, Korteweg-de Vries, 2D advection-diffusion, Keller-Segel, and electrophoresis data.
Methods
The method represents PDE discovery as sparse regression over a neural-network-derived function library. Automatic differentiation, normalization, thresholding, and regularization identify the active terms while controlling overfitting.
- Model formulation: DeepMoD seeks the PDE underlying u(x,t) by selecting a sparse combination of candidate functions and their spatial derivatives.The one-dimensional library includes polynomial terms, derivatives up to third order, and combinations such as u^2u_xx.
- Neural-network library: The neural network maps spatial and temporal coordinates to an approximation of u, which is used to construct the candidate-term library.Automatic differentiation supplies spatial and temporal derivatives for the library.
- Training objective: The cost function combines mean squared error, regression loss, and L1 regularization on ξ.The regression term updates ξ alongside the network weights and biases, while L1 regularization promotes sparsity.
- Training objective: The regression loss regularizes the neural network and helps prevent overfitting noisy data despite the large function library.The coefficient vector is optimized after the MSE has converged in the illustrated training process.
- Scaling and thresholding: Thresholding sets negligible coefficients to zero, after which the network is retrained without the L1 penalty to estimate unbiased coefficients.For Burgers’ data, the required u_xx and uu_x terms stand out from other terms at 1%, 10%, and 25% noise.
- Scaling and thresholding: Normalization makes the regression terms dimensionless and places coefficient magnitudes near O(1), enabling comparison across terms with different scales.The normalization also supports selecting terms with different orders of magnitude.
Results
DeepMoD recovers governing PDEs across nonlinear, coupled, higher-dimensional, shock-containing, and experimental systems from randomly sampled noisy data. Its benchmarks include recovery with as few as 100–200 samples, noise levels up to 75%, and successful identification of electrophoresis advection-diffusion structure.
- Benchmark scope: DeepMoD benchmarks Burgers’ equations, Korteweg–de Vries, 2D advection-diffusion, and coupled Keller-Segel PDEs across nonlinear, shock, coupled, higher-dimensional, and experimental settings.The benchmark suite tests the method on nonlinear equations, solutions containing shock waves, coupled PDEs, higher-dimensional data, and experimental data.
- Burgers’ equation: 1% coefficient error was obtained for a Burgers’ equation reconstruction in the initial benchmark.The reconstructed solution and underlying PDE were recovered with coefficients having as little as 1% error.
- Robustness: 100 data-points sufficed at vanishing noise, while 5000 data points supported recovery with noise levels up to 75%.Recovery became intermittent between regimes where all five runs and none of the five runs returned the correct equation, showing the importance of sampling.
- Comparison: DeepMoD recovered the correct Burgers’ equation up to noise levels > 50% with 10^3 samples, compared with 10^5 samples for competing methods.PDE-Find recovered up to 1% Gaussian noise and PDE-Stride up to 5% on the compared dataset with 10^5 data points.
- Shock wave solutions: 5% coefficient error was achieved for Burgers’ equation with a shock wave using 2000 points and 10% white noise.Automatic differentiation was used for this shock-containing case, whose numerical derivatives are difficult because of discontinuities.
- Coupled differential equations: The Keller-Segel experiment recovered both coupled PDEs from 10000 points with 5% white noise, including the small coupled term uxwx.The library contained 36 terms, including spatial derivatives, cross terms, and first-order polynomial terms.
- Higher-dimensional data: For 2D advection-diffusion, 200 noiseless samples recovered the equation, while 5000 samples recovered the PDE and velocity vector up to approximately 25% noise.The simulated system used D = 0.5 and velocity vector (0.25, 0.5).
- Experimental data: In electrophoresis data, 5000 sampled points consistently recovered the advection term uy and diffusive terms uxx and uyy.Analysis of a second dye gave nearly identical drift velocity v ≈ 0.3 and diffusion coefficient D ≈ 0.01, while some higher-order nonlinear terms remained small but non-zero.
Discussion
DeepMoD integrates neural-network approximation, automatic differentiation, and sparse regression to discover PDEs from noisy spatio-temporal data. The authors report robustness across artificial and experimental datasets, while identifying library completeness and thresholding as key conditions for success.
- Contributions: DeepMoD discovers governing PDEs by combining neural networks with sparse model discovery for spatio-temporal datasets.The method is demonstrated on five artificial case studies and an experimental advection-diffusion dataset.
- Scope: DeepMoD supports higher-dimensional inputs and outputs, coupled PDEs, and inference of diffusive or chemotactic equations from single-particle-tracking data.The demonstrated coupled systems include 2D advection-diffusion and Keller-Segel equations.
- Method and robustness: Automatic differentiation and regression-based regularization enhance derivative approximation in noisy data and improve robustness to noise levels and sample size.The authors describe this combination as enabling model selection on highly noisy, low-resolution experimental data.
- Limitations: The method’s success depends strongly on a complete candidate-function library and a threshold for retaining small non-zero coefficients.If the underlying functions are absent from the library, DeepMoD will not return the correct equation; cross-validation can identify this problem.
- Additional consequences: DeepMoD also denoises data and approximates derivatives from noisy measurements, with potential applications to physics-informed enhancement of super-resolution images.The paper characterizes derivative estimation as difficult for classical interpolation and finite-difference schemes.
SUPPLEMENTARY INFORMATION FOR DEEPMOD: DEEP LEARNING FOR MODEL DISCOVERY IN NOISY DATA
The supplementary material details DeepMoD’s sparse-regression pipeline, from neural-network approximation and automatic differentiation to coefficient normalization and thresholding. It also documents convergence behavior, noise-related MSE interpretation, and the computational cost of enlarging the candidate library.
- Method: DeepMoD represents the governing PDE as a sparse coefficient vector over a library of polynomial functions and spatial derivatives.The model-discovery task is reduced to finding a sparse vector that fits the data.
- Method: A feed-forward neural network maps input coordinates to the field, while automatic differentiation supplies derivatives used to construct the candidate library.The activation function must be differentiable to at least the highest derivative order included in the library.
- Method: The regression problem is embedded in the network cost function and combined with L1 regularization to promote sparse coefficients.The supplementary material describes optimization of the network and coefficient vector together.
- Library size: Increasing the library to 20 terms still recovered the correct Burgers’ and Korteweg-de Vries equations with identical coefficient accuracy, but substantially increased computational time.The enlarged library included fourth-order derivatives and third-order polynomials, compared with 12 terms in the main-text example.
- Training and noise: Approximately 8000 epochs were required for convergence in both noiseless and noisy Burgers datasets, with noisy-data cost dominated by the MSE.For noisy data, the regression term prevents overfitting and the residual MSE reflects the difference between measured and underlying dynamics.
- Training and noise: For the noisy case, an MSE of approximately 10^-4 matched the expected minimum, while substantially larger networks showed no observed overfitting.The supplementary analysis identifies lower-than-expected MSE as a possible sign of overfitting.
- Normalization and thresholding: Coefficient normalization puts library terms on comparable scales before thresholding, enabling selection among terms with different orders of magnitude.The normalized coefficients are typically O(10^-1 − 10^0), after which components near the median are thresholded using a standard-deviation criterion.
- Normalization and thresholding: The standard-deviation threshold performed well in the presented benchmarks, but its criterion is described as somewhat arbitrary.Normalization is important when physically meaningful coefficients differ substantially in magnitude, such as Burgers’ nonlinear and diffusion terms.
Synthetic data
DeepMoD was evaluated on synthetic PDEs spanning one- and two-dimensional problems, varying noise and sample size. It recovered higher-order and two-dimensional equations from relatively sparse, noisy data, although sampling affected recovery reliability.
- DeepMoD was tested on five synthetic case studies, including Burgers’, KdV, 2D advection diffusion, and Keller-Segel equations.
- The synthetic evaluation added Gaussian white noise, typically ranging from 5% to 75%, to assess robustness against noise and dataset size.
- Korteweg-de Vries equation: 20% white noise and 2,000 randomly sampled points still yielded the correct KdV PDE, although recovery depended strongly on sampling.
- 2D advection diffusion: 5000 randomly sampled points recovered the 2D advection-diffusion PDE and velocity vector for noise levels up to approximately 25%.
- 2D advection diffusion: 200 samples were sufficient to recover the correct 2D advection-diffusion equation at vanishing noise levels.
Gel electrophoresis
The electrophoresis experiment provided noisy concentration profiles for discovering an advection-diffusion equation. The study also examined coefficient selection, training behavior, and sensitivity to network architecture across synthetic and experimental settings.
- A low-voltage electrophoresis experiment tracked a charged dye moving through agarose gel under an approximately constant-velocity electric field.
- The experimental analysis used a 12-term library and 5,000 randomly sampled points across 300×150 pixels and 50 time frames.
- Coefficient selection: Two of ten electrophoresis runs incorrectly retained terms after thresholding the coefficient vector.
- Network architecture: At least two network layers produced results whose size had negligible effect on the recovered equations.
- Synthetic benchmarks: For Burgers’ data, the accuracy analysis varied noise at 500 samples and sample size at vanishing noise levels, marking failed PDE recovery separately.