PaperlayerView exact revision on arXiv

Source-linked AI summary

PDEBENCH: An Extensive Benchmark for Scientific Machine Learning

Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, Mathias Niepert

arXiv:2210.07182v7cs.LGcs.CVphysics.flu-dynphysics.geo-ph

TL;DR

Scientific ML lacks accessible benchmarks that are both easy to use and challenging across representative PDE problems. PDEBENCH addresses this gap with diverse datasets, extensible generation code, baseline models, and physics-aware metrics; its experiments show that model performance depends on the task and that RMSE alone is insufficient.

  • Problem

    Scientific ML lacks a widely accessible, practically simple, and statistically challenging benchmark with ready-to-use datasets for comparing methods across PDE problems.

  • Method

    PDEBENCH combines datasets from 11 PDEs, forward and inverse tasks, a uniform extensible interface, downloadable and generative data, baseline models, and Scientific ML metrics.

  • Results

    PDEBENCH experiments find no one-size-fits-all model and show that RMSE is not a good proxy in turbulent and non-smooth regimes; FNO provides the best prediction for most metrics among baseline surrogate models.

  • Takeaways & Limitations

    The benchmark identifies challenging tasks for current ML methods and provides standardized resources for comparing and extending Scientific ML approaches.

  • Takeaways & Limitations

    PDEBENCH focuses on time-dependent flow problems and does not currently cover other physics, quantum mechanics, multi-phase flows, or non-rectangular domains.

Abstract

from arXiv · show

Machine learning-based modeling of physical systems has experienced increased interest in recent years. Despite some impressive progress, there is still a lack of benchmarks for Scientific ML that are easy to use but still challenging and representative of a wide range of problems. We introduce PDEBench, a benchmark suite of time-dependent simulation tasks based on Partial Differential Equations (PDEs). PDEBench comprises both code and data to benchmark the performance of novel machine learning models against both classical numerical simulations and machine learning baselines. Our proposed set of benchmark problems contribute the following unique features: (1) A much wider range of PDEs compared to existing benchmarks, ranging from relatively common examples to more realistic and difficult problems; (2) much larger ready-to-use datasets compared to prior work, comprising multiple simulation runs across a larger number of initial and boundary conditions and PDE parameters; (3) more extensible source codes with user-friendly APIs for data generation and baseline results with popular machine learning models (FNO, U-Net, PINN, Gradient-Based Inverse Method). PDEBench allows researchers to extend the benchmark freely for their own purposes using a standardized API and to compare the performance of new models to existing baseline methods. We also propose new evaluation metrics with the aim to provide a more holistic understanding of learning methods in the context of Scientific ML. With those metrics we identify tasks which are challenging for recent ML methods and propose these tasks as future challenges for the community. The code is available at https://github.com/pdebench/PDEBench.

1 Motivation

Scientific ML lacks an accessible, simple, and statistically challenging benchmark with ready-to-use datasets. PDEBENCH addresses this gap with diverse PDE problems, extensible data and code, baseline models, and physics-aware evaluation.

  • Scientific ML lacks a widely accessible benchmark combining practical simplicity, ready-to-use datasets, and statistically challenging PDE problems.
  • The benchmark is motivated by costly dataset generation, which requires substantial compute, storage, and specialized expertise.PDEBENCH offers quick, off-the-shelf training data and an easy path for extension.
  • PDEBENCH provides diverse datasets based on 11 PDEs, covering forward and inverse problems through a uniform, extensible interface.The suite includes downloadable data, generation code, and pretrained models for comparison.
  • Its datasets span multiple parameter values, initial and boundary conditions, and resolutions sufficient to capture local dynamics.
  • PDEBENCH evaluates models with metrics beyond RMSE that incorporate properties of the underlying physics.

2 Related Work

Scientific ML benchmarks must address PDE-specific datasets and diverse solver-learning methods rather than relying on monolithic datasets from conventional machine learning.

  • PDE datasets can occupy gigabytes or terabytes while containing relatively few samples, and each PDE task requires a task-specific dataset.
  • Scientific ML methods learn neural surrogates that approximate outputs from ground-truth PDE solvers.Relevant approaches include physics-informed neural networks and neural operators.

3 PDEBENCH: A Benchmark for Scientific Machine Learning

PDEBENCH formalizes PDE surrogate learning across varied spatial-temporal problems, datasets, baselines, and physics-aware metrics. Its benchmark spans stylized and advanced flow problems designed to test challenging dynamics and generalization.

  • 3.1 General Problem Definition: A PDE solution is a vector-valued function over spatial domain, temporal index, and possibly function-valued parameters.
  • 3.1 General Problem Definition: The forward propagator maps a PDE state at one timestep to the state at the next timestep.
  • 3.1 General Problem Definition: Scientific ML learns an emulator approximating the discretized forward operator from datasets of simulated PDE solutions.Multiple previous timesteps may be used to represent temporal derivatives through finite-difference approximations.
  • 3.1 General Problem Definition: Inverse settings estimate unknown initial conditions or PDE parameters from observed outputs using differentiable forward surrogates.
  • 3.1 General Problem Definition: The benchmark includes 1D-to-3D, time-dependent and time-independent datasets generated with varied parameters, initial conditions, and boundary conditions.
  • 3.2 Overview of Datasets and PDEs: The dataset collection combines six basic PDEs with three advanced real-world problems exhibiting shocks, sensitivity to initial conditions, diverse boundaries, and spatial heterogeneity.Advanced problems include compressible and incompressible Navier–Stokes equations and shallow-water equations.
  • 3.2 Overview of Datasets and PDEs: Compressible Navier–Stokes equations model phenomena including shock-wave formation, while incompressible equations apply to sub-sonic and turbulent flows.
  • 3.3 Overview of Metrics: RMSE alone may miss small-scale spatial changes and physics violations, motivating metrics for conserved quantities, boundaries, frequency bands, local maxima, and extrapolation.

4 A Selection of Experiments

The experiments compare baseline surrogate models, autoregressive U-Net training strategies, temporal extrapolation, and inference runtimes across PDEBench tasks. Results highlight FNO’s strong overall accuracy, pushforward-trained U-Net stability, increasing extrapolation error, and efficient trained-model inference.

  • Experiments: The section presents selected experiments, with additional results, tables, and plots provided in the appendix.The reported experiments do not constitute an exhaustive discussion of all PDEBENCH results.
  • Baseline Setups: The experiments train U-Net, FNO, and PINN baselines using 90% training data and 10% test data.FNO follows its original implementation and protocols, while U-Net uses autoregressive training with the pushforward trick.
  • Baseline Performance: FNO provides the best prediction for most metrics and learns differential operators with low errors for conserved quantities and boundaries.Across many problems, its error remains about 4 × 10^-4 across the frequency spectrum.
  • Temporal Error Analysis: Pushforward-trained U-Net provides better training stability across all problems and better accuracy over longer prediction horizons.Fully autoregressive training produced instabilities, while open-loop testing accumulated error over unrolled steps.
  • Temporal Error Analysis: After training only on half the time steps, FNO’s RMSE increases monotonically for 1D Advection and Burgers equations when t > 1.The experiment tests whether the model captures temporal dependencies with limited training-time information.
  • Inference Time Comparison: Once trained, ML predictions can be computed multiple orders of magnitude more efficiently than numerical simulations.The runtime comparison is hardware-dependent, and ML training has the highest computational demand.

5 Conclusions and Limitations

PDEBENCH provides accessible, extensible resources for evaluating Scientific ML methods across diverse PDE problems. Its current scope focuses on time-dependent flow problems and excludes several physics settings, including multiphase flows and non-rectangular domains.

  • Conclusions: PDEBENCH combines precomputed datasets, configurable data-generation code, diverse PDE tasks, scientific evaluation metrics, and an inverse-modeling example.The benchmark spans simple 1D equations through challenging 3D coupled systems with complex boundary conditions.
  • Limitations: The benchmark focuses on time-dependent flow problems and does not cover other physics or quantum mechanics.These areas are explicitly outside the paper’s scope.
  • Limitations: Current flow-problem limitations include the absence of multiphase flows and non-rectangular domains.The authors leave both extensions for future work.

Checklist

Scientific ML lacks a broadly accessible and challenging benchmark, while existing frameworks cover only parts of the method space or impose restrictions. PDEBench addresses this gap with broader interoperability and extensibility.

  • Machine learning has established benchmarks across domains, but Scientific ML still lacks a widely accessible, simple, and statistically challenging benchmark.
  • Differentiable PDE solvers can support training, optimization, control, sensitivity analysis, and inverse problems, complementing surrogate-model benchmarks.
  • Existing unified Scientific ML frameworks either face intellectual-property restrictions or focus largely on PINN and DeepONet methods.

B Detailed metrics description

PDEBench evaluates forward and inverse Scientific ML models with global, local, conservation-aware, boundary-aware, and frequency-aware metrics. Its inverse evaluation separately measures errors in estimated quantities and their resulting predictions.

  • The benchmark uses RMSE, normalized RMSE, and maximum error to measure global emulation performance and prediction stability.
  • cRMSE measures deviation from a conserved value, while bRMSE evaluates whether the model properly represents boundary conditions.
  • fRMSE measures prediction error across low-, middle-, and high-frequency Fourier regions to assess dependence on wavelength.
  • Inverse evaluation measures both error in the estimated initial condition and error in predictions generated from that estimate.
  • Inverse metrics include MSE, normalized L2 and L3 norms, and corresponding FFT-based measures across low-, middle-, and high-frequency ranges.

C Training Protocol and Hyperparameters

The benchmark combines standardized training protocols with diverse PDE formulations, numerical solvers, initial and boundary conditions, and compressible-flow datasets. These settings span forward and challenging physical dynamics, including rapid reaction, nonlinear transport, shocks, and nonstandard boundaries.

  • Training protocol: FNO training uses 500 Adam epochs, an initial learning rate of 10^-3, halved every 100 epochs, with 90% training and 10% validation/testing splits.
  • Training protocol: The inverse setting predicts a specified future state t = T, using 20 training epochs and T = 15, except T = 5 for CFD.The setup estimates 64 hidden initial-condition values, tests 100 samples, and uses a gradient-method learning rate of 0.2.
  • PDE configurations: The 1D advection equation models linear transport with constant speed β and exact solution u(t, x) = u0(x − βt).Datasets use periodic boundaries and sinusoidal-wave initial conditions, with second-order upwind finite differences for numerical solutions.
  • PDE configurations: The diffusion-reaction equation combines diffusion with a source term whose dependence on u can produce potentially exponential growth and rapid dynamics.Its numerical solution uses second-order central differences and a piecewise-exact solution method for the source term.
  • PDE configurations: Burgers’ equation combines nonlinear fluid behavior and diffusion, with constant viscosity ν; its Reynolds-number regime distinguishes stronger nonlinear behavior from diffusion.The numerical scheme uses upwind differences for advection and central differences for diffusion.
  • Advanced and applied datasets: PDEBench includes compressible Navier-Stokes datasets across one to three dimensions, random-field, turbulence, and shock-tube settings, with periodic or outgoing boundaries and varied parameter regimes.The benchmark also includes a diffusion-sorption problem with nonlinear diffusion, a singularity at u = 0, and a derivative boundary condition relevant to groundwater contaminant transport.

D.10 Gradient-Based Inverse Method

The Gradient-Based Inverse Method estimates initial conditions from final-time observations and evaluates errors in spatial and frequency domains. For the 1D advection example, errors concentrate in mid-high frequencies and are smaller at the final-time prediction than in the estimated initial condition.

  • Method: The inverse problem minimizes prediction loss to infer an initial condition from a final-time state.The estimated initial condition is generated deterministically, with parameters using bilinear interpolation.
  • Evaluation: For the 1D advection example, the estimated initial condition and final-time prediction are compared in both spatial and frequency domains.The evaluation uses spatial-domain and frequency-domain errors across four datasets with FNO and U-Net surrogate models.
  • Results: The final-time prediction error is smaller than the error in the estimated initial condition.Figure 12 shows the true and estimated initial conditions on the left and true and predicted final-time values in the middle.
  • Results: The largest inverse-problem error is concentrated in the higher frequencies.This pattern is also visible in the frequency metrics reported in Tables 3 and 4.
  • Evaluation: Baseline performance is summarized with spatial, conserved-value, boundary, maximum-error, and low-, medium-, and high-frequency metrics across several PDE tasks.The reported tables cover diffusion-sorption, diffusion-reaction, shallow-water, advection, Burgers’, Darcy flow, and compressible Navier-Stokes equations.

F Detailed Runtime Comparison

This section compares the computation time of the PDE data-generation solver with FNO, U-Net, and PINN baseline runs. Timing measurements use specified CPU, GPU, and software configurations, with separate comparisons for 2D and 3D settings.

  • Timing setup: Computation times are compared between a PDE solver, single forward runs of FNO, U-Net, and PINN, and one training epoch.Table 15 reports all times in seconds.
  • 2D scenarios: The 2D timing measurements cover diffusion-sorption, 2D diffusion-reaction, and shallow-water equation scenarios.PyClaw is used for shallow-water data generation.
  • Hardware and software: The 2D measurements use two AMD EPYC 7742 CPUs, one NVIDIA Volta V100 GPU, PyTorch 1.11, and CUDA 11.3.These system details are listed in Table 14.
  • 3D scenarios: The 3D experiment reduces resolution to 64^3 because of severe memory usage, although the official dataset includes 128^3 data.Its shorter times also reflect fewer time steps and samples than the 2D cases.

G Resolution Sensitivity of Inference Time

Inference-time scaling is evaluated for classical simulations and ML methods on 2D and 3D compressible Navier-Stokes cases using the same hardware. ML inference is reported as substantially faster and less sensitive to diffusion strength.

  • Experimental setup: The comparison plots inference time versus resolution for classical simulation and ML methods on 2D and 3D compressible Navier-Stokes cases.The measurements use identical hardware resources for fairness.
  • Results: Nearly 3 orders of magnitude separate ML inference time from classical simulation time.This result is reported for the compressible Navier-Stokes resolution-dependence experiment.
  • Resolution scaling: Both ML models show resolution dependence similar to the inviscid classical simulation method.The comparison concerns the scaling trend rather than a single resolution.
  • Diffusion dependence: ML inference time is generally independent of the diffusion coefficient, whereas classical simulation time increases with it because of the CFL stability condition.For the explicit method, the stated condition is Δt ∝ Δx^2/η.

H Error Comparison with PDE Solver

The PDE-solver comparison tests whether baseline predictions remain competitive with higher-resolution reference data and presents visual comparisons across multiple benchmark equations. For 2D diffusion-reaction, low-resolution solver data can contain larger discretization error than baseline prediction error.

  • Resolution sensitivity: Higher-resolution 2D diffusion-reaction data at 512×512 are downsampled to 128×128 and treated as ground truth for error comparison.U-Net, FNO, PINN, and low-resolution PDE-solver predictions are compared against this reference.
  • Results: Low-resolution PDE-solver data already accumulate high discretization error relative to baseline model prediction error.The comparison is reported for the 2D diffusion-reaction scenario.
  • Caveat: The authors state that further resolution sensitivity analysis is required before confirming that the chosen resolution is sufficiently fine as ground truth.This qualifies the interpretation of the higher-resolution reference experiment.
  • Visual comparisons: Visualizations compare generated datasets with baseline predictions for diffusion-sorption and 2D diffusion-reaction equations.The listed figures include diffusion-sorption data, FNO, and U-Net predictions, plus diffusion-reaction data and FNO predictions.
  • Visual comparisons: The visualization set also covers shallow-water, advection, Burgers’, reaction-diffusion, Darcy flow, and compressible Navier-Stokes equations.These cases are referenced across Figures 18–26.

J Visualization of Initial Conditions

PDEBench visualizes diverse initial conditions and simulation outputs across multiple PDE families, while documenting dataset composition, acquisition, and scope. The benchmark is intended to represent challenging physical systems, although no finite collection can represent the entire space of such systems.

  • Initial-condition visualizations: Figures visualize initial-condition diversity for shallow-water, diffusion-sorption, diffusion-reaction, advection, Burgers, reaction-diffusion, and CFD datasets.The visualizations include varying perturbation radii, random uniform conditions, random noise, scalar-function samples, and CFD samples.
  • Dataset composition: Each dataset instance is a simulation-generated array containing PDE solution values for variables of interest.The arrays represent solutions of specific partial differential equations and their physical-system variables.
  • Dataset scope: PDEBENCH includes datasets with longer time steps, non-linearly coupled variables, and varied parameters, initial conditions, and boundary conditions.The dataset documentation presents these properties as part of its effort to cover challenging physical systems.
  • Data acquisition: The dataset was acquired through computationally expensive numerical simulations rather than direct observation.Its instances are generated by performing computational simulations of the governing PDEs.
  • Scope boundary: High-dimensional data, especially 3D data, has only 100 samples because storage and individual sample sizes are limiting factors.The authors note that this may reduce ML accuracy, while source code allows users to generate additional data.

K.9 Reading and using the dataset

PDEBench stores simulation arrays in HDF5 and provides APIs and examples for loading them into machine-learning workflows. Its standardized layout identifies samples, time, spatial dimensions, and state variables, with special handling for large CFD variables.

  • Reading the dataset: Downloaded HDF5 files can be read with standard HDF5 software, while provided APIs load data into PyTorch data loaders.The project also supplies training and dataset-reading information through its home page.
  • Data-format example: Figure 34 illustrates PDEBENCH's data format for PDEs containing density and velocity terms.The figure provides a visual example of the documented data organization.
  • File format: PDEBENCH stores realizations in HDF5 files whose names encode the PDE, parameters, and configuration.Each HDF5 file contains a documented group that may include multiple tensor datasets.
  • Array convention: Arrays follow the convention (b×t×x1 · · ·×xd ×v), representing samples, time, spatial dimensions, and system-state variables.Some datasets omit dimensions when parameters are time-invariant.
  • Large CFD data: Compressible Navier-Stokes variables are stored independently rather than in a shared channel dimension to reduce array size and support spatial subsampling.The documented examples include density and pressure as separate variables.
Loading 2210.07182v7…