PaperlayerView exact revision on arXiv

Source-linked AI summary

RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data

Peiyan Hu, Haodong Feng, Hongyuan Liu, Tongtong Yan, Wenhao Deng, Tianrun Gao, Rong Zheng, Haoren Zheng, Chenglei Yu, Chuanrui Wang, Kaiwen Li, Zhi-Ming Ma, Dezhi Zhou, Xingcai Lu, Dixia Fan, Tailin Wu

arXiv:2601.01829v2cs.LG

TL;DR

Scientific ML models are mostly trained and validated on simulations, while real-world data are scarce and differ from simulated data. RealPDEBench pairs measurements with simulations and defines tasks and evaluation resources for complex physical systems. The benchmark reveals real–simulated discrepancies, while simulated pretraining improves prediction performance and finetuning convergence.

  • Problem

    Scientific ML is mostly trained and validated on simulated data, while scarce real-world datasets limit evaluation in real scenarios and work on sim-to-real transfer.

    Source: §1 · p. 2

  • Method

    RealPDEBench pairs real-world measurements and simulations across physical systems and defines tasks comparing real-world training, simulated training, and simulated pretraining with real-world finetuning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    RealPDEBench reveals a gap between real-world and simulated data, while reasonably leveraging simulated data improves real-world prediction and simulated pretraining speeds finetuning convergence.

    Source: §1 · p. 2, §4.3 · p. 9

  • Takeaways & Limitations

    RealPDEBench provides a foundation for exploring how to combine real-world and simulated data for improved models.

    Source: §1 · p. 2, §5 · p. 10

    The benchmark does not systematically explore strong out-of-distribution regimes and currently covers a limited range of physical domains.

    Source: F LIMITATIONS AND FUTURE WORKS · p. 37

Abstract

from arXiv · show

Predicting the evolution of complex physical systems remains a central problem in science and engineering. Despite rapid progress in scientific Machine Learning (ML) models, a critical bottleneck is the lack of expensive real-world data, resulting in most current models being trained and validated on simulated data. Beyond limiting the development and evaluation of scientific ML, this gap also hinders research into essential tasks such as sim-to-real transfer. We introduce RealPDEBench, the first benchmark for scientific ML that integrates real-world measurements with paired numerical simulations. RealPDEBench consists of five datasets, three tasks, eight metrics, and ten baselines. We first present five real-world measured datasets with paired simulated datasets across different complex physical systems. We further define three tasks, which allow comparisons between real-world and simulated data, and facilitate the development of methods to bridge the two. Moreover, we design eight evaluation metrics, spanning data-oriented and physics-oriented metrics, and finally benchmark ten representative baselines, including state-of-the-art models, pretrained PDE foundation models, and a traditional method. Experiments reveal significant discrepancies between simulated and real-world data, while showing that pretraining with simulated data consistently improves both accuracy and convergence. In this work, we hope to provide insights from real-world data, advancing scientific ML toward bridging the sim-to-real gap and real-world deployment. Our benchmark, datasets, and instructions are available at https://realpdebench.github.io/.

1 INTRODUCTION

Scientific ML is commonly trained and evaluated on simulations despite scarce, costly real-world data and substantial simulation–measurement gaps. RealPDEBench pairs measurements with simulations and benchmarks tasks, metrics, and baselines to study those gaps and their potential to be bridged.

1 INTRODUCTION

  • Real-world datasets remain scarce because acquiring them requires costly experimental setups and measurement expertise, limiting scientific ML development and evaluation.The scarcity also hinders work on noisy real-world data, sim-to-real transfer, and the effects of measurement limitations on model performance.Source: §1 · p. 2
  • More than 700 trajectories across five fluid-dynamics and combustion scenarios pair real measurements with simulations under distinct operating conditions.Each trajectory exceeds 2000 frames, and each parameter setting includes real-world and simulated data of equal temporal duration.Source: §1 · p. 2
  • Three tasks compare real-world training, simulated training, and simulated pretraining followed by real-world finetuning.These settings support comparisons of the strengths and limitations of real-world and simulated data.Source: §1 · p. 2
  • RealPDEBench exposes a gap between real-world and simulated data, while experiments indicate that reasonably leveraging simulations can improve real-world prediction.The benchmark is intended to support further exploration of how to combine the advantages of both data types.Source: §1 · p. 2
  • The benchmark includes nine evaluation metrics and ten baselines, spanning data-driven scientific ML models, state-of-the-art architectures, a pretrained PDE foundation model, and a traditional method.The metrics include data-oriented and physics-oriented measures, plus a dedicated metric for investigating simulated-data pretraining.Source: §1 · p. 2

2 RELATED WORK

Existing scientific ML benchmarks address PDEs, fluid scenarios, inverse problems, or data quality, while some scientific observational datasets are not designed for ML and have limited coverage. RealPDEBench pairs measurements and simulations across multiple physical systems and supports evaluation on real-world data and transfer from simulation.

2 RELATED WORK

  • RealPDEBench distinguishes itself by pairing real-world measurements with numerical data across multiple physical systems for real-world evaluation and simulation-to-reality transfer.The paper reports more than 700 real-world physics experiments and numerical simulations.Source: §2 · p. 3
  • Prior benchmarks target particular scopes, including PDEs, fluid scenarios, inverse problems, or data quality and volume.PDEBench spans different PDEs, while the Well emphasizes high resolution and large data volume.Source: §2 · p. 3
  • Some works provide real observational data, but are not designed for machine learning and have limited data and sparse operating conditions.A concurrent benchmark, REALM, targets neural surrogates on multiphysics reactive flows in realistic regimes.Source: §2 · p. 3
  • Scientific ML baselines range from small models trained on single datasets to pretrained or foundation models trained on large-scale datasets.RealPDEBench can be used to verify these models' performance on real-world measurements.Source: §2 · p. 3

3 REALPDEBENCH: BRIDGING REAL-WORLD AND SIMULATED DATA

RealPDEBench organizes prediction of physical-system evolution around paired real-world and simulated data, with three training setups and evaluation on real-world data. It covers five physical scenarios and provides metrics and baseline models for comparing predictions.

3.1 TASK DEFINITION: ROLES OF REAL-WORLD AND SIMULATED DATA

  • The three tasks train on real-world samples, simulated samples, or simulated pretraining followed by real-world finetuning, using the same held-out real-world validation and test sets.The setup reflects scenarios where real-world data are scarce and simulated data are abundant.Source: §3.1 · p. 4

3.2 OVERVIEW OF DATASETS AND PDES

  • Five scenarios—Cylinder, Controlled Cylinder, FSI, Foil, and Combustion—span 736 real-world and simulated trajectories across fluid and combustion systems.Real-world fluid data are measured with PIV, combustion data with chemiluminescence imaging, and simulations are generated using CFD methods.Source: §3.2 · pp. 4–5

3.3 OVERVIEW OF METRICS

  • The benchmark defines nine metrics for the three tasks, grouped into data-oriented and physics-oriented measures.The metrics are intended to evaluate different aspects of scientific-ML predictions.Source: §3.3 · p. 5

3.3.1 DATA-ORIENTED METRICS

  • Data-oriented evaluation includes RMSE, MAE, and Relative L2 Error, R2, and an Update Ratio comparing the updates needed to reach the best real-world-training RMSE.The Update Ratio compares simulated pretraining with real-world finetuning against real-world training from scratch.Source: §3.3.1 · p. 6

3.3.2 PHYSICS-ORIENTED METRICS

  • Physics-oriented measures include Fourier-space error across frequency bands and Frequency Error, which compares spectra of spatially summed temporal signals.The Frequency Error uses a 1D FFT and mean absolute error between the predicted and target spectra.Source: §3.3.2 · p. 6
  • Kinetic Energy Error evaluates velocity fields, while Mean Velocity Profile Error compares time-averaged velocity at selected probes and is used for long-term autoregressive evaluation.Mean Velocity Profile Error is also used to compare simulated and real-world data.Source: §3.3.2 · p. 6

3.4 BASELINE MODELS

  • The baselines include a data-driven reduced-order model and neural operators, including pretrained DPOT models in 30M- and 509M-parameter sizes.The baseline set also includes architectures such as U-Net, FNO, DeepONet, and Transolver.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments find a substantial gap between simulated and real-world data, while simulated pretraining followed by real-world finetuning improves accuracy and convergence. Baseline strengths differ across local-error, periodicity, and long-term prediction measures.

4.1 EXPERIMENT SETUP

  • Training, validation, and test data are split by parameter level, with RMSE, Relative L2 Error, fRMSE, and Update Ratio reported across three training categories.Simulated-data training adds noise and randomly masks unmeasured modalities; autoregressive evaluation is included to assess long-term prediction.Source: §4.1 · p. 7
  • The experiments primarily compare baselines on real-world data, a setting that differs from most of their prior applications.Source: §4.1 · p. 7

4.2 GAP OF REAL-WORLD AND SIMULATION

  • Real-world training improved Rel L2 by 9.39% to 78.91% over simulated training when both were evaluated on the same real-world test data.Simulated training also produced much higher Frequency Errors than real-world training.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 IMPROVEMENT WITH SIMULATED DATA PRETRAINING

  • Real-world finetuning after simulated pretraining lowered errors versus training on the same amount of real-world data and reached optimal performance in fewer updates for most datasets and baselines.The authors attribute the accuracy improvement to broader simulated trajectories and dynamic information in unmeasured modalities; Figure 3b shows faster validation RMSE decline on Combustion.Source: §4.3 · pp. 8–9, Figure 3 · p. 9

4.4 COMPARISON OF BASELINES

  • In RMSE–Frequency Error comparisons, DPOT-L-FT is closest to the origin; U-Net and CNO favor lower RMSE, while MWT generally favors periodicity.The comparison uses RMSE for local pixel-level error and Frequency Error for global periodicity; proximity to the origin indicates better performance.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 OTHER ANALYSIS

  • On Cylinder, CNO’s error grows faster than other methods across autoregressive rounds; at 10 rounds, DPOT performs best and DMD has limitations under MVPE.The evaluation feeds each prediction back as input and measures error across the resulting NT steps.Source: Figure 3 · p. 9, §4.5 · p. 10, Figure 5 · p. 10
  • On Foil, CNO’s relative performance improves as frequency increases, a pattern that may relate to its design for eliminating aliasing errors.The analysis compares normalized fRMSE across low-, mid-, and high-frequency bands.Source: §4.5 · p. 10, Figure 6 · p. 10

5 CONCLUSION

RealPDEBench pairs real-world and simulated data for evaluating scientific ML on complex physical systems, and the authors present it as a step toward bridging the gap between them.

5 CONCLUSION

  • RealPDEBench is introduced as the first real-world scientific ML benchmark for predicting complex physical systems, with paired real-world and simulated data.Source: §5 · p. 10
  • The benchmark incorporates three task categories, nine evaluation metrics, and ten baselines.Source: §5 · p. 10
  • The authors hope the benchmark and datasets will spur methods that integrate real-world and simulated data more effectively.Source: §5 · p. 10

ETHICS STATEMENT

The benchmark uses data from physical experiments and numerical solvers, and the authors caution against direct safety-critical use without further validation.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The authors provide code, data, checkpoints, logs, documentation, and scripts to support reproducibility and validation of data acquisition and processing.

REPRODUCIBILITY STATEMENT

A OTHER EXPERIMENT RESULTS

A.1 OTHER METRICS

On additional metrics, U-Net and DPOT-L-FT perform relatively well overall, while Combustion remains more challenging.

A.1 OTHER METRICS

  • Combustion’s R2 is significantly lower than on the other datasets, and KE is not applied because its modality is light intensity.Source: §A.1 · p. 18
  • Across additional MAE, R2, KE, and FE metrics, U-Net and DPOT-L-FT perform relatively well overall, while Combustion is more challenging.The table presents results for these additional metrics; its caption notes that bolded values are best.Source: Table 2 · p. 18, §A.1 · p. 18

A.2 AUTOREGRESSIVE EVALUATION

The appendix reports long-horizon baseline behavior and frequency-resolved results, alongside dataset, data-collection, and measurement specifications.

A.2 AUTOREGRESSIVE EVALUATION

  • U-Net variants lead across most metrics and datasets; transformers are competitive short term but degrade faster with longer autoregressive horizons.Frequency errors increase markedly from two to three autoregressive steps; the tables report results for two and three rounds.Source: §A.2 · p. 19, Table 3 · p. 20, Table 4 · p. 21
  • In 10-round visualizations, velocity field u is easier to capture than v.Figures 7 and 8 show the respective predicted mean velocity profiles.Source: §A.2 · p. 23, Figure 7 · p. 23, Figure 8 · p. 24

A.3 LOW-, MID- AND HIGH-FREQUENCY FRMSE

  • On Controlled Cylinder and FSI, U-Net underperforms at low-frequency fRMSE but outperforms at mid- and high-frequency fRMSE.The reported comparison indicates stronger U-Net performance on higher-frequency information.Source: Table 6 · p. 25, §A.3 · p. 25

B.1 DATA FORMAT

  • The dataset overview tabulates trajectories, frames, resolution, memory, and modalities, including velocity, pressure, and combustion luminescence intensity.Simulated combustion data include additional chemistry, temperature, pressure, and velocity variables.Source: Table 7 · p. 26, §B.1 · p. 26

B.2 DATA COLLECTION

  • Simulations are paired to experimental parameter settings, with spatial domains and time steps adjusted for convergence and approximation; data are collected after stabilization.Source: §B.2 · p. 26
  • The five experimental regimes vary Reynolds number, with additional control frequency, FSI mass ratio, foil angle of attack, or combustion-mixture and equivalence-ratio settings.The Cylinder, Controlled Cylinder, FSI, Foil, and Combustion descriptions specify these operating parameters.Source: §B.2 · p. 27

B.3 DETAILS OF DATA MEASUREMENT

  • Time-resolved PIV produces velocity fields that are filtered for spurious vectors, with free-stream velocities checked against the tunnel’s calibrated bulk velocity.The measurements use different PIV processing software and interrogation-window settings for FSI versus the other fluid datasets.Source: §B.3 · pp. 29–30

B.4 COMPARISON BETWEEN SIMULATED AND REAL-WORLD DATA

The appendix compares simulated and measured data, describes combustion frequency analysis, and documents training, preprocessing, and optimization options.

B.4 COMPARISON BETWEEN SIMULATED AND REAL-WORLD DATA

  • 0.05650 is the simulated-data 200-step MVPE on Cylinder; every real-world-finetuned model has lower MVPE, while full-trajectory mean velocity is close to real data.Source: Table 9 · p. 30, §B.4 · p. 30, Table 8 · p. 30
  • Combustion frequency analysis compares measured OH chemiluminescence with an LES-derived signal after detrending and FFT to show dominant instability frequencies.The figure separates the real-world and simulated frequency analyses into panels (a) and (b).Source: §B.4 · pp. 30–31, Figure 12 · p. 32

C.1 SCRIPTS

  • Training-script flags distinguish numerical or real data and whether real-world training is finetuning; a separate command specifies a checkpoint path.Source: §C.1 · pp. 31–32

C.2 DATA PRE-, POST-PROCESSING, OPTIMIZER AND SCHEDULER

  • The framework supports no normalization, Gaussian standardization, or range normalization, with Adam and step-decay or cosine-annealing learning-rate schedules.Source: §C.2 · p. 32

C.3 DATASET MODULE

The dataset module supports configurable distribution and parameter-regime test selection, while a unified model class standardizes baseline operations.

C.3 DATASET MODULE

  • Dataset test modes select all, in-distribution, out-of-distribution, seen, or unseen parameter regimes for generalization evaluation.‘Seen’ uses regimes observed during training; ‘unseen’ enables testing on regimes absent from training.Source: §C.3 · pp. 32–33

C.4 MODEL MODULE

  • A unified Model class standardizes baseline forward passes, training-loss computation, and checkpoint loading.Source: §C.4 · p. 34

D EXPERIMENT DETAILS

The experiments specify strategies for handling modality differences and describe the traditional and learned baselines used to predict physical-system evolution. The methods span convolutional, operator-learning, transformer, and pretrained models.

D.1 MASK-TRAINING STRATEGY

  • Mask-training randomly hides simulated-only modalities so models can use them during training and accept real-world inputs with zero vectors for unobserved modalities.The authors state that training with the additional modalities lets models learn more information about the dynamics.Source: §D.1 · p. 34

D.2 COMBUSTION SURROGATE MODEL TRAINING

  • For Combustion, a U-Net surrogate maps simulated modalities to corresponding real-world modalities and supplies its outputs alongside simulated data during training.The surrogate is trained on paired simulated and real-world data from the same periods.Source: §D.2 · p. 34

E.1 DMD

  • DMD decomposes flow snapshots to extract coherent structures, then retains the 10 modes with largest amplitudes for prediction.Modal amplitudes are fitted to the initial condition; future states are predicted by linear superposition.Source: §E.1 · p. 34

E.2 U-NET; E.3 CNO

  • The U-Net uses an encoder-decoder with skip connections, while CNO uses spectrally motivated filters and a residual-block bottleneck to map between function spaces.The U-Net outputs target fluid variables; CNO's lifting and projection layers map fields into and out of a latent space.Source: §E.2 · p. 35, §E.3 · p. 35

E.4 DEEPONET; E.5 FNO

  • DeepONet combines branch-network input features with trunk-network coordinates, whereas FNO parameterizes an integral kernel in the Fourier domain.The benchmark's DeepONet uses a 3D CNN branch and fully connected trunk; FNO's design supports zero-shot super-resolution.Source: §E.4 · p. 35, §E.5 · p. 35

E.6 WDNO; E.7 MWT

  • WDNO combines wavelets with diffusion models to capture abrupt changes and model high-dimensional distributions, while MWT combines multiscale wavelet decomposition with sparse Fourier transforms.The MWT setup uses filter size 3, spectral width 5, channel multiplier 4, and four operator blocks.Source: §E.6 · p. 35, §E.7 · p. 36

E.8 GK-TRANSFORMER; E.9 TRANSOLVER

  • The Galerkin Transformer baseline replaces its original Fourier-transform output regressor with an FNO layer, while Transolver uses global self-attention to capture long-range dependencies.The Galerkin Transformer modification is described as enhancing volumetric feature capture and recovery of fine-scale structures.Source: §E.8 · p. 36, §E.9 · p. 36

E.10 DPOT

  • The benchmark uses pretrained DPOT models of 30M and 509M parameters, resizing data to 128×128 and tuning optimization hyperparameters while preserving model structure.Inputs and outputs are padded with ones for datasets with fewer than four channels; projection layers are reinitialized for datasets with more than four.Source: §E.10 · p. 36

F LIMITATIONS AND FUTURE WORKS

The authors identify scope, metric, and task boundaries for RealPDEBench and describe visualized baseline predictions and their use of language models. They propose expanding the benchmark and its evaluation in future work.

F LIMITATIONS AND FUTURE WORKS

  • RealPDEBench currently lacks systematic evaluation in strong out-of-distribution regimes, and no physics-oriented metric is dedicated specifically to Combustion.The authors also identify limited domain scope and propose adding electromagnetics, structural mechanics, and aerodynamics.Source: F LIMITATIONS AND FUTURE WORKS · p. 37

G VISUALIZATION

  • The visualization section presents one sample of ground truth and predictions for all baselines and datasets under real-world-data finetuning.The accompanying figures include results for u and v on Cylinder, Controlled Cylinder, FSI, and Foil, and I on Combustion.Source: G VISUALIZATION · p. 37, Figure 13 · p. 38, Figure 14 · p. 39

H THE USE OF LARGE LANGUAGE MODELS (LLMS)

Related papers

Loading 2601.01829v2…