PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysicsBench: A Unified Leaderboard for Generative and Predictive Models in Engineering Design and Simulation

Sang Won Lee, Hyogu Jeong, Namwoo Kang

arXiv:2608.24056v1cs.LGcs.CE

TL;DR

Engineering AI models are evaluated in isolation, often on unconstrained academic scales with inconsistent procedures, limiting evidence for realistic model selection. PhysicsBench standardizes evaluation across generative and predictive tasks and finds that the best model frequently changes with data scale, with no architecture leading across tasks.

  • Problem

    Engineering AI models lack a unified evaluation procedure spanning realistic limited-data scales, consistent metrics, and comparable generative and predictive tasks.

  • Method

    PhysicsBench evaluates generative and predictive models across seven 1D–3D tasks, controlled S–XL data scales, common metrics, and debiased BenchRank rankings.

  • Results

    The best model often changes with data scale, and no single architecture is best across the benchmark’s tasks.

  • Takeaways & Limitations

    Model recommendations should be conditioned on both the engineering task and the available data scale rather than assumed from a universal leader.

  • Takeaways & Limitations

    Two tasks use single baseline runs because rerun sets are incomplete, and the leaderboards provide no per-cell confidence intervals.

Abstract

from arXiv · show

Generative and predictive artificial intelligence models are increasingly used to generate geometry and to predict physical fields and scalar quantities in engineering design and simulation. Yet these models are typically evaluated in isolation, on academic datasets at unconstrained scales, with inconsistent metrics and procedures. We present PhysicsBench, a unified benchmark and leaderboard that evaluates generative and predictive models under one standardized procedure. PhysicsBench spans seven generation and prediction tasks across 1D, 2D, and 3D domains and ranks 66 models on nine datasets, comprising industrial-scale CAD/CFD/FEA simulations and public references, expanded into 28 configurations. One procedure and ranking apply to both families, each ranked within its own tasks. Evaluation spans realistic, limited data scales from S to XL rather than the unlimited training sets common in academic benchmarks. A common metric suite captures geometric fidelity with distributional distances, physical-field and scalar accuracy, and engineering-specific field- and shape-validity. BenchRank debiases correlated metrics and ranks by PageRank over a head-to-head dominance graph, so every reported quality metric is also ranked, with computational cost in a separate efficiency view. Across tasks, an architecture's large-scale academic standing weakly predicts its small-data ranking. The top model changes with data scale in six of the seven tasks, and no model leads more than one task. PhysicsBench turns "state-of-the-art" from a self-reported claim into an openly published foundation for model selection.

1 Introduction

PhysicsBench addresses fragmented engineering-AI evaluation with a unified leaderboard for generative and predictive models, using one standardized procedure across realistic, limited data scales. It combines broad task coverage, engineering-validity metrics, and deterministic BenchRank evaluation to support fair, reproducible model comparison.

  • Motivation: Existing engineering benchmarks are narrow, often simplified, and constrained by scarce high-fidelity data and non-standardized evaluation splits.These limitations can omit production-scale geometric complexity and hinder fair comparison.
  • Unified evaluation: PhysicsBench ranks generative and predictive models under one standardized procedure across industrial engineering datasets.Uniform training, inference, metric computation, and ranking isolate algorithmic differences from evaluation-setup differences.
  • Scope and scale: The benchmark spans seven 1D, 2D, and 3D tasks and evaluates controlled data scales from S to XL.The scale sweep exposes data-efficiency curves, small-data collapse, and best-model crossovers hidden by fixed-scale evaluation.
  • Ranking: BenchRank down-weights correlated metrics, applies PageRank to a head-to-head dominance graph, gates non-viable models, and aggregates datasets by geometric mean.The procedure is deterministic and reproducible.
  • Validity metrics: PhysicsBench ranks engineering-validity metrics alongside conventional errors, covering field-validity for prediction and shape-validity for generation.Field-validity includes modal assurance, sign, and extremal agreement; shape-validity includes manifold-Δ and uniformity-Δ scores.

2 Related Work

Related work covers generative models for engineering geometry and fields, neural surrogates for physical prediction, and benchmarks that standardize evaluation. PhysicsBench addresses the resulting fragmentation by connecting generative and predictive evaluation under shared comparison needs, especially in engineering’s small-data regime.

  • Generative models: Generative models produce engineering designs, physical fields, and 3D shapes using GANs, VAEs, diffusion models, implicit networks, normalizing flows, and point-cloud methods.Examples include DeepSDF and PointFlow for 3D generation.
  • Generative models: Academic shape benchmarks use FID, Chamfer distance, COV, and MMD to measure fidelity and diversity, but not valid, evenly sampled shape manifolds.PhysicsBench promotes this structural shape-validity test to a ranked evaluation component.
  • Predictive models: Neural surrogates replace expensive solvers by predicting physical fields or scalar quantities, with neural operators extending across discretizations and irregular engineering geometries.Representative methods include Fourier Neural Operator, DeepONet, Geometry-Informed Neural Operator, Transolver, and GNOT.
  • Benchmark fragmentation: Models are usually evaluated on separate datasets with method-specific metrics and training budgets, preventing cross-method and cross-dataset comparison and disconnecting generative from predictive research.Both model families serve the same engineering design loop despite this evaluation separation.
  • Small-data evaluation: Engineering operates in a small-data regime where industrial surrogates train on tens to a few hundred simulations and rankings are least stable, with the best model changing by data scale.The passage identifies high-fidelity CFD/FEA evaluation as the dominant cost of surrogate modeling.
  • Existing benchmarks: Existing engineering benchmarks standardize complementary areas, including time-dependent PDE simulations, RANS airfoil flow, automotive aerodynamics, and domain-specific 3D automotive design.Examples include PDEBench, The Well, AirfRANS, DrivAerNet++, DrivAerML, and CarBench.

3 Benchmark

PhysicsBench standardizes evaluation of generative and predictive engineering models across seven tasks, nine datasets, 28 configurations, and controlled data scales from S to XL. Its BenchRank leaderboard combines geometric, physical, and engineering-validity metrics through debiased pairwise ranking with safeguards against correlated metrics and degenerate models.

  • Scope and task design: PhysicsBench evaluates generation and prediction across 1D, 2D, and 3D engineering tasks using a unified input-to-output taxonomy and standardized procedure.The procedure fixes data, training, inference, metric computation, and ranking for every model; design optimization is outside scope.
  • Benchmark coverage: 66 publicly ranked models and the official AB-UPT are evaluated on nine source datasets expanded into 28 configurations across four controlled data scales from S to XL.AB-UPT is evaluated on DrivAerML and ranked in the paper, but withheld from the public leaderboard under its research license.
  • Evaluation protocol: Every model is measured under the same fixed-seed, per-cell protocol, with additional reruns at two seeds and task-level means providing the reporting basis for pairwise comparison.The benchmark avoids ranking averaged models against un-averaged models, preserving comparable run-to-run variability across competitors.
  • Metric suite: Five ranked engineering-validity metrics complement conventional pixel-wise and point-wise errors: sign agreement, extremal agreement, manifold-Δ, uniformity-Δ, and MAC.The first four are introduced by PhysicsBench, while Modal Assurance Criterion (MAC) is adapted from structural dynamics.
  • Ranking methodology: BenchRank uses a viability-gated pairwise dominance graph and re-normalized relative centrality instead of a single metric or averaged normalized score.The gate removes models that are median-absolute-deviation outliers at k=3 on all gated metrics; total scores place the top model at 100 and range from 32 to 40 here.

4 Results and Discussion

Results use a common single-GPU evaluation setup and BenchRank quality scores. Rankings vary with data scale and dataset, with no architecture best everywhere, while coverage differences require cautious interpretation.

  • Evaluation setup: The full suite trains and evaluates each model, dataset, and scale cell on a single NVIDIA RTX PRO 6000 with 96 GB memory.The 96 GB card accommodates every model because DoMINO exceeds the 80 GB capacity of an A100.
  • Ranking procedure: BenchRank filters nonviable models, debiases correlated metrics, applies direction-aware weighting, and ranks models through PageRank centrality.The viability gate excludes models worse than median_j + kMAD_j on every gated metric; scores are rescaled so the top model scores 100.
  • Main findings: Within tasks, the best model often changes with data scale, and no single architecture is best across the benchmark.Rankings reorder with scale and dataset even among models competing head-to-head.
  • Coverage and limitations: GeoFLARE’s 3D-field pooled rank reflects evaluation on DrivAerML only and narrower coverage than full-coverage baselines.Its results should therefore be interpreted per dataset; AB-UPT is excluded from the public leaderboard under a research license.

4.1 No Universal Winner

PhysicsBench finds no universal winner: no architecture wins across tasks, and the top model changes with training-set size in six of seven tasks, except 3D field prediction.

  • Cross-task leadership: No single architecture wins across tasks, so model leadership is task-specific.Table 5 reports the top-ranked model for each task at each data scale.
  • Scale dependence: 6 of 7 tasks change their top model as training-set size grows; 3D field prediction is the only exception.The result is reported across the task-by-data-scale rankings.
  • Ranking scope: The 1D scalar task is split into tabular regression on Concrete and Airfoil and time-series RUL on CMAPSS because the model sets are disjoint.A single pooled ranking would compare models that never competed head-to-head.

4.2 Data-Scale Crossovers and Small-Data Behavior

Controlled S-to-XL evaluations reveal data-scale crossovers, data-efficiency differences, and non-monotonic quality changes that fixed-budget evaluations can miss. Because some reversals persist across seeds and may reflect training instability, scale-dependent rankings are treated as indicative and affected cells are flagged.

  • Data-efficiency effects: Controlled S-to-XL scales expose data-efficiency effects that fixed-budget academic evaluations miss.In 3D geometry generation, DeepSDF is most data-efficient on DeepJEB and DeepWheel at smaller scales, but its cross-dataset FPD is highest at S because of collapse on non-watertight DrivAerNet cars.
  • Non-monotonic scaling: DeepSDF’s DeepJEB 3D FPD rises monotonically from 7.5 at S to 17.7 at XL.This reversal is interpreted as a symptom of training instability rather than data efficiency.
  • Non-monotonic scaling: GAN (Basic)’s DeepJEB 2D FID rises from 164.3 at M to 179.8 at L.The repeated 2D result at further seeds indicates a model property rather than a single random draw.
  • Non-monotonic scaling: 3D-GAN’s cross-dataset FPD rises from 38 at L to 100 at XL.Because more data should not degrade a well-trained generator, the reversal is treated as training instability.
  • Interpretation and limitations: Scale-dependent orderings are presented as indicative, with affected cells flagged rather than over-interpreted.The qualification follows repeated-seed evidence that at least one 2D reversal is a model property, while the broader reversals are read as instability symptoms.

4.3 Efficiency–Accuracy Trade-off · 4.4 Per-Task Results

The benchmark exposes task- and scale-dependent efficiency–accuracy trade-offs, while per-task results show substantial crossovers, failures, and dataset-specific differences. No single model dominates across the reported prediction and generation settings.

  • 4.3 Efficiency–Accuracy Trade-off: 0.8M-parameter Point Transformer-Small and 0.4M-parameter PCT-Small sit on or near the scalar-regression efficiency frontier, outperforming full-capacity counterparts in small-data settings.Parameter count is a poor utility proxy in this regime, so the efficiency view re-ranks models after accounting for cost.
  • 4.4.1 Field Prediction: GeoFLARE leads every scale on DrivAerML 3D field prediction, but its standalone coverage differs from models evaluated across all 3D-field datasets.GeoFLARE, GeoTransolver, DoMINO, and official AB-UPT are evaluated only on DrivAerML, whereas several other models cover all 3D-field datasets.
  • 4.4.1 Field Prediction: Ranks six through eight differ by at most 0.75% and 0.36% of total score, while models below the top-eight cut remain within 1.6 points of rank six.This near-tie can be reordered by metric reweighting, so the paper avoids interpreting an exact order within that band.
  • 4.4.1 Field Prediction: 0.93 to 0.98 R^2 on displacement contrasts with 0.76 to 0.83 on stress at scale XL, while every modal model has R^2 < 0 at all four scales.The results expose component-level differences and show that the modal task is never learned, regardless of data size.
  • 4.4.1 Field Prediction: Six of eight models reach R^2>0.9 on the DeepJEB displacement target at scale XL, whereas DPT-Hybrid reaches 0.37 after falling to R^2≈−8 at scale S.In 2D field prediction, five of eight models lead at some scale or dataset, with U-Net++, Attention U-Net, and FPN (ResNet-18) trading first place most often.
  • 4.4.2 Scalar Prediction: R^2≈0.60 at scale S makes a capacity-reduced Point-Transformer variant the 3D leader for both DeepWheel mass and modal, rising to 0.93 and 0.89 at scale XL.The intermediate scales remain the least settled portion of the scalar-regression trajectories.
  • 4.4.2 Scalar Prediction: 0.51 for XGBoost exceeds the from-scratch MLP and TabNet scores of 0.35 and 0.31 on Concrete, while pooled tabular leadership shifts from TabPFN to NODE at XL.On Airfoil, leadership rotates through Ridge, TabPFN, FT-Transformer, and MLP as data scale grows.
  • 4.4.3 Generation: A VAE ranks first at scale S across all three 2D image-generation datasets, but DDPM takes the top rank from M onward and at XL.DeepWheel remains difficult: the top-ranked model’s FID is ≈143 at XL versus ≈79 on DrivAerNet, reflecting domain difficulty.

4.5 Academic vs. Industrial Performance

Large-scale academic standing is a weak predictor of performance in the small-data, real-geometry regime faced by engineers. Academic results are used as qualitative references, and architectures leading academic leaderboards are often not the best choice at scale S.

  • 4.5 Academic vs. Industrial Performance: Academic standing weakly predicts performance in small-data, real-geometry engineering regimes.The benchmark takes academic standing from originating publications rather than re-measuring it.
  • 4.5 Academic vs. Industrial Performance: Academic standing is treated as a qualitative reference drawn from originating publications.
  • 4.5 Academic vs. Industrial Performance: Architectures that headline academic leaderboards are frequently not the best choice at scale S.The passage specifically includes diffusion models for generation among such architectures.

Anti-Trend Cases Are Genuine

Several models genuinely reverse the usual more-data-is-better trend, with failures tied to model behavior rather than pipeline artifacts. The audited cases include field-specific GLPN collapse, capacity–data mismatch in 3D scalar prediction, and instability or weight-clipping weaknesses in generation.

  • Anti-Trend Cases Are Genuine: GLPN produces negative R^2 on Darcy-pressure and AirfRANS eddy-viscosity fields at every scale, despite exceeding 0.9 on larger-scale AirfRANS velocity components.It remains mid-pack on DeepJEB structural channels, confirming field-specific behavior rather than a universal pipeline failure.
  • Anti-Trend Cases Are Genuine: On the smallest 3D scalar tasks, full-capacity Point Transformer and PCT models collapse toward mean prediction, while reduced Point Transformer dips at M before recovering.These failures indicate a capacity–data mismatch and motivate capacity-reduced variants.
  • Anti-Trend Cases Are Genuine: 3D-GAN degrades at the largest 3D scale from GAN training instability, while weight-clipping WGAN-CP trails at larger 2D scales.The latter reflects the documented weakness of weight clipping.

Worked Example

For the DeepJEB structural field problem with roughly 100 high-fidelity simulations, the dataset-specific 2D-field L leaderboard recommends FPN (ResNet-18). It ranks first in quality, ahead of Attention U-Net and U-Net, with its lead attributed to higher structural similarity.

  • Worked Example: With roughly 100 high-fidelity simulations, the DeepJEB structural field problem falls in the L data bucket.Practitioners consult the dataset’s own 2D-field L leaderboard.
  • Worked Example: FPN (ResNet-18) ranks first in quality, ahead of Attention U-Net and U-Net.FPN has 15.6M parameters, compared with 8.0M for Attention U-Net and 7.8M for U-Net.
  • Worked Example: SSIM 0.89 explains FPN’s lead through higher structural similarity.The example identifies FPN as the default recommendation, unlike the pooled winner reported in Table 5.

5 Conclusion

PhysicsBench provides a unified, standardized leaderboard for generative and predictive engineering models across tasks, domains, datasets, and controlled data scales. Its conclusions are bounded by incomplete reruns and absent confidence intervals, while future work targets broader coverage and task-aware model recommendation.

  • Contribution: PhysicsBench evaluates generative and predictive models under one standardized procedure across seven 1D, 2D, and 3D engineering tasks and controlled data scales.The benchmark combines industrial and public CAD/CFD/FEA data in a unified Engineering AI Leaderboard.
  • Contribution: BenchRank combines a common metric suite with debiased graph-based ranking, ranking every reported quality metric while reporting computational cost separately.This separates quality ranking from efficiency assessment.
  • Limitations: 5 of 7 tasks use means over repeated training runs, while 3D generation and 3D field prediction use a single baseline run because rerun sets are incomplete.The differing run bases can make closely spaced ranks reflect run-to-run noise.
  • Limitations: Leaderboards provide no per-cell confidence intervals, so closely spaced ranks and crossover narratives should be interpreted as indicative.The reported value is the mean over a cell’s repeated training runs, with the basis fixed per task rather than per model.
  • Future Work: Future work includes broader datasets and tasks, multi-channel prediction, geometry validity metrics, continuously updated rankings, and agentic model recommendation under task, dataset, and resource constraints.The agentic recommendation is identified as the most immediately promising direction.

Appendix A Metric Definitions

Appendix A defines PhysicsBench’s engineering-specific validity measures alongside standard generative-modeling, vision, geometry, depth, and reconstruction metrics. The definitions specify what each metric measures, its direction, and important complementarities or applicability limits.

  • Engineering-specific validity metrics: Sign Agreement measures the fraction of points where predicted and true fields share sign, indicating whether field loads point in the correct direction.It uses an Iverson-bracket indicator over points.
  • Engineering-specific validity metrics: Extremal Agreement measures recall of the top-10% largest-magnitude regions, assessing overlap between predicted and true stress or pressure hot spots.It uses k = ⌈0.1N⌉ and computes |T_k(ŷ) ∩ T_k(y)|/k.
  • Geometry metrics: Manifold-Δ and Uniformity-Δ separately assess local surface dimensionality and point-sampling evenness, because clean surfaces can still contain seam clumping and under-resolved regions.Uniformity-Δ is |c_gen − c_real| using the coefficient of variation of k-nearest-neighbor distances with k = 8; Manifold-Δ uses k = 20.
  • Field metrics: MAC measures sign- and scale-invariant alignment of predicted and reference vector or modal fields, while remaining undefined for scalar stress and pressure components.MAC = |φ̂·φ|²/(∥φ̂∥²∥φ∥²), and it complements Sign Agreement because it is sign-invariant.

Appendix B BenchRank Ranking Robustness

BenchRank rankings are largely robust to hyperparameter, run-averaging, normalization, and aggregation perturbations under the published task-seed basis and model pool. Run averaging changes one of 28 task-scale boards, while released per-run scores enable reproducibility except for withheld AB-UPT 3D-field rows.

  • Evaluation basis: BenchRank hyperparameters are fixed by construction, and robustness analyses use post-processing only, without retraining, on the published per-task seed basis and model pool.The pool adds official AB-UPT on DrivAerML to the published set.
  • Run averaging: 1 of 28 task-scale boards changes its top-ranked model when every task is recomputed using means across repeated runs.The change occurs in 3D generation at M, where the leading models move from 42.33% apart to within 0.38% after averaging.
  • Run averaging: Mean Kendall 𝜏-b is 0.97, with a worst case of 0.73, for pooled rankings recomputed from mean repeated-run results.The changed 3D-generation-at-M board does not reverse a decisive result but dissolves one.
  • Run-to-run variation: In 2D generation, FID’s run-to-run standard deviation is 3.9% of the cell value at the median and 10.2% at the ninetieth percentile.The leading model’s median score margin is 48.5%, compared with run-to-run score spreads of 1.2% at the median and 6.1% at the ninetieth percentile.
  • Alternative aggregators: BenchRank correlates positively but imperfectly with min–max normalized mean and Borda aggregation, at mean Kendall 𝜏-b values of 0.72 and 0.75.The comparison uses the same cross-dataset metric matrix; Elo-style aggregation was not tested.
  • Reproducibility and limitations: The analysis is reproducible from released per-run scores under the stated perturbations, except for AB-UPT rows on 3D-field boards withheld by its research license.The seed basis remains fixed per task rather than per model to avoid comparing averaged models with unaveraged competitors within one board.
Loading 2608.24056v1…