PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Meets Physics: Data-Efficient Pre-Training for Unstructured Neural PDE Solvers

Luis Medrano-Navarro, Giacomo Baldan, Qiang Liu, Benjamin Holzschuh, Jan Hagnberger, Mathias Niepert, Nils Thuerey

arXiv:2610.03363v3cs.AI

TL;DR

Unstructured PDE surrogate pre-training often depends on expensive, large offline datasets. GMP instead generates geometry or PDE supervision online for steady-state and transient problems, and reports faster convergence, greater data efficiency, and higher fine-tuning accuracy. Its lightweight setup is presented as suitable for standard pre-processing, though overfitting remains a challenge in highly complex, low-data settings.

  • Problem

    Pre-training for unstructured PDE surrogates still depends heavily on costly offline datasets, while many approaches target structured grids without geometric variation.

    Source: §1 · p. 1

  • Method

    GMP uses online geometric descriptors for steady-state pre-training and online canonical PDE solutions for transient pre-training, without storing large precomputed datasets.

    Source: §1 · p. 2

  • Results

    GMP reports faster convergence, greater data efficiency, and higher downstream accuracy, particularly in low-data regimes.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The computationally lightweight pre-training setup can serve as a standard pre-processing step without substantially increasing total training time.

    Source: §5 · p. 10

    Overfitting remains a challenge in highly complex, low-data scenarios.

    Source: §5 · p. 10

Abstract

from arXiv · show

Neural surrogate models for Partial Differential Equations (PDEs) on unstructured 3D geometries are often limited by poor generalization and the high cost of generating large-scale training datasets. Consequently, pre-training on massive datasets of related PDE dynamics has emerged as a critical alternative to enhance the robustness and scalability of these models. However, this strategy is neither compute- nor data-efficient, as it relies on massive pre-computed data that is very costly to generate. In this work, we introduce a disk-data-free pre-training framework tailored to both steady-state and transient regimes. For steady-state problems, we propose a geometry-driven strategy that leverages intrinsic shape descriptors to learn representations of complex 3D domains. For transient problems, we introduce a physics-driven approach based on online generation of synthetic PDE data, enabling scalable pre-training without reliance on expensive datasets. Across multiple experiments, our approach achieves faster convergence, greater data efficiency, and higher accuracy during fine-tuning, particularly under realistic low-data regimes. This methodology provides a practical pathway toward data-efficient neural emulators for large-scale simulations. Code and models are available at https://github.com/tum-pbs/GMP.

1 Introduction

Unstructured PDE pre-training is constrained by the cost of generating large offline datasets, motivating a storage-free approach that treats steady-state and transient problems differently. GMP pairs geometry-driven and physics-driven pre-training and reports improved fine-tuning performance, especially in low-data regimes.

1 Introduction

  • Existing PDE pre-training still relies on costly offline datasets, leaving efficient pre-training for unstructured simulations without precomputed data an open question.The introduction also notes that most existing approaches target structured grids without geometric variation.Source: §1 · p. 1
  • Steady-state fields are determined by geometry and operating conditions, whereas transient dynamics also depend on prior physical states.This distinction motivates separate geometry- and physics-driven strategies.Source: §1 · pp. 1–2
  • GMP generates supervision online: random geometric primitives provide descriptors for steady-state pre-training, while canonical PDE solutions provide transient training data.The geometric targets include surface normals, curvature, and Vector Distance Fields.Source: §1 · p. 2
  • GMP is a modular, architecture-agnostic framework that avoids storing precomputed PDE solutions and expensive CAD geometries, with reported gains in convergence, data efficiency, and fine-tuning accuracy.The reported benefits are particularly pronounced in low-data regimes.Source: §1 · p. 2

2 Related work

Neural PDE surrogates have progressed from structured-grid models to architectures for unstructured meshes, while scalability and training-data requirements remain challenges. Prior pre-training approaches address parts of this landscape, but the passage identifies a gap in modular, storage-free pre-training for unstructured simulations.

2 Related work

  • CNNs [19], DeepONet, and Fourier Neural Operators [22] are among earlier neural PDE approaches, while Transformers have also been adapted to PDEs [13] [27] [28] [29].Conventional CNNs and vision Transformers remain constrained to structured grids.Source: §2 · p. 2
  • GNNs support irregular domains but faced scalability bottlenecks on large meshes; later mesh and point-cloud models use local neighborhoods, anchor points, or sliced attention to address attention costs.The passage identifies standard attention's quadratic complexity O(N^2) as prohibitive for high-resolution 3D simulations.Source: §2 · pp. 2–3
  • Physics pre-training includes online spectral-solver data limited to regular grids [11] and geometry-focused methods such as GeoPT [16] and occupancy prediction, but the passage identifies no prior modular framework for storage-free unstructured pre-training.The cited approaches also tend to target autoregressive rollouts rather than direct geometry-to-physics mappings.Source: §2 · p. 3

3 Method

The method pairs a unified surrogate formulation with distinct online pre-training strategies for steady and transient problems. It then fine-tunes the resulting representations and uses a modular architecture across downstream engineering tasks.

3 Method

  • The unified formulation treats transient prediction as temporal evolution and steady-state prediction as a direct mapping from geometry and conditions to fixed-point fields.The surrogate maps discretized inputs to the next-time-step fields, while steady-state solutions are treated as fixed points.Source: §3 · p. 3

3.1 Geometric pre-training

  • Randomized assemblies of cubes, spheres, and tetrahedra provide geometries for learning surface normals, curvature, and volumetric Vector Distance Fields.For continuous-output architectures, masked operator regression predicts descriptors at disjoint query points; smooth, intersection-free Gaussian-deformed grids provide volumetric discretizations.Source: §3.1 · p. 4

3.2 Physics pre-training

  • Online spectral solvers generate canonical PDE trajectories for masked autoencoding, avoiding storage of precomputed simulations during transient pre-training.A snapshot is sampled, interpolated at surface and deformed-volume coordinates, and reconstructed at disjoint query points.Source: §3.2 · p. 5

3.3 Fine-tuning

  • Fine-tuning evaluates pre-trained models against from-scratch baselines; transient models move from masked reconstruction to single-step dynamics.For steady-state tasks, models map geometry directly to target fields; the protocol initializes projection layers from scratch and initially freezes backbone weights.Source: §3.3 · p. 5

3.4 Architecture

  • GMP-Net combines geometry, surface-state, and volume-state encoders, with query-aware decoders and learned weights to predict the next state.Branches can be activated according to problem requirements, enabling separate geometric and online physics pre-training; steady-state use activates only the geometry branch.Source: §3.4 · p. 6

3.5 Downstream application datasets

  • The downstream evaluation uses unseen engineering benchmarks spanning steady aerodynamics and structural mechanics, plus synthetic and established transient tasks.Steady-state datasets include DrivAerML and SHIFT-Wing; transient tests include Kuramoto–Sivashinsky solutions, flow past an ellipse, and SHIFT-Crash rollouts.Source: §3.5 · pp. 6–7

4 Experiments

Pre-training improves fine-tuning accuracy, convergence, and data efficiency across steady-state and transient tasks, with gains especially visible in low-data settings. The experiments also compare pre-training cost and geometric descriptor choices.

4.1 Results for steady-state problems

  • 47% maximum accuracy improvement for GMP-Net and Transolver++ at 16 training samples; GMP-Net achieves the lowest errors overall.Across the 144-sample evaluation setup, AB-UPT shows no measurable gain in highly data-sparse regimes because of severe overfitting.Source: §4.1 · p. 7, Table 3 · p. 8
  • Fine-tuning with 16 samples outperforms training from scratch with 32, while pre-training also speeds convergence and improves lift-coefficient agreement, especially in low-data regimes.The convergence comparison is on DrivAerML; the aerodynamic-coefficient analysis concerns CL and CD.Source: §4.1 · p. 7, Figure 7 · p. 8, Figure 8 · p. 9
  • 4 hours of geometric pre-training on one NVIDIA A100 compares with 80 hours to train SMART on SHIFT-Wing from scratch for 200 epochs.The authors describe pre-training as a light preprocessing step.Source: §4.1 · p. 7
  • GMP pre-training matches GeoPT in low-data regimes except on SHIFT-Crash, and outperforms it as downstream training data grows.Under identical hardware, GeoPT takes about 12 hours per epoch and about 10 days to reach test MSE 0.02; the comparison uses controlled reimplementations.Source: §4.1 · p. 8, Table 3 · p. 8
  • Surface normals and curvature improve downstream transfer more than VDF alone; binary-occupancy pre-training shows no improvement in GMP.The authors connect differential surface geometry to resolving steep aerodynamic boundary-layer gradients.Source: §4.1 · p. 8, Table 4 · p. 8

4.2 Results for time-dependent problems

  • 10–25% lower rollout error across transient scenarios with pre-training, including transfer from 3D single-channel pre-training to 2D multi-channel tasks.GMP-Net from scratch already has the lowest error among evaluated baselines for 2D KS and 3D SHIFT-Crash, and is comparable to Transolver++ on 2D Ellipse.Source: §4.2 · p. 9, Table 5 · p. 9, Figure 10 · p. 10
  • Around 30% relative 1-step NMSE gains persist across 16, 32, and 64 training trajectories, with larger absolute gains in smaller-data regimes.The comparison averages final test loss across three datasets.Source: §4.2 · p. 9, Table 6 · p. 9

5 Conclusions, limitations and future work

GMP combines online physics-data generation with geometric descriptors to improve downstream accuracy, data efficiency, and convergence. The conclusion highlights unresolved overfitting and descriptor-selection questions.

5 Conclusions, limitations and future work

  • GMP improves downstream accuracy, data efficiency, and convergence, and its lightweight pre-training can be integrated as preprocessing without a substantial increase in total training time.The authors frame the resulting benefit as amortizing initial computational costs through faster downstream convergence and reduced data requirements.Source: §5 · p. 10
  • Highly complex, low-data scenarios remain vulnerable to overfitting.The authors also identify geometric descriptor selection and its potential effect on robustness and generalization as open questions.Source: §5 · p. 10

Technical appendices and supplementary material

The appendices collect ablations and supporting implementation and evaluation material. The point-resolution study weighs downstream accuracy gains against pre-training cost.

Technical appendices and supplementary material

  • The supplementary material lists ablations, GeoPT comparison details, hyperparameters, baseline comparisons, coefficient computation, mesh generation, regional results, and additional plots.These topics are organized across appendices A–I.Source: Technical appendices and supplementary… · p. 15

A Ablation studies for pre-training

A.1 Number of points during pre-training

  • Increasing the pre-training point budget yields diminishing downstream accuracy gains while adding prohibitive computation; the default uses an intermediate resolution.Discretization invariance allows pre-training at lower spatial resolution than downstream fine-tuning, whose target meshes can have millions of vertices.Source: §A.1 · p. 15, Table 7 · p. 15

A.2 Loss weighting during pre-training

The appendix examines geometric-loss balancing and the scope of online physics pre-training, then studies model-size and point-density effects during fine-tuning.

A.2 Loss weighting during pre-training

  • Geometric pre-training jointly predicts curvature, surface normals, and vector distance fields, with dynamic weighting or ConFIG used to manage competing objectives.Dynamic relative weighting uses running loss magnitudes; ConFIG projects task gradients toward a unified, conflict-free descent direction.Source: §A.2 · p. 15
  • ConFIG pre-training yields the lowest downstream prediction errors, while its loss components generally converge better than with the alternative weighting strategy.The comparison is evaluated on DrivAerML with 64 samples.Source: §A.2 · p. 16, Table 8 · p. 16, Figure 11 · p. 16

A.3 Family of PDEs

  • Online PDE pre-training selects equations and initial conditions to span spatial features and wavenumbers, using single-channel crops to remain system-agnostic.The selected features include smooth diffusive gradients, sharp shocks, and multiscale turbulent vortices.Source: §A.3 · p. 16

A.4 Physics pre-training results

  • The self-supervised masked reconstruction objective reaches NMSE on the order of 10^-4 for single snapshots, before fine-tuning to learn dynamics between time steps.The online synthesis setup has no static validation set; the reported reconstruction result concerns the pre-training autoencoding objective, not dynamics.Source: §A.4 · p. 16

A.5 Model size

  • Erwin is more susceptible to overfitting as capacity rises, whereas SMART scales consistently, with only marginal gains at its largest sizes.The study evaluates SMART variants with 11 M, 39 M, and 79 M parameters and Erwin variants with 2 M, 8 M, and 30 M parameters.Source: §A.5 · pp. 16–17, Figure 13 · p. 17, Figure 14 · p. 17

A.6 Number of points during fine-tuning

  • Increasing SMART training density from 2^13 to 2^16 points improves test nMSE by 3% and reduces point-wise error standard deviation by 14%.The nMSE is evaluated on entire samples; higher density is associated with greater local stability and fewer high-error outliers.Source: §A.6 · p. 18, Figure 15 · p. 18

B More on the comparison against other pre-training strategies

The comparison argues that static geometric pre-training can improve downstream fine-tuning without synthetic dynamics or high-fidelity CAD models. It highlights enriched descriptors, conflict-aware optimization, and staged adaptation, while noting limits to direct comparison with published GeoPT results.

Geometric pre-training comparison

Geometric descriptors

  • Adding explicit surface normals and curvature to signed distance fields or binary occupancy improves representation quality and downstream transfer.Binary occupancy-only pre-training showed no improvement in GMP and was omitted from the final variants.Source: B More on the comparison against other… · p. 18

Conflict-free optimization

  • ConFIG addresses negative transfer between geometric pre-training objectives that scalarized loss balancing does not fully exploit.The comparison concerns conflicting gradient directions across objectives.Source: B More on the comparison against other… · p. 18

Comparison scope

  • The GeoPT comparison uses a controlled reimplementation, but its original published numbers are not directly comparable because datasets and other experimental settings differ.Only DrivAerML is the same dataset across the compared scenarios; model sizes, splits, predicted fields, and hyperparameters may also differ.Source: B More on the comparison against other… · p. 18, Table 9 · p. 19

C Training hyperparameters

The appendix documents physics and geometric pre-training settings, benchmark configurations, and model-specific baseline hyperparameters to support reproducibility.

C Training hyperparameters

D Baselines hyperparameters

E Base architecture selection

The architecture study selects SMART for its accuracy across steady-state datasets despite efficiency advantages of other models. The appendix also describes how predicted surface fields are converted into aerodynamic forces and compares pre-trained and scratch-trained lift predictions.

E Base architecture selection

  • SMART shows superior accuracy across both datasets and surface and volume fields, making it the selected backbone despite Erwin and Transolver++ having better computational efficiency.The model comparison uses approximately 10M parameters and evaluates DrivAerML, SHIFT-Wing, and SHIFT-Crash; Table 20 reports averaged relative L2 errors.Source: E Base architecture selection · pp. 23–24, Table 20 · p. 23, Table 24 · p. 24

F Aerodynamic forces calculation

  • Predicted pressure and wall shear stress are integrated over surface cells by quadrature to obtain total aerodynamic force; drag and lift are its horizontal and vertical components.For aerospace applications, the angle of attack is also accounted for.Source: F Aerodynamic forces calculation · p. 24
  • For SHIFT-Wing, GMP pre-training improves lift-coefficient prediction over training from scratch at both 32 and 256 training samples, though 32 samples remain too sparse for accurate forces.Figure 16 compares the test-sample lift coefficients for the two sample counts.Source: F Aerodynamic forces calculation · p. 24, Figure 16 · p. 25

G Intersection-free meshes via Gaussian random fields

The mesh construction uses smooth Gaussian random fields to deform a Cartesian grid while preserving valid, non-overlapping geometry. A global Lipschitz constraint and rescaling provide geometric guarantees, and the resulting model is tested on larger deformations.

G Intersection-free meshes via Gaussian random fields

  • The construction normalizes the displacement to satisfy |Dd| < 1, yielding a globally injective deformation without self-intersections.The stated condition implies |λ_i| < 1 for Dd and det DΦ > 0.Source: G Intersection-free meshes via… · p. 25
  • A Cartesian grid is deformed by a smooth displacement field built from independent Gaussian random fields with a frequency-controlled spectrum.The deformation is Φ(x, y) = (x, y) + d(x, y).Source: G Intersection-free meshes via… · p. 25
  • The sampled displacement is rescaled using an estimated maximum Lipschitz constant, and deformation strength α is limited to at most 1 to keep the mesh intersection-free.The implementation samples α from [0.5, 1.0], uses ℓ = 0.1, and checks that each cell has positive signed area.Source: G Intersection-free meshes via… · pp. 25–26
  • Fine-tuning with α = 1.4 samples shows no degradation on the KS test set, which includes larger deformations than the training range.The authors interpret this as robust generalization to out-of-distribution deformations.Source: G Intersection-free meshes via… · p. 26

H Additional results

Additional experiments report results on complete DrivAerML data, regional aerodynamic errors, and qualitative plots across steady and autoregressive tasks. The reported gains include faster convergence and improvements in selected regional and rollout comparisons.

H Additional results

  • On complete DrivAerML data, pre-training still slightly improves accuracy, although gains are marginal for some models, and significantly accelerates convergence.The table reports relative L2 error and pre-training improvement across models.Source: Table 25 · p. 26, H Additional results · p. 26
  • With 64 training samples and GMP-Net, point-wise errors are substantially reduced in aerodynamically critical regions including side mirrors and front bumpers.Table 26 provides a regional breakdown for automotive and aerospace benchmarks.Source: H Additional results · p. 27, Table 26 · p. 27

I Extra plots

  • Figures 18, 21, and 24 report lower test error and faster convergence with pre-training for GMP-Net on DrivAerML, SHIFT-Wing, and SHIFT-Crash, respectively.Figures 18–26 show loss evolutions across model and dataset combinations.Source: Figure 18 · p. 27, I Extra plots · p. 27, Figure 21 · p. 28
  • For 2D Kuramoto–Sivashinsky rollouts, GMP improves on scratch training at all timesteps and GMP-Net has lower errors than Transolver++; ellipse and crash comparisons also favor GMP-Net over scratch.Ellipse comparisons against Transolver++ vary by sample and rollout stage, while the two crash examples report lower GMP-Net errors.Source: Figure 29 · p. 30, Figure 32 · p. 31, Figure 35 · p. 32

Related papers

Loading 2610.03363v3…