PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Aware Operator Transformer as an Efficient and Accurate Neural Surrogate for PDEs on Arbitrary Domains

Shizheng Wen, Arsh Kumbhat, Levi Lingsch, Sepehr Mousavi, Yizhou Zhao, Praveen Chandrashekar, Siddhartha Mishra

arXiv:2505.18781v4cs.LG

TL;DR

The paper addresses the need for accurate and efficient PDE solution operators on arbitrary domains, where existing models can trade accuracy against computational efficiency. It proposes GAOT, combining geometry-aware multiscale graph neural operator encoders and decoders with geometry embeddings and a vision-transformer processor. GAOT is reported to be significantly more accurate, robust, computationally efficient, and scalable across diverse PDE datasets, including state-of-the-art performance on three-dimensional industrial CFD benchmarks.

  • Problem

    Repeated numerical PDE solves are costly for many-query engineering tasks, while existing neural operators face limitations on arbitrary domains and an accuracy-efficiency trade-off.

  • Method

    GAOT combines multiscale graph neural operator encoders and decoders, geometry embeddings, and a patch-based vision-transformer processor for PDEs on arbitrary domains.

  • Results

    GAOT is reported to be significantly more accurate, robust, computationally efficient, and scalable across diverse PDE datasets, achieving state-of-the-art performance on three-dimensional industrial CFD benchmarks.

  • Takeaways & Limitations

    GAOT provides a neural operator framework designed to handle arbitrary point-cloud inputs and produce PDE solutions at query points while remaining computationally efficient and scalable.

Abstract

from arXiv · show

The very challenging task of learning solution operators of PDEs on arbitrary domains accurately and efficiently is of vital importance to engineering and industrial simulations. Despite the existence of many operator learning algorithms to approximate such PDEs, we find that accurate models are not necessarily computationally efficient and vice versa. We address this issue by proposing a geometry aware operator transformer (GAOT) for learning PDEs on arbitrary domains. GAOT combines novel multiscale attentional graph neural operator encoders and decoders, together with geometry embeddings and (vision) transformer processors to accurately map information about the domain and the inputs into a robust approximation of the PDE solution. Multiple innovations in the implementation of GAOT also ensure computational efficiency and scalability. We demonstrate this significant gain in both accuracy and efficiency of GAOT over several baselines on a large number of learning tasks from a diverse set of PDEs, including achieving state of the art performance on three large scale three-dimensional industrial CFD datasets.

1 Introduction

PDE surrogates are needed because repeated numerical solves are costly, while existing neural operators face an accuracy-efficiency trade-off on arbitrary domains. GAOT addresses this gap with geometry-aware graph encoders and decoders, a patch-based transformer processor, and efficiency-oriented implementation strategies.

  • Motivation: Repeated PDE solves make many-query tasks such as uncertainty quantification, control, and inverse problems computationally expensive.This motivates fast and efficient neural surrogates for PDE solvers.
  • Motivation: Many neural operators are restricted to Cartesian grids and therefore cannot directly handle most engineering and industrial systems with complex geometries.This creates a need for operators that learn PDEs on arbitrary domains represented as point clouds.
  • Motivation: Existing models exhibit an accuracy-efficiency trade-off: robust models such as RIGNO are not necessarily efficient or scalable, whereas efficient models such as GINO may lack accuracy.The paper motivates evaluating accuracy, computational efficiency, and scalability together.
  • GAOT: GAOT combines a multiscale attentional graph neural operator encoder, geometry embeddings, a patch-based transformer processor, and a MAGNO decoder for arbitrary-domain PDE learning.The encoder processes arbitrary point clouds into a coarser latent grid, while the decoder can generate solutions at query points.
  • Evaluation: GAOT is tested on 28 benchmarks spanning time-dependent and time-independent PDEs, varied geometries, and comparisons with widely used baselines.The evaluation includes regular grids, random point clouds, and highly unstructured adapted grids.
  • Evaluation: GAOT achieves state-of-the-art performance on the large-scale three-dimensional DrivAerNet++ automobile-aerodynamics benchmark and is tested on two further industrial datasets.The additional datasets are DrivaerML for automobile aerodynamics and NASA-CRM for aerospace applications.

2 Methods.

GAOT learns PDE solution operators from arbitrary-domain data by mapping inputs to latent representations, processing them globally, and decoding solutions at query points. Its architecture combines flexible latent domains with multiscale, geometry-aware encoding and transformer processing.

  • Problem Formulation: GAOT formulates operator learning by mapping PDE inputs, including coefficients, forcing, boundary values, and domain information, to the solution field.For time-dependent PDEs, the operator maps coefficients and initial data to solution trajectories from discretely sampled spatial and temporal data.
  • GAOT Model Architecture: The encode-process-decode architecture maps arbitrary point-cloud inputs to latent tokens, processes them, and predicts the solution at any query point.The latent domain may be a structured grid, a randomly downsampled point cloud, or a projected low-dimensional grid.
  • Encoder: MAGNO extends graph-neural-operator encoding with multiple neighborhood scales and attention-based aggregation to integrate information across spatial resolutions.Scale radii are selected as rm = smr0, while attention-based weights determine contributions from neighboring points at each scale.
  • Geometry Embeddings: Geometry embeddings augment coordinate information with local statistics and shape descriptors, providing the model with additional information about domain geometry.Descriptors include neighbor counts, distance statistics, and PCA features used to represent local shape anisotropy.
  • Transformer Processor: A transformer processor globally exchanges information among latent tokens, using a patch-based vision transformer for regular latent grids and a standard transformer for randomly sampled points.The regular-grid processor combines tokens into patches before attention and feed-forward processing.
  • Decoder: The MAGNO decoder reverses the multiscale aggregation around each query point to produce the predicted field, while time-dependent problems add current time and lead-time inputs.The decoder selects increasing-radius neighborhoods around the query point and applies corresponding attention-based graph operations.

3 Results.

GAOT is evaluated across diverse PDE benchmarks, emphasizing accuracy, robustness, computational efficiency, scalability, and industrial-scale applicability. It performs strongly across datasets while retaining advantages on large three-dimensional CFD problems and transfer settings.

  • Computational Efficiency and Scalability: GAOT has the highest training throughput and fastest inference latency, outperforming the second-most efficient model, GINO, by almost 50% and 15%, respectively.The comparison measures throughput in samples/s and latency in milliseconds on an NVIDIA GeForce RTX 4090 using float32 precision.
  • Computational Efficiency and Scalability: GAOT’s training throughput scales more favorably than the baselines as both input size and model size increase.The comparison includes GAOT, GINO, RIGNO, and Transolver.
  • Industrial scale 3D datasets: On DrivAerNet++, GAOT improves state-of-the-art performance for pressure and wall shear stress prediction, including approximately 30% greater accuracy for wall-shear-stress mean absolute error.The dataset contains approximately 500K points per car shape across 8K car geometries.
  • Industrial scale 3D datasets: GAOT is almost twice as accurate as GINO on wall shear stress for the 9M-point DrivAerML dataset and significantly outperforms GINO on the NASA-CRM industrial dataset.DrivAerML uses high-fidelity LES simulations, making the learning problem more physically detailed and computationally demanding than DrivAerNet++.
  • Generality, Generalization and Scaling: GAOT generalizes well across regular grids, sub- and super-resolution settings, unseen bluff-body shapes, and few-shot transfer learning.Fine-tuning provides an almost order-of-magnitude accuracy gain over training from scratch in the bluff-body transfer experiment.
  • Ablation studies: Ablation studies attribute GAOT’s performance to the combined effects of MAGNO encoder-decoder components, regular latent grids, statistical geometry embeddings, multiscale features, and the ViT processor.The studies also report robustness to latent-grid size and favor time-derivative marching with γ = 1 and δ = τ.

4 Discussion

GAOT is presented as an accurate, robust, efficient, and scalable neural operator for PDEs on arbitrary domains. Its evaluation spans diverse PDE datasets and includes large-scale industrial simulations.

  • GAOT combines a multiscale GNO encoder/decoder, geometric embeddings, and a vision transformer processor for arbitrary-domain PDEs.
  • The implementation is designed to improve computational efficiency and scalability while supporting point-cloud inputs and arbitrary query points.
  • GAOT is significantly more accurate, robust, and computationally efficient than many baselines across diverse two-dimensional PDE datasets.
  • GAOT achieves state-of-the-art performance on three large-scale three-dimensional industrial simulation datasets in automobile and aerospace applications.

Supplementary Material for: Geometry Aware Operator Transformer as an Efficient and Accurate Neural Surrogate for PDEs on

The supplementary material formalizes operator learning for time-dependent and steady-state PDEs, defining solution operators that map discretized inputs to continuous solution fields. It also describes the GAOT architecture and its geometry-aware, multiscale transformer components.

  • PDE formulations: The paper considers time-dependent PDEs with solutions, parameters, boundary values, initial conditions, and spatial differential operators on domains D.
  • PDE formulations: Steady-state PDEs are treated as time-independent problems, with solutions represented as limiting cases of time-dependent dynamics when applicable.
  • Solution operators: The solution operator maps initial data and parameter functions to time-dependent solutions, while the steady-state operator maps parameter or boundary data to a solution field.
  • Operator-learning task: The operator-learning task approximates these solution operators from discretized data pairs without repeatedly using expensive high-fidelity numerical solvers.
  • GAOT architecture: GAOT uses geometry-aware latent tokens, multiscale attentional graph neural operators, and transformer processing to represent discretized PDE data on arbitrary domains.

B.1 Choice of Latent Grid

GAOT primarily uses a structured stencil grid for latent tokens, balancing geometric coverage and efficient patching against dimensionality-related costs. Alternative strategies trade adaptive token allocation or reduced 3D token counts for patching difficulty or projection error.

  • Structured Stencil Grid: Structured stencil grids provide fine coverage and support patching, which reduces the token count before transformer processing.Large patches can speed training with negligible observed performance effect.
  • Structured Stencil Grid: Structured-grid token counts grow exponentially with dimension, making dense 3D grids potentially prohibitively large.Tokens may also be underutilized when inputs lie on a lower-dimensional manifold.
  • Downsampled Unstructured Points: Downsampled unstructured points allocate tokens adaptively to denser regions and avoid empty-region tokens.However, unstructured tokens are harder to patch effectively and can underperform structured grids even for partially low-dimensional domains.
  • Projected Low-Dimensional Grid: Projected low-dimensional grids drastically reduce 3D token counts while retaining structured-plane patching.Projection can lose local-neighborhood information and introduce approximation errors that degrade accuracy relative to direct strategies.
  • GAOT Choice: GAOT mainly adopts structured stencil grids because of their robustness and simplicity, observing fast convergence and strong generalization despite empty tokens.The model also supports latent-domain choices including downsampled point clouds and projected low-dimensional grids.
  • Supporting Components: Multiscale attentional graph operators aggregate neighborhoods across multiple radii with attention-based quadrature weighting, while geometry embeddings encode local shape information.Scale-adapted neighborhoods capture fine details at small radii and broader context at large radii.

D Datasets

The benchmarks span diverse PDE types, geometries, discretizations, and flow conditions, including newly introduced datasets for circular Poisson problems and compressible flow around varied bodies. The datasets also include large, highly unstructured adaptive meshes and industrial-scale settings.

  • The study evaluates 28 benchmarks covering time-independent and time-dependent PDEs, regular grids, random point clouds, and highly unstructured adapted grids.
  • The dataset suite includes hyper-elasticity, Poisson, incompressible Navier–Stokes, compressible Euler, and wave-equation problems under varied conditions and geometries.
  • Poisson-C-Sines defines operator learning from finite-element source projection coefficients to nodal solution values on a circular domain.
  • The airfoil dataset varies Mach number, angle of attack, and 500 perturbations across NACA2412, NACA0012, and RAE2822 shapes, using anisotropic adaptive refinement for shock resolution.
  • The bluff-body dataset covers varied geometries and compressible Euler flows at 0.3 ≤Ma ≤1.3 and 0.5°≤α ≤15.0°, using pseudo-steady finite-time solutions.
  • Runtime comparisons note that theoretical costs may diverge from actual hardware performance because graph operations use sparse, irregular GPU memory access.

E.2 Runtime Profiling

Runtime profiling compares GAOT and GINO using component-level timing and memory considerations, while defining accuracy and efficiency metrics for broader model evaluation. GAOT reduces graph-processing overhead and addresses arbitrary-geometry batching bottlenecks through cached graphs and hybrid execution.

  • The comparison uses batch size 1 on NACA0012 for component-level timing, with GINO at 6.07M parameters and GAOT at 5.62M parameters.
  • GINO takes 13.79 ms per sample, with encoder and decoder graph building each consuming almost 2 ms.
  • GAOT precomputes graphs, loads them from cache, sequentially processes encoder and decoder components, and batch-processes the transformer processor to avoid memory overflow.
  • Accuracy is measured with relative L1 error, using medians across samples and components and the final snapshot for time-dependent tasks.
  • Training throughput counts samples processed per second across forward, backward, and update steps, with measurement batch sizes selected below the maximum OOM-free size.
  • Inference latency is the batch-size-one forward-pass time, and computational metrics are averaged over 100 measurements on one NVIDIA-4090 GPU.

E.4 Results for Radar Chart in Main Text.

The radar-chart analysis combines accuracy, robustness, throughput, latency, scalability, and peak memory to compare models across dataset categories. GAOT’s patching reduces effective latent-token counts, while GAOT-8 improves computational performance without substantial accuracy degradation.

  • The radar chart reports accuracy, robustness, training throughput, inference latency, input scalability, and model scalability across time-independent and time-dependent datasets.
  • GAOT exhibits the lowest peak GPU memory usage among compared models, with scaling plots showing memory behavior as input grid and model size increase.
  • Input scalability is evaluated at 50,000 input points, while model scalability is evaluated at approximately 70 million parameters.
  • The scalability evaluation points were limited by single-4090 benchmarking constraints for RIGNO-18 and Transolver, although GAOT can scale beyond those tested limits.
  • GAOT-8 uses transformer patch size 8 instead of the default size 2 and achieves enhanced computational performance without substantial accuracy degradation.

E.5 Regular Grid Dataset

GAOT performs strongly on structured-grid, time-dependent PDE datasets in comparison with several neural-operator baselines. It ranks among the top two on six of seven datasets and ranks first on five.

  • GAOT is compared with RIGNO-18, RIGNO-12, CNO, scOT, and FNO on time-dependent PDE datasets with regular-grid inputs.
  • GAOT ranks within the top two on six of seven structured-grid benchmark datasets and achieves first place on five.
  • On Cartesian datasets, ViT alone is comparable to or slightly better than GAOT in some cases, while GAOT is significantly superior in many others.

E.6 Model and Dataset Scaling

GAOT scaling experiments show that increasing model capacity and training data generally improves accuracy, while resolution-invariance tests demonstrate strong spatial generalization. Larger models require more computation, and full-resolution training can still improve accuracy.

  • Model Size: CE-Gauss error falls from 48.4% to 6.88% as THS increases from 32 to 256, while epoch time rises from roughly 84 to 143 seconds.The comparison fixes TL=5 and LC=32.
  • Model Size: LC=32 consistently performs strongly, whereas LC=16 slightly degrades performance and LC=64 or 128 provides only marginal gains.The authors interpret LC=32 as sufficient for capturing encoder-level geometry information.
  • Model Size: CE-Gauss error decreases from 25.0% at TL=1 to 5.28% at TL=10, while the model reaches 11.2M parameters and 260-second epochs.At TL=3, the error is 9.0%.
  • Data Size: Errors consistently decline as training trajectories or static samples increase, often with near-linear scaling for fluid datasets.Fluid experiments use 128–1024 trajectories, while static tasks use up to 2048 samples, with elasticity limited by data availability.
  • Resolution Invariance: GAOT trained at 2048 points achieves its best performance when tested at 8192 points, outperforming baselines trained and tested at 8192 points.This supports resolution-invariant prediction across discretizations.
  • Resolution Invariance: Training with only 10% of input points and testing at full 500K-point resolution is less accurate than fully trained GAOT, although it still outperforms several baselines.Neural-field evaluation therefore does not guarantee accuracy retention under coarse training.

E.9 Training at full vs. low Resolution for the DrivAernet++ Dataset.

The DrivAerNet++ comparison examines whether low-resolution training can support full-resolution inference. The results favor full-resolution training for accuracy, while warning that neural-field capability alone does not ensure performance retention.

  • Comparison: Full-resolution-trained GAOT is twice more accurate than all other models in the reported DrivAerNet++ comparison.Its performance is also on par with models trained and tested at low resolution.
  • Scope: Neural-field capability does not necessarily preserve performance after low-resolution training, and possible accuracy degradation requires further investigation.The passage explicitly identifies this issue as unresolved.
  • Dataset: The evaluated geometries include diverse bluff-body shapes and contact-surface orientations used in the Bluff-Body benchmark and transfer-learning fine-tuning.The shapes vary by semicircle, cone, and rectangle configurations.

E.11 Training Randomness

The supplementary experiments assess training variability, solver runtimes, architectural alternatives, tokenization, and latent-token scaling. Together, they characterize GAOT’s computational behavior and robustness across PDE benchmarks.

  • Training Randomness: GAOT’s six Bluff-Body training runs have a relative L1 test-error standard deviation of 0.12 across random seeds.Each run uses a different pseudorandom-number-generator seed.
  • Runtime Comparison: GAOT inference takes 8.95–10.14 ms on 2D datasets and around 365 ms on 3D DrivAerNet++, yielding speedups of 1–5 orders of magnitude and up to 10 orders of magnitude.DrivAerML graph construction substantially increases runtime; excluding it, inference is 446.06 ms.
  • Runtime Comparison: For NACA0012, coarsening the classical solver reduces runtime from 7 minutes to approximately 1 minute, while GAOT remains approximately 10 ms and achieves a 6000 speedup.The coarsened solver error is 4.5%, compared with GAOT’s 6.8%.
  • Architecture Comparison: Across six PDE benchmarks, GAOT is compared with RANO, RFNO, and GINO using median relative L1 errors.The encode–process–decode variants differ in their graph, Fourier, and transformer components.
  • Tokenization: Strategy I achieves the best tokenization performance on Elasticity and Poisson-C-Sines, while Strategies II and III can fail to converge on Poisson-C-Sines.The strategies perform similarly on the simpler Elasticity dataset.
  • Token Scaling: Increasing latent-token counts generally improves accuracy and convergence, but transformer attention costs rise quadratically with token count.Larger patch sizes reduce transformer tokens and training time without necessarily degrading performance.

F.3 Time-Stepping Method

GAOT supports output, residual, and derivative time-stepping formulations for time-dependent PDEs. Derivative stepping generally produces the lowest final-time errors and is adopted as the default.

  • Performance: Derivative time stepping yields the lowest final-time errors on five of six fluid benchmarks, while Output performs slightly better on CE-RP.The comparison reports median relative L1 errors at final time t14.
  • Formulations: GAOT can approximate the output, residual, or time derivative by selecting different γ and δ values in its time-stepping formulation.The paper uses derivative stepping by default for time-dependent PDE experiments.

F.4 Geometric Embedding

The ablations evaluate geometric embeddings and multiscale neighborhood features across unstructured and industrial datasets. Explicit geometry encoding and multiscale neighborhoods improve performance, while encoder-only geometry embedding offers a favorable accuracy–compute trade-off for industrial benchmarks.

  • Geometric embedding approaches: Statistical geometric embedding consistently outperforms PointNet on irregular meshes, while PointNet training appears unstable on Poisson-C-Sines.Poisson-C-Sines reaches 23.7% error with PointNet.
  • Geometric embedding configurations: Geometric embeddings improve DrivAerNet++ and DrivAerML performance relative to omitting them.The ablation compares no embedding, encoder-only embedding, and encoder-plus-decoder embedding.
  • Geometric embedding configurations: Encoder-only geometric embedding achieves better performance with less computational effort than embedding in both encoder and decoder.On DrivAerML, decoder embedding degrades performance, which the authors associate with increased overfitting risk near final predictions.
  • Multiscale features: Multiscale neighborhoods reduce mean relative L1 error on unstructured datasets compared with a single fixed radius.The multiscale configuration uses radii [0.022, 0.033, 0.044], whereas the single-scale configuration uses radius 0.033.
  • Multiscale features: 4.66% to 3.04%: Poisson-C-Sines error decreases when single-scale features are replaced by multiscale neighbors.Multiple radii help accommodate varying mesh density on unstructured domains.

G Visualizations of Datasets

This section visualizes model inputs, ground-truth solutions, and model estimates across diverse structured, unstructured, and industrial PDE datasets. The visualizations also include point distributions for several airfoil and bluff-body datasets and compare temporal predictions at later times for unstructured datasets.

  • Dataset visualizations: Figures G.1–G.3 visualize inputs, ground-truth solutions, and model estimates for Poisson-C-Sines, Poisson-Gauss, and Elasticity test samples.
  • Dataset visualizations: Figures G.4–G.6 and G.7 show inputs, ground-truth solutions, model estimates, and point distributions for airfoil and bluff-body datasets.The airfoil datasets are NACA0012, NACA2412, and RAE2822; the bluff-body dataset is shown separately.
  • Industrial CFD visualizations: Figures G.8–G.13 visualize model estimates against ground-truth solutions for surface pressure, wall shear stress, or friction coefficient on DrivAerNet++, DrivAerML, and NASA-CRM.
Loading 2505.18781v4…