PaperlayerView exact revision on arXiv

Source-linked AI summary

Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries

Huakun Luo, Haixu Wu, Hang Zhou, Lanxiang Xing, Yichen Di, Jianmin Wang, Mingsheng Long

arXiv:2502.02414v2cs.LG

TL;DR

Industrial PDE simulations require accurate solutions on million-scale geometries, beyond the scale handled by previous neural solvers. Transolver++ adds eidetic states and an optimized parallel framework to Transolver, achieving gains across standard benchmarks and high-fidelity industrial tasks while scaling across GPUs. Its comparisons rely on splitting inputs for models that cannot directly process million-scale meshes.

  • Problem

    Previous neural PDE solvers were limited in handling the million-scale geometries needed for accurate industrial simulations.

  • Method

    Transolver++ upgrades Transolver with eidetic states, local adaptive aggregation, and a highly optimized multi-GPU parallelism framework.

  • Results

    13% relative gain averaged across six standard benchmarks and over 20% improvement on high-fidelity million-scale industrial datasets.

  • Takeaways & Limitations

    Transolver++ accurately solves PDEs on million-scale geometries, supports 1.2 million mesh points on one GPU, and scales input size linearly with additional GPUs.

  • Takeaways & Limitations

    Models unable to process million-scale meshes directly are evaluated by splitting inputs, independently inferring pieces, and concatenating outputs.

Abstract

from arXiv · show

Although deep models have been widely explored in solving partial differential equations (PDEs), previous works are primarily limited to data only with up to tens of thousands of mesh points, far from the million-point scale required by industrial simulations that involve complex geometries. In the spirit of advancing neural PDE solvers to real industrial applications, we present Transolver++, a highly parallel and efficient neural solver that can accurately solve PDEs on million-scale geometries. Building upon previous advancements in solving PDEs by learning physical states via Transolver, Transolver++ is further equipped with an extremely optimized parallelism framework and a local adaptive mechanism to efficiently capture eidetic physical states from massive mesh points, successfully tackling the thorny challenges in computation and physics learning when scaling up input mesh size. Transolver++ increases the single-GPU input capacity to million-scale points for the first time and is capable of continuously scaling input size in linear complexity by increasing GPUs. Experimentally, Transolver++ yields 13% relative promotion across six standard PDE benchmarks and achieves over 20% performance gain in million-scale high-fidelity industrial simulations, whose sizes are 100$\times$ larger than previous benchmarks, covering car and 3D aircraft designs.

1. Introduction

Neural PDE solvers target costly numerical simulations, but prior models struggle with the million-scale geometries required for accurate industrial applications. Transolver++ combines adaptive physical-state modeling with optimized parallelism to scale neural PDE solving while retaining accuracy.

  • Numerical PDE methods discretize equations into meshes and approximate solutions there, but industrial simulations can take days or months.
  • Existing models fail beyond 400k points, although industrial applications typically require million-scale meshes whose fineness affects simulation accuracy.
  • Transolver reaches 700k input points but uses simpler experimental data than real applications, motivating extension to million-scale geometries.
  • Transolver++ addresses scaling bottlenecks with a highly optimized parallelism framework and a local adaptive mechanism for capturing physical states from massive meshes.
  • 1.2 million mesh points fit on one GPU while accuracy is maintained, and the framework provides linear scalability across GPUs.
  • 13% relative gain across six standard benchmarks and over 20% improvement on million-scale industrial datasets cover car and 3D aircraft design tasks.

2. Related Work

Neural PDE solver research spans operator learning, efficient Transformers, and graph neural networks for irregular geometries. Transolver instead models interactions among learned physical states, while large-scale industrial processing remains an open challenge addressed through parallelism.

  • Neural PDE solvers use deep models as efficient surrogates because numerical PDE solutions often require high computational cost.
  • Operator learning: Operator-learning methods approximate mappings from input functions to solutions, including Fourier-domain, latent-grid, and multiscale architectures.
  • Transformers: Transformer-based PDE solvers use attention, but standard attention has quadratic complexity and mesh-point sequences can limit geometric learning efficiency.
  • Transolver: Transolver groups mesh points into physical states and applies attention among states to model intrinsic physical correlations more effectively.
  • Graph neural networks: GNN-based solvers explicitly pass messages over unstructured mesh nodes and edges, but prior work also reports geometric instability concerns.
  • Large-scale geometries: Large-scale geometry processing is crucial for industrial design, yet previous research had not explored the problem; parallel frameworks address memory from large inputs.

3. Revisiting Transolver

Transolver reduces attention complexity by aggregating mesh points into learned physical states, but million-scale inputs create state-homogeneity and GPU-memory bottlenecks. These limitations motivate Transolver++'s adaptive state mechanism and parallel implementation.

  • Physics-Attention: Physics-Attention learns slice weights that assign mesh-point representations to physical states, aggregates points into states, and applies attention among those states.
  • Physics-Attention: Updated physical states are mapped back to mesh space through a deslice operation, replacing standard Transformer attention in Transolver.
  • Scaling challenges: O(N^2) canonical attention complexity is reduced to O(M^2), where M is constant and typically set to 32 or 64.
  • Scaling challenges: Uniform slice weights cause attention to degenerate into average pooling, and large meshes can produce homogeneous physical states that weaken physics modeling.
  • Scaling challenges: Embedding million-scale point representations with feedforward layers consumes huge GPU memory, creating a stability bottleneck despite nearly constant Physics-Attention cost.

4. Transolver++

Transolver++ addresses scaling bottlenecks in physics learning and computation by learning eidetic physical states and distributing computation across GPUs. Its local-adaptive state learning improves physical-state distinctions, while optimized parallelism reduces input-size-dependent communication and increases single-GPU capacity.

  • 4.1. Physics-Attention with Eidetic States: Transolver++ avoids attention degeneration by learning eidetic physical states through local-adaptive temperature control and slice reparameterization.The local mechanism adjusts state-distribution sharpness from each point’s local properties, while reparameterization models point-to-state assignment.
  • 4.1. Physics-Attention with Eidetic States: Slow-changing regions concentrate on one physical state, whereas fast-changing regions receive multimode slice-weight distributions.These assignments adapt to intricate physics fields on complex geometries.
  • 4.2. Parallel Transolver++: Transolver++ distributes mesh points across GPUs and communicates only when computing attention among eidetic states.Each GPU computes partial weighted sums, which are synchronized with AllReduce before state attention is updated.
  • 4.2. Parallel Transolver++: O(#gpu × M(C + 1)) communication remains invariant to input size, unlike parallel methods whose communication overhead grows with mesh-point count.The framework communicates partial sums of sizes O(MC) and O(M) across GPUs.
  • 4.2. Parallel Transolver++: 1.2 million mesh points fit on a single GPU after eliminating the redundant f projection without sacrificing model performance.The implementation also reports 0.25MB transferred with 32 GPUs, independent of input mesh size.
  • 4. Transolver++: 13% averaged relative promotion is achieved across six standard PDE benchmarks, with the lowest relative L2 error in every PDE-solving task.The comparison covers the top-performing baselines selected by overall performance.

5. Experiments

Experiments evaluate Transolver++ on standard PDE benchmarks and million-scale industrial geometries, assessing accuracy, physical-state learning, efficiency, and scalability. The model consistently improves over baselines while handling substantially larger meshes.

  • Experimental setup: Transolver++ is evaluated on six standard PDE benchmarks and industrial car and 3D aircraft simulations spanning broad mesh sizes.Industrial datasets include DrivAerNet++ and a newly simulated AirCraft dataset.
  • Standard benchmarks: Over 13% improvement is achieved over the dataset-specific second-best baseline averaged across six standard benchmarks.The comparison uses the averaged relative promotion reported in Figure 4(b).
  • Large geometries: Transolver++ achieves over 20% average promotion across large-geometry datasets and handles 2.5 million meshes within 4 A100 GPUs.On DrivAerNet++, relative promotions are 11.0% for volume fields and 12.6% for surface fields; surface-field leads reach 24.1% on DrivAerNet++ Surface and 30.4% on AirCraft.
  • Large geometries: Transolver++ captures sharper intrinsic physics variations with lower relative error, whereas other models tend to produce over-smoothed predictions.The reported comparison focuses on regions with drastic physical changes.
  • Ablations: A local adaptive mechanism significantly improves performance, while speed-up optimization and reparameterization together produce the best ablation result.The ablations are conducted on AirCraft.
  • Analysis and scalability: Transolver++ learns more diverse physical states and maintains a favorable performance-efficiency balance, including the lowest memory usage at equal input size.The model also benefits consistently from larger training datasets and model sizes.

6. Conclusion

Transolver++ extends Transolver toward practical neural PDE solving on million-scale geometries through eidetic states and an optimized parallel framework. It improves industrial design performance while offering favorable efficiency and scalability.

  • 6. Conclusion: Transolver++ accurately solves PDEs discretized on million-scale geometries using eidetic states and a highly optimized parallel framework.The design targets both physics learning and computation efficiency.
  • 6. Conclusion: The model achieves significant advancement in industrial design tasks and can serve as a backbone for PDE-solving foundation models.The conclusion explicitly connects its efficiency and scalability with this potential role.

A. Full Results on Standard Benchmarks

Full benchmark results show that Transolver++ performs best across six standard PDE benchmarks under relative L2 error, with carefully aligned reproduced baselines supporting fair comparison.

  • Full benchmark results: Transolver++ achieves the lowest relative L2 error across all six standard PDE-solving benchmarks, with an averaged relative promotion of 13%.The full results supplement the main benchmark comparison.
  • Comparison protocol: Baseline reproductions marked with an asterisk use closely aligned parameter counts and configurations for fair comparison.The implementation adjusts LNO’s default dimension in elasticity because its original parameter count was substantially larger.

B. Implementation Details

The implementation-details section describes benchmark, baseline, and setup information intended to support reproducibility and facilitate comparisons.

  • Implementation details: Benchmark, baseline, and implementation setups are documented to ensure reproducibility and facilitate comparisons.

B.1. Benchmarks

The paper evaluates Transolver++ on six standard PDE benchmarks and two industrial aerodynamic datasets spanning varied physics, geometries, resolutions, and prediction targets.

  • Standard and industrial benchmarks: Eight benchmarks cover elasticity, plasticity, transonic airfoil flow, pipe flow, 2D Navier–Stokes, Darcy flow, car aerodynamics, and aircraft aerodynamics.The standard benchmarks include six PDE tasks, while DrivAerNet++ and AirCraft provide industrial aerodynamic settings.
  • Benchmark organization: The benchmark descriptions report dataset splits, mesh-point counts, geometric types, and input-output specifications to support reproducible evaluation.These details are compiled in Table 6.
  • Industrial benchmarks: AirCraft contains simulations of over 30 aircraft designs under five incoming-flow conditions, each discretized into approximately 300,000 3D mesh points.The varying conditions include Mach number, angle of attack, and sideslip angle.

B.2. Metrics

Evaluation uses relative L2 error as the primary metric, with separate field and coefficient errors for large-scale datasets and R2 as an additional coefficient metric.

  • Primary metric: Relative L2 error measures the difference between predicted outputs and ground truth and is reported for direct model comparisons on standard benchmarks.The metric is defined for a predicted output field and its ground truth.
  • Large-scale errors: Large-scale evaluation separates relative L2 error for surface fields, volume fields, and physical coefficients.This decomposition distinguishes surface and surrounding-area predictions while also assessing inferred system coefficients.
  • Physical coefficients: AirCraft lift coefficient quantifies aerodynamic lift using surface pressure, surface normals, lift direction, shear stress, and aircraft surface.The definition combines pressure and shear-stress contributions over the aircraft surface.
  • Physical coefficients: Car-design evaluation uses drag coefficient to quantify aerodynamic drag and assess vehicle performance and fuel-efficiency-related design objectives.Its definition uses surface pressure, surface normals, drag direction, shear stress, and vehicle surface.
  • Coefficient metric: R2 evaluates coefficient prediction accuracy, with values closer to 1 indicating better performance.The score compares predicted coefficients with ground-truth coefficients relative to their mean.

B.3. Baselines and Implementations

Transolver++ is compared with more than 20 PDE-solving baselines under controlled training and parameter settings, including special handling for million-scale meshes.

  • Baseline selection: The comparison includes neural operators, Transformer-based PDE solvers, and graph neural networks tested under shared training configurations.Open-source implementations were obtained and checked against the original papers before training.
  • Controlled comparisons: Baseline parameter sizes are aligned by adjusting blocks or hidden dimensions when necessary to make performance comparisons fair.LNO, for example, is reduced from its default dimension of 256 to 192 in elasticity because its parameter count is substantially larger.
  • Baseline results: Published results are directly adopted for several baselines whose data settings exactly match the paper’s benchmarks.These include FNO, Geo-FNO, GNOT, OFormer, and Transolver.
  • Million-scale evaluation: Most baselines cannot process million-scale industrial meshes directly, so they are evaluated using 50k-point subsampling and KNN mesh reconstruction.Testing aggregates predictions from multiple subsampled sets to reconstruct full outputs.
  • Million-scale evaluation: On full DrivAerNet++, Transolver ranks second-best, while Transolver++ handles approximately 2.5 million points directly using eidetic states and a parallelism framework.The Transolver++ configuration uses 32 slices, 256 channels, and four layers.
  • Additional analyses: Supplementary analyses visualize eidetic states and provide showcase comparisons across industrial and standard datasets.The visualizations compare Transolver++ with Transolver, while showcase comparisons include GNOT for industrial datasets.

C.1. Eidetic States Visualization

The paper visualizes learned physical or eidetic states from final model layers across standard and industrial geometries, comparing Transolver++ with Transolver and other baselines.

  • Visualization scope: Visualizations cover DrivAerNet++ Surface, AirCraft, Airfoil, and Elasticity, with final-layer representations compared across models.The comparison is intended to examine learned representations under complex physical geometries.
  • Industrial geometries: Figures 8 and 9 compare 32 learned physical or eidetic states on DrivAerNet++ Surface and AirCraft, where lighter colors indicate higher state weights.Both Transolver and Transolver++ are plotted for direct comparison.
  • Standard benchmarks: Figures 10–13 visualize 64 final-layer learned states for Transolver++ and Transolver on Airfoil and Elasticity.The figures distinguish eidetic states in Transolver++ from physical or learned states in Transolver.
  • Error-map showcases: Figures 14–16 provide showcase comparisons using lighter error-map colors to indicate better performance.Transolver is compared on standard benchmarks, while Transolver and GNOT are compared on industrial datasets.
Loading 2502.02414v2…