Source-linked AI summary
Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries
Hang Zhou, Haixu Wu, Haonan Shangguan, Yuezhou Ma, Huikun Weng, Jianmin Wang, Mingsheng Long
TL;DR
Industrial neural PDE solvers must process high-resolution geometries whose memory demands exceed typical GPU capacity. Transolver-3 combines architectural, amortized-training, and cached-inference strategies to scale prediction, handling over 160 million cells across three industrial benchmarks. Its scope remains primarily stationary aerodynamic simulation, with transient and tightly coupled multi-physics generalization not fully explored.
Problem
Neural PDE solvers face prohibitive memory complexity when scaling high-fidelity prediction to industrial geometries exceeding 10^8 cells.
Method
Transolver-3 combines Physics-Attention optimizations, geometry amortized training, and decoupled inference with global physical-state caching.
Results
Transolver-3 achieves consistent state-of-the-art performance across three industrial benchmarks and scales to 160 million mesh cells.
Takeaways & Limitations
The framework demonstrates practical high-fidelity neural PDE solving on industrial-scale geometries under limited GPU memory.
Takeaways & Limitations
Validation primarily covers stationary aerodynamic benchmarks, while generalization to highly transient or tightly coupled multi-physics systems remains unexplored.
Abstract
from arXiv · showhide
Deep learning has emerged as a transformative tool for the neural surrogate modeling of partial differential equations (PDEs), known as neural PDE solvers. However, scaling these solvers to industrial-scale geometries with over $10^8$ cells remains a fundamental challenge due to the prohibitive memory complexity of processing high-resolution meshes. We present Transolver-3, a new member of the Transolver family as a highly scalable framework designed for high-fidelity physics simulations. To bridge the gap between limited GPU capacity and the resolution requirements of complex engineering tasks, we introduce two key architectural optimizations: faster slice and deslice by exploiting matrix multiplication associative property and geometry slice tiling to partition the computation of physical states. Combined with an amortized training strategy by learning on random subsets of original high-resolution meshes and a physical state caching technique during inference, Transolver-3 enables high-fidelity field prediction on industrial-scale meshes. Extensive experiments demonstrate that Transolver-3 can handle meshes with over 160 million cells, achieving impressive performance across three challenging simulation benchmarks, including aircraft and automotive design tasks. Code is available at https://github.com/thuml/Transolver-3.
1. Introduction
Neural PDE solvers promise faster high-fidelity simulation, but industrial geometries exceed current memory limits. Transolver-3 addresses this scaling bottleneck through training and inference changes designed for meshes beyond 10^8 cells.
- Motivation: Industrial CFD meshes can exceed 100 million cells because higher-fidelity simulations require greater mesh resolution.Large-scale meshes support physics fidelity and numerical stability but increase computational demands.
- Motivation: A 128-feature cell mesh requires around 51.2 GB in FP32, quickly exceeding the 80 GB capacity of high-end GPUs.This makes geometry scaling under current hardware limitations a central requirement for practical neural PDE solvers.
- Prior limitations: Existing Transolver handles complex geometries but is limited to 700k cells, while Transolver++ uses multi-GPU parallelism for million-scale meshes.These limits motivate a different approach to industrial-scale geometry processing.
- Approach: Transolver-3 combines Physics-Attention optimizations with geometry amortized training to bridge hardware constraints and industrial-scale geometries.Training uses random subsets of high-fidelity meshes to learn the underlying physics laws.
- Approach: A decoupled inference framework separates physical-state estimation from field prediction and uses cached global states for full-resolution meshes.This design aggregates global mesh information without exceeding memory limits.
- Results: Transolver-3 achieves consistent state-of-the-art performance across three industrial-level benchmarks with hundred-of-million-scale meshes.The reported result emphasizes both predictive performance and scalability.
2. Related Work
Related neural PDE solvers use physics-informed, operator-learning, graph, latent-space, and efficient-attention approaches. Recent work increasingly targets generalization and high-resolution industrial meshes, while Transolver uses physical states to improve attention efficiency.
- Neural PDE solvers: Physics-informed neural networks encode PDEs in loss functions but often face optimization challenges and limited generalization.Neural operators instead learn mappings from input functions to solutions from data.
- Neural PDE solvers: Fourier, geometric, latent-space, and graph-based methods address operator learning across regular and irregular geometries.Examples include FNO, Geo-FNO, LSM, Graph-UNet, GNO, MeshGraphNet, and GINO.
- Transformer solvers: Standard Transformers have quadratic complexity, motivating linear-attention and other efficient-attention models for large meshes.OFormer, GNOT, ONO, and FactFormer are cited as examples.
- Transformer solvers: Transolver groups mesh points into physical states and applies attention across those states rather than directly over massive meshes.This Physics-Attention mechanism is presented as central to Transolver’s geometry flexibility.
- Generalization: Recent studies scale neural PDE solvers through broader PDE-family pretraining, shared latent spaces, and denoising auto-regressive pretraining.The cited methods include large-scale FNO pretraining, MPP, DPOT, and Unisolver.
- Geometry scaling: High-resolution scalability has progressed from roughly 10^5-cell limits toward industrial-scale meshes through latent-space architectures and multi-GPU methods.AB-UPT and Transolver++ are cited as later advances in this direction.
3. Transolver-3
Transolver-3 addresses geometry-scaling bottlenecks in Physics-Attention through architectural optimizations, tiled computation, amortized training, and decoupled inference. These changes reduce memory demands while supporting high-fidelity prediction on industrial-scale meshes.
- Revisit Transolver Complexity: Physics-Attention remains resolution-dependent because intermediate projections and slice weights require O(NC) and O(NM) memory.Self-attention in the slice domain costs O(M^2C), but operations tied to mesh resolution N remain bottlenecks.
- Geometry Scaling at the Training Phase: Faster slice and deslice reorder matrix multiplications without changing the computation, removing two of five O(N) operations.The optimized formulation also eliminates two O(NC) intermediate tensors by switching linear projections into the latent space.
- Geometry Scaling at the Training Phase: Geometry slice tiling partitions physical-state computation across tiles, reducing peak slice-weight memory from O(NM) to O(NtM).Tile contributions are accumulated into global state and normalization accumulators, so the full N × M weight matrix is not materialized.
- Geometry Scaling at the Training Phase: Geometry amortized training learns from random subsets of high-fidelity meshes rather than processing every cell in each iteration.The sampled subset size n is approximately 10^5–10^6, while varying subsets encourage learning on the continuous geometry manifold.
- Geometry Scaling at the Inference Phase: Decoupled inference caches global physical states from high-resolution meshes and then decodes fields on the full mesh using the cache.The framework separates physical-state estimation from field prediction, allowing the expensive cache computation to be performed once.
4. Experiments
Transolver-3 is evaluated on three industrial aerodynamic benchmarks, including aircraft and automotive geometries, with meshes reaching 160M cells. It consistently outperforms competing solvers while improving memory efficiency, computational speed, and high-resolution prediction quality.
- Benchmark setup: Transolver-3 is evaluated on NASA-CRM, AhmedML, and DrivAerML, covering aircraft, bluff-body, and high-fidelity automotive aerodynamic simulations.The experiments assess point-wise errors and design-oriented quantities such as drag and lift coefficients.
- Main results: 9 out of 10 metrics favor Transolver-3 over all baselines, with only AhmedML volume pressure pv narrowly favoring AB-UPT.The comparison uses relative L2 errors across surface and volume fields on the three benchmarks.
- Main results: Over 160 million DrivAerML volume-field cells are handled successfully, surpassing competing models while preserving the full expressive capacity of Physics-Attention.Transolver-3 retains the Linear1 layer by shifting it into the slice domain to reduce memory consumption.
- Model analysis: Geometry slice tiling boosts single-GPU capacity by roughly 90%, while faster slice and deslice support approximately 10% more mesh cells than Transolver++.Further scaling beyond 10^8 cells relies on combining these optimizations with amortized training and decoupled inference.
- Model analysis: Transolver-3 reduces Physics-Attention latency by approximately 60% and GFLOPs by 20% through optimized operation ordering.The practical latency reduction is larger than the theoretical GFLOPs reduction.
- Model analysis: Smaller and more uniform error distributions are observed across benchmarks, including difficult high-curvature, rear-region, front, and underbody areas.The case study reports precise surface-pressure and volumetric-velocity predictions for DrivAerML.
5. Conclusions
Transolver-3 addresses memory bottlenecks in neural PDE solvers for industrial-scale geometries across training and inference. It scales prediction to 160 million mesh cells through architectural optimizations, amortized training, and physical state caching.
- Transolver-3 targets practical neural PDE solving on industrial-scale geometries exceeding 10^8 cells.
- Training combines two structural optimizations with geometry amortized training to bridge high-fidelity representations and limited GPU memory.
- Inference separates physical state caching from full mesh decoding to aggregate global mesh information.
- 160 million mesh cells demonstrate Transolver-3's potential for real-world AI-CAE workflows.
- Further geometric pre-training is proposed as a direction for improving generalization across cross-domain industrial topologies.
Impact Statement
The paper advances neural PDE solvers for high-fidelity industrial applications by improving memory efficiency, computation speed, and practical training and inference strategies. Its focus is enabling prediction on industrial-scale geometries.
- Transolver-3 is designed to advance neural PDE solvers for high-fidelity industrial applications.
- The method analyzes Physics-Attention and introduces architectural optimizations for memory efficiency and faster computation.
- Practical training and inference strategies enable prediction on industrial-scale geometries.
A.1. Benchmarks
The benchmarks cover aircraft and automotive CFD datasets with high-fidelity flow fields, while evaluation uses field errors and integrated aerodynamic coefficients. The metrics assess both local predictions and global aerodynamic behavior.
- Benchmarks: NASA-CRM evaluates a full wing–body–horizontal tail aircraft configuration under realistic 1g flight shape deformation using six input parameters.
- Benchmarks: AhmedML contains 500 Ahmed bluff-body configurations with approximately 20 million cells per case and hybrid RANS-LES simulations.
- Benchmarks: DrivAerML contains 500 parametrically deformed DrivAer vehicle variants developed for machine-learning applications in high-fidelity automotive aerodynamics.
- Evaluation metrics: Relative L2 error evaluates point-wise pressure, wall shear stress, velocity, and pressure fields across the benchmarks.
- Integrated coefficients: Drag coefficient Cd and lift coefficient Cl are derived by integrating predicted pressure and wall shear stress over the full high-resolution surface mesh.
- Evaluation metrics: R2 complements field errors by measuring prediction quality for integrated aerodynamic coefficients across the test set.
Mean Absolute Error (MAE) for Integrated Coefficients
MAE evaluates integrated drag and lift coefficients under geometry-scaling experiments. Increasing sampled surface cells during inference consistently reduces coefficient MAE, linking evaluation resolution to global prediction precision.
- MAE measures errors in predicted integrated aerodynamic coefficients Cd and Cl under resolution constraints.
- Lower MAE indicates higher predictive precision for design-critical aerodynamic quantities.
- Increasing sampled surface cells during inference consistently reduces MAE on Cd and Cl.
- Scaling training/input and evaluation/output resolution is essential for high-fidelity industrial-scale simulation results.
A.3. Baselines and Implementations
The section describes baseline architectures, implementation choices, and evaluation procedures for comparing Transolver-3 with neural PDE solvers on irregular and industrial-scale geometries.
- Implementation: All baselines use the same training strategy, with full meshes for NARA-CRM and geometry amortized training for larger benchmarks.The table defines subset size as the number of randomly sampled points used during amortized training.
- Implementation: The experiments reproduce or reuse baseline results across NASA-CRM, AhmedML, and DrivAerML, with implementation details varying by benchmark.Some results are taken from prior work, while Transolver and Transolver++ use new experiments and GAOT is reproduced for comparison.
- Transformer-based Models: AB-UPT uses separate geometry, surface, and volume branches connected by shared physics blocks and anchored attention for scalable coupled-field modeling.The architecture targets unstructured meshes exceeding 10^8 cells.
- Transformer-based Models: Transolver maps mesh points to learnable physics-aware slices, applies self-attention at slice level, and projects updated representations back to the mesh.This reduces computational and memory costs relative to point-wise attention.
- Results: Transolver-3 significantly outperforms baseline models across repeated experiments, except for volume pressure on AhmedML.The comparison uses the second-best model as a strong baseline, and experiments are repeated three times.
C.1. Physics States Visualization
The visualization examines the physics-aware slice weights learned by Transolver-3 on the DrivAerML surface, showing how surface points are organized into physical states.
- Slice-weight visualization: The visualization extracts representative slice weights from the first Physics-Attention layer on the DrivAerML surface dataset.It is intended to show how the model organizes surface points under complex automotive geometries.
- Slice-weight visualization: The learned tokens form coherent, spatially meaningful physical states that reflect salient aerodynamic patterns on high-resolution surface meshes.These states are interpreted through the spatial organization of the slice weights.
- Slice-weight visualization: Lighter colors indicate higher weight in the corresponding physical state.
C.2. Showcases
The showcases compare Transolver-3 with Transolver++ and AB-UPT across three industrial-scale benchmarks, multiple physical quantities, and prediction error maps.
- Cross-benchmark comparisons: The figures compare predictions and error maps from Transolver-3, Transolver++, and AB-UPT on NASA-CRM, AhmedML, and DrivAerML.The comparisons cover surface and volume quantities across aircraft and automotive geometries.
- Error-map interpretation: Transolver-3 is more effective in certain key and challenging geometric regions, such as regions involving the Cf coefficient.Lighter error-map colors correspond to lower prediction errors.
- Physical quantities: The showcases cover surface pressure, skin friction, wall shear stress, and the x-component of volume velocity.AhmedML and DrivAerML velocity comparisons include side-view and bottom-view slices.
D. Limitations and Future Work
Transolver-3 scales to industrial-grade geometries, but its efficiency and validated scope remain bounded by computation–memory trade-offs and benchmark coverage.
- Computational trade-offs: Geometry slice tiling trades additional computation for spatial capacity, so tile size must balance processing time against hardware constraints.
- Validation scope: The framework has primarily been validated on stationary aerodynamic benchmarks, leaving highly transient and tightly coupled multi-physics systems insufficiently explored.Future work proposes broader cross-domain generalization through structural geometry pre-training.