PaperlayerView exact revision on arXiv

Source-linked AI summary

Transolver: A Fast Transformer Solver for PDEs on General Geometries

Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, Mingsheng Long

arXiv:2402.02366v2cs.LGmath.NA

TL;DR

PDE solving on large, complex meshes requires models that capture physical correlations without attending inefficiently to every point. Transolver learns intrinsic physical states through Physics-Attention, which forms learnable slices and physics-aware tokens for linear-complexity attention. It achieves consistent state-of-the-art results across six benchmarks and excels on industrial car and airfoil simulations, with a reported 22% relative gain across the standard benchmarks.

  • Problem

    Transformers applied directly to large, complex PDE meshes face computational and relation-learning difficulties, while regular patches do not fit unstructured geometries or expose hidden physical states.

  • Method

    Transolver uses Physics-Attention to group mesh points with similar physical states into learnable slices, encode physics-aware tokens, and attend to those tokens in linear complexity.

  • Results

    22% relative gain across six standard benchmarks accompanies consistent state-of-the-art performance, while Transolver also excels in large-scale car and airfoil simulations.

  • Takeaways & Limitations

    Learning intrinsic physical states gives Transolver geometry-general modeling capacity for complex geometries and supports favorable efficiency, scalability, and out-of-distribution generalizability.

  • Takeaways & Limitations

    Some baselines are inapplicable to unstructured-mesh industrial datasets, and several Transformer models experience unstable training with massive mesh-point counts.

Abstract

from arXiv · show

Transformers have empowered many milestones across various fields and have recently been applied to solve partial differential equations (PDEs). However, since PDEs are typically discretized into large-scale meshes with complex geometries, it is challenging for Transformers to capture intricate physical correlations directly from massive individual points. Going beyond superficial and unwieldy meshes, we present Transolver based on a more foundational idea, which is learning intrinsic physical states hidden behind discretized geometries. Specifically, we propose a new Physics-Attention to adaptively split the discretized domain into a series of learnable slices of flexible shapes, where mesh points under similar physical states will be ascribed to the same slice. By calculating attention to physics-aware tokens encoded from slices, Transovler can effectively capture intricate physical correlations under complex geometrics, which also empowers the solver with endogenetic geometry-general modeling capacity and can be efficiently computed in linear complexity. Transolver achieves consistent state-of-the-art with 22% relative gain across six standard benchmarks and also excels in large-scale industrial simulations, including car and airfoil designs. Code is available at https://github.com/thuml/Transolver.

1. Introduction

Transolver addresses the difficulty of modeling complex physical interactions on large, irregular PDE meshes by learning intrinsic physical states rather than attending directly to every mesh point. Its Physics-Attention forms learnable slices and physics-aware tokens, achieving linear complexity and strong benchmark and industrial results.

  • PDE solvers often use discretized meshes because analytic solutions are difficult, while conventional numerical solutions for complex structures can take hours or days.
  • Directly applying Transformers to massive, irregular mesh points is inefficient and struggles to learn complex spatiotemporal interactions among multiple physical quantities.Canonical attention has quadratic complexity, and pointwise features or regular patches do not adequately represent physical correlations under unstructured geometries.
  • Transolver groups mesh points with similar physical states into learnable slices and applies attention to physics-aware tokens encoded from those slices.The learned slices are intended to capture interactions behind the discretized domain rather than relying only on superficial mesh structure.
  • 22% relative gain across six standard benchmarks accompanies Transolver’s consistent state-of-the-art performance.
  • Transolver also excels in large-scale industrial simulations, including car and airfoil designs, with favorable efficiency, scalability, and out-of-distribution generalizability.

2. Related Work

Prior PDE-learning approaches include physics-informed models, neural operators, geometric deep learning, and Transformer solvers. They differ in how they encode physical constraints, operator mappings, irregular geometries, and computational structure.

  • Physics-informed neural networks: Physics-informed neural networks optimize outputs to satisfy PDE equations, initial conditions, and boundary conditions, but require exact PDE formalization.This requirement can make them difficult to apply to partially observed real-world applications.
  • Neural operators: Neural operators learn input-output mappings for PDE-governed tasks, with Fourier, U-Net, wavelet, factorized, and latent-space variants targeting multiscale or high-dimensional correlations.
  • Irregular geometries: Graph neural operators and geometric Fourier methods address irregular meshes by using graph representations or projecting domains into uniform latent meshes.GINO combines graph-based local modeling with geometric Fourier global modeling, while 3D-GeoCA adds pretrained 3D vision initialization.
  • Transformer PDE solvers: Transformer PDE solvers use multigrid, low-rank, or factorized attention, but several assume uniform grids and therefore have limited applicability to unstructured meshes.
  • Geometric deep learning: Geometric deep-learning methods such as graph neural networks, PointNet, and Point Transformer provide representations for irregular graphs or scattered point clouds.

3. Method

Transolver addresses the efficiency and correlation challenges of applying Transformers to PDE meshes by learning intrinsic physical states rather than relying on direct mesh-point relations. Its Physics-Attention forms learnable slices and physics-aware tokens, applies token-level attention, and maps the resulting representations back to mesh points with linear complexity.

  • Motivation: PDE solvers must model physical quantities on discretized meshes, but massive and complex geometries make direct Transformer attention inefficient and unreliable for learning physical relations.The limitations are especially pronounced for unstructured geometries, spatially distant points with similar states, and multiphysics interactions.
  • Learning Physics-Aware Tokens: Transolver assigns mesh points with similar learned geometry and physics features to learnable slices, aiming to represent intrinsic physical states behind the discretized domain.Slice weights are produced point-wise and sharpened with Softmax so slices capture more distinguishable physical patterns.
  • Learning Physics-Aware Tokens: Each slice is spatially aggregated and normalized into a physics-aware token containing information about a specific physical state.The token is a weighted combination of mesh-point features, normalized by the sum of slice weights.
  • Learning Physics-Aware Tokens: Unlike classical computation-area splitting, learned slices can group spatially distant mesh points that share physical states, such as a car windshield and license plate.This distinction targets physically coherent subsets rather than only spatially local regions.
  • Transolver: Physics-Attention applies attention among physics-aware tokens to capture correlations between physical states, then deslices the transformed tokens back to mesh points.The Transolver layer replaces standard attention while retaining a Transformer-style layered architecture.
  • Transolver: With M treated as a constant and M ≪ N, Physics-Attention has complexity O(NMC + M 2C), which is linear in the number of mesh points.The architecture embeds geometry and optional observations, repeatedly applies Physics-Attention, and projects the final features to predictions.

4. Experiments

Experiments evaluate Transolver across varied geometries, standard PDE benchmarks, industrial design tasks, efficiency, ablations, attention behavior, scalability, and out-of-distribution samples. The results show strong accuracy, efficiency, physical-state modeling, scalability, and generalization.

  • Benchmark coverage: Experiments cover six standard benchmarks spanning point clouds, structured and unstructured meshes, regular grids, and 2D and 3D geometries.The evaluation also includes car and airfoil design tasks.
  • Standard benchmarks: Transolver achieves consistent state-of-the-art performance across six widely used benchmarks covering solid and fluid physics.Reported promotions include 25.6% on Elasticity, 29.4% on Plasticity, and 29.7% on Pipe.
  • Attention behavior: Physics-Attention produces more informative geometry tokens than individual mesh points and reduces token count, improving complex fluid-structure interaction modeling.Direct mesh-point linear attention is reported as closer to a degenerated uniform distribution.
  • Practical design: Transolver estimates more accurate physics fields in both volume and surface on complex car and airfoil design tasks.It also performs best on drag and lift coefficients and their Spearman rank correlations, which quantify design-ranking ability for shape optimization.
  • Efficiency: 25.6% error reduction (0.0064 vs. 0.0086) on Elasticity accompanies 5x fewer parameters and 1.3x running speed than GNOT.On large meshes, linear-complexity Physics-Attention also surpasses Galerkin Transformer and OFormer in running time and final performance.
  • Physics-Attention analysis: Learned slices capture similar physical states, remain precise with 50% of Elasticity’s input mesh, and outperform fixed regular squares.Softmax sharpens slice weights, while increasing slice count captures finer states at greater computational cost.
  • Case study: Performance is strongest around boundaries and multiple-material junctions, including elasticity extrusion zones, Darcy medium boundaries, and curved car surfaces.These regions involve intricate physical interactions requiring accurate geometry and hidden-physics modeling.
  • Scalability and generalization: Transolver maintains consistent performance as PDE resolution increases to 25x and benefits from up to 5x more training data and model parameters.It also performs well on out-of-distribution samples, with Spearman rank correlations near 99% on unseen Reynolds numbers and angles of attack.

5. Conclusions and Future Work

Transolver applies attention to learned physical states rather than directly to mesh points, targeting PDEs on general geometries. The paper reports strong benchmark and practical-design performance, alongside analyses of efficiency, scalability, and out-of-distribution generalizability.

  • Conclusion: Transolver applies attention to learned physical states, providing geometry-general capacity and supporting physical-correlation modeling.The paper positions this design for PDEs on general geometries and complex practical simulations.
  • Conclusion: The model performs strongly on established benchmarks and practical design tasks involving complex geometries and multiphysics interactions.The paper also analyzes performance, efficiency, scalability, and out-of-distribution generalizability.
  • Future work: Future work will explore large-scale pre-training of Transolver toward foundation models for PDE solving.

Impact Statement

The work aims to advance deep learning for PDE solving by capturing correlations among learned physical states. It reports potential usefulness for future research and industrial production while emphasizing its scientific focus.

  • Scientific impact: The proposed idea of capturing correlations among learned physical states could inspire future PDE-solving research.
  • Industrial relevance: The model’s performance on large-scale design tasks could be useful in industrial production.
  • Scope: The paper mainly focuses on the scientific problem and states a commitment to ethical considerations during development.

A. Proof of Theorem 3.4

The appendix formalizes canonical attention as a Monte-Carlo approximation of an integral operator and extends this view to Physics-Attention over a slice domain. It establishes an isomorphic input-to-slice projection under stated assumptions and identifies attention among slice tokens as the relevant operator.

  • Integral-operator interpretation: Canonical Transformer attention is derived as a Monte-Carlo approximation of an integral operator over the input domain.The derivation begins by formalizing the integral operator and matching its Monte-Carlo approximation to query, key, and value projections.
  • Integral-operator interpretation: The resulting discrete expression corresponds to attention computed with linear layers for queries, keys, and values.The approximation yields the standard attention calculation after applying the kernel construction and sampling steps.
  • Slice-domain construction: A countable input domain can be mapped bijectively to a slice domain by assigning each input element a slice weight subject to an unassigned-slice constraint.The construction defines an isomorphic projection between the input domain and slice domain.
  • Physics-Attention operator: The appendix states that Physics-Attention is theoretically equivalent to a learnable integral operator on the original domain under the stated assumptions.The construction connects the slice-domain attention mechanism back to the PDE-solving operator on the input domain.
  • Slice-domain construction: Under smooth slice weights, the isomorphic projection can be treated as a diffeomorphism between continuations of the input and slice domains.The theorem proof uses this projection to transfer the operator formulation from mesh points to slices.
  • Physics-Attention operator: Physics-Attention uses a kernel between mesh points and slices, attention among slice tokens, and a slice-token value function to represent the corresponding integral.Permutation invariance permits simplifying the Jacobian determinant to one, and the approximation is performed in the slice domain.

B.1. Benchmarks

The experiments cover eight PDE benchmarks spanning structured, regular-grid, point-cloud, and unstructured-mesh settings in two and three dimensions. They evaluate physics-field errors alongside drag/lift coefficient metrics for design-oriented tasks.

  • Benchmark coverage: Eight benchmarks cover solid-material, fluid, and porous-medium PDEs across point-cloud, structured-mesh, regular-grid, and unstructured-mesh settings.The benchmark suite includes Elasticity, Plasticity, Airfoil, Pipe, Navier-Stokes, Darcy, Shape-Net Car, and AirfRANS.
  • Standard PDE benchmarks: Elasticity predicts pointwise material stress from a 972-point structure representation using 1,000 training and 200 test samples.The output is stress at each discretized point.
  • Standard PDE benchmarks: Plasticity predicts deformation over 20 future time steps from an arbitrarily shaped die, while Airfoil predicts meshwise Mach number from deformed airfoil shapes.Plasticity uses a 101 × 31 structured mesh and four deformation directions; Airfoil uses a 221 × 51 mesh.
  • Standard PDE benchmarks: Pipe estimates horizontal fluid velocity from pipe geometry, Navier-Stokes predicts the next 10 fluid states, and Darcy predicts pressure through porous media.These tasks use structured grids and varying training/test sample configurations.
  • Practical design benchmarks: Shape-Net Car and AirfRANS evaluate physics-field prediction in large unstructured meshes for car and airfoil design simulations.Shape-Net Car uses 32,186 mesh points, while AirfRANS uses 32,000 points and varies airfoil shape, Reynolds number, and angle of attack.
  • Evaluation metrics: Evaluation combines relative L2 errors for physics fields and drag/lift coefficient metrics, including Spearman correlation for design ranking.Higher Spearman correlation indicates that model-predicted coefficients more readily support finding the best design.

B.3. Implementations

The implementation aligns Transolver configurations with baseline model sizes and efficiency, while benchmark-specific handling addresses structured and unstructured meshes. Several baselines are excluded from practical design tasks because of mesh, boundary, or training-stability constraints.

  • Transolver configuration: Transolver uses 256 channels for high-dimensional inputs and 128 channels otherwise, with 32 or 64 slices respectively.The slice counts are selected to balance efficiency and align parameters and running efficiency with Transformer operators.
  • Baseline handling: For unstructured meshes, Project() is configured as a single Linear layer, while baseline results are taken from official papers or prior reported experiments where applicable.The implementation includes special handling for models that cannot directly process unstructured meshes.
  • Training configuration: Training configurations are generally adopted from previous works without extra tuning, with an additional spatial-gradient regularization term for Darcy.The configuration table distinguishes volume and surface losses and adds Lg for Darcy following ONO.
  • Baseline handling: geo-FNO degenerates on Shape-Net Car and AirfRANS, which the authors associate with its periodic-boundary assumption under complex boundaries.Consequently, related neural operators based on geo-FNO are not tested on those practical design tasks.
  • Baseline handling: ONO and OFormer are not reported on Shape-Net Car and AirfRANS because of unstable execution or optimization with large unstructured meshes.The stated issues involve ONO’s positive-semidefinite channel-attention requirement and OFormer’s attention over more than 32,000 mesh points.

C. Full Ablations

The ablations examine slice count, adaptive versus fixed slicing, and full attention across resolutions. They show that slice count trades performance against computation, adaptive slices outperform regular squares, and Physics-Attention becomes more advantageous as mesh resolution grows.

  • Number of slices: Increasing the number of slices generally improves performance but adds computation and can introduce noise or performance drops at excessive counts.The reported fluctuations include M = 1024 for Elasticity and Navier-Stokes.
  • Learnable versus fixed slices: Fixed regular squares damage performance across all benchmarks, including originally regular-grid Navier-Stokes and Darcy tasks.The ablation attributes this pattern to intricate and spatially deformed physical states.
  • Comparison with full attention: The full-attention comparison uses Darcy resolutions from 484 to 168,921 tokens while replacing Physics-Attention with canonical attention at identical architecture elsewhere.Ground-truth downsampling causes serious performance drops at the two smallest settings, so relative promotions remain the comparison focus.
  • Comparison with full attention: Transolver performs stably from 7,225 to 168,921 tokens, while its advantage over the plain Transformer grows at larger mesh sizes.At the smallest resolution, the two models perform similarly.

D.1. Learned Slices

Transolver’s learned slices capture diverse physical-state patterns across benchmark geometries. The same approach remains precise on resampled meshes and outperforms prior models on showcased predictions.

  • D.1. Learned Slices: Learned slices capture pressure, flow, swirling, and fluid-structure patterns across Shape-Net Car, Airfoil, Pipe, Navier-Stokes, and Darcy benchmarks.The Navier-Stokes slices also learn a periodic diagonal pattern corresponding to the benchmark’s periodic external force.
  • D.1. Learned Slices: 50%-80% of the original mesh points can be retained while Transolver still precisely captures physical states on broken resampled meshes.These experiments test whether learned physical states depend on concrete discretization.
  • D.1. Learned Slices: Transolver better captures Plasticity deformation, Airfoil shock waves, Pipe-end fluid, and Navier-Stokes swirling than LSM and GNOT.GNOT fails to predict the future deformation of Plasticity in the comparison.

E.1. Model Scalability

Model behavior is examined across parameter sizes, slice granularities, multiscale configurations, and additional datasets. Larger models generally help, while learned slices permit flexible multiscale settings and results remain statistically robust.

  • E.1. Model Scalability: Most benchmarks benefit from increasing model depth from 8 to 40 layers, especially Elasticity and Airfoil.Elasticity Relative L2 improves from 0.0064 at 8 layers to 0.0047 at 40 layers.
  • E.1. Model Scalability: A larger slice count M produces finer-grained physical-state modeling, and M can vary across layers for multiscale modeling.The expected number of mesh points per slice is N/M.
  • E.1. Model Scalability: Multiscale configurations can outperform the official fixed-slice configurations in some cases, although increasing M generally boosts performance.The multiscale setup uses half as many slices in the last four layers as in the first four.
  • E.1. Model Scalability: The slice design is not constrained by exact discretization divisions, allowing arbitrary M values without inflexible padding operations.The paper mainly evaluates the official configuration and leaves further exploration of multiscale settings for future work.
  • E.1. Model Scalability: Transolver surpasses the second-best model with high confidence across repeated experiments, including 95% confidence on the limited-data AirfRANS setting.Experiments are repeated three times, with standard deviations reported; AirfRANS contains 1,000 simulations.

F. Full Efficiency Analysis

Transolver’s efficiency is evaluated as mesh size grows against other Transformer-based methods. Its linear-complexity Physics-Attention yields the least running-time growth, particularly on large meshes.

  • F. Full Efficiency Analysis: Transolver presents the least running-time growth among five Transformer-based methods as input mesh size increases.The comparison records model parameters, running time, and GPU memory across mesh sizes from 2^10 to 2^15.
  • F. Full Efficiency Analysis: The efficiency comparison covers Elasticity Relative L2 and Shape-Net Car ρD across unstructured-mesh Transformer methods.Running time is measured per epoch over 10^3 iterations, and “/” denotes a baseline failure.
Loading 2402.02366v2…