Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations

• Dense spatial discretizations make standard numerical solvers increasingly costly with grid resolution, often rendering high-fidelity simulations computationally infeasible.Source: §1 · p. 2
• Many scientific machine-learning methods require retraining for each PDE instance, whereas trained neural operators predict solutions for unseen configurations.Source: §1 · p. 2
• MLP-based operators can struggle with complex nonlinear behavior, while spectral operators face increased cost or lossy interpolation on general unstructured domains.Source: §1 · p. 2
• Transformer attention captures long-range dependencies but scales quadratically, while linear attention can oversmooth and Transolver’s slice compression loses local detail and shares head weights.Source: §1 · p. 3
• CANO dynamically clusters queries while retaining keys and values in their original coordinate space, avoiding slice compression loss and attention-head weight sharing with linear scaling in the number of points.The model maintains global receptive fields.Source: §1 · p. 3
• Across 12 diverse PDE benchmarks, CANO achieves up to 41.67% lower error on solid mechanics and 33.73% on turbulent flows than previous models.Source: §1 · p. 3

• Neural operators act as fast surrogates that accelerate inference by learning mappings between parameter spaces and corresponding PDE solution spaces.Source: §2 · p. 4
• Operator learning approximates a solution operator mapping parameter fields to PDE solutions, with a parameterized neural operator trained on input–solution pairs by minimizing empirical mean squared error.Source: §2 · p. 4
• Green’s-function-motivated kernel operators update continuous hidden representations through non-local integration, combining local input features with global interactions before decoding the solution field.Source: §2 · pp. 4–5
• FNO uses a translation-invariant kernel to reduce non-local integration to frequency-domain convolution, but standard FFTs make it difficult to apply directly to complex unstructured meshes.Source: §2 · p. 5
• DeepONet represents the solution as a dot product of branch and trunk networks, but its MLP learning dynamics can struggle with sharp, high-frequency oscillations.Source: §2 · p. 5
• Standard attention has quadratic cost, while linear attention reduces complexity through kernel factorization but produces low-rank, smooth maps that limit high-frequency detail.Source: §2 · pp. 5–6
• Transolver projects physical nodes into fewer slice tokens and performs attention there, but slice compression discards local high-frequency detail and shared head weights restrict distinct head patterns.Source: §2 · p. 6

3 Cluster Attention Neural Operator (CANO)

• CANO clusters queries while keeping keys and values at full physical resolution, retaining physical detail and global context.Unlike symmetric slice compression, this asymmetric cross-attention avoids compressing K and V.Source: §3 · p. 7
• A dynamic assignment matrix C, derived from Q, maps N physical nodes to Kc cluster centroids, with Kc ≪ N.CANO first projects the input into query, key, and value spaces, then derives C from the query matrix.Source: §3 · p. 7
• Compact clustered queries Qc attend to the full, uncompressed key matrix K to extract relevant features from V.Source: §3 · p. 7
• The same assignment matrix C dispatches aggregated cluster features Zc back to the original spatial resolution.Source: §3 · p. 7
• CANO’s attention costs O(NKc d), giving linear complexity in N when Kc ≪ N, while allowing independent multihead computations without shared weights.The paper contrasts this with Transolver’s weight sharing across attention heads.Source: §3 · p. 7

4.1 Standard PDE Benchmarks

• CANO achieves lower relative L2 errors than Transolver across all six standard PDE benchmarks, with reductions of 41.67% on Plasticity and 29.79% on Navier–Stokes.The reported largest improvements are on (2+1)-D Plasticity and turbulent Navier–Stokes.Source: §4.1 · pp. 9–10, Table 3 · p. 12
• Across six standard datasets, CANO has lower median errors; its Navier–Stokes energy spectrum also matches the reference across wavenumbers, including small scales.The spectrum comparison averages 200 test samples, with mean ± 3 standard deviation regions largely overlapping.Source: §4.1 · pp. 10, 12, Figure 2 · p. 13, Figure 5 · p. 16

4.3 Long-time rollout benchmarks

• On Airfoil and Elasticity, CANO captures key spatial features, while Navier–Stokes rollouts preserve main flow structures through timestep +10 with limited error accumulation.The spectrum comparison also reports agreement with the reference at small spatial scales.Source: §4.1 · p. 10, Figure 3 · p. 14

4.1 Standard PDE Benchmarks

• CANO uses fewer parameters than Transolver on five of six standard benchmarks, reducing Navier–Stokes parameters from 12.28M to 8.12M.CANO uses slightly more parameters on Elasticity, while the reported overall comparison is higher accuracy with fewer parameters.Source: §4.1 · p. 13, Table 4 · p. 13

4.2 Irregular domain benchmarks

• On three irregular point-cloud benchmarks, CANO has the lowest relative L2 errors, including a 33.73% reduction versus the second-best baseline on Pipe Turbulence.The tests cover Irregular Darcy, Pipe Turbulence, and Composite; CANO's error distributions shift downward and suppress some outliers, though Composite still has outliers.Source: §4.2 · p. 17, Table 6 · p. 17, Figure 6 · p. 18

7: Navier-Stokes

• Models train on 5-step input and prediction windows, then are evaluated at 5 steps and at 15 steps for Navier-Stokes or 50 for ICP Plasma and Heat Transfer.The benchmarks cover regular-grid Navier-Stokes and point-cloud multiphysics datasets.Source: §7 · pp. 18–19, Table 7 · p. 19
• CANO maintains lower relative L2 errors than Transolver across datasets and rollout horizons, despite rising errors for both models.Table 8 reports the long-term comparison.Source: §7 · p. 19, Table 8 · p. 19
• At long horizons, Transolver has a more pronounced upper error-distribution tail than CANO, while both models’ distributions spread out.Figure 7 shows per-step error accumulation and short- and long-horizon error distributions.Source: §7 · pp. 19–20, Figure 7 · p. 20
• CANO and Transolver have comparable early spatial errors, but Transolver’s errors become more pronounced over time on ICP Plasma and Heat Transfer.Figures 8 and 9 visualize electron-temperature and velocity-magnitude absolute errors, respectively.Source: §7 · p. 20, Figure 8 · p. 21, Figure 9 · p. 21

Training Sample Efficiency

• With 200 training samples, CANO’s relative L2 errors are 1.80×10−2 on Darcy Flow and 3.09×10−1 on Navier-Stokes, below Transolver’s 2.17 × 10−2 and 3.76 × 10−1.Both models’ errors decrease as the training sample count grows from 200 to 1000.Source: Training Sample Efficiency · p. 22, Table 9 · p. 22

Sensitivity to Cluster Number

• The best cluster count depends on the system: complex flows reach lowest error at Kc = 128, while Pipe, Airfoil, and Plasticity favor Kc = 16, 32, and 64, respectively.The passage links choosing Kc to spatial complexity to help prevent representational redundancy.Source: Table 10 · p. 22, Sensitivity to Cluster Number · p. 23

5 Conclusions and Discussion

• Across 12 PDE benchmarks spanning regular and irregular geometries and long-horizon autoregression, CANO consistently improves predictive accuracy over the considered state-of-the-art baselines.The conclusion presents asymmetric cluster-based attention as an alternative to symmetric latent compression when fine-scale information and long-term stability matter.Source: §5 · p. 23
• CANO fixes the cluster count a priori, and optimized clustering and cross-attention implementations remain important for extending it to large-scale three-dimensional applications.The authors identify adapting cluster count to solution complexity as a future direction.Source: §5 · p. 23

A Theoretical Connection to Linear Attention

• Generalized linear attention represents attention using feature mappings and an intermediate operation, providing the framework used to analyze efficient neural operators.The formulation is introduced to decouple quadratic Softmax computation.Source: A Theoretical Connection to Linear… · p. 24
• Transolver aggregates physical nodes into latent slices, applies self-attention there, and maps back; its slicing and deslicing mappings share a parameter, while compressed interactions risk losing fine-grained dynamics.The passage attributes these structural limitations to forced parameter sharing and interaction within the compressed slice space.Source: A Theoretical Connection to Linear… · p. 24
• CANO dynamically clusters queries for cross-attention instead of compressing both queries and keys, with full-resolution keys and values retained.Its query-side mapping acts as a dynamic cluster assignment, while the other mapping processes clustered queries against full-resolution keys.Source: A Theoretical Connection to Linear… · pp. 24–25

A Govering Equations

• The section presents governing equations for both standard PDE benchmarks and irregular-domain benchmarks.Source: B Govering Equations · p. 25

Airfoil.

• The Airfoil benchmark models transonic flow, treats viscosity as negligible under its low-viscosity conditions, and maps mesh locations to Mach number.The governing equations are the Euler equations for mass, momentum, and energy; the mesh resolution is 221 × 51.Source: Airfoil · p. 25

Pipe.

• The Pipe benchmark models incompressible flow through a pipe and predicts horizontal fluid velocity from a 129×129 structured mesh.Its governing equations are based on the Navier-Stokes relations for Newtonian fluids.Source: Pipe · pp. 25–26

Plasticity.

• The Plasticity benchmark simulates a plastic material impacted by an arbitrary-shaped die and predicts mesh-point deformation over the future 20 time steps.The die shape is supplied on a 101×31 structured mesh; the governing equation is an equilibrium equation for solid material dynamics.Source: Plasticity · p. 26

Navier-Stokes.

• The Navier-Stokes benchmark models incompressible viscous flow on a unit torus and predicts 10 future frames from 10 past frames on a 64 × 64 grid.With constant fluid density, the dynamics use a vorticity formulation and viscosity ν = 10−5.Source: Navier-Stokes · p. 26

Darcy Flow.

• Darcy Flow models fluid diffusion through porous media using a 2D steady-state elliptic equation.The dataset tests resolution of high-frequency spatial discontinuities in the permeability coefficient field a(x) on an 85×85 regular grid.Source: Darcy Flow · p. 26
• The Darcy Flow task uses the permeability coefficient field a(x) as its spatially varying input.The dataset evaluates whether the model resolves high-frequency discontinuities in this field.Source: Darcy Flow · p. 26

Elasticity.

• The Elasticity benchmarks estimate inner stress in an incompressible material with an arbitrary central void under external tension.They share the Plasticity dataset’s equilibrium equation and map material structure to inner stress.Source: Elasticity · p. 26

Irregular Darcy.

• Irregular Darcy uses 2,290 unstructured nodes and tests a Darcy-flow problem with highly irregular geometric boundaries.It maps the varying permeability coefficient field a(x) to pressure field u(x), sharing the standard Darcy flow’s 2D steady-state elliptic equation.Source: Irregular Darcy · p. 27

Pipe Turbulence.

• Pipe Turbulence predicts future velocity states from previous observations for incompressible flow in an unstructured pipe with 2,673 nodes.The governing equations are the same as those used for the Pipe benchmark.Source: Pipe Turbulence · p. 27

Composite.

• The Composite benchmark covers thermoelastic deformation of a 3D composite driven by temperature profiles across 8,232 nodes.It maps input 3D temperature profiles to structural stress fields under a steady-state equilibrium equation coupled with thermal expansion; the stress includes a thermal-stress term induced by temperature change.Source: Composite · p. 27