Predicting Physics in Mesh-reduced Space with Temporal Attention

1 INTRODUCTION

• Next-step models accumulate error and can drift in volume, energy, phase, or shape, motivating strategies such as training noise or solver-in-the-loop data augmentation.Training noise can be difficult to tune and can bound achievable accuracy; drift is less easily mitigated.Source: §1 · p. 1
• Transformers can capture long-term dependencies that help preserve phase and conserved quantities, but full-sequence memory grows with sequence length and spatial extent.This makes direct application to large unstructured meshes difficult.Source: §1 · pp. 1–2
• The method uses a GNN to aggregate local information onto pivotal nodes, applies whole-sequence transformer attention to the resulting low-dimensional latent, then decodes with a second GNN.The coarse-scale, upsampling, and fine-scale update sequence is related to a multigrid V-cycle.Source: §1 · p. 2
• The proposed GNN-and-transformer model outperforms MeshGraphNets [34] on challenging fluid dynamics tasks and produces stable rollouts without training noise or strong phase drift.The reported tasks include systems from sonic shocks to vascular flow.Source: §1 · p. 2

2 RELATED WORK

• GNN-based learned simulators support irregular or adaptive meshes and have been applied across systems including articulated dynamics and particle systems.The cited applications include [37], [23], and [38].Source: §2 · p. 2
• Prior sequence models address temporal dependencies, but high memory overhead limits their use on high-dimensional systems, while POD and CNN autoencoders have mesh or domain constraints.POD cannot handle variable-length state vectors, and CNN autoencoders require rectangular domains and uniform grids.Source: §2 · p. 2
• The proposed approach locally reduces the input by aggregating information on a coarsened graph, drawing on GNNs’ local rules and connections to multilevel methods and graph coarsening.The passage relates the approach to multi-level methods [50] [15] and graph coarsening [52] [2].Source: §2 · p. 2

3.1 PROBLEM DEFINITION AND OVERVIEW

• The target is a rapid rollout of mesh-discretized states from an initial state and system parameters, avoiding the expense of traditional numerical integration.The systems are governed by parameterized PDEs, and finite volume discretization represents the domain as mesh cells.Source: §3.1 · p. 3

3.2 GRAPH REPRESENTATION AND MESH REDUCTION

• Graph Mesh Reducer (GMR) locally summarizes the graph at selected pivotal nodes, yielding rich representations intended to approximate system dynamics on a coarser graph.The graph is built from mesh cells and neighboring relations, and pivotal nodes are generally selected by uniform sampling.Source: §3.2 · pp. 3–4
• The pivotal-node representations are concatenated into a low-dimensional latent, and GMUS restores a full-graph state by interpolating non-pivotal node representations and applying message passing.GMR and GMUS are trained as an autoencoder over time steps and sequences.Source: §3.2 · pp. 4–5

3.3 ATTENTION-BASED SIMULATOR

• The autoregressive transformer predicts latent states from the initial latent and system-parameter token, attending over the preceding latent sequence.The model can reduce to a next-step model under specified choices, while pivotal-node reduction and a longer input range lower computation and memory use.Source: §3.3 · p. 5
• Compared with next-step models, whole-sequence attention propagates gradients through the sequence and can use prior steps to support stable predictions with less drift and error accumulation.The latent prediction model is trained on full sequences in reduced space, avoiding joint training of the more memory-intensive encoder and decoder modules.Source: §3.3 · p. 5, §3.4 · p. 6

3.4 MODEL TRAINING AND TESTING

• At inference, the system parameters and initial state are encoded as tokens, the transformer predicts the latent sequence, and GMUS decodes each predicted latent into a mesh state.The training procedure incrementally adds loss terms from later time steps, which the authors found more stable than directly minimizing the overall objective.Source: §3.4 · p. 6

• Across cylinder, sonic, and vascular flows, predictions remain stable without noise injection and closely match CFD, including varying parameters and mesh sizes.The evaluation covers fixed, moving, and varying unstructured meshes.Source: §4 · p. 6
• On long-rollout cylinder and vascular tasks, the attention-based model's RMSE is significantly lower than next-step baselines; it outperforms baselines in most scenarios.Table 1 compares MeshGraphNet with and without noise injection against the LSTM, GRU, and Transformer variants.Source: §4 · p. 7, Table 1 · p. 7
• Mesh reduction raises initial error, but it grows slowly; MeshGraphNet accumulates error more strongly, noise injection only partly helps, and the Transformer outperforms GRU and LSTM variants.The paper attributes the sequence model's long-span error behavior to passing gradients through all previous steps.Source: §4 · pp. 7–8, Figure 3 · p. 8
• At long rollouts, the model retains cylinder-flow phase while MeshGraphNet drifts, and it preserves vascular left inflow as MeshGraphNet's inflow diminishes.The comparison includes cylinder flow at t = 250 and vascular flow at t = 228.Source: §4 · p. 8, Figure 4 · p. 8
• The encoder-decoder compresses state latents to at most 4% of the original representation with RMSE below 1.5%.Source: §4 · p. 8
• For cylinder flow, GMR latents separate trajectories at different Reynolds numbers, whereas PCA of system states makes those trajectories hard to distinguish.The GMR trajectories enter distinct periodic phases.Source: §4 · p. 8, Figure 5 · p. 9
• In cylinder flow, attention shifts from the parameter token toward vortex-shedding cycles, whose frequency matches the data; in sonic flow, parameter-token attention quickly decays.The cylinder-flow attention pattern is reported across the tested Reynolds-number range.Source: §4 · p. 9, Figure 7 · p. 9, Figure 6 · p. 9

• The proposed approach combines graph-based mesh reduction and temporal attention for efficient autoregressive prediction of complex physical dynamics.Source: §5 · p. 9
• Attention to whole simulation trajectories can help identify conserved properties or fundamental frequencies of the system.Source: §5 · p. 9
• Compared with next-step methods, the approach predicts with higher accuracy and less drift.Source: §5 · p. 9

• Red and blue nodes in the schematic mark pivotal nodes.Source: Figure 8 · p. 14
• GMR and GMUS reduce and recover a graph through information flow analogous to a V-cycle in multigrid methods.The schematic uses a seven-node graph and two message-passing layers as its example.Source: §A.1 · p. 14, Figure 8 · p. 14
• Increasing node size represents information aggregated from surrounding nodes during message passing.Source: §A.1 · p. 14, Figure 8 · p. 14

• GMR and GMUS require O(NLd*) computation, where N is the graph node count, L the layer count, and d* the maximum hidden-unit count.Source: §A.2 · p. 14
• The transformer requires O(T · (d′)²) computation per prediction, considering O(T) previous steps and a latent vector of length d′ at each step.Source: §A.2 · p. 14
• In the authors’ calculations, the transformer takes much less time than GMR or GMUS because the latent dimension d′ is small.Source: §A.2 · p. 14
• Without GMR, applying the transformer directly to system states would cost O(TN²d²), which the authors say is infeasible for typical computation servers.Source: §A.2 · p. 14

• GMR and GMUS are first trained to minimize the graph reconstruction loss Lgraph.Source: §A.3 · p. 15
• After encoder-decoder training, its parameters are frozen and temporal-attention parameters are trained on prediction loss Latt with early stopping.The algorithm describes updating the attention parameters until latent-prediction RMSE falls below a threshold.Source: §A.3 · p. 15
• The parameter-token encoder mlpp is trained jointly with temporal attention.Source: §A.3 · p. 15

A.4 DATASET DETAILS

• Cylinder, sonic, and vascular flow datasets are generated with OpenFOAM using the finite volume method on adaptive triangular meshes.The cylinder and vascular cases use incompressible Navier–Stokes equations; sonic flow uses compressible Navier–Stokes equations.Source: §A.4 · p. 15
• Cylinder and sonic flow each provide 51 training and 50 test trajectories, while vascular flow provides 11 training and 10 test trajectories.Source: §A.4 · p. 15
• Cylinder flow varies Reynolds number on a fixed mesh, sonic flow varies initial temperature on a fixed topology with moving node coordinates, and vascular flow varies thrombus radius across different mesh topologies.Source: §A.4 · p. 16

A.5 MODEL DETAILS

• GraphNet node and edge representations have width 128, and both GMR and GMUS use three GraphNet blocks with ReLU-activated MLP functions of hidden size 128.Source: §A.5 · p. 16
• The final MLP layer has output size 4 for cylinder and sonic flow and 2 for vascular flow.Source: §A.5 · p. 17
• The transformer uses one layer and four attention heads, with latent sizes 1024 for cylinder and sonic flow and 800 for vascular flow.Latent size is the number of pivotal nodes multiplied by the final GraphNet output size.Source: §A.5 · p. 17

A.6 PIVOTAL POINTS SELECTION

• Pivotal nodes are generally sampled uniformly: 256 of 1,699 cells for cylinder, 256 of 1,900 for sonic, and 400 of 7,561 for vascular flow.Vascular pivotal nodes are manually reduced in the aneurysm region, where flow features are not rich.Source: §A.6 · p. 17, Figure 12 · p. 18

A.7 DATA COMPRESSION

• GMR and GMUS compress original physical data to about 3% of its size, with slight reconstruction-accuracy loss; the transformer is not part of that compression process.Source: §A.7 · p. 19, Table 5 · p. 19

A.8 INFERENCE TIME

• The learned model predicts without subtimestepping and is significantly faster than finite-volume simulation across all three datasets.Inference separately predicts latent states with the transformer and decodes them with GMUS.Source: §A.8 · p. 19, Table 6 · p. 19

A.9 OTHER VARIABLE CONTOUR

• Pressure-contour rollouts are described as accurate over long sequences under different condition settings.Source: §A.9 · p. 19, Figure 13 · p. 20

A.10 FURTHER ANALYSIS OF LATENT ENCODINGS AND ATTENTION

• In sonic flow, different initial temperatures lead to rapidly diverging dynamics despite overlapping parameter tokens, while attention to z−1 rapidly decays.Attention distributions are shown as time-time log-scale contours, with each row summing to 1 after softmax.Source: §A.10 · p. 20, Figure 14 · p. 20, Figure 15 · p. 21

A.11 LONGER ROLLOUTS

• A cylinder model trained on 400-step trajectories is evaluated at twice that length, using a sliding attention window of n recent steps plus the parameter token.The window is used to limit memory for very long rollouts while retaining the same training complexity.Source: §A.11 · p. 21, Figure 16 · p. 21

A.12 PREDICTION ON CONVECTION-DIFFUSION FLOW

• RMSE is 1.03 × 10^-3 for the proposed model versus 2.79 × 10^-3 for MeshGraphNet on non-oscillatory convection-diffusion rollouts.The model is also reported to accurately predict these sequences.Source: §A.12 · p. 22

A.13 CARDIAC FLOW PREDICTION

• The model accurately predicts 10 cardiac cycles and is reported to perform significantly better than MeshGraphNet on cardiac-flow rollout error.Source: Table 8 · p. 22, §A.13 · p. 22, Figure 18 · p. 22