PaperlayerView exact revision on arXiv

Source-linked AI summary

Distance-Aware Attention and Wall-Distance Expert Routing for Transformer-Based 3D Flow Prediction

Sanghyeon Kim, Sunwoong Yang, Namwoo Kang

arXiv:2609.07222v1cs.AI

TL;DR

Transformer flow surrogates process all points with wall-blind retrieval and shared transformations, despite sharply different near-wall and outer-flow physics. The paper adds wall- and geometry-conditioned retrieval and expert routing to AB-UPT and Transolver-3. On DrivAerML, the combined method reduces volume pressure error by 12.5% at 50 training cases and improves results across larger-data, backbone, and unseen-body evaluations.

  • Problem

    Transformer surrogates retrieve from compressed representations and transform every point without explicitly using wall-relative physical structure, although baseline errors concentrate near walls and in separated flow.

  • Method

    DA-CA modulates each volume query by wall distance before cross-attention, while SVMoE routes volume and surface points through expert mixtures using wall distance and local geometry.

  • Results

    The combined method reduces volume pressure error by 12.5% on DrivAerML with 50 training samples, while also improving near-wall and far-zone performance relative to the baseline.

  • Takeaways & Limitations

    Wall-related conditioning improves retrieval and transformation across the evaluated surrogate settings, with volume experts organizing into near-wall, transition, and free-stream bands without assignment supervision.

  • Takeaways & Limitations

    The interpretation of expert organization is limited to contiguous regions along a prescribed physical axis; comparisons with canonical y+ thresholds and detailed surface-expert analysis remain future work.

Abstract

from arXiv · show

Transformer surrogates for 3D flow prediction compress an industrial mesh into a small set of tokens from which every prediction point reads. Two operations follow: the retrieval step in which a point gathers information from the compressed representation, and the feed-forward layer that transforms what it retrieved. In current backbones both are blind to where the point sits in the flow. We condition both on wall-related physical signals. Distance-aware cross-attention (DA-CA) reshapes each volume query by its wall distance before retrieval, so that a point deep in the boundary layer draws different geometric information than one in the outer flow. Surface-volume mixture-of-experts (SVMoE) replaces the shared feed-forward layer with a small set of experts, routed by wall distance for volume points and by local geometry for surface points. Neither mechanism is tied to one architecture, so we apply both unchanged to AB-UPT and Transolver-3. On DrivAerML with 50 training cases, DA-CA reduces the volume pressure error by 10.1%, and DA-CA and SVMoE together reduce it by 12.5%; DA-CA improves the near-wall region at some cost in the far region, which SVMoE recovers, and the volume experts settle into near-wall, transition, and free-stream bands without routing supervision. Retrained on 300 cases, the conditioning improves every field quantity, reducing volume pressure and velocity errors by 33.1% and 18.6% on AB-UPT and by 21.4% and 21.3% on Transolver-3. Under Leave-One-Body-Out evaluation on DrivAerNet++, it reduces the volume pressure error on unseen body types by up to 14.2%.

1 Introduction

Industrial 3D flow surrogates must preserve localized structures near walls and in wakes, but scalable transformer backbones retrieve and transform every point without wall-aware physical conditioning. The paper introduces DA-CA and SVMoE to inject wall distance and local geometry into these two operations.

  • Motivation: Thin boundary layers, separated shear layers, recirculation zones, and wakes occupy little volume but control engineering quantities such as drag, lift, heat transfer, and pressure loss.A model can therefore perform well over most of the domain while failing where design decisions are determined.
  • Motivation: A single DrivAerML hybrid RANS-LES case requires about 60,000 core-hours, making high-fidelity simulation prohibitive for early-stage exploration.The dataset contains roughly 8.8 million surface cells and 160 million volume cells per vehicle variant.
  • Motivation: Transformer surrogates compress industrial meshes into compact tokens, after which prediction points retrieve information and apply a shared feed-forward transformation.AB-UPT uses anchor tokens, while Transolver uses learned physical-state slices.
  • Problem: The retrieval and feed-forward operations ignore a point’s wall-relative location, despite different physical regimes in boundary layers, wakes, surfaces, and free stream.Under limited high-fidelity supervision, the baseline learns wall-dependent behavior inefficiently and attention varies only weakly with distance.
  • Approach: DA-CA conditions each volume query on wall distance before retrieval, while SVMoE routes gathered information through experts using wall distance for volume points and local geometry for surface points.DA-CA changes what a token retrieves; SVMoE changes how the retrieved information is transformed.

2 Related Work

The paper situates its approach among geometry-aware 3D flow surrogates, position-conditioned attention, conditional modulation, mixture-of-experts models, and DrivAer-based aerodynamic benchmarks. Its key distinction is conditioning retrieval on a token’s physical wall distance rather than on pairwise token position.

  • UPT compresses irregular inputs into fixed-size latent tokens and decodes solutions at arbitrary query locations.
  • Existing attention methods encode relative token position through bucketed offsets, rotary embeddings, or distance penalties applied to pairwise scores.
  • DA-CA uses a token’s physical wall distance to modify its query representation, changing what information it retrieves rather than only its coupling strength.
  • Conditional computation commonly modulates features using auxiliary signals, while CFD surrogates often incorporate geometry-derived signals as input features.
  • Mixture-of-experts models increase capacity by routing tokens to a subset of experts, with sparse and differentiable routing as established variants.
  • DrivAerML supplies high-fidelity fields for 500 morphed notchback variants, whereas DrivAerNet++ supports out-of-distribution evaluation across multiple body types.

3 Backbones and Experimental Setup

AB-UPT and Transolver-3 compress mesh information differently but share a retrieval step followed by a point-shared feed-forward transformation. The experiments examine wall-aware conditioning under controlled architectural, data, and compute choices, including a two-branch AB-UPT configuration and training-set saturation.

  • AB-UPT and Transolver-3 implement retrieval and transformation differently, providing two backbones for applying the proposed mechanisms.
  • AB-UPT separates geometry, surface, and volume processing, with cross-attention allowing volume points to draw on nearby surface information.
  • Transolver compresses point features into physical-state slice tokens, applies self-attention among slices, and reconstructs point features from the same assignments.
  • Both backbones first retrieve information from compressed tokens and then apply a feed-forward network shared across points, without conditioning either step on flow location.
  • All AB-UPT experiments omit the geometry branch and compare the unconditioned and proposed models using the same surface-volume configuration.
  • Relative L2 error saturates near N_train = 50 under fixed compute, with training-set expansion to 200 samples leaving the three test cases largely unchanged.
  • Under the relevant fixed-wall-clock budget, the authors interpret accuracy as being limited more by surrogate inductive bias than by training-data quantity.

4 Distance-Aware Cross-Attention (DA-CA)

DA-CA targets the baseline’s wall-blind retrieval by conditioning each volume query on wall distance before cross-attention. It changes attention toward physically relevant regions and lowers field error, while exposing a wake trade-off that motivates subsequent feed-forward specialization.

  • Motivation: Cross-attention is the dominant accuracy contributor, because removing it raises volume velocity error from 7.74% to 9.73%, while removing self-attention leaves accuracy near baseline.The ablation keeps twelve attention blocks fixed while redistributing them between self- and cross-attention.
  • Limitations of vanilla attention: The baseline’s attention kernel is wall-blind: its score uses learned query and key features without explicitly receiving wall distance.Although positional encodings contain coordinates, the learned projections do not reliably organize attention along the wall-distance axis under the studied data and compute budgets.
  • Limitations of vanilla attention: Baseline error concentrates in the near-wall band and rear wake, decreasing monotonically as wall distance increases.This localization identifies small wall distance as the region where conditioning is most needed.
  • The DA-CA mechanism: DA-CA reshapes each volume query using its wall distance before retrieval, while leaving keys, values, self-attention, and the surrounding anchor structure unchanged.The learned gain and bias act only on the query branch that reads from surface anchors.
  • Results and analysis: DA-CA learns nonzero, wall-distance-dependent modulation and shifts cross-attention toward nearer distance-ranked anchors.The conditioning begins as an exact copy of unconditioned attention, then develops learned scale and shift functions; nearest-bin weight mass rises by roughly an order of magnitude.
  • Results and analysis: 10.1%: DA-CA reduces DrivAerML volume pressure error under the 50-sample training regime.The improvement is accompanied by better near-wall and side-region behavior, while the wake shows little improvement because the shared feed-forward layer remains unchanged.

5 Surface-Volume Mixture-of-Experts (SVMoE)

SVMoE replaces the shared feed-forward transformation with separate surface and volume expert mixtures, routing volume tokens by wall distance and surface tokens by local geometry. The learned routing forms physically interpretable regions and complements DA-CA by improving outer-flow and wake accuracy while preserving near-wall gains.

  • Motivation and design: SVMoE replaces the shared feed-forward layer with partially different nonlinear maps for surface tokens and volume tokens at different depths.The design keeps the backbone unchanged while conditioning how retrieved information is transformed.
  • Routing mechanisms: Volume tokens use three experts routed only by wall distance, while surface tokens use three experts routed by a local geometric descriptor.The volume router encodes log wall distance; the surface descriptor includes normal alignment, neighborhood shape, and normal variation.
  • Emergent expert structure: 77.8% of volume routing mass goes to the near-wall expert, 8.3% to the transition expert, and 13.9% to the free-stream expert.The experts dominate disjoint wall-distance intervals: dw < 0.045, 0.045 ≤ dw ≤ 0.128, and dw > 0.128.
  • Emergent expert structure: The volume partition emerges during training without expert-assignment supervision, gradually separating into bands consistent with boundary-layer stratification.Routers receive only dw and ϕs; the authors qualify emergence as organization along a prescribed physical axis rather than discovery of higher-dimensional structure.
  • Accuracy and complementarity: DA-CA lowers near-wall velocity error from 12.5% to 11.3% but raises far-zone error from 5.5% to 6.1%, whereas adding SVMoE lowers them to 10.9% and 5.3%.This complements DA-CA’s wall-proximal information aggregation with a separate nonlinear transformation for the outer band and wake.
  • Accuracy and complementarity: With 300 training samples, conditioning improves every field quantity on both AB-UPT and Transolver-3, including volume pressure reductions of 33.1% and 21.4%, respectively.Volume velocity errors fall by 18.6% on AB-UPT and 21.3% on Transolver-3.

6 Generalization across Geometric Families

The proposed model transfers across DrivAerML and DrivAerNet++ despite substantial geometric distribution shift, improving predictions on pooled and unseen body types. Under Leave-One-Body-Out evaluation, gains persist across all held-out bodies and field quantities, while routing proportions remain stable.

  • Geometric distribution shift: DrivAerNet++ forms three body-type clusters that overlap DrivAerML only slightly, creating a geometric-family shift concentrated near the rear silhouette.The rear-quarter variation changes wake topology and rear-pressure recovery, testing transfer across geometric families.
  • Case 1: Multi-body evaluation: Across all three body silhouettes and four field quantities, DA-CA + SVMoE improves on the unconditioned baseline when training includes every body type.Case 1 evaluates heterogeneous multi-body training rather than unseen-shape generalization.
  • Case 2: Leave-One-Body-Out: Up to 14.2% error reduction is achieved across all twelve Leave-One-Body-Out cells for unseen body types and quantities.The proposed model improves on the baseline for Estateback, Fastback, and Notchback, although errors are higher under genuinely out-of-distribution evaluation.
  • Routing stability: Volume routing retains near-wall, transition, and free-stream proportions across LOBO sub-cases despite receiving only wall distance and no body-type information.The stability links the unseen-body gains to routing behavior that transfers across geometric families.
  • Case 2: Leave-One-Body-Out: The relative ordering of DA-CA + SVMoE and the unconditioned baseline is preserved across all three held-out body types.Figure 13 reports the same ordering in field comparisons for Estateback, Fastback, and Notchback.

7 Conclusion

Transformer surrogates normally retrieve and transform tokens without conditioning those operations on a point’s physical location, leaving near-wall errors poorly addressed. The paper adds wall-distance-aware retrieval and physically routed experts, improving accuracy across data scales, backbones, and unseen geometries, while identifying limits around unsteady flows, larger datasets, wake structure, and OOD backbone coverage.

  • Problem: Transformer surrogates use feature-similarity retrieval and a shared feed-forward mapping despite different physics across surface, boundary-layer, and free-stream points.Baseline cross-attention remains nearly flat with distance, while errors concentrate near the wall and do not change beyond roughly 50 training cases.
  • Method: DA-CA conditions retrieval on wall distance, while SVMoE routes the feed-forward transformation using wall distance for volume points and geometry for surface points.Together they insert physical distinctions into the two computation stages that were previously location-blind.
  • Results: 33.1% and 18.6% reductions in volume pressure and velocity errors occur on AB-UPT, versus 21.4% and 21.3% on Transolver-3 after retraining on 300 cases.The same components improve both evaluated backbones under the identical protocol.
  • Limitations: The evidence covers steady-state flows, at most 300 training cases, scalar wall-distance routing, and OOD evaluation on AB-UPT only.Wake points at equal wall distance are treated alike, and substantially larger datasets or unsteady flows remain untested.
Loading 2609.07222v1…