DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training

• PDE pre-training is challenging because datasets differ in dimensions, temporal steps, resolution, geometry, regularity, and numerical ranges.Source: §1 · p. 2
• DPOT combines autoregressive denoising with a Fourier-based transformer, using a nonlinear learnable frequency-space transform for kernel integral transforms.Source: §1 · p. 2
• The model is pre-trained on more than 100k trajectories from 10+ diverse PDE datasets and is scalable up to 0.5B parameters.Source: §1 · p. 2
• DPOT achieves state-of-the-art results on PDEBench, PDEArena, and CFDBench, reducing error by up to 52%, with transfer to 3D, high-resolution, and steady-state tasks.Source: §1 · p. 2

2.1. Neural Operators

• Neural-operator approaches include branch-and-trunk networks, frequency-domain methods, geometry-adapted extensions, time-dependent strategies, and physics-informed combinations.Source: §2.1 · p. 2
• Transformer architectures add techniques such as patchification and linear attention, while frequency-domain attention offers low memory and computational cost.Source: §2.1 · p. 2
• Despite architectural advances, neural operators generally require task-specific training and domain-specific data, motivating efforts to improve data efficiency.Source: §2.1 · p. 3

2.2. Pre-training in Scientific Machine Learning

• Pre-training has succeeded in NLP and vision and has emerging applications in scientific machine learning, including protein, molecule, climate, and weather modeling.Source: §2.2 · p. 3
• Early PDE pre-training explored contrastive learning, shared training across steady-state systems, in-context learning, and autoregressive training, but remained limited to specific equations and data.Source: §2.2 · p. 3

3.1. Overview of DPOT

• DPOT models time-dependent PDEs from observed trajectories, implicitly inferring unknown PDE parameters from adjacent frames while addressing uneven trajectory lengths and varied data shapes.The method combines denoising pre-training, preprocessing and sampling, and a flexible transformer architecture to handle these challenges.Source: §3.1 · p. 3

3.2. Auto-regressive Denoising Pre-training

• Injecting small-scale noise into previous frames trains the model to predict the next timestep, regularizing it and improving robustness while reducing the training–inference gap.The training objective samples one-step transitions across datasets.Source: §3.2 · pp. 3–4

3.3. Data Preprocessing and Sampling

• Resampling to H=128, channel padding, and geometry masks standardize inputs; dataset-weighted sampling balances representation independently of dataset size.Lower-resolution data are upscaled and higher-resolution data are downscaled or sampled; irregular geometries receive an additional mask channel.Source: §3.3 · p. 4

3.4. Model Architecture

• Patchification and temporal aggregation precede Fourier-domain global convolution; shared, parallelizable MLPs provide expressive spatial mixing that scales with depth and width.The architecture also uses a multi-head Fourier mixer to process feature groups in separate subspaces.Source: §3.4 · p. 5

3.5. Analysis of Our Model

• The analysis establishes universal approximation: Fourier attention layers can approximate continuous operators on compact subsets of Sobolev spaces to arbitrary ε>0.The theorem concerns mappings between Sobolev spaces.Source: §3.5 · pp. 5–6

4.1. Setup.

• Pre-training uses 12 datasets from four sources covering PDEs including Navier–Stokes, diffusion-reaction, and shallow-water equations.Across model scales, training used AdamW at a learning rate of 1 × 10−3 for 1000 epochs; prediction quality was measured with L2RE unless stated otherwise.Source: §4.1 · p. 6

4.2. Baselines

• Comparisons include operator-learning architectures and pre-trained MPP-AViT, alongside a larger FNO-m baseline.Except for MPP-AViT/FNO-m, the other methods were trained and evaluated on each sub-dataset individually.Source: §4.2 · p. 6

4.3. Main Results

• DPOT-L (0.5B) achieves SOTA in 9 of 12 tasks and reduces L2RE by more than 50% on 8 of 12 tasks versus zero-shot methods.More fine-tuning steps are usually better, with a trade-off in computational cost.Source: §4.3 · pp. 6–7, Table 1 · p. 7
• Small DPOT models achieve SOTA on 9 of 12 datasets, with up to 52% average improvement (0.116 to 0.0553 on FNO-1e −4).For larger zero-shot models, DPOT-L improves over DPOT-S on 10 datasets by up to 39%, though MPP-L has lower L2RE on PDB-DP.Source: §4.3 · p. 6, Table 1 · p. 7

4.4. Downstream Tasks Experiments

• Pre-trained initialization improves results on high-resolution turbulence, reduces 3D error from 41% to 22.6% despite 2D-only pre-training, and gives the best steady-state results.The downstream comparisons use the same number of training steps for DPOT from scratch and with pre-trained weights.Source: Table 2 · p. 7, §4.4 · p. 7

4.5. Scaling Experiments

• As model size scales from 7M to 500M parameters, zero-shot L2RE falls from 0.06 to 0.028; DPOT generalizes better across layer counts than AViT.The 7M DPOT model outperforms the 30M AViT model; fine-tuning further improves performance.Source: Figure 3 · p. 8, §4.5 · p. 8

4.6. Ablation Experiments

• Eight Fourier-attention heads yield the lowest testing loss, while patch size P = 8 gives significantly lower testing error than the tested alternatives.A small patch size such as P = 2 performs poorly.Source: Table 3 · p. 8, §4.6 · p. 8
• Test error decreases from 0.0178 to 0.0152 and then rises as injected-noise intensity increases, while training error rises with noise.The authors interpret an appropriate noise level as beneficial to generalization.Source: Table 4 · p. 8, §4.6 · p. 8

• DPOT combines a new auto-regressive denoising strategy with a model architecture based on Fourier attention.Source: §5 · p. 8
• The approach was scaled to 1B parameters using more than 10 PDE datasets.Source: §5 · p. 8
• Experiments examine PDE benchmarks and downstream PDE tasks, and the paper explores scaling behavior for PDE pre-training.The authors describe this as a pioneering large-scale exploration and point to potential future applications in practical problems.Source: §5 · p. 8

Broader Impact

• Neural-network predictions of physical systems can entail errors and lack interpretability, posing risks where safety and robustness are important.The paper identifies industrial manufacturing and scientific discovery as potential application areas for PDE pre-training.Source: Broader Impact · p. 9

A.1. Model sizes and training details

• Pre-training uses a 1 × 10^-3 learning rate, a One-cycle schedule over 1000 epochs, and eight A800 GPUs with total batch size 160.The setup uses 200 warm-up epochs, weight decay 1 × 10^-6, patch size 8, and predicts T = 10 next frames.Source: §A.1 · p. 12
• For downstream dimensions, Fourier attention shares parameters across frequency components, while FFT/IFFT, patch embedding, and positional embedding are adapted.For 3D data, the paper proposes replacing 2D FFT with 3D FFT while reusing existing parameters.Source: §A.1 · p. 12

A.2. Details of Datasets

• FNO-ν uses vorticity as its quantity of interest, PDEBench-SWE predicts water depth, and PDEBench-DR predicts density.Source: §A.2 · p. 12
• CFDBench predicts velocity and pressure on irregular geometries, while PDEBench-CMS and PDEArena-NS1/2 predict velocity, pressure, and density fields.Source: §A.2 · pp. 12–13
• The appendix includes a table describing configurations of DPOT models of different sizes.Source: Table 5 · p. 13

B.1. Results of DPOT with 1B Parameters

• DPOT-Huge nearly dominates the zero-shot results against MPP-Large and DPOT-L, with a slight decrease on CFDBench.The 1B model reports performance boosts of over 50% on some datasets, including FNO datasets and PDEArena-NS, compared with DPOT-L.Source: §B.1 · p. 13

B.2. Timing experiments

• Inference speed is measured as the average time to predict a single timestep.The results are presented in Table B.2.Source: §B.2 · p. 13

B.3. Comparison with AFNO architecture

• In the DPOT–AFNO comparison, enforcing frequency-domain sparsity raises training loss and worsens test performance.The authors conclude that soft shrink might not suit PDE operator learning.Source: §B.3 · p. 13

B.4. Performance under varying resolutions

• For resolution generalization, the authors modify DPOT’s convolutional patchification layers following CNO and measure L2RE at different resolutions.The appendix identifies Table B.4 as the results location but gives no result values in the supplied passage.Source: §B.4 · p. 13

B.5. Supplementary Experiments on Datasets with Long Trajectories

• Pretrained DPOT’s prediction error increases slowly as trajectories lengthen, while the randomly initialized model’s error grows quickly.At very short step sizes, pre-training adds little because prediction accuracy is already high.Source: §B.5 · p. 13

B.6. Additional Results on SuperBench Dataset

• On SuperBench’s 8× upsampling task, pretrained DPOT significantly outperforms EDSR, while non-pretrained DPOT performs comparably to EDSR.The task upsamples Navier-Stokes Kraichnan turbulence data at Re=16000 from 128×128 to 1024×1024; RFNE is lower-is-better.Source: §B.6 · p. 14

B.7. Supplementary experiments on Kolmogorov turbulence flow

• 33.5% full-trajectory test error for DPOT-pretrained compares with 82.2% for vanilla DPOT and 104% for FNO on Kolmogorov turbulence.The authors report that the latter models learned almost nothing, especially as time progressed.Source: §B.7 · p. 14, Table 12 · p. 16

B.8. Comparison between Equal Data Sampling and Our Balanced Data Sampling

• Balanced sampling outperformed equal sampling overall and produced more even convergence across datasets.Equal sampling was slightly better on some datasets, including PDB-M1, but showed poor convergence on others such as FNO-1e-5, PDB-DR, and PDB-SWE.Source: §B.8 · p. 14, Table 13 · p. 16

B.9. Comparison between Self-attention and Fourier Mixer

• About 25% lower test errors were reported for the Fourier mixer than for standard self-attention, which required more parameters and computational resources.The comparison used subsets of Compressible Navier-Stokes equations from PDEBench.Source: §B.9 · p. 15, Table 14 · p. 16

C. Proof of Universal Approximation of Fourier Attention

• The paper states that Fourier attention layers can approximate any continuous operators, establishing the claimed universal-approximation capability of DPOT’s core layers.The argument is framed for functions on a periodic torus.Source: C. Proof of Universal Approximation of… · p. 15, §C.1 · p. 15

C.1. Preliminaries and Definition

• DPOT is defined as a neural operator built from Fourier-attention layers, with lifting into hidden space and projection back to the output space.The setup uses discrete Fourier transforms on values represented on a regular mesh.Source: §C.1 · pp. 15, 17

C.2. Proof of the theorem

• The proof aims to show that a DPOT can approximate any operator on a d-dimensional torus with mesh M.It proceeds in three steps: construct an equivariant proxy, approximate it with a Sumformer, then fit that Sumformer with DPOT.Source: §C.2 · p. 18
• The proxy target is made equivariant by attaching a permutation to each input and undoing that permutation in the target output.The proof establishes the proxy’s equivariance under input reordering before using the Sumformer approximation theorem.Source: §C.2 · p. 18
• The Sumformer approximates the equivariant proxy within ε in the infinity norm.This is the intermediate approximation step used before constructing a corresponding DPOT.Source: §C.2 · pp. 18–19
• A DPOT is constructed to approximate the Sumformer by using Fourier-attention layers to realize its component mappings.The construction uses lifting with positional encoding, a Fourier-domain mapping, and an MLP mapping to approximate ψ and produce the Sumformer output.Source: §C.2 · pp. 19–20

D. Limitations of the work.

• The authors identify risk-sensitive scenarios as a safety concern because neural networks have intricate mathematics and theoretical research is relatively scarce.The paper characterizes its contribution as an empirical and numerical study.Source: D. Limitations of the work · p. 20