PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Local Accuracy: Benchmarking the Rollout Reliability of PDE Foundation Models

Jaewan Park, Seid Koric, Giri Krishnan, Sudhakar Pamidighantam, Surresh Marru

arXiv:2610.31935v1cs.LG

TL;DR

Short-horizon performance does not establish whether PDE foundation-model advantages persist over full autoregressive rollouts. The paper benchmarks models across paired horizons and finds that favorable comparisons against scratch baselines can coexist with widespread error amplification.

  • Problem

    Whether a PDE foundation model’s short-horizon advantage persists over a full autoregressive rollout remains an open evaluation question.

    Source: §1 · p. 2

  • Method

    The benchmark compares short- and full-horizon performance across six paired datasets while varying the scoring horizon within matched model, dataset, budget, and checkpoint comparisons.

    Source: §1 · p. 3, §3.1 · p. 5, §6.3 · p. 15

  • Results

    At least one fine-tuned foundation model beats both scratch baselines in 26 of 30 short-horizon cells, while full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired configurations.

    Source: §7 · p. 16

  • Takeaways & Limitations

    Evaluation should report paired-horizon errors, amplification, and failures rather than rely on a single local forecast score.

    Source: §7 · pp. 16–17

    The benchmark compares complete model recipes and therefore does not isolate the causal effect of pretraining.

    Source: §6.4 · p. 16

Abstract

from arXiv · show

PDE foundation models enable data-efficient adaptation but are deployed autoregressively, so short-horizon accuracy need not imply reliable simulation. We ask whether the short-horizon advantage of a foundation-model recipe persists when its predictions are repeatedly fed back as inputs for a long horizon. We define error amplification as G=E(H_f)/E(H_s), which measures full-horizon error relative to short-horizon error. We evaluate seven models on seven two-dimensional time-dependent PDE datasets: scratch-trained FNO and DeepONet; fine-tuned DPOT, GPhyT, Poseidon, and Walrus; and frozen in-context VICON. We report VRMSE, relative L_2, and relative MAE. Six datasets provide paired short- and full-horizon scores, while CE-RM contributes short-rollout sweeps only. At least one fine-tuned foundation model outperforms both scratch baselines in 26 of 30 short-rollout dataset--budget cells under every metric. Nevertheless, full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired trained configurations, with a median error amplification of $2.04$. The count below VRMSE one falls from 104 to 53, and 12 full rollouts exceed ten although none does at the short horizon. Walrus achieves the lowest full-horizon VRMSE in 16 of 24 cells, yet its median error amplification is $2.70$. Relative L_2 and relative MAE show the same qualitative pattern. These results show that favorable short-horizon rankings can coexist with substantial error amplification. PDE foundation-model benchmarks should therefore report horizon-resolved absolute errors, amplification statistics, and failure incidence rather than a single local forecast score.

1 Introduction

Short-horizon accuracy alone cannot establish whether a PDE foundation model remains useful during autoregressive rollout. This benchmark compares short- and full-horizon performance and finds that foundation-model advantages can coexist with substantial error amplification.

Motivation

  • Repeated self-conditioning changes the rollout inputs from ground-truth states to model-generated states, so local accuracy does not establish full-trajectory reliability.Small phase, amplitude, or conservation errors can affect subsequent predictions.Source: §1 · p. 2

Evaluation rationale

  • Error amplification is the ratio of full-horizon error to short-horizon error, and neither this ratio nor short-horizon error alone describes rollout behavior.Low amplification can reflect consistently poor predictions, while low short-horizon error can conceal later divergence.Source: §1 · p. 2

Main findings

  • Error amplification exceeds one in nearly every paired configuration, even when model rankings remain stable across horizons.Configurations can also cross from VRMSE below one to above one during full rollout.Source: §1 · p. 2

Benchmark scope

  • The benchmark compares short- and full-horizon errors across six datasets and reports VRMSE, relative L2, and RMAE.CE-RM adds a short-rollout data-scaling study, but its trajectories do not support paired horizon analysis.Source: §1 · pp. 2–3

Main findings

  • At least one fine-tuned foundation model outperforms both scratch baselines in 87% of all cases.The comparison is evaluated at the same downstream-data budget.Source: §1 · pp. 1, 3

2 Related work

PDE operator learning includes task-specific neural operators and pretrained models intended to transfer representations across physical systems. This benchmark combines existing datasets and evaluates complete model recipes across horizons, data budgets, and error metrics.

Neural operators for PDEs

  • DeepONet and FNO provide complementary task-specific baselines, using branch-and-trunk networks and Fourier-space spectral convolutions, respectively.In this benchmark, both are trained from scratch on each downstream dataset.Source: §2 · p. 3

PDE foundation models

  • PDE foundation models are pretrained on trajectories spanning multiple equations, parameter regimes, resolutions, or domains, then transferred by fine-tuning or in-context conditioning to reduce downstream data or adaptation effort.In the benchmark, DPOT, GPhyT, Poseidon, and Walrus are fine-tuned across downstream trajectory budgets, while VICON remains frozen and uses in-context demonstrations.Source: §2 · pp. 3–4

PDE datasets and benchmarks

  • Existing PDE benchmarks support cross-equation evaluation, while datasets such as RealPDEBench also pair real measurements with numerical simulations [4] [6] [14].Other collections broaden available physical and dataset coverage, including PDEGym and The Well [8].Source: §2 · p. 4

Benchmark contribution

  • The benchmark combines seven existing two-dimensional, time-dependent datasets rather than introducing a new physical dataset.Six datasets support paired short- and full-horizon evaluation; CE-RM instead supplies a trajectory-budget sweep through N = 1024 because its trajectories contain only 21 frames.Source: §2 · p. 4

Benchmark design

  • Within paired comparisons, the model, dataset, trajectory budget, and checkpoint are held fixed while the scoring horizon changes.This design connects conditional evaluation of foundation models with rollout-focused assessment of autoregressive surrogates [13] [14] [15].Source: §2 · pp. 4–5

3 Benchmark question and measurements

The benchmark evaluates autoregressive predictions over paired scoring horizons while preserving each model’s native temporal configuration. It combines field-level error metrics with horizon-amplification measures, while treating rollout reliability as a finite-horizon empirical concept rather than a formal stability guarantee.

3.1 Autoregressive rollout

  • During rollout, model-generated states re-enter the model-specific input history, so evaluation probes both local transitions and the dynamics produced by iteration.Input-history length and the number of states emitted per call may vary by model.Source: §3.1 · p. 5

3.3 Metric and Temporal Coverage

  • Six datasets use paired short- and full-horizon scores, with Hs covering 11 predicted frames and Hf varying by dataset; CE-RM contributes short-rollout results only.The six trained models have all three metrics at both scoring intervals; CE-RM’s 21-frame trajectories preclude paired horizon-amplification analysis.Source: §3.1 · p. 5, §3.3 · p. 6

3.2 Metrics

  • VRMSE compares error with field spatial variability; relative L2 and relative MAE provide checks against conclusions driven by that normalization.A metric value greater than ten is an operational flag for catastrophic rollout failure, not a formal stability certificate.Source: §3.2 · p. 5

3.4 Operational Meaning of Rollout Reliability

  • The horizon-amplification factor compares errors aggregated over full and short scoring intervals; values above one indicate amplification, but the ratio cannot be interpreted alone.The benchmark also considers the paired errors, threshold crossings, winner changes, and operationally defined catastrophic failures.Source: §3.4 · p. 6

4 Benchmark design

The benchmark spans seven PDE datasets and six swept trained models, with frozen VICON as a separate reference. Six datasets support paired short- and full-horizon evaluation; CE-RM supplies short-horizon results only.

4.1 Datasets and splits

  • The benchmark uses extrapolative RealPDEBench test splits, held-out DR and CondNS tests, and a fixed CE-RM test set that enables the sweep to N = 1024.The other datasets end their training sweeps at N = 32 or 64.Source: §4.1 · p. 6
  • Six datasets provide short- and full-horizon scores, while CE-RM provides short-horizon scores only; RealPDEBench entries use simulated fields, not experimental measurements.All seven datasets provide VRMSE, relative L2, and relative MAE.Source: Table 1 · p. 7

4.2 Models and adaptation tracks

  • Six models have checkpoints at each available N: scratch-trained FNO and DeepONet, plus fine-tuned DPOT, GPhyT, Poseidon, and Walrus; frozen VICON is a separate in-context reference.Model comparisons align nominal trajectory budgets, test trajectories, and metric definitions, but not parameter count, optimization compute, or subset construction; they compare adaptation recipes, not the causal effect of pretraining alone.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Evaluation protocol

  • Models retain their native temporal and rollout procedures, while short- and full-horizon intervals determine which predictions are averaged.For FNO, DeepONet, DPOT, and GPhyT, checkpoints are selected by one-step teacher-forced validation loss; Poseidon uses one-step supervised loss, whereas Walrus uses multi-step rollout loss.Source: §4.3 · pp. 7–8
  • Trained models are scored at native resolution, whereas VICON uses a fixed 128 × 128 grid and is evaluated as a separate reference track.Predictions are scored in raw physical units, with FNO and DeepONet de-normalized before evaluation.Source: §4.3 · p. 8

5 Results

Across paired evaluations, full-horizon errors usually exceed short-horizon errors, even when foundation-model recipes often outperform scratch baselines. The results also show that model rankings and rollout behavior depend on dataset, budget, and inference setting.

5.1 Longer horizons expose widespread error amplification

  • At least one fine-tuned foundation model beats both scratch baselines in 26 of 30 short-rollout cells and 21 of 24 full-horizon cells under every metric.The VRMSE winner is retained in 18 of 24 cells, including 15 where Walrus wins at both horizons, but 104 of 360 pairwise VRMSE orderings reverse.Source: §5.1 · p. 9, Table 3 · p. 10
  • Full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired configurations, with median amplification G = 2.04; 12 full rollouts exceed ten, versus none at the short horizon.VRMSE below one falls from 104 of 144 paired short-horizon configurations to 53 at the full horizon; relative L2 and relative MAE show the same qualitative pattern.Source: §5.1 · pp. 9–10, Figure 1 · p. 9

5.2 The horizon gap persists at the largest matched data budget

  • At the largest matched budgets, full-horizon VRMSE exceeds short-horizon VRMSE in all 36 model–dataset configurations, with median amplification G = 2.91.Walrus rises from 0.1093 to 0.7401 on FSI and from 0.1483 to 0.5498 on CondNS; full-horizon winners vary across datasets.Source: §5.2 · p. 10, Table 4 · p. 12
  • Cylinder is an exception to foundation-model dominance: at N = 64, scratch-trained FNO has short/full VRMSE of 0.0043/0.0147 and lower full-horizon error than every fine-tuned recipe.The short-horizon winner is retained on four of six paired datasets; controlled cylinder changes from GPhyT to DPOT, and foil from Walrus to DPOT.Source: §5.2 · p. 10

5.3 Data-efficiency trends and rollout failures are model-specific

  • Across short-horizon sweeps, Walrus wins 19 of 30 dataset–budget cells, while CE-RM shifts from FNO wins at N = 1 and 4 to Poseidon wins from N = 16 through 1024.Walrus's CE-RM VRMSE is non-monotone, rising from 0.1659 at N = 64 to 0.2001 at N = 256 before falling to 0.1133 at N = 1024.Source: Figure 2 · p. 11, §5.3 · p. 12
  • No short-horizon configuration exceeds VRMSE ten, but 12 of 144 paired full-horizon configurations do, including extreme failures at N = 16 on foil and controlled cylinder.At N = 16, DPOT reaches 3.63 × 10^6 on foil and GPhyT reaches 1.07 × 10^12 on controlled cylinder; scratch FNO also wins select cylinder and CE-RM regimes.Source: §5.3 · p. 12

5.4 Frozen in-context inference changes the question

  • Frozen in-context VICON has short-horizon VRMSE from 0.7660 to 0.9933 on six paired datasets, but all six full-horizon scores exceed one.Its median amplification is G = 1.38, while CE-RM supplies only a short-horizon reference and no full-horizon or N-sweep result.Source: §5.4 · p. 12, Table 6 · p. 14

6 Discussion

Fine-tuned foundation-model recipes often outperform scratch baselines, but that relative advantage does not ensure accurate long rollouts. Error amplification exposes widespread loss of absolute fidelity, while model-selection choices and the benchmark’s scope constrain interpretation.

6.1 What transfers and what does not

  • At least one fine-tuned foundation model beats both scratch baselines in 21 of 24 full-horizon cells, but this does not guarantee faithful dynamics.The corresponding short-horizon count is 26 of 30 cells, and the relative advantage and absolute rollout fidelity are distinct.Source: §6.1 · p. 14

6.2 What error amplification reveals

  • Full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired configurations, with median amplification 2.04.Of 104 configurations below VRMSE one at the short horizon, 51 cross to at least one; 12 full-horizon outcomes exceed ten although none do at the short horizon.Source: §6.2 · pp. 14–15
  • Amplification should be read alongside both absolute errors: a near-one ratio can reflect poor accuracy at both horizons, and it is not a stability certificate.It is a within-dataset diagnostic, not a per-step growth rate or evidence of monotonic error increase.Source: §6.2 · p. 15

6.3 Implications for model selection and benchmark design

  • Five of six swept recipes select checkpoints using one-step supervised validation loss, whereas Walrus uses multi-step rollout validation and early stopping.The study does not establish that one-step selection causes the horizon gap; Walrus also has median VRMSE amplification of 2.70.Source: §6.3 · p. 15
  • Benchmarks should report paired near-term and deployment-window errors, amplification, and failure incidence, with transparent checkpoint-selection protocols.Error–time curves and per-trajectory distributions can distinguish gradual degradation from a small number of severe failures.Source: §6.3 · p. 15

6.4 Scope and limitations

  • The evidence covers seven two-dimensional, grid-based simulation datasets, with CE-RM limited to short-horizon analysis and no test of experimental noise or sim-to-real transfer.The metrics quantify field-level error but do not identify its physical or numerical cause or establish formal stability.Source: §6.4 · p. 16

7 Conclusion

The paper evaluates PDE models across time-dependent systems to test whether foundation-model recipes retain their short-horizon advantage under autoregressive rollout. Their relative advantage often persists, but absolute fidelity commonly deteriorates as the horizon extends.

7 Conclusion

  • Full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired configurations, with median amplification 2.04.Fifty-one configurations cross from VRMSE below one to at least one, and 12 exceed ten only over the full horizon.Source: §7 · p. 16
  • At least one fine-tuned foundation model outperforms both scratch baselines in 26 of 30 short-horizon cells and 21 of 24 full-horizon cells under every metric.The secondary metrics show the same qualitative horizon-dependent pattern.Source: §7 · p. 16
  • A retained leaderboard rank or low near-term error does not establish that repeated self-conditioning remains faithful.Horizon amplification should be interpreted together with short- and full-horizon absolute errors.Source: §7 · pp. 16–17
  • Because the benchmark compares complete model recipes, it establishes a conditional empirical advantage, not the isolated causal effect of pretraining.The recommended evaluation uses paired horizons, explicit failure reporting, and transparent adaptation and checkpoint-selection protocols.Source: §7 · p. 17

Reproducibility and supporting material

The supporting material describes the benchmark harness and stored run artifacts.

Reproducibility and supporting material

  • The repository maintains the benchmark harness, model adapters, protocol documentation, and per-model instructions; each evaluated cell stores its rollout archive and training log.The horizon-resolved harvest computes all three metrics from saved predictions in raw units.Source: Reproducibility and supporting material · p. 17

A Complete full-horizon VRMSE results

The appendix identifies the full-horizon VRMSE table’s coverage and scoring intervals.

A Complete full-horizon VRMSE results

B Complete relative-L2 results

Table 8 reports relative L2 results for all seven datasets and every available checkpoint. The first six datasets have short/full entries, while CE-RM has short-rollout entries only.

B Complete relative-L2 results

C Complete relative-MAE results

Table 9 reports relative MAE results for every checkpoint. The first six datasets have short/full entries, while CE-RM has short-rollout entries only.

C Complete relative-MAE results

D Complete frozen in-context reference metrics

The appendix provides frozen in-context VICON metrics, additional Walrus full-pool endpoints, compute and result provenance, field visualizations, and per-frame rollout error curves.

D Complete frozen in-context reference metrics

  • Table 10 gives frozen in-context VICON metrics across all seven datasets, with short/full entries for six and short-only CE-RM entries.VICON is excluded from the trained-model aggregate counts in Table 3.Source: Table 10 · p. 21

E Additional Walrus full-pool endpoints

  • Additional Walrus full-pool endpoints are reported as short/full metrics, outside same-N aggregates and rankings.These endpoints cover four RealPDEBench datasets; exact pool sizes are provided for FSI, controlled cylinder, and foil, but not cylinder.Source: Table 11 · p. 21, E Additional Walrus full-pool endpoints · p. 21

F Compute and result provenance

  • Retained rollout archives include predictions, ground truth, and training logs, enabling trajectory-level uncertainty and physics-based diagnostics without rerunning inference.Walrus and VICON also retain per-trajectory prediction archives.Source: F Compute and result provenance · p. 21

G Field visualizations

  • Field figures compare ground truth with model rollouts across t = 10–20 and show a t = 20 diagnostic as training trajectories grow.Panels share a perceptually uniform ground-truth-percentile color scale; columns are matched physical fields, and Walrus and VICON are fixed references in data-efficiency panels.Source: G Field visualizations · p. 22

H Per-frame rollout error curves

  • Per-frame curves plot VRMSE, relative-L2, and RMAE against absolute prediction frame at the largest training budget.Values are medians over test trajectories of channel-averaged error; averaging t = 10–20 reproduces scalar table entries within one to two percent, and the vertical axis is logarithmic. CE-RM models use different frame grids.Source: H Per-frame rollout error curves · p. 22

Related papers

Loading 2610.31935v1…