Towards a Physics Foundation Model

Florian Wiesner; Zoë J. Gray; Matthias Wessling; Stephen Baek

Source-linked AI summary

Towards a Physics Foundation Model

Florian Wiesner, Zoë J. Gray, Matthias Wessling, Stephen Baek

arXiv:2509.13805v4Sep 17, 2025cs.LGcs.AIstat.ML

TL;DR

Existing physics-aware models are specialized and often require retraining for new systems, motivating a general Physics Foundation Model. GPhyT learns dynamics from contextual state sequences using a Transformer differentiator, numerical integration, and diverse simulation data. It outperforms multi-physics baselines on known tasks and produces physically plausible predictions for entirely new physics, while long-range prediction still requires major accuracy and stability improvements.

Problem
Existing physics models are specialized to single systems and struggle with new governing equations or boundary conditions without retraining.
Method
GPhyT combines a Transformer-based neural differentiator with numerical integration and trains on a diverse 1.8 TB simulation corpus to infer dynamics from contextual state sequences.
Results
GPhyT outperforms other multi-physics architectures on known physics and produces physically plausible predictions for entirely new physical systems.
Takeaways & Limitations
A single pretrained model can adapt through context alone to diverse physical systems, including unseen boundary conditions and novel phenomena.
Takeaways & Limitations
Long-range prediction will require orders-of-magnitude improvements in both accuracy and stability.

Abstract

from arXiv · show

Foundation models have revolutionized natural language processing through a ``train once, deploy anywhere'' paradigm, where a single pre-trained model adapts to countless downstream tasks without retraining. Access to a Physics Foundation Model (PFM) would be transformative - democratizing access to high-fidelity simulations, accelerating scientific discovery, and eliminating the need for specialized solver development. Yet current physics-aware machine learning approaches remain fundamentally limited to single, narrow domains and require retraining for each new system. We present the General Physics Transformer (GPhyT), trained on 1.8 TB of diverse simulation data, that demonstrates foundation model capabilities are achievable for physics. Our key insight is that transformers can learn to infer governing dynamics from context, enabling a single model to simulate fluid-solid interactions, shock waves, thermal convection, and multi-phase dynamics without being told the underlying equations. GPhyT achieves three critical breakthroughs: (1) superior performance across multiple physics domains, outperforming specialized architectures by more than 7x, (2) plausible zero-shot generalization to entirely unseen physical systems through in-context learning, and (3) more stable long-term predictions through long-horizon rollouts. By establishing that a single model can learn generalizable physical principles from data alone, this work opens the path toward a universal PFM that could transform computational science and engineering.

1. Introduction

Physics foundation models aim to support diverse physical simulations without task-specific retraining, but existing approaches remain specialized. GPhyT addresses this gap by inferring dynamics from context and testing broad, stable, and zero-shot prediction capabilities.

Current physics-aware models generally solve narrowly defined systems and require new training to handle different physics or boundary conditions.
A general Physics Foundation Model must handle diverse phenomena and infer governing dynamics from input data rather than require complete system descriptions.
GPhyT is trained on a diverse 1.8 TB simulation corpus to emulate in-context learning from short sequences of prior physical states.
The study evaluates whether one transformer can model disparate systems, remain stable during extended rollouts, and generalize zero-shot to unseen conditions.
GPhyT outperforms multi-physics architectures on seen tasks and produces physically plausible predictions for phenomena absent from its training data.

2. Related work

Prior physics surrogate methods improve simulation efficiency but remain largely specialized to individual systems. Foundation-model research is beginning to pursue broader scientific generalization through quantitative-data models, meta-learning, transfer learning, and multi-task training.

PINNs incorporate governing PDEs as soft loss constraints, while Neural Operators learn mappings from PDE parameters to solution spaces.
PINNs and Neural Operators typically struggle with new equations, boundary conditions, and complex multi-physics phenomena without transfer learning or retraining.
Scientific foundation models include language models fine-tuned on scientific corpora and models operating directly on quantitative scientific data.
Physics researchers have explored meta-learning, transfer learning, and multi-task models to adapt pretrained systems across PDEs and physical systems.
Transformers use self-attention to capture long-range dependencies and have been adapted from language to images and spatiotemporal video data.

3. General Physics Transformer

GPhyT combines a spatiotemporal Transformer differentiator with numerical integration to model diverse physical dynamics. It is trained on varied simulation data and augmentation schemes designed to promote in-context inference across scales and sampling rates.

3.1. Architecture: GPhyT couples a Transformer-based neural differentiator that learns temporal dynamics with a numerical integrator that extrapolates future physical states.
3.1. Architecture: The differentiator consumes multiple state snapshots, tokenizes spatiotemporal patches, applies attention, and reconstructs the predicted time derivative.
3.1. Architecture: Spatial and temporal derivatives are concatenated with the original fields to provide explicit local information, especially for sharp gradients.
3.1. Architecture: Forward Euler integrates the learned time derivative to produce the next system state, while more accurate integrators remain possible.
3.2. Datasets: The training corpus contains over 2.4 million snapshots totaling 1.8 TB across seven simulation datasets spanning fluid, thermal, obstacle, and solid-interaction settings.
3.2. Datasets: These design choices explicitly encourage in-context learning instead of memorization of one fixed physical system.
3.2. Datasets: Variable time increments and per-dataset normalization force the model to infer temporal scales and absolute magnitudes from contextual dynamics.

4. Results

GPhyT achieves strong single-step accuracy across multiphysics datasets, remains competitive during long-horizon rollouts, and generalizes to novel boundary conditions and entirely new physical systems without fine-tuning.

4.1. Multi-physics learning: 7× higher NMSE for DPOT than GPhyT makes GPhyT the strongest overall single-step model in the multiphysics comparison.All models were trained identically to predict the subsequent frame, and GPhyT achieves the lowest error in all but one dataset.
4.1. Multi-physics learning: GPhyT achieves the lowest NMSE on every evaluated physics dataset except Euler, while model errors increase on more challenging systems.The reported error bars also show substantial variability across prediction accuracy.
4.2. Long-range Prediction: GPhyT performs exceptionally well in autoregressive rollouts through 24 steps, although Poseidon slightly surpasses it in later stages.All models accumulate error during rollouts, and GPhyT and Poseidon are competitive across most datasets except shear flow.
4.2. Long-range Prediction: GPhyT preserves global dynamics and physical plausibility better than the compared models in Euler shockwave predictions, but no model reaches numerical-solver fidelity.The paper states that practical engineering applications require orders-of-magnitude improvements in accuracy and stability.
4.3. In-context learning: GPhyT is the only model below NMSE 1 on novel physical systems and maintains stable long-horizon predictions without fine-tuning.It significantly outperforms baselines on new boundary conditions and achieves lower errors than all baselines on completely new physics, despite higher initial errors.

5. Conclusion

GPhyT demonstrates that a simple transformer-based model can learn dynamics across diverse physical systems and adapt to novel phenomena from input context alone. Its long-horizon physical consistency supports progress toward Physics Foundation Models, though the approach remains less precise than numerical solvers.

GPhyT learns and predicts diverse physical dynamics without explicit physics-specific features.
GPhyT outperforms state-of-the-art multi-physics models on known physics tasks.
GPhyT infers new boundary conditions and novel physical phenomena from input prompts alone.
GPhyT maintains physical consistency during long-horizon rollouts, although it does not yet match numerical solvers.
The foundation-model paradigm is achievable for physics through a pre-trained model adapting to novel tasks through context alone.

Impact Statement

The paper reports no societal consequences requiring specific emphasis.

The authors identify no potential societal consequences that require specific highlighting.

A.1. Ablation: What Makes a Good Physics Foundation Model

The ablations identify derivative-based prediction, explicit derivative features, model scale, and input context length as important design factors. Larger models and longer contexts improve accuracy, while four input timesteps offer a favorable accuracy-efficiency trade-off.

Predicting time derivatives instead of next states yields substantially lower errors across rollout horizons.This decouples learned dynamics from numerical time integration.
Removing explicit spatial and temporal derivatives degrades extended-rollout accuracy by nearly an order of magnitude.The derivatives provide local gradient information for sharp gradients and discontinuities.
Increasing model capacity from 9M to 385M parameters produces consistent accuracy improvements.
Additional input timesteps improve accuracy with diminishing returns, following a log-linear trend.The largest gain occurs when increasing from one to two timesteps.
Ninput = 4 provides a favorable accuracy-efficiency trade-off while requiring fewer input frames than prior work.Longer prompts improve accuracy but increase self-attention cost quadratically.

A.2. Limitations

GPhyT remains constrained by two-dimensional training data, limited physics coverage, fixed resolution, and lower long-term precision than numerical solvers. These boundaries limit its current scope for comprehensive engineering use.

The current model is restricted to 2D systems because of data scarcity and computational limitations.The architecture is described as directly extensible to 3D systems.
GPhyT falls considerably short of numerical solvers in long-term prediction precision.Lower errors are identified as essential for practical engineering applications.
GPhyT is trained exclusively on fluid dynamics and heat transfer systems.Broader coverage would require domains such as mechanics, chemistry, molecular dynamics, and optics.
The model is trained on fixed 256×128 resolution images.Widespread adoption may require variable domain sizes and resolutions.

A.3. Long-horizon accuracy per dataset

Figure 6 reports median normalized mean square error across test datasets after 12 rollout steps, with losses grouped by dataset and overall loss.

12-step rollouts are evaluated using median normalized mean square error across the test datasets.The figure reports grouped losses for each dataset and the overall loss; error bars indicate the 25th and 75th percentile errors.

A.4.1. GENERAL PHYSICS TRANSFORMER

The section describes GPhyT’s training setup, shared physical-field representation, evaluation protocol, and the diverse flow systems included in its datasets. It also specifies the governing physics and boundary conditions underlying these simulations.

Evaluation: Figure 7 compares median normalized mean square error across test datasets over 24 rollout steps, with error bars spanning the 25th and 75th percentile errors.The figure groups losses by dataset and reports an overall loss.
Training setup: Baseline models use the same datasets, variable strides, four input time steps where required, and a total batch size of 256 for 1 million AdamW steps.Baseline training uses two A100 GPUs, while GPhyT models S, M, and L use four H100 or A100-80GB GPUs in parallel.
Data representation: GPhyT models 2D time series of physical snapshots, representing pressure, density, temperature, and two velocity components as input fields.Absent fields are zeroed, and training uses 256 x 128 spatial dimensions with train/validation/test splits of 0.8/0.1/0.1.
Simulation domains: The datasets cover incompressible shear flow, compressible shock-wave flow, Rayleigh–Bénard convection, turbulent radiative mixing, obstacle flows, and heated flow through channels.These systems involve Navier–Stokes, Euler, heat-transport, radiative-cooling, and surface-tension-related physics, depending on the dataset.
Simulation domains: The turbulent radiative-layer dataset represents hot dilute gas mixing with cold dense gas, where intermediate-temperature gas cools and transfers mass between phases.Its boundaries are periodic in x and zero-gradient in y, while the instability is seeded by simulation-varying small-scale noise.
Simulation domains: Heated-flow simulations combine compressible gas transport with heating or cooling structures, producing interactions between density-driven and forced convection.The configurations include heating rods or cooling rods with a heated wall.