Transformers for Modeling Physical Systems

Nicholas Geneva; Nicholas Zabaras

Source-linked AI summary

Transformers for Modeling Physical Systems

Nicholas Geneva, Nicholas Zabaras

arXiv:2010.03957v6Oct 4, 2020cs.LGphysics.comp-ph

TL;DR

The paper addresses the limited use of transformers outside NLP for modeling physical dynamical systems. It uses Koopman-based embeddings to represent physical states for transformer prediction, and reports accurate predictions across varied systems with performance exceeding alternative scientific machine-learning models.

Problem
Transformer applicability outside natural language processing has been limited, motivating their evaluation for physical dynamical systems.
Method
The model combines a GPT-2-style transformer decoder with Koopman observable embeddings that represent dynamical-system states as vectors.
Results
The proposed transformer surrogate outperforms alternative models widely used in recent scientific machine learning literature.
Takeaways & Limitations
Self-attention can use many past time steps directly, supporting prediction of physical systems with longer or multi-time-scale dependencies.
Takeaways & Limitations
Future work is needed on unsupervised pre-training and on understanding what attention mechanisms imply for physical systems.

Abstract

from arXiv · show

Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicted by a transformer. The proposed model is able to accurately predict various dynamical systems and outperform classical methods that are commonly used in the scientific machine learning literature.

1. Introduction

This paper extends transformer models from natural language processing to physical dynamical systems, targeting prediction that can replace expensive numerical solvers. It combines self-attention with Koopman-inspired embeddings and evaluates the approach on diverse high-dimensional physical problems.

Motivation and contribution: Transformer models are proposed for predicting physical dynamical systems as surrogate models that can replace expensive numerical solvers.The work frames this as using transformers to learn the language of physics.
Motivation and contribution: Existing surrogate-modeling approaches include Gaussian processes, polynomial chaos expansions, reduced-order models, reservoir computing, and deep neural networks.Surrogate models approximate physical phenomena inexpensively and can represent distributions of initial or boundary conditions.
Motivation and contribution: Current neural models often rely on recent time steps or fixed temporal windows, limiting their generalizable time-aware prediction of multi-time-scale phenomena.The motivating examples include turbulent fluid flow, multi-scale materials modeling, and molecular systems.
Methodological contributions: The paper introduces self-attention transformers for physical dynamics and Koopman dynamics embeddings that connect high-dimensional physical states with NLP-style embeddings.The contributions also discuss relations between self-attention and traditional numerical time integration.
Evaluation scope: The proposed model is demonstrated on high-dimensional partial differential equations involving chaotic dynamics, fluid flows, and reaction-diffusion systems.The examples span chaotic, periodic fluid, and three-dimensional reaction-diffusion dynamics.

2. Methods

The method combines a Koopman-based embedding of discretized physical dynamics with a transformer decoder that predicts the resulting time series. The embedding is trained and then frozen before transformer training, while self-attention provides a learned temporal integration mechanism.

Problem: The problem is formulated as predicting discretized physical states over time for systems described by ordinary or partial differential equations.The state is sampled at spatial points and time steps, producing a time-series representation.
Transformer Decoder: The transformer receives an embedded dynamical system Ξ=(ξ0,ξ1,…,ξT), with each embedded state ξi∈R^e, plus sinusoidal positional encodings.The model uses a GPT-2-style decoder with masked attention for sequential prediction.
Transformer Decoder: Training uses an autoregressive Gaussian likelihood in embedded space, equivalent to an L2 loss rather than a token softmax, because physical states are not finite dictionaries of tokens.The lower-dimensional embedded representation can reduce transformer training costs for high-dimensional physical states.
Embedding Model: The embedding model learns mappings F:R^(n×d)→R^e and G:R^e→R^(n×d) between discretized physical states and one-dimensional vectors.This provides a physics-specific alternative to NLP tokenization and word embeddings.
Embedding Model: Approximate Koopman dynamics define the embedding observables, using a learnable banded Koopman operator optimized with an encoder-decoder model.The banded structure reduces parameters and memory, encourages meaningful dynamical modes, and the learned operator is discarded after embedding training.
Training and Dynamics: The embedding loss combines reconstruction, Koopman dynamics, and parameter-decay terms, while self-attention can represent explicit Adams integration schemes and more complex temporal dependencies.The embedding model is trained first, frozen, and then used to embed data for transformer training; predicted embeddings are decoded back into physical states during testing.

3. Experiments and Results

The experiments evaluate transformer surrogates on chaotic Lorenz dynamics, 2D fluid flow, and 3D reaction-diffusion systems. Across these settings, Koopman embeddings and self-attention support accurate long-horizon predictions, while data and embedding quality constrain performance in the most complex case.

Experiments: The study evaluates the proposed transformer on Lorenz dynamics, Navier-Stokes flow around a cylinder, and a 3D Gray-Scott reaction-diffusion system.The Lorenz system provides a chaotic benchmark, fluid flow tests predictions across Reynolds numbers, and Gray-Scott dynamics test scalability in three dimensions.
3.1. Chaotic Dynamics: 320-step Lorenz predictions remain extremely accurate beyond the transformer’s context length, and 25k-step trajectories qualitatively preserve the numerical solution’s structure.The transformer receives only the initial state for the plotted test cases, while the long trajectory comparison uses ∆t = 0.01.
3.1. Chaotic Dynamics: After extrapolating beyond the trained context range, the transformer achieves the best Lorenz accuracy at later times, although deep Koopman performs best within the training range.Relative test MSE is reported over intervals segmented by the transformer’s context length.
3.1. Chaotic Dynamics: With 1% training noise, the transformer is the best-performing model, while with 5% noise it remains competitive particularly at later time-steps.Each clean-data model was retrained using observations perturbed by 1% and 5% noise scaled by state-variable magnitude.
3.2. 2D Fluid Dynamics: For 2D fluid flow, Koopman embeddings enable the transformer to match instantaneous states more accurately and provide more reliable predictions than ConvLSTM with fewer learnable parameters.The results also indicate that both the embedding method and self-attention contribute to the model’s success.
3.3. 3D Reaction-Diffusion Dynamics: For 3D reaction-diffusion dynamics, the transformer produces acceptable predictions with structures similar to the numerical solver, but added depth yields no considerable test-error improvement.The authors attribute the remaining limitation primarily to training-data quantity and embedding inaccuracies from dimensionality reduction.

4. Conclusion

The paper demonstrates that transformers can model physical dynamics, with self-attention capturing longer temporal dependencies and Koopman embeddings providing dynamical context. The authors identify embedding design and future investigation of pre-training and attention interpretation as open directions.

4. Conclusion: Transformers can learn longer and more complex temporal dependencies than alternative machine-learning methods for physical phenomena.The paper highlights potential benefits for systems evolving across multiple time scales or containing multiple phases.
4. Conclusion: The approach is demonstrated on physical systems including turbulent fluid flow and chemical reactions, where multi-scale or multiphase dynamics are relevant.
4. Conclusion: Koopman dynamics provide physics-inspired embeddings that address the challenge of representing high-dimensional physical states for transformer prediction.The embeddings impose dynamical context on the learned representation.
4. Conclusion: The proposed transformer surrogate outperforms alternative models widely used in scientific machine learning literature.
4. Conclusion: Unsupervised pre-training and improved understanding of attention mechanisms in physical systems remain subjects for future work.

Appendix A. Lorenz Supplementary Results

The supplementary Lorenz results show that the transformer remains accurate longer than alternative learned models and compares favorably with Euler integration. Its predictions also preserve the qualitative structure of the numerical solution over long horizons.

Appendix A. Lorenz Supplementary Results: The transformer remains consistently accurate over the plotted Lorenz prediction horizon, while alternative machine-learning methods quickly deviate after several time-steps.The comparison uses three test cases and predictions over 128 time-steps.
Appendix A. Lorenz Supplementary Results: The transformer clearly outperforms the less accurate Euler time-integration method when compared against Runge-Kutta solutions for 256 time-steps.
Appendix A. Lorenz Supplementary Results: The transformer and numerical Lorenz solutions have the same structure over 25k time-steps, providing qualitative evidence that the prediction maintains physical dynamics.
Appendix A. Lorenz Supplementary Results: The supplementary experiments include clean and contaminated training data with two noise levels used for the reported results.

Appendix B. Cylinder Supplementary Results

The cylinder-flow supplementary results compare transformer embeddings and recurrent alternatives on velocity, pressure, and learned dynamical modes. Transformer-Koopman predictions are qualitatively good and outperform the alternatives in the reported comparison.

Appendix B. Cylinder Supplementary Results: Transformer-Koopman predictions outperform alternative models in qualitative velocity and pressure field comparisons.The alternatives include transformer models with auto-encoder and PCA embeddings, ConvLSTM, and LSTM-KM.
Appendix B. Cylinder Supplementary Results: All tested models have similar training and validation errors, indicating minimal overfitting in the cylinder-flow experiment.
Appendix B. Cylinder Supplementary Results: The learned Koopman embedding projects cylinder-flow dynamics onto dominant eigenvectors, revealing an initial transition region before periodic vortex shedding.

Appendix C. Gray-Scott Supplementary Results

The Gray-Scott supplementary results show reliable early prediction of reaction-diffusion fields, while accuracy degrades as interacting reaction fronts create complex structures. The figures also compare embedding choices and visualize the predicted species fields.

Appendix C. Gray-Scott Supplementary Results: The transformer reliably predicts earlier Gray-Scott time-steps with reasonable accuracy before complex reaction structures develop.The visualizations include volume plots for both species and contour plots for three test cases.
Appendix C. Gray-Scott Supplementary Results: The supplementary comparison evaluates transformer predictions using Koopman, auto-encoder, and PCA embeddings against ConvLSTM and LSTM-KM models.The velocity-magnitude comparison is shown for a test case at Re = 633.
Appendix C. Gray-Scott Supplementary Results: As reaction fronts interact, the transformer degrades in accuracy but maintains notable consistency with the true solution.
Appendix C. Gray-Scott Supplementary Results: Pressure predictions are compared across transformer embedding methods, ConvLSTM, and LSTM-KM in the Gray-Scott supplementary visualizations.
Appendix C. Gray-Scott Supplementary Results: Training and validation MSE are plotted for the transformer embedding variants and recurrent baselines.
Appendix C. Gray-Scott Supplementary Results: Additional Gray-Scott figures visualize test-case volumes, isosurfaces spanning u, v = [0.3, 0.5], and x −y contours at z = 16.