Source-linked AI summary
$β$-Variational autoencoders and transformers for reduced-order modelling of fluid flows
Alberto Solera-Rico, Carlos Sanmiguel Vila, M. A. Gómez, Yuning Wang, Abdulrahman Almashjary, Scott T. M. Dawson, Ricardo Vinuesa
TL;DR
Chaotic fluid flows involve nonlinear interactions that can limit linear reduced-order representations. This paper combines a β-VAE for compact latent representations with a transformer for temporal prediction, obtaining more efficient flow reconstruction and stronger prediction performance than the compared models.
Problem
Chaotic flows exhibit nonlinear interactions, while linear methods such as POD can lack performance and dominant modes may miss complex flow behavior.
Method
The framework combines a β-VAE that learns a compact latent space with a transformer that predicts its temporal dynamics, using reconstruction and KL-divergence losses.
Results
Using 20 β-VAE modes reconstructed 89.8% of train-data energy versus 64.4% for 20 POD modes, while the combined ROM produced predictions with E of 78.1% and 64.6%.
Takeaways & Limitations
The β-VAE learns compact near-orthogonal nonlinear modes with shedding frequencies equivalent to energetic POD modes, and the transformer provides more robust temporal prediction than LSTM or KNF.
Takeaways & Limitations
The POD comparison is computed only from snapshots covering the training interval [0, ttrain].
Abstract
from arXiv · showhide
Variational autoencoder (VAE) architectures have the potential to develop reduced-order models (ROMs) for chaotic fluid flows. We propose a method for learning compact and near-orthogonal ROMs using a combination of a $β$-VAE and a transformer, tested on numerical data from a two-dimensional viscous flow in both periodic and chaotic regimes. The $β$-VAE is trained to learn a compact latent representation of the flow velocity, and the transformer is trained to predict the temporal dynamics in latent space. Using the $β$-VAE to learn disentangled representations in latent-space, we obtain a more interpretable flow model with features that resemble those observed in the proper orthogonal decomposition, but with a more efficient representation. Using Poincaré maps, the results show that our method can capture the underlying dynamics of the flow outperforming other prediction models. The proposed method has potential applications in other fields such as weather forecasting, structural dynamics or biomedical engineering.
Analysis of the latent-spaces
The study evaluates β-VAE latent spaces against POD for periodic and chaotic two-dimensional viscous flows, focusing on reconstruction energy, mode structure, disentanglement, and generalisation. β-VAE representations reproduce key periodic-flow features and provide compact nonlinear modes for the more complex chaotic case.
- Flow cases: The experiments use periodic Re = 40, α = 90° and chaotic Re = 100, α = 80° two-dimensional viscous flows, with 1,000 and 150,000 snapshots respectively.The data domain is 96c × 28c on a 300 × 98 grid, with different snapshot intervals for the two cases.
- Periodic benchmark: The periodic flow is represented by two POD modes capturing E = 98.4% of fluctuating kinetic energy, providing a benchmark for β-VAE physical soundness.The two dominant POD modes represent a shedding wake.
- Periodic benchmark: β-VAE modes reproduce the periodic shedding pattern and characteristic frequency, with E = 97.5% and cross-correlation R12 = 0.0015.The comparison indicates that the latent-space contains physically meaningful flow phenomena.
- Latent-space size and β: β = 0.05 is selected to balance near-orthogonality, reconstruction error, disentanglement, and generalisation; lower β values increase train-test imbalance, while β = 1 performs poorly.The β-VAE losses compare reconstruction accuracy with KL regularisation during training.
- Latent-space size and β: With d = 20, β-VAE reconstruction reaches E = 89.8% on training data and E = 80.1% on test data, whereas d = 10 reaches E = 82.0% and E = 61.6%.The smaller bottleneck loses reconstruction and generalisation capability, while larger latent spaces can contain modes with negligible values.
- Chaotic latent space: For the chaotic case, β-VAE modes are nonlinear combinations of relevant flow features and separate phenomena with characteristic frequencies more clearly than POD.Frequencies associated with energetic POD modes also appear in the β-VAE dynamics.
Latent-space predictor models
The study compares transformer, LSTM, and KNF predictors for β-VAE latent dynamics, evaluating both prediction errors and whether reconstructed fields preserve chaotic-flow behavior.
- Predictor models: Four models predict successive latent-space vectors: self-attention and easy-attention transformers, KNF, and LSTM.Each model is trained on previous latent vectors and then used recursively for future steps.
- Prediction behavior: All predictors diverge from the original chaotic trajectory after several time steps and show similar quantitative performance initially.This divergence is expected for a chaotic dynamical system.
- Prediction behavior: The easy-attention transformer has among the slowest error growth rates, especially when snapshots are sampled at Δt = 0.2tc.The authors suggest this may reflect improved representation of multiple time scales.
- Long-term dynamics: Poincaré maps show that KNF fails to reproduce correlation variability, whereas LSTM and especially transformer models accurately capture the chaotic latent dynamics.The KNF behavior is associated with convergence toward harmonic dynamics during prediction.
- Field reconstruction: The β-VAE decoder converts predicted latent vectors back into velocity fields, enabling comparison of actual, reconstructed, and predicted flow fields.The complete ROM is assessed through reconstructed energy and POD modes over the prediction horizon.
- Field reconstruction: Predicted fields retain POD modes similar to the original data, reinforcing that the combined β-VAE–transformer ROM reproduces the flow dynamics.The energy analysis also shows better reconstruction for latent-space size d = 20 than for d = 10.
Discussion
The framework combines β-VAE latent representations with transformer-based temporal prediction for periodic and chaotic two-dimensional viscous flows. It achieves more efficient reconstruction than POD and preserves relevant flow patterns while predicting unseen chaotic-flow data.
- Discussion: The framework combines β-VAE architectures for robust nonlinear latent spaces with transformers for temporal prediction in periodic and chaotic flow regimes.The study uses two-dimensional viscous flow around two collinear flat plates.
- Discussion: For the periodic case, β-VAE modes reproduce the vortex-shedding features and equivalent number of energetic modes identified by POD.This comparison is used to assess the physical soundness of the learned representations.
- Discussion: 89.8% energy is reconstructed with 20 β-VAE modes on training data, compared with 64.4% for POD using 20 modes; POD requires 69 modes to match the β-VAE result.The β-VAE latent space is described as compact and near-orthogonal.
- Discussion: The β-VAE modes exhibit shedding frequencies equivalent to those of the most energetic POD modes, suggesting they represent important nonlinear flow features.This links the learned latent dynamics to established POD modal behavior.
- Discussion: On previously unseen data, the combined ROM reconstructs 78.1% energy at t + tc and 64.6% at t + 10tc while capturing the original flow patterns.LSTM and easy-attention transformer models outperform KNF, with the transformer providing more robust prediction across snapshot intervals.
Dataset description and pre-processing
The study uses simulated incompressible two-dimensional viscous flow over two collinear flat plates, with velocity data processed for β-VAE learning. The β-VAE encodes flow fields into a latent representation and decodes latent inputs into spatial modes.
- The source data are incompressible, two-dimensional viscous flow fields around two collinear flat plates generated by numerical simulation.
- The dataset is downsampled to 300 × 98 grid points, with streamwise and crosswise velocity fluctuations stored as separate channels and standardised component-wise.
- The VAE combines reconstruction loss with KL divergence to reconstruct inputs while keeping latent distributions close to a standard normal distribution.
- β controls the trade-off between reconstruction accuracy and latent-space disentanglement, with higher β promoting disentanglement but potentially reducing accuracy.
- The encoder outputs latent means and variances, while the decoder reconstructs spatial modes from one-hot latent input vectors.
- For the chaotic case, varying one latent input changes the shedding-wake pattern, and 20 β-VAE modes reconstruct E = 89.8% versus POD’s E = 64.4%.
Proper orthogonal decomposition
POD provides the classical reference for decomposing the velocity fluctuations into spatial basis functions and time-dependent coefficients. The snapshot method forms a correlation matrix whose eigenvectors yield orthonormal temporal modes.
- POD decomposes mean-subtracted streamwise and crosswise velocities into fluctuating components before representing them with spatial basis functions.
- Each fluctuation field is approximated as a sum of spatial basis functions weighted by time-dependent coefficients.
- The snapshot method treats each image as an Np-dimensional vector and arranges the data into an Nt × Np snapshot matrix.
- The two-point correlation matrix G = u uT + v vT is eigendecomposed to obtain eigenvalues and temporal-mode eigenvectors.
- The POD comparison uses snapshots from the training interval [0, ttrain].
Time-series prediction models
The temporal model predicts the β-VAE latent dynamics with transformer architectures and compares them against LSTM and Koopman with Non-linear Forcing models. Transformer inputs contain delayed latent sequences and temporal-spatial embeddings.
- The β-VAE encoder produces the latent time series, with 90% of snapshots used for training and the final 10% reserved for testing.
- The transformer predicts the next latent vector from the previous 64 time steps using mean-squared-error training.
- Time-space embeddings provide temporal and spatial information so the transformer can distinguish latent vectors from different time steps.
- Transformer encoder blocks combine multi-head attention with feed-forward networks, using self-attention and easy attention mechanisms.
- The temporal predictions are compared with four-layer LSTMs and Koopman with Non-linear Forcing models.
Training setup
The β-VAE and transformer are trained separately with Adam-based optimization and fixed data splits. The transformer architecture uses a 64-step delay and multi-head attention components.
- The β-VAE and transformer are implemented in Torch 2.0 and trained on NVIDIA GeForce RTX 4090 and A100 GPUs, respectively.
- β-VAE training uses Adam for 1,000 epochs with batch size 256 and a one-cycle learning-rate schedule from 1×10^-4 to 5×10^-6.
- The architecture summary identifies T as the time-delay dimension and dmodel as the embedding size, alongside feed-forward and LSTM hidden-state dimensions.
- The encoder and decoder each contain 1.06×10^6 trainable parameters, and the first 90% of snapshots are used for training.
- Transformer training uses Adam, batch size 256, an initial learning rate of 1 × 10^-3, exponential decay, and up to 1,000 epochs.