PhyMo: A Physical-Field Modality for Multimodal AI4Physics

• Generic text or token representations can omit the mathematical constraints governing how physical quantities interact across space and time.Wind speed and solar irradiance are associated with transport, advection, or diffusion processes describable by PDEs.Source: §1 · pp. 1–2
• The physical-field modality organizes measurements through PDEs that characterize their spatial and temporal dynamics, with PDE residual losses adding physics-based constraints during training.Source: §1 · p. 2
• PhyMo pretrains a physical-field encoder, aligns its embeddings with visual representations, and uses fused representations for downstream prediction heads.Source: §1 · p. 2
• PhyMo’s experiments report consistently enhanced multimodal representation quality and downstream forecasting over conventional numerical or text representations.Source: §1 · p. 2

• Weather and climate forecasting approaches predominantly use a single structured physical-data representation rather than explicitly exploiting complementary modalities.Source: §2 · p. 3
• Multimodal scientific systems align varied observations, including images, spectra, scalar measurements, and text, for downstream tasks.Examples include astronomy, materials science, weather analysis, and neutrino classification.Source: §2 · p. 3
• These approaches generally represent physical information as arrays or symbolic or textual descriptions, not as a dedicated modality constrained by governing physical laws.Source: §2 · p. 3

3.1 Motivation

• The proposed physical-field modality addresses representations that treat measurements as generic tokens without preserving constraints on their spatiotemporal evolution.Its design organizes heterogeneous physical measurements through PDE-associated structures.Source: §3.1 · pp. 3–4

3.2 The Framework of PhyMo

• PhyMo uses three stages: PDE-supervised physical-field pretraining, alignment with visual representations, and downstream prediction from aligned embeddings.The framework maps heterogeneous measurements into a physical-field modality rather than treating them as generic numerical or textual inputs.Source: §3.1 · p. 3, §3.2 · p. 4, Figure 1 · p. 4
• Physical fields combine PDE-operator tokens with measured values, and reconstruction is constrained by the governing PDEs.The encoder is pretrained to reconstruct the fields under PDE residual supervision.Source: §3.2 · pp. 4–5, §3.1 · p. 4
• An adapter projects image and physical embeddings into a shared space; contrastive and regression objectives align pairs, whose concatenation feeds horizon-specific MLP heads.The prediction heads are trained with mean squared error, with k-fold validation used to assess stability and select the prediction model.Source: §3.2 · p. 5

3.3 Theoretical Analysis

• Theorem 3.1 states that combining image and physical representations cannot raise Bayes-optimal risk and may lower it when the modalities contain complementary predictive information.The analysis frames this as an information-theoretic motivation for using both modalities.Source: §3.3 · p. 6
• PDE-structured inputs can distinguish systems with identical instantaneous measurements but different local dynamics, unlike state-only numerical representations.The paper uses this distinction to motivate organizing physical information by its governing structure.Source: §3.3 · p. 6

4.1 Experiment Setup

• The evaluation uses five datasets spanning power, radiance, weather, and autonomous-driving speed prediction, with regression as its primary downstream evaluation paradigm.The paper describes PhyMo as a general representation-learning framework and reports five experiments over varying random seeds for mean R2 and sample-standard-deviation evaluation.Source: §4.1 · p. 7, §4 · p. 7

4.2 Performance Comparison

• PhyMo achieves SOTA performance on SKIPP’D, Folsom, NREL, and MeteoNet, and ranks second close to the best-performing method on Boreas.The reported comparisons span solar forecasting, meteorological prediction, and autonomous-driving scenarios.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Ablation Analysis

• R2 is 84.80 with both modalities, versus 82.90 without physical fields and 83.45 without images; early fusion scores 84.80 versus 83.94 for late fusion.The authors interpret the modality ablations as complementary information and the fusion comparison as favoring representation-level interaction.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 Hyperparameter Sensitivity Analysis

• Performance remains within a narrow range across cross-modal embedding sizes and image spans, while reconstruction performance improves through K3 and is comparable at K4 and K6.The authors attribute diminishing returns at longer horizons partly to SKIPP’D PV forecasting’s greater reliance on short-period histories; compact representations, recent visual context, and moderately diverse horizons offer the preferred balance.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 Prediction Visualization

• On SKIPP’D, predictions track sunny-day PV curves closely but show larger errors under cloudy conditions with abrupt, short-lived fluctuations.The visualization reports that the model captures the overall generation envelope and major ramps under cloudy conditions, while errors increase with ground-truth volatility.Source: §4.5 · p. 9, Figure 2 · p. 9

5 Conclusion

• PhyMo organizes heterogeneous physical quantities through PDE-associated operators and uses a three-stage procedure of encoder pretraining, cross-modal alignment, and downstream prediction.The physical-field encoder is pretrained through field reconstruction with PDE residual supervision.Source: §5 · pp. 9–10
• Across five datasets, PhyMo achieves SOTA performance compared with the strongest baseline on each dataset.The conclusion describes physical fields as a principled modality for multimodal AI4Physics and presents PhyMo as a framework scalable to additional quantities, governing equations, and scientific domains.Source: §5 · pp. 9–10

A Detailed Theoretical Analysis

• The analysis uses squared prediction loss, consistent with the MSE objective used for downstream prediction.Source: A Detailed Theoretical Analysis · p. 13

A.1 Preliminaries

• It denotes the target by Y and the learned image and physical-field representations by Zimg and Zphy, respectively.For arbitrary Z, it introduces Bayes-optimal prediction risk under squared loss.Source: §A.1 · p. 13
• The stated Bayes-optimal predictor identity provides the basis for Theorems 3.1 and 3.3.Source: §A.1 · p. 13

A.2 Proof of Multimodal Bayes-Risk Dominance

• For the image representation, the law of total variance yields an additional non-negative conditional-variance term when conditioning on both representations.Source: §A.2 · p. 13
• The proof derives a conditional-variance inequality given Zimg and then takes expectation over Zimg.Source: §A.2 · p. 13
• It repeats the argument by conditioning first on Zphy, then combines the resulting inequalities.Source: §A.2 · p. 13

A.3.1 Physical Admissible Set

• The joint physically admissible set S consists of underlying physical fields satisfying their governing PDEs.The proof assumes the equations are correctly specified and that S is nonempty, closed, and convex.Source: §A.3.1 · p. 14
• Physics-constrained reconstruction is defined by projecting an unconstrained reconstruction onto S.The Euclidean projection is uniquely defined under the stated properties of S.Source: §A.3.1 · p. 14

A.3.2 Projection-Based Reconstruction Bound

• The proof uses the projection bound and non-negativity of an error term to establish a pointwise reconstruction-risk inequality that remains valid in expectation.The supplied proof passages state that taking expectation preserves the pointwise inequality.Source: §A.3.2 · pp. 14–15

A.4 Proof of the Advantage of PDE-Structured Physical Representations

• The comparison contrasts physical quantities observed at a state, X, with a representation that also uses the collection of PDE operators, D(X).The proof sets up prediction risks for numerical-only and PDE-structured physical information.Source: §A.4 · p. 15
• The proof expresses the risk difference through a conditional-variance term and notes that this term is non-negative.The strict case occurs when that term is positive with positive probability.Source: §A.4 · p. 15
• PDE-structured physical information provides a strict risk advantage when the differential structure contains predictive information unavailable from the instantaneous numerical state alone.The stated condition is positive conditional variance of E[Y | X, D(X)] given X.Source: §A.4 · p. 15

B.1 Dataset Details

• The five-dataset evaluation combines solar and meteorological forecasting settings with Boreas, a multi-season autonomous-driving dataset.Boreas contains synchronized camera, lidar, radar, and navigation streams, with repeated routes under varied seasonal and weather conditions.Source: §B.1 · pp. 16–17

B.2 Baseline Details

• The baseline suite ranges from classical structured-data models and temporal forecasters to scientific and domain-specific multimodal models.Examples include LASSO, Random Forest, TabM, LSTM, TSMixer, xLSTM, BiMamba, AstroCLIP, Maven, and AION-1.Source: §B.2 · pp. 17–18

B.3 Metrics

• RMSE, MAE, and R2 assess forecasting from complementary perspectives: large deviations, average error magnitude, and explained target variability.The section reports all three metrics for a more comprehensive evaluation of downstream forecasting performance.Source: §B.3 · p. 19

C Limitations

• Downstream evaluation primarily covers regression forecasting; the authors plan to extend assessment to classification, reconstruction, and physical-state estimation.Available physical variables and their spatiotemporal resolutions also vary by dataset.Source: C Limitations · p. 19
• The three-stage training procedure and PDE residual computation add computational overhead compared with conventional data-driven models.Proposed efficiency directions include lightweight encoders, shared-stage optimization, and more efficient PDE-operator computation.Source: C Limitations · p. 20

AI Use Statement

• The authors report using generative AI to assist with translation, code implementation, and experimental feedback, and state that they reviewed the assisted work.They report that two authors verified and tested LLM-generated code.Source: AI Use Statement · p. 20