PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

Oskar Bohn Lassen, Joao Paulo de Souza Boger, Simon Driscoll, Stephen I. Thomson, Sebastian Schemm, Filipe Rodrigues, Francisco C. Pereira

arXiv:2609.34627v1cs.LG

TL;DR

State-based prediction errors and selected physical constraints provide limited evidence about whether learned trajectories reproduce the underlying dynamics. The paper evaluates predictive relationships discovered in expert-defined representations, finding that weather models differ in preserving ERA5 structure in ways conventional forecast errors do not capture.

  • Problem

    Low state-prediction error and satisfaction of selected physical constraints do not establish that a model reproduces the underlying dynamics or learned relationships.

    Source: §2 · pp. 3–4

  • Method

    Experts define physical representations, and predictive relationships discovered in reference trajectories are frozen as tests of model-generated trajectories.

    Source: §1 · p. 2

  • Results

    For GraphCast, FengWu, and Pangu-Weather, process-level evaluation reveals differences in preserving ERA5 relationships that conventional forecast errors do not capture.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Accurate state prediction alone does not establish preservation of reference predictive structure.

    Source: §7 · p. 9

    The representation space need contain relevant sources of predictability, but need not characterize the complete system dynamics.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Machine-learning models for physical systems are currently evaluated primarily through errors between predicted and reference states and, increasingly, through tests of physical consistency. These metrics assess whether predictions are accurate and satisfy selected physical requirements, but provide limited insight into whether learned trajectories reproduce the underlying dynamics. Domain experts examine such relationships through physical representations that expose relevant processes, interactions, and responses, but these analyses are often separated from typical machine-learning evaluation. We introduce a practical framework for evaluating dynamical fidelity through predictive structure in physical representation spaces. Experts define the representations, while reference trajectories determine which relationships are predictive and retained as evaluation tests. We demonstrate the approach in atmospheric forecasting using ERA5 representations of planetary-wave activity and Northern Annular Mode evolution, and evaluate Pangu-Weather, GraphCast, and FengWu. The models exhibit distinct departures from reference predictive structure that are not reflected by conventional forecast errors. The framework thereby turns domain-expert representations into systematic tests of learned physical dynamics without prescribing the relationships in advance.

1 Introduction

State-based errors and selected physical-consistency tests provide limited evidence about whether learned trajectories reproduce system dynamics. The paper introduces a framework that uses expert-defined representations and reference-discovered relationships to evaluate dynamical fidelity, then demonstrates it on stratospheric circulation forecasts.

1 Introduction

  • State-based evaluation measures whether a model reaches the correct state, while process-oriented diagnostics examine whether reference processes and response relationships are preserved.Such diagnostics are commonly developed within individual domains and evaluated separately from standard machine-learning metrics.Source: §1 · p. 2
  • Reference trajectories determine which relationships are predictive and retained as tests, while experts define the physical vocabulary.The framework does not prescribe in advance which relationships models should satisfy.Source: §1 · p. 2
  • The framework formalizes state-space accuracy, physical consistency, and dynamical fidelity, and discovers process-level tests within expert-defined representations.Physical consistency does not by itself imply dynamical fidelity.Source: §1 · p. 2
  • Across GraphCast, FengWu, and Pangu-Weather, process-level evaluation reveals differences not captured by conventional forecast error.The tests examine whether models preserve ERA5-discovered conditional evolution in stratospheric wave–mean-flow dynamics and Northern Annular Mode evolution.Source: §1 · p. 2

2 From Physical Consistency to Dynamical Fidelity

State prediction and tests of physical consistency each provide partial evidence about learned dynamics: low prediction error does not identify learned relationships, and a limited set of physical constraints does not establish dynamical fidelity. Process-level diagnostics can instead test whether particular aspects of reference evolution are preserved.

2 From Physical Consistency to Dynamical Fidelity

  • Learned state-transition models generate autoregressive trajectories from reference initial states, while common training minimizes prediction error over observed transitions.The one-step notation also applies to models conditioned on multiple preceding states or predicting multiple future states jointly.Source: §2 · pp. 2–3
  • Low prediction error does not identify which relationships a model learned, and similarly performing models can rely on different relationships.Those relationships may behave differently outside the training distribution or over long rollouts, where errors can accumulate and trajectories diverge.Source: §2 · p. 3
  • A complete collection of physical relations can characterize dynamics when it is sufficiently rich to distinguish different vector fields.The paper contrasts that complete case with practical evaluation using only a subset of known and testable relations.Source: §2 · p. 3
  • A limited set of testable physical constraints restricts admissible dynamics but does not guarantee that the remaining dynamics are close to the true system.Only a sufficiently rich, complete characterization can distinguish dynamics; empirical tests also use finite-resolution approximations of physical relations.Source: §2 · p. 3
  • Even when a learned system satisfies evaluated physical relations, it may realize different transport, propagation, interaction, or response behaviour.Finite spatial and temporal resolution further coarsens tests of continuous physical relations.Source: §2 · pp. 3–4
  • Process diagnostics expose particular aspects of system evolution, and agreement in a diagnostic indicates agreement in the aspect it exposes.They may characterize flux, transport, propagation, forcing, or response, and can return a scalar, field, or space–time diagnostic.Source: §2 · p. 4

3 Evaluating dynamical fidelity through expert-defined representations and relationship discovery

The framework separates the definition of a physically meaningful representation space from the predictive relationships tested within it. Experts define the representations; reference trajectories identify relationships that are then frozen for evaluation.

3 Evaluating dynamical fidelity through expert-defined representations and relationship discovery

  • The framework makes process-level evaluation more systematic by separating expert definition of physical representations from discovery of the relationships evaluated within them.Reference trajectories determine which relationships are retained, and those relationships are frozen before model evaluation.Source: §3 · p. 4, §3.2 · p. 5

3.1 Expert-defined physical representations

  • Expert-defined representations are physically meaningful transformations chosen to expose relevant sources, transports, interactions, and responses.The representation space need only contain physically relevant sources of predictability, not characterize the complete system dynamics.Source: §3.1 · p. 4

3.2 Discovering predictive relationships and evaluating dynamical fidelity

  • Relationship discovery identifies source–target pairs whose source strongly conditions the target distribution in reference trajectories.The candidate catalogue may be informed by physical theory and prior literature, while the reference determines which relationships are retained.Source: §3.2 · p. 5
  • The discovery procedure may be model-based or model-free, causal or purely predictive, and may search candidates individually, greedily, or jointly.Candidate pairs are evaluated on reference trajectories and mapped to a retained set.Source: §3.2 · p. 5
  • Retained relationships act as process diagnostics and are evaluated by comparing conditional distributions or selected statistics, such as means and variances.The comparison uses an application-appropriate discrepancy; the tests can detect errors that invariants and budgets miss.Source: §3.2 · p. 5

4 Instantiating Dynamical Fidelity in Stratospheric Circulation

The study applies predictive-structure evaluation to stratospheric circulation, using ERA5 representations to identify robust precursors of future NAM evolution and test forecasts on frozen subsets.

4.1 Scientific target: Northern Annular Mode evolution

  • NAM evolution is measured over the 50–100 hPa layer from approximately neutral initial states, using 772 admissible ERA5 winter initializations.The target is the subsequent 10-day NAM trajectory.Source: §4.1 · p. 6

4.2 Physical representations and candidate precursor space

  • Candidate precursors span background circulation, planetary-wave propagation and forcing, and circulation response, across regions, windows, and temporal operators.The first-level search covers 7,875 candidate relationships; Figure 1 breaks down maxima by region, representation/wavenumber, and window/operator.Source: §4.2 · pp. 6–7, Figure 1 · p. 6

4.3 Relationship discovery and frozen model evaluation

  • From 7,875 first-level candidates, 283 relationships passed filtering; a conditional search then retained 672 relationships from frozen branches.The searches required at least 95% directional stability and permutation-FDR calibration; the second-level search examined more than 4.3 million candidates.Source: §4.3 · p. 7, Figure 2 · p. 7
  • GraphCast, FengWu, and Pangu-Weather are evaluated on the same frozen initialization subsets using conditional NAM mean and variance errors alongside conventional T, u, and v forecast errors.Figure 3 compares conditional NAM errors across the retained terminal conditions with conventional forecast MAE on the same subsets.Source: §4.3 · p. 7, Figure 3 · p. 8

5 ERA5 reveals sparse and conditional predictive structure

ERA5 contains sparse, hierarchical predictive structure for the subsequent evolution of initially neutral NAM states.

6 ERA5 reveals sparse and conditional predictive structure

  • Predictive signals concentrate in particular combinations of representation, region, temporal window, and aggregation operator rather than broad diagnostic families.Only 295 of 7,875 first-level candidates reproduce their separation direction across winters, and 283 survive permutation calibration.Source: §5 · p. 8
  • Conditioning on first-level states reveals 672 robust second-level relationships, showing that much reference predictability emerges only after conditioning on preceding circulation.The retained relationships describe multiple physically distinct routes from comparable initial NAM states toward different subsequent evolutions.Source: §5 · p. 8

6 Dynamical fidelity in ML forecasts

The models differ in how they preserve ERA5 conditional NAM evolution, and their relative fidelity depends on lead time and predictive structure. Conventional forecast errors do not capture these differences.

6 Dynamical fidelity in ML forecasts

  • FengWu most closely reproduces conditional NAM mean and variance over the first three lead days and has the lowest T, u, and v MAE at every lead; from day 4, Pangu-Weather best matches conditional NAM.FengWu’s conditional errors grow fastest and exceed both other models’ by day 10.Source: §6 · p. 8, Table 1 · p. 9
  • For strong negative NAM shifts, GraphCast’s conditional-mean error is 0.27 versus FengWu’s 0.33, reversing to 0.13 versus 0.09 for weak negative shifts.Pangu-Weather has the lowest conditional-mean error across reported representation and evolution subsets, while FengWu retains the lowest conventional forecast error.Source: §6 · p. 8, Table 1 · p. 9
  • Conventional forecast skill can conceal where model behaviour agrees with or departs from the reference system’s physical dynamics.The reported table compares conditional NAM mean and variance errors with conventional T, u, and v forecast MAE.Source: §6 · p. 9, Table 1 · p. 9

7 Conclusion

The framework uses expert-defined physical representations and robust relationships discovered from reference trajectories to test whether learned models preserve predictive structure.

7 Conclusion

  • Applied to stratospheric circulation, the framework reveals model differences in preserving ERA5 wave–mean-flow relationships and NAM evolution that conventional forecast errors do not capture.Accurate state prediction alone does not establish preservation of reference predictive structure.Source: §7 · p. 9
  • The identified relationships describe predictive, not causal, structure; ERA5 is the reference rather than ground truth.Some precursors summarize histories longer than those directly available to forecast models, so disagreement may reflect missing initialization information or incorrect subsequent evolution.Source: §7 · p. 9
  • Stability filtering and permutation calibration reduce sensitivity to chance relationships within the finite ERA5 winter cohort but do not establish transfer to other periods, reanalyses, or physical systems.Source: §7 · p. 9

AI use statement

Appendix A relates evaluation tests to a differential residual, showing how budgets, invariants, and process diagnostics capture different partial views of dynamics. It also records the authors’ AI-use disclosure.

AI use statement

  • The authors report using generative AI tools for methodological development, coding, analysis, literature work, and manuscript drafting, with authors retaining final responsibility.They state that AI-assisted code and analyses were tested and verified, and that literature and scientific claims were checked against underlying sources.Source: AI use statement · p. 9

A Budget and algebraic constraints as coarsenings of the differential residual

  • Budgets and invariants are not sufficient to establish the reference dynamics, and process diagnostics are not ordered with them.Budgets can miss residual errors that cancel across a fixed interval; invariants and process diagnostics can each miss aspects detected by the other test family.Source: A Budget and algebraic constraints as… · p. 13, §A.1 · p. 14, §A.3 · p. 15
  • The differential residual compares a trajectory’s actual rate of change with the rate prescribed by the reference dynamics; tests on predictions inspect it only at visited states.A zero residual identifies a trajectory the reference system could produce; a non-zero residual signals otherwise.Source: A Budget and algebraic constraints as… · p. 13

A.1 Budgets

  • Budgets check changes in tracked quantities against accumulated tendencies, but finite budgets over fixed intervals can miss errors that cancel in their sums.For example, transport that is too fast in one part of a step and too slow in another can leave the interval budget satisfied.Source: §A.1 · p. 14

A.2 Invariants

  • Invariants are weaker than budgets: a model can preserve an invariant while following the reference dynamics at the wrong speed.An invariant is a budget with an identically zero tendency and can be checked from snapshots without evaluating the reference tendency.Source: §A.2 · p. 14

A.3 Process diagnostics

  • Process diagnostics compare a selected aspect of predicted and reference evolution, but matching them does not ensure the full trajectories agree.Diagnostics and budgets or invariants are not ordered: either family can pass while the other fails.Source: §A.3 · p. 15

A.4 Example: harmonic oscillator

The oscillator examples show that passing one kind of evaluation test does not guarantee agreement with the reference dynamics. The appendix also qualifies its flow-map assumption, especially for large-step emulators.

A.4 Example: harmonic oscillator

  • The speed-scaled oscillator preserves invariants but changes period, while damping preserves period but violates energy; neither reproduces the reference dynamics.The budget of q detects the speed error in the first model, while the damped model fails the energy invariant and every budget built on it.Source: §A.4 · p. 15

A.5 Remark on vector fields and flow maps.

  • The flow-map assumption is an idealization; large-step machine-learning emulators have no stated backward-error guarantee, though finite-difference tests remain possible with discretization error.For numerical one-step methods with small steps, backward error analysis provides an exponentially small error bound.Source: §A.5 · p. 15

B Hybrid and Physics-Informed Architectures

This section explains why architectural constraints do not necessarily guarantee dynamical fidelity and defines atmospheric process diagnostics and a winter-balanced search for predictive relationships. It then tests those relationships for reproducibility across held-out winters and calibrates the search against cross-winter permutation nulls.

B.1 Hybrid models

  • Hybrid models can conserve enforced invariants yet leave closure errors in wave–mean-flow coupling and circulation response unconstrained, making process-level tests necessary.A differentiable-solver model combines resolved equations with a learned closure; invariant conservation constrains closure error only along the invariant gradient.Source: §B.1 · p. 16

B.2 Constrained architectures; B.3 Physics-informed losses

C.1 Zonal Means and Eddy Covariances

  • Zonal eddy covariances encode joint heat and momentum transports, so plausible individual wind and temperature wave fields can still yield suppressed or reversed transport.These transports redistribute heat, affect jets and storm tracks, and mediate wave forcing of the stratospheric polar vortex.Source: §C.1 · pp. 16–17

C.2 Fourier Interpretation

  • Fourier diagnostics separate wave scales and amplitudes from phase alignment: misaligned waves can cancel or reverse covariance, while matching-wavenumber contributions identify transport errors by scale.The main-text diagnostics use total covariance summed across resolved zonal wavenumbers.Source: §C.2 · p. 18

C.3 Eliassen–Palm Flux and Wave Forcing

  • The Eliassen–Palm flux diagnoses meridional and vertical wave propagation, while its divergence measures wave forcing of the circulation; during SSWs, upward wave convergence decelerates polar-vortex winds.The vertical flux component points toward lower pressure for upward propagation.Source: §C.3 · p. 19

C.4 Northern Annular Mode

  • The dimensionless NAM index is a standardized projection onto a pressure-level circulation pattern: positive values indicate a strong vortex, while negative values indicate weakening or disruption.After an SSW, negative NAM values often emerge first in the stratosphere and then appear at progressively lower altitudes.Source: §C.4 · pp. 19–20

D.1 Data and cohort; D.2 Candidate source space

  • The discovery cohort contains 772 neutral initializations from 46 winters, searched across 7,875 candidate sources spanning field diagnostics and past NAM.Neutrality is defined by |N50:100(tc)| ≤ 0.3; candidates combine field representations with latitude regions, pressure regions, temporal windows, and operators.Source: §D.1 · p. 20, §D.2 · p. 20

D.3 Split criterion and eligibility; D.4 Winter-level stability filtering

  • Thresholds exhaustively maximize day-6–10 NAM trajectory variance explained subject to child-size and multiwinter coverage rules; a second-stage filter tests separation direction across held-out winters.Each winter receives equal total weight, and the selected child must meet specified distinct-winter coverage requirements.Source: §D.3 · p. 21, §D.4 · p. 21

D.4 Winter-level stability filtering

  • The filter used 1,000 shared 70/30 winter-level splits, keeping each winter’s initialization dates together and re-estimating thresholds on training winters only.The candidate source specification remained fixed, and the held-out data played no role in selecting the source variable or threshold.Source: §D.4 · pp. 21–22
  • Relationships passed the stability filter only when their direction reproduced in at least 95% of held-out-winter evaluations after threshold re-estimation; this is not a significance test.All 283 retained first-level relationships had at least 999 valid resamples. Among 672 retained second-level relationships, the median was 998; 61 had fewer than 500 and 23 fewer than 20.Source: §D.4 · p. 22

D.5 Permutation-FDR calibration

  • The FDR calibration used 100 null universes that reassigned complete future NAM trajectories across winters, preserving within-trajectory dependence while breaking precursor associations.The complete discovery pipeline, including candidate searches, threshold optimization, and the 1,000-split stability filter, was repeated for each null universe.Source: §D.5 · p. 22
  • The retained FVE threshold was the smallest one meeting a conservative upper FDR bound estimated by bootstrapping complete permutation universes.The method used the 97.5th percentile of 20,000 bootstrap replicates; permutation universes, rather than individual null candidates, were the resampling units.Source: §D.5 · p. 23

D.6 First- and second-level discovery

The section discovers ERA5 relationships in two levels, then freezes the retained structure for evaluating three weather models on common forecast protocols. The evaluations compare preservation of conditional NAM behavior with conventional state-space errors.

D.6 First- and second-level discovery

  • 283 first-level relationships produced 566 fixed branches; the conditional search evaluated 4,358,681 candidates, with 71,494 passing stability and 672 retained.The retained second-level relationships form the evaluation panel.Source: §D.6 · p. 23

D.7 Retained relationship panel

  • The 672 ERA5-discovered relationships form the primary panel; model tests use frozen subsets, with a 287-motif equal-weight analysis as a redundancy sensitivity check.The motifs are not assumed to represent statistically independent or distinct physical mechanisms.Source: §D.7 · p. 24
  • Pangu-Weather has the lowest conditional-mean and conditional-variance errors, while FengWu has the lowest T, u, and v MAE; the ranking is unchanged under motif weighting.FengWu’s conditional-mean error rises from 0.148 to 0.169 under motif weighting, nearly reaching GraphCast’s 0.171.Source: §D.7 · p. 24, Table 2 · p. 24

E.1 GraphCast (operational)

  • GraphCast uses an operational checkpoint pretrained on ERA5 and fine-tuned on operational IFS HRES analyses.It uses a 0.25° grid, 13 pressure levels, and a native 6-hour forecast step.Source: §E.1 · p. 25

E.2 Pangu-Weather (6-hour)

  • The 6-hour Pangu-Weather configuration combines its published 24-hour and 6-hour networks for forecasts throughout the rollout.The system operates at 0.25° resolution on 13 pressure levels and was trained using ERA5 data from 1979–2017.Source: §E.2 · p. 25

E.3 FengWu (operational)

  • The evaluated FengWu checkpoint is a single autoregressive model trained on ERA5 without the later transfer learning on operational analyses.It uses a native 6-hour step, 0.25° resolution, 13 pressure levels, and 69 input variables.Source: §E.3 · p. 25

E.4 Common forecast protocol

  • All three models are initialized on 772 neutral-NAM dates and evaluated with common variables, pressure levels, forecast horizons, and verification procedures.Forecasts run autoregressively for 11 days; conventional T, u, and v errors and forecast NAM are both computed.Source: §E.4 · p. 25

E.5 Forecast coverage

  • All 2,316 model–initialization rollouts are complete through lead days 1–10, with no forecast values masked, clipped, or imputed.The largest absolute forecast N50:100 values are 3.51 for GraphCast, 3.71 for FengWu, and 3.89 for Pangu-Weather.Source: §E.5 · p. 26

E.6 NAM forecast error

NAM forecast error does not capture the relationship-level differences found by conditional errors. The reference forecasts perform much worse overall, though weak-shift rows are an exception for conditional-mean error.

E.6 NAM forecast error

  • 0.453 for GraphCast, 0.294 for FengWu, and 0.275 for Pangu-Weather are their NAM forecast errors over all initializations; Pangu’s advantage over FengWu is smaller than for conditional errors.The conditional-mean error comparison is 0.091 against 0.148, and the conditional-variance error comparison is 0.107 against 0.210.Source: §E.6 · p. 26
  • FengWu has lower NAM forecast error than GraphCast in every table row, yet GraphCast has lower conditional-mean error in five rows, including strong negative shifts.For a strong negative shift, NAM forecast error is 0.590 against 0.456, while conditional-mean error is 0.275 against 0.329.Source: §E.6 · p. 26
  • GraphCast has the smallest unconditional mean error over all initializations, 0.010 against 0.067 for FengWu and 0.032 for Pangu-Weather, but the largest conditional-mean error.Using the ERA5 50/100 hPa mean changes the reported values by at most 0.004.Source: §E.6 · p. 26
  • Climatology and persistence have NAM forecast errors of 0.954 and 0.937, and terminal-leaf conditional-mean errors of 0.714 and 0.668 and conditional-variance errors of 1.172 and 1.144.These reference errors are several times those of all three models, except that weak-negative-shift conditional-mean errors are comparable; reference conditional-variance errors there remain 5–10 times larger.Source: §E.6 · p. 26

E.7 Training-period overlap with the evaluation cohort

The evaluation cohort overlaps the forecast systems’ training and development periods, so it is not a strictly held-out benchmark of general forecast skill. The models are instead compared on the same reference-defined dynamical relationships across the full cohort.

E.7 Training-period overlap with the evaluation cohort

  • 648 of 772 initializations fall in 2017 or earlier, within all three models’ ERA5 training period; 66 fall in GraphCast’s HRES fine-tuning period, and 58 fall in 2022–2025.The reference cohort spans winters from 1979–2025; later years provide temporally out-of-sample cases.Source: §E.7 · p. 26
  • Because substantial portions of the cohort overlap model development or training periods, the experiment is not intended as a strictly held-out benchmark of general forecast skill.The overlap should be considered when interpreting absolute forecast performance.Source: §E.7 · p. 26
  • All three models are evaluated on the same reference-defined dynamical relationships across the full available cohort.Source: §E.7 · p. 26
Loading 2609.34627v1…