PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Predictor-Corrector Dependence in Hard-Constrained Climate Emulators

Tianyu Zhang, Xinyu Wang, Zhenghan Tai, Xiao-Wen Chang, Yoshua Bengio

arXiv:2610.23561v1cs.CYcs.CE

TL;DR

Corrected forecasts alone conflate the neural predictor’s accuracy with the effect of physical correction. The paper separates those contributions with complementary readouts, crossed training–inference conditions, and matched controls; results across ACE2 and ConCerNet show that predictor–correction interactions depend on training conditions, horizon, and system.

  • Problem

    Scoring only corrected forecasts cannot distinguish an accurate learned transition from an inaccurate one whose errors are removed by correction.

    Source: §1 · p. 1

  • Method

    The framework compares corrected forecasts, pre-correction outputs along corrected trajectories, and uncorrected rollouts, while crossing correction during training and inference and using matched controls.

    Source: §1 · p. 1

  • Results

    Across tested ACE2 and ConCerNet settings, predictor–correction interactions depend on training conditions, forecast horizon, and system.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Separately evaluating predictor accuracy and correction effects provides a clearer account of forecast skill in models that use physical correction.

    Source: §7 · p. 9

    The climate experiments use ACE2-family checkpoints whose 1940 evaluation months fall within pretraining, and generalization to unseen periods remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Climate emulators often enforce physical constraints by correcting each state predicted by a neural network, but evaluating only the corrected forecast conflates predictor skill with the effect of the correction. We introduce an evaluation framework that separates these contributions by comparing corrected forecasts, pre-correction predictions along corrected trajectories, and autonomous rollouts with correction disabled. We further cross correction during training and inference to distinguish its immediate effect from how it shapes the learned predictor, and use matched nonphysical controls to test whether improvements are specific to the physical correction. In ACE2, correction substantially improves physical consistency. In a three-seed ACE2 pilot, models trained with correction produce less accurate uncorrected precipitation forecasts at short horizons than models trained without correction, but more accurate forecasts at 10 and 30 days. This contradicts our preregistered hypothesis that the deficit would grow with forecast horizon. In a separate nine-seed experiment, directly supervising the uncorrected output reverses its precipitation forecast deficit at 72 hours while maintaining the accuracy of corrected forecasts at 24 hours. In ConCerNet's heat-equation system, projection-trained models have higher uncorrected trajectory-based vector-field RMSE than models trained without projection, while restoring the physical projection recovers accuracy far more effectively than a matched nonphysical control. Together, these results show that hard corrections can reshape the predictors trained alongside them, creating dependencies that are not visible from corrected forecasts alone. Hard-constrained emulators should therefore be evaluated by separately measuring predictor accuracy, correction effects, and their interaction during training.

1 INTRODUCTION

Corrected forecasts alone cannot distinguish an accurate neural transition from one whose errors the corrector removes. The paper proposes separating predictor skill, correction effects, and their interaction through complementary readouts and crossed training–inference evaluation.

Introduction

  • Corrected-output evaluation alone cannot distinguish an accurate learned transition from an inaccurate transition whose errors are subsequently removed.Physical operations can enforce conservation, positivity, diagnostic closure, or prescribed boundary conditions.Source: §1 · p. 1
  • The evaluation compares matched controls, pre-correction network outputs along corrected trajectories, and rollouts with correction disabled, while keeping learned weights fixed across inference modes.Disabling correction throughout forecasting exposes feedback-dependent input distributions.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Correction changes uncorrected precipitation skill differently by horizon: correction-trained models have higher RMSE at 24 and 72 hours but lower RMSE at 10 and 30 days.ConCerNet shows a different pattern: projection-trained models rely strongly on physical projection to recover predictive accuracy.Source: §1 · p. 2
  • The framework attributes forecast skill to the learned predictor, its corrector, and their interaction during training, and is evaluated in ACE2, ConCerNet, and ACE2S-SHiELD+.The paper also uses matched nonphysical controls to test whether the physical correction contributes beyond an equal computation budget or nonphysical perturbation.Source: §1 · pp. 1–2

2 RELATED WORK

Related work spans neural weather and climate models, physics-guided and closed-loop learning, and methods that enforce constraints in learned predictions. Prior work on correction dependence motivates testing raw outputs and autoregressive feedback with nonphysical controls.

Neural atmospheric models

  • GraphCast, Pangu-Weather, NeuralGCM, and ACE2 exemplify neural atmospheric forecasting, learned components coupled to dynamics, and weather-to-climate rollouts.The cited models include GraphCast and Pangu-Weather, NeuralGCM (Kochkov et al., 2024), and ACE2 (Watt-Meyer et al., 2025).Source: §2 · p. 2

Physics-guided and closed-loop learning

  • Physics-guided and closed-loop learning includes analytic atmospheric constraints, hard-constrained turbulence architectures, learned solver corrections, and feedback-induced distribution shifts.These lines of work motivate evaluating learned components when their outputs become future inputs.Source: §2 · pp. 2–3

Corrector dependence and negative controls

  • Prior work found precipitation rescaling can hide deterioration in raw network outputs; this paper additionally disables correction in autoregressive feedback and crosses training with inference conditions.Spatial shams disrupt physical alignment as target-independent negative controls.Source: §2 · p. 3

Constraint-enforcing operations

  • Constraint-enforcing approaches include optimization layers, completion-and-correction, homeomorphic projections, and constrained generation; ConCerNet uses vector-field projection to preserve invariants.ConCerNet provides a setting for crossed training–inference evaluation (Zhang et al., 2023).Source: §2 · p. 3

3 METHOD

The method separates corrected forecasts, network outputs before correction, and autonomous uncorrected rollouts, then crosses correction conditions during training and inference. It also uses matched controls, an auxiliary raw-output objective, and a constrained heat-equation system to assess predictor–corrector dependence.

3.1 PREDICTOR–CORRECTOR FORMULATION

  • Corrected rollouts score the corrected state and feed it forward; pre-correction readouts score the network output while retaining corrected-state feedback; uncorrected rollouts recursively feed network outputs back.The uncorrected rollout measures forecast accuracy including effects of earlier predictions on later inputs; its first prediction matches the pre-correction output, but later predictions can differ.Source: §3.1 · p. 3

3.2 MATCHED-CONTROL EVALUATION IN ACE2

  • ACE2 matched-control evaluation compares physical correction with no correction and three spatial shams, using nine seeds, four 1940 months, two horizons, and three variables.Shams disrupt spatial alignment; the evaluation considers predictive accuracy, conservation, positivity, correction magnitude, and decorrelation.Source: §3.2 · p. 3

3.3 TRAINING–INFERENCE EVALUATION IN ACE2

  • The ACE2 pilot crosses correction during adaptation and forecasting, yielding four conditions while holding each adapted model’s weights and evaluation settings fixed across inference conditions.It uses three seeds, four months, and horizons from 24 hours to 30 days; positive adaptation effect A_s indicates lower uncorrected RMSE after correction-trained adaptation.Source: §3.3 · p. 4

3.4 AUXILIARY SUPERVISION OF PRE-CORRECTION PREDICTIONS

  • The auxiliary objective supervises pre-correction surface precipitation with a normalized squared-error term while leaving the main loss unchanged.The weight is selected on validation forecasts subject to a corrected-forecast accuracy threshold, and all models receive 2400 adaptation updates.Source: §3.4 · p. 4

3.5 EVALUATION IN A HEAT-EQUATION SYSTEM

  • In ConCerNet’s 101-dimensional heat-equation system, physical projection subtracts the mean vector-field component, enforcing a zero-sum vector field; a norm-matched sham is also evaluated.Trajectory-based vector-field RMSE evaluates the model and reference vector fields at their respective forecast and reference states, combining vector-field and evaluation-state differences.Source: §3.5 · p. 4

3.6 STATISTICAL ANALYSIS

  • ACE2 matched-control analyses use seed-level inference and Holm correction, while pilot analyses with three seeds remain descriptive; direct-supervision results use nine adaptation seeds and ConCerNet uses nine independent seeds.The pilot’s normality cannot be assessed reliably, and its minimum one-sided exact sign-test p-value is 0.125; ConCerNet’s confirmatory horizon is 1.0, with other horizons descriptive.Source: §3.6 · p. 5

4 RESULTS

Across ACE2, correction strongly improves physical consistency, while its precipitation-forecast effects depend on horizon and training conditions. ConCerNet results likewise show that physical projection can recover accuracy more effectively than a matched sham.

4.2 PHYSICAL CONSISTENCY AND LONG-HORIZON ERROR

  • With official correction during adaptation and forecasting, mean maximum dry-air drift is 0.0625 Pa versus 1332 Pa without correction, and negative precipitation falls from 33.1% to 0%.Mean water-budget residual magnitude is 1.15 × 10−12 versus 7.93 × 10−5 kg m−2 s−1.Source: §4.2 · p. 5
  • With correction at both stages, precipitation RMSE is higher than with neither stage corrected by 2.46% over 10 days and 1.75% over 30 days, without observed divergence.These RMSE effects average error over each rollout window.Source: §4.2 · p. 5

4.1 PRACTICAL GAINS IN TEMPERATURE

  • Across 24 matched-regime comparisons, 23 favor official correction and remain significant after Holm correction; all eight effects meeting practical-importance thresholds are for temperature.The 72-hour precipitation comparison against no correction is inconclusive.Source: §4.1 · p. 5, Table 1 · p. 6, §4.3 · p. 6

4.3 HORIZON-DEPENDENT REVERSAL OF THE UNCORRECTED GAP

  • At 10 days, correction-trained predictors’ 13.48% lower uncorrected precipitation RMSE contrasts with 11.51% higher temperature RMSE and 9.17% higher height RMSE.At 24 hours, adaptation with correction increases uncorrected 500 hPa height RMSE by 93.72% on average, while enabling inference correction reduces it by 51.58%.Source: §4.3 · p. 6

4.4 DIRECT SUPERVISION REDUCES THE SHORT-HORIZON GAP

  • Direct supervision reverses the uncorrected 72-hour precipitation RMSE gap in all nine seeds, changing the mean gap from +4.92% to −4.43%, while lowering corrected 24-hour RMSE on average.The direct-supervision evaluation uses June and September 1940, whereas the training–inference pilot averages four months.Source: Table 2 · p. 6, §4.4 · p. 6

4.3 HORIZON-DEPENDENT REVERSAL OF THE UNCORRECTED GAP

  • With correction disabled throughout forecasting, correction-trained predictors have 59.7% higher precipitation RMSE at 24 hours and 7.13% higher at 72 hours, but 13.48% and 15.69% lower at 10 and 30 days.The reversal contradicts the preregistered hypothesis that the uncorrected deficit would grow with forecast horizon.Source: §4.3 · p. 6, Figure 2 · p. 7

4.5 WITHIN-FAMILY ENERGY-CORRECTION EVALUATION

  • At fixed ACE2S-SHiELD+ checkpoints trained with energy correction, enabling it reduces 24-hour 500 hPa height RMSE by 4.29% and 32.00% in rs0 and rs1; at checkpoints trained without it, effects are below 0.1%.At the latter checkpoints, enabling correction increases 30-day precipitation RMSE by 6.68% and 9.85%; the separate between-checkpoint comparison is confounded by training-budget and correction-condition differences.Source: §4.5 · p. 7, Figure 3 · p. 8

4.6 PROJECTION DEPENDENCE IN CONCERNET

  • At t = 1, projection-trained ConCerNet models have 26.87 times the uncorrected vector-field error of models trained without projection; physical projection reduces their error 92.65%, versus an approximately 2.82% increase with the norm-matched sham.At the primary horizon, the physical–sham error-reduction difference is 95.47 percentage points.Source: §4.6 · pp. 7, 9, Figure 4 · p. 8

5 DISCUSSION

Correction effects vary with adaptation history and forecast horizon, while direct supervision can improve short-horizon uncorrected precipitation accuracy and corrected accuracy. The discussion also proposes diagnostics and reports a distinct projection-versus-sham result in ConCerNet.

5 DISCUSSION

  • At longer horizons, the correction-adapted predictor has lower uncorrected precipitation RMSE, yet enabling its corrector increases RMSE; correction reduces mean RMSE at every evaluated horizon for the predictor adapted without it.Source: §5 · p. 9
  • The authors propose tracking precipitation bias, variability, and spatial agreement alongside other-field errors, plus correction magnitudes and water-budget residuals over forecast lead.These diagnostics could assess whether changes in applied correction coincide with the horizon-dependent reversal.Source: §5 · p. 9
  • Direct supervision of pre-correction precipitation reverses the 72-hour uncorrected RMSE gap and improves corrected 24-hour accuracy.The intervention links the short-horizon deficit to the adaptation objective.Source: §5 · p. 9
  • In ConCerNet, physical projection recovers prediction accuracy more effectively than a norm-matched sham.This extends predictor-corrector dependence to a constrained dynamical system with an analytic invariant.Source: §5 · p. 9

6 LIMITATIONS

The climate results come from adapted ACE2-family checkpoints under prescribed sea-surface temperature and sea ice, with evaluation months inside pretraining. The authors note limits on tested horizons, seeds, and model families.

6 LIMITATIONS

  • 1940 evaluation months fall within the released models’ pretraining period, leaving generalization to unseen periods untested.Source: §6 · p. 9
  • The horizon-dependent reversal is based on three seeds, and longer-horizon matched-regime comparisons remain descriptive.Source: §6 · p. 9
  • Direct supervision was tested at 72 hours without correction and 24 hours with correction; extension beyond ACE2-family models and heat-equation ConCerNet remains open.Source: §6 · p. 9

7 CONCLUSION

The conclusion argues that separating corrected forecasts, pre-correction outputs, and uncorrected rollouts clarifies predictor-correction interactions. Across the tested ACE2 and ConCerNet settings, these interactions vary with training conditions, forecast horizon, and system.

7 CONCLUSION

  • Separating corrected forecasts, pre-correction outputs, and uncorrected rollouts reveals how physical correction interacts with the learned predictor.Source: §7 · p. 9
  • Across tested ACE2 and ConCerNet settings, the interaction depends on training conditions, forecast horizon, and system.Source: §7 · p. 9
  • Direct supervision improves the short-horizon ACE2 predictor while preserving corrected accuracy, supporting separate evaluation of predictor accuracy and correction effects.Source: §7 · p. 9

AI USE STATEMENT

The authors used generative AI tools across coding, experiments, editing, and draft illustrations, and state that they reviewed the outputs. The appendix defines how the disabled corrector and raw outputs are represented.

AI USE STATEMENT

  • Generative AI assisted with coding, experiment management, manuscript editing, and draft figure illustrations; the authors reviewed the outputs and take responsibility for the final materials.Source: AI USE STATEMENT · p. 10

A.1 PREDICTION AND CORRECTION

  • The appendix writes the disabled corrector as C0(zt+1, xt, bt) = zt+1 and reconstructs raw outputs by subtracting recorded correction from corrected outputs at each rollout step.Source: §A.1 · p. 12

A.2 EVALUATION CONTRASTS

The evaluation contrasts correction during training and inference, then defines how forecast errors and relative effects are aggregated. The direct-supervision experiment is triggered by prespecified pilot criteria and evaluated across nine seeds.

A.2 EVALUATION CONTRASTS

  • The four training–inference contrasts separate correction during adaptation from correction at every forecast step.The reported contrast between inference effects is calculated within each seed and expressed in percentage points.Source: Table 3 · p. 12, §A.2 · p. 12

A.3 AGGREGATION OF PREDICTION ERRORS

  • Matched-control relative effects are averaged equally across four months and then across nine training seeds.Each seed- and month-level RMSE averages seven paired forecast starts; long-horizon RMSE instead averages saved six-hour leads and initialization dates within each rollout window.Source: §A.3 · p. 12

B.1 DIRECT SUPERVISION OF UNCORRECTED OUTPUTS

  • The direct-supervision experiment starts when the pilot meets specified short-horizon deficit and corrected-forecast criteria, and evaluates nine adaptation seeds across four forecast horizons.It compares supervised models with correction enabled and disabled against three matched baselines; its intervention criteria use the 72-hour uncorrected precipitation gap and 24-hour corrected precipitation RMSE change.Source: §B.1 · p. 13

B.2 STATISTICAL ANALYSIS

The analysis treats training seeds as independent units, with months and forecast starts nested within seeds, and reports uncertainty and practical-effect thresholds. It also describes physical-consistency, matched-control, long-horizon, direct-supervision, and energy-correction evaluations.

B.2 STATISTICAL ANALYSIS

  • The matched-control analysis uses seed-level effects, hierarchical-bootstrap intervals, and Holm-adjusted sign-flip tests across 24 prespecified comparisons.Practical-effect thresholds are 0.5% for temperature and 5% for height and precipitation; long-horizon analyses are descriptive secondary outcomes.Source: §B.2 · p. 13

B.3 PHYSICAL-CONSISTENCY PILOT

  • The physical-consistency pilot aggregates dry-air drift, water-budget residuals, and negative-precipitation fraction over seven 72-hour forecast starts and three seeds.Dry-air drift is the maximum absolute change in area-weighted global-mean dry-air surface pressure relative to initialization.Source: §B.3 · p. 13

C.1 MATCHED-CONTROL COMPARISONS

  • Figure 5 displays all 24 prespecified matched-condition comparisons of the official correction’s relative RMSE reduction against controls, alongside pointwise hierarchical-bootstrap 95% confidence intervals; nine independent training seeds were used.Source: Figure 5 · p. 14

C.2 LONG-HORIZON COMPARISONS

  • The long-horizon matched-regime evaluation compares window-averaged RMSE changes for 10- and 30-day forecasts against matched no-correction models.It uses four initialization dates per month for 10-day forecasts and one for 30-day forecasts; Figure 6 marks negative changes as lower RMSE.Source: §C.2 · p. 14

C.4 DIRECT-SUPERVISION SEED COMPARISONS

  • Figure 7 shows the individual paired-seed changes summarized in Table 2.Source: §C.4 · p. 14

D.1 WITHIN-FAMILY ENERGY-CORRECTION EVALUATION

  • The ACE2S-SHiELD+ evaluation uses two seed families and four 1979 forecast starts; its between-checkpoint comparison is budget-confounded and disables inference correction.Figure 8 therefore does not isolate a causal effect of correction during adaptation.Source: §D.1 · p. 14

D.2 CONCERNET EVALUATION CRITERIA

The ConCerNet criteria report seed-level prediction-error statistics and distinguish physical-projection effects from sham-projection effects. The supplied figures also describe long-horizon RMSE changes and an unequal-budget checkpoint comparison.

D.2 CONCERNET EVALUATION CRITERIA

  • Figure 6 organizes long-horizon RMSE changes by rollout length and forecast variable; negative values indicate lower RMSE.The dashed line denotes no change, and the figure shows no confidence intervals.Source: Figure 6 · p. 15
  • Figure 8 compares checkpoints adapted with correction using unequal adaptation budgets, so the between-checkpoint contrast does not isolate correction’s causal effect.Both checkpoint families are evaluated without the inference energy corrector.Source: Figure 8 · p. 17
  • Invariant drift measures the mean absolute change in the conserved state sum from its initial value.The passage defines its reduction under physical projection, but the supplied text does not include the resulting value.Source: §D.2 · p. 17
  • All prediction errors are trajectory-based vector-field RMSE, and the reported lower interval endpoint is a two-sided 95% bootstrap bound over nine seeds.The excess ratio is the total error ratio minus one; the physical–sham contrast is reported in percentage points.Source: §D.2 · p. 18, Table 5 · p. 18

Related papers

Loading 2610.23561v1…