Source-linked AI summary
Too Rare to Learn: Prescribed Cyclone Tracks Degrade a Bay of Bengal Ocean Emulator
Sumaiya Islam
TL;DR
Regional ocean emulators need to forecast unseen cyclones, but evidence for prescribing cyclone information in the intermediate regime of rare conditioning is limited. This paper withholds 15 Bay of Bengal cyclones and compares matched U-Nets with and without four prescribed track channels. The ocean-only model beats persistence in every run, whereas storm conditioning loses to persistence, with the failure associated with rare channel activation and reversible by silencing the cyclone map at inference.
Problem
Evidence is limited on whether an informative but rarely activated cyclone channel improves regional ocean emulation for unseen storms.
Method
The study withholds 15 whole Bay of Bengal cyclones and compares matched U-Nets with and without four prescribed best-track cyclone channels.
Results
Every ocean-only run beats persistence, every storm-conditioned run loses to it, and silencing the cyclone map improves held-out storm RMSE by 7.5% to 16.4% in every seed.
Takeaways & Limitations
Track-derived conditioning should be judged by how often its signal is exercised during training, not only by how informative it is when present.
Takeaways & Limitations
The claim is scoped to track-derived conditioning at a one-day lead, using one basin, architecture, and GLORYS12 reanalysis.
Abstract
from arXiv · showhide
Neural ocean emulators are being proposed for regional forecasting in cyclone-exposed coastal seas, and a natural design choice is to hand the network the cyclone as a prescribed input. We test that choice in the Bay of Bengal and find it harmful. We withhold 15 whole cyclones spanning 65 to 150 kt from GLORYS12 reanalysis and compare two U-Nets that are identical except for four prescribed cyclone-track channels. Across three seeds the ocean-only model beats persistence in every run and the storm-conditioned model loses to it in every run, with the two skill ranges disjoint (p = 3.1e-5, paired across storms). The cause is exposure frequency rather than signal content: the channels are non-zero on only 7.9% of training days, so they are out of distribution the moment they activate. The extra error falls inside the prescribed storm footprint, and replacing the real cyclone map with a no-storm map at inference improves held-out storm forecasts by 7.5 to 16.4% in every seed. The conditioned network has learned a response to a rare signal that is confidently wrong.
1 Introduction
Coastal cyclone forecasting in the Bay of Bengal motivates ocean emulators that can anticipate unseen storms. This study measures the previously untested intermediate regime where informative cyclone inputs are prescribed but rare during training, finding that rare conditioning performs worse than no conditioning.
- Motivation: Bay of Bengal coastal warnings need accurate next-day forecasts of surface temperature and salinity during unseen tropical cyclones.Sea surface temperature affects storm enthalpy and cold wakes, while salinity controls the barrier layer that suppresses cooling.
- Motivation: Ocean-only neural forecast systems cannot know that a cyclone is arriving because they use only past ocean states.Their unseen-storm behavior is therefore fixed by patterns inferred from ordinary days.
- Research gap: Prior work brackets but does not answer whether prescribing a cyclone helps when the cyclone channel is informative but rare.Cyclone-only datasets expose models to storms in every sample, unlike operational regional emulators.
- Approach and contribution: The study withholds 15 whole Bay of Bengal cyclones and compares otherwise nearly identical U-Nets with and without four prescribed track channels.The models differ by only 864 parameters out of roughly 101,000, and the conditioned model loses to persistence in all six runs while the unconditioned model beats it.
- Approach and contribution: The reported failure is attributed to sparse exposure to the conditioning signal rather than to its content.The channels are non-zero on 7.9% of training days, and the penalty concentrates where they activate.
2 Experimental setup
The experiment uses event-disjoint GLORYS12 reanalysis data, prescribed best-track cyclone fields, and matched U-Nets evaluated against persistence. Whole storms are withheld across intensity and monsoon-season strata, with checkpoint selection restricted to storm-free windows.
- Data and inputs: The four prescribed channels encode cyclone presence, intensity, and two motion components as Gaussian-rendered best-track fields.They are provided for the forecast target day and contain no ocean state.
- Data splits: Fifteen severe cyclones spanning 65 to 150 kt and both monsoon seasons are withheld as whole events, yielding 272 evaluation windows.Each cyclone is buffered by seven days to prevent event overlap between training and evaluation.
- Models and training: The ocean-only and storm-conditioned U-Nets use six and ten input channels, respectively, but differ in capacity by only 0.9%.Both predict the next-day state from three days of history using masked MSE and AdamW across seeds 1, 2, and 3.
- Metric: Skill is measured against persistence using S = 1 − RMSE_model/RMSE_pers, where persistence repeats the most recent observed state.The study reports both quiet-selected and final-epoch checkpoints.
- Metric: RMSE is reported in degrees Celsius for temperature and PSU for salinity.
3 Results
Prescribed cyclone conditioning reverses the model’s skill relative to persistence on held-out storms. The advantage of the ocean-only model is concentrated in salinity, while per-cyclone results show a limited but consistent pattern rather than universal failure.
- Pooled skill: [+0.048, +0.161] versus [−0.477, −0.075]: ocean-only skill exceeds persistence in every run, while storm-conditioned skill falls below it.The ranges are disjoint, and the ocean-only arm beats the conditioned arm across the 15 storms at p = 3.1 × 10^-5.
- Per-cyclone results: Ocean-only exceeds persistence on 11 to 14 of 15 cyclones depending on seed, with one 1998 storm as the consistent exception.Storm-conditioned runs reach persistence on six cyclones in seeds 1 and 2, but none of the four most intense storms.
- Per-cyclone results: Figure 2 orders cyclones by persistence difficulty and shows one point per seed with a bar at each seed mean.
- Variable-specific results: 0.036 PSU: ocean-only improves on persistence in salinity, while storm-conditioned does not improve at all.Both learned models are worse than persistence for temperature, so the pooled advantage comes from salinity.
- Variable-specific results: 0.141°C and 0.226 PSU: persistence has lower temperature and salinity RMSE than storm-conditioned values of 0.318°C and 0.236 PSU.Ocean-only RMSE is 0.168°C and 0.190 PSU, respectively.
4 Why conditioning hurts
The conditioning penalty appears when cyclone channels activate: rare exposure produces a wrong response concentrated in storm conditions and locations. Withholding the cyclone map improves forecasts, reinforcing that exposure frequency—not signal content—drives the failure.
- Rare exposure: 7.9% of training days contain non-zero cyclone channels, leaving the storm response estimated from a small, unrepresentative slice.On the remaining 92.1% of days, the network sees a constant no-storm field.
- Regime dependence: Conditioning costs more on held-out cyclone days than storm-free days in every seed, including seed 1's −0.026 calm-day versus +0.123 storm-day contrast.The penalty is mildly beneficial while channels are off but harmful once they activate.
- No-storm intervention: 15.3%, 16.4% and 7.5% RMSE improvements result when the real cyclone map is replaced by a no-storm map at inference.The improvement occurs on held-out cyclone days in every seed.
- Spatial localization: 70.6%, 97.8% and 101.5% salinity-RMSE penalties occur inside the prescribed storm footprint, versus 6.2%, 15.8% and 48.5% outside.The inside-to-outside penalty ratios are 11.3, 6.2 and 2.1, supporting exposure frequency rather than signal content as the explanation.
5 Conclusions and future work
The paper concludes that prescribed cyclone tracks can harm one-day regional ocean emulators on unseen cyclones when the conditioning signal is rarely exercised during training. It scopes the claim to track-derived conditioning and proposes denser forcing and multi-day rollouts as future work.
- Conclusion: Rare track conditioning made the emulator worse than persistence on unseen cyclones, while the same network without it beat persistence.The proposed criterion is to judge conditioning by how often it is exercised during training, not only by its informativeness when present.
- Scope and future work: The claim is scoped to track-derived conditioning at a one-day lead.Future work will add dense ERA5 forcing and extend evaluation to multi-day rollouts.
Climate impact statement
Short-range upper-ocean forecasts matter for coastal Bangladesh and eastern India during cyclone season. The study evaluates whether an intuitive storm-aware input helps across a deliberately structured set of held-out cyclones.
- Climate impact: Short-range upper-ocean forecasts support coastal Bangladesh and eastern India during cyclone season.The statement motivates the practical setting of the study.
- Evaluation scope: 15 held-out cyclones are selected by intensity terciles crossed with monsoon season rather than by fame.The design aims to test when conditioning fails across the sampled regimes.
B Per-variable results
Per-variable evaluation shows that the ocean-only advantage is confined to salinity, while both models underperform persistence on temperature. Regime-level reporting compares ocean-only and storm-conditioned skill relative to persistence.
- Temperature: Both model arms are worse than persistence at temperature at a one-day lead in every run.The pooled physical-unit RMSE table reports lower values as better and compares final-epoch checkpoints by seed.
- Regime comparison: Table 4 reports ocean-only skill minus storm-conditioned skill by regime, with positive values meaning conditioning hurts.The comparison is evaluated separately for calm and cyclone days against each regime's persistence baseline.
D Channel occupancy
Cyclone conditioning is rare and concentrated in weak storms, leaving the network with little exposure to the intense signals present in the held-out test set.
- The event-disjoint split withholds 15 test cyclones and 7-day buffers, preventing their days and adjacent-week days from entering training.
- 542 of 885 active training days carry peak winds below 30 kt, while only 19 exceed 90 kt.
- Only 5 training days, or 0.045% of the split, contain storms above 120 kt.The test set includes four cyclones at that intensity and seven above 90 kt.
E Channel construction and training configuration
The experiment compares nearly identical U-Nets using cyclone maps derived from best-track data, with storm-free checkpoint selection and explicit scope limits around forcing, rollout, and generality.
- Channel construction: The cyclone channels encode position, intensity, and motion, while both arms use the same one-level U-Net architecture.The storm-conditioned arm receives ten input channels versus six for the ocean-only arm.
- Channel occupancy: Figure 6 shows no storm on 10,310 of 11,195 training days and a storm above 120 kt on only five.
- Training configuration: Checkpoint selection uses 33 storm-free 21-day blocks, excluding cyclone periods from validation and leaving Phailin as a monitored validation cyclone.This avoids tuning reported weights to the storm regime under test.
- Limitations: The experiment does not establish that dense physical forcing would fail similarly because the prescribed channels omit ocean state, wind stress, and heat flux.The sparsity argument may not transfer to a field present every day.
- Seed 3: Seed 3 is a poorly converged outlier, with pooled skill −0.477 versus about −0.076 for seeds 1 and 2.The authors characterize this pattern as failed optimisation rather than a stronger conditioning effect.
- Seed 3: The restarted seed-3 run is not reproducible from the same seed because CUDA non-determinism changes the trajectory after the machine entered swap.Seeds 1 and 2 alone still produce a disjoint storm-conditioned skill range of [−0.078, −0.075].
H Where the penalty falls
The conditioned model’s excess error is concentrated inside the prescribed storm footprint rather than spread uniformly across held-out windows, supporting a footprint-specific failure pattern.
- 112 of 272 test windows carry a prescribed storm footprint above the 0.2 presence threshold.The remaining windows lie in the 7-day buffers and have no footprint to split on.
- The conditioned arm’s penalty inside the footprint is 11.3, 6.2, and 2.1 times the outside penalty for seeds 1 to 3.The penalty is larger inside the footprint for every seed, although the margin narrows as the run degrades.
- The footprint analysis distinguishes a localized response to the cyclone map from uniform under-training or under-parameterisation.The associated table reports salinity RMSE inside and outside the footprint, with penalty defined relative to the ocean-only arm.
- Figure 7 compares observed salinity change, ocean-only error, storm-conditioned error, and the cyclone channel for one day of Cyclone Fani.The conditioned arm shows a broad southern-basin error where the cyclone channel is active.
I Classical baselines
Classical baselines show that damped persistence is nearly pure persistence, while the ocean-only model remains better than both classical references and the storm-conditioned model performs worse than all three.
- Classical baselines: α = 0.97 at a one-day lead, making damped persistence almost pure persistence.Climatology alone is nearly four times worse.
- Classical baselines: +0.010 skill is the gain of damped persistence over persistence.The blend uses α fitted on 594 storm-free validation windows.
- Classical baselines: +0.048 to +0.161 is the ocean-only skill range, clearing both persistence and the strongest classical baseline.The storm-conditioned arm ranges from −0.075 to −0.477, below all three baselines.
- Classical baselines: Table 6 evaluates climatology and damped persistence on the same 272 held-out cyclone windows as the main comparison.Scores use the same aggregation as Table 1, with lower RMSE better.
- Correctness checks: The comparison required correcting persistence indexing, checkpoint-channel mismatches, and the standardized value used to silence cyclone channels.These pipeline errors could otherwise have produced confident but wrong numbers.