PaperlayerView exact revision on arXiv

Source-linked AI summary

Digital Typhoon Dataset V3: Benchmarking for Multi-spectral Extension and Analyzing Systematic Bias in Intensity Regression

Asanobu Kitamoto, Victor Le Louarn, Noelia Falczuk, Ngoc Anh Tong, Anja Veleva

arXiv:2610.21654v1cs.CV

TL;DR

Digital Typhoon V3 adds multispectral channels to a nearly 50-year typhoon satellite dataset, enabling tests of whether multi-channel inputs improve regression and analyses of intensity bias. The benchmarks find no clear pressure-regression improvement from multi-channel inputs in the current setting, while weighting strategies do not fully eliminate underestimation for strong typhoons.

  • Problem

    The study asks whether multi-spectral inputs improve regression and how to address intensity-regression underestimation for strong typhoons.

    Source: §1 · p. 2

  • Method

    The paper adds three Himawari channels, benchmarks basic vision models on single- and multi-channel data, and tests sample weighting and inverse-density weighting inspired by BalancedMSE.

    Source: §1 · p. 2, §5.3 · p. 17

  • Results

    Multi-channel inputs do not clearly improve pressure regression in the current setting, and weighting strategies reduce extreme-case bias but do not fully eliminate regression-to-the-mean behavior.

    Source: §6 · p. 19

  • Takeaways & Limitations

    Benchmark interpretation must account for the reduced dataset size and temporal coverage caused by requiring all four channels to be available.

    Source: §4.7 · p. 13

    The multi-channel dataset has shorter temporal coverage and fewer observations because it retains only times when all four channels are available.

    Source: §4.7 · p. 13

Abstract

from arXiv · show

This paper presents the Digital Typhoon Dataset V3, a new version of the longest typhoon satellite image dataset spanning nearly 50 years, designed to benchmark machine learning models on long-term spatio-temporal data. The latest addition to Dataset V3 is multispectral satellite data, enabling multi-channel models. We investigate data-splitting strategies and present benchmarking results for the Northern and Southern Hemispheres. Experimental results show that, under the current dataset construction and experimental setting, multi-channel inputs do not clearly outperform single-channel inputs for pressure regression. Among the evaluated model architectures, image types, and spectral channels, ResNet-18, full-resolution images, and Channel 1 generally provide the best single-channel performance for pressure regression. Performance also differs between the WP and AU datasets by regression target, with lower RMSE on WP for pressure regression and on AU for wind regression. These results highlight the importance of considering dataset-specific and task-specific characteristics when interpreting benchmark performance. Additionally, we analyzed systematic bias in intensity regression, which leads to underestimation of intensity for strong typhoons. We tested imbalance-aware weighting strategies, including inverse-density weighting inspired by BalancedMSE, and showed that these approaches are not sufficient to remove the bias. The dataset is publicly available at http://agora.ex.nii.ac.jp/digital-typhoon/dataset/ and https://github.com/kitamoto-lab/digital-typhoon/.

1 Introduction

Digital Typhoon spans nearly 50 years of tropical-cyclone satellite observations and supports machine-learning tasks. Version 3 adds multispectral channels, prompting benchmarks of single- and multi-channel models and analysis of intensity-regression bias.

1 Introduction

  • The dataset began in November 2023 with typhoon-centered satellite images and Japan Meteorological Agency best-track data, spanning 1978 to 2025.It was designed to support machine-learning tasks including intensity forecasting.Source: §1 · p. 1
  • Version 2 added the Southern Hemisphere and increased dataset size from 54GB to 77GB.This enabled comparisons across both hemispheres to examine regional differences.Source: §1 · p. 1
  • Version 3 adds a second infrared, water vapor, and near-infrared channel to the prior single-infrared dataset, expanding its total size to 231.3GB.The WP dataset is 164.9GB and the AU dataset is 66.4GB.Source: §1 · p. 2
  • The new channels motivate testing whether multi-spectral models outperform single-channel models and analyzing systematic underestimation for strong typhoons.The study trains basic vision models to compare single- and multi-channel performance.Source: §1 · p. 2

2 Digital Typhoon Dataset V3

Dataset V3 extends the record with 2025 observations and additional Himawari channels, while updating Pyphoon and correcting identified data issues. Its channel-matching policy favors continuity, and the corrected release is reported to preserve the paper’s benchmark conclusions.

2.1 Temporal Dimension

  • The dataset adds the 2025 typhoon season for WP and the 2024–2025 cyclone season for AU, increasing the number of tropical cyclones and images.WP data use JMA best tracks, while AU data use BoM best tracks.Source: §2.1 · p. 2

2.2 Spectral Dimension

  • V3 adds Himawari channels 2–4 and assigns channel numbers by wavelength overlap to prioritize continuity, although wavelength definitions differ across satellites.Channel 3 is water vapor, and Channel 4 is near-infrared; visible channels are excluded because their day-night patterns and long-term noise complicate quantitative analysis.Source: §2.2 · pp. 2–3

2.3 Updates on Pyphoon

  • Pyphoon3 handles single- and multi-channel datasets by discarding observation times when any channel is unavailable, which makes multi-channel sets smaller.It also reorganizes directories by channel and then hemisphere to support both task types.Source: §2.3 · pp. 3–4

2.4 Dataset Availability

  • The dataset is available under CC BY 4.0, and its version naming distinguishes major updates, annual seasons, and same-cycle corrections.The experiments used V3-2025.R1; the paper identifies V3-2025.R2 as the corrected release.Source: §2.4 · p. 4

2.5 Data Corrections in V3-2025.R2

  • The corrected V3-2025.R2 release regenerated 4,056 images and removed 46; retraining found conclusions hold, with median evaluation RMSE improving about 0.1 hPa on WP and AU differences within run variability.Benchmarks used V3-2025.R1; evaluation sets are identical across releases, while only training sets differ.Source: §2.5 · p. 5

3 Use Cases of the Digital Typhoon Dataset

The dataset is used across general-purpose machine-learning benchmarks, tropical-cyclone intensity work, image forecasting, track analysis, and environmental research. Its long homogeneous image sequence and best-track records serve different research needs, while published machine-learning use has largely focused on WP data.

Testbed for general-purpose models and toolkits

  • General-purpose model and toolkit benchmarks use the dataset for tasks including central-pressure forecasting, wind-speed estimation, and video prediction.In this role, the dataset is valued as a long, homogeneous, openly licensed spatio-temporal image sequence with scalar labels.Source: §3 · p. 5

Intensity estimation and forecasting

  • Intensity studies span estimation and forecasting approaches, while several works identify the intense, data-sparse regime and underestimation of extreme intensity as recurring challenges.A recent benchmark also reports that vision foundation models’ visual shortcuts collapse for intense tropical cyclones.Source: §3 · p. 6

Other use cases

  • Published uses include cloud-image prediction and generation, best-track-based track studies, and meteorological, climatological, and environmental analysis.These span image-based forecasting, track modeling, and analyses using best-track records.Source: §3 · p. 6

Use patterns

  • Most machine-learning uses still rely on WP data, while AU data introduced in V2 has mainly served as a held-out set for cross-hemisphere generalization.The dataset is also increasingly used to probe general-purpose models rather than only tropical-cyclone-specific models.Source: §3 · p. 6

4 Benchmarking the Digital Typhoon Dataset V3

Section 4 defines the hemisphere- and channel-specific datasets and the benchmark protocol, including a temporally intersected multi-channel set and a held-out recent-season evaluation split.

4.1.1 Single-Channel Datasets

  • A single-channel dataset contains all observations from one channel for one hemisphere; WP-1 and AU-1 denote Channel 1 datasets, with the same naming pattern for Channels 2–4.Table 2 reports cyclone and image counts for each single-channel dataset.Source: §4.1.1 · p. 6

4.1.2 Multi-Channel Datasets

  • The multi-channel dataset retains only times when all four channels are available, and the evaluated inputs use Channel 1 + 2, Channel 1 + 3, or Channel 1 + 2 + 3.The same four-channel intersection underlies each input configuration, although each supplies a different channel subset.Source: §4.1.2 · pp. 6–7

4.2 Methodology

  • Benchmarking compares ResNet-18 with ResNet-50 and evaluates full-resolution, resized, and cropped images across channels and model configurations.Full images are 512 × 512; resized and cropped inputs are 224 × 224, with cropping preserving original resolution.Source: §4.2 · p. 7
  • Experiments assess pressure regression primarily and wind regression for single-channel comparisons, separately for WP and AU, reporting mean RMSE and standard deviation over five runs.Each run uses the same training and evaluation partitions.Source: §4.2 · p. 7

4.3 Data Splitting

  • The primary SS-L2 split holds out the 2024 and 2025 seasons for evaluation and uses earlier seasons for training.The held-out subset is used to monitor performance and report results, while model parameters and weighting schemes are fitted using training data alone; Tables 4 and 5 show per-channel dataset, training, and evaluation sizes.Source: §4.3 · p. 7, Table 4 · p. 8, Table 5 · p. 8
  • Because SS-L2 fixes the held-out seasons, evaluation-set percentages vary across channels, while SQ-R reduces percentage differences but remains affected by channel-specific dataset sizes.Source: §4.3 · p. 8
  • Future split options include an SQ-L1 design excluding active cyclones and separate validation and test sets for choosing hyperparameters and training duration.The authors also plan to retain the same split scheme in future dataset updates, which can create overlap between successive evaluation periods.Source: §4.3 · p. 8

4.4 Optimizer selection

The authors revisited optimizer choice for Dataset V3 after observing convergence and stability issues with SGD in earlier benchmarks. Adam converged faster and more stably in the evaluated experiments, so subsequent experiments used Adam.

4.4 Optimizer selection

  • In V1 and V2 benchmarking, SGD showed slower convergence and greater learning-rate sensitivity, affecting training stability and reproducibility.Source: §4.4 · p. 8
  • Adam converges faster and more stably than SGD across the evaluated experiments.Source: §4.4 · p. 8
  • All subsequent experiments use Adam and report RMSE after 50 epochs, rather than the best value within those epochs, except in Appendix A.The fixed training duration maintains a consistent evaluation protocol despite Adam generally converging well before 50 epochs.Source: §4.4 · p. 8

4.5 Benchmarking Single-Channel Models

Single-channel benchmarks compare pressure and wind regression across WP and AU, varying model architecture, spectral channel, and image type. Performance patterns differ by task and dataset, with ResNet-18, Channel 1, and full images generally strongest for pressure regression.

4.5 Benchmarking Single-Channel Models

  • The single-channel benchmark evaluates pressure on WP and AU and includes wind regression comparisons, with results reported as mean RMSE and standard deviation over five runs.The experimental setting is summarized in Table 6.Source: §4.5 · p. 8, Table 6 · p. 9

4.5.1 Pressure Regression Task

  • ResNet-18 with Channel 1 and Full images gives the best pressure RMSE: 6.56 ± 0.06 hPa on WP and 7.20 ± 0.12 hPa on AU.Full images generally outperform resized or cropped inputs, while higher-numbered channels generally have higher errors; ResNet-50 is usually worse and less stable.Source: §4.5.1 · pp. 9–10, Table 7 · p. 10, Table 8 · p. 10

4.5.2 Wind Regression Task

  • For wind regression, WP’s lowest RMSE is 15.67 ± 0.80 kt with ResNet-18, Channel 2, and Full images; AU’s is 10.75 ± 0.17 kt with ResNet-50, Channel 1, and Full images.On WP, Channel 2’s result is similar to Channel 1’s 15.75 ± 1.09 kt; AU sometimes favors ResNet-50, and Full images are generally strongest.Source: §4.5.2 · p. 10, Table 9 · p. 11, Table 10 · p. 11

4.5.3 Comparison between Single-Channel Models

  • WP has lower pressure RMSE, while AU has lower wind RMSE, showing that benchmark performance differs by target variable.For pressure, the best reported RMSEs are 6.56 hPa on WP and 7.20 hPa on AU; for wind, the best are 15.67 kt on WP and 10.75 kt on AU.Source: §4.5.3 · p. 11

4.6 Benchmarking Multi-channel Models

Multi-channel pressure benchmarks use a shared four-channel intersection dataset to compare single- and multi-channel inputs. The tested combinations show comparable performance rather than a clear advantage for multi-channel inputs.

4.6 Benchmarking Multi-channel Models

  • Multi-channel inputs do not clearly improve pressure regression: WP’s best combination is 6.8308 hPa versus 6.9270 hPa for Channel 1, while AU’s best multi-channel result, 8.4443 hPa, exceeds Channel 1’s 8.2839 hPa.The WP difference is comparable to run-to-run variability; across both datasets, gains from combining channels are marginal.Source: Table 11 · p. 12, Table 12 · p. 12, §4.6 · pp. 12–13
  • The four-channel intersection enables matched single- and multi-channel comparisons, but filtering reduces dataset size and temporal coverage.The benchmark uses ResNet-18, Full images, and SS-L2 with 2024–2025 as the evaluation seasons.Source: §4.6 · pp. 11–12
  • Channel 1 and Channel 2 perform similarly on the intersection dataset, while Channel 4 is worst on AU and, with Channel 3, on WP.The reported RMSEs for Channels 1 and 2 are 6.93 and 6.96 hPa on WP, and 8.28 and 8.24 hPa on AU.Source: §4.6 · p. 12

4.7 Trade-off between More Spectral Data and Smaller Dataset Size

The multi-channel intersection dataset enables spectral-combination experiments but retains observations only when all four channels are available. Its reduced size and coverage accompany worse single-channel benchmark results for several comparisons.

4.7 Trade-off between More Spectral Data and Smaller Dataset Size

  • On the original versus intersection datasets, Channel 1 RMSE rises from 6.56 to 6.93 hPa on WP and from 7.20 to 8.28 hPa on AU.WP-1 has 202,986 observations in the single-channel dataset versus 96,579 in the multi-channel dataset; WP-4 differs little because Channel 4 constrains intersection coverage.Source: §4.7 · p. 13
  • The multi-channel intersection retains only observations with all four channels available, reducing dataset size and temporal coverage and creating a trade-off with spectral completeness.The resulting coverage reduction must be considered when interpreting multi-channel benchmark results.Source: §4.7 · p. 13
  • Channel 1, like infrared Channel 2, is more informative than the other channels, but adding channels does not clearly or consistently improve performance on either dataset.The results should not be taken to mean additional spectral information is intrinsically uninformative; channel combinations using arithmetic operations were not explored.Source: §4.7 · p. 13
  • The study concludes that reducing dataset size for multi-channel inputs cannot be compensated by their performance advantage under the tested setting.The observed WP Channel 1 + 2 + 3 result is comparable to, rather than a clear improvement over, Channel 1.Source: §4.7 · p. 13

4.8 Key Benchmarking Findings

The benchmark finds that optimizer choice, single-channel configurations, target variable, and dataset construction all shape reported performance. Under the tested setting, multi-channel pressure inputs do not clearly outperform the strongest single-channel configuration.

4.8 Key Benchmarking Findings

  • Adam converges faster and more stably than SGD and is used for all subsequent experiments.Source: §4.8 · p. 14
  • For single-channel pressure regression, ResNet-18 generally performs better and more stably than ResNet-50, with Channel 1 and Full images yielding 6.56 hPa on WP and 7.20 hPa on AU.Source: §4.8 · p. 14
  • For wind regression, the best reported WP result is 15.67 kt with ResNet-18 and Channel 2, while AU’s is 10.75 kt with ResNet-50 and Channel 1.Source: §4.8 · p. 14
  • WP has lower pressure RMSE, whereas AU has substantially lower wind RMSE, reflecting dataset- and task-specific characteristics.The best single-channel wind results are 15.67 kt on WP and 10.75 kt on AU.Source: §4.8 · p. 14
  • Channel 1 with Full-resolution images and ResNet-18 is the strongest and most consistent single-channel pressure configuration, while additional spectral inputs provide no clear performance advantage.The best multi-channel pressure RMSEs are 6.83 hPa on WP and 8.44 hPa on AU.Source: §4.8 · p. 14

5 Analyzing Systematic Bias in Intensity Regression

The section characterizes pressure-prediction errors across typhoon intensities and tests weighting strategies for rare or difficult pressure ranges. Weighting reduces bias magnitude but leaves an intensity-dependent pattern, while stronger weighting trades lower extreme-case bias for higher global RMSE.

5.1 Systematic Bias in Intensity Regression

  • The bias analysis used 47 overlapping consecutive-year validation splits, training without each target pair and evaluating on that pair.This sliding-window analysis differs from repeated runs on a fixed partition.Source: §5.1 · p. 15
  • Below 920 hPa, mean bias exceeded 10 hPa in extreme cases; predictions underestimated strong and overestimated weak typhoons, without establishing a single cause.Less than 1% of samples had pressures below 920 hPa; the authors describe the error pattern rather than attributing it to one cause.Source: §5.1 · p. 15, Figure 3 · p. 15

5.2 Sample Weighting

  • Adaptive weights used training-set per-bin RMSE, while the held-out 2024–2025 seasons were reserved for evaluation; results combine weighted loss with weighted sampling.Each weighting configuration used five runs on a fixed partition.Source: §5.2 · pp. 16–17
  • RMSE was 8.59 with 1/f weighting and 7.04 with 1/f 0.25; stronger weighting reduced extreme-case bias but worsened global performance.The reported weighting experiments combine weighted loss and weighted sampling rather than testing them independently.Source: Figure 4 · p. 17, §5.2 · p. 17

5.3 Inverse-Density Weighting Inspired by BalancedMSE

  • Global RMSE was 8.42 with inverse-density weighting, which reduced low-pressure bias; comparisons with Figure 3 are not direct because validation protocols differ.The implementation is a simpler inverse-frequency weighting strategy, not the original probabilistic BalancedMSE formulation.Source: §5.3 · pp. 17–18, Figure 5 · p. 18

5.4 Future Directions for Intensity Regression

  • Future work could better exploit existing observations through imbalance-aware or calibrated models and temporal or meteorological inputs; conventional image augmentation is considered inappropriate.The authors caution that transformations may alter physically meaningful structures or create unrealistic cyclone configurations.Source: §5.4 · pp. 18–19

6 Conclusion

Dataset V3 adds spectral data for multi-channel tropical-cyclone analysis, but multi-channel inputs do not clearly improve pressure regression in the current setting. Weighting reduces bias on extreme cases without eliminating regression-to-the-mean behavior.

6 Conclusion

  • Multi-channel inputs do not clearly improve pressure regression in the current setting, and missing-channel data may limit robust learning, especially for under-represented strong typhoons.The paper identifies both dataset constraints and model limitations as contributors to the difficulty of analyzing strong typhoons.Source: §6 · p. 19
  • Sample weighting and inverse-density weighting inspired by BalancedMSE reduce bias for extreme cases but do not fully eliminate regression-to-the-mean behavior.The paper suggests dataset constraints and model limitations both contribute to the difficulty of analyzing strong typhoons.Source: §6 · p. 19
  • Dataset development and modeling strategies provide a foundation for future work on tropical-cyclone intensity regression, particularly for extreme events.This conclusion follows the paper’s discussion of dataset constraints and persistent bias in strong-typhoon regression.Source: §6 · p. 19

A Comparison of V3-2025.R1 and V3-2025.R2

The V3-2025.R1 and R2 comparison tests how corrected training images affect benchmark results. Corrupted images strongly affect training error but have only a limited, roughly 0.1 hPa effect on generalization where detectable.

Setup

  • The comparison uses ResNet-18, full 512 × 512 images, an SS-L2 split, 50 epochs, and five runs; evaluation sets are identical across releases.Training sets differ only in corrected images and, for AU-1 and AU-2, 23 removed GOES-9 images; WP-2 was excluded because its 1,253 corrected images are almost the same observations as WP-3.Source: A Comparison of V3-2025.R1 and… · p. 22

Evaluation metric

  • The evaluation RMSE fluctuates through training and can spike for an AU dataset, so the authors recommend the median over the last ten epochs for future comparisons using this protocol.They also report the last-five-epoch median and epoch-50 value to show sensitivity to the metric choice.Source: A Comparison of V3-2025.R1 and… · p. 22

Results

  • R1 comparison runs were retrained, and their epoch-50 results differ from Tables 7 and 8 because of run-to-run variability and single-epoch spikes.For WP-1, the epoch-50 value is 6.66 hPa here versus 6.56 hPa in the earlier table.Source: A Comparison of V3-2025.R1 and… · p. 22
  • 2.0–2.9 hPa versus 1.3 hPa: training RMSE at epoch 50 is higher on V3-2025.R1 than R2 with 0.3–0.9% corrupted observations; AU-1 shows no difference at 0.06%.Blank images carry no information about their valid labels and dominate remaining training error when they cannot be fitted.Source: A Comparison of V3-2025.R1 and… · pp. 22–23
  • 0.10 hPa and 0.12 hPa: V3-2025.R2 improves the median evaluation RMSE over the last ten epochs for WP-1 and WP-3, respectively.The improvements are larger than variability across runs.Source: A Comparison of V3-2025.R1 and… · p. 23
  • −0.13, +0.04, and −0.06 hPa: AU-1, AU-2, and AU-3 evaluation differences fall within run variability, with signs depending on the metric.The AU-1 difference, with 39 corrupted observations, illustrates variation that can arise from run-to-run variability alone on the smaller AU evaluation set.Source: A Comparison of V3-2025.R1 and… · p. 23

Related papers

Loading 2610.21654v1…