Source-linked AI summary
Season-Aware Hybrid Convolutional-Transformer for Antarctic Sea Ice Concentration Forecasting
Danyang Li, John Taylor, Thang Bui, Quanling Deng
TL;DR
Antarctic SIC forecasting is difficult because it combines complex spatial structure, long-range temporal dependencies, and strong seasonal variability. The paper proposes a hybrid Convolutional-Transformer framework with seasonal priors, achieving competitive classification and regression performance while improving short- and long-range predictions.
Problem
Antarctic SIC forecasting is important for climate monitoring and downstream polar applications but remains challenging because of coupled processes, sharp ice-edge gradients, and a strong non-stationary seasonal cycle.
Method
The framework combines convolutional spatial feature extraction with Transformer spatio-temporal dependency modeling and adds Fourier-based month-aware positional encoding plus seasonal temporal bias attention.
Results
Competitive performance was achieved across classification and regression metrics, with seasonal priors providing consistent benefits in both short- and long-range prediction.
Takeaways & Limitations
Hybrid convolutional and attention-based models with periodic prior information show potential for Antarctic SIC forecasting.
Takeaways & Limitations
The attention maps provide only a qualitative view of learned dependencies, motivating systematic interpretability analyses across seasons, regions, and forecast horizons.
Abstract
from arXiv · showhide
Antarctic sea ice concentration (SIC) forecasting is an important yet challenging task due to the coexistence of complex spatial structure, long-range temporal dependencies, and strong seasonal variability. Conventional convolution-based models are effective at capturing local spatial patterns, but often have limited ability to model long-term temporal evolution. To address these challenges, we build on a hybrid Convolutional-Transformer forecasting framework for monthly Antarctic SIC forecasting. This framework combines convolutional encoding for spatial feature extraction with factorised self-attention for spatio-temporal dependency modelling. We further introduce two seasonal prior mechanisms: a month-aware positional encoding that injects calendar-month information into the token representation, and a seasonal temporal bias that encourages attention to periodically related historical states. Experimental results show that the proposed framework achieves better performance than convolutional and recurrent baselines across both classification and regression metrics. Ablation studies further indicate that the seasonal prior mechanisms provide consistent additional gains in both short- and long-horizon prediction. These results demonstrate the value of combining convolutional structures, attention mechanisms, and periodic prior information for Antarctic SIC forecasting.
1 Introduction
Antarctic SIC forecasting is important but difficult because of nonlinear coupled processes, spatial heterogeneity, long-range dependencies, and strong seasonal variability. The proposed framework combines convolutional spatial extraction, compressed Transformer mixing, and two seasonal prior mechanisms.
- Antarctic SIC prediction supports climate monitoring, polar logistics, and marine operations but remains challenging because of coupled processes, sharp ice-edge gradients, and a non-stationary seasonal cycle.
- Physics-based forecasting is physically consistent but computationally expensive and demanding to calibrate and assimilate.
- Traditional statistical models are efficient but may inadequately represent nonlinear sea-ice interactions under extreme or rapidly changing conditions.
- CNNs learn spatial SIC features effectively but lack explicit mechanisms for modelling long-range temporal dependencies.
- The proposed Convolutional-Transformer uses a convolutional encoder-decoder and compressed Transformer bottleneck for feasible global spatio-temporal mixing.
- MonthPE factorizes spatial, temporal, and Fourier-based seasonal embeddings, while SeasonBias modulates temporal attention using circular annual-cycle distance.
2 Problem Statement and Data
The study formulates Antarctic SIC forecasting as deterministic sequence-to-sequence prediction from historical monthly maps to future SIC fields. It uses masked spatial inputs and dual continuous-classification outputs to support both regression assessment and autoregressive forecasting.
- Antarctic SIC forecasting is framed as learning a neural mapping from T historical monthly maps to K future monthly SIC fields.
- A binary land mask restricts prediction to valid ocean pixels because Earth-system geography is fixed.
- The dataset is the NSIDC Sea Ice Index Version 3, based on multi-decadal passive-microwave observations mapped at 25×25km2 resolution.
- Each forecast step produces a continuous normalized SIC field for skill evaluation and autoregressive rollout.
- The classification target assigns valid pixels to Open Water, Marginal Ice, or Full Ice using SIC thresholds below 15%, from 15% to 80%, and above 80%.
- Training uses T = 12 monthly maps to predict up to K = 12 future months, with binary ice extent additionally evaluated using the conventional 15% threshold.
3 Transformer and Overall Framework
The framework combines convolutional spatial encoding and decoding with a compressed, factorized Transformer bottleneck for efficient spatio-temporal SIC forecasting. Month-aware positional information and a learnable seasonal attention bias encode annual periodicity in token representations and temporal interactions.
- Spatio-Temporal Tokenization: A frame-wise CNN encoder extracts local SIC morphology and produces compact bottleneck features before Transformer processing.The encoder is applied independently to each monthly frame with shared weights.
- Spatio-Temporal Tokenization: Convolutional tokenization reduces sequence length from THW to TS, where S ≪ HW, enabling efficient factorized attention while retaining local spatial structure.Temporal and spatial attention are applied sequentially rather than as global attention over all spatio-temporal tokens.
- Month-Aware Positional Encoding: MonthPE factorizes positional information into spatial, temporal, and Fourier-based calendar-month embeddings instead of using an intractable 3D lookup table.The resulting month embedding is added to every spatial token at its timestep and represents a 12-month periodic prior.
- Separable Spatio-Temporal Attention: Seasonal bias adds a learnable circular-distance prior to temporal attention logits, directly shaping query-key interactions between calendar months.Positive α favors seasonally aligned months, while negative α permits the model to reverse that preference; α is learned per temporal attention layer.
- Forecast Reconstruction: The decoder reshapes latent tokens back to spatial grids and reconstructs forecasts using either U-Net skip connections or a direct CNN decoder.Long-horizon prediction uses an autoregressive rollout, with multi-step training for K_train = 2 and evaluation up to K_eval = 12 months.
4 Experiments
The experiments compare hybrid Convolutional-Transformer models with baselines across multi-horizon SIC forecasting, ablations, spatial forecasts, and temporal attention. Results indicate that seasonal priors improve forecasting performance and produce more seasonally structured temporal attention.
- Overall Performance Comparison: The evaluation compares CNN, UNet, ConvLSTM, PredRNN, and Transformer-based variants using multi-horizon continuous and categorical SIC forecasting.Figure 2 covers a 12-month autoregressive horizon, while the experiments also examine short- and long-horizon ablations.
- Overall Performance Comparison: Hybrid Transformer models with seasonal priors generally outperform convolutional and recurrent baselines across the 12-month forecasting horizon.RMSE and classification accuracy both degrade with forecast lead time, but the proposed models remain competitive and generally achieve better performance.
- Overall Performance Comparison: For 2023 classification forecasts, Transformer models with MonthPE and SeasonBias better preserve seasonal ice-boundary evolution than recurrent and convolutional baselines.The differences are most visible near the marginal ice zone, where spatial variability makes classification more difficult.
- Ablation at Short- and Long-Horizon Forecast Steps: At step 1, combining MonthPE and SeasonBias gives the best performance for both CNN+Trans and UNet+Trans across all reported metrics.For CNN+Trans, the combined setting reduces RMSE from 0.0679 to 0.0653 and increases Cls3 accuracy from 0.9371 to 0.9420.
- Ablation at Short- and Long-Horizon Forecast Steps: At step 12, the seasonal priors generally improve long-horizon performance, although the UNet+Trans response varies by metric.For CNN+Trans, the combined setting improves Cls3 accuracy from 0.9036 to 0.9155 and reduces RMSE from 0.1169 to 0.1058; UNet+Trans does not select the combined setting for every metric.
- Temporal Attention: Attention visualisations show that MonthPE introduces month-dependent variation, while SeasonBias creates clearer diagonal and banded seasonal query-key structures.The combined model preserves periodic structure while allowing attention weights to vary with the target month, consistent with the proposed attention design.
5 Conclusions
The paper presents a hybrid Convolutional-Transformer framework for Antarctic SIC forecasting that combines spatial representation learning, spatio-temporal dependency modeling, and seasonal priors. Results indicate consistent benefits from seasonal information, while future work includes physical knowledge integration and broader interpretability analysis.
- The framework combines convolutional spatial representation learning with Transformer-based spatio-temporal dependency modeling for Antarctic SIC forecasting.
- Fourier-based positional encoding and seasonal bias attention provide consistent benefits in both short- and long-range prediction.
- The current framework does not explicitly use physical knowledge such as thermodynamic constraints, sea-ice dynamics, or coupled ocean-atmosphere information.
- The proposed design may extend to Earth science and time-series forecasting problems with strong seasonal or cyclic behavior.
- Future interpretability analyses could compare attention patterns across seasons, regions, and forecast horizons to assess reliance on physically meaningful information.
Network Structure and Parameters
The proposed network uses a convolutional encoder, a Transformer bottleneck, and a decoder to produce full-resolution regression and classification forecasts. Experiments standardize the input-output setting and training protocol across compared models, while bottleneck modules differ in temporal modeling.
- Network Structure: The convolutional encoder extracts local spatial features and reduces resolution before spatio-temporal tokens enter the Transformer bottleneck.
- Network Structure: Positional encodings and factorized self-attention model temporal and spatial dependencies at the bottleneck.
- Network Structure: The decoder upsamples bottleneck features to the original spatial resolution and produces regression and classification outputs.
- Experimental Setup: All compared models use the same input resolution, historical window, forecast horizon, loss functions, and training protocol unless otherwise specified.
- Bottleneck Comparisons: Pure CNN and UNet baselines use convolutional bottlenecks, whereas ConvLSTM and PredRNN introduce recurrent temporal modules.
- Architecture Parameters: The architecture tables specify T = 12, H = W = 352, S = 44 × 44, and Cin = 256 for the proposed UNet-Transformer and CNN-Transformer models.
Additional ablation
Additional ablations examine SeasonBias strength and historical input length. Moderate SeasonBias is robust and performs best among tested settings, whereas extending the input window does not clearly improve forecasting.
- SeasonBias Strength: α = 0.12 achieves the best overall step-12 performance among tested SeasonBias strengths on the UNet+Trans backbone with MonthPE.
- SeasonBias Strength: The model is relatively robust to SeasonBias strength, with only moderate performance variations across the tested range.
- Input Window: Increasing the input window from 12 to 24 months does not provide a clear overall improvement at step 1 and step 12.
- Input Window: The longer-window results suggest that extending historical context alone may not be beneficial without more effective selection or weighting of temporal information.
Additional experiments results
Qualitative comparisons examine twelve-step classification forecasts initialized in June and December 2023. All methods reproduce major sea-ice distributions, while differences are most visible near the ice edge.
- The qualitative comparisons cover multi-step classification forecasts initialized in June and December 2023 across forecast steps 1–12.
- All methods reproduce the major large-scale sea-ice distribution, with visual differences most apparent near the ice edge.
- Each comparison organizes forecast lead times by columns and ground truth or forecasting methods by rows.
- The three classification classes are open water, marginal ice, and full ice.