Source-linked AI summary
Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts
Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett, Farnoush Banaei-Kashani
TL;DR
Sea-ice stage-of-development charts contain disagreement among expert annotators, while learned models introduce additional predictive uncertainty. This paper measures both sources, compares supervision and uncertainty-estimation strategies, and finds that soft supervision best aligns model uncertainty with disagreement overall, especially near the ice edge, while Monte Carlo dropout is best calibrated.
Problem
Operational ice charts are commonly treated as exact training targets even though independent ice services may assign different stage-of-development labels to the same observation.
Method
The study measures multi-annotator disagreement with first Wasserstein distance on the ordinal stage ladder, compares five supervision strategies, and evaluates predictive, ensemble, dropout, evidential, and conformal uncertainty.
Results
Soft supervision achieves the highest overall entropy–disagreement correlation, ρ(H, D) = 0.256, while Monte Carlo dropout achieves the lowest expected calibration error, 0.050.
Takeaways & Limitations
Model uncertainty tracks multi-annotator disagreement most strongly near the ice edge, reaching 0.704 within 0–10 km for most supervision strategies.
Takeaways & Limitations
The results come from a single region and season, one architecture, and one calibration split, so supervision rankings may not transfer to other settings.
Abstract
from arXiv · showhide
Sea-ice stage of development (SoD) describes the age and associated thickness of sea ice and provides important information for navigation, and operational ice monitoring. SoD labels are obtained from operational ice charts, where trained analysts interpret satellite observations and assign standardized stage codes to regions with similar ice conditions. These codes often represent ranges of compatible ice thicknesses rather than exact physical values. Deep-learning methods can automate SoD mapping and commonly adopt operational ice charts as reference labels for training. These annotations are not exact, however; this is because chart interpretation relies on analyst judgement and on the observations available at the time, so different ice services may assign different SoD labels to the same conditions. We term this variation across independently produced expert annotations multi-annotator label uncertainty; collapsing the annotations into a single deterministic target discards this variation. A second source of uncertainty originates in the learned model itself. In this paper, we quantify both sources: annotation uncertainty from disagreement among independent ice-service charts and model uncertainty from the learned predictive models. We then evaluate their relationship by testing whether model uncertainty is higher where ice services disagree. We observe that supervision incorporating information from multiple annotators can improve this correspondence, with soft supervision achieving the highest overall correlation of 0.256. The relationship becomes substantially stronger near the ice edge, where model predictive uncertainty closely tracks multi-annotator disagreement, reaching a correlation of 0.704 within 0--10 km. Among the uncertainty-estimation approaches, Monte Carlo dropout provides the best-calibrated confidence estimates, with an expected calibration error of 0.050.
1 Introduction
Automated SoD mapping commonly treats operational ice-chart labels as exact ground truth, although independent ice services may assign different labels to the same observations. The study therefore quantifies annotation and model uncertainty and evaluates how they correspond.
- Motivation: Independent ice services can assign different SoD labels to the same observation because charting relies on ambiguous imagery, environmental context, and expert judgement.This disagreement is termed multi-annotator label uncertainty.
- Motivation: Deterministic targets can turn legitimate interpretation differences into apparent label noise and obscure the distinction between prediction error and reference-data ambiguity.They may also encourage unjustified confidence.
- Approach: The framework compares five supervision strategies and four model-uncertainty forms while measuring disagreement directly from independently produced ice charts.A direct disagreement-regression control estimates how much disagreement is recoverable from observations.
- Contributions: The dataset contains 404 Sentinel-1 scenes independently charted by at least two of four national ice services.The services are NIC, DMI, CIS, and NOAA.
2 Related Work
Prior work has advanced automated sea-ice classification and model-centered uncertainty estimation, while independent studies show that expert-generated ice charts also vary. This motivates treating reference-label uncertainty as a distinct consideration in automated SoD mapping.
- Prior approaches: Deep-learning approaches increasingly automate sea-ice classification and ice-chart generation from SAR and multisource observations, usually using operational charts as reference labels.These labels therefore carry uncertainty relevant to model training.
- Research gap: Existing uncertainty studies primarily characterize predictive-model or output uncertainty, whereas separate work documents disagreement among analysts and ice services producing charts from the same SAR observations.The two uncertainty sources are therefore addressed in different research strands.
- SoD representation: Table 1 organizes SIGRID-3 stage codes into thickness-based SoD representations.The supplied passages identify the table’s scope but do not provide its cell values.
3 Methodology
The methodology preserves uncertainty in operational SoD annotations while comparing supervision strategies and model-uncertainty estimates. It represents charted ice compositions on an ordered thickness ladder, measures inter-service disagreement, and tests whether predictive uncertainty tracks it.
- SoD label representation: Each operational chart represents polygon-level ice composition using partial concentrations and stage codes harmonized onto a common, thickness-ordered SoD ladder.Open-water proportions are removed because the task is conditioned on the presence of ice.
- SoD label representation: A pixel inherits the complete SoD composition of its chart polygon, while independently produced service charts can differ through polygon delineation, composition, and stage assignment.This preserves both within-chart ranges and cross-service variation.
- Supervision strategies: Hard supervision keeps one dominant stage, consensus averages services before selecting a dominant stage, and soft supervision retains the complete cross-service mean composition.Soft supervision preserves relative stage proportions, including differences among services.
- Disagreement measurement: Multi-annotator disagreement is measured with the first Wasserstein distance on the ordered SoD ladder, so neighboring-stage disagreements count less than distant-stage disagreements.The resulting D(x) is used as a continuous disagreement measure.
- Supervision strategies: Disagreement-weighted supervision uses w(x)=1/(1+D(x)), giving greater training weight to pixels on which services agree.The approach tests whether contested annotations should contribute less to training.
- Model uncertainty: Model uncertainty is evaluated through predictive entropy, deep ensembles, Monte Carlo dropout, evidential modeling, conformal prediction sets, and a direct disagreement-regression control.Entropy and epistemic estimates are compared with measured disagreement, while conformal sets are assessed for coverage and size relationships.
4 Experimental Evaluation
The evaluation uses a multimodal 2025 dataset of 404 co-charted Sentinel-1 scenes from four ice services, then compares supervision strategies and uncertainty measures against inter-service disagreement. Results show that preserving annotation distributions improves uncertainty alignment, especially near the ice edge, while Monte Carlo dropout provides the best calibration.
- Data and setup: The multimodal dataset contains 404 Sentinel-1 scenes independently charted by at least two of NIC, DMI, CIS, and NOAA.The experiments use SAR alone or combined with AMSR2 observations and harmonize service-specific Egg Codes onto a common SoD ladder.
- Data and setup: Service annotations differ beyond total concentration: 57.7% of pixels with agreeing concentration codes have incompatible stage compositions.Their mean composition Wasserstein distance is five times that of fully agreeing cells, motivating explicit treatment of stage disagreement.
- Results: Soft supervision achieves the highest overall entropy–disagreement correlation at 0.256, while Monte Carlo dropout attains the lowest expected calibration error at 0.050.The deterministic consensus target performs best on point and ordinal accuracy, but soft supervision provides the strongest uncertainty alignment.
- Results: Adding AMSR2 strengthens entropy–disagreement alignment for consensus from 0.193 to 0.253 and soft supervision from 0.141 to 0.256, but adds little to SoD class discrimination.The brightness-temperature channels do not consistently improve prediction across supervision strategies.
- Results: Conformal set size tracks measured disagreement positively, reaching ρ=0.315 for consensus, although hard and weighted coverage fall below the nominal level.At α=0.1, empirical coverage is 0.875 for hard supervision and 0.883 for weighted supervision.
- Results: Predictive entropy aligns most strongly with disagreement near the ice edge, reaching 0.52–0.70 within 10 km for most SAR supervision strategies.The relationship generally declines farther into the ice pack, with hard supervision as the main exception.
5 Conclusion and Limitations
The study compares multi-annotator label uncertainty with model uncertainty in sea-ice stage mapping, finding their strongest alignment near the ice edge and best calibration from Monte Carlo dropout. These findings are qualified by limits in geographic, seasonal, architectural, annotation, and evaluation scope.
- 0.704 correlation was reached within 0–10 km of the ice edge, compared with a maximum of 0.256 across the full dataset.The SAR consensus model showed the strongest near-edge alignment between predictive uncertainty and expert disagreement.
- Monte Carlo dropout achieved the best calibration among evaluated uncertainty methods, with ECE = 0.050.
- The results quantify multi-annotator uncertainty from independent national ice-service charts and compare it with predictive, ensemble, dropout, evidential, and conformal model uncertainty.Inter-annotator disagreement is measured with the first Wasserstein distance on the ordinal stage ladder.
- The conclusions are limited by a single region and season, one architecture and calibration split, coarse service-based disagreement, polygon-level labels, and fixed thickness mappings.These choices may constrain transferability and affect how disagreement and thickness error are interpreted.