Source-linked AI summary
Uncertainty-Aware Learning from Multi-Expert Interval Targets
Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett, Farnoush Banaei-Kashani
TL;DR
Expert labels can encode multiple plausible interpretations, while interval annotations add within-label imprecision to variation across experts. This work harmonizes label vocabularies, retains expert intervals in a mixture-of-Betas predictor, and evaluates decomposed uncertainty; on sea-ice concentration it reduces MAE by 31% versus hard labels with SAR+AMSR2.
Problem
Expert labels can reflect multiple plausible interpretations, and interval-valued labels contain within-label uncertainty alongside variation across experts.
Source: §I · pp. 1–2Method
The framework harmonizes heterogeneous label vocabularies, retains individual expert intervals in a Mixture-of-Betas predictor, and evaluates decomposed predictive uncertainty against label-side sources.
Source: §I · p. 2Results
31% lower MAE with SAR+AMSR2 relative to hard supervision on sea-ice concentration.
Source: §V · p. 9Takeaways & Limitations
Predictive uncertainty components do not automatically align with their intended sources, while decomposition matching improves alignment between between-component and between-label uncertainty.
Source: §V · p. 9
Abstract
from arXiv · showhide
Many machine learning (ML) applications rely on expert labels, and qualified experts may provide different but plausible interpretations of the same observation. Such variation across expert labels may reflect genuine disagreement or ambiguity rather than annotation error. When individual experts additionally report intervals rather than exact values, the supervision contains two distinct sources of label uncertainty: within-label imprecision and between-expert variation. Existing methods treat these forms separately: multi-expert approaches collapse labels to a consensus, interval-target methods often yield a single prediction, and predictive-uncertainty methods rarely validate their uncertainty estimates against observed expert disagreement. To address this problem, we propose an approach that preserves individual expert intervals, separates within-label imprecision from between-expert variation, and validates the corresponding predictive uncertainty components. First, heterogeneous label vocabularies are harmonized into a common probabilistic label space, separating encoding differences from expert judgement. Second, individual label intervals are retained and modeled with a mixture of Beta distributions trained using a proper Cram\'er-distance objective, preserving distinct expert-reported labels. Third, we decompose predictive uncertainty into within-component, between-component, and model uncertainty, and evaluate whether these components correspond to within-label uncertainty, between-label uncertainty, and model error, respectively. Because this correspondence is not guaranteed, we introduce decomposition matching, which aligns the predictive components to their intended label-side sources. On sea-ice concentration the model reduces MAE by 31\% over hard labels and outperforms aggregation, interval-distribution, and interval-regression baselines.
I. INTRODUCTION
Expert labels can encode genuine disagreement, ambiguity, and interval imprecision, creating distinct label-side uncertainty alongside model uncertainty. The paper preserves individual interval labels and tests whether predictive uncertainty components correspond to these sources.
I. INTRODUCTION
- Expert intervals create within-label imprecision, while differences among experts create a separate between-label uncertainty that should not be collapsed with model uncertainty.Existing approaches typically address only parts of this setting, without jointly distinguishing and validating all three sources using observed multi-expert interval labels.Source: §I · pp. 1–2
- The framework harmonizes heterogeneous label vocabularies, retains individual intervals in a Mixture-of-Betas target, and trains with a proper Cramér-distance objective.It evaluates predictive within-component, between-component, and model uncertainty against label imprecision, expert variation, and model error, respectively.Source: §I · p. 2
- The study uses independently produced operational ice charts to examine three uncertainty sources across 404 satellite scenes charted by up to three ice services.Sea-ice concentration estimates can differ across analysts under intermediate concentrations, marginal ice, poorly resolved floes, and melt-season imagery.Source: §I · p. 2
- Decomposition matching explicitly aligns predictive uncertainty components with their intended label-side sources because the correspondence does not emerge reliably on its own.The introduction identifies between-component uncertainty as recovering between-expert variation reliably only under explicit matching.Source: §I · p. 2
II. RELATED WORK
Prior work variously resolves expert disagreement, preserves label variation, learns from interval targets, or estimates predictive uncertainty. This paper combines multiple interval labels with uncertainty decomposition and examines the components against observed label variation.
A. Learning from Multiple Experts
- Consensus and reliability methods model expert differences but generally seek a single latent target, whereas disagreement-preserving methods capture variation but usually use point-valued annotations.This work instead retains multiple expert intervals and evaluates whether predictive components recover both within-label imprecision and between-expert variation.Source: A. Learning from Multiple Experts · p. 2
B. Learning from Interval Targets
- Interval-target methods generally use a single interval, while this setting retains multiple expert intervals to separate within-label imprecision from between-expert variation.Earlier approaches include interval regression and interval-censored regression; regression with interval targets constrains predictions using observed bounds,.Source: B. Learning from Interval Targets · pp. 2–3
C. Probabilistic Prediction and Uncertainty
- Bayesian approximations, deep ensembles, and evidential methods estimate predictive or model uncertainty, but that uncertainty does not itself measure observed expert variation.A single Beta distribution cannot represent separated modes, motivating the paper's use of mixture models for bounded targets.Source: C. Probabilistic Prediction and… · p. 3
D. Uncertainty in SIC Estimation
- Earlier sea-ice concentration work estimates predictive uncertainty from model outputs, whereas this paper preserves independent expert intervals and checks predictive uncertainty against observed label sources.Reported differences in sea-ice assessments vary with concentration, location, and season, with larger differences at intermediate concentrations.Source: D. Uncertainty in SIC Estimation · p. 3
III. METHODOLOGY
- The methodology harmonizes heterogeneous label codes, constructs a probabilistic target retaining both forms of label uncertainty, and decomposes predictive uncertainty for comparison with label uncertainty and model error.The stages are described as harmonization, target construction, and predictive-uncertainty decomposition.Source: §III · p. 3
A. Problem Formulation and Label Harmonization
The formulation represents each expert's code as an interval in a shared ordered label space, then harmonizes intervals for comparison while retaining the original intervals for learning and uncertainty measurement.
A. Problem Formulation and Label Harmonization
- Each instance is a pixel with input features, and each expert code is decoded through an expert-specific map into an interval on the shared label space [0, 1].Equal interval bounds represent an exact-valued label, and the observed label set contains the intervals from the experts available for that instance.Source: A. Problem Formulation and Label… · p. 3
- Because expert vocabularies and quantization conventions differ, each interval is treated as uniform and discretized onto a common grid to form a comparable label distribution.For nonzero-width intervals, bin mass is proportional to overlap length; an exact label assigns all mass to its containing bin.Source: A. Problem Formulation and Label… · p. 3
- The harmonized distributions support comparison across coding conventions, while original intervals remain available for distributional learning and label-uncertainty measurement.Between-label uncertainty reflects variation among experts' labels for the same instance.Source: A. Problem Formulation and Label… · p. 3
- Between-expert variation is quantified by mean pairwise Wasserstein-1 distance, but is undefined when only one expert labels an instance.Within-label uncertainty is the average variance induced by interval widths; between-label uncertainty is the variance of interval centers, normalized for the equal-weight mixture.Source: A. Problem Formulation and Label… · pp. 3–4
B. Distributional Learning and Uncertainty Decomposition
The method retains each expert interval as a distribution and fits a mixture predictor to the full target, then decomposes predictive variability to assess alignment with label-side uncertainty and model error.
B. Distributional Learning and Uncertainty Decomposition
- Each expert-reported interval is represented as a uniform distribution, and the target averages the distributions while retaining overlapping or separated expert support.Exact labels reduce to point masses; conflicting intervals are not merged into one interval.Source: B. Distributional Learning and… · p. 4
- A mixture of Beta distributions models broad or separated probability regions over the bounded label space [0, 1].A single Beta distribution cannot represent separated high-probability regions without assigning probability between them.Source: B. Distributional Learning and… · p. 4
- The predictive CDF is trained against the full target CDF using Cramér distance, an objective equivalent to expected CRPS up to a target-dependent term.This trains the model to reproduce the full target distribution rather than only its mean.Source: B. Distributional Learning and… · p. 4
- Cramér loss constrains only the overall predictive distribution, so component-to-label correspondence is not guaranteed without decomposition matching.Decomposition matching adds auxiliary losses to align within- and between-component uncertainties with their label-derived counterparts; between-label matching applies where at least two labels are available.Source: B. Distributional Learning and… · pp. 4–5
- Predictive variance is decomposed into within-component, between-component, and model variability, with model variability measured across independently trained ensemble members.The components are evaluated against within-label uncertainty, between-label uncertainty, and prediction error, respectively.Source: B. Distributional Learning and… · p. 5
IV. EXPERIMENTAL EVALUATION
The section introduces the experimental evaluation; its supplied figure depicts one Sentinel-1 scene with sensor inputs, agency SIC charts, and pairwise label differences.
IV. EXPERIMENTAL EVALUATION
- Fig. 1 combines HH and HV backscatter with AMSR2 brightness temperatures for one Sentinel-1 EW scene in Davis Strait.The scene is dated 2025-12-26.Source: Figure 1 · p. 5
- The figure shows CIS, DMI, and NIC SIC charts at the midpoint of each agency's reported interval.Source: Figure 1 · p. 5
- Mean pairwise W1 distance among agency labels is concentrated near the marginal ice zone in the depicted scene.Source: Figure 1 · p. 5
A. Dataset
The dataset pairs satellite observations with independent operational ice charts, harmonizes agency vocabularies, and uses the resulting interval labels to characterize expert variation.
A. Dataset
- Agency SIC vocabularies are harmonized to a common 11-bin representation before expert reports are compared.The comparison excludes one DMI–NIC scene-pair without shared valid pixels.Source: A. Dataset · p. 5
- 0.80–0.89 tenths are the mean W1 distances for DMI–NIC, CIS–NIC, and NIC–NOAA; CIS–NOAA shows closer agreement.The comparison is across shared pixels after label harmonization.Source: A. Dataset · p. 6, Table I · p. 6
- Approximately 83% of target variance is between experts and 17% is within individual labels.These provide label-side references for evaluating the model uncertainty decomposition.Source: A. Dataset · p. 6
- 404 scenes contain matched satellite observations and independent agency charts, with scenes rasterized to a common Sentinel-1 grid for pixelwise comparison.The dataset includes NIC, DMI, CIS, and NOAA; 364 scenes have charts from two agencies and 40 from three.Source: A. Dataset · pp. 5–6
B. Experimental Setup
The experimental setup holds the architecture and optimization schedule fixed across methods, selects models by validation MAE, and evaluates on held-out scenes.
B. Experimental Setup
- All methods use the same U-Net architecture, crop sampling, optimization schedule, and random seeds, without method-specific hyperparameter tuning.Training uses random 256 × 256 crops for 30 epochs of 400 iterations, with batch size 16.Source: B. Experimental Setup · p. 6
- Fig. 2 summarizes inter-agency SIC differences across 404 scenes by distance to the ice edge and by month for DMI–NIC and NIC–NOAA.Source: Figure 2 · p. 6
- Model selection uses the lowest validation MAE relative to the consensus midpoint, and three random-seed models also serve as deep-ensemble members.The scene split is 282 training, 41 validation, and 81 test scenes.Source: B. Experimental Setup · p. 6
C. Baselines
The baselines differ in whether they aggregate expert labels, retain intervals, or represent multiple plausible predictions. Distribution-matching controls isolate the effect of using a mixture and decomposition matching.
C. Baselines
- Label-aggregation baselines train on an 11-bin prediction, using a reference expert midpoint, mean expert midpoint, or averaged harmonized distributions.These are the Hard, Consensus, and Soft baselines, respectively.Source: C. Baselines · p. 6
- Interval-distribution baselines retain expert intervals but use one continuous Beta distribution, averaging interval losses or spanning experts with a consensus range.The two variants are AvgNLL and Cons.Source: C. Baselines · p. 6
- Deterministic interval-regression baselines produce point predictions, either penalizing predictions outside intervals or minimizing worst-case squared error.These are LIM and MM, respectively.Source: C. Baselines · p. 6
- Distribution matching compares K = 1 with K = 2 mixtures; a second K = 2 variant adds decomposition matching with λ = 1.The K = 1 control shares the multi-expert target and Cramér objective but has no between-component variation.Source: C. Baselines · p. 6
D. Metrics
The metrics assess point accuracy, distributional fit, uncertainty alignment, and selective prediction. Results show that decomposition matching improves benchmark accuracy and source alignment, while transfer and error-ranking performance depend on the setting and uncertainty objective.
D. Metrics
- MAE measures predictive-mean error against the consensus midpoint, while entropy and expected-CRPS assess probabilistic predictions; Spearman correlations and E-AURC assess uncertainty alignment and selective prediction.E-AURC compares retained error with an oracle that retains predictions in increasing order of actual error.Source: D. Metrics · p. 7
E. Main Results
- 19× lower CRPS: the two-component mixture preserves separated expert modes and reduces unsupported gap mass from 0.61 to 0.20.The target CDF has a flat plateau across the unsupported region, which only the mixture reproduces.Source: E. Main Results · p. 7, Figure 3 · p. 7
- 0.402 is the K=2+decomp predictive-width correlation with W1, versus 0.200 for plain K=2; uncertainty aligned with label variation is less effective for ranking errors.AvgNLL has the lowest E-AURC (0.082), while distribution-matching models have E-AURC ≥0.204; lower E-AURC indicates better selective prediction.Source: E. Main Results · pp. 7–8, Table III · p. 8
- 0.559 MAE with SAR+AMSR2 is the table’s lowest, versus 0.567 for K = 2; with SAR alone, decomposition matching reaches 1.230, the lowest SAR-only error.The K = 2 model improves over K = 1 with SAR+AMSR2 (0.567 vs. 0.709) but underperforms it with SAR alone (1.714 vs. 1.484).Source: E. Main Results · p. 7, Table II · p. 8
- Under temporal transfer, decomposed K=2 reaches 0.562 ±.016 MAE and ρb = 0.528; under cross-agency transfer, MAE is 0.684 versus 0.727, but alignment correlations drop.Cross-agency ρtot ≤0.150 and ρb ≤0.197; point predictions transfer to unseen agency pairs, while uncertainty channels reflect training-agency annotation conventions.Source: Table V · p. 8, E. Main Results · p. 8
- Decomposition improves between-component alignment with between-label uncertainty (0.500) and direct disagreement (0.396), but cross-correlations remain.For plain K=2, between-component uncertainty instead tracks error (0.635); with decomposition that error correlation falls to 0.233.Source: E. Main Results · p. 8, Table IV · p. 8
- The irreducible MAE floor is highest within 0–10 km of the ice edge (1.339) and falls to 0.180 with no edge present; pooled Floor/MAE rises from 38% to 56%.The proposed model has lower MAE than hard supervision in every stratum; near the edge, both models are already close to the floor (59% and 61%).Source: Table VI · p. 9, E. Main Results · p. 9
- Under synthetic noise, both variants remain at ceiling correlation across noise levels, but the noise control is near zero; decomposition shifts variance share upward as σs rises.The between-component share increases with σs from 0.832–0.949 with decomposition, versus a decrease from 0.783–0.618 for plain K=2; the share is meaningful only alongside ρ(sb, s).Source: Table VII · p. 9, E. Main Results · p. 9
1) Sensitivity Analysis:
The sensitivity analyses select K=2 as a useful mixture size and show that decomposition weights affect alignment of the uncertainty components.
Sensitivity Analysis
- MAE improves from 0.709 to 0.567 from K=1 to K=2, then saturates; CRPS is best at K=2 (0.0243).For K ≥2, Keff ≈1.1 indicates little use of additional components, and ρerr peaks at K=2 (0.635); the study therefore uses K=2 throughout.Source: Table VIII · p. 9, 1) Sensitivity Analysis · p. 9
- At (1, 1), ρb=0.500 and ρw=0.677 with MAE 0.559 versus 0.567 without supervision; over-weighting (3, 3) raises MAE to 0.663.λb alone raises ρb from 0.107 to 0.491 at (0, 1), while supervising ub alone lowers within-label alignment from 0.667 to 0.454.Source: Table IX · p. 9, 1) Sensitivity Analysis · p. 9
V. CONCLUSION
The framework learns from multiple expert interval labels while separating label-side uncertainty sources from predictive model uncertainty. On sea-ice concentration, it improves MAE over hard supervision and shows that predictive components do not automatically align with their intended sources.
V. CONCLUSION
- The Mixture-of-Betas model reduces MAE by 45% with SAR alone and 31% with SAR+AMSR2 relative to hard supervision, outperforming single-component and deterministic interval-target baselines.Decomposition matching substantially improves between-component/between-label alignment and width–W1 correlation without degrading point accuracy.Source: §V · p. 9
- Predictive uncertainty components do not automatically align with their intended label-side sources, and uncertainty aligned with between-expert variation and model error remains complementary.Decomposition matching improves between-component and between-label alignment and increases width–W1 correlation without degrading point accuracy.Source: §V · p. 9
- Future work includes spatially structured uncertainty, semi-supervised decomposition with multi-expert labels for only some scenes, and expert-conditioned modeling of agency-specific disagreement.The authors also identify jointly representing between-expert variation and model error within one predictive distribution as a future direction.Source: §V · p. 10