Source-linked AI summary
TwinTrack: Post-hoc Multi-Rater Calibration for Medical Image Segmentation
Tristan Kirscher, Alexandra Ertl, Klaus Maier-Hein, Xavier Coubez, Philippe Meyer, Sylvain Faisan
TL;DR
Ambiguous PDAC boundaries make single-ground-truth confidence scores difficult to interpret, motivating calibration to multi-rater responses. TwinTrack applies post-hoc isotonic calibration of ensemble probabilities to the voxelwise MHR, yielding the best overall ambiguity-aware performance and the challenge’s top ranking.
Problem
PDAC segmentation has genuine inter-rater ambiguity, while standard single-mask models can produce poorly calibrated scores that overstate certainty.
Method
TwinTrack applies post-hoc isotonic regression to calibrate fixed ensemble tumor probabilities to the voxelwise MHR, the fraction of raters labeling each voxel as tumor.
Results
TwinTrack achieves the best overall performance across ambiguity-aware metrics, improving TDSC and lowering ECE and CRPS versus the uncalibrated pipeline.
Takeaways & Limitations
Calibrated probabilities are directly interpretable as the expected fraction of experts assigning the tumor label, while preserving voxel-prediction ranking.
Abstract
from arXiv · showhide
Pancreatic ductal adenocarcinoma (PDAC) segmentation on contrast-enhanced CT is inherently ambiguous: inter-rater disagreement among experts reflects genuine uncertainty rather than annotation noise. Standard deep learning approaches assume a single ground truth, producing probabilistic outputs that can be poorly calibrated and difficult to interpret under such ambiguity. We present TwinTrack, a framework that addresses this gap through post-hoc calibration of ensemble segmentation probabilities to the empirical mean human response (MHR) -the fraction of expert annotators labeling a voxel as tumor. Calibrated probabilities are thus directly interpretable as the expected proportion of annotators assigning the tumor label, explicitly modeling inter-rater disagreement. The proposed post-hoc calibration procedure is simple and requires only a small multi-rater calibration set. It consistently improves calibration metrics over standard approaches when evaluated on the MICCAI 2025 CURVAS-PDACVI multi-rater benchmark.
1. Introduction
PDAC boundaries on contrast-enhanced CT are ambiguous, so expert disagreement reflects genuine image uncertainty rather than mere label noise. TwinTrack addresses the resulting calibration gap by fitting fixed segmentation probabilities to the voxelwise mean human response (MHR) using a small multi-rater calibration set.
- Motivation: Expert disagreement in PDAC tumor delineation reflects genuine ambiguity because tumor boundaries are often ill-defined on contrast-enhanced CT.This matters for staging and treatment planning, where tumor extent must be delineated.
- Problem: Single-mask training encourages interpreting model scores as confidence in a unique boundary, which is unsatisfactory for ambiguous segmentation tasks.Poorly calibrated scores can overstate certainty.
- Prior work: Prior ambiguity-aware approaches mainly modify training, while post-hoc medical-image calibration has primarily addressed single-rater settings.The distinction motivates a multi-rater post-hoc formulation.
- Contribution: The pipeline uses a coarse model to define a high-recall region of interest before high-resolution ensemble prediction and post-hoc MHR calibration.Figure 1 presents these three stages in sequence.
- Contribution: TwinTrack calibrates a fixed segmentation model to the voxelwise MHR with a small calibration set, a setting the paper states was not explicitly formalized previously.MHR denotes the fraction of experts labeling each voxel as tumor.
2. Method
TwinTrack combines a coarse-to-fine ensemble segmenter with isotonic regression that calibrates tumor probabilities to the empirical MHR. The monotone mapping preserves voxel-ranking information while making probabilities interpretable as rater-labeling fractions.
- Segmentation pipeline: A low-resolution nnU-Net localizes the pancreas and defines a dilated high-recall ROI, while K=3 high-resolution nnU-Nets produce the pooled voxelwise score.The ensemble outputs are averaged before calibration and binary class merging.
- Calibration: Isotonic regression maps tumor probabilities to the voxelwise MHR using post-hoc calibration rather than retraining the segmentation model.The MHR is the fraction of N raters labeling voxel x as tumor.
- Calibration: The monotone calibration mapping preserves the ranking of voxel-wise predictions while changing their probabilistic interpretation.This separates ordering of predictions from their interpretation as probabilities.
- Interpretation: MHR alignment maps high-disagreement voxels to intermediate probabilities and interprets calibrated outputs as the expected fraction of raters assigning the tumor label.This avoids forcing ambiguous voxels into arbitrary hard labels.
- Theory: The MHR target is theoretically justified because multi-rater calibration is equivalent to calibrating predictions toward the MHR with the same isotonic-regression procedure.The justification is derived from a multi-rater isotonic-regression objective.
3. Experiments and Results
On the CURVAS–PDACVI test set, TwinTrack provides the strongest overall ambiguity-aware calibration among methods sharing the same coarse-to-fine segmenter. It improves multiple calibration metrics over the uncalibrated pipeline and outperforms single-rater and hard-label targets overall.
- Main results: TwinTrack achieves the best overall performance across ambiguity-aware metrics on the CURVAS–PDACVI test set.All compared methods use the same coarse-to-fine segmentation pipeline and differ only in calibration target.
- Main results: Relative to the uncalibrated pipeline, TwinTrack improves TDSC and lowers both ECE and CRPS.The absolute ECE reduction is modest but statistically significant.
- Main results: TwinTrack has the best VI performance for PORTA, SMV, AORTA, and CELIAC, whereas the hard-label target is best only for SMA.The vessel-level result is not uniformly won by one calibration target.
- Calibration targets: Single-rater and hard-label calibration markedly reduce TDSC and worsen ECE and CRPS relative to MHR calibration.These comparisons support MHR as the more suitable target for this ambiguous multi-rater setting.
- Benchmark outcome: TwinTrack achieved the top ranking in the MICCAI 2025 CURVAS–PDACVI challenge.This ranking is reported alongside the benchmark metric results.
A.1. Notation
The notation defines multi-rater voxel annotations, prediction bins, bin statistics, and the monotone isotonic mapping used for calibration.
- Each voxel has binary annotations from N raters and a model prediction in [0, 1] for the tumor class.
- Voxels in the calibration dataset are partitioned into M equal-mass bins ordered by predicted probability.Each bin contains approximately the same number of voxels.
- The weight of each bin represents its fraction of all calibration-dataset voxels.
- For each bin, the method computes average predicted confidence and each rater’s empirical positive rate.
- Single-rater isotonic regression learns a monotone mapping by minimizing weighted squared discrepancies between mapped confidence and empirical accuracy.The constrained problem is solved efficiently with PAVA, producing a piecewise-constant bin solution that can be linearly interpolated.
- Equal-mass binning regularizes dense-prediction calibration by stabilizing estimates and reducing computational cost and overfitting risk.
A.3. Calibration with several raters
The multi-rater extension minimizes a monotone weighted least-squares objective whose target is the mean human response averaged within each prediction bin.
- The multi-rater extension learns a monotone mapping that minimizes a weighted objective over prediction bins.
- After sum-of-squares decomposition, terms independent of the mapping are separated from the calibration-dependent objective.
- The bin-level target is the average fraction of raters labeling voxels as tumor, equal to the mean human response averaged over the bin.
- The resulting optimization is equivalent to calibrating predictions toward the MHR.It is theoretically justified and uses the same isotonic regression procedure as the single-rater case.
Appendix B. Evaluation Metrics
The evaluation protocol uses four complementary metrics covering soft overlap, calibration, volume uncertainty, and vascular invasion.
- TDSC measures soft multi-rater overlap by averaging Dice scores across multiple probability thresholds against the MHR.
- ECE measures calibration per rater and averages the resulting errors across raters using the official challenge metric.Predictions are partitioned into 50 uniform bins.
- CRPS measures volume uncertainty by comparing predicted soft tumor volume with a multi-rater reference volume distribution.
- VI evaluates vascular invasion independently for five vessels using the most restrictive plane-wise involvement and Wasserstein distance between invasion distributions.
Appendix C. Quantitative Results
Table 2 reports quantitative CURVAS–PDACVI test-set results with bootstrap uncertainty estimates for methods sharing the same segmenter but differing in calibration target.
- Table 2 reports full quantitative results with 95% bootstrap confidence intervals.
- The CURVAS–PDACVI test set contains n = 64 cases, and values are bootstrap means over 5000 resamples with 95% CIs.
- All methods use the same coarse-to-fine segmenter and differ only in the post-hoc calibration target.
Appendix D. Additional Calibration Diagnostics
TwinTrack improves agreement between predicted tumor probabilities and the empirical mean human response, bringing reliability curves closer to the perfect-calibration diagonal. Evaluation uses 50 uniform-width bins, while calibration learning uses 250 equal-mass bins.
- TwinTrack brings the test-set reliability curve much closer to the perfect calibration diagonal than the uncalibrated ensemble.
- Calibration learning uses M = 250 equal-mass bins, while reliability visualization and ECE reporting use 50 uniform-width bins on [0, 1].
- Reliability diagrams compare predicted confidence with the empirical fraction of tumor labels represented by the MHR.
- Residual non-monotonicity can reflect finite-sample fluctuations, distribution shift, and differing training and evaluation binning strategies.
Appendix E. Qualitative Results
The qualitative comparison shows that uncalibrated TwinTrack confidence is overconfident relative to the MHR, whereas calibrated confidence better matches graded uncertainty and expert-disagreement extent.
- Calibrated TwinTrack confidence is more consistent with the MHR’s graded uncertainty and spatial extent of expert disagreement.Each row shows a representative CURVAS–PDACVI case with the same zoomed lesion region across panels.
- The figure compares MHR on CT, uncalibrated confidence, and calibrated confidence using overlay probabilities from 0 to 1.White contours indicate iso-probability levels.