PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark

M. Riera-Marín, O. K. Sikha, J. Rodríguez-Comas, M. S. May, T. Kirscher, X. Coubez, P. Meyer, S. Faisan, Z. Pan, X. Zhou, X. Liang, C. Hémon, V. Boussot, J. -L. Dillenseger, J. -C. Nunes, K. -C. Kahl, C. Lüth, J. Traub, P. -H. Conze, M. M. Duh, A. Aubanell, R. de Figueiredo Cardoso, S. Egger-Hackenschmidt, J. García-López, M. A. González-Ballester, A. Galdran

arXiv:2604.27582v1cs.CV

TL;DR

Reliable assessment of pancreatic tumor vascular invasion is limited by ambiguous tumor-vessel boundaries and missing public multi-rater datasets. This paper introduces the CURVAS-PDACVI benchmark and uncertainty-aware evaluation framework, showing that global segmentation performance does not guarantee reliable localized vascular-invasion assessment, especially when expert agreement is low.

  • Problem

    PDAC vascular-invasion assessment remains challenging because tumor boundaries are ill-defined, adjacent to major vessels, and subject to expert disagreement.

  • Method

    The CURVAS-PDACVI benchmark combines five expert annotations per CT scan with probabilistic calibration and vessel-specific vascular-invasion evaluation.

  • Results

    Strong global segmentation performance did not necessarily translate into reliable vascular-invasion assessment at anatomically critical tumor-vessel boundaries, particularly in low-consensus cases.

  • Takeaways & Limitations

    Reliable PDAC vascular-invasion assessment requires evaluating localized boundary performance and inter-rater ambiguity alongside volumetric segmentation accuracy.

  • Takeaways & Limitations

    The relatively small test cohort and limited high-complexity cases reduce statistical power and warrant caution when interpreting ranking differences.

Abstract

from arXiv · show

Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i.e., tumor extension into adjacent critical vessels. Despite its importance for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge, an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In particular, methods optimized for binary segmentation perform competitively on average overlap metrics, but often degrade in high-complexity cases with low expert consensus, either collapsing in volume or overextending at uncertain boundaries. In contrast, methods that model inter-rater disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic models for preoperative decision-making.

1. Introduction · 2. The CURVAS-PDACVI Benchmark: Dataset

PDAC vascular invasion assessment is clinically important but difficult because tumor boundaries and tumor-vessel involvement are ambiguous, producing substantial expert disagreement and limiting deterministic model reliability. CURVAS-PDACVI addresses these limitations with five expert annotations per scan, uncertainty-aware evaluation, and a curated, stratified benchmark dataset.

  • 1. Introduction: Accurate PDAC segmentation supports tumor extent assessment, vascular invasion evaluation, and surgical resectability decisions, but heterogeneous appearance, ill-defined boundaries, and vascular proximity make the task challenging.
  • 1. Introduction: Expert radiologists often disagree on PDAC boundaries and vascular involvement, while black-box neural networks remain limited in reliability and clinical readiness.
  • 1. Introduction: Consensus-based fusion and single-gold-standard evaluation can obscure annotator disagreement and yield incomplete or biased assessments of model performance.
  • 1. Introduction: Clinical vascular invasion criteria are subjective, causing suboptimal specificity and inter-observer variability that deterministic predictions cannot capture for surgical decision-making.
  • 1. Introduction: CURVAS-PDACVI provides five independent expert annotations per CT scan and evaluates probabilistic predictions against empirical multi-rater vascular-invasion distributions.
  • 2.1. Multi-Rater Annotations in CURVAS-PDACVI: The dataset selected 125 curated CT volumes from PANORAMA, excluding cases without pathological confirmation or fully manual tumor and pancreas annotations.
  • 2.1. Multi-Rater Annotations in CURVAS-PDACVI: Vascular annotations were completed across five anatomical components: the Aorta, Celiac Trunk, Porta, Superior Mesenteric Vein (SMV), and Superior Mesenteric Artery (SMA).
  • 2.2. Data Splitting: The 109 cases were stratified into training (N = 40), validation (N = 5), and test (N = 64) partitions while balancing key demographic, imaging, and tumor-related confounders.

3. The CURVAS-PDACVI Benchmark: Competition

The CURVAS-PDACVI competition established a standardized, uncertainty-aware benchmark for PDAC segmentation and vascular invasion under multi-rater ambiguity. Six nnU-Net-based algorithms were evaluated using complementary spatial, calibration, probabilistic-volume, and vessel-specific metrics, with final ranking based on averaged team ranks.

  • Challenge organization: The open challenge compared uncertainty-aware PDAC segmentation and vascular invasion methods within a common evaluation framework focused on multi-rater ambiguity.The competition complemented the public CURVAS-PDACVI dataset and examined method behavior in a clinically relevant setting.
  • Challenge organization: The challenge used Grand Challenge for standardized data distribution, containerized submissions, and independent evaluation across training, validation, and testing phases during 2025.All submissions followed the same protocol for fair comparison, while organizer baselines were reference-only and prize-ineligible.
  • Evaluation framework: Methods were assessed with four metric families: segmentation quality, multi-rater calibration, probabilistic volume assessment, and vascular invasion analysis.This framework extended conventional overlap benchmarking to evaluate spatial accuracy and reliability under uncertainty.
  • Evaluation framework: Segmentation quality used conventional Dice Similarity Coefficient (DSC) against binarized STAPLE consensus and threshold-averaged Dice (Thr-DSC) over predicted probabilities and averaged annotations.Thr-DSC preserved the probabilistic nature of predictions and multi-rater ground truth, while DSC used binarized masks.
  • Evaluation framework: Vascular invasion was evaluated independently for the Porta, SMV, SMA, Celiac Trunk, and Aorta using Wasserstein-based distances between predicted and reference invasion-score distributions.The distributions were derived from multiple thresholds and annotators, capturing uncertainty at critical tumor-vessel boundaries.
  • Participants and ranking: Six unique algorithms from six international teams were evaluated, all based on nnU-Net variants but differing in multi-rater supervision, uncertainty modeling, and vascular-precision tradeoffs.Separate rankings covered DSC, Thr-DSC, MR-ECE, CRPS, and five vessel-specific VI scores; the final leaderboard averaged team ranks across metrics.

TwinTrack - University of Strasbourg, France · CorpuSeg - Shenzhen Institute of Advanced Technology · MIC DKFZ - German Cancer Research Center

The benchmark submissions used two-stage or coarse-to-fine segmentation pipelines while incorporating multi-rater variability through post-hoc calibration, prediction-level fusion, Bayesian inference, or ensemble-based uncertainty estimation. These strategies differed in whether annotations were integrated during training or uncertainty was modeled at inference.

  • TwinTrack - University of Strasbourg, France: TwinTrack used a two-stage nnU-Net cascade, localizing anatomy at low resolution before high-resolution refinement within a region of interest.The refinement stage used a deep ensemble.
  • TwinTrack - University of Strasbourg, France: TwinTrack modeled inter-rater variability post hoc by applying isotonic regression to align predicted probabilities with mean human consensus.Multiple annotations were not integrated during training.
  • CorpuSeg - Shenzhen Institute of Advanced Technology: CorpuSeg used coarse tumor localization followed by five independent second-stage models, each trained on one expert annotation.The model outputs were averaged during inference to reflect annotator variability.
  • CorpuSeg - Shenzhen Institute of Advanced Technology: CorpuSeg represented multi-rater supervision through prediction-level fusion, averaging the five probabilistic outputs at inference.This design allowed the final prediction to capture annotator variability.
  • CorpuSeg - Shenzhen Institute of Advanced Technology: BreizhSeg combined coarse-to-fine segmentation with Bayesian uncertainty modeling using a pretrained ResidualEncoderUNet converted into an Adaptable Bayesian Neural Network.Stochastic perturbations were introduced into normalization layers.
  • CorpuSeg - Shenzhen Institute of Advanced Technology: BreizhSeg approximated the posterior predictive distribution through multiple stochastic forward passes and fused probabilistic segmentations into a consensus prediction.The stochastic passes were performed at inference time.
  • MIC DKFZ - German Cancer Research Center: MIC DKFZ fine-tuned a pretrained nnU-Net ResEnc-L ensemble using all available annotations, including the STAPLE consensus.Uncertainty was estimated from ensemble softmax outputs and calibrated using temperature scaling.
  • MIC DKFZ - German Cancer Research Center: MIC DKFZ emphasized strong baseline segmentation while explicitly addressing calibration at inference time through temperature scaling.The approach derived uncertainty from ensemble softmax outputs.

ROISeg - Chinese Academy of Sciences · OrdSTAPLE - Universitat Pompeu Fabra, Sycai Medical

ROISeg uses a conventional STAPLE-consensus segmentation strategy, whereas OrdSTAPLE explicitly models annotator disagreement through ordinal prediction. Benchmark results show that uncertainty-aware modeling is especially valuable when expert consensus breaks down, despite strong global overlap from binary methods.

  • ROISeg - Chinese Academy of Sciences: ROISeg fused five expert annotations into one STAPLE consensus and trained a full-resolution nnU-Net without explicit uncertainty modeling.
  • OrdSTAPLE - Universitat Pompeu Fabra, Sycai Medical: OrdSTAPLE combined a STAPLE-trained binary model with an ordinal model predicting increasing annotator-agreement levels, then merged their probabilistic outputs.
  • 3.4.1. Quantitative Benchmark Results: Mean DSC exceeded 65% and Thr-DSC approached 60-65% for the strongest methods, while BreizhSeg ranked first in DSC, Thr-DSC, and MR-ECE.ROISeg achieved the lowest CRPS.
  • 3.4.1. Quantitative Benchmark Results: TwinTrack achieved the lowest W1 error for four of five vascular structures, whereas CorpuSeg performed best for the SMA and remained competitive elsewhere.The Aorta was the least challenging structure across methods.
  • 3.4.1. Quantitative Benchmark Results: Strong global segmentation performance did not necessarily translate into accurate localized vascular-involvement assessment at boundaries relevant to PDAC resectability.
  • 3.4.2. Statistical Robustness of the Benchmark Ranking: Across 500 bootstrap iterations, BreizhSeg consistently ranked highly on DSC, Thr-DSC, and MR-ECE, while TwinTrack and CorpuSeg remained competitive for vascular invasion.
  • 3.4.2. Statistical Robustness of the Benchmark Ranking: Global metric differences were often significant, but most vessel-specific comparisons were not, reflecting the strong influence of highly ambiguous cases.The analyses supported population-level ranking robustness while showing that fine-grained vascular assessment remained limited by case difficulty and inter-rater ambiguity.
  • 3.4.3. Performance in high-complexity cases: OrdSTAPLE became the top-performing method on cases with mean pairwise human DSC below 30%, while BreizhSeg and MIC DKFZ lost spatial overlap and TwinTrack and ROISeg incurred larger vascular invasion errors.Explicit disagreement modeling appeared to improve robustness when expert consensus broke down, whereas binary strategies tended toward under-segmentation or boundary overextension.

4. Discussion

The benchmark shows that aggregate segmentation performance can mask unreliable vascular-invasion assessment at ambiguous tumor-vessel interfaces. Its findings support calibrated, uncertainty-aware evaluation while highlighting limitations from cohort size, annotation-based reference standards, and limited architectural diversity.

  • Clinical relevance: Strong global volumetric agreement did not guarantee reliable vascular-invasion assessment, particularly at ambiguous tumor-vessel interfaces and challenging venous structures.Similar aggregate results masked clinically relevant differences when expert consensus was low.
  • Method behavior: BreizhSeg and MIC DKFZ performed strongly on global overlap, whereas TwinTrack and CorpuSeg were more competitive at the vascular interface.The ranking changed substantially when evaluation was restricted to high-complexity cases, where average-case spatial methods degraded.
  • Clinical relevance: Clinical usefulness depends on calibrated and anatomically reliable boundaries around structures such as the SMV, PORTA, and SMA, not tumor-volume recovery alone.The benchmark therefore advocates aggregate overlap alongside uncertainty-aware, multi-metric evaluation tied to downstream clinical goals.
  • Uncertainty modeling: Binary-target methods produced sharp, overconfident boundaries under weak expert agreement, while disagreement-aware methods generated diffuse probability maps reflecting annotation uncertainty.Probabilistic outputs can communicate confidence and support downstream review when imaging evidence and expert labels remain uncertain.
  • Limitations: The benchmark’s small test cohort and limited high-complexity cases reduce statistical power, while expert-annotation ground truth inherits interpretive uncertainty and potential bias.All submissions also used nnU-Net variants, limiting architectural diversity and emphasizing training strategy, uncertainty modeling, and annotation handling.

5. Conclusion · CRediT authorship contribution statement

The conclusion presents CURVAS-PDACVI as a public multi-rater benchmark and community challenge for uncertainty-aware PDAC vascular invasion assessment. The authorship statement assigns contributions spanning conceptualization, methodology, software, data curation, supervision, writing, funding, and related activities.

  • 5. Conclusion: The CURVAS-PDACVI benchmark combines a publicly released multi-rater PDAC segmentation dataset with an open community challenge on vascular invasion under uncertainty.
  • 5. Conclusion: Strong global segmentation performance does not necessarily ensure reliable characterization of pancreatic tumor-vessel interfaces.
  • 5. Conclusion: The benchmark identifies low expert agreement as a particularly challenging condition for reliable tumor-vessel interface characterization.
  • CRediT authorship contribution statement: Meritxell Riera-Marín contributed conceptualization, methodology, software, investigation, original drafting, review and editing, data curation, formal analysis, and validation.
  • CRediT authorship contribution statement: Sikha O K contributed conceptualization, software, and visualization, while J´ulia Rodríguez-Comas contributed conceptualization and funding acquisition.
  • CRediT authorship contribution statement: Matthias Stefan May contributed conceptualization, data curation, and funding acquisition, while Pierre-Henri Conze contributed conceptualization and writing review and editing.
  • CRediT authorship contribution statement: The remaining listed contributors provided data curation, supervision, writing review and editing, funding acquisition, software, investigation, and conceptualization across their assigned roles.

6. Appendix

The appendix documents the released datasets, curated benchmark cohort, challenge protocol, submission requirements, and evaluation framework. It also specifies vascular invasion scoring across five vessels and rank aggregation across nine evaluation axes.

  • Data and resources: The PANORAMA scans and CURVAS-PDACVI benchmark, including multi-rater annotations and evaluation resources, are publicly available through Zenodo releases.The repository uses standardized case file organization, including original PANORAMA segmentations and additional annotations.
  • Challenge protocol: The Grand Challenge protocol progressed from training and open sanity-check and validation phases to closed testing, followed by organizer analysis and MICCAI 2025 winner announcements.The listed phases ran from May through September 2025.
  • Submission requirements: Participants submitted binarized and probabilistic PDAC segmentations, while vascular structures were supplied for local testing and evaluation rather than participant segmentation.The official evaluation code is publicly available in the challenge repository.
  • Evaluation framework: Vascular invasion was evaluated independently for PORTA, SMV, SMA, CELIAC TRUNK, and AORTA using maximum slice-wise contact angles across coronal, sagittal, and axial planes.The final leaderboard averaged ranks across DSC, Thr-DSC, MR-ECE, CRPS, and five vessel-specific vascular invasion scores.

CorpuSeg - Shenzhen Institute of Advanced Technology

CorpuSeg combined coarse low-resolution localization with five full-resolution models trained on separate expert annotations. It fused their probabilistic outputs through voxel-wise averaging.

  • Architecture: CorpuSeg used a low-resolution nnU-Net for coarse localization before applying five full-resolution second-stage models.Each second-stage model was trained on a different expert annotation.
  • Fusion: Final predictions were produced by voxel-wise averaging of the five probabilistic outputs, exemplifying prediction-level fusion of multi-rater supervision.

BreizhSeg - University of Rennes 1

BreizhSeg adapted a pretrained ResidualEncoderUNet into a Bayesian model by replacing normalization layers with Bayesian Normalization Layers. Five stochastic inference passes approximated the posterior predictive distribution, which was later combined with STAPLE.

  • BreizhSeg initialized a pretrained ResidualEncoderUNet from PANORAMA Batch 4 and MSD pancreas tumor data.
  • The deterministic network became an Adaptable Bayesian Neural Network by replacing normalization layers with Bayesian Normalization Layers.
  • Five stochastic forward passes approximated the posterior predictive distribution at inference, followed by combination with STAPLE.

MIC DKFZ - German Cancer Research Center

MIC DKFZ adapted a pretrained nnU-Net ResEnc-L model to CURVAS-PDACVI using all five annotations and STAPLE-derived labels, with temperature scaling before ensembling. Inference omitted overlapping patches and test-time augmentation because of computational constraints.

  • MIC DKFZ pretrained an nnU-Net ResEnc-L model on PANORAMA Batch 4 before fine-tuning it on CURVAS-PDACVI.
  • The team fine-tuned an ensemble of five models using all five annotations together with the STAPLE label as image–ground-truth pairs.
  • Temperature scaling preceded ensembling, while computational constraints led to inference without overlapping patches or test-time augmentation.

ROISeg - Chinese Academy of Sciences

ROISeg trained a full-resolution nnU-Net on STAPLE consensus labels, simplifying processing while discarding explicit inter-rater disagreement. Benchmark analyses showed that volumetric overlap was weakly related to vessel-specific errors, whose variance was driven by recurrent ambiguous cases.

  • Implementation: ROISeg trained a full-resolution nnU-Net directly on STAPLE consensus labels, avoiding multi-stage processing and ensembling but discarding explicit inter-rater disagreement.The method collapsed multi-rater annotations into a single fused target before training.
  • Metric relationships: DSC and Thr-DSC were strongly correlated, whereas their association with vessel-specific vascular invasion errors was markedly weaker.This supported evaluating overlap, calibration, and vascular invasion with multiple metrics rather than relying on global volumetric agreement alone.
  • Statistical comparisons: Several pairwise method differences remained significant for calibration and overlap, while most vessel-specific vascular invasion comparisons did not reach significance.Case-wise variance in vascular assessment was strongly influenced by a subset of highly ambiguous studies shared across methods.
  • Case-wise failure analysis: Recurrent hard studies induced aligned degradation patterns across several metrics and architectures, indicating that benchmark variance was substantially driven by intrinsically ambiguous cases.Methods nevertheless exhibited distinct failure profiles across overlap, calibration, and vascular metrics.
Loading 2604.27582v1…