PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified CT and MRI Pancreas Segmentation for Label-Efficient Cross-Modality Subregion Transfer

Ziliang Hong, Hongyi Pan, Halil Ertugrul Aktas, Andrea Bejar, Elif Keles, Frank H. Miller, Michael B. Wallace, Rajesh N. Keswani, Gorkem Durak, Ulas Bagci

arXiv:2609.13043v1cs.CVcs.AI

TL;DR

Pancreas segmentation is challenged by anatomical variability and CT–MRI domain shifts. This work uses domain-adversarial learning to train a unified model and transfers its learned representations to subregion segmentation with limited single-modality labels. The reported results support cross-modality segmentation and transfer to CT without CT subregion training labels.

  • Problem

    Pancreas segmentation is difficult across CT and MRI because anatomy varies and modality-induced domain shifts can make learned representations less robust across modalities.

    Source: §I · p. 1

  • Method

    The framework uses domain-adversarial learning for unified CT–MRI pancreas segmentation and transfers the learned encoder to subregion segmentation using limited single-modality labels.

    Source: §I · p. 1, 2) Pancreas Subregion Segmentation · p. 3

  • Results

    The framework showed robust cross-modality performance and external OOD generalization, and transferred to CT head–body–tail segmentation without CT subregion training labels.

    Source: §I · p. 1, §V · p. 5

  • Takeaways & Limitations

    The findings support a practical approach to fine-grained label scarcity across imaging modalities by enabling CT subregion segmentation without CT subregion annotations during downstream training.

    Source: §V · p. 5

    The study aligns modality-level CT–MRI differences, leaving scanner and protocol variation for future work, and focuses on pancreas segmentation.

    Source: D. Discussion · p. 5

Abstract

from arXiv · show

Robust medical image segmentation across imaging modalities is challenging because of large differences in appearance and intensity distributions. Models trained on a single modality often show substantial performance drops when applied to unseen domains. In this work, we develop a unified 3D pancreas segmentation framework that applies domain-adversarial learning to 4,604 heterogeneous CT and MRI scans to learn anatomical representations. A shared nnU-Net encoder-decoder is trained for whole-pancreas segmentation, with a latent domain discriminator encouraging CT-MRI feature alignment. The learned encoder is subsequently transferred to pancreatic head-body-tail segmentation using limited MRI-only subregion annotations. An average Dice score of 87.31% on the in-distribution test set and Dice scores ranging from 84.20% to 88.09% across external OOD datasets were achieved in whole pancreas segmentation. Dice scores of 80.53% on MRI and 83.05% on CT were achieved for downstream subregion segmentation, without using CT subregion annotations. These results demonstrate that a unified anatomical representation can support both cross-modality pancreas segmentation and label-efficient downstream transfer.

I. INTRODUCTION

Pancreas segmentation is difficult because of anatomical variability, irregular shape, low contrast, and pronounced CT–MRI appearance differences. The work proposes unified modality-invariant representations for whole-pancreas segmentation and limited-label subregion transfer.

I. INTRODUCTION

  • Large anatomical variability, irregular shape, low contrast, and differing CT–MRI appearance make accurate pancreas segmentation challenging.The introduction also notes that accurate segmentation supports clinical and research applications including diagnosis, surgical planning, and downstream analysis.Source: §I · p. 1
  • Separate models for clinical imaging modalities and MRI sequences are inefficient and often infeasible when annotations are limited.Directly combining CT and MRI can degrade performance because of strong modality-induced domain shifts.Source: §I · p. 1
  • The work develops a unified CT–MRI framework using domain-adversarial learning to learn modality-invariant pancreatic representations.The stated goal is a single model that segments the pancreas across multiple imaging modalities.Source: §I · p. 1
  • The learned representations transfer to downstream segmentation with limited annotations from one modality, including CT head–body–tail segmentation without CT subregion labels.The authors frame this as addressing fine-grained label scarcity in the target modality.Source: §I · p. 1

II. RELATED WORK

Related work covers domain-adversarial alignment across imaging domains and the evolution of pancreas segmentation methods. It identifies robust, transferable representations across modalities as an unresolved need.

A. Domain Adversarial Learning

  • Domain-Adversarial Neural Networks use a gradient reversal layer to align feature distributions between source and target domains through adversarial training.The approach has inspired extensions in computer vision and medical image analysis.Source: A. Domain Adversarial Learning · p. 1
  • Medical-imaging studies have applied adversarial learning to cross-dataset and cross-modality segmentation, including CT–MRI feature alignment for organ segmentation.The passage cites brain-lesion adaptation and cross-modality organ-segmentation work,.Source: A. Domain Adversarial Learning · p. 2

B. Pancreas Segmentation

  • Pancreas segmentation progressed from multi-atlas and handcrafted methods to deep-learning approaches, with U-Net variants and nnU-Net becoming widely used.nnU-Net adapts its network configuration to datasets and performs strongly across medical segmentation tasks.Source: B. Pancreas Segmentation · p. 2
  • Pancreas segmentation research remains CT-centered, while MRI is less explored because annotated datasets are limited and T1W and T2W appearance differs.Robust, transferable representations across imaging modalities remain incompletely addressed.Source: B. Pancreas Segmentation · p. 2

III. METHODOLOGY

The problem statement defines a multimodal segmentation setting with image, mask, and modality labels, then specifies a segmentation objective. It notes that MRI–CT domain shift can reduce generalizability.

A. Problem Statement

  • The multimodal dataset contains N images, ground-truth masks, and modality labels, with MRI encoded as 0 and CT as 1.The standard network uses an encoder E and decoder D to predict a segmentation mask from each input image.Source: A. Problem Statement · p. 2
  • The segmentation network predicts masks with D(E(x_i)) and minimizes a joint Dice and cross-entropy loss against ground-truth masks.The passage says the two losses capture global overlap and voxel-wise accuracy, respectively.Source: A. Problem Statement · p. 2
  • Significant MRI–CT domain shift can degrade a single-modality model’s generalizability to unseen distributions.Source: A. Problem Statement · p. 2

B. Modality-Invariant Representation Learning

The method adds domain-adversarial supervision to encourage modality-invariant features. The encoder is trained to resist modality-specific cues while retaining representations useful for segmentation.

B. Modality-Invariant Representation Learning

1) Domain Discriminator and Joint Optimization:

The domain discriminator is trained to identify CT versus MRI from encoder features, while gradient reversal sends an adversarial signal to the encoder. The joint objective balances segmentation and domain losses, with the adversarial weight increased gradually during training.

1) Domain Discriminator and Joint Optimization:

  • The discriminator minimizes cross-entropy to distinguish MRI from CT using latent features produced by the encoder.This modality-level supervision supplies the domain signal used in adversarial training.Source: 1) Domain Discriminator and Joint… · p. 3
  • A gradient reversal layer reverses gradients during backpropagation, encouraging the encoder to suppress modality-specific information.The reversal is paired with an identity transformation during the forward pass.Source: 1) Domain Discriminator and Joint… · p. 3
  • The total objective combines segmentation and domain losses, with λ gradually increased from 0 to 0.5 during warm-up.λ scales the influence of the adversarial signal; the warm-up is used to support meaningful anatomical features and training stability.Source: 1) Domain Discriminator and Joint… · p. 3

2) Pancreas Subregion Segmentation:

The downstream task transfers a frozen encoder to head, body, and tail segmentation using limited single-modality subregion labels. The study evaluates whole-pancreas segmentation across CT and MRI and subregion segmentation on specified in-distribution and external datasets.

2) Pancreas Subregion Segmentation:

A. Tasks and Datasets

  • Whole Pancreas Segmentation evaluates binary whole-organ segmentation across CT and MRI, while the downstream task partitions the pancreas into head, body, and tail.Whole-pancreas performance is assessed on in-distribution test sets and out-of-distribution datasets; downstream evaluation includes Cyst-X and CT from AMOS and BTCV.Source: A. Tasks and Datasets · p. 3
  • The in-distribution cohort contains 4,604 CT and MRI scans, and external generalization is evaluated on AMOS, BTCV, and U-Mamba.Only Cyst-X provides whole-pancreas and subregion masks among the listed in-distribution datasets; expert radiologists annotated subregions for 7 AMOS and 10 BTCV CT scans.Source: A. Tasks and Datasets · p. 3

B. Implementation Details

The implementation uses a standard nnU-Net backbone with 3D patch training and sliding-window inference. Training proceeds through a segmentation warm-up followed by adversarial alignment, with domain supervision distinguishing CT from MRI rather than individual MRI sequences.

B. Implementation Details

  • The model uses nnU-Net preprocessing, resampling images and annotations to median training-set spacing and applying z-score intensity normalization.Training uses 80×160×192-voxel 3D patches; sliding-window inference produces whole-volume predictions.Source: B. Implementation Details · pp. 3–4
  • Training first uses segmentation loss for 2,000 epochs, then adds the domain discriminator for 1,000 adversarial-alignment epochs.The segmentation network and discriminator are jointly optimized with Adam at an initial learning rate of 0.001.Source: B. Implementation Details · p. 4
  • Domain supervision distinguishes CT from MRI rather than specific MRI sequences, reflecting their relatively congruent latent feature distributions.The passage attributes this choice to shared MR imaging physics and preliminary t-SNE visualization.Source: B. Implementation Details · p. 4

1) Whole Pancreas Segmentation:

Whole-pancreas segmentation achieved 87.31% average Dice on the in-distribution test set and Dice scores above 84% across external datasets. Transferred subregion segmentation scored 80.53% on MRI and 83.05% on CT without CT subregion annotations.

1) Whole Pancreas Segmentation:

2) Pancreas Subregion Segmentation:

3) Ablation study of partial single-modality fine-tuning:

The benefit of domain-adversarial pretraining under partial single-modality fine-tuning varied with MRI sequence and supervision amount. Bottleneck-feature visualizations showed reduced modality-specific clustering with the discriminator, while the study leaves finer-grained distribution shifts for future work.

3) Ablation study of partial single-modality fine-tuning:

  • Domain-adversarial pretraining matched or outperformed the baseline on MRI across fine-tuning settings, while CT gains varied by MRI sequence and supervision amount.The qualitative visualization compares baseline and proposed results across MRI T1W/T2W and CT under different fine-tuning settings.Source: 3) Ablation study of partial… · p. 4, Figure 2 · p. 5

D. Discussion

  • Without domain-adversarial training, CT and MRI bottleneck features separated; adding the discriminator made modality-specific clustering less pronounced, consistent with quantitative robustness and transfer results.The t-SNE visualization shows MRI sequences and CT in separate colors, with an additional view grouping MRI sequences together.Source: D. Discussion · pp. 4–5, Figure 3 · p. 5
  • The study aligns modality-level CT and MRI shifts, leaving scanner and protocol variation for future work; extending the framework beyond pancreas segmentation also remains future work.Source: D. Discussion · p. 5

V. CONCLUSION

The framework learns modality-invariant anatomical representations for pancreas segmentation across CT and MRI, with evaluation showing robust cross-modality performance and generalization. Its learned representations also transfer to pancreatic subregion segmentation on CT without CT subregion annotations.

V. CONCLUSION

  • Evaluation on 4,604 multi-center scans and external OOD datasets demonstrated robust cross-modality pancreas segmentation and generalization.The framework uses domain-adversarial learning to learn modality-invariant anatomical representations for a single model across CT and MRI.Source: §V · p. 5
  • The learned representations supported pancreatic head–body–tail segmentation on CT without CT subregion annotations during downstream training.The paper presents this as a practical solution to fine-grained label scarcity across imaging modalities.Source: §V · p. 5
  • Modality-invariant representation learning outperformed or matched the baseline across all partial single-modality fine-tuning settings.The figure compares the proposed approach (green) with the baseline (gray).Source: Figure 4 · p. 6
Loading 2609.13043v1…