PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

Cheng Chen, Qi Dou, Hao Chen, Jing Qin, Pheng Ann Heng

arXiv:2002.02255v1eess.IVcs.CV

TL;DR

Medical image segmentation suffers when models encounter unlabeled target domains with heterogeneous characteristics, especially across MRI and CT. SIFA combines image and feature alignment through adversarial learning, deep supervision, and a shared encoder, improving target-domain segmentation across two bidirectional tasks. The method is reported to outperform state-of-the-art adaptation approaches, with abdominal results close to the supervised upper bound.

  • Problem

    Cross-modality domain shift degrades medical image segmentation, while acquiring expert target-domain annotations is expensive or sometimes infeasible.

  • Method

    SIFA synergistically combines image alignment and feature alignment using adversarial learning, deep supervision, and a shared feature encoder in an end-to-end framework.

  • Results

    SIFA consistently improves target-modality segmentation on cardiac and abdominal tasks, outperforms state-of-the-art approaches, and achieves abdominal results close to the supervised upper bound.

  • Takeaways & Limitations

    Bidirectional unsupervised MRI–CT adaptation is achieved on two multi-class medical segmentation tasks without target labels.

  • Takeaways & Limitations

    The framework uses 2D networks because combining image and feature alignment in 3D is memory intensive and training prohibitive.

Abstract

from arXiv · show

Unsupervised domain adaptation has increasingly gained interest in medical image computing, aiming to tackle the performance degradation of deep neural networks when being deployed to unseen data with heterogeneous characteristics. In this work, we present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA), to effectively adapt a segmentation network to an unlabeled target domain. Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. In particular, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features by leveraging adversarial learning in multiple aspects and with a deeply supervised mechanism. The feature encoder is shared between both adaptive perspectives to leverage their mutual benefits via end-to-end learning. We have extensively evaluated our method with cardiac substructure segmentation and abdominal multi-organ segmentation for bidirectional cross-modality adaptation between MRI and CT images. Experimental results on two different tasks demonstrate that our SIFA method is effective in improving segmentation performance on unlabeled target images, and outperforms the state-of-the-art domain adaptation approaches by a large margin.

I. INTRODUCTION

Medical segmentation models can fail across MRI and CT because cross-modality domain shift changes image appearance. SIFA addresses this challenge by jointly aligning images and features for unlabeled target domains.

  • Motivation: Cross-modality differences can cause deep networks trained on one modality to fail when segmenting the same anatomy in the other.MRI and CT depict cardiac and abdominal anatomy with significantly different visual appearances.
  • Motivation: Unsupervised domain adaptation is attractive because target-domain annotations are expensive or sometimes infeasible to obtain.Medical annotation requires expert knowledge, while this setting requires no target labels.
  • Research gap: Existing approaches align domains at either the image level or feature level, but these perspectives are complementary and had not been synergistically combined.Image alignment changes input appearance, whereas feature alignment seeks domain-invariant representations.
  • SIFA framework: SIFA jointly transforms labeled source images toward target appearance and aligns remaining feature-level differences using adversarial learning and deep supervision.Its feature alignment operates in multiple aspects, including semantic prediction and generated image spaces.
  • Evaluation: SIFA is evaluated for bidirectional MRI–CT adaptation on cardiac substructure and abdominal multi-organ segmentation.The experiments target unlabeled target images and compare the method with state-of-the-art approaches.
  • Evaluation: Unsupervised adaptation produces abdominal results very close to the supervised training upper bound.The paper reports this outcome alongside extensive bidirectional experiments and comparisons.

II. RELATED WORK

Medical image domain adaptation addresses performance degradation caused by inter-scanner and cross-modality variation. SIFA combines image- and feature-level alignment in one shared-encoder framework so both adaptations can benefit from end-to-end training.

  • Domain shift in medical image analysis commonly arises from inter-scanner and cross-modality variations.
  • Adversarial adaptation methods address domain shift through image-level alignment, feature-level alignment, or combinations of both.
  • A shared feature encoder allows image alignment and feature alignment to mutually benefit each other during end-to-end training.
  • Image and feature alignment are complementary, but their simultaneous use remains insufficiently explored for severe cross-modality medical-image shifts.
  • SIFA synergistically merges image and feature adaptation in a unified network for unsupervised cross-modality segmentation.

A. Appearance Transformation for Image Alignment

SIFA performs appearance transformation between source and target domains while preserving image content through adversarial and cycle-consistency training. Its generators and discriminators create bidirectional source-target transformations, with the encoder and decoder forming the reverse generator.

  • Image alignment transforms source images toward target-domain appearance while preserving their original structural semantics.
  • A generator G_t produces target-like source images, while discriminator D_t distinguishes transformed images G_t(x_s) from real target images x_t.
  • The target-domain generator and discriminator are optimized adversarially as a minimax two-player game.
  • The feature encoder E and upsampling decoder U form the reverse generator G_s = E ◦ U for target-to-source reconstruction.
  • Cycle consistency encourages reconstructed source and target images to remain close to their respective originals through pixel-wise loss.

2) Segmentation Network for Target Data:

The target-domain segmentation network applies the shared encoder and pixel-wise classifier to source images transformed into target-like appearance. It is trained with a hybrid cross-entropy and Dice loss to address class imbalance.

  • The composition of encoder E and classifier C forms the segmentation network for target-domain images.
  • The network extracts features from transformed source images and forwards them to a pixel-wise classifier to predict segmentation masks.
  • The hybrid segmentation loss combines cross-entropy and Dice losses for training on transformed image–label pairs.The Dice term helps address class imbalance in medical image segmentation.
  • The framework connects the encoder with pixel-wise classifiers while discriminators distinguish their inputs for adversarial learning.

B. Adversarial Learning for Feature Alignment

Because image alignment alone can remain insufficient under severe cross-modality shift, SIFA aligns synthesized target-like and real target features through adversarial learning in compact spaces.

  • Image alignment may be insufficient when cross-modality domain shift is severe, motivating additional feature-alignment discriminators.
  • SIFA avoids direct high-dimensional feature alignment by using adversarial learning through two compact lower-dimensional spaces.
  • A semantic-space discriminator classifies segmentation outputs from transformed source and real target images to encourage modality-consistent predictions.
  • When semantic predictions remain distinguishable, adversarial gradients are back-propagated to the encoder to reduce feature-distribution differences.

2) Deeply Supervised Adversarial Learning in Semantic Prediction Space:

Deep supervision extends semantic-prediction-space alignment to multiple encoder depths, strengthening gradient guidance for lower-level features and using separate classifiers and discriminators.

  • Deep supervision directly guides upper and lower encoder layers to improve gradient propagation to low-level features.
  • An additional pixel-wise classifier is connected to lower encoder-layer outputs for supervised feature alignment.
  • The segmentation and adversarial losses are extended across two classifier–discriminator pairs attached to different encoder layers.
  • The source discriminator also distinguishes reconstructed source-like images from images transformed from real target images, supervising domain-invariant features.

C. Shared Encoder for Synergistic Learning

SIFA shares encoder E across image and feature alignment, allowing their objectives and gradients to interact in one end-to-end framework. The implementation uses sequential module updates and a specified multi-module architecture.

  • C. Shared Encoder for Synergistic Learning: Encoder E is shared between image and feature alignment so both adaptive perspectives can mutually benefit during training.
  • C. Shared Encoder for Synergistic Learning: The encoder receives gradients from image-alignment losses and feature-alignment discriminators, fitting it to a multi-task learning scenario.
  • C. Shared Encoder for Synergistic Learning: Training updates modules sequentially in the order Gt → Dt → E → Ci → U → Ds → Dpi.
  • D. Network Configurations and Implementation Details: Target-domain testing applies encoder E followed by classifier C1 to obtain segmentation without target-domain annotations.
  • D. Network Configurations and Implementation Details: The encoder uses residual connections and dilation rate 2 to enlarge receptive fields while preserving spatial resolution for dense predictions.
  • D. Network Configurations and Implementation Details: The framework was trained for 20k iterations with batch size 8 using Adam at a learning rate of 2×10^-4.The same learning rate was used across network modules.

IV. EXPERIMENTS

The experiments evaluate bidirectional unsupervised MRI↔CT adaptation on cardiac and abdominal segmentation using unpaired datasets with target labels reserved for evaluation.

  • Experimental Tasks: Two multi-class tasks assess adaptation between MRI and CT: cardiac substructures and abdominal organs.The cardiac task segments AA, LAC, LVC, and MYO; the abdominal task segments liver, both kidneys, and spleen.
  • Experimental Protocol: Adaptation is evaluated in both directions, from MRI to CT and from CT to MRI.MRI and CT data are unpaired and collected from different patient cohorts.
  • Experimental Protocol: Target-domain ground-truth masks are used only for evaluation and are withheld during training.Each modality is randomly split into 80% training and 20% testing scans.
  • Datasets: The cardiac dataset contains 20 MRI and 20 CT training volumes, while the abdominal dataset contains 20 MRI and 30 CT training volumes.Ground-truth masks are provided for the cardiac structures and abdominal organs in both datasets.

B. Evaluation Metrics

Evaluation combines voxel-wise overlap and 3D boundary-distance metrics, with comparisons against no-adaptation and supervised-training bounds. SIFA substantially improves cardiac and abdominal cross-modality segmentation, approaching supervised performance on abdominal data.

  • Evaluation Metrics: Dice measures voxel-wise segmentation accuracy, while ASD measures average 3D distances between predicted and ground-truth surfaces.Higher Dice and lower ASD indicate better segmentation.
  • Cardiac Segmentation: Without adaptation, average Dice falls to 17.2% for MRI→CT cardiac segmentation and 15.7% for CT→MRI.The gaps to supervised upper bounds are 73.7 and 67.9 percentage points, respectively.
  • Cardiac Segmentation: 74.1% average Dice and 7.0 average ASD are achieved for cardiac CT, while cardiac MRI reaches 63.4% Dice and 5.7 ASD.These results are reported across the four cardiac structures.
  • Abdominal Segmentation: Without adaptation, abdominal average Dice is 58.2% for CT and 57.7% for MRI, with approximately 30-point gaps to supervised training.The reported cross-modality degradation is less severe than for cardiac segmentation.
  • Qualitative Results: The visual comparisons arrange raw images, no-adaptation results, competing methods, SIFA, supervised results, and ground truth across columns.Cardiac structures are color-coded as AA, LAC, LVC, and MYO; abdominal organs are separately color-coded.
  • Abdominal Segmentation: 83.7% average Dice and 1.3 average ASD are achieved for abdominal CT, while abdominal MRI reaches 85.4% Dice and 1.5 ASD.The gaps to supervised training are 5 percentage points in Dice and 0.1 ASD for CT, and 1.9 Dice points with the same ASD for MRI.

D. Comparison with State-of-the-art Methods

SIFA is compared with unsupervised adaptation methods using image alignment, feature alignment, or both. It significantly outperforms the comparison approaches, while methods combining both alignment types perform better than single-perspective methods.

  • Compared Methods: The comparison includes PnP-AdaNet, SynSeg-Net, AdaOutput, CycleGAN, and CyCADA under a common segmentation-network architecture.PnP-AdaNet results on the cardiac dataset are taken from its paper because it used the same dataset and backbone.
  • Compared Methods: AdaOutput uses feature alignment, CycleGAN uses image alignment, and CyCADA combines image and feature alignment.PnP-AdaNet and SynSeg-Net are dedicated to cross-modality MRI/CT segmentation using feature and image alignment, respectively.
  • Results: SIFA significantly outperforms the other comparison approaches by a large margin on the reported quantitative and visual evaluations.Quantitative results are presented in Tables I and II, while visual comparisons appear in Figures 3 and 4.
  • Results: Methods using both image and feature alignments obtain better segmentation results than methods adopting only one alignment perspective.The passage specifically identifies CyCADA and SIFA as the methods using both alignments.

E. Ablation Analysis of Key Components

Ablations show that combining image alignment with deeply supervised feature alignment progressively improves cardiac MRI-to-CT adaptation. Broader experiments indicate consistent gains across tasks and directions, while 2D processing and balanced-domain assumptions remain limitations.

  • Component ablation: Average Dice rose from 58.0% with image alignment alone to 67.7% after adding semantic-space feature alignment.The added components were FA-P1 and FA-P2.
  • Component ablation: 74.1% average Dice was achieved after additionally applying feature alignment in the generated image space, defining the full SIFA model.The progressive gains support joint alignment across image and feature spaces.
  • Comparison with prior SIFA: SIFA consistently improved average Dice and reduced ASD versus prior SIFA, especially for small or irregular structures such as LAC, MYO, and spleen.The authors attribute this improvement to auxiliary adversarial learning in semantic prediction space.
  • Comparison with prior SIFA: SIFA produced higher discriminator loss than prior SIFA, indicating that segmentation-prediction feature distributions were harder to distinguish by domain.The comparison concerns discriminator Dp1 on cardiac MRI-to-CT and CT-to-MRI adaptation.
  • Cross-task evaluation: The method improved target-modality segmentation across cardiac and abdominal tasks without task-specific architecture or hyperparameter tuning.The performance gap to supervised training was larger for cardiac data, where contrast and image quality were more challenging.
  • Bidirectional adaptation: Adaptation improved in both MRI-to-CT and CT-to-MRI directions, but cardiac adaptation was easier from MRI to CT while abdominal results were similarly high.The authors suggest adaptation difficulty may depend more on the task than on direction.
  • Limitations: The framework uses 2D networks for volumetric segmentation because combining image and feature alignment in 3D is memory intensive and training prohibitive.The study also assumes relatively balanced source and target data, leaving imbalanced and target-limited settings for future work.

VI. CONCLUSION

SIFA combines image and feature alignment through adversarial learning and a shared encoder for unsupervised medical-image domain adaptation. Experiments on bidirectional MRI–CT adaptation across cardiac and abdominal segmentation tasks demonstrate improved target-domain segmentation.

  • VI. CONCLUSION: SIFA synergistically combines image and feature alignment through adversarial learning and a shared feature encoder during end-to-end training.The framework targets unsupervised adaptation for medical image segmentation.
  • VI. CONCLUSION: Experiments on two challenging multi-class tasks validate unpaired bidirectional adaptation between MRI and CT.The evaluated tasks are cardiac substructure segmentation and abdominal multi-organ segmentation.
  • VI. CONCLUSION: SIFA improves segmentation performance in the unlabeled target domain.This is the paper’s reported conclusion across the evaluated tasks.
Loading 2002.02255v1…