Source-linked AI summary
Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation
Cheng Chen, Qi Dou, Hao Chen, Jing Qin, Pheng-Ann Heng
TL;DR
Domain shift can substantially degrade medical image segmentation, while labeled target-domain data are expensive to obtain. SIFA jointly adapts image appearance and feature representations using shared encoders and adversarial learning, recovering performance in unpaired MR-to-CT cardiac segmentation.
Problem
Cross-modality domain shifts degrade DCNN performance on medical images, while annotating each new target domain is prohibitively expensive.
Method
SIFA jointly transforms source-image appearance toward the target domain and learns domain-invariant segmentation features with shared encoder layers and adversarial losses.
Results
SIFA recovered average Dice from 17.2% without adaptation to 73.0% and reduced average ASD to 8.1 in MR-to-CT cardiac segmentation.
Takeaways & Limitations
SIFA outperformed previous methods, including CyCADA, on Dice and ASD for cross-modality cardiac structure segmentation.
Abstract
from arXiv · showhide
This paper presents a novel unsupervised domain adaptation framework, called Synergistic Image and Feature Adaptation (SIFA), to effectively tackle the problem of domain shift. Domain adaptation has become an important and hot topic in recent studies on deep learning, aiming to recover performance degradation when applying the neural networks to new testing domains. Our proposed SIFA is an elegant learning diagram which presents synergistic fusion of adaptations from both image and feature perspectives. In particular, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features towards the segmentation task. The feature encoder layers are shared by both perspectives to grasp their mutual benefits during the end-to-end learning procedure. Without using any annotation from the target domain, the learning of our unified model is guided by adversarial losses, with multiple discriminators employed from various aspects. We have extensively validated our method with a challenging application of cross-modality medical image segmentation of cardiac structures. Experimental results demonstrate that our SIFA model recovers the degraded performance from 17.2% to 73.0%, and outperforms the state-of-the-art methods by a significant margin.
Introduction
Medical imaging creates severe cross-modality domain shifts that degrade DCNN segmentation, while acquiring labeled target-domain data is prohibitively expensive. SIFA addresses this challenge by jointly adapting image appearance and feature representations within one adversarial framework.
- Different imaging modalities produce substantially different appearances, causing DCNNs trained on MR images to fail on CT images.
- Target-domain annotation can recover performance but is prohibitively expensive because medical labeling requires expertise.
- Unsupervised domain adaptation seeks competitive target-domain performance using only source-domain annotations and unlabeled target data.
- Image adaptation aligns domain appearances through pixel-to-pixel transformations, typically constrained by cycle consistency to preserve image contents.
- Feature adaptation extracts domain-invariant representations by adversarially discriminating source and target feature distributions, including compact semantic or image spaces.
- SIFA synergistically combines image transformation and feature adaptation, sharing the encoder so both perspectives benefit during unified end-to-end training.
Related Work
Prior unsupervised adaptation methods address domain shift at the image or feature level, with newer work exploring their combination. SIFA targets the severe cross-modality medical segmentation setting where either strategy alone may be insufficient.
- Image-level adaptation: Image-level methods transform source or target images so models can operate across domains at the DCNN input level.
- Feature-level adaptation: Feature-level methods reduce domain shift by learning domain-invariant features through statistical alignment or adversarial learning.
- Feature-level adaptation: Recent approaches use compact semantic prediction or image spaces for discriminator-based feature adaptation.
- Combined adaptation: Image and feature adaptations are complementary, motivating combined strategies such as CyCADA for semantic segmentation adaptation.
- Combined adaptation: In severe cross-modality medical images, adaptation from either perspective alone may be insufficient, motivating SIFA’s unified combination of both processes.
Methods
SIFA unifies image-level appearance alignment and feature-level domain-invariance learning for unsupervised cross-modality medical image segmentation. A shared encoder connects both adaptations in an end-to-end framework trained with adversarial, cycle-consistency, and segmentation objectives.
- Image Adaptation for Appearance Alignment: A GAN generator Gt transforms labeled source images into target-like images, while discriminator Dt distinguishes transformed images from real target images.The adversarial game aligns source-image appearance with the target-domain distribution.
- Image Adaptation for Appearance Alignment: Cycle consistency uses reverse reconstruction to preserve the original semantic contents of source-to-target and target-to-source image transformations.The reconstruction constraints encourage U(E(Gt(xs))) ≈ xs and Gt(U(E(xt))) ≈ xt.
- Image Adaptation for Appearance Alignment: The segmentation network applies classifier C to encoder features from adapted source images and trains predictions with a hybrid cross-entropy and Dice loss.The hybrid loss balances its two components using α to address class imbalance in medical image segmentation.
- Feature Adaptation for Domain Invariance: Feature adaptation uses discriminators in compact semantic-prediction and generated-image spaces to reduce domain-specific information in encoder features.Dp compares segmentation outputs from adapted source and real target images, while Ds uses generated source-like images to supervise feature invariance.
- Synergistic Learning Diagram: SIFA shares feature encoder E across image and feature adaptations, allowing both perspectives to contribute gradients during unified end-to-end training.The encoder receives image-adaptation losses and feature-adaptation gradients in a multi-task learning scenario.
Experimental Results
Experiments on unpaired MR-to-CT cardiac segmentation show that SIFA combines image and feature adaptation to improve performance under severe domain shift. Ablations and visual comparisons indicate complementary gains from the adaptation components.
- Comparison with state-of-the-art methods: Average Dice rises from 17.2% without adaptation to 73.0% with SIFA, while average ASD falls to 8.1.
- Comparison with state-of-the-art methods: SIFA outperforms prior unsupervised adaptation methods by a large margin on both Dice and ASD.
- Effectiveness of key components: Image adaptation alone increases average Dice to 58.0%, showing that transforming source images toward the target appearance improves segmentation.
- Effectiveness of key components: Adding semantic-space feature adaptation raises average Dice from 58.0% to 65.7%, supporting complementary contributions from image and feature adaptation.
- Effectiveness of key components: Further feature adaptation in generated-image space improves SIFA predictions, producing clearer cardiac shapes and less noisy masks.
Conclusion
SIFA combines image appearance transformation with domain-invariant feature learning for unsupervised cross-modality medical image segmentation. On unpaired MR-to-CT cardiac segmentation, it outperforms competing methods on both Dice and ASD.
- SIFA simultaneously transforms image appearance and learns domain-invariant features through adversarial learning with partial parameter sharing.The unified network exploits mutual benefits between image and feature adaptations during end-to-end training.
- The method was validated on unpaired MR-to-CT adaptation for cardiac segmentation against various state-of-the-art methods.
- SIFA achieved superior segmentation performance to competing methods in both Dice and ASD.
- The framework is general and can be extended to other unsupervised domain-adaptation segmentation applications.