Source-linked AI summary
Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation
Yue Zhang, Shun Miao, Tommaso Mansi, Rui Liao
TL;DR
The paper addresses multi-organ X-ray segmentation when pixel-level X-ray labels are difficult to obtain. It trains DI2I on labeled CT-derived DRRs and uses TD-GAN for task-driven translation and segmentation on unlabeled X-rays, achieving performance close to supervised training.
Problem
Pixel-level X-ray annotation is difficult and time-consuming, motivating segmentation from labeled CT-derived synthetic data.
Method
DI2I is trained on labeled DRRs, while TD-GAN combines cycle-GAN translation with DI2I-based segmentation supervision for unlabeled X-rays.
Results
The proposed model significantly improves topogram segmentation over direct DI2I application and reaches the same accuracy level as supervised training.
Takeaways & Limitations
The framework is general and can be adapted to other tasks by replacing the supervision network.
Takeaways & Limitations
The pre-trained DI2I must remain frozen during TD-GAN training to avoid disturbed supervision from fake DRRs.
Abstract
from arXiv · showhide
Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 85% which achieves the same level accuracy of supervised training (88%).
1 Introduction
The paper targets multi-organ segmentation in X-ray images without pixel-level X-ray labels, using labeled CT-derived synthetic data and task-driven domain adaptation. Its framework combines image translation with segmentation supervision for real X-ray parsing.
- X-ray multi-organ segmentation is clinically important for diagnosis, treatment evaluation, surgical planning, and automated workflows.
- Projective imaging creates overlapping anatomy, fuzzy boundaries, and complex textures, making accurate pixel-level X-ray annotation difficult and time-consuming.
- The framework trains a DI2I segmentation network on labeled DRRs generated from segmented CT scans, then adapts it to real X-ray images.
- TD-GAN combines modified cycle-GAN translation between DRRs and X-rays with DI2I-based segmentation consistency supervision.
- The proposed pipeline uses unpaired, totally unlabeled X-ray images and is designed for unsupervised domain adaptation in segmentation.
2 Methodology
The methodology combines DI2I segmentation trained on labeled DRRs with TD-GAN for pixel-to-pixel translation and segmentation between DRRs and real X-ray images. Task-driven losses use DI2I predictions and cycle segmentation consistency to preserve organ information during adaptation.
- Dense Image to Image Network for Segmentation on DRRs: DI2I is trained on pixel-wise labeled DRRs generated from segmented CT scans for multi-organ segmentation.The target organs are lung, heart, liver, and bone.
- Dense Image to Image Network for Segmentation on DRRs: The DI2I uses a dense-block encoder-decoder UNet whose five output channels represent background and four organs.Dense blocks iteratively concatenate feature outputs, while separate organ channels simplify overlapped-organ segmentation into binary classifications.
- Task Driven Generative Adversarial Networks (TD-GAN): TD-GAN modifies cycle-GAN with two generators, two discriminators, and four paths for translation and reconstruction between DRRs and X-ray images.The paths are real DRR → fake X-ray, real X-ray → fake DRR, real X-ray → reconstructed X-ray, and real DRR → reconstructed DRR.
- Task Driven Generative Adversarial Networks (TD-GAN): The real X-ray → fake DRR path feeds DI2I-predicted organ probability maps with generated DRRs to the discriminator for conditional adversarial training.The probability maps remain unbinarized so the loss remains differentiable, and the discriminator evaluates image-label pair realism.
- Task Driven Generative Adversarial Networks (TD-GAN): The pretrained DI2I remains frozen during TD-GAN training so generated DRRs do not disturb its supervision.The framework can be adapted to other tasks by replacing the pretrained task network U(·).
- Task Driven Generative Adversarial Networks (TD-GAN): The real DRR → reconstructed DRR path adds segmentation loss to cycle reconstruction, requiring the reconstructed DRR to retain DI2I segmentation performance.The total TD-GAN objective is a weighted sum of the losses across the paths.
3 Experiments and Results
Experiments evaluate DI2I and TD-GAN on labeled DRRs and topograms, including direct, cycle-GAN, task-driven, and supervised comparisons. TD-GAN improves topogram segmentation over direct DI2I and reaches supervised-training accuracy.
- Dataset and setup: 815 labeled DRRs and 153 topograms form the evaluation dataset.Topogram labels are used only for evaluation, while training uses DRRs and an unlabeled topogram subset.
- DI2I on DRRs: DI2I achieves mean Dice scores of 0.9417 for lung, 0.923 for heart, 0.894 for liver, and 0.910 for bone on DRRs.Results are reported as mean ± standard deviation using five-fold cross-validation.
- Qualitative results: Figure 4 compares topogram predictions from direct DI2I and TD-GAN against red ground-truth boundaries.Colored regions represent predictions, with the direct DI2I results shown above and TD-GAN results below.
- Topogram segmentation: TD-GAN significantly improves segmentation over direct DI2I application on topograms and reaches the same accuracy level as supervised training.The direct application of learned DI2I fails completely on topograms; supervised training uses topogram labels under the same data split.
- Model comparison: TD-GAN-A and TD-GAN-S outperform cycle-GAN, while the final TD-GAN achieves the best performance among the compared models.Cycle-GAN performs image style transfer only, whereas the task-driven variants incorporate segmentation-related design.
4 Discussions and Conclusions
The paper addresses multi-organ segmentation on unlabeled X-ray images using labeled DRRs. It combines image style transfer with task-driven segmentation and presents the framework as adaptable to other computer-aided diagnosis tasks.
- Conclusions: The framework uses a cycle-GAN substructure for style transfer and add-on modules for simultaneous organ segmentation.The approach is applied to multi-organ segmentation over totally unlabeled X-ray images with labeled DRRs.
- Conclusions: Replacing DI2I with other supervision networks can adapt the framework to lesion classification, landmark localization, and abnormal motion detection.These extensions are identified as future directions.