Source-linked AI summary
Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training
Faisal Mahmood, Richard Chen, Nicholas J. Durr
TL;DR
Medical imaging lacks large annotated datasets, and conventional synthetic-to-real adaptation is difficult for diverse human tissues. The paper reverses the adaptation direction, transforming real endoscopy images into synthetic-like representations while self-regularization preserves clinically relevant features. Depth estimation improved by 88% on porcine colon data and 48% on colon phantom data over raw images.
Problem
Large annotated medical-image datasets are difficult to acquire, while making synthetic images realistic is hard for diverse patient-specific tissue features.
Method
The method trains a reverse domain-adaptation transformer on synthetic and unlabeled real endoscopy images, using adversarial loss and self-regularization.
Results
88% SSIM improvement for porcine colon data and 48% for colon phantom data were obtained by transforming inputs into synthetic-like representations.
Takeaways & Limitations
Synthetic-like domain-adapted endoscopy images enabled a depth network trained on synthetic data to predict depth better from real endoscopy images.
Takeaways & Limitations
The synthetic-data framework requires forward models and anatomically accurate organ models that represent substantial biological variation and rare conditions.
Abstract
from arXiv · showhide
To realize the full potential of deep learning for medical imaging, large annotated datasets are required for training. Such datasets are difficult to acquire because labeled medical images are not usually available due to privacy issues, lack of experts available for annotation, underrepresentation of rare conditions and poor standardization. Lack of annotated data has been addressed in conventional vision applications using synthetic images refined via unsupervised adversarial training to look like real images. However, this approach is difficult to extend to general medical imaging because of the complex and diverse set of features found in real human tissues. We propose an alternative framework that uses a reverse flow, where adversarial training is used to make real medical images more like synthetic images, and hypothesize that clinically-relevant features can be preserved via self-regularization. These domain-adapted images can then be accurately interpreted by networks trained on large datasets of synthetic medical images. We test this approach for the notoriously difficult task of depth-estimation from endoscopy. We train a depth estimator on a large dataset of synthetic images generated using an accurate forward model of an endoscope and an anatomically-realistic colon. This network predicts significantly better depths when using synthetic-like domain-adapted images compared to the real images, confirming that the clinically-relevant features of depth are preserved.
1. Introduction
Medical imaging lacks large annotated datasets because collection and annotation are costly, time-intensive, privacy-constrained, and expertise-dependent. The paper reverses conventional synthetic-to-real adaptation by transforming real images into synthetic-like representations while preserving clinically relevant information.
- Large annotated medical-image datasets remain scarce because data collection, annotation, and dissemination involve cost, time, expertise, privacy, and regulatory barriers.
- Synthetic data can address annotation shortages, but adapting synthetic images toward realism is difficult for medical images containing diverse patient-specific tissue features.
- Cross-patient generalization is difficult because endoscopy combines patient-specific low-level texture with clinically relevant high-level texture.Vascular patterns are described as non-diagnostic, whereas high-level texture should generalize across patients.
- The proposed reverse adaptation transforms real medical images to look synthetic while preserving clinically relevant information, bridging synthetic source and real target domains.
- The study generates perfectly annotated synthetic endoscopy data and evaluates adaptation using monocular depth estimation on phantom, porcine, and human endoscopy data.
2. Related Work
Related work addresses limited medical-image data through augmentation, prior information, transfer learning, GANs, and adversarial biomedical imaging. However, medical applications remain constrained by limited training data and tissue diversity.
- Improving deep-learning performance with limited medical-imaging data remains an active research area.
- Existing approaches include elastic augmentation, application-specific prior information, and transfer of knowledge from conventional computer vision.
- GANs train competing generator and discriminator networks, with the discriminator classifying generated images as real or fake and feeding back to the generator.
- Adversarial training has been applied to biomedical noise reduction, segmentation, detection, reconstruction, classification, and image synthesis.
3. Generating Synthetic Medical Data
The framework generates synthetic medical images by combining accurate device modeling, anatomically accurate organ modeling, and varied rendering conditions. Endoscopy demonstrates the approach with pixel-wise depth supervision, while anatomical and device complexity constrain realism.
- Synthetic medical data is generated by modeling the imaging device, modeling the organ anatomically, and rendering images across positions, angles, and parameters.
- Medical forward models and anatomical models must represent greater complexity and variation than typical cameras and object models.
- The endoscopy dataset uses a wide-angle monocular camera, two to three light sources, an anatomically accurate colon, varied virtual-endoscope conditions, and pixel-wise ground-truth depth.
- 260,000 rendered endoscopy images with ground-truth depth train efficient depth networks, but those networks do not generalize effectively to real-world images.
4. Reverse Domain Adaptation
Reverse domain adaptation trains a transformer to make real images synthetic-like while retaining image properties through self-regularization. Adversarial discrimination, pixel-level similarity, and stabilized alternating updates define the training pipeline.
- The transformer maps real images to synthetic-like representations, using adversarial training to reduce the gap between real and synthetic domains.
- The transformer loss combines an adversarial term that promotes synthetic-like outputs with self-regularization that penalizes large deviations from the real image.The self-regularization weight λ controls how strongly deviations are penalized.
- The discriminator classifies synthetic images and transformed real images, operating at patch level to help prevent artifacts.
- Training alternates transformer and discriminator updates using mini-batches of synthetic and transformed real images.Buffered transformed outputs are used to increase adversarial-training stability.
- At equilibrium, the transformer aims to fool the discriminator while keeping transformed images close to their real inputs.
5. Depth Estimation from Monocular Endoscopy Images
The method trains a CNN-CRF depth estimator on synthetic endoscopy images with ground-truth depths, then evaluates reverse domain adaptation across three target domains. Its energy model combines unary superpixel depth regression with pairwise similarity-based smoothness and is optimized using regularized stochastic-gradient backpropagation.
- The evaluation trains a depth-estimation network on synthetically generated endoscopy data and tests adaptation to three target domains.The stated goal is to bring target-domain distributions closer to the synthetic source domain through adversarial training.
- The depth estimator uses a CNN-CRF framework in which each synthetic image is divided into superpixels with a depth value for each superpixel.The synthetic training images include ground-truth depths for supervision.
- Depth prediction solves a maximum a posteriori problem using a conditional distribution defined by an energy function.The energy function determines the probability of candidate depth vectors given an image.
- The energy combines unary potentials that regress individual superpixel depths with pairwise potentials that encourage smoothness between neighboring superpixels.The unary network processes a single superpixel patch, while the pairwise component uses network parameters and a superpixel similarity matrix.
- Intensity differences and grayscale histograms provide pairwise similarities because inverse intensity is a valuable cue for endoscopic depth estimation.These similarities are expressed using a general ℓ2 form.
- The objective minimizes negative log likelihood with regularization penalties on the learned parameter vectors and is solved by stochastic-gradient backpropagation.The regularization terms penalize heavily weighted vectors through λθ and λβ.
6. Experiments
The experiments evaluate synthetic-trained depth estimation across colon phantom, porcine colon, and human endoscopy data, using domain adaptation for real images. Domain adaptation improves quantitative depth estimation, especially for biologically variable porcine tissue, while human-data reconstructions remain qualitative.
- Datasets: Quantitative evaluation uses 2,160 colon phantom images, 1,400 CT-registered porcine colon images, and publicly available human colonoscopy images without ground-truth depth.The phantom and porcine datasets support quantitative evaluation; human images support qualitative assessment only.
- Data generation: Figure 4 derives phantom ground truth from a 3D model and porcine ground truth by registering optical endoscopy with CT-based views.The two acquisition pipelines provide corresponding depth maps for quantitative evaluation.
- Network and adaptation: The depth-estimation network is trained on synthetic endoscopy data, while real test images are transformed into synthetic-like representations before inference.The transformer uses reverse domain adaptation, and the synthetic-trained depth estimator therefore expects synthetic-like inputs.
- Reverse domain adaptation: Patient-specific texture is removed from transformed endoscopy images while shape information is preserved for depth estimation and polyp identification.The examples specifically show vasculature removal alongside preservation of clinically relevant shape information.
- Quantitative results: 88% SSIM improvement occurs for porcine colon data and 48% for colon phantom data after transforming inputs with the proposed domain-adaptation method.The reported improvement is marginal for homogeneous phantom material and more significant for naturally variable porcine tissue.
- Qualitative results: Human endoscopy depth estimates and topographical reconstructions are demonstrated qualitatively because corresponding ground-truth depth is unavailable.The human-data examples show the complete transformation, depth-estimation, and surface-reconstruction pipeline without quantitative validation.
7. Conclusions and Future Work
The paper proposes reverse domain adaptation for transforming real medical images into synthetic representations while preserving clinically relevant features, and validates it for endoscopic depth estimation.
- 88% and 48% improvement in depth prediction over raw images was achieved for a real porcine colon and a colon phantom, respectively.The evaluation used synthetic training data and synthetic-like domain-adapted real images.
- The method was validated on monocular endoscopy depth estimation using a large synthetic dataset for learning depth.
- Future work will apply reverse domain adaptation to other medical imaging modalities and use predicted depth for automated polyp segmentation and classification.