Source-linked AI summary
Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations
Chen Qin, Bibo Shi, Rui Liao, Tommaso Mansi, Daniel Rueckert, Ali Kamen
TL;DR
Multi-modal registration is difficult because modalities can have complex intensity relationships and features that do not appear in both images. This paper learns registration from disentangled shared shape representations, reporting improved accuracy and speed against conventional approaches. Its reported experiments are on 2D images, with real 3D application left for future work.
Problem
Multi-modal registration is challenging because image modalities have unknown, complex intensity relationships and may contain features missing from the other modality.
Source: §1 · p. 2Method
The method disentangles images into shared latent shape and modality-specific appearance representations, then trains registration with latent-space and adversarial image-space similarity metrics.
Source: §1 · p. 2, §2.2 · p. 5, §2.3 · p. 6Results
The authors report improved accuracy and speed over conventional approaches, with UMDIR outperforming competing methods on Dice, MCD, and HD and matching or improving their Grad Det-Jac range.
Source: §3.3 · p. 10, §4 · p. 11Takeaways & Limitations
The learned registration function supports one-shot bidirectional registration at noticeably higher processing speed than MIND's direction-specific training.
Source: §1 · p. 2, §3.3 · p. 11The method is demonstrated on 2D images; extending it to real 3D volumes using patch-based methods remains future work.
Source: §1 · p. 2, §4 · p. 12
Abstract
from arXiv · showhide
We propose a fully unsupervised multi-modal deformable image registration method (UMDIR), which does not require any ground truth deformation fields or any aligned multi-modal image pairs during training. Multi-modal registration is a key problem in many medical image analysis applications. It is very challenging due to complicated and unknown relationships between different modalities. In this paper, we propose an unsupervised learning approach to reduce the multi-modal registration problem to a mono-modal one through image disentangling. In particular, we decompose images of both modalities into a common latent shape space and separate latent appearance spaces via an unsupervised multi-modal image-to-image translation approach. The proposed registration approach is then built on the factorized latent shape code, with the assumption that the intrinsic shape deformation existing in original image domain is preserved in this latent space. Specifically, two metrics have been proposed for training the proposed network: a latent similarity metric defined in the common shape space and a learningbased image similarity metric based on an adversarial loss. We examined different variations of our proposed approach and compared them with conventional state-of-the-art multi-modal registration methods. Results show that our proposed methods achieve competitive performance against other methods at substantially reduced computation time.
1 Introduction
Multi-modal registration is important for combining complementary medical-image information, but differing intensity relationships and missing features make alignment difficult. The paper proposes an unsupervised approach that learns registration from disentangled shape representations and reports competitive accuracy with faster processing.
1 Introduction
- Medical modalities capture complementary tissue information, but different scanners and acquisition times can yield images with anatomical changes that need registration.Source: §1 · pp. 1–2
- Registration is challenging because modalities have unknown intensity relationships and may contain features absent from the other image.Existing approaches use mutual information or landmarks, but the former often ignores spatial information and the latter may be difficult or impossible to localize.Source: §1 · p. 2
- The proposed unsupervised method learns multi-modal deformable registration without ground-truth deformation fields, aligned image pairs, or anatomical landmarks.It factorizes images into domain-invariant shape and domain-specific appearance representations, then learns registration in the shared shape space.Source: §1 · p. 2
- The authors report accuracy competitive with state-of-the-art methods and substantially faster processing.They describe the method as a one-shot bidirectional solution.Source: §1 · p. 2
- Prior approaches synthesize a modality or map images to a common domain, while unsupervised deep registration methods mainly address mono-modal images.Information-theoretic methods such as normalized and local mutual information can overlook anatomical spatial information.Source: §1 · p. 3
2 Proposed Method
The method reduces unsupervised multi-modal registration to registration in a domain-invariant latent space, without ground-truth deformation fields, landmarks, or aligned multi-modal training pairs. It disentangles shared shape from modality-specific appearance, then combines latent-space alignment with an adversarial image-space similarity metric.
2 Proposed Method
- The framework has three components: unpaired image-to-image translation for disentangling, latent-space deformable registration, and an adversarial network that learns image-space similarity.It targets training without ground-truth deformation fields, anatomical landmarks, or aligned multi-modal image pairs.Source: §2 · p. 3
2.1 Image disentangling via unpaired image-to-image translation
- The translation network separates domain-invariant shape from domain-specific appearance and uses reconstruction, adversarial, and cross-cycle consistency losses.Cross-cycle consistency is added to better preserve shape information.Source: §2.1 · pp. 4–5
2.2 Multi-modal image registration via disentangled representations
- A diffeomorphic registration network predicts a velocity field from latent shape representations and aligns the modalities bidirectionally, with deformation-gradient penalties for smoothness.The deformation is defined as the exponential map of the velocity field.Source: §2.2 · pp. 5–6
2.3 Multi-modal image registration via combined metrics
- The combined objective adds a patch-GAN image-space similarity metric to latent-space alignment to address the possibility that latent similarity misses detailed deformations.The discriminator is trained to distinguish well-aligned from registered image patches, using cross-cycle consistency to reduce the influence of translation-distribution mismatch.Source: §2.3 · pp. 6–7
3 Experiments
The experiments evaluate UMDIR on COPDGene data with a synthesized modality and BraTS T1/T2 images, using established multimodal baselines and task-specific measures. The proposed methods outperform conventional baselines on COPDGene accuracy, are competitive on BraTS, and provide faster processing; combined similarity metrics further improve performance in reported comparisons.
3.1 Datasets
- COPDGene uses end-inspiration and end-expiration lung CT with a synthesized modality, while BraTS uses aligned T1 and T2 images with synthetic deformations reserved for evaluation ground truth.The experiments use COPDGene data to assess clinical deformations and BraTS to assess real multimodal images.Source: §3.1 · pp. 7–8
3.2 Experimental Settings
- Evaluation compares mutual-information, MIND, diffeomorphic Demons with learned image translation, and UMDIR variants, using dataset-specific accuracy measures plus deformation smoothness and runtime.The methods and parameters were selected or tuned using validation-set performance; UMDIR-Lat, UMDIR-GAN, and UMDIR-LaGAN combine or use different proposed similarity criteria.Source: §3.2 · pp. 8–9
3.3 Results
- On COPDGene, UMDIR outperforms traditional and image-translation-plus-registration baselines on Dice, MCD, and HD, with Grad Det-Jac no larger than or in the same range as competitors.Combining latent and cross-cycle adversarial similarity further improves accuracy, and UMDIR methods are faster than traditional baselines and I2I+GAN.Source: Table 1 · p. 9, §3.3 · p. 10
- On BraTS, UMDIR is competitive with other methods and produces smooth, regular deformations; its latent method registers both directions without further training.MIND attains lower RMSE(I) for T1 →T2, but requires separate training per direction; UMDIR offers one-shot bidirectional registration at noticeably higher processing speed.Source: Table 2 · p. 10, §3.3 · pp. 10–11
- For T1-to-T2 registration, the combined metrics outperform either metric alone, while error-map arrows mark regions where UMDIR-Lat or UMDIR-GAN performs better.The figure compares pixel-wise intensity differences between warped images and target moving images.Source: Figure 4 · p. 11, §3.3 · p. 11
4 Conclusion
The paper presents fully unsupervised multimodal deformable registration using disentangled latent representations and latent- and image-space similarity losses. The models improve on conventional approaches in reported accuracy and speed, while future work will examine joint training and real 3D-volume application.
4 Conclusion
- The fully unsupervised model reduces multimodal registration to a monomodal problem using disentangled latent embeddings learned through unpaired image translation.Registration training uses a distance loss in latent shape space and a cross-cycle adversarial loss in image space.Source: §4 · p. 11
- Experiments show improved accuracy and speed over conventional approaches, while future work proposes investigating joint MUNIT–UMDIR training to refine domain-invariant representations.The authors also propose extending the application to real 3D volumes using patch-based methods for efficient training.Source: §4 · pp. 11–12