Source-linked AI summary
Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation
Moab Arar, Yiftach Ginger, Dov Danon, Ilya Leizerson, Amit Bermano, Daniel Cohen-Or
TL;DR
Multi-modal registration must recover spatial correspondence despite difficult cross-modality similarity measures and differing sensor appearances. The paper jointly trains a spatial transformation network with a geometry-preserving image translator, enabling mono-modality metrics and accurate unsupervised registration across modality pairs. Its scope includes paired same-scene inputs and a proposed future direction for geometry-preserving generators beyond registration.
Problem
Cross-modality similarity measures are difficult to design for registration, while differing sensor parameters make spatial alignment necessary.
Method
The method jointly trains a spatial transformation network and an image-to-image translation network through two flows that encourage translation to preserve geometry.
Results
The method produces smooth and accurate registration using adversarial learning and mono-modality metrics, even with little training data.
Takeaways & Limitations
Learning translation replaces direct cross-modality comparison with simple mono-modality metrics and can adapt to any pair of modalities.
Takeaways & Limitations
The method assumes paired images representing the same scene, captured by different modality sensors.
Abstract
from arXiv · showhide
Many applications, such as autonomous driving, heavily rely on multi-modal data where spatial alignment between the modalities is required. Most multi-modal registration methods struggle computing the spatial correspondence between the images using prevalent cross-modality similarity measures. In this work, we bypass the difficulties of developing cross-modality similarity measures, by training an image-to-image translation network on the two input modalities. This learned translation allows training the registration network using simple and reliable mono-modality metrics. We perform multi-modal registration using two networks - a spatial transformation network and a translation network. We show that by encouraging our translation network to be geometry preserving, we manage to train an accurate spatial transformation network. Compared to state-of-the-art multi-modal methods our presented method is unsupervised, requiring no pairs of aligned modalities for training, and can be adapted to any pair of modalities. We evaluate our method quantitatively and qualitatively on commercial datasets, showing that it performs well on several modalities and achieves accurate alignment.
1. Introduction
Multi-modal registration requires spatial correspondence despite differing sensor appearances, while cross-modality similarity measures are difficult to design and optimize. The paper addresses this with unsupervised translation-based registration using a geometry-preserving translator and mono-modality metrics.
- Multi-modal sensors provide richer scene representations but often have different extrinsic parameters, making non-rigid registration necessary.
- Classic methods optimize nonlinear spatial transformations with cross-modality similarity measures, whose design and computation remain challenging.
- Supervised registration commonly relies on synthesized data because ground-truth registration parameters are difficult to collect.
- Mono-modality metrics such as MSE are unsuitable across modalities, motivating the use of statistical measures including NMI, NCC, and SSIM.
- The proposed unsupervised method learns translation between modalities so registration can use mono-modality metrics instead of hand-crafted cross-modality measures.
- Its main challenge is jointly training registration and translation networks while making translation geometry preserving through adversarial training.
2. Related Works
Related methods address multi-modal registration through domain translation, modality-independent features, adversarial losses, or disentangled representations, but each leaves specific alignment or fidelity concerns.
- CycleGAN-based approaches regularize deformation mappings with adversarial discrimination between deformed and real images.
- Encoder-decoder methods create modality-independent features for STN-based transformations, but MSE preserves local similarity without enforcing global fidelity.
- Adversarial-only STN training produces realistic registered images but loses local coherence with the target.
- A disentangled embedding approach combines adversarial and similarity losses, while learned disentanglement can introduce local inconsistencies.
3. Overview
The method learns modality translation alongside spatial registration, replacing cross-modality comparison with mono-modality evaluation. Two jointly trained flows encourage translation to preserve geometry while registration carries the deformation.
- The core idea is to learn translation between modalities rather than use a cross-modality metric.
- The translated spatially transformed image can be compared with the source using a uni-modality metric.
- The approach is designed to generalize and adapt to any pair of given modalities.
- Registration uses a spatial transformation network and an image-to-image translation network trained simultaneously through two flows.
- The same deformation field is applied either before translation or after translation, encouraging geometry to remain encoded in the spatial transformation network.
- At test time, only the spatial transformation network aligns paired images from different modalities of the same scene.
4. Method
The method jointly learns a non-rigid registration network and an image-to-image translator, using two compositions to encourage geometry-preserving translation. Reconstruction, adversarial, and smoothness objectives train the system so registration handles spatial alignment while translation handles photometric differences.
- Architecture: The method trains a registration network R, translation network T, and discriminator D jointly for unsupervised multimodal alignment.R predicts a deformation field, T translates between modalities, and D supports adversarial training.
- Architecture: R is a spatial transformation network with a fully convolutional deformation-field generator and a re-sampling layer for non-rigid alignment.The deformation field assigns a two-dimensional displacement to each input pixel, and the re-sampler applies it to a source image.
- Registration regularization: Bilateral smoothness regularization encourages neighboring pixels to receive similar deformations while reducing oversmoothing near image boundaries.The bilateral weights are computed from the transformed image on each forward pass and treated as constant during backpropagation.
- Geometry-preserving translation: Geometry preservation is encouraged because translation can otherwise generate images aligned with the ground truth even when registration is inaccurate.Reducing the translator’s capacity improves registration performance but still prevents R from performing all registration.
- Geometry-preserving translation: The two training flows apply translation before registration or registration before translation, using the same deformation field in both compositions.Their outputs are constrained toward the target modality through an L1 reconstruction loss, encouraging T and R to commute.
- Training objectives: Conditional adversarial training jointly optimizes T and R to produce realistic target-domain samples, alongside reconstruction and smoothness objectives.The implementation uses encoder-decoder and U-NET-based architectures with residual connections and Instance Normalization.
5. Experimental Results
Experiments evaluate the method on a commercial RGB/IR/depth dataset using landmark-based registration accuracy, qualitative alignments, and ablations of translation, loss, and smoothness components. The results support geometry-preserving translation, combined losses, and bilateral filtering as important for accurate registration.
- Dataset and evaluation: The commercial dataset contains RGB, IR, and depth images of banana plants captured by calibrated sensors.The dataset contains 6100 frames, each with RGB, IR, and depth images.
- Dataset and evaluation: Registration accuracy is measured from manually annotated landmark pairs on salient objects and throughout the scene.The accuracy is based on average Euclidean distance between target points and deformed source points.
- Quantitative evaluation: Prescribed cross-modality similarity measures perform poorly, with NCC producing noisy registrations and SSIM producing smoother but less accurate results.The evaluation compares NCC and edge-based SSIM losses for training the registration network.
- Quantitative evaluation: CycleGAN translation combined with SIFT fails on the dataset because the descriptors are not designed for multi-modal data.The method instead uses learned translation to enable mono-modality comparisons for registration.
- Qualitative evaluation: Qualitative results show successful RGB-to-depth and RGB-to-IR alignment, with deformation concentrated on backgrounds when salient objects already align.Overlaying the target salient object before and after registration provides a visual alignment check.
- Ablation studies: Training both compositions improves registration accuracy over either composition alone, while bilateral filtering improves precision and balances train and test accuracy.Using only reconstruction loss creates noisy artifacts, using only adversarial loss gives inexact alignment, and jointly updating all network weights gives the best accuracy.
6. Summary and Conclusions
The paper presents an unsupervised multi-modal registration method using geometry-preserving image translation and mono-modality metrics. It reports smooth, accurate registration with little training data and identifies geometry-preserving generators as a direction for broader applications.
- The method combines image-to-image translation with mono-modality metrics, avoiding direct comparison between different modalities.A geometry-preserving translation network enables this comparison for registration.
- Adversarial learning with mono-modality metrics produces smooth and accurate registration results even with little training data.
- The authors propose exploring geometry-preserving generators beyond image registration.They also suggest alternating layers or operators across flows to encourage commutativity and achieve non-trivial properties.