Source-linked AI summary
Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration
Di Wang, Jinyuan Liu, Xin Fan, Risheng Liu
TL;DR
Misaligned infrared-visible fusion is hindered by spatial deformation, cross-modality discrepancy, and resulting ghost artifacts. The paper addresses this with cross-modality generation-registration and adaptive feature interaction fusion, reporting superior capability on misaligned cross-modality image fusion and extension to existing IVIF methods.
Problem
Existing learning-based IVIF methods primarily target pre-registered images and can produce serious ghosting on misaligned pairs because cross-modality differences complicate alignment and fusion.
Method
The framework generates geometry-preserving pseudo infrared images with CPSTN, converts alignment into mono-modality registration, and adaptively fuses registered infrared and visible features with IFM.
Results
The proposed method demonstrates superior capability on misaligned cross-modality image fusion, while CPSTN improves VIF by 0.14 on TNO and 0.22 on RoadScene.
Takeaways & Limitations
The generation-registration paradigm can be extended to existing IVIF methods to improve fusion performance on misaligned cross-modality images.
Abstract
from arXiv · showhide
Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy. To overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF). Specifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as input. Benefiting from the favorable geometry preservation ability of the CPSTN, the generated pseudo infrared image embraces a sharp structure, which is more conducive to transforming cross-modality image alignment into mono-modality registration coupled with the structure-sensitive of the infrared image. In this case, we introduce a Multi-level Refinement Registration Network (MRRN) to predict the displacement vector field between distorted and pseudo infrared images and reconstruct registered infrared image under the mono-modality setting. Moreover, to better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM) to adaptively select more meaningful features for fusion in the Dual-path Interaction Fusion Network (DIFN). Extensive experimental results suggest that the proposed method performs superior capability on misaligned cross-modality image fusion.
1 Introduction
Misaligned IVIF remains difficult because cross-modality differences and spatial deformation produce ghosting, while the proposed framework combines cross-modality generation, registration, and adaptive fusion to address these problems.
- Problem: Pre-registered IVIF methods are labor-intensive and time-sensitive, while slight offsets or deformations can cause serious ghosting artifacts.The underlying challenge is the intensity discrepancy between misaligned infrared and visible images.
- Problem: Large cross-modality variation makes it impractical to bridge infrared and visible domains directly in a shared feature space.The absence of cross-modality similarity constraints has limited attempts to fuse misaligned image pairs.
- Approach: CGRP uses cross-modality generation to narrow modality differences before registering the resulting images under a mono-modality setting.The paradigm is motivated by infrared imagery emphasizing sharp geometry structures over texture details.
- Approach: CPSTN generates pseudo infrared images from visible images while using perceptual style transfer and cross regularizations to preserve geometry structure.This supports a unified representation for the distorted infrared image and its pseudo infrared counterpart.
- Contributions: The framework includes an interaction fusion module that adaptively fuses multimodality features, avoiding feature smoothing and emphasizing faithful texture details.The complete method is reported as a robust unsupervised framework for misaligned IVIF with superior experimental capability.
2 Method
The method sequentially generates a pseudo infrared image, registers the distorted infrared image under a mono-modality setting, and adaptively fuses registered infrared and visible features. Its losses jointly constrain perceptual style transfer, cross-cycle consistency, registration, and fusion quality.
- Cross-modality generation and registration: The cross-modality generation-registration paradigm translates visible images into pseudo infrared images to reduce spatial offsets and ghosting in misaligned fusion.CPSTN generates the pseudo infrared image while preserving sharp geometry structure.
- Cross-modality generation and registration: CPSTN generates the pseudo infrared image with a perceptual style-transfer constraint and inter-path cross regularization across two cycle routes.The cross regularization contains content and edge terms, while the perceptual style-transfer loss contains perceptual and style terms.
- Cross-modality generation and registration: MRRN predicts deformation fields between distorted and pseudo infrared images through multi-level coarse-to-fine refinement, then reconstructs the registered infrared image with a resampler.The final deformation field is obtained at the highest feature-extractor level, and spatial transformation performs registration.
- Dual-path interaction fusion: DIFN extracts infrared and visible features through dual paths and uses IFM to adaptively select and recalibrate meaningful features before producing the fused image.The feature interaction module uses feature responses from both modalities to perform adaptive fusion.
- Loss functions: Training minimizes a combined objective comprising perceptual style-transfer, cross regularization, adversarial, registration, and fusion losses.The fusion loss includes MS-SSIM, joint gradient, and self-visual-saliency terms; the registration loss combines similarity and smoothness constraints.
3 Experiments
Experiments evaluate the proposed registration-fusion framework on synthetically misaligned TNO and RoadScene data, showing strong registration, fusion, efficiency, and ablation results. The analyses indicate that CPSTN, MRRN, and their coordinated use improve misaligned IVIF performance.
- Experimental Setup: Experiments use affine and elastic deformations to create misaligned infrared images from pre-registered TNO and RoadScene data.RoadScene provides 221 training images, while TNO is used only for testing because of its limited size.
- Comparison with SOTA: The proposed method ranks first across CC, VIF, and SSIM on both RoadScene and TNO, outperforming eight state-of-the-art fusion methods.On RoadScene, the method reaches CC 0.621, VIF 0.895, and SSIM 0.507; on TNO, it reaches CC 0.481, VIF 1.016, and SSIM 0.473.
- Ablation Studies: CPSTN improves cross-modality registration and fusion, producing 0.14 and 0.22 VIF gains on TNO and RoadScene, respectively.The authors also report reduced distortion and negligible ghosts in qualitative comparisons.
- Ablation Studies: MRRN improves IVIF performance compared with directly fusing misaligned infrared and visible images.The ablation isolates MRRN without CPSTN to assess its contribution to cross-modality registration.
- Additional Analysis: Removing CPSTN perceptual and cross-cycle constraints causes structural degradation, while IFM replacement produces smoother, lower-contrast textures.The full CPSTN yields sharper geometry, and IFM adaptively selects features to produce richer, sharper, higher-contrast fusion results.
- Additional Analysis: CPSTN generalizes across FlowNet+STN and VoxelMorph registration models and improves existing IVIF methods on misaligned images.The proposed overall framework still outperforms these CPSTN-enhanced methods, indicating that its performance depends on cooperation among components.
4 Conclusion
The framework uses generation-registration and feature interaction to mitigate ghosting in unsupervised misaligned infrared-visible fusion. Its registration paradigm also extends to existing IVIF methods and improves fusion performance on misaligned images.
- The proposed framework targets ghosting in unsupervised fusion of misaligned infrared and visible images.
- Generation-registration simplifies cross-modality alignment into mono-modality registration, while feature interaction adaptively selects meaningful infrared and visible features.
- Ablation analyses examine CPSTN loss functions, the IFM, and CPSTN generalization on the RoadScene dataset.
- The generation-registration paradigm can be extended to existing IVIF methods to improve fusion performance on misaligned cross-modality images.