Source-linked AI summary
CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte, Luc Van Gool
TL;DR
MM image fusion must preserve complementary modality information while modeling cross-modality features and separating shared from modality-specific content. CDDFuse addresses this with dual-branch Transformer-CNN extraction, correlation-driven decomposition, and LT/INN-based fusion, and reports leading fusion performance with improved downstream recognition accuracy. Its decomposition relies on an assumption about correlated low-frequency and modality-specific high-frequency information.
Problem
MM image fusion needs to preserve complementary modality information while modeling cross-modality features and decomposing modality-shared and modality-specific features.
Method
CDDFuse uses a dual-branch encoder with LT blocks for global low-frequency features, INN blocks for local high-frequency features, and a correlation-driven loss for feature decomposition.
Results
CDDFuse achieves leading image-fusion performance for infrared-visible and medical image fusion and improves downstream multi-modal pattern-recognition task accuracy.
Takeaways & Limitations
The unified benchmark shows that CDDFuse fusion images can facilitate downstream infrared-visible object detection and semantic segmentation.
Takeaways & Limitations
The decomposition assumes low-frequency cross-modality features are correlated and high-frequency features represent modality-specific information.
Abstract
from arXiv · showhide
Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.
1. Introduction
MM image fusion combines complementary modality information, but existing CNN-based pipelines struggle to separate shared and modality-specific features and to capture global context. CDDFuse addresses these challenges with dual-branch feature extraction, correlation-driven decomposition, and complementary Transformer-CNN components.
- Motivation: IVF preserves infrared thermal radiation and visible-image texture, while MIF fuses modalities to reveal abnormalities for diagnosis and treatment.IVF fusion can also provide clearer scene and object representations for downstream recognition tasks.
- Challenges: Existing CNN-based auto-encoder pipelines inadequately extract cross-modality features because shared encoders miss modality-specific information and private encoders miss shared information.CNNs also operate within relatively small receptive fields, limiting context modeling.
- Design rationale: The paper assumes low-frequency inputs are correlated across modalities as shared information, whereas high-frequency features are less correlated and represent modality-specific characteristics.For infrared-visible fusion, low frequencies include shared background and environmental structure, while high frequencies include independent textures.
- Proposed approach: CDDFuse uses a dual-branch Transformer-CNN framework to extract and fuse global and local features corresponding to modality-shared and modality-specific information.The architecture combines Transformer long-range modeling with CNN local-context extraction.
- Proposed approach: Correlation-driven decomposition makes cross-modality base features correlated and detailed high-frequency features decorrelated, while INN blocks support lossless information transmission.The model reconstructs the fused image through a decoder after dual-branch feature extraction.
2. Related Work
Prior MMIF work includes GAN-, auto-encoder-, unified-, and algorithm-unrolling-based CNN models, alongside increasingly capable Transformer and INN components. CDDFuse combines lightweight long-range modeling, Restormer processing, and invertible feature extraction with correlation-based decomposition to address limitations of conventional auto-encoders.
- CNN-based MMIF: Deep-learning MMIF methods include GAN-based, auto-encoder-based, unified, and algorithm-unrolling models.These categories respectively target distributional and perceptual quality, encoder-decoder transformation, cross-task learning, and model-driven interpretability.
- Transformer components: Transformer-based vision models provide global feature extraction through self-attention but introduce substantial computational overhead.Lite Transformer structures reduce parameters through long-short range attention and flattened feed-forward processing.
- Transformer components: Restormer facilitates multi-scale local-global representation learning on high-resolution images, and CDDFuse adopts Restormer and Lite Transformer blocks.The paper positions these blocks as complementary components for MMIF feature extraction.
- Invertible neural networks: Invertible neural networks are normalized-flow modules with lossless information-preserving properties that have been applied across image-processing tasks.CDDFuse uses INNs as part of its feature-extraction and fusion design.
- Relation to existing approaches: Compared with conventional auto-encoders, CDDFuse uses different structures for local and long-range features and applies correlation-based decomposition to suppress redundant information.The paper presents this design as more reasonable and intuitive than a pure CNN framework.
3. Method
CDDFuse decomposes cross-modality features into low-frequency base and high-frequency detail components using a dual-branch encoder, then fuses and decodes them through a two-stage training scheme. Lite Transformer and INN branches target long-range base information and locally detailed information, while correlation-driven losses encourage modality-shared and modality-specific representations.
- Encoder: CDDFuse uses Restormer-based shared feature encoding, followed by Lite Transformer and INN branches for base/detail feature decomposition.The BTE extracts low-frequency base features, while the DCE extracts high-frequency detail information.
- Encoder: The Lite Transformer branch models long-distance dependencies while reducing computational cost through a flattened feed-forward structure.The LT block shrinks the Transformer embedding to reduce parameters while preserving performance.
- Encoder: The INN detail branch is designed for lossless information transmission, preserving high-frequency edge and texture information through mutually generated input and output features.Its affine coupling layers use bottleneck residual blocks as mapping functions to balance computation and feature extraction ability.
- Decoder and fusion: Base/detail fusion layers use LT and INN blocks respectively, matching the encoder’s inductive bias for low- and high-frequency feature processing.The base and detail fusion layers output fused features that are passed to the decoder.
- Decoder and fusion: The decoder concatenates decomposed features and reconstructs source images during stage I or produces fused images during stage II.Restormer blocks are retained in the decoder because its inputs contain cross-modality and multifrequency features.
- Two-stage training: Two-stage training first reconstructs paired infrared and visible images while learning decomposition, then uses the trained encoder to generate fused images.Stage-I losses combine infrared reconstruction, visible reconstruction, and feature decomposition terms; the decomposition loss promotes correlated base features and less-correlated detail features.
- Two-stage training: The correlation-driven decomposition loss makes base features more correlated and detail features less correlated, consistent with shared versus modality-specific information.The paper reports that the detail correlation term becomes larger under gradient-descent guidance from Ldecomp.
4. Infrared and visible image fusion
CDDFuse is evaluated for infrared-visible fusion across three benchmarks, with qualitative, quantitative, decomposition, ablation, and downstream-task analyses. The results show strong fusion quality, effective feature decomposition, and improved object detection and segmentation performance.
- Setup: CDDFuse is evaluated on MSRS, RoadScene, and TNO test datasets using eight fusion-quality metrics.Training uses 1,083 MSRS image pairs, while RoadScene and TNO are used without fine-tuning to assess generalization.
- Comparison with SOTA methods: The fused images preserve infrared thermal radiation and visible-image textures, while clarifying dark-region targets and low-illumination background contours.These visual properties support clearer scene and object interpretation.
- Comparison with SOTA methods: CDDFuse achieves excellent performance on almost all eight quantitative fusion metrics across varied illumination conditions and target categories.The comparison includes DIDFuse, U2Fusion, SDNet, RFNet, TarDAL, DeFusion, and ReCoNet.
- Comparison with SOTA methods: The decomposed base features activate relevant background information, while detail features separate infrared object highlights from visible textures.This visualization indicates that modality-specific features are extracted in the intended feature groups.
- Ablation studies: Removing or altering the correlation-driven decomposition loss weakens feature decomposition, while replacing the LT-INN combination with single-block alternatives reduces fusion effectiveness.The ablations also report serious information loss with CNN-based fusion and unsatisfactory results without two-stage training.
- Downstream IVF applications: CDDFuse obtains the best reported detection performance on M3FD and improves segmentation accuracy by integrating thermal, edge, and contour information.The downstream evaluations use mAP@0.5 for detection and IoU for semantic segmentation.
5. Medical image fusion
CDDFuse is evaluated on MRI-CT, MRI-PET, and MRI-SPECT medical image fusion datasets, both with and without MIF-specific training. It preserves detailed texture and highlighted structure information while achieving leading performance on almost all metrics.
- Results: Qualitative results for MRI-PET-16 are presented in Figure 6, alongside quantitative MIF results in Table 5.Table 5 marks the best values in boldface and the second-best values with underlining.
- Experimental setup: The MIF evaluation uses 286 medical image pairs, with MRI-CT, MRI-PET, and MRI-SPECT subsets in the test data.The test sets contain 21 MRI-CT, 42 MRI-PET, and 73 MRI-SPECT pairs.
- Experimental setup: CDDFuse is compared both as an IVF-trained model testing generalization and as CDDFuse∗ trained directly on MIF data.The IVF-trained comparison includes TarDAL, RFNet, DeFusion, and ReCoNet; the MIF-trained comparison includes U2Fusion, SDNet, and EMFusion.
- Results: CDDFuse preserves detailed texture and highlighted structure information in medical image fusion.This behavior is reported for CDDFuse whether or not it is trained on the MIF dataset.
- Results: CDDFuse achieves leading performance on almost all metrics with and without MIF-specific training.The reported comparison covers both the IVF-trained methods and models trained on the MIF dataset.
6. Conclusion
The paper introduces a dual-branch Transformer-CNN architecture for multi-modal image fusion. CDDFuse uses Restormer, Lite Transformer, invertible neural network blocks, and correlation-driven decomposition to improve feature extraction and downstream recognition accuracy.
- Architecture: CDDFuse uses a dual-branch Transformer-CNN architecture to extract modality-specific and modality-shared features.The architecture combines Restormer, Lite Transformer, and invertible neural network blocks.
- Feature decomposition: A correlation-driven decomposition loss makes feature decomposition more intuitive and effective.The loss is presented as part of the mechanism for better extracting modality-specific and modality-shared features.
- Evaluation: Experiments demonstrate CDDFuse’s fusion effectiveness and improved accuracy in downstream multi-modal pattern recognition tasks.The conclusion explicitly links the method to both image fusion and downstream recognition evaluation.