PaperlayerView exact revision on arXiv

Source-linked AI summary

CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion

Jinyuan Liu, Runjia Lin, Guanyao Wu, Risheng Liu, Zhongxuan Luo, Xin Fan

arXiv:2211.10960v3cs.CV

TL;DR

IVIF methods can preserve complementary information poorly because their losses neglect modality relationships and their architectures may suffer feature degeneration. CoCoNet uses coupled contrastive constraints, data-sensitive weighting, and multi-level attention, with experiments reporting SOTA performance across IVIF and medical image fusion. Its broader transfer depends on whether meaningful contrastive masks can be constructed for new modalities.

  • Problem

    Existing learning-based fusion methods neglect inter-relationships between modalities and feature transmission, causing redundant or invalid information and vital information degeneration.

  • Method

    CoCoNet combines coupled contrastive constraints, data-sensitive loss weights, and a multi-level attention module for end-to-end multimodal image fusion.

  • Results

    CoCoNet achieves state-of-the-art performance, outperforming nine state-of-the-art IVIF methods and extending to medical image fusion.

  • Takeaways & Limitations

    The method preserves salient thermal targets and faithful textural details while reducing redundant features through contrastive constraints and adaptive feature transmission.

  • Takeaways & Limitations

    Applying contrastive learning to other fusion tasks requires target modalities capable of generating interpretable and meaningful contrastive masks.

Abstract

from arXiv · show

Infrared and visible image fusion targets to provide an informative image by combining complementary information from different sensors. Existing learning-based fusion approaches attempt to construct various loss functions to preserve complementary features, while neglecting to discover the inter-relationship between the two modalities, leading to redundant or even invalid information on the fusion results. Moreover, most methods focus on strengthening the network with an increase in depth while neglecting the importance of feature transmission, causing vital information degeneration. To alleviate these issues, we propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion in an end-to-end manner. Concretely, to simultaneously retain typical features from both modalities and to avoid artifacts emerging on the fused result, we develop a coupled contrastive constraint in our loss function. In a fused image, its foreground target / background detail part is pulled close to the infrared / visible source and pushed far away from the visible / infrared source in the representation space. We further exploit image characteristics to provide data-sensitive weights, allowing our loss function to build a more reliable relationship with source images. A multi-level attention module is established to learn rich hierarchical feature representation and to comprehensively transfer features in the fusion process. We also apply the proposed CoCoNet on medical image fusion of different types, e.g., magnetic resonance image, positron emission tomography image, and single photon emission computed tomography image. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation, especially in preserving prominent targets and recovering vital textural details.

1 Introduction

Infrared and visible image fusion combines complementary sensing strengths, but existing methods can retain redundancy, lose essential information, and struggle with feature transmission and supervision. CoCoNet addresses these issues with coupled contrastive learning, data-driven weighting, and multi-level attention, achieving strong IVIF and medical image-fusion results.

  • Infrared images preserve thermal targets in difficult conditions, while visible images provide high-resolution background details, motivating their fusion into one informative image.
  • Existing learning-based IVIF methods face unsupervised training, manually tuned loss trade-offs, and feature degeneration that can produce redundant or degraded information.
  • CoCoNet couples contrastive constraints with data-driven source-image weights to retain complementary targets and details while reducing redundancy and manual parameterization.
  • A multi-level attention module learns hierarchical representations and improves feature transmission during end-to-end fusion.
  • Extensive experiments on multiple datasets report that CoCoNet outperforms nine state-of-the-art IVIF methods and extends effectively to medical image fusion.

2 Related Works

Related work spans traditional transform, representation, decomposition, optimization, hybrid, and learning-based fusion methods. Deep models improve flexibility but retain challenges involving handcrafted fusion rules, modality imbalance, information loss, and generalization across fusion tasks.

  • Traditional Fusion Methods: Traditional IVIF methods include multi-scale transform, sparse representation, saliency, subspace, model-based, hybrid, and related approaches.
  • Traditional Fusion Methods: Multi-scale transforms use tools such as wavelets, contourlets, and edge-preserving filters to decompose images and preserve details during fusion.
  • Traditional Fusion Methods: Sparse-representation methods learn over-complete dictionaries or compact subdictionaries to represent source patches and reduce dictionary redundancy.
  • Deep Learning-Based Fusion Methods: Deep learning fusion methods use autoencoders, CNNs, GANs, and transformers, but may rely on manually designed fusion rules or omit important modality information.
  • Deep Learning-Based Fusion Methods: General fusion networks and medical-image methods seek efficient cross-task reuse, while traditional approaches remain constrained by handcrafted features, complex rules, and limited scene adaptability.

3 The Proposed Method

CoCoNet addresses unsupervised infrared-visible fusion by coupling contrastive constraints with self-adaptive loss weighting and multi-level feature attention. The method uses modality-specific targets, details, and masks to guide feature selection while improving feature representation and transmission.

  • Motivation and overview: CoCoNet combines coupled contrastive learning, self-adaptive loss weighting, and a multi-level attention module for end-to-end image fusion.The design targets complementary information preservation, redundancy reduction, and comprehensive feature representation.
  • Coupled contrastive learning: Target constraints treat infrared foreground regions as positives and corresponding visible regions as negatives, while detail constraints reverse these roles for background texture preservation.This pairing encourages salient infrared targets and vivid visible details in the fused image.
  • Coupled contrastive learning: Manually generated foreground and background masks provide explicit guidance for separating saliency from texture during contrastive feature extraction and fusion.The masks are generated for corresponding image pairs using typical infrared images from the TNO dataset.
  • Self-adaptive learning weight: The fusion loss combines structure similarity and intensity similarity terms, with SSIM measuring contrast, illumination, and structural differences between source and fused images.The loss is written as LP = αLS + LN, where LS is the structure similarity loss and LN is the intensity similarity loss.
  • Self-adaptive learning weight: Image-specific weights replace fixed empirical proportions: average gradient guides SSIM weighting, while entropy gives greater intensity-loss weight to the modality containing more information.Average gradient reflects basic intensity changes and is matched to structural similarity; entropy is computed at pixel level and informs intensity weighting.
  • Network architecture: The multi-level attention module integrates hierarchical source features to strengthen extracted representations and comprehensively transfer them into the fused image.The module is designed to use higher-level features from the source images rather than relying only on direct feature transmission.

4.1 Experimental Settings

The experiments evaluate CoCoNet on public TNO and RoadScene image pairs using two-stage training and six image-quality metrics.

  • Datasets: TNO and RoadScene provide the public infrared-visible image pairs used for evaluation; RoadScene contains 221 realistic driving-scene pairs.TNO is used to train the network because of its high-quality images and distinctive scenarios.
  • Training strategy: The framework is trained on TNO through training and fine-tuning phases, with contrastive constraints added during fine-tuning.The initial stage uses self-adaptive loss, while fine-tuning combines it with contrastive constraints.
  • Training strategy: Training uses 46 image pairs, 1,410 64×64 patches, normalization to [-1, 1], Adam optimization, learning rate 0.0001, and batch size 30.The patches are intended to improve perception of subtle gradients and entropy.
  • Training strategy: Fine-tuning uses 18 masked TNO images and one positive with three negative samples for contrastive constraint learning.Two negatives are randomly selected from other negative patches.
  • Evaluation metrics: Evaluation uses EN, AG, SF, SD, SCD, and VIF to measure information abundance, intensity change, contrast, correlation, and visual information fidelity.The selected metrics are commonly used for image-quality measurement.

4.2 Results and Analysis on IVIF

CoCoNet is compared visually and quantitatively with state-of-the-art fusion methods on TNO and RoadScene datasets, emphasizing target visibility and texture preservation.

  • Qualitative comparison on TNO: On TNO, CoCoNet produces brighter foreground targets, higher contrast, and clearer visible-image textures than the compared methods.The reported advantage is especially apparent in enlarged patches containing thermal targets and leaf details.
  • Qualitative comparison on RoadScene: On RoadScene, CoCoNet makes pedestrians more conspicuous and preserves vivid truck textures in relatively low-light scenes.The comparison also reports sharper and clearer images across varied driving scenarios.
  • Quantitative comparison: On TNO, CoCoNet achieves the highest results on all six evaluated metrics.The paper attributes the result to effective use of vital source features; SF and AG are reported as 58% higher than the second-best methods.
  • Quantitative comparison: On RoadScene, CoCoNet achieves the best results on EN, SF, AG, SCD, and VIF, and SOTA results on SD.The evaluation covers fourteen methods on the TNO and RoadScene datasets.
  • Architecture analysis: The full MAM architecture combines skip connections, channel attention, and backbone features for multi-level feature processing.Figure 11 compares progressively richer MAM variants and the full proposed design.

4.3 Ablation Study

Ablations show that CoCoNet’s multi-level attention, self-adaptive weighting, and coupled contrastive regularization jointly improve feature preservation, detail quality, and source-image adaptation.

  • Multi-level attention module: Removing pretrained VGG features significantly reduces metrics on both TNO and RoadScene, confirming the value of complementary high-level features.Channel attention further improves EN, SF, SD, and VIF by rearranging modality-derived channel weights.
  • VGG feature ablation: Integrating all VGG feature levels achieves the best performance across all metrics, while omitting low-level features causes the worst results.Low-level features provide edge and color information; the complete model also preserves richer details and prominent highlights.
  • Self-adaptive learning: Self-adaptive σ and γ weights generally outperform hand-crafted weights because they adapt to diverse source-image characteristics.On TNO, the strategy reaches around 0.2 points higher than fixed SSIM weights on SCD, although fixed weights slightly outperform it on VIF when σ=0.8.
  • Contrastive regularization: The coupled contrastive regularization preserves salient thermal targets and vivid visible details, whereas missing constraints or mismatched patch sampling introduce noise, blur, or reduced brightness.Both target and detail constraints are described as indispensable, and correct correspondence is identified as especially important.
  • Mask analysis: Different mask-generation strategies produce similar visual features, indicating that the proposed method is nearly independent of how masks are obtained.The reported preserved properties include real details and high contrast.

4.4 Computational Complexity Analysis

The complexity analysis evaluates CoCoNet’s efficiency alongside fusion quality, reporting faster execution than several competing methods despite the additional complexity of multi-level feature processing.

  • Running time: CoCoNet runs two times faster than DenseFuse and U2Fusion, three times faster than PMGI, and four times faster than FusionGAN, GANMcC, and RFN.The evaluation uses 64×64 inputs and averages running time over ten TNO images.
  • Model complexity: Despite being more complicated than FusionGAN, CoCoNet achieves faster speed, while its MAM module requires substantial parameters for multi-level feature extraction and transmission.Removing MAM reduces computational complexity but changes the architecture’s feature-processing capacity.

4.5 Limitations

CoCoNet can preserve substantial information under slight source-image misalignment, but significant pixel deviations produce halos and artifacts in the fused results.

  • Misalignment boundary: Significant pixel deviation between infrared and visible inputs causes halos and artifacts, despite the method preserving much vital information under slight misalignment.The study evaluates tiny, moderate, and extreme affine and elastic translations on TNO and Roadscene images.

4.6 Results and Analysis on MIF

On medical image fusion tasks, CoCoNet balances modality-specific intensity, color, and detail information, with qualitative and quantitative comparisons favoring the proposed method over competing approaches.

  • MRI-PET fusion: CoCoNet preserves high-value information across MRI-PET fusion by combining multiscale attention with MRI-mask contrastive learning.The reported mechanism helps prevent MRI details from being covered by other modalities’ intensity information.
  • MRI-PET fusion: On MRI-PET images, CoCoNet achieves the optimal visualization effect by balancing color retention and detail preservation.EMFusion produces color distortion, while SwinFusion retains color but leaves MRI details covered.
  • MRI-SPECT fusion: MRI-mask contrastive learning preserves MRI details with less interference on MRI-SPECT fusion, where SPECT intensity information is sparse.Other methods exhibit modality-specific information loss, including MRI darkening or color defects.
  • Quantitative comparison: CoCoNet is superior to other deep learning methods on the reported MRI-PET and MRI-SPECT evaluation metrics.The quantitative comparisons are presented in Table 5.

4.7 Task-driven evaluation

The paper evaluates CoCoNet through downstream object detection and semantic segmentation, alongside visual comparisons, to assess whether fused images retain useful semantic information. CoCoNet performs strongly among common fusion methods, while complete contrastive constraints improve segmentation and detection-related outcomes over variants.

  • Evaluation motivation: Task-driven evaluation measures how fusion results support object detection and semantic segmentation beyond visual quality and statistical indicators.Object detection uses retrained YOLOv5 models on M3FD, while semantic segmentation uses fine-tuned SegFormer models on MFNet.
  • Object detection: CoCoNet ranked third in mAP@.5 and second in mAP@.5:.95 for object detection among the compared common fusion methods.TarDAL achieved the best mAP@.5 result and DIDFuse ranked second; mAP@.5:.95 reflects performance across different IoU values.
  • Ablation analysis: In object-detection ablations, the complete method had a clear mAP advantage, whereas adding only object or detail constraints negatively affected overall detection results.The object-only variant nevertheless showed stronger sensitivity to salient categories such as person and bus.
  • Semantic segmentation: The complete contrastive constraint achieved the best semantic-segmentation performance across multiple categories and both average indicators.The single target constraint improved overall segmentation more than the single detail constraint, while variants without contrastive constraints performed poorly.

4.8 Extension to other fusion tasks

CoCoNet is extended to GFP–PC and NIR–VIS fusion, where it produces promising qualitative results by combining complementary intensity, color, structure, detail, foreground, and background information. The authors also identify modality-specific limits to transferring the coupled contrastive strategy.

  • Extension: CoCoNet is applied to GFP–PC and NIR–VIS fusion as examples of other multi-modality image-fusion tasks.The extensions are intended to illustrate the method’s applicability beyond infrared and visible fusion.
  • Modality roles: GFP supplies intensity and color information while PC supplies structure and detail, paralleling PET/SPECT and MRI roles respectively.NIR provides rich background information, whereas VIS offers clear foreground content.
  • Qualitative results: Qualitatively, CoCoNet preserves PC edge information while introducing GFP color with low deterioration, and retains NIR texture while producing high contrast in NIR–VIS fusion.The reported visual results are shown in Figure 24.
  • Generality boundary: PC images lack the broader information needed to provide MRI segmentation masks because they mainly capture gradient details along lines.This difference limits the correspondence between GFP–PC fusion and MRI–PET fusion.
  • Generality boundary: The authors state that transferring coupled contrastive learning requires interpretable and meaningful contrastive masks for the target modalities.Whether the modalities can generate such masks is identified as a precondition for efficacy and a barrier to generalizability.

5 CONCLUSION

The conclusion presents CoCoNet as an end-to-end fusion network combining twin contrastive constraints, multi-level attention, and self-adaptive weighting. The reported experiments support strong visual and quantitative performance, ablation effectiveness, and extension to medical image fusion.

  • Conclusion: CoCoNet integrates twin contrastive constraints and multi-level features for infrared and visible image fusion.The constraints preserve typical features while avoiding redundant features during fusion.
  • Conclusion: A multi-level attention module learns hierarchical representations and improves feature transmission, while self-adaptive weighting replaces manually crafted loss trade-offs.These components are described as part of the network and loss-function design.
  • Conclusion: Qualitative and quantitative experiments report SOTA performance with high efficiency, and ablations validate the effectiveness of the proposed method.The conclusion specifically highlights salient thermal targets and faithful textural details.
  • Conclusion: The extended CoCoNet also achieves superior performance against other state-of-the-art approaches on medical image fusion.The conclusion reports this extension as an additional result of the proposed method.
Loading 2211.10960v3…