PaperlayerView exact revision on arXiv

Source-linked AI summary

Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption

Jinyuan Liu, Guanyao Wu, Zhu Liu, Di Wang, Zhiying Jiang, Long Ma, Wei Zhong, Xin Fan, Risheng Liu

arXiv:2501.10761v1cs.CV

TL;DR

IVIF research needs to integrate infrared and visible information while addressing registration, task performance, robustness, evaluation, and efficiency challenges. This survey organizes learning-based methods across data, fusion, and task dimensions, reviews representative techniques and datasets, and compares registration, fusion, and downstream perception results. It provides a structured synthesis of current advances and open directions for practical IVIF.

  • Problem

    Practical IVIF remains constrained by difficult cross-sensor registration, limited adaptation to high-level perception tasks, robustness challenges, evaluation gaps, and efficiency demands.

  • Method

    The survey organizes learning-based IVIF methods across data, fusion, and task dimensions, covering techniques, datasets, architectures, loss functions, and practical application pipelines.

  • Results

    The survey provides comparative analyses of registration, fusion, and downstream tasks, including qualitative and quantitative object-detection results across representative methods.

  • Takeaways & Limitations

    The synthesis clarifies technology alignment with data types, practical applications, and evaluation standards while identifying directions for continued IVIF research.

Abstract

from arXiv · show

Infrared-visible image fusion (IVIF) is a critical task in computer vision, aimed at integrating the unique features of both infrared and visible spectra into a unified representation. Since 2018, the field has entered the deep learning era, with an increasing variety of approaches introducing a range of networks and loss functions to enhance visual performance. However, challenges such as data compatibility, perception accuracy, and efficiency remain. Unfortunately, there is a lack of recent comprehensive surveys that address this rapidly expanding domain. This paper fills that gap by providing a thorough survey covering a broad range of topics. We introduce a multi-dimensional framework to elucidate common learning-based IVIF methods, from visual enhancement strategies to data compatibility and task adaptability. We also present a detailed analysis of these approaches, accompanied by a lookup table clarifying their core ideas. Furthermore, we summarize performance comparisons, both quantitatively and qualitatively, focusing on registration, fusion, and subsequent high-level tasks. Beyond technical analysis, we discuss potential future directions and open issues in this area. For further details, visit our GitHub repository: https://github.com/RollingPlain/IVIF_ZOO.

I. INTRODUCTION

IVIF combines complementary infrared and visible information for reliable perception, but practical deployment still faces data compatibility, task adaptation, and efficiency challenges. This survey organizes learning-based methods multidimensionally and compares their roles across registration, fusion, and downstream tasks.

  • Motivation: Infrared and visible images provide complementary information, supporting high-reliability perception in complex and extreme scenarios.Infrared sensing contributes thermal information, while visible imagery supplies environmental texture, although both have adverse-condition limitations.
  • Motivation: Since 2018, deep learning has driven learning-based IVIF toward stronger visual quality, robustness, and computational efficiency than conventional approaches.These methods frequently achieve state-of-the-art performance across benchmarks spanning visual enhancement, data compatibility, and task adaptability.
  • Survey scope: Existing surveys primarily cover traditional or learning-based IVIF, whereas this work examines learning-based approaches through data, fusion, and task dimensions.The survey reviews more than 180 learning-based approaches.
  • Contributions: The survey analyzes architectures and loss functions and provides a lookup table summarizing representative methods’ core ideas.The table is intended to support researchers entering the field and planning subsequent studies.
  • Contributions: It systematically reviews techniques and datasets for practical applications and compares preliminary registration with downstream tasks including object detection and semantic segmentation.The survey also identifies challenges, open issues, emerging trends, and prospective directions.

B. Organization

The paper frames practical IVIF as a pipeline spanning data compatibility, fusion, and perception tasks. Its taxonomy then organizes methods by dimensions and reviews architectures, losses, datasets, and representative fusion strategies.

  • Practical pipeline: Practical IVIF must address registration difficulty and downstream perception performance in addition to visual enhancement.Pixel alignment is difficult because infrared and visible sensors differ in viewpoints, pixel distributions, and resolutions; visually enhanced outputs may still underperform on high-level tasks.
  • Taxonomy: The survey’s taxonomy uses a multidimensional perspective and introduces common architecture and loss-function components.The overview is presented through Figure 4 and Table I.
  • AE-based approaches: Auto-encoder methods pretrain an encoder-decoder, then combine encoded features using manual fusion rules or a second training stage.The encoder extracts features and the decoder reconstructs the fused image.
  • AE-based approaches: AE developments emphasize improved fusion rules and data integration alongside architectural changes such as new layers and modified connections.These directions target multimodal feature synthesis and network design.
  • Representative methods: Representative methods include DenseFuse’s addition or l1-norm softmax fusion rules and RFN-Nest’s residual fusion architecture for preserving deep feature details.These examples illustrate the survey’s separation of fusion-rule and network-architecture innovations.

2) CNN-based Approaches:

CNN-based IVIF methods generally extract features, fuse them, and reconstruct images, while the broader taxonomy also contrasts optimization, loss, architecture, GAN, and Transformer strategies. These approaches seek better multimodal integration, but adversarial and modality-balance issues remain.

  • CNN-based approaches: CNN-based IVIF algorithms generally perform feature extraction, fusion, and image reconstruction, learning complex and high-level features from data.The CNN pipeline is illustrated alongside other major IVIF paradigms.
  • Optimization-inspired methods: Optimization-inspired CNN methods embed iterative integration or replace numerical operations with learnable modules, often guided by modality priors.LRRNet applies low-rank representation to the network design.
  • Loss functions: Loss-function innovations shape unsupervised IVIF outcomes, including illumination-aware and salient-target-guided objectives.PIAFusion uses a lighting-assessment subnetwork, while STDFusionNet uses a salient target mask.
  • Architectural methods: Architectural research modifies residual, dense, and U-Net designs, while NAS automatically searches for suitable fusion architectures.Other designs combine CNN extraction with graph interaction or diffusion-based modeling.
  • GAN-based approaches: GAN methods use single or dual discrimination to guide fused images toward one modality or balance information from both modalities.Single discrimination risks modality imbalance, whereas dual discrimination must still extract distinct thermal and visible characteristics.

4) Transformer-based Approaches:

Transformer-based IVIF methods combine local feature extraction with attention-based modeling of long-range dependencies and cross-modality relationships. Their stronger global context comes with substantial computational and memory demands that challenge real-world deployment.

  • 4) Transformer-based Approaches:: Transformer-based fusion methods commonly combine CNN blocks for shallow features with transformer blocks for long-range dependence.This design aggregates local feature extraction and global dependency modeling.
  • 4) Transformer-based Approaches:: Multi-scale auto-encoding and spatiotransformer strategies aggregate local and global feature information for image fusion.SwinFusion additionally uses self- and cross-attention to model modality-specific and cross-domain dependencies.
  • 4) Transformer-based Approaches:: Prompt-based learning from Vision-Language Models uses semantic prompts to guide fusion, targeting target identification and fusion quality.
  • 4) Transformer-based Approaches:: Transformer-based approaches capture long-range dependencies and retain global context, but require high computational resources and large memory.The resource demands make real-world deployment challenging.
  • Application-oriented IVIF: Object detection and semantic segmentation are central downstream tasks in application-oriented IVIF, with methods using fusion, detection, and cross-modality interactions.Examples include detection-guided fusion, object-aware losses, and cross-modality feature alignment.

2) Semantic Segmentation:

Application-oriented IVIF extends beyond image quality by integrating fusion with semantic perception and by addressing imperfect registration. Methods span semantic-guided fusion, two-stream segmentation, and registration strategies based on pseudo-labels, shared latent spaces, or mutual reinforcement.

  • 2) Semantic Segmentation:: SeAFusion cascades fusion with segmentation and uses semantic loss to enrich fused information through loop-based training.SuperFusion jointly addresses multi-modality registration, fusion, and semantic perception, while PSFusion injects semantics progressively at the feature level.
  • 2) Semantic Segmentation:: Two-stream infrared-visible segmentation fuses modality-specific features in either encoder or decoder stages.Encoder-based methods aggregate features early, whereas decoder-based methods combine them during reconstruction.
  • 2) Semantic Segmentation:: Unified approaches use semantic perception to guide fusion, including bi-level dynamic learning and automated loss-function search.These methods target visually appealing fused images that can serve downstream perception tasks.
  • Other perception tasks: Infrared-visible data also support object tracking, crowd counting, salient object detection, and depth estimation through modality fusion.Reported mechanisms include pixel-wise, feature-level, and decision-based fusion, as well as spectral and style transfer.
  • 1) Registration-free approaches:: Fusion for imperfectly aligned multi-modality images is divided into style-transfer-based and latent-space-based approaches.These categories address registration by generating pseudo-labels or mapping modalities into a shared feature space.
  • 1) Registration-free approaches:: Style-transfer methods jointly learn modality transfer and spatial transformation networks to generate pseudo-labels and predict spatial displacements.The modality-transfer network transforms one modality, after which the spatial-transformation network estimates displacement.
  • 1) Registration-free approaches:: Registration methods include bidirectional mono-modality training, cross-modality generation-registration, shared latent spaces, and mutual reinforcement between registration and fusion.RFNet combines coarse-to-fine registration with interactive learning so fusion and registration promote each other.

2) General fusion approach:

The survey presents application-oriented IVIF as a pipeline spanning fusion and downstream perception, while highlighting adversarial robustness as an unresolved concern. It also organizes representative methods and aligned datasets for comparison.

  • 2) General fusion approach:: Adversarial perturbations can fool neural-network estimates, and PAIFusion uses image fusion to enhance robustness against adversarial attacks.The survey identifies this robustness problem as an urgent future research topic for real-world applications.
  • 2) General fusion approach:: Representative methods include TarDAL for joint fusion and detection, DetFusion for detection-guided fusion, and SeAFusion for cascaded fusion and segmentation.The table also lists SegMiF as jointly learning fusion and segmentation features through cross-modality attention.
  • 2) General fusion approach:: Aligned multi-modality datasets are catalogued as part of the survey’s comparative resource overview.The dataset download links are organized in the GitHub repository referenced by the paper.

D. Architectures Summary and Discussion

IVIF architectures are evolving toward increasingly complex structures and loss functions to model complementary modalities and improve fusion quality. The survey organizes this evolution across network design, iterative refinement, and loss-function dimensions.

  • D. Architectures Summary and Discussion: Deep learning-based fusion architectures are categorized by network structure as existing architectures, complex stacked networks, multi-batch architectures, and recursive architectures.
  • D. Architectures Summary and Discussion: Existing CNN architectures use cascaded, residual, or densely connected blocks for feature extraction and information integration.
  • D. Architectures Summary and Discussion: Complex stacked networks intertwine multiple block types to deepen processing and exploit complementary information from source images.
  • D. Architectures Summary and Discussion: Multi-batch architectures use specialized modules and parameters to optimize fusion across diverse infrared-visible inputs.
  • D. Architectures Summary and Discussion: Recursive architectures and diffusion models progressively refine fusion quality by reusing prior information during iterative enhancement.This design is described as suitable for sequential data.
  • D. Architectures Summary and Discussion: Overall, IVIF research is moving toward more complex and refined network structures for increasingly complex fusion demands.
  • Loss Functions: Unsupervised fusion losses are organized by pixel-level comparisons, evaluative metrics, and data characteristics.Examples include L1, MSE, SSIM, gradients, visual salience maps, spatial frequency, edge, perceptual, and contrastive losses.
  • Loss Functions: Researchers can select, combine, and optimize loss functions according to source-image characteristics and task requirements.

A. Benchmark

The benchmark spans datasets for visual fusion, target detection, and semantic segmentation, with metrics tailored to fusion, registration, and downstream perception. Evaluation uses nine fusion metrics and three registration metrics alongside standard segmentation and detection measures.

  • Datasets: Datasets are grouped into early fusion, target-detection, and semantic-segmentation categories.TNO, RoadScene, and M3FD are used for fusion comparisons.
  • Fusion Metrics: Nine fusion metrics comprise five reference-based measures and four no-reference measures.The reference-based metrics are MI, VIF, CC, SCD, and QAB/F; the no-reference metrics are EN, SF, SD, and AG.
  • Registration Metrics: Registration evaluation uses MSE, MI, and NCC to assess alignment and similarity.Lower MSE and higher MI indicate better alignment; NCC evaluates similarity between corresponding windows.
  • Downstream Task Metrics: Semantic segmentation is evaluated with IoU, mIoU, Acc, and mAcc, while detection uses Recall, Precision, AP, and mAP.Segmentation metrics measure pixel or category overlap and accuracy; detection metrics average precision across classes.

1) Qualitative Comparisons:

Qualitative and quantitative comparisons reveal different strengths among fusion strategies under aligned and misaligned conditions. Style-transfer methods preserve structural alignment well, while latent-space methods perform better on several no-reference metrics.

  • Qualitative Comparisons: CAF and MoEFusion partially suppress smoke effects while avoiding the background color deviation observed in other methods.IGNet avoids most smoke impact but leans toward infrared, whereas CoCoNet highlights targets but introduces background color deviation.
  • Qualitative Comparisons: UMFusion, IMF, and SuperFusion effectively correct structural distortion and edge ghosts in misaligned-image fusion.Latent-space methods RFVIF, SemLA, and ReCoNet retain residual deformations.
  • Qualitative Comparisons: IMF, MURF, and UMFusion rank among the top three methods for object saliency and texture richness.The comparison concerns fused-image visual properties under misaligned cross-modality inputs.
  • Quantitative Comparisons: SuperFusion performs strongly on reference-based metrics, whereas style-transfer methods achieve suboptimal reference-based results and latent-space methods lead several no-reference metrics.SuperFusion benefits from ground-truth deformation-field supervision; latent-space strengths include EN, SF, SD, and AG.
  • Quantitative Comparisons: The contrasting metric profiles indicate complementary strengths between style-transfer and latent-feature-space methods.The survey identifies combining these advantages as a promising direction.

C. Image Fusion for Object Detection

The detection evaluation trains a common YOLO-v5 setup on M3FD fusion outputs and tests robustness using held-out consecutive-image sequences. Detection performance depends on preserving thermal targets while handling degraded, weak-texture scenes.

  • Evaluation Setup: The evaluation uses YOLO-v5 with fusion images from all methods under a unified M3FD training and testing setup.A consistent detector and dataset make the fusion methods directly comparable for detection.
  • Evaluation Setup: The test set contains 100 randomly selected groups of 10 consecutive M3FD images to reduce scene overlap with training data.This split is intended to validate robustness and limit overfitting from continuously captured images.
  • Qualitative Detection Results: TarDAL and PAIFusion detect all pedestrians in a degraded small-target scene, unlike low-contrast or artifact-producing methods.Fog, rain, strong light, and weak textures make pedestrians difficult to detect; thermal-target visibility and degradation handling are decisive factors.
  • Quantitative Detection Results: TarDAL achieves the highest detection precision by preserving thermal target details and textural information.It performs especially well for cars and persons; AP@0.5, precision, recall, mAP@0.5, and mAP@0.5:0.95 are reported.
  • Quantitative Detection Results: SegMiF and TIMFusion show strong detection performance through semantic feature interjection and task-loss guidance.U2Fusion and SDNet balance information preservation across infrared and visible modalities.
  • Segmentation Context: Extreme lighting remains difficult for segmentation, with dark pedestrians and an overexposed bus causing missed outlines and class confusion.Many methods misclassify the bus as a car and fail to capture its full outline.

2) Quantitative Comparisons:

The survey compares IVIF methods on segmentation and computational efficiency, while identifying registration and adversarial robustness as important unresolved practical concerns.

  • Semantic segmentation: Perception-guided fusion methods, including MoEFusion and DetFusion, achieve competitive semantic segmentation performance through task-specific losses or learning strategies.These results are reported on the FMB dataset.
  • Computational efficiency: Average runtime, network parameters, and FLOPS provide complementary measures of IVIF computational efficiency.Runtime measures task speed, while parameters and FLOPS reflect model size and forward-propagation computational volume.
  • Computational efficiency: Increasing parameter counts generally correspond to higher FLOPS, and higher FLOPS usually result in longer runtime.The survey frames these relationships as general tendencies rather than strict equivalences.
  • Computational efficiency: Efficiency experiments use ten randomly selected 1024×768 M3FD images on an Nvidia GeForce 4090, with GPU timing measured using CUDA events.The reported results use milliseconds, millions of parameters, and gigaflops.
  • Computational efficiency: DDFM reaches 280.8k ms runtime and substantially exceeds other methods in parameters and FLOPS, suggesting relatively low practical efficiency.The comparison is reported for the M3FD computational-efficiency experiment.
  • Registration and robustness: Few IVIF studies address image registration, although accurate pixel alignment is difficult because infrared and visible images arise from different imaging principles and spectral ranges.The survey proposes registration-compatible fusion networks or registration-aware loss components as future directions.
  • Registration and robustness: Robustness against physical distortions, degradation, digital parameter perturbations, and especially cross-modality attacks remains challenging for image fusion networks.PAIFusion is cited as exploring parameter perturbation, while real-world physical and cross-modality attacks remain formidable.

B. Developing Benchmarks

The survey reviews benchmark expansion for IVIF perception tasks while emphasizing persistent gaps in registration data, evaluation metrics, efficiency, and deployment on resource-limited devices.

  • Benchmark development: Early IVIF benchmarks such as TNO, RoadSenece, and VIFB had limited scene diversity and resolution.Later datasets expanded toward perception tasks and more varied scenarios.
  • Benchmark development: MS and LLVIP provide object-detection labels but focus mainly on road and surveillance scenes, while M3FD adds adverse weather and varied scenarios.MFNet and FMB extend benchmark coverage to semantic tasks with extensive labels.
  • Open benchmark issues: Registration benchmarks with real-world discrepancies and registration ground truth remain necessary because existing benchmarks primarily use pixel-aligned pairs.The survey highlights imaging differences and sensor baselines as factors to represent.
  • Evaluation metrics: Traditional metrics such as EN, MI, CC, and SCD each capture only one image-quality aspect and may diverge from subjective evaluations, especially under high noise.The survey therefore calls for metrics combining visual and perceptual quality.
  • Efficiency constraints: Improving fused-image quality by enlarging neural networks increases parameters and harms operational efficiency.Neural Architecture Search, pruning, and atrous convolution have been proposed, but their effectiveness remains dependent on advanced GPU capacity.
  • Efficiency constraints: Lightweight architectures are needed because UAVs and handheld devices cannot generally support the computational demands of heavy GPUs.The survey frames efficient, less resource-intensive network design as a future research priority.
  • Task applications: IVIF can assist downstream vision tasks including autonomous navigation, depth estimation, and 3D reconstruction.The supplied discussion connects multimodal imagery with environmental understanding across lighting and weather conditions.
  • Conclusion: The survey synthesizes methodological and practical issues through comparative analyses of registration, fusion, and related tasks, while identifying future research areas.Its stated goal is to guide both beginners and experienced researchers.
Loading 2501.10761v1…