Source-linked AI summary
VIFB: A Visible and Infrared Image Fusion Benchmark
Xingchen Zhang, Ping Ye, Gang Xiao
TL;DR
Visible-infrared image fusion lacks a shared benchmark, standardized evaluation, and convenient algorithm integration for gauging the state of the art. The paper introduces VIFB with a 21-pair dataset, 20-algorithm library, and 13 metrics, then evaluates the integrated methods. Results show no dominant method across metrics, while deep-learning methods do not yet outperform conventional methods overall.
Problem
The field lacks a recognized dataset, comprehensive evaluation standard, and convenient interfaces for fair large-scale comparison of fusion algorithms.
Method
VIFB combines 21 registered visible-infrared image pairs, 20 integrated fusion algorithms, and 13 metrics covering four evaluation categories.
Results
No fusion method dominates across all or most metrics, and deep-learning methods do not currently show superiority over non-learning methods.
Takeaways & Limitations
Comprehensive evaluation should use multiple metrics because fusion algorithms can perform differently across evaluation criteria.
Takeaways & Limitations
Most evaluated fusion algorithms are computationally expensive for practical RGB-infrared tracking, limiting real-time feasibility.
Abstract
from arXiv · showhide
Visible and infrared image fusion is one of the most important areas in image processing due to its numerous applications. While much progress has been made in recent years with efforts on developing fusion algorithms, there is a lack of code library and benchmark which can gauge the state-of-the-art. In this paper, after briefly reviewing recent advances of visible and infrared image fusion, we present a visible and infrared image fusion benchmark (VIFB) which consists of 21 image pairs, a code library of 20 fusion algorithms and 13 evaluation metrics. We also carry out large scale experiments within the benchmark to understand the performance of these algorithms. By analyzing qualitative and quantitative results, we identify effective algorithms for robust image fusion and give some observations on the status and future prospects of this field.
1. Introduction
Visible and infrared image fusion lacks a shared benchmark, standardized evaluation, and convenient code integration, limiting fair large-scale comparison. VIFB addresses these gaps with a dataset, algorithm library, and comprehensive metrics.
- Existing studies lack a recognized dataset, making comparisons difficult because experiments use different image pairs.
- Different metrics capture different aspects of fusion quality, and no single metric is superior to all others.
- Inconsistent interfaces and environments make large-scale evaluation of publicly available algorithms inconvenient and time-consuming.
- VIFB provides 21 visible-infrared image pairs, 20 publicly available fusion algorithms, and 13 evaluation metrics.
- The benchmark integrates algorithms into an accessible code library and makes comprehensive comparison results available to researchers.
2. Related Work
Visible-infrared fusion includes traditional transform, representation, subspace, saliency, hybrid, and emerging deep-learning methods. However, the field still lacks a common benchmark and integrated evaluation resources for objective comparison.
- Traditional visible-infrared fusion methods include multi-scale transform, sparse representation, subspace, saliency-based, hybrid, and other theory-driven approaches.
- Recent methods use CNNs, GANs, Siamese networks, and autoencoders to learn features and adaptive fusion weights.
- The field lacks a widely recognized dataset, so studies commonly use different image pairs and objective comparison is difficult.
- Existing datasets are difficult to use for gauging the state of the art because code libraries, evaluation metrics, and benchmark results are lacking.
3. Visible and Infrared Image Fusion Benchmark
VIFB combines a varied registered test set, integrated fusion algorithms, and metrics spanning four evaluation categories. Its experiments compare qualitative, quantitative, and runtime behavior across methods.
- Dataset: The VIFB test set contains 21 registered visible-infrared pairs covering indoor, outdoor, low-illumination, and over-exposure conditions.
- Experiments: Figures 3 and 4 qualitatively compare 20 methods on fight and manlight image pairs.
- Code library: VIFB integrates 20 recent fusion algorithms and provides an interface for adding other algorithms or fusion results.
- Evaluation metrics: The benchmark implements 13 metrics spanning information-theoretic, feature-based, structural-similarity, and human-perception categories.
- Experiments: Figure 5 compares six metrics for selected methods across all 21 image pairs, using average values over the dataset.
4. Experiments
Experiments compare VIFB algorithms qualitatively, quantitatively, and by runtime. Results show trade-offs across visual quality, metrics, and computational cost rather than a universally dominant method.
- Experimental setup: Experiments used authors’ default parameters and pretrained deep learning models without retraining.The study reports only part of the results in the paper, with more provided in supplementary materials.
- Qualitative performance comparison: Hybrid MSD, MGFF, TIF, and VSMWLS produced more natural fused images while preserving more visible-image detail on the fight pair.Most methods revealed people hidden in the visible image’s shadows, but several introduced artifacts or lost visible details.
- Qualitative performance comparison: On the manlight pair, many methods failed to clearly reveal over-exposed people, while CBF, GFCE, and NSCT SR introduced artifacts.The comparison highlights method-specific weaknesses under over-exposure.
- Quantitative performance comparison: NSCT SR achieved the best overall quantitative performance, while LatLRR and DLF tied for second-best overall performance.NSCT SR had 3 best and 1 third-best metric values; LatLRR and DLF each had 3 best values.
- Quantitative performance comparison: Different methods led different metric families: NSCT SR excelled in CE, EN, and MI; LatLRR in AG, EI, and SF; and DLF in RMSE, SSIM, and PSNR.The results support using diverse metrics for comprehensive evaluation.
- Quantitative performance comparison: NSCT SR and LatLRR combined strong quantitative results with relatively poor qualitative performance, showing that both evaluation types are necessary.Their artifacts, detail loss, and target-visibility failures were reflected in weaker QCB and QCV performance.
- Runtime comparison: Runtime varied substantially: CBF took more than 50 times longer than GFF, while ResNet required 4.80 seconds per image pair.Multi-scale methods were generally fast, whereas deep learning methods were slower even with GPU assistance.
- Runtime comparison: Most fusion algorithms were computationally expensive for RGB-infrared tracking, limiting their feasibility for real-time trackers.The paper specifically identifies LatLRR and NSCT SR as examples of time-consuming methods.
5. Concluding Remarks
The paper concludes by presenting VIFB as a platform for evaluating visible and infrared image fusion methods. Its experiments identify unresolved differences across methods, metrics, visual judgments, and computational efficiency.
- Concluding remarks: VIFB combines 21 image pairs, 20 fusion algorithms, 13 evaluation metrics, and released results into a benchmark for gauging new methods.The authors describe it as the first visible and infrared image fusion benchmark to date.
- Concluding remarks: Deep learning-based fusion methods did not yet outperform non-learning algorithms, although the authors identify deep learning as an important future direction.The conclusion attributes this future relevance to deep learning’s strong representation ability.
- Concluding remarks: Comprehensive evaluation requires multiple metric types and both qualitative and quantitative comparisons because algorithms perform differently across metrics and visual assessments.The conclusion frames this as a central observation from the benchmark experiments.
- Concluding remarks: Computational efficiency remains a barrier to real-time applications such as tracking and detection.The authors identify efficiency improvement as an ongoing need.
- Concluding remarks: The authors plan to extend VIFB’s dataset and code library and implement additional evaluation metrics.They present the current benchmark as a starting point for researchers in visible and infrared image fusion.