PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Comparing Multi-exposure Image Fusion Algorithms

Xingchen Zhang

arXiv:2007.15156v1cs.CVeess.IV

TL;DR

MEF lacks a benchmark for fair and comprehensive comparisons, while hand-crafted features are limited under varying input conditions. The paper presents MEFB, a benchmark with a larger test set, algorithms, metrics, and fused images, finding that performance varies across metrics and deep learning-based methods are not dominant.

  • Problem

    MEF lacks extensive experiments and a benchmark for fair, comprehensive comparison, while under- or over-exposure can obscure image details.

  • Method

    The paper presents MEFB, comprising 100 image pairs, 16 MEF algorithms, 20 evaluation metrics, and 1600 fused images.

  • Results

    Performance varies significantly across metric types, and some deep learning-based algorithms do not show competitive performance compared with conventional algorithms.

  • Takeaways & Limitations

    MEF algorithms should be evaluated using different kinds of metrics rather than relying on a single metric.

  • Takeaways & Limitations

    Hand-crafted features are not robust to varying input conditions, limiting the performance of conventional MEF approaches.

Abstract

from arXiv · show

Multi-exposure image fusion (MEF) is an important area in computer vision and has attracted increasing interests in recent years. Apart from conventional algorithms, deep learning techniques have also been applied to multi-exposure image fusion. However, although much efforts have been made on developing MEF algorithms, the lack of benchmark makes it difficult to perform fair and comprehensive performance comparison among MEF algorithms, thus significantly hindering the development of this field. In this paper, we fill this gap by proposing a benchmark for multi-exposure image fusion (MEFB) which consists of a test set of 100 image pairs, a code library of 16 algorithms, 20 evaluation metrics, 1600 fused images and a software toolkit. To the best of our knowledge, this is the first benchmark in the field of multi-exposure image fusion. Extensive experiments have been conducted using MEFB for comprehensive performance evaluation and for identifying effective algorithms. We expect that MEFB will serve as an effective platform for researchers to compare performances and investigate MEF algorithms.

1. Introduction

MEF combines information from differently exposed images to recover details that a single under- or over-exposed capture may miss, but existing methods and evaluations lack robustness and standardization. MEFB addresses this gap with a shared benchmark, algorithms, metrics, fused outputs, and toolkit for broader comparison.

  • Motivation: A single camera image may fail to reveal all details because of under-exposure or over-exposure, while MEF combines information from multiple exposures.The paper motivates MEF as a way to address the limited capture range of common imaging sensors.
  • Existing methods: Conventional MEF approaches use hand-crafted features whose performance is limited because they are not robust to varying input conditions.The introduction contrasts these methods with newer deep learning approaches that offer strong representation abilities.
  • Evaluation gap: Fair comparison is difficult because studies use different test images, evaluation metrics, and compared algorithms.The paper identifies these inconsistent choices as obstacles to determining which MEF algorithm performs better.
  • Evaluation gap: The absence of a benchmark leaves deep learning-based and conventional MEF approaches without extensive, comprehensive performance comparisons.Several deep learning methods had been proposed, but the introduction notes that benchmark-based comparisons were lacking.
  • Proposed benchmark: MEFB enables extensive experiments comparing deep learning-based approaches with conventional algorithms, with the dataset made publicly available.Its test set covers diverse environments to assess fusion-algorithm generalization, while the toolkit facilitates performance evaluation.

2. Related work

MEF combines information from multiple images, but comparing MEF algorithms is difficult because studies use inconsistent test images, metrics, and evaluation practices. Existing methods and datasets do not provide a unified benchmark with shared resources for large-scale comparison.

  • MEF combines information from multiple images into a single, more informative image for downstream applications.
  • Image fusion methods operate at pixel, feature, or decision level in spatial or transform domains across several task types.
  • Conventional methods use hand-crafted features, while deep learning approaches include CNNs, GANs, Siamese networks, and autoencoders.
  • MEF evaluation is difficult because ground-truth images are normally unavailable and qualitative assessment is time-consuming, labor-intensive, and observer-dependent.
  • No evaluation metric is better than all others, so studies use different metrics, making performance comparison under a common standard infeasible.
  • Existing datasets provide test images but lack code libraries and fusion results, while MEFB combines a code library, metrics, fused images, and toolkit.

3.1. Dataset

MEFB is a test set of 100 under-/over-exposed image pairs assembled from Internet, existing datasets, and author-captured images. Its varied cameras, environments, resolutions, scenes, and objects are intended to assess robustness and generalization.

  • MEFB contains 100 multi-exposure image pairs, each comprising an under-exposed image and an over-exposed image.
  • The dataset is for testing rather than training and focuses on fusing one under-exposed image with one over-exposed image.
  • The dataset can be extended to image sequences containing more than two images.
  • The test set combines images from the Internet, existing MEF datasets, and images captured by the authors.
  • Images from varied cameras and places cover diverse working environments, supporting tests of algorithm robustness and generalization to varying inputs.
  • MEFB includes varied resolutions, indoor and outdoor scenes, day and night settings, and diverse objects.

3.2. Integrated algorithms

MEFB integrates 16 algorithms spanning dedicated MEF methods, general image-fusion methods, and recent deep learning approaches. Its toolkit also allows additional algorithms or fused images to be added.

  • MEFB integrates 16 algorithms, including DEM, DSIFT EF, FMMEF, GD, GFF, IFCNN, MEFAW, MEFCNN, MEFDSIFT, MEFNet, MEFOpt, MGFF, MTI, PMEF, PWA, and SPD-MEF.
  • FMMEF and SPD-MEF were designed specifically for MEF, whereas IFCNN and MGFF were originally designed for general image fusion including MEF.
  • IFCNN, MEFCNN, and MEFNet are recent deep learning-based MEF methods published in top journals or conferences.
  • Thirteen of the 16 integrated algorithms were published in the last four years.
  • The integrated methods cover multiple kinds of MEF methods and are intended to represent state-of-the-art MEF approaches.
  • Additional algorithms can be incorporated by adding source code through the toolkit interface or adding fused images.

3.3. Evaluation metrics

MEFB implements more than 20 evaluation metrics spanning four categories because MEF algorithms can perform differently across metric types. The toolkit therefore supports broader quantitative comparison of fused-image quality.

  • Metric categories: Image feature-based metrics measure gradients, edges, or textures, while structural metrics assess similarity between fused and source images.Examples include AG, EI, QAB/F, QP, SD, SF, QC, QW, QY, and MEF-SSIM.
  • Metric categories: MEFB groups its metrics into information theory-based, image feature-based, structural similarity-based, and human perception-inspired categories.The implemented metrics include CE, EN, FMI, NMI, PSNR, QNCIE, TE, AG, EI, QAB/F, QP, SD, SF, QC, QW, QY, MEF-SSIM, QCB, QCV, and VIF.
  • Metric coverage: MEFB implements more than 20 metrics covering all four metric categories, enabling quantitative assessment of fused-image quality.The paper describes this as the first MEF work implementing more than 20 metrics across all four kinds.
  • Metric interpretation: Larger values generally indicate better fusion performance, except for CE and QCV; EN is usually auxiliary because noise can strongly affect it.The paper also states that large values of QAB/F, SD, SF, SSIM, and related metrics indicate better fusion performance.
  • Metric categories: Information-theory metrics quantify information or feature transfer, whereas human-perception metrics evaluate fusion according to perceptual criteria.MI measures information transferred from source images, and FMI measures transferred feature information; QCB, QCV, and VIF are human-perception-inspired metrics.

3.4. The software toolkit

The MEFB software toolkit facilitates algorithm execution and performance evaluation through interfaces for images, algorithms, and metrics. Its extensible workflow supports both in-tool fusion and external fused images.

  • Toolkit interfaces: The toolkit provides separate interfaces for adding test or fused images, MEF algorithms, and evaluation metrics.These interfaces are designed to make MEFB easy to extend.
  • Usage modes: Users can add algorithms to MEFB, generate fused images, and compute metrics within the toolkit.After settings are configured, running algorithms and computing metrics can be performed with a single button press.
  • Usage modes: Users can also import fused images produced outside the toolkit and compare their algorithms with methods already included in MEFB.This provides an alternative workflow when users do not want to integrate their algorithm directly.

4. Experiments and results

Experiments on MEFB compare 16 MEF approaches qualitatively, quantitatively, and computationally using publicly available implementations with default settings. Results show strong method- and metric-dependent variation, with MGFF frequently performing well but qualitative and quantitative rankings not always agreeing.

  • Experimental setup: The experiments evaluate publicly available implementations with default settings, using an RTX2070 GPU and i7-9750H CPU; MEFB is used only for evaluation, not training.Pre-trained deep-learning models were supplied by their authors.
  • Qualitative comparison: MGFF gives the best qualitative result on the buildingRoad image pair.The comparison also reports artifacts, color distortion, and over-exposure in outputs from several other methods.
  • Qualitative comparison: MGFF produces the best qualitative result on the flower image pair, with the most uniform intensity around the flower among GD, MGFF, and MTI.Most methods produce severe artifacts in this challenging case; IFCNN introduces color distortion, while PWA and SPD MEF remain somewhat under-exposed.
  • Quantitative comparison: MGFF, IFCNN, and FMMEF rank among the top three quantitatively, but each performs well in different metric categories.MGFF is strongest mainly on image-feature and human-perception metrics, IFCNN on information-theory metrics and selected feature metrics, and FMMEF on feature and structural-similarity metrics.
  • Quantitative comparison: Different metric categories examine MEF approaches from different aspects, so evaluating algorithms with only a few metric types is inadequate.The paper argues that broad metric coverage is essential for comprehensive performance comparison.
  • Evaluation consistency: Quantitative and qualitative rankings are not always consistent: IFCNN is second quantitatively but produces severe color distortion qualitatively.The experiments therefore use both quantitative and qualitative evaluation methods.
  • Computational cost: Running time varies substantially across methods; GPU-assisted deep-learning methods are generally efficient, whereas patch-based methods cost more than pixel-based methods.MEFOpt takes around seven minutes per image pair because of iterative optimization.

5. Discussions

The discussion presents MEFB as a test bed for fairer MEF evaluation by standardizing a larger test set and broad metric coverage. It also notes that deep-learning performance varies substantially and that the dataset should be expanded.

  • Benchmark role: MEFB addresses the lack of platforms for fair MEF performance evaluation by providing a reusable test bed.The authors compare its intended role for MEF with OTB and VOT for object tracking.
  • Dataset scope: The MEFB dataset contains 100 image pairs, which is larger than the image sets previously used in MEF literature.The authors identify the dataset as a test set rather than a training set.
  • Limitations and future work: The authors plan to add more image pairs because the current 100-pair test set, although larger than prior sets, could be expanded.This is the stated future scope boundary for the dataset.
  • Algorithm comparison: IFCNN is competitive with conventional approaches, whereas the other examined deep-learning methods fail to outperform many conventional algorithms.The authors suggest that the previous lack of benchmarks made comprehensive testing across metric perspectives difficult.

6. Conclusions

The paper presents MEFB as a benchmark and toolkit for comprehensive comparison of multi-exposure image fusion methods. Experiments using MEFB show that deep learning methods are not dominant, while evaluation outcomes vary substantially across metrics.

  • Benchmark contribution: MEFB provides a benchmark platform and toolkit for comprehensive performance comparison of MEF algorithms.It is described as a first benchmark for multi-exposure image fusion.
  • Benchmark contents: MEFB currently contains 100 image pairs, 16 MEF algorithms, 20 evaluation metrics, 1600 fused images, and a software toolkit.
  • Extensibility: The toolkit can be extended with more images, methods, and evaluation metrics.
  • Experimental findings: Deep learning-based methods are not dominant in MEF because some do not show competitive performance against conventional algorithms.
  • Experimental findings: MEF algorithm performance varies significantly across metrics, making evaluation with different kinds of metrics essential.The findings also suggest considering several metrics together when designing loss functions instead of relying only on MEF-SSIM.
Loading 2007.15156v1…