Source-linked AI summary
TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework using Self-Supervised Multi-Task Learning
Linhao Qu, Shaolei Liu, Manning Wang, Zhijian Song
TL;DR
MEF seeks to fuse differently exposed images despite limited training data, non-real ground truth, and CNN difficulty modeling long-range dependencies. TransMEF uses self-supervised multi-task reconstruction in an encoder-decoder with CNN and transformer modules, trained on natural images. On a benchmark against 11 methods, it achieves state-of-the-art or best subjective and objective performance.
Problem
Deep learning MEF methods require substantial multi-exposure data, while CNNs have difficulty modeling long-range dependencies; synthesized ground truth can also be non-real.
Method
TransMEF trains an encoder-decoder on large natural image datasets using three self-supervised reconstruction tasks and a CNN-transformer encoder.
Results
TransMEF achieves the best subjective and objective performance against 11 competitive traditional and deep learning-based methods on the latest released MEF benchmark dataset.
Takeaways & Limitations
The framework learns multi-exposure characteristics from natural-image training while using both local and global information during feature extraction.
Takeaways & Limitations
Traditional MEF methods remain constrained by weak hand-crafted representations with low generalizability.
Abstract
from arXiv · showhide
In this paper, we propose TransMEF, a transformer-based multi-exposure image fusion framework that uses self-supervised multi-task learning. The framework is based on an encoder-decoder network, which can be trained on large natural image datasets and does not require ground truth fusion images. We design three self-supervised reconstruction tasks according to the characteristics of multi-exposure images and conduct these tasks simultaneously using multi-task learning; through this process, the network can learn the characteristics of multi-exposure images and extract more generalized features. In addition, to compensate for the defect in establishing long-range dependencies in CNN-based architectures, we design an encoder that combines a CNN module with a transformer module. This combination enables the network to focus on both local and global information. We evaluated our method and compared it to 11 competitive traditional and deep learning-based methods on the latest released multi-exposure image fusion benchmark dataset, and our method achieved the best performance in both subjective and objective evaluations.
1 Introduction
TransMEF addresses training-data and feature-extraction limitations in multi-exposure image fusion with self-supervised multi-task learning and a CNN-transformer encoder. Evaluated against 11 methods on a benchmark dataset, it achieves the best subjective and objective performance.
- Multi-exposure fusion combines differently exposed LDR images into one HDR image because single sensors can miss scene luminance ranges.
- Existing deep learning methods require substantial multi-exposure training data, while CNNs struggle to model long-range dependencies because of their small receptive fields.
- TransMEF trains an encoder-decoder network on large natural image datasets using self-supervised reconstruction, avoiding training with multi-exposure images.
- Three self-supervised reconstruction tasks are conducted simultaneously through multi-task learning to learn multi-exposure characteristics and extract more generalized features.
- The encoder combines CNN and transformer modules so feature extraction uses both local and global information.
- The method achieves the best subjective and objective performance against 11 traditional and deep learning-based methods on the latest released MEF benchmark dataset.
2 Related Work
MEF research spans traditional spatial and transform-domain methods and newer deep learning approaches. TransMEF responds to limitations involving hand-crafted representations, non-real ground truth, training-data demands, and domain discrepancy when using natural-image training.
- Traditional MEF methods use spatial-domain or transform-domain fusion, with transform methods combining frequency components after domain-specific transforms.
- Traditional methods are limited by weak hand-crafted representations with low generalizability, which hinders further improvement.
- Deep learning MEF networks can use ground truth fusion images or similarity metric-based losses, but both approaches require substantial multi-exposure training data.
- Synthesized ground truth images are not real and can lead to inferior fusion performance.
- Natural-image-trained encoder-decoder networks may not effectively learn multi-exposure characteristics because of domain discrepancy.
- TransMEF uses three self-supervised reconstruction tasks to train on large natural image datasets while learning multi-exposure characteristics.
3 Method
TransMEF trains an encoder-decoder through three self-supervised reconstruction tasks, then fuses source-image features during inference. Its encoder combines CNN and transformer modules to integrate local and global information.
- Framework Overview: The framework reconstructs original images from inputs whose subregions are altered by three transformations, using simultaneous multi-task learning.The transformations are gamma-based, Fourier-based, and global region shuffling.
- Framework Overview: During fusion, the trained encoder extracts feature maps from two source images, a fusion rule combines them, and the decoder reconstructs the fused image.The described pipeline initially concerns single-channel grayscale fusion; color fusion is addressed separately.
- TransBlock: TransBlock combines a CNN path and a transformer path to model local and global image dependencies.The CNN processes the destroyed image directly, while image patches form the transformer input sequence.
- Self-Supervised Tasks: The three reconstruction tasks target luminance and scene content, frequency-domain texture and detail, and structure and semantic information.Gamma-based transformation addresses luminance-related content, Fourier-based transformation targets texture and detail, and global region shuffling targets structure and semantics.
4 Experiments and Results
Experiments evaluate TransMEF on a 100-pair benchmark using subjective assessment and 12 objective metrics, with comparisons against 11 traditional and deep learning methods. TransMEF provides strong visual results and achieves the best performance on nine of 12 objective metrics.
- Dataset and setup: The test set contains 100 multi-exposure image pairs covering varied scenes, while MS-COCO supplies more than 70,000 natural training images resized to 256 × 256 grayscale.
- Evaluation metrics: The evaluation uses subjective quality assessment and 12 objective metrics spanning information theory, image features, structural similarity, and human perception.
- Compared methods: TransMEF is compared with 11 competitive traditional and deep learning-based MEF methods on the benchmark dataset.
- Subjective results: For the first example, TransMEF maintains the best luminance and contrast while displaying more details and better visual perception than the comparison methods.
- Subjective results: For the second example, TransMEF maintains optimal luminance and contrast while retaining more detailed information than competing methods.
- Objective results: 9 of 12 metrics: TransMEF achieves the best objective performance, with only small gaps from the best results on the remaining three metrics.
5 Ablation Study
Ablation studies using 20% of the training data isolate the effects of TransBlock and the three self-supervised reconstruction tasks. TransBlock consistently improves fusion performance, and joint multi-task training performs best overall.
- TransBlock: Adding TransBlock always improves fusion performance, regardless of whether the proposed self-supervised reconstruction tasks are used.
- TransBlock: The TransBlock model reconstructs better details than the traditional CNN architecture in the visualization study.
- Self-supervised tasks: Each self-supervised reconstruction task alone improves fusion performance, while conducting all three simultaneously through multi-task learning achieves the overall best performance.
6 Conclusion
The paper concludes that TransMEF combines self-supervised multi-task learning with an encoder integrating CNN and transformer modules for multi-exposure fusion. Extensive experiments report state-of-the-art subjective and objective performance against competitive methods.
- TransMEF uses an encoder-decoder structure trained on large natural image datasets, integrating CNN and transformer modules to capture local and global information.
- Three self-supervised reconstruction tasks are conducted simultaneously through multi-task learning to learn multi-exposure image characteristics during reconstruction.
- Extensive experiments show state-of-the-art subjective and objective performance compared with existing competitive methods.
- The proposed TransBlock and self-supervised reconstruction tasks have potential applications in other image fusion tasks and image processing areas.
A Illustration of Three Specific Self-Supervised Image Reconstruction Tasks
The reconstruction tasks perturb image luminance, frequency-domain information, or spatial regions, then train the network to recover the original image. These transformations target scene content, luminance, texture, and detail information.
- A.1 Gamma-based Transformation: Gamma-based transformation randomly changes luminance in selected image subregions by applying a Gamma transform with gamma uniformly sampled from [0, 3].
- A.1 Gamma-based Transformation: The Gamma task trains reconstruction from luminance-transformed subregions to learn scene content and luminance information.
- A.2 Fourier-based Transformation: The Fourier reconstruction task is designed to learn texture and detail information from the frequency domain.
- A.2 Fourier-based Transformation: The Fourier-based task blurs the amplitude spectrum and randomly shuffles phase values before reconstructing the original image.
- A.2 Fourier-based Transformation: The Fourier procedure transforms selected subregions by computing amplitude and phase spectra, modifying them, and applying inverse Fourier reconstruction.
- A.3 Global Region Shuffling: Global region shuffling selects subregions, selects another region, and swaps them to create transformed inputs for reconstruction.
B Comparisons of Image Reconstruction Results with CNN and Transformer
The TransMEF encoder combines CNN and transformer modules to use local and global information, improving reconstructed-image quality over CNN alone. Visual and objective comparisons support clearer reconstructions with TransBlock.
- The encoder combines CNN and transformer modules to model both local and global information during feature extraction.This design addresses CNNs’ difficulty establishing long-range dependencies.
- The comparison includes original images, CNN-only reconstructions, and TransBlock reconstructions across two examples.
- TransBlock produces clearer reconstructed images than CNN architecture alone, whose results are slightly blurred.Numbers in the examples are nearly as clear as in the original images with TransBlock.
- TransBlock yields lower MSE and higher SSIM and PSNR than CNN architecture alone.These metrics indicate better preservation of image details and reconstruction quality.
C Evaluation Metrics
The evaluation uses 12 objective metrics spanning information transfer, image features, and structural similarity. These metrics assess how well fused images preserve source information and image quality.
- The objective evaluation selects 12 metrics from four perspectives for a fair and comprehensive comparison.The listed metrics include QMI, QTE, QNICE, PSNR, FMI, QA/BF, and QP.
- QMI measures information transferred from source images to the fused image, while QTE measures correlation between source and fused-image Tsallis entropy.
- QNICE measures nonlinear source–fused-image correlation, whereas PSNR measures the ratio of peak-value power to noise power.
- FMI measures transferred feature information, and QA/BF measures transferred edge information from source images to fused images.
D More Fusion Results Visualization
Across three outdoor scenes, TransMEF preserves luminance and contrast while retaining image details. Competing methods more often produce dark, low-contrast, or artifact-prone fused images.
- In the first outdoor scene, TransMEF retains optimal luminance and contrast while displaying excellent details.Several methods produce dark images, while others maintain luminance but lose contrast or detail.
- In the second outdoor scene, TransMEF preserves the best luminance and contrast and retains more detailed information.Most methods show poor luminance or artifacts, while PMGI has unclear details.
- In the third outdoor scene, TransMEF maintains optimal luminance and contrast while presenting sufficient details.Several competing methods exhibit severe artifacts, low contrast, or lost detail information.
D.2 Indoor Scenes
In three indoor scenes, TransMEF consistently maintains strong luminance and contrast while preserving more details. Other methods variously show overexposure, dimness, artifacts, or detail loss.
- In the first indoor scene, U2Fusion and TransMEF achieve the relatively best visual perception.Other methods show dimness, overexposure, or poor contrast.
- In the second indoor scene, TransMEF and PMGI maintain optimal luminance and contrast while preserving more details.Competing methods introduce artifacts, dimness, low contrast, or detail loss.
- In the third indoor scene, TransMEF retains optimal luminance and contrast with more details.Other methods show artifacts, dimness, or low contrast with lost detail information.