Source-linked AI summary
SESF-Fuse: An Unsupervised Deep Model for Multi-Focus Image Fusion
Boyuan Ma, Xiaojuan Ban, Haiyou Huang, Yu Zhu
TL;DR
Multi-focus fusion must combine differently focused images because limited depth of field leaves objects at different distances unequally sharp. SESF-Fuse trains an unsupervised encoder-decoder, measures activity with spatial frequency on deep features, and adjusts a decision map through consistency verification. The authors report promising or state-of-the-art performance in objective and subjective assessments compared with existing fusion methods.
Problem
Limited depth of field makes objects at different distances difficult to capture sharply in one image, motivating fusion of differently focused views.
Method
An unsupervised encoder-decoder extracts deep features, spatial frequency measures their activity, and consistency verification adjusts the decision map for fusion.
Results
SESF-Fuse generally outperforms other methods and achieves state-of-the-art multi-focus fusion performance in objective and subjective assessments.
Takeaways & Limitations
The paper demonstrates the viability of combining unsupervised learning with traditional image-processing algorithms for image fusion.
Takeaways & Limitations
The approach assumes that in-focus objects are sharp while objects outside the depth of field are likely blurred.
Abstract
from arXiv · showhide
In this work, we propose a novel unsupervised deep learning model to address multi-focus image fusion problem. First, we train an encoder-decoder network in unsupervised manner to acquire deep feature of input images. And then we utilize these features and spatial frequency to measure activity level and decision map. Finally, we apply some consistency verification methods to adjust the decision map and draw out fused result. The key point behind of proposed method is that only the objects within the depth-of-field (DOF) have sharp appearance in the photograph while other objects are likely to be blurred. In contrast to previous works, our method analyzes sharp appearance in deep feature instead of original image. Experimental results demonstrate that the proposed method achieves the state-of-art fusion performance compared to existing 16 fusion methods in objective and subjective assessment.
Introduction
Multi-focus image fusion addresses the limited depth of field that prevents all scene objects from appearing sharp in one photograph. SESF-Fuse uses unsupervised deep features and spatial frequency to fuse differently focused images, achieving state-of-the-art performance against 16 existing methods.
- Motivation: Limited optical depth of field makes it difficult to capture all objects at different distances in focus within one shot.Fusion combines images with different focus points to create an all-in-focus result.
- Motivation: Fused all-in-focus images support human and computer operators as well as segmentation, feature extraction, and object recognition.
- Related approaches: Prior deep-learning fusion methods include supervised CNN-based multi-focus fusion and unsupervised approaches for multi-exposure or infrared-visible fusion.The cited CNN method uses synthetic images with different blur levels to train a two-class focus classifier.
- Key assumption: For multi-focus fusion, the authors assume in-focus objects are sharp while other objects are likely blurred, making feature gradient more relevant than feature intensity.This motivates measuring activity from feature gradients rather than the L1 norm of feature vectors.
- Contribution: The proposed unsupervised deep convolutional method extracts deep features and uses spatial frequency to measure activity level for fusion.The paper reports state-of-the-art performance against 16 existing fusion methods in objective and subjective assessment.
Related work
Earlier multi-focus fusion methods use transform-domain coefficients or spatial-domain gradient information to estimate activity. Deep CNNs introduced learned representations, but the field spans diverse transform, block-based, pixel-based, and deep-learning strategies.
- Method categories: Image fusion methods are broadly classified into transform-domain and spatial-domain approaches.
- Transform-domain methods: Transform-domain methods estimate source-image activity from coefficients produced by a selected transform, making transform choice important.Examples include Laplacian and ratio-of-low-pass pyramids, wavelet transforms, curvelet transforms, sparse representation, and image matting.
- Spatial-domain methods: Early spatial-domain methods measured activity with fixed-size blocks and spatial frequency, but could produce undesirable artifacts.Adaptive block methods were proposed to optimize block size, while later approaches used pixel-level gradient information.
- Spatial-domain methods: Pixel-based spatial-domain methods include gradient-based approaches such as dense SIFT-based fusion.
- Deep-learning methods: Deep CNNs have been applied to image fusion, including supervised multi-focus fusion, unsupervised multi-exposure fusion, and DenseFuse for infrared-visible fusion.
Method
The method learns deep features with an unsupervised encoder-decoder, then uses feature-based spatial frequency to build and refine a decision map for fusing two registered multi-focus images.
- Feature extraction: An auto-encoder reconstructs input images during training, after which its fixed encoder extracts deep features for fusion.The network omits fusion during training and reconstructs the input image instead.
- Feature extraction: The encoder uses convolutional and SEDense blocks without pooling, while the decoder reconstructs images using layers C2 through C5.The training loss combines pixel loss and SSIM loss, with λ normalizing their contributions.
- Spatial frequency calculation: Unlike DenseFuse, the method measures activity from feature gradients using spatial frequency rather than feature intensity.Spatial frequency is adapted to the encoder’s high-dimensional feature vector at each image pixel.
- Spatial frequency calculation: Corresponding spatial frequencies from the two source images are compared to produce an initial pixel-based decision map.The modified calculation uses row and column vector frequencies on deep features, with same padding at feature-map borders.
- Consistency verification: Consistency verification alternates morphological opening and closing, removes small regions, and applies guided filtering to improve the decision map.The small-region threshold is usually 0.01 × H × W, and the initial fused image guides the edge-preserving filter.
- Fusion: The refined decision map selects pixels from two pre-registered source images through a pixelwise weighted-average fusion rule.The algorithm is designed for two source images, while additional images can be fused sequentially.
Experiments
Experiments evaluate SESF-Fuse on public multi-focus image pairs using ablations, visual comparisons, and three objective fusion-quality metrics. The method generally outperforms competing approaches, while its ablation results support using spatial frequency on deep features and decision-map fusion.
- Experimental Setup: 38 public multi-focus image pairs were used for testing, while the unsupervised encoder-decoder was trained and validated on MS-COCO images.The training set contained about 82,783 images and the validation set 40,504 images; images were resized to 256 × 256 and converted to grayscale.
- Experimental Setup: SESF-Fuse was compared with 16 representative fusion methods, including transform-domain, sparse-representation, and CNN-based approaches.The comparison included LP, RP, NSCT, DWT, DTCWT, SR, and CNN-Fuse among others.
- Ablation Experiments: Spatial-frequency fusion of deep features outperformed max, abs-max, average, and L1-norm alternatives, while decision-map fusion improved over directly decoding spatial-frequency-fused features.The ablation evaluated seven fusion modes and found feature gradient, calculated by spatial frequency, suited to multi-focus fusion; the direct deep-feature fusion could not completely recover every in-focus detail.
- Ablation Experiments: Squeeze-and-excitation decision-map fusion achieved higher average Qg and Qm scores than dense spatial-frequency decision-map fusion and had the highest first-place count.The authors associate this robustness with dynamic recalibration of features by the squeeze-and-excitation structure.
- Visual Comparison: SESF-Fuse preserved focused regions more accurately than several competing methods in visual tests, including leaf boundaries, a koala ear, a beer bottle, and a person.Difference-image comparisons showed stronger detection in both center and boundary regions of near-focused areas, whereas several methods retained visible contours or misdetected regions.
- Objective Comparison: SESF-Fuse and CNN-Fuse clearly exceeded the other 15 methods on average Qg and Qcb scores, while SESF-Fuse achieved the highest first-place count for Qm despite a lower average score than LP.CNN-Fuse and SESF-Fuse had comparable Qg performance, but CNN-Fuse required supervised training with synthetic blurred images whereas SESF-Fuse used an unsupervised model.
- Overall Results: Across subjective visual quality and objective metrics, SESF-Fuse generally outperformed the alternatives and achieved state-of-the-art multi-focus image-fusion performance.This conclusion combines the visual comparisons with the three metric-based evaluations.
Conclusion
The paper presents an unsupervised deep model that combines encoder-decoder features with spatial frequency for multi-focus image fusion, achieving promising performance against existing fusion methods in objective and subjective assessment.
- SESF-Fuse trains an encoder-decoder network unsupervised to acquire deep features, then combines them with spatial frequency to calculate activity levels and decision maps.
- The fusion pipeline uses the calculated activity levels and decision maps to perform image fusion.
- The proposed approach achieves promising fusion performance compared with existing fusion methods in objective and subjective assessment.
- The authors describe the work as demonstrating the viability of combining unsupervised learning with traditional image-processing algorithms.
- The authors suggest that the same strategy could be applied to multi-exposure, infrared-visible, and medical image fusion.