Source-linked AI summary
DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs
K. Ram Prabhakar, V. Sai Srikar, R. Venkatesh Babu
TL;DR
Multi-exposure fusion needs methods that handle varying conditions and extreme exposure pairs despite limited training data and absent reference outputs. DeepFuse addresses this with an unsupervised CNN trained using a no-reference quality metric, and extensive evaluations report better performance than state-of-the-art approaches across varied scenarios.
Problem
Existing MEF methods rely on hand-crafted features that are not robust to varying inputs and perform poorly on extreme exposure pairs, while reference ground truth for training is unavailable.
Method
DeepFuse learns exposure-pair fusion with a CNN that combines common low-level features and uses MEF SSIM as a no-reference loss without reference ground truth.
Results
DeepFuse outperforms existing approaches across varied input scenarios and produces artifact-free, perceptually pleasing results for extreme exposure image pairs.
Takeaways & Limitations
Unsupervised perceptual-quality training can support CNN-based static multi-exposure fusion without reference images, including extreme exposure pairs.
Takeaways & Limitations
The input exposure images are assumed to be registered and aligned, avoiding camera and object motion.
Abstract
from arXiv · showhide
We present a novel deep learning architecture for fusing static multi-exposure images. Current multi-exposure fusion (MEF) approaches use hand-crafted features to fuse input sequence. However, the weak hand-crafted representations are not robust to varying input conditions. Moreover, they perform poorly for extreme exposure image pairs. Thus, it is highly desirable to have a method that is robust to varying input conditions and capable of handling extreme exposure without artifacts. Deep representations have known to be robust to input conditions and have shown phenomenal performance in a supervised setting. However, the stumbling block in using deep learning for MEF was the lack of sufficient training data and an oracle to provide the ground-truth for supervision. To address the above issues, we have gathered a large dataset of multi-exposure image stacks for training and to circumvent the need for ground truth images, we propose an unsupervised deep learning framework for MEF utilizing a no-reference quality metric as loss function. The proposed approach uses a novel CNN architecture trained to learn the fusion operation without reference ground truth image. The model fuses a set of common low level features extracted from each image to generate artifact-free perceptually pleasing results. We perform extensive quantitative and qualitative evaluation and show that the proposed technique outperforms existing state-of-the-art approaches for a variety of natural images.
1. Introduction
The paper targets exposure fusion for difficult and extreme lighting by learning from exposure pairs rather than relying on hand-crafted features. It introduces an end-to-end CNN trained without reference images and reports strong performance across varied conditions.
- HDRI captures a wider range of scene brightness, helping avoid overly dark or saturated regions in difficult lighting.
- Conventional MEF commonly fuses multiple differently exposed LDR images, but small exposure gaps often require more than two images.Using more images increases storage, processing time, and power requirements.
- The proposed method learns to fuse exposure-bracketed image pairs with a CNN that extracts and combines common low-level features.Its network is trained end-to-end using a no-reference image-quality loss rather than reference ground truth.
- The model is trained and tested on diverse exposure stacks spanning indoor, outdoor, day, night, side-lighting, and back-lighting conditions.The authors report no need for parameter fine-tuning across varying inputs.
- The work contributes a CNN-based unsupervised fusion algorithm, a benchmark dataset, and comparisons with 7 state-of-the-art algorithms.
2. Related Works
Earlier exposure-fusion methods generally combine inputs using hand-crafted weights or features, with reported artifacts and sensitivity to input conditions. The paper motivates CNN-based feature learning directly from raw pixels as an alternative.
- Traditional exposure-fusion algorithms commonly compute local or pixel-wise weights and form a weighted sum of the input images.
- Existing methods use hand-crafted measures such as local energy, correlation, contrast, saturation, luminance, gradients, or color consistency.
- Reported weaknesses include block artifacts, hallucinated edges, mismatched colors, dark regions, and halo effects from particular fusion strategies.
- Hand-crafted approaches may require parameter changes for different input conditions, including exposure type, filter size, and image size.
- The proposed alternative uses CNNs to learn suitable fusion features directly from raw pixels instead of relying on hand-crafted inputs.
3. Proposed Method
The proposed method uses a CNN to learn exposure fusion from aligned image pairs without reference outputs, combining shared low-level features and optimizing a no-reference quality loss. It fuses luminance through the network, handles chrominance separately, and evaluates blending choices and training data for practical deployment.
- Problem formulation: The CNN models fusion as a feed-forward mapping from an aligned exposure sequence, avoiding camera and object motion.The input exposure sequence is assumed to be registered and aligned using existing registration algorithms.
- Network architecture: The architecture extracts common low-level features with weight-tied pre-fusion channels, merges paired features, and reconstructs the fused image.The network comprises feature-extraction, fusion, and reconstruction components; shared weights encourage corresponding inputs to produce the same feature type.
- MEF SSIM loss function: MEF SSIM supplies the no-reference loss because the expected fused output is unavailable for supervised error computation.It compares input-patch statistics with the fused result and measures structural integrity and luminance consistency across multiple scales.
- Color-channel fusion: The method fuses the luminance channel with the CNN, then combines it with chrominance channels to form the final RGB image.Luminance is selected because structural details and brightness variation are prominent there; chrominance is fused separately because MEF SSIM is formulated for grayscale images.
- MEF SSIM loss function: MEF SSIM discards local luminance comparison, combines contrast and structure across scales, and backpropagates the resulting image-wide loss.The desired contrast uses the highest input contrast, while structural weights depend on consistency between input patches; the total loss is computed over image pixels.
- Training and evaluation: The training corpus combines 25 public and 50 curated exposure stacks, each containing two images separated by ±2 EV, with 30000 cropped 64 ×64 patches.Images were resized to 1200 × 800 and collected across indoor and outdoor scenes using a standard camera setup and tripod.
4. Experiments and Results
Experiments compare DeepFuse with seven state-of-the-art methods using MEF SSIM and qualitative image sequences spanning varied lighting and exposure conditions. DeepFuse generally preserves details and brightness while reducing artifacts, and it also extends to multi-focus fusion.
- Experimental setup: The evaluation covers indoor and outdoor, day and night, natural and artificial lighting, and linear and non-linear exposure sequences against seven MEF algorithms.MEF SSIM is used for objective evaluation because the cited alternatives do not adequately predict subjective opinions.
- Loss comparison: SSIM-trained CNN outputs are sharper and more artifact-free than outputs trained with ℓ2 or ℓ1 losses.The ℓ2 result is blurred, while the ℓ1 result is sharper but exhibits halo effects along edges.
- Baseline comparison: DeepFuse’s unsupervised version achieves superior MEF SSIM performance over the baseline in almost all test sequences.The baseline is bounded by using Mertens or Li results as ground truth and does not exceed both methods.
- Comparison with state-of-the-art: DeepFuse produces consistent brightness and preserves finer structural details where Mertens et al. loses tile and lamp details from the under-exposed image.The paper attributes this preservation to learned edge- and texture-extracting filters in C1 and C2.
- Comparison with state-of-the-art: Li et al. and Shen et al. exhibit non-uniform brightness or contrast-loss artifacts, whereas DeepFuse provides clearer texture details and avoids the reported distortions.For Shen et al., the cited brightness distortion appears in cloud regions between balloons.
- Comparison with state-of-the-art: Ma et al.’s patch-strength weighting introduces halo effects around lamp edges and along the lighthouse roof, while DeepFuse is also applied to multi-focus fusion.On a multi-focus dataset, the CNN produces an all-in-focus result and its filters identify suitable regions from varying-focus inputs.
5. Conclusion and Future work
DeepFuse efficiently fuses varied-exposure static image pairs through common-feature merging and reconstruction, producing artifact-free, perceptually pleasing results without parameter fine-tuning. The method performs better than state-of-the-art approaches across diverse input scenarios and shows promise for broader fusion tasks.
- Conclusion: DeepFuse extracts common low-level features from each input, merges feature pairs, and reconstructs the final fused image.The network is trained end-to-end with a no-reference image quality loss.
- Conclusion: DeepFuse produces artifact-free, perceptually pleasing outputs for static image pairs with varied exposure levels.
- Conclusion: DeepFuse requires no parameter fine-tuning across varying input conditions.Training and testing use exposure stacks captured under diverse settings.
- Conclusion: Extensive quantitative and qualitative evaluations show DeepFuse performs better than state-of-the-art approaches across a wide range of input scenarios.
- Future work: DeepFuse achieves better fusion quality for extreme exposure image pairs and supports using perceptual metrics such as SSIM and MEF SSIM without ground-truth images.The authors suggest adapting the generic fusion method to other fusion problems and future motion-image fusion.