Source-linked AI summary
Attention-guided Network for Ghost-free High Dynamic Range Imaging
Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, Yanning Zhang
TL;DR
Dynamic HDR reconstruction is challenged by ghosting from motion, misalignment, and saturation, while existing alignment and deep-learning methods remain limited in severe cases. AHDRNet uses reference-guided attention and dilated residual dense blocks in an end-to-end network without required alignment preprocessing. The authors report high-quality ghost-free HDR images under large motion and saturation, with extensive experiments validating the method.
Problem
Dynamic-scene HDR methods remain vulnerable to ghosting, alignment errors, large movements, occlusion, and extensive saturation.
Method
AHDRNet uses reference-guided attention to merge LDR features and dilated residual dense blocks to preserve details and enlarge receptive fields.
Results
AHDRNet produces high-quality HDR images under large image motion and saturation, with experiments reporting better numerical performance than other methods.
Takeaways & Limitations
AHDRNet offers the prospect of more extensive HDR imaging applications in scenes containing large motion and saturation.
Takeaways & Limitations
Existing deep HDR methods still cannot handle large areas of saturation, particularly when occlusion is also present.
Abstract
from arXiv · showhide
Ghosting artifacts caused by moving objects or misalignments is a key challenge in high dynamic range (HDR) imaging for dynamic scenes. Previous methods first register the input low dynamic range (LDR) images using optical flow before merging them, which are error-prone and cause ghosts in results. A very recent work tries to bypass optical flows via a deep network with skip-connections, however, which still suffers from ghosting artifacts for severe movement. To avoid the ghosting from the source, we propose a novel attention-guided end-to-end deep neural network (AHDRNet) to produce high-quality ghost-free HDR images. Unlike previous methods directly stacking the LDR images or features for merging, we use attention modules to guide the merging according to the reference image. The attention modules automatically suppress undesired components caused by misalignments and saturation and enhance desirable fine details in the non-reference images. In addition to the attention model, we use dilated residual dense block (DRDB) to make full use of the hierarchical features and increase the receptive field for hallucinating the missing details. The proposed AHDRNet is a non-flow-based method, which can also avoid the artifacts generated by optical-flow estimation error. Experiments on different datasets show that the proposed AHDRNet can achieve state-of-the-art quantitative and qualitative results.
1. Introduction
HDR imaging combines differently exposed LDR images to recover a broad illumination range, but motion, saturation, and alignment errors can cause ghosting and missing details. AHDRNet addresses these issues with reference-guided attention and dilated residual dense blocks, supported by experiments and ablations.
- Motivation: HDR imaging merges differently exposed LDR images to represent a broader range of illuminations than individual sensors can capture.Natural luminance spans several orders of magnitude, while digital sensors measure only a limited fraction.
- Challenges: Motion between exposure images causes significant ghosting and blurring, despite high-quality results in static scenes.Multiple-exposure methods typically designate one LDR image as the reference.
- Challenges: Optical-flow alignment remains vulnerable to estimation errors, while existing deep methods struggle with large movement, occlusion, and extensive saturation.These limitations can leave distortions, ghosting, or missing details in HDR outputs.
- Proposed Approach: AHDRNet uses learnable attention maps guided by the reference image to emphasize complementary features and exclude motion or severe saturation.Unlike early feature stacking, attention guides the merging process before the final HDR reconstruction.
- Proposed Approach: Dilated residual dense blocks preserve hierarchical details, enlarge receptive fields, and help hallucinate content in saturated regions.The blocks merge attention-guided feature maps from the LDR images.
- Evaluation: Extensive experiments across different datasets and ablations evaluate AHDRNet’s performance and the roles of its components.The paper reports superiority of the proposed model in its experimental evaluation.
2. Related Work
Prior HDR methods address ghosting through pixel rejection, optical-flow or patch-based correspondences, and joint alignment with reconstruction. These approaches use image differences, variance, motion vectors, or optimization to identify or align corresponding content.
- Methods relying on pixel rejection: Pixel-rejection methods classify pixels as static or moving under a global-registration assumption.Grosch used an error map based on input color differences, while Jacobs et al. detected ghost regions using weighted variance.
- Patch-based methods: Patch-based methods use dense patch correspondences or joint optimization to address alignment and HDR reconstruction.Sen et al. proposed patch-based energy minimization integrating alignment with HDR reconstruction; the passage also describes patch-based dense correspondences.
- Optical-flow and correspondence methods: Optical-flow methods estimate dense correspondences and warp pixels across differently exposed images.Bogoni estimated motion vectors for pixel warping, and Kang et al. computed optical flow after transforming LDR intensities into the luminance domain using exposure times.
3. Attention-guided Network for HDR Imaging
AHDRNet merges three exposure-varied LDR images through reference-guided attention and a DRDB-based merging network, without alignment preprocessing. Attention suppresses misaligned or saturated features, while DRDBs and global residual learning support detailed HDR reconstruction.
- Input and reference image: AHDRNet uses three exposure-varied LDR images, with the middle-exposure image serving as the reference.The inputs are gamma-corrected into the HDR domain and concatenated with their original LDR images before network processing.
- Network overview: The end-to-end network can process original input images without optical-flow or other alignment preprocessing.Its attention mechanism is integrated into the HDR network rather than applied as a separate registration stage.
- Merging network: The merging network uses DRDBs and global residual learning to exploit features and reconstruct plausible HDR details.DRDBs enlarge receptive fields for recovering details contaminated by saturation and moving objects, while the residual connection uses shallow reference features.
- Attention network: Reference-guided attention compares encoded reference and non-reference features, then weights non-reference features before merging.The attention network extracts 64-channel feature maps, predicts attention maps for non-reference images, and concatenates the attended features with the reference features.
- Attention network: The attention maps suppress misaligned and saturated regions while highlighting useful non-reference details when reference regions are saturated or noisy.The maps are bounded in [0,1] through convolutional layers followed by sigmoid activation.
- Training loss: Training minimizes the ℓ1 distance between tonemapped estimated and ground-truth HDR images, because ℓ1 was observed to preserve details better than ℓ2.The final HDR result is displayed through tonemapping.
4. Experiments
Experiments evaluate AHDRNet on ground-truth and ground-truth-free datasets, including architecture ablations, loss comparisons, and state-of-the-art baselines. Results show that attention and dilated residual dense blocks improve ghost suppression, detail recovery, and overall HDR quality.
- Experimental setup: AHDRNet is trained on 74 HDR samples and tested on 15 samples using three LDR exposures with biases of {−2, 0, +2} or {−3, 0, +3}.Testing also includes Kalantari’s dataset and Sen’s and Tursun’s datasets without ground truth.
- Model architecture: The attention module suppresses ghosting while retaining background information, improving over variants without attention.The ablation compares AHDRNet with DRDB-Net, A-RDB-Net, RDB-Net, RB-Net, and Deep-RB-Net.
- Model architecture: Dilated residual dense blocks capture more content and alleviate ghosts more effectively than residual-block baselines, even when network depth is increased.The results associate dilated convolutions with larger receptive fields, missing-detail hallucination, and higher PSNR values.
- Training loss: The ℓ1 loss is selected because the loss comparison indicates stronger detail preservation.The authors train AHDRNet using ℓ1 loss.
- Evaluation on ground-truth data: On Kalantari et al.’s testing data, AHDRNet achieves the best PSNR-µ and PSNR-M results among the compared methods.The evaluation averages results over 15 testing images; Ours+OF is slightly better or competitive, and outperforms under the same alignment process.
- Evaluation without ground truth: On datasets without ground truth, AHDRNet produces appealing results with reduced geometry distortion, color artifacts, and noise.Visual comparisons report sharper details and fewer artifacts than existing methods.
5. Conclusion
The paper concludes that attention-based HDR reconstruction addresses ghosting and saturation artifacts in multiple-exposure imaging. AHDRNet generates high-quality HDR images under large motion and saturation, suggesting broader HDR applications.
- Conclusion: AHDRNet overcomes ghosting and saturation limitations that restrict multiple-exposure HDR imaging.The conclusion specifically emphasizes dynamic scenes with large image motion and saturation.
- Conclusion: AHDRNet generates high-quality HDR images even in the presence of large image motion and saturation.The authors state that this offers the prospect of more extensive HDR imaging applications.