Source-linked AI summary
Deep High Dynamic Range Imaging with Large Foreground Motions
Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, Chi-Keung Tang
TL;DR
HDR imaging from LDR bursts is difficult when foreground motion, occlusion, saturation, and exposure differences defeat optical-flow alignment. This paper replaces flow-based merging with image translation, reporting fewer artifacts and distortions while recovering plausible missing highlight details across varied inputs. The approach still faces challenges when massive saturated regions must be recovered from very few LDR inputs.
Problem
Large foreground motions make optical-flow alignment unreliable and cause artifacts or distortions when merging multiple LDR exposures into HDR images.
Method
The paper formulates HDR merging as image translation and uses an end-to-end encoder-decoder CNN without optical-flow alignment.
Results
The non-flow-based CNN outperforms state-of-the-art HDR methods, avoids flow-induced distortions and artifacts, and recovers plausible details in saturated regions across varied inputs.
Takeaways & Limitations
Learned multi-LDR translation provides a practical HDR synthesis alternative for dynamic scenes with large foreground motions and missing or weakly captured details.
Takeaways & Limitations
Recovering massive saturated regions remains challenging when only a minimal number of LDR inputs is available.
Abstract
from arXiv · showhide
This paper proposes the first non-flow-based deep framework for high dynamic range (HDR) imaging of dynamic scenes with large-scale foreground motions. In state-of-the-art deep HDR imaging, input images are first aligned using optical flows before merging, which are still error-prone due to occlusion and large motions. In stark contrast to flow-based methods, we formulate HDR imaging as an image translation problem without optical flows. Moreover, our simple translation network can automatically hallucinate plausible HDR details in the presence of total occlusion, saturation and under-exposure, which are otherwise almost impossible to recover by conventional optimization approaches. Our framework can also be extended for different reference images. We performed extensive qualitative and quantitative comparisons to show that our approach produces excellent results where color artifacts and geometric distortions are significantly reduced compared to existing state-of-the-art methods, and is robust across various inputs, including images without radiometric calibration.
1 Introduction
The paper targets HDR reconstruction from misaligned LDR stacks with large foreground motions, where optical-flow alignment and existing merging methods can introduce artifacts or discard useful content. It instead treats HDR merging as image translation and uses an end-to-end network that avoids optical flow while hallucinating missing details.
- Problem: Large foreground motions create unavoidable foreground-background misalignments that make HDR composition difficult.Small camera motions can often be addressed with homography, but foreground motion causes artifacts, ghosting, or rejected pixels.
- Problem: Optical-flow-based HDR methods can produce geometric distortions and color artifacts when large object motions and different exposures make flows unreliable.These errors may persist through the final HDR synthesis.
- Approach: The paper formulates multi-exposure HDR merging as image translation rather than optical-flow alignment.The proposed approach is motivated by learned image mappings, CNN-based misalignment handling, and hallucination of missing details.
- Contributions: The end-to-end network avoids artifacts and distortions caused by erroneous flows and can hallucinate plausible details missing from all LDR inputs because of saturation and occlusion.The framework is also described as faster and more practical than prevailing flow-based approaches.
- Evaluation: Extensive qualitative and quantitative comparisons report that the method outperforms state-of-the-art HDR synthesis approaches and remains robust across varied LDR inputs.The reported input variations include different exposure separations and images without radiometric calibration.
2 Related Work
Prior HDR methods address camera or object motion through registration, pixel rejection, ghosting correction, optical flow, or patch-based correspondence, each with important limitations. The paper positions multi-LDR translation as a way to use burst inputs while avoiding flow-related artifacts and single-image saturation failures.
- Static-scene HDR: Early multi-exposure HDR methods work well for static scenes and static cameras.Their setting does not address the harder case of moving objects.
- Camera motion: Camera-motion methods register LDR images before merging and often transform images to handle brightness changes during registration.Examples include luminance-domain and gradient-domain transformations.
- Object motion: Object-motion methods either reject moving pixels during merging or detect ghosting afterward, thereby failing to fully use available image content.These approaches treat misaligned pixels as unusable rather than reconstructing their HDR information.
- Flow-based methods: Optical-flow methods address dense pixel correspondence but can introduce artifacts and distortions for large displacements and complicate HDR merging.The paper distinguishes its method from related CNN merging approaches that first align inputs with optical flow.
- Patch-based methods: Patch-based methods can produce excellent results but have longer running times and often fail with large motions or large saturated regions.These limitations motivate a learned alternative for difficult dynamic scenes.
- Single-image reconstruction: Single-LDR CNN reconstruction can hallucinate weakly represented details but often fails in highly contrastive scenes because it relies on one input under large-scale saturation.The paper therefore considers multiple burst-captured LDR images as a richer source of HDR content.
- Scope: HDR production commonly includes radiometric calibration, tone-mapping, and dynamic-range compression, while this work focuses on the merging process.The paper contrasts this focus with more expensive special capture devices.
3 Approach
The approach treats multi-exposure HDR reconstruction as image translation, using an end-to-end network to produce a ghost-free HDR image aligned with a reference input. It combines exposure-aware preprocessing with encoder–merger–decoder architectures and tonemapped-image training.
- 3 Approach: The method maps multiple LDR exposures to a ghost-free HDR image aligned with a chosen reference LDR.The experiments use three LDR images with the middle exposure as reference, while the same network can support more inputs and other reference images.
- 3.1 Network Architecture: The framework uses symmetric encoder–decoder translation networks in Unet and ResNet variants.ResNet replaces the middle layers with residual blocks, whereas Unet uses encoder–decoder skip connections.
- 3.1 Network Architecture: Separate early encoders extract information from each exposure before middle layers merge features and later layers decode the HDR output.This design avoids duplicating the whole network while preserving exposure-specific information before merging.
- 3.2 Processing Pipeline and Loss Function: Non-RAW inputs are first linearized with an estimated inverse camera response function and then gamma-corrected, although visually plausible results can also be produced without radiometric calibration.The paper attributes this robustness to gamma being a rough approximation of the camera response function.
- 3.2 Processing Pipeline and Loss Function: The pipeline maps sorted LDR inputs into HDR-domain representations using simple gamma encoding, then concatenates the original and transformed inputs into a 6-channel network input.The LDRs help detect misalignment and saturation, while exposure-adjusted HDR inputs improve robustness across exposure levels.
- 3.2 Processing Pipeline and Loss Function: Training computes the loss on tonemapped HDR outputs rather than directly in the HDR domain, using µ-law range compression with µ set to 5000.The paper motivates this choice because HDR images are usually displayed after tonemapping and other tonemappers may be difficult to differentiate.
4 Datasets
The study uses a ground-truth HDR dataset with dynamic and static exposure captures, supplemented by two additional datasets. Training emphasizes motion-rich patches, while evaluation includes runtime, quantitative, and visual comparisons.
- 4 Datasets: The primary dataset contains 89 scenes with ground-truth HDR images, including dynamic and static three-exposure captures.Exposure biases are {−2.0, 0.0, +2.0} or {−3.0, 0.0, +3.0}, with static sets used to construct ground truth.
- 4 Datasets: Table 1 compares average test-set running time under CPU conditions, while the figure-based comparisons use images from Kalantari’s test set.Figure 3 reports PSNR for tonemapped images and compares zoomed HDR regions against other methods.
- 4 Datasets: Table 2 averages PSNR, SSIM, and HDR-VDP-2 across 15 Kalantari test images, comparing tonemapped and linear outputs with ground truth.The table caption distinguishes the tonemapped and linear-image metrics and identifies HDR-VDP-2 as the final row.
- 4 Datasets: Backgrounds are first aligned with a simple homography so that learning concentrates on foreground motions without introducing artifacts or distortions.The paper compares this setup with training without background alignment in Section 5.4.
- 4 Datasets: The dataset split provides 74 training examples and 15 testing examples, with 256 × 256 crops, augmentation, and roughly 19000 patches.Flipping and rotation increase the training data by 8 times.
- 4.2 Data Augmentation and Patch Generation: Training focuses on foreground motion by detecting patches with low cross-exposure structural similarity and replicating those motion-rich patches.This counters the large proportion of patches containing mostly static background regions.
5 Experiments and Results
The experiments compare the proposed HDR framework with flow-based and other architectures, including runtime tests and visual examples. The implementation includes multi-stage encoding and decoding, with Unet and ResNet variants evaluated on different inputs.
- Implementation Details: The network uses separate encoders for radiance maps concatenated with corresponding LDR images, followed by feature merging and decoding.Encoding uses stride-2 convolutions, while decoding uses deconvolutions.
- Implementation Details: Figure 4 compares the proposed method with a flow-based method on images from the Kalantari and Tursun datasets.
- Implementation Details: Figure 5 contrasts hallucination from only a medium-exposure shot with hallucination from low-, medium-, and high-exposure shots.
- Implementation Details: The Unet processes 256 × 256 patches through eight encoding and eight decoding layers, while ResNet uses residual blocks between encoding and decoding stages.
- Runtime Comparison: 0.225s and 0.239s are the GPU runtimes of Unet and ResNet, respectively, for three 896 × 1408 LDR inputs.The optical-flow alignment in takes 59.4s on average, while the all-method CPU comparison used an i7-4790K system.
5.3 Evaluation and Comparison
The evaluation combines qualitative and quantitative comparisons with state-of-the-art HDR methods. On Kalantari’s test set, the proposed methods achieve comparable SSIM and slightly higher HDR-VDP-2 scores than the flow-based baseline.
- Evaluation Setup: The evaluation compares two patch-based methods, a motion-rejection method, a flow-based CNN merger, and a single-image HDR method.
- Qualitative Results: Figure 6 compares hallucinated details, including missing trunk texture, between the proposed network and methods without hallucination.
- Evaluation Setup: PSNR and SSIM are measured before and after µ-law tonemapping, while HDR-VDP-2 evaluates visual quality under specified display conditions.
- Quantitative Results: On Kalantari’s test set, achieves slightly higher PSNR, while the proposed methods obtain comparable SSIM and slightly higher HDR-VDP-2 scores.ResNet tends to achieve higher scores than Unet.
Qualitative Comparison
Qualitative comparisons show that large object motion, especially in saturated regions, exposes artifacts and distortions in existing methods. The proposed networks produce results without obvious artifacts or distortions and avoid the flow-related failures highlighted in Figure 4.
- Qualitative Comparison: In regions without object motion, all methods produce decent results, but large motion in saturated regions causes artifacts in several competing methods.
- Highlight Regions: Figure 7 compares highlight regions from the Sen dataset, where competing methods are assessed against the proposed results.
- Reference Images: Figure 8 shows HDR results produced with each of three LDR inputs serving as the reference image.
- Qualitative Comparison: Flow-based method produces geometric distortions, while the motion-rejection method produces ghosting when applied with three inputs.
- Qualitative Comparison: The proposed Unet and ResNet produce comparably good results without obvious artifacts or distortions, with ResNet generally outperforming Unet.
- Flow-Based Comparison: Figure 4 illustrates that erroneous optical-flow alignment introduces severe distortions and color artifacts, whereas the proposed method avoids them and is more computationally efficient.
Hallucination
The framework can hallucinate details missing from exposure stacks, including in saturated or occluded regions. Experiments also examine additional inputs, uncalibrated images, and highlight-region recovery against competing methods.
- Hallucination: The network can hallucinate details that conventional optimization approaches find nearly impossible to recover.
- Input Variation: Figure 9 evaluates HDR generation using different numbers of input LDR images.
- Radiometric Calibration: Figure 10 presents HDR results without radiometric calibration using novel cellphone images captured with different camera response functions.
- Hallucination: With only a medium exposure, the network hallucinates grass texture in saturated regions; with two additional exposures, it incorporates information such as ground texture.
- Hallucination: Hallucination is useful when total occlusion leaves over-exposed or under-exposed content missing from every LDR input.
- Highlight Regions: In highlight regions, competing methods often fail to recover details or introduce artifacts and distortions, while the proposed method generally works well.Hu’s method performs poorly generally, and Kalantari’s method introduces evident distortions and color artifacts.
Highlight
The image-translation framework supports flexible reference-image selection and can benefit from additional LDR inputs, improving details in saturated and under-exposed regions.
- Background Alignment: Parallax effects can produce blurry results that homography transformation cannot resolve.
- Reference Images: Different reference images can be selected by re-arranging the input LDRs, producing perceptually compelling pseudo-HDR outputs.These outputs may not represent accurate radiance values but resemble real HDR images perceptually.
- More Input LDRs: The framework may support more than three input LDRs, although the paper lacks a suitable dataset for fully exploring this extension.The authors use Sen’s dataset for a brief extensibility experiment, whose HDR ground truths are not perfect but sufficient for testing.
- More Input LDRs: Using more input LDRs markedly improves details in saturated and under-exposed regions compared with three inputs.
Cellphone Example
The model produces good results on novel cellphone images from different devices and remains robust without radiometric calibration, while background alignment remains important for avoiding blurry edges.
- Cellphone Robustness: Novel cellphone images from different cameras produce good results even without radiometric calibration.The tested phones had different camera response functions.
- Background Alignment: Without background alignment, the network tends to produce blurry edges where the background is substantially misaligned.The authors attribute this to confusion from background motion, which CNNs are generally weak at handling.
- Background Alignment: Homography may fail to align saturated regions under parallax effects, leaving the final HDR output blurry.
6 Conclusion and Future Work
The paper presents deep CNN image translation as a non-flow-based approach for HDR imaging with large foreground motions, reporting improved quality, efficiency, extensibility, and input robustness. It remains challenged by massive saturated regions when only a few LDR inputs are available.
- Conclusion: The method formulates HDR imaging as image translation and uses deep CNNs to avoid optical-flow alignment.
- Conclusion: Extensive experiments show that the non-flow-based approach outperforms state-of-the-art methods, especially with large foreground motions.
- Conclusion: The translation network avoids distortions and artifacts from erroneous optical flows, is computationally more efficient, and hallucinates plausible details in saturated regions.
- Conclusion: The system can be extended to more inputs and different reference images, and is robust to inputs without radiometric calibration.
- Future Work: Recovering massive saturated regions remains challenging when only a minimal number of input LDRs is available.The authors propose incorporating high-level knowledge and developing a more powerful solution in future work.