Source-linked AI summary
HDR-GAN: HDR Image Reconstruction from Multi-Exposed LDR Images with Large Motions
Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, Rynson W. H. Lau
TL;DR
Dynamic-scene HDR reconstruction must handle severe exposure-image misalignment and missing content caused by moving objects. HDR-GAN uses adversarial learning, reference-based feature merging, and deep HDR supervision to reconstruct such images, achieving state-of-the-art performance over prior HDR methods on diverse scenes.
Problem
Dynamic scenes make multi-exposed LDR fusion difficult because moving objects cause severe misalignment and missing content that multiple exposures may not compensate.
Method
HDR-GAN combines adversarial learning with a reference-based residual merging module for feature-domain motion alignment and deep HDR supervision for artifact reduction.
Results
HDR-GAN achieves state-of-the-art HDR reconstruction performance over prior HDR methods on diverse scenes.
Takeaways & Limitations
The model can fuse multi-exposed LDR images with large object motions without explicit alignment while producing faithful information in regions with missing content.
Takeaways & Limitations
Existing HDR datasets contain limited challenging scenes or lack ground truth, and reconstructing HDR images from very high-resolution inputs remains constrained by CNN memory resources.
Abstract
from arXiv · showhide
Synthesizing high dynamic range (HDR) images from multiple low-dynamic range (LDR) exposures in dynamic scenes is challenging. There are two major problems caused by the large motions of foreground objects. One is the severe misalignment among the LDR images. The other is the missing content due to the over-/under-saturated regions caused by the moving objects, which may not be easily compensated for by the multiple LDR exposures. Thus, it requires the HDR generation model to be able to properly fuse the LDR images and restore the missing details without introducing artifacts. To address these two problems, we propose in this paper a novel GAN-based model, HDR-GAN, for synthesizing HDR images from multi-exposed LDR images. To our best knowledge, this work is the first GAN-based approach for fusing multi-exposed LDR images for HDR reconstruction. By incorporating adversarial learning, our method is able to produce faithful information in the regions with missing content. In addition, we also propose a novel generator network, with a reference-based residual merging block for aligning large object motions in the feature domain, and a deep HDR supervision scheme for eliminating artifacts of the reconstructed HDR images. Experimental results demonstrate that our model achieves state-of-the-art reconstruction performance over the prior HDR methods on diverse scenes.
I. INTRODUCTION
HDR-GAN addresses the severe misalignment and missing-content problems that arise when fusing multi-exposed LDR images in dynamic scenes. It combines adversarial learning with feature-domain motion handling and deep HDR supervision, and outperforms prior HDR models on a public benchmark.
- Motivation: Large foreground motions create severe LDR misalignment and missing content in over-/under-exposed regions, producing blur and ghosting artifacts.Moving-object occlusions can prevent multiple exposures from compensating for missing details.
- Contribution: The generator uses a reference-based residual merging block to align large object and camera motions in the feature domain.The module transforms features from multiple exposures toward the median-exposure reference image without explicitly aligning the input images.
- Results: The model fuses multiple LDR images with large object motions into a ghost-free HDR image without explicit alignment, restoring details in dis-occluded regions.In the moving-car example, competing methods exhibit ghosting artifacts while HDR-GAN restores missing details.
- Contribution: HDR-GAN is presented as the first GAN-based method for reconstructing HDR images from multi-exposed LDR images.Adversarial learning is intended to produce faithful information where input LDR images contain missing content.
- Contribution: Deep HDR supervision generates high-resolution HDR outputs at multiple scales to reduce artifacts in the reconstructed HDR image.Merged features are progressively upsampled and passed through decoding blocks for supervision.
- Results: Experiments on a public benchmark show that HDR-GAN outperforms state-of-the-art HDR models.The reported comparison covers the benchmark's evaluation metrics.
II. RELATED WORK
HDR reconstruction research spans static-scene methods that assume aligned exposures and dynamic-scene methods that address motion, but large foreground motion still creates alignment and missing-content challenges.
- HDR Reconstruction for Static Scenes: Static-scene HDR methods merge aligned multi-exposure LDR images using hand-crafted or deep features.
- HDR Reconstruction for Static Scenes: Single-LDR HDR methods learn exposure expansion or synthesis, but generally underperform methods using multiple exposures.
- HDR Reconstruction for Dynamic Scenes: Dynamic-scene methods estimate homographies or optical flow before fusing LDR images to address camera and object motion.
- HDR Reconstruction for Dynamic Scenes: CNN-based dynamic HDR methods use image translation, attention, or non-local processing, yet large foreground motions cause severe misalignment and missing contents.
- HDR Reconstruction for Dynamic Scenes: HDR-GAN is proposed as a GAN-based model addressing large object motions without explicitly aligning the LDR images.
III. OUR PROPOSED METHOD
HDR-GAN converts and fuses three multi-exposed LDR inputs through a multi-scale generator with reference-aligned feature merging, deep HDR supervision, and adversarial training.
- Overall Framework: HDR-GAN uses three exposure-ordered LDR images, with the median exposure as reference, to construct an HDR image.
- Overall Framework: The generator receives each LDR image concatenated with its HDR-domain version and outputs an HDR image through encoding, merging, and decoding stages.
- Multi-scale LDR Encoder: Three consecutive downsampling encoders extract multi-scale features from each input using residual blocks and stride-2 convolutions.
- Reference-aligned Feature Fusion: Reference-based merging concatenates same-scale features and uses the median-exposure features in a residual path to implicitly align other exposures.
- Deep HDR Supervision: Deep HDR supervision upsamples intermediate merged features through an auxiliary decoding path to generate multiple full-resolution HDR outputs and reduce artifacts.
B. The Discriminator
The discriminator uses PatchGAN features projected onto a hypersphere, where moment distances to a north-pole reference define the adversarial comparison.
- Discriminator Architecture: PatchGAN with five convolutional layers serves as the discriminator for HDR-GAN.
- Hypersphere Projection: The discriminator reshapes its output into an n-dimensional vector and projects it onto a hypersphere using inverse stereographic projection.
- Adversarial Objective: Adversarial optimization compares generated and ground-truth mappings through hypersphere distances measured relative to the north pole.
- Adversarial Objective: The hypersphere distance uses first-, second-, and third-moment distances, while the hybrid loss balances its terms with λ set to 1.
A. Datasets and Metrics
The evaluation uses a ground-truth HDR benchmark for training and testing, plus ground-truth-free datasets, and reports quality in tonemapped and linear HDR domains.
- Datasets: Training exposures use bias combinations of {−2, 0, +2} or {−3, 0, +3}.
- Datasets: Sen’s and Tursun’s datasets are used for testing despite lacking ground-truth HDR images.
- Metrics: Evaluation reports PSNR and SSIM in both µ-law tonemapped and linear HDR domains, with µ = 5000 for tonemapping.
- Metrics: HDR-VDP-2 additionally assesses HDR-image visibility and quality across luminance conditions.
B. Implementation and Details
The model is evaluated on high-resolution training data with standard augmentation and optimization settings, alongside benchmark comparisons summarized in Table I.
- Training setup: Training uses 1500×1000 images, 512 × 512 patches, random cropping, rotation, and flipping.These settings augment the training set while preserving relatively large spatial inputs.
- Evaluation reporting: Table I compares HDR-GAN with state-of-the-art methods, with higher values indicating better performance.The table marks the best performance values in bold and identifies results sourced from prior works.
- Optimization: The learning rate starts at 10−4, then decays to 10−5 after 114,300 iterations and 10−6 after 706,400 iterations.Training runs for 1,215,000 iterations with batch size 2 using Adam.
C. Comparison with the State-of-the-Arts
HDR-GAN is compared with patch-based and deep-learning HDR methods on multiple datasets, using quantitative and qualitative evaluations of dynamic scenes.
- Quantitative comparison: The benchmark includes two patch-based methods and four deep-learning methods, with differing requirements for input alignment.Kalantari et al. use optical flow, DeepHDR requires homography-aligned backgrounds, while AHDRNet and HDR-GAN do not need that preprocessing.
- Quantitative comparison: HDR-GAN outperforms all comparison methods on every reported metric in Table I.Its results lead on PSNRL, PSNRµ, SSIMµ, SSIML, and HDR-VDP-2 according to the reported comparison.
- Qualitative comparison: Across qualitative examples, HDR-GAN recovers details such as hair and sticks while reducing blur, noise, color distortion, and artifacts.The examples cover dynamic objects, low-light conditions, global camera motion, and fast person motion.
D. Ablation Study
Ablations examine network scale, residual merging blocks, deep HDR supervision, output branches, and the discriminator to identify the components supporting reconstruction quality.
- Network scale: The proposed three-scale architecture performs better than the two-scale and four-scale variants.The two-scale network underperforms, while the deeper four-scale network has slightly lower performance and may overfit in its last branch.
- Merging blocks: Removing the residual merging blocks significantly lowers PSNR and prevents effective alignment and content recovery.Visual results show failed alignment of the three input images without these blocks.
- Deep HDR supervision: The best result uses both ˆH1 and ˆH2 for loss computation while using ˆH2 as the output.This agrees with the quantitative ablation results and indicates the value of deep HDR supervision.
OR ˆH2) AND THE GROUND-TRUTH IMAGE AS INPUTS. THE VALUES
The discriminator ablation evaluates whether adversarial training and multiple generated HDR outputs improve reconstruction performance.
- Discriminator: Adding discriminator D apparently boosts the performance of the proposed model.The generator without GAN remains comparable with state-of-the-art methods, while adversarial training further improves its result.
- Multiple discriminator inputs: Using both ˆH1 and ˆH2 as discriminator inputs slightly improves generation ability compared with using only one output.The discriminator must distinguish both generated outputs from the ground-truth image.
V. CONCLUSION AND FUTURE WORKS
HDR-GAN reconstructs HDR images from three multi-exposed LDR inputs while addressing large-motion misalignment and missing content. Experiments report state-of-the-art reconstruction performance, while the authors identify challenging-scene coverage and high-resolution processing as future needs.
- Conclusion: HDR-GAN generates HDR images from three multi-exposed LDR images and targets artifacts caused by large object motions.The model uses adversarial learning, reference-based residual merging blocks, and deep HDR supervision.
- Conclusion: The model achieves state-of-the-art reconstruction performance compared with prior HDR methods.
- Future Work: Existing HDR datasets are limited by relatively few challenging scenes, and several alternatives lack ground-truth HDR images.The authors call for a larger dataset containing more diverse and challenging scenes.
- Future Work: Reconstructing HDR images from very high-resolution photos remains difficult for CNN-based models because of limited memory resources.The authors propose investigating models that process high-resolution images efficiently and effectively.