Source-linked AI summary
Single Image HDR Reconstruction Using a CNN with Masked Features and Perceptual Loss
Marcel Santana Santos, Tsang Ing Ren, Nima Khademi Kalantari
TL;DR
Single-image HDR reconstruction must recover information missing from saturated LDR regions, where existing methods can produce artifacts or blurry textures. The paper combines feature masking, HDR-specific perceptual loss, staged inpainting pretraining, and challenging-patch sampling. It reports visually pleasing HDR results that outperform state-of-the-art methods across simulated and real-world scenes.
Problem
Existing single-image HDR methods struggle to recover plausible saturated-region textures and can produce blurriness, checkerboard artifacts, and halo artifacts.
Method
The approach uses feature masking, an adapted VGG-based perceptual loss, inpainting pretraining followed by HDR fine-tuning, and challenging-patch sampling.
Results
The approach produces better results than state-of-the-art methods on simulated images and images captured with real-world cameras.
Takeaways & Limitations
The system can reconstruct high-luminance regions and hallucinate textures in saturated areas while reducing halo and checkerboard artifacts.
Takeaways & Limitations
The method does not always reconstruct all details and can introduce nonexistent patterns or incorrect colors when the input lacks sufficient texture information.
Abstract
from arXiv · showhide
Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to hallucinate plausible textures, producing results with artifacts in the saturated areas. In this paper, we present a novel learning-based approach to reconstruct an HDR image by recovering the saturated pixels of an input LDR image in a visually pleasing way. Previous deep learning-based methods apply the same convolutional filters on well-exposed and saturated pixels, creating ambiguity during training and leading to checkerboard and halo artifacts. To overcome this problem, we propose a feature masking mechanism that reduces the contribution of the features from the saturated areas. Moreover, we adapt the VGG-based perceptual loss function to our application to be able to synthesize visually pleasing textures. Since the number of HDR images for training is limited, we propose to train our system in two stages. Specifically, we first train our system on a large number of images for image inpainting task and then fine-tune it on HDR reconstruction. Since most of the HDR examples contain smooth regions that are simple to reconstruct, we propose a sampling strategy to select challenging training patches during the HDR fine-tuning stage. We demonstrate through experimental results that our approach can reconstruct visually pleasing HDR results, better than the current state of the art on a wide range of scenes.
1 INTRODUCTION
The paper reconstructs missing HDR information in saturated LDR regions using feature masking, perceptual loss, staged training, and challenging-patch sampling. The authors report visually pleasing results that outperform state-of-the-art methods across simulated and real-world scenes.
- Single-image HDR reconstruction addresses luminance information lost when cameras capture scenes exceeding their limited dynamic range.
- Existing CNN methods use contextual information but often produce blurriness, checkerboard artifacts, and halo artifacts in saturated areas.
- Feature masking reduces saturated features' contribution, addressing training ambiguity from applying identical filters to well-exposed and saturated pixels.
- The adapted VGG-based perceptual loss is designed to reconstruct sharp textures in saturated regions more effectively than pixel-wise losses.
- The network is pretrained for image inpainting, fine-tuned for HDR generation, and trained with patches selected to emphasize challenging textured HDR areas.
- The approach reconstructs high-luminance regions and produces better results than state-of-the-art methods on simulated and real-camera images.
2 RELATED WORK
Prior single-image HDR methods use global or local heuristics, user interactions, or deep CNNs to expand luminance and recover saturated details. The paper positions its learning-based approach as systematic across scenes rather than dependent on heuristics or manual inputs.
- Single-image HDR reconstruction, also called inverse tone-mapping, recovers details from regions with missing content.
- Global inverse tone-mapping methods expand luminance using functions such as linear, exponential, gamma, or mid-level mappings.
- Local heuristic methods identify high-luminance areas and apply expand-maps, filtering, or edge-stopping functions to enhance saturated regions.
- User-interaction methods can transfer textures from selected regions but may require several minutes of expert interaction.
- The proposed learning-based approach aims to reconstruct HDR images across varied scenes without heuristic strategies or user inputs.
- Deep CNN methods generate bracketed exposures or directly reconstruct HDR images from a single LDR input.
- Existing learning methods often retain halo and checkerboard artifacts and lack saturated-region textures despite using adversarial losses in some cases.
3 APPROACH
The approach reconstructs saturated HDR regions with a CNN that combines exposed input content with predicted highlights, while masking invalid features and using perceptual supervision to synthesize textures. It also addresses limited HDR data through inpainting pre-training and challenging-patch selection.
- 3 APPROACH: The CNN combines well-exposed LDR content with network predictions in saturated regions to form the reconstructed HDR image.A soft exposure mask identifies how well-exposed each pixel is; γ = 2.0 maps the input to the linear domain.
- 3.1 Feature Masking: Feature masking multiplies each layer’s feature maps by soft masks, reducing contributions from invalid saturated content while retaining weak signals.Mask values indicate valid-input contribution, and masks are propagated through the network using normalized convolutional filters.
- 3.1 Feature Masking: The masks become blurrier and more uniform in deeper layers because deeper features have larger receptive fields.The mechanism applies the corresponding mask before convolution at each layer.
- 3.2 Loss Function: The loss combines HDR reconstruction with VGG and style terms to encourage perceptually similar textures instead of only pixel-wise agreement.The reconstruction loss is computed in saturated regions in the logarithmic domain, while the style term compares global Gram-matrix statistics.
- 3.2 Loss Function: The VGG perceptual term compares reconstructed and ground-truth features, while range compression makes HDR inputs suitable for the VGG network.The implementation uses λ3 = 1.0, λ4 = 120.0, and μ = 500.
- 3.3 Inpainting Pre-training: The training strategy uses inpainting pre-training and selects HDR fine-tuning patches containing complex saturated-region textures.Texture patches are detected from detail-layer Sobel gradients, with a mean-gradient threshold of 0.85 in the implementation.
4 IMPLEMENTATION
The implementation uses a U-Net with feature masking throughout convolutional layers, separate datasets for inpainting and HDR fine-tuning, and patch selection focused on saturated content.
- Architecture: The network uses a U-Net architecture with feature masking in all convolutional layers and nearest-neighbor upsampling in the decoder.
- Architecture: Encoder layers use Leaky ReLU, decoder layers use ReLU except the final linear layer, and skip connections link corresponding encoder and decoder layers.
- Dataset: The inpainting stage uses MIT Places, containing approximately 2.5M diverse scene images, while HDR fine-tuning uses simulated LDR images and corresponding HDR targets.
- Dataset: The final training dataset contains 100K input and ground-truth patches after discarding patches without saturated content.
- Training: The network is trained to convergence first on inpainting and then on HDR reconstruction using Adam with a learning rate of 2 × 10^-4.
5 RESULTS
The method is evaluated quantitatively and visually against prior single-image HDR approaches on synthetic and real images. It consistently produces sharper textures and fewer artifacts, including as saturation increases, while ablations support feature masking, inpainting pre-training, perceptual loss, and textured-patch sampling.
- Quantitative comparison: On a test set of 75 HDR images, the method achieves significantly better MSE and HDR-VDP-2 results than existing learning-based approaches.The test images range from 1024 × 768 to 2084 × 2844 resolutions.
- Synthetic scenes: The method synthesizes saturated-region textures better than prior methods while maintaining similar or better quality in bright highlights across diverse synthetic scenes.
- Saturation robustness: As the percentage of saturated pixels increases, competing methods degrade quickly, whereas the proposed method retains sharp details and bright highlights.
- Real images: On real images from smartphones and a Canon 5D Mark IV, the method increases dynamic range by synthesizing realistic textures in saturated regions.
- Ablation studies: Inpainting pre-training outperforms HDR pre-training numerically and visually, producing sharper textures in saturated areas.
- Ablation studies: Feature masking produces fewer checkerboard, halo, and blur artifacts than standard convolution, gated convolution, and input masking.
- Ablation studies: Textured-patch sampling yields more detailed textures and fewer objectionable artifacts than training without patch sampling.
- Ablation studies: The proposed perceptual loss produces visually realistic saturated-region textures, whereas pixel-wise l1 loss tends to produce blurry images.
6 LIMITATIONS AND FUTURE WORK
The method has several reconstruction failure modes, including missed details, invented textures, and incorrect colors, while perceptual loss improves texture realism. Video reconstruction is additionally limited by temporal instability because frames are synthesized independently.
- Limitations: The method fails to reconstruct some details, such as curtain wrinkles, although it can outperform competing approaches in the example.The comparison is qualitative and concerns a failure case shown in Figure 13.
- Limitations: Perceptual loss produces visually realistic textures, whereas pixel-wise loss produces blurry results.This comparison is illustrated in Figure 12.
- Limitations: When input information is insufficient, the method may introduce textures absent from the ground truth or reconstruct saturated regions with incorrect colors.The reported color error is attributed to training-data associations between trees and skies.
- Future work: HDR video reconstruction is not temporally stable because the network synthesizes each frame independently.The paper suggests temporal regularization as future work.
7 CONCLUSION
The paper presents a CNN-based system for single-image HDR reconstruction that combines feature masking, perceptual loss, staged training, and challenging-patch sampling. It reports realistic high-frequency detail synthesis and better performance than prior methods on synthetic and real-world images, while failure cases remain possible.
- Conclusion: The system uses feature masking with automatic mask updating to reduce halo and checkerboard artifacts caused by saturated pixels.The masking mechanism reduces the contribution of features computed on saturated areas.
- Conclusion: The approach can fail to reconstruct curtain wrinkles, introduce nonexistent textures, or assign incorrect colors in saturated regions.These failure cases are documented in Figure 13.
- Conclusion: A task-specific perceptual loss enables the network to synthesize visually realistic textures in saturated areas.The loss is designed for HDR reconstruction rather than relying only on pixel-wise supervision.
- Conclusion: Two-stage training pre-trains the network on inpainting, then fine-tunes it for HDR generation using challenging HDR patches.The sampling strategy encourages texture synthesis during HDR fine-tuning.
- Conclusion: The model reconstructs high-frequency details realistically and outperforms previous methods on synthetic and real-world images.The paper reports both quantitative and qualitative improvements across these image types.