Source-linked AI summary
Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang
TL;DR
Single-image HDR reconstruction must recover missing under- and over-exposed details from one quantized, saturated LDR image. The paper reverses the camera pipeline with specialized networks for dequantization, linearization, and hallucination, then refines the full model jointly. Across diverse datasets, it reports favorable performance against state-of-the-art methods and visually pleasing reconstructions with faithful details.
Problem
Single-image HDR reconstruction is challenging because under- and over-exposed regions lose information, while HDR pixels have much greater variation than 8-bit LDR pixels.
Method
The model reverses dynamic-range clipping, camera-response mapping, and quantization through specialized dequantization, linearization, and hallucination networks with physical constraints.
Results
The method performs favorably against state-of-the-art single-image HDR reconstruction methods across four datasets and real-world LDR images, with visually pleasing results and faithful details.
Takeaways & Limitations
Explicitly modeling the camera pipeline supports physical constraints during training and improves generalization to unseen scenes.
Takeaways & Limitations
The model focuses on selected camera-pipeline components and represents demosaicing, white balancing, gamut mapping, and noise reduction with a refinement network.
Abstract
from arXiv · showhide
Recovering a high dynamic range (HDR) image from a single low dynamic range (LDR) input image is challenging due to missing details in under-/over-exposed regions caused by quantization and saturation of camera sensors. In contrast to existing learning-based methods, our core idea is to incorporate the domain knowledge of the LDR image formation pipeline into our model. We model the HDRto-LDR image formation pipeline as the (1) dynamic range clipping, (2) non-linear mapping from a camera response function, and (3) quantization. We then propose to learn three specialized CNNs to reverse these steps. By decomposing the problem into specific sub-tasks, we impose effective physical constraints to facilitate the training of individual sub-networks. Finally, we jointly fine-tune the entire model end-to-end to reduce error accumulation. With extensive quantitative and qualitative experiments on diverse image datasets, we demonstrate that the proposed method performs favorably against state-of-the-art single-image HDR reconstruction algorithms.
1. Introduction
Single-image HDR reconstruction addresses the limitations of multi-exposure capture by recovering HDR content from one LDR image. The proposed method reverses the modeled camera pipeline with specialized networks and performs favorably against state-of-the-art methods.
- Motivation: Multi-exposure HDR capture works well for static scenes but can produce ghosting on dynamic scenes and is infeasible when only one LDR image exists.Existing LDR images on the Internet are one example where capturing exposure brackets is unavailable.
- Challenge: Single-image HDR reconstruction is difficult because under- and over-exposed regions lose information, while HDR pixels span far more variation than 8-bit LDR pixels.Recent direct LDR-to-HDR methods therefore face a substantially larger output space than the input representation.
- Prior limitations: Prior methods recover over-exposed regions or synthesize multiple exposure variants, but quantization and inaccurate camera response functions remain only implicitly addressed.These approaches do not explicitly reverse the image-formation operations that create the input LDR image.
- Proposed approach: The proposed model decomposes reconstruction into dequantization, linearization, and hallucination, corresponding to reversing clipping, camera-response mapping, and quantization.Three specialized deep networks tackle these sub-tasks instead of learning one generic direct mapping.
- Evaluation: Explicitly reversing the camera pipeline reduces the difficulty of training a single HDR reconstruction network and supports visually pleasing results with faithful details.The method was evaluated on four datasets and real-world LDR images using quantitative, qualitative, and user-study evaluations.
- Contributions: The method combines physical constraints, task-specific features and losses, and two HDR datasets, achieving favorable HDR-VDP-2 scores and visual quality against state-of-the-art methods.The datasets include synthetic-LDR and real-LDR images for training and evaluation.
2. Related Work
Related work spans multi-image fusion, direct single-image HDR reconstruction, dequantization, radiometric calibration, and image completion. This paper distinguishes itself by directly modeling and reversing selected camera-pipeline components while using a refinement network for the remaining processes.
- Multi-image HDR reconstruction: Multi-image HDR methods fuse bracketed exposures, using alignment and post-processing to reduce artifacts in dynamic scenes.The paper instead focuses on reconstructing HDR from a single LDR image.
- Single-image HDR reconstruction: Single-image HDR reconstruction avoids ghosting artifacts but is significantly more challenging than multi-exposure reconstruction.Deep CNN methods have learned direct LDR-to-HDR mappings, including approaches focused on over-exposed regions.
- Camera pipeline: Unlike prior methods, the proposed approach directly reconstructs HDR by modeling the inverse image-formation pipeline.Figure 2 compares the LDR pipeline, existing single-image methods, and the proposed method.
- Dequantization and decontouring: The method targets 32-bit floating-point HDR recovery from 8-bit LDR input, rather than restoring an 8-bit image from lower-bit-depth input.Quantization can create contouring or banding artifacts that become more visible after HDR tone mapping.
- Radiometric calibration: Single-image CRF recovery requires statistical priors that may not apply broadly to images in the wild.This motivates careful treatment of radiometric calibration within the reconstruction pipeline.
- Image completion: Hallucination-Net incorporates the constraint that missing over-exposed pixels have values equal to or greater than other image pixels.This distinguishes HDR completion from generic image completion.
- Camera pipeline: The paper models dynamic-range clipping, CRF mapping, and quantization, while leaving demosaicing, white balancing, gamut mapping, and noise reduction to a refinement network.The selected components are presented as especially important for HDR reconstruction.
3. Learning to Reverse the Camera Pipeline
The method reconstructs HDR images by explicitly reversing three modeled LDR formation steps—clipping, non-linear camera-response mapping, and quantization—using specialized networks and end-to-end fine-tuning.
- 3. Learning to Reverse the Camera Pipeline: The complete model explicitly reverses the image-formation pipeline rather than learning only a generic direct LDR-to-HDR mapping.This design is intended to reduce the difficulty of training a single reconstruction network and limit accumulated errors through joint fine-tuning.
- 3.1. LDR image formation: The modeled LDR pipeline applies dynamic range clipping, non-linear mapping through a camera response function, and quantization.The forward process is represented as L = Φ(H) = Q(F(C(H))).
- 3.2. Design methodology: The reconstruction task is decomposed into dequantization, linearization, and hallucination, each targeting the inverse of one pipeline step.Three CNNs receive corresponding supervisory signals and physical constraints, then are jointly fine-tuned end-to-end to reduce error accumulation.
- 3.2. Dequantization: The Dequantization-Net learns to reduce quantization artifacts by predicting a correction added to the input LDR image.Its output is trained against the ground-truth non-linear image using an ℓ2 loss.
- 3.3. Linearization: The Linearization-Net estimates an inverse CRF from the non-linear LDR image, edge maps, and spatial-aware histogram features, while enforcing monotonicity and endpoint constraints.The inverse CRF is represented with PCA coefficients, reconstructed by integration and normalization, and used to map the dequantized image into a linear LDR image.
- 3.4. Hallucination: The Hallucination-Net recovers missing over-exposed content as positive residuals blended through a soft over-exposure mask.The reconstructed HDR image is modeled as ˆH = ˆIlin + α · C^-1(ˆIlin), with α based on γ = 0.95; a ReLU enforces positive residuals.
4. Experimental Results
The method is evaluated on four datasets through quantitative, qualitative, user-study, and ablation experiments. It performs favorably against existing methods, while fine-tuning and stage-wise component training substantially improve reconstruction quality.
- Experimental setup: The evaluation uses HDR-VDP-2 on HDR-SYNTH, HDR-REAL, RAISE, and HDR-EYE, with additional PSNR, SSIM, and LPIPS metrics on tone-mapped images.Comparisons include HDRCNN, DrTMO, ExpandNet, Deep chain HDRI, and Deep recursive HDRI.
- Quantitative comparisons: The proposed method performs favorably on all four datasets, and Ours+ improves HDR-VDP-2 by 1.57 on HDR-REAL, 0.41 on RAISE, and 0.5 on HDR-EYE.Ours+ denotes fine-tuning on the HDR-REAL training set after training on HDR-SYNTH.
- Visual comparisons: On real images without ground-truth HDR references, the method restores details in both under- and over-exposed regions while competing methods show artifacts, blur, or low contrast.HDRCNN exhibits noise, banding, or over-saturated colors; DrTMO is blurry or washed out; ExpandNet produces artifacts.
- User study: Users preferred the proposed results in 70% of with-reference comparisons and 69% of no-reference comparisons across the HDR-REAL test set.The study used 70 HDR images, 30 comparisons per participant, and 200 unique participants.
- Ablation studies: The Dequantization-Net outperforms existing methods, while edge and histogram features plus a monotonically increasing constraint improve Linearization-Net reconstruction.The linearization analysis evaluates inverse-CRF reconstruction error and PSNR of the reconstructed linear image.
- Ablation studies: Positive residual learning improves Hallucination-Net performance by 1.19 HDR-VDP-2 scores, whereas end-to-end training from scratch drops performance by 4.43 and 3.48 scores on HDR-SYNTH and HDR-REAL.Resize convolution reduces checkerboard artifacts, and perceptual loss helps generate realistic details.
5. Conclusions
The method leverages domain knowledge of the LDR image formation pipeline to design modules that reverse the imaging process. Experiments and comparisons validate its effectiveness across challenging scenes.
- The method uses the LDR image formation pipeline to design network modules for reversing the imaging process.
- Explicit camera-pipeline modeling imposes physical constraints during network training.
- Experiments and comparisons validate the approach for restoring visually pleasing details across varied challenging scenes.