PaperlayerView exact revision on arXiv

Source-linked AI summary

HDR image reconstruction from a single exposure using deep CNNs

Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K. Mantiuk, Jonas Unger

arXiv:1710.07480v1cs.CVcs.GRcs.LG

TL;DR

The paper addresses HDR reconstruction from a single exposure by predicting information lost in saturated image regions. It uses a specialized deep CNN trained with simulated saturation and HDR data, achieving visually convincing reconstructions across varied cameras and scenes, while remaining limited by severe saturation and compression artifacts.

  • Problem

    Single-exposure LDR images lose information in saturated regions, creating a challenge for reconstructing HDR images without combining multiple exposures.

  • Method

    A hybrid dynamic range autoencoder uses an LDR encoder, a log-domain HDR decoder, domain-transformation skip-connections, and HDR-specific training data.

  • Results

    The method produces high-resolution, visually convincing HDR reconstructions, generalizes to arbitrary and low-end cameras, and performs well in image-based lighting and subjective HDR-display evaluation.

  • Takeaways & Limitations

    The results support automatic HDR reconstruction from a single exposed LDR image across a wide range of situations.

  • Takeaways & Limitations

    Reconstruction quality is constrained by content-dependent amounts of saturation and by blocking compression artifacts around highlights.

Abstract

from arXiv · show

Camera sensors can only capture a limited range of luminance simultaneously, and in order to create high dynamic range (HDR) images a set of different exposures are typically combined. In this paper we address the problem of predicting information that have been lost in saturated image areas, in order to enable HDR reconstruction from a single exposure. We show that this problem is well-suited for deep learning algorithms, and propose a deep convolutional neural network (CNN) that is specifically designed taking into account the challenges in predicting HDR values. To train the CNN we gather a large dataset of HDR images, which we augment by simulating sensor saturation for a range of cameras. To further boost robustness, we pre-train the CNN on a simulated HDR dataset created from a subset of the MIT Places database. We demonstrate that our approach can reconstruct high-resolution visually convincing HDR results in a wide range of situations, and that it generalizes well to reconstruction of images captured with arbitrary and low-end cameras that use unknown camera response functions and post-processing. Furthermore, we compare to existing methods for HDR expansion, and show high quality results also for image based lighting. Finally, we evaluate the results in a subjective experiment performed on an HDR display. This shows that the reconstructed HDR images are visually convincing, with large improvements as compared to existing methods.

1 INTRODUCTION

The paper reconstructs HDR images from single-exposure LDR inputs by estimating saturated highlights with a specialized CNN. Its high-quality results are supported by experiments and a subjective HDR-display evaluation.

  • The method estimates missing highlight information in saturated regions of a single-exposure LDR image to reconstruct HDR output.
  • The hybrid dynamic range autoencoder encodes spatial context from the LDR input and decodes an HDR image in the log domain.Skip-connections transfer information between the LDR encoder and HDR decoder.
  • The approach targets limitations of existing inverse tone-mapping operators, which often fail to reconstruct saturated pixels.Prior operators commonly focus on plausible HDR-display appearance or rough image-based-lighting estimates.
  • The system reconstructs high-quality HDR images from arbitrary single-exposed LDR images when saturated areas are reasonably small.The reconstructed images are evaluated through examples and a subjective experiment.
  • The trained CNN and parameters are made available online for prediction from LDR images.

2 RELATED WORK

Related work addresses HDR capture and inverse tone-mapping through exposure multiplexing, pixel transformations, saturation reconstruction, and deep learning. The paper positions its automatic CNN-based approach against these alternatives.

  • Exposure multiplexing captures a scene’s luminance range, but dynamic scenes require robust alignment and specialized alternatives introduce calibration, size, or resolution limitations.
  • Global inverse tone-mapping transforms all pixels without reconstructing information lost through saturation.These methods may use linear scaling or nonlinear functions for display-oriented HDR expansion.
  • Saturation-reconstructing inverse tone-mapping operators modify highlight regions to produce results resembling reference HDR images.
  • The proposed approach is reported to achieve significantly better results than existing inverse tone-mapping operators while running automatically within a second on modern hardware.
  • Related CNN applications include image restoration, HDR video reconstruction from multiple exposures, and single-exposure HDR panorama reconstruction for image-based lighting.
  • A concurrent deep-learning method predicts shorter- and longer-exposure LDR images before combining them with standard HDR reconstruction methods.

3 HDR RECONSTRUCTION MODEL

The model reconstructs saturated HDR highlights from arbitrary LDR images using a fully convolutional hybrid autoencoder, domain-aware skip-connections, and a perceptually motivated illuminance–reflectance loss. It blends predicted highlights with linearized input pixels while preserving non-saturated regions, and training results show benefits from skip-connections, the I/R loss, and pre-training.

  • 3.1 Problem formulation and constraints: The CNN predicts saturated pixels and blends its log-domain outputs with linearized LDR values to reconstruct HDR images.A linear ramp beginning at threshold τ preserves non-saturated input regions and reduces banding at transitions.
  • 3.2 Hybrid dynamic range autoencoder: The fully convolutional hybrid autoencoder encodes LDR images into a low-resolution feature representation and decodes HDR images in the log domain.Because the latent representation remains a multi-channel image rather than a fully connected layer, the network supports variable image resolutions.
  • 3.3 Domain transformation and skip-connections: Skip-connections transfer high- and low-level encoder features to the decoder, helping recover high-frequency details around saturated regions.Their domain transformation maps LDR display values to logarithmic HDR, while optimized log-domain feature combinations correspond to multiplicative operations in linear HDR space.
  • 3.4 HDR loss function: The illuminance–reflectance loss separates brightness and reflectance components, with λ controlling their relative importance.Greater illuminance weight favors high-intensity prediction, whereas greater reflectance weight favors local colors and details.
  • 3.4 HDR loss function: The I/R loss generally produces fewer artifacts in large saturated areas than the direct pixel loss.The authors suggest neighborhood influence from the Gaussian low-pass filter may provide a regularizing effect.

4 HDR IMAGE DATASET

The authors assemble a large HDR dataset and augment it with virtual-camera captures, producing training samples that support generalization across cameras. Pre-training comparisons show improved reconstruction of smaller highlights.

  • 1121 HDR images and 67 HDR video sequences were gathered for the reconstruction dataset.
  • Randomized virtual-camera crops, calibration, flipping, and resampling produced paired LDR-HDR training examples.Crops were resized to 320×320 pixels.
  • ∼125K training samples were captured from the HDR dataset, helping the model generalize across images from different cameras.
  • Pre-training made reconstructions more consistent and improved recovery of smaller highlights.
  • The LDR data contain peaks near the maximum pixel value, reflecting information lost through saturation.

5 TRAINING

Training combines pretrained initialization, simulated HDR data, and optimization on an HDR loss. Synthetic pre-training improves highlight recovery and reduces artifacts in saturated regions.

  • The encoder uses VGG16 weights from Places, while decoder deconvolutions and skip fusions are initialized for bilinear upsampling and feature addition.
  • ADAM optimization used a learning rate of 5×10^-5 for 800K steps with mini-batches of 8.Training took approximately 6 days on an Nvidia Titan X GPU.
  • A Places subset excluding images with substantial saturation provided approximately 600K images for simulated HDR pre-training.The subset retained images with fewer than 50 maximum-valued pixels.
  • Exposure-corrected, transformed, quantized, and clipped inputs with 5% saturated pixels yielded visually convincing reconstructions corresponding well to ground-truth HDR images.
  • The simulated HDR dataset was created by linearizing selected Places images with the inverse camera curve and increasing exposure.
  • Pre-training significantly improved performance: small highlights were better recovered, large saturated regions had fewer artifacts, and Table 1 reported error reduced by more than 10%.The synthetic dataset used 224×224-pixel images without resampling.

6 RESULTS

The CNN reconstructs saturated highlights convincingly across synthetic and real camera inputs, outperforming inverse tone-mapping methods while retaining limitations for extreme intensities and compression artifacts.

  • Training strategies: 10.7% lower error is achieved with pre-training and the I/R loss compared with no pre-training.Skip-connections additionally reduce error by 24%, while the I/R loss reduces error by 5.8% compared with the alternative loss.
  • Comparisons to ground truth: Synthetic test images show convincing recovery of lights, illuminated facades, specular reflections, colors, and details in many saturated regions.Spatial context can infer colors even when all channels are saturated, while residual information in one channel supports larger-area reconstruction.
  • Reconstruction with real-world cameras: Real camera JPEGs, including Canon 5DS R and degraded iPhone 6S images, also produce reconstructions of comparable quality.iPhone inputs can reconstruct skin tones, fire, and caustics, but JPEG blocking artifacts near highlights impair performance.
  • Varying information loss: Shorter-exposure inputs preserve more highlight information and therefore provide more detail in the reconstructed images.Predicted illuminance remains approximately at the same level across exposure settings.
  • Comparison to iTMOs: The CNN predicts saturated-region colors and structural content more convincingly than existing inverse tone-mapping methods.The comparison indicates that local highlight scaling can boost apparent dynamic range without recovering local information.
  • Image-based lighting: Reconstructed highlights enable visually convincing image-based lighting that is much closer to ground truth than LDR input or brightness-boosted iTMO results.Details remain unrecovered in large saturated window regions, but estimated luminance supports the rendering.

7 EVALUATION

A subjective pairwise experiment on a calibrated HDR display compared LDR, ground-truth HDR, an established iTMO, and the CNN reconstruction for perceived naturalness.

  • Experimental design: 15 participants judged image pairs for naturalness in a two-alternative forced-choice experiment on a calibrated HDR display.The display provided a 50,000:1 contrast range, and luminance was normalized across methods.
  • Experimental design: The experiment compared clamped LDR images, ground-truth HDR images, Banterle et al.’s iTMO, and the CNN output.The CNN output used the pre-training and I/R-loss configuration.
  • Results: The CNN reconstruction surpassed LDR images and was comparable to original HDR images in most cases on the JOD scale.Figure 16 reports values relative to LDR images with 95% confidence intervals.
  • Results: The iTMO was judged least natural, even below LDR, while CNN images were very likely better than their original LDR counterparts.Inaccurate colors and reduced contrast were identified as explanations for the iTMO result.

8 CONCLUSION AND FUTURE WORK

The paper concludes that a hybrid dynamic range autoencoder can reconstruct HDR from a single exposed LDR image, while performance is bounded by missing information, compression, and training-data limitations.

  • Conclusion: The hybrid dynamic range autoencoder is designed around HDR characteristics in its architecture, training data, and optimization procedure.The paper demonstrates reconstruction quality and versatility through examples and subjective evaluation.
  • Limitations: A content-dependent limit makes it difficult to quantify how much missing information the network can handle.Fully saturated regions may lack inferable structures, although illuminance can still support image-based lighting.
  • Limitations: Blocking compression artifacts around highlights impair reconstruction performance to some extent.The paper identifies compression-degraded inputs as a practical limitation for real-world reconstruction.
  • Future work: Recovering dark regions lost through quantization and noise is a complementary problem outside the saturated-pixel reconstruction considered here.The paper distinguishes this problem from saturated-pixel recovery because noise becomes important when exposure is increased.
  • Future work: Augmenting training data with compression artifacts is proposed as future work, but may reduce quality on images without compression.The paper also identifies robust high-resolution estimation with GANs as a future challenge.

APPENDIX A: DATA AUGMENTATION

The appendix specifies the virtual camera used to augment HDR images for training.

  • The virtual camera is used to augment HDR images for the training data described in Section 4.

A.1 Random cropping

The HDR training data is converted into many randomly positioned and resized sub-images. This produces approximately 125K training images while covering varied input scales.

  • Ten sub-images are randomly selected from each megapixel of HDR data.The crops use random positions and sizes.
  • The crop sizes span 20%–60% of the original image dimensions.Each crop is bilinearly resampled to 320x320 pixels.
  • The procedure yields a final training set of approximately 125K images.

A.2 Exposure

Training exposures are simulated by scaling HDR crops so clipping removes a randomly selected fraction of image information. The clipped fraction is controlled through a percentile-based threshold.

  • The clipped fraction v is uniformly sampled from 0.05 to 0.15.Thus, simulated saturation removes 5%–15% of image information.
  • An exposure scaling s is defined from v to control the amount of clipping.The scaling is applied before clipping the image.
  • The threshold Hth is the 1−v percentile of the HDR-image histogram.This percentile determines the scaling that removes the selected information fraction when clipping occurs.

A.3 Camera curve

Camera-response variation is simulated with a parametric sigmoid curve and randomized parameters. Additional color, noise, and flip augmentations broaden the training examples.

  • A parametric sigmoid function is used to approximate different camera response curves.
  • The fitted camera-curve parameters are n = 0.9 and σ = 0.6, with randomized training values sampled around them.The sampling distributions are n ∼N(0.9, 0.1) and σ ∼N(0.6, 0.1).
  • The curve is normalized so that f(1) = 1.The scaling 1 + σ provides this normalization.
  • Hue and saturation are randomly perturbed, while Gaussian noise and horizontal flips further augment the images.Noise has a randomly selected standard deviation in the range σ ∈[0, 0.01], and horizontal flipping has probability 0.5.
Loading 1710.07480v1…