Source-linked AI summary
ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content
Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, Kurt Debattista
TL;DR
Most imaging content remains LDR, although HDR better represents real-world lighting and supports HDR viewing and processing. ExpandNet uses a supervised, multiscale CNN to expand LDR images to HDR end to end, without upsampling or heuristics. It mostly outperforms traditional expansion operators and handles heavily under- and over-exposed content with fewer artefacts in certain cases.
Problem
Most imaging content remains LDR even though HDR handles real-world lighting, creating a need to convert LDR content for HDR viewing and pipelines.
Method
ExpandNet is a supervised, fully automatic, end-to-end multiscale CNN that reconstructs HDR from LDR using local, dilation, and global features without upsampling.
Results
ExpandNet mostly outperforms traditional expansion operators and performs better than non-dedicated CNN architectures, with fewer artefacts in certain heavily under- and over-exposed cases.
Takeaways & Limitations
A dedicated CNN architecture can expand single-exposure LDR content to HDR without upsampling, while remaining competitive with tested alternatives.
Takeaways & Limitations
Existing expansion operators often struggle with under- and over-exposed LDR content, and further investigation is required to completely remove artefacts.
Abstract
from arXiv · showhide
High dynamic range (HDR) imaging provides the capability of handling real world lighting as opposed to the traditional low dynamic range (LDR) which struggles to accurately represent images with higher dynamic range. However, most imaging content is still available only in LDR. This paper presents a method for generating HDR content from LDR content based on deep Convolutional Neural Networks (CNNs) termed ExpandNet. ExpandNet accepts LDR images as input and generates images with an expanded range in an end-to-end fashion. The model attempts to reconstruct missing information that was lost from the original signal due to quantization, clipping, tone mapping or gamma correction. The added information is reconstructed from learned features, as the network is trained in a supervised fashion using a dataset of HDR images. The approach is fully automatic and data driven; it does not require any heuristics or human expertise. ExpandNet uses a multiscale architecture which avoids the use of upsampling layers to improve image quality. The method performs well compared to expansion/inverse tone mapping operators quantitatively on multiple metrics, even for badly exposed inputs.
1. Introduction
HDR imaging represents real-world lighting, but most existing content remains LDR and must be converted for HDR viewing and pipelines. ExpandNet provides a fully automatic, end-to-end CNN-based approach with a novel multiscale architecture.
- Motivation: Most current and legacy imaging content remains LDR despite HDR’s applications and improved handling of real-world lighting.LDR content therefore needs conversion for improved viewing and use in future HDR pipelines.
- Motivation: Existing LDR-to-HDR methods are primarily model-driven, parameterized, difficult for non-experts, and unsuitable for all content types.
- Contribution: ExpandNet uses a multiscale CNN architecture in which local, dilation, and global branches learn detail, larger-neighbourhood information, and global context.The architecture avoids upsampling to reduce blocking and haloing artefacts.
- Contribution: ExpandNet is a fully automatic, end-to-end, parameter-free method for expanding LDR content using a novel CNN architecture.The approach is based on learned abstract representations acquired directly from data rather than simplistic pixelwise processing.
- Contribution: The paper reports competitive results against other tested approaches, including CNN architectures, and adds data augmentation for limited HDR training content.It also provides a comprehensive quantitative comparison of LDR-to-HDR expansion methods.
2. Related Work
Prior LDR-to-HDR expansion methods include global, local, and learning-based approaches, but many rely on heuristics and struggle with poorly exposed content. ExpandNet addresses these limitations with a dedicated end-to-end architecture without upsampling.
- LDR-to-HDR methods: Expansion operators generate HDR content from LDR input, with global methods applying uniform functions and local methods using spatially varying expand maps.
- LDR-to-HDR methods: Earlier approaches include power-function and gamma-based expansion, inverse tone mapping, inpainting, and classification-based methods.
- Limitations: Most existing expansion operators derive added information from heuristics and have difficulty handling under- and over-exposed LDR content.
- Deep learning: Deep learning has been applied to image translation, super-resolution, upsampling, inpainting, HDR imaging, and tone mapping.
- Deep learning: Concurrent HDR-expansion methods use U-Net-like prediction of saturated regions or multiple exposures, whereas ExpandNet performs dedicated end-to-end expansion without upsampling.The paper links upsampling in related architectures to artefacts such as checkerboard or tiling effects.
3. ExpandNet
ExpandNet is a three-branch multiscale CNN that expands LDR images into predicted HDR images while avoiding upsampling layers. Its branches capture local, medium-range, and global information, which are fused before RGB HDR prediction.
- Architecture: ExpandNet uses local, dilation, and global CNN branches to capture high-frequency detail, medium-range features, and image-wide context.The global branch processes a resized 256×256 input, while the local and dilation branches process the LDR image without downsampling.
- Architecture: The local and dilation branches avoid downsampling and upsampling, while the global branch only downsamples to capture whole-image context.This design aims to reduce blocking, banding, checkerboard, haloing, and information-bleeding artefacts associated with upsampling.
- CNN foundations: ExpandNet uses convolutional layers that exploit spatial correlations while reducing learnable parameters and supporting efficient image-processing implementations.The network is presented as a CNN architecture composed of layered nonlinear transformations and convolutional operations.
- Branch contributions: Branch masking shows that the local branch preserves fine detail, the dilation branch contributes medium-range frequencies, and the global branch adds low frequencies and changes overall sharpness.Masking the global branch produces darker, sharper outputs, while masking the local branch removes most fine details.
- Feature fusion: The three branch outputs are concatenated, global features are replicated across pixels, and 1×1 convolutions fuse information across scales before final RGB prediction.A final convolutional layer produces the predicted HDR image in the [0,1] range.
- Activations: The network uses SELU activations in hidden layers and a Sigmoid output activation that maps predictions to the [0,1] range.SELU is described as supporting self-normalizing activations, while the output layer constrains the predicted image range.
4. Training and Implementation
ExpandNet is trained end-to-end on augmented LDR-HDR pairs generated from HDR images, using varied crops, dynamic-range operators, and relative-luminance targets. Evaluation materials cover scene- and display-referred scaling under optimal exposure and culling.
- Dataset: The dataset contains 1,013 training images and 50 test images, with only 100 training images containing calibrated luminance values.Images have linear RGB values and resolutions from 800×800 to 4,916 × 3,273.
- Data augmentation: Each training epoch generates a different LDR-HDR pair by randomly cropping an HDR image and applying a dynamic-range reduction operator.Crop positions favor the image center, crop sizes favor smaller regions, and the minimum crop size is 384 × 384.
- Preprocessing: The cropped HDR target is resized to 256×256 and mapped to [0,1], while the network predicts relative luminance because few images have absolute luminance values.Training inputs are produced with five operators: Photoreceptor, ALM, display, Bilateral, and Exposure.
- Evaluation setup: The reported experiments use single exposures for testing; tone-mapping operators are used only for training augmentation.Evaluation includes optimal exposure and culling conditions for scene-referred and display-referred outputs.
- Optimization: Training minimizes the Equation 7 loss with λ = 5 using mini-batch gradient descent, backpropagation, Adam, and learning-rate reduction after plateaus.The initial learning rate is 7e−5 with batch size 12; L2 regularization is also used.
5. Results
ExpandNet is evaluated against established expansion operators and CNN architectures on four metrics across exposure and luminance-scaling conditions. It ranks in the top statistical group in most comparisons and is especially effective for culling inputs, while augmentation and branch ablations explain performance differences.
- Quantitative evaluation: The evaluation uses PSNR, SSIM, MS-SSIM, and HDR-VDP-2.2 on 50 full-HD single-exposure LDR inputs created by optimal exposure or culling.Culling clips the top and bottom 10% of values, causing more information loss and distortion than optimal exposure.
- Comparisons: ExpandNet is compared with seven prior expansion methods and four CNN architectures, with CNN baselines trained on the same dataset and framework where applicable.The CNN comparison includes U-Net, a colorization architecture, and Eilertsen et al.’s LDR-to-HDR network.
- Quantitative results: 13 of 16 pairwise result groups place EXP in the top statistically indistinguishable group across optimal, culling, scene-referred, and display-referred conditions.The exceptions are optimal-condition pu-SSIM for both scaling types and scene-referred pu-MMSIM.
- Quantitative results: For culling, EXP appears to reproduce HDR better than other methods when substantial over- and under-exposure requires reconstruction.EIL also performs very well for optimal exposure, where fewer pixels require CNN prediction.
- Visual and perceptual results: HDR-VDP-2.2 maps show EXP performing better than most methods on the examined culling scenes, while EIL performs well in one challenging scene.The maps encode perceptual visibility probability, with blue indicating imperceptible and red perceptible differences.
- Visual and perceptual results: CNN visual comparisons show blocking, banding, and information bleeding in UNT and COL, with these artefacts present to a lesser extent in EIL.EXP reduces bleeding into large overexposed areas and recovers high-frequency contrast, although sharp-edge artefacts remain.
- Data augmentation: Training with all listed tone-mapping operators outperforms specialization on mismatched inputs, whereas EXP-Photo reaches PSNR 24.28 versus 21.52 when tested on Photoreceptor inputs.EXP-Photo averages PSNR 19.93 for display-referred culling, but specializes effectively to Photoreceptor-generated inputs.
- Further investigation: Adding camera response functions produces results within 1% of EXP, while branch analysis finds the global branch contributes most and the full architecture converges fastest with lowest loss.The local and dilation branches improve each other’s performance when combined.
6. Conclusions
ExpandNet expands single-exposure LDR content to HDR using a three-branch CNN architecture, mostly outperforming traditional expansion operators across multiple metrics. It also compares favorably with non-dedicated CNNs and handles heavily under- and over-exposed content with fewer artefacts in certain cases.
- 6. Conclusions: ExpandNet expands single-exposure LDR content to HDR using CNNs and a dedicated three-branch architecture.The branches address different aspects of expansion through local detail, larger neighbourhoods, and global context.
- 6. Conclusions: ExpandNet mostly outperforms traditional expansion operators across the evaluated metrics.
- 6. Conclusions: ExpandNet performs better than non-dedicated CNN architectures based on UNT and COL.
- 6. Conclusions: Compared with other dedicated CNN methods, ExpandNet does well in certain cases and exhibits fewer artefacts for heavily under- and over-exposed content.
- 6. Conclusions: Further investigation is required to completely remove artefacts, including examining network receptive fields; dynamic content also requires temporal-coherence solutions.The paper identifies careful design and possible recurrent approaches as directions for dynamic methods.
Revisions
The revision corrected results in Figure 5 and LAN culling in Table 2, with minor changes that do not alter the reported outcomes or conclusions.
- Revisions: Corrections to Figure 5 and LAN culling in Table 2 were minor and did not alter the outcomes or conclusions.