PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Tone Mapping Operator for High Dynamic Range Images

Aakanksha Rana, Praveer Singh, Giuseppe Valenzise, Frederic Dufaux, Nikos Komodakis, Aljosa Smolic

arXiv:1908.04197v1eess.IVcs.CVcs.GR

TL;DR

Existing tone-mapping operators often need scene-by-scene tuning, limiting their use across varied HDR content. DeepTMO uses a conditional GAN trained with TMQI-selected targets and a multi-scale design; the authors report that viewers significantly preferred its images over the best TMQI-rated outputs for most tested scenes.

  • Problem

    Existing TMOs require extensive scene-specific parameter tuning and successfully handle only a limited number of HDR images.

    Source: A. Tone Mapping Operators for HDR… · p. 3

  • Method

    DeepTMO maps HDR to LDR with a conditional GAN and multi-scale generator-discriminator, trained using targets selected by TMQI.

    Source: §I · pp. 1–2, A. Target Tone Mapped Images · p. 8

  • Results

    Viewers significantly preferred DeepTMO over the best TMQI-rated tone-mapped images for most tested scenes.

    Source: 3) Results · p. 12

  • Takeaways & Limitations

    DeepTMO produces tone-mapped outputs across varied scene categories, and the subjective study reports preference for its contrast-preserving images.

    Source: §I · p. 1, 3) Results · p. 12

    The training targets are noisy tone-mapped ground-truth images; the authors suggest weakly supervised or unsupervised learning as future alternatives.

    Source: A. Limitations and Future Work · p. 13

Abstract

from arXiv · show

A computationally fast tone mapping operator (TMO) that can quickly adapt to a wide spectrum of high dynamic range (HDR) content is quintessential for visualization on varied low dynamic range (LDR) output devices such as movie screens or standard displays. Existing TMOs can successfully tone-map only a limited number of HDR content and require an extensive parameter tuning to yield the best subjective-quality tone-mapped output. In this paper, we address this problem by proposing a fast, parameter-free and scene-adaptable deep tone mapping operator (DeepTMO) that yields a high-resolution and high-subjective quality tone mapped output. Based on conditional generative adversarial network (cGAN), DeepTMO not only learns to adapt to vast scenic-content (e.g., outdoor, indoor, human, structures, etc.) but also tackles the HDR related scene-specific challenges such as contrast and brightness, while preserving the fine-grained details. We explore 4 possible combinations of Generator-Discriminator architectural designs to specifically address some prominent issues in HDR related deep-learning frameworks like blurring, tiling patterns and saturation artifacts. By exploring different influences of scales, loss-functions and normalization layers under a cGAN setting, we conclude with adopting a multi-scale model for our task. To further leverage on the large-scale availability of unlabeled HDR data, we train our network by generating targets using an objective HDR quality metric, namely Tone Mapping Image Quality Index (TMQI). We demonstrate results both quantitatively and qualitatively, and showcase that our DeepTMO generates high-resolution, high-quality output images over a large spectrum of real-world scenes. Finally, we evaluate the perceived quality of our results by conducting a pair-wise subjective study which confirms the versatility of our method.

I. INTRODUCTION

Existing tone mapping operators often require extensive per-scene parameter tuning, making adaptation across varied HDR content difficult. DeepTMO uses a conditional GAN and multi-scale architecture to produce high-resolution tone-mapped images for diverse scenes.

I. INTRODUCTION

  • Existing TMOs may handle only a limited range of HDR images and require extensive manual parameter tuning for each new scene.The paper describes this process as slow, tedious, expensive, and impractical for large varieties of HDR content.Source: §I · p. 1
  • The cGAN conditions generation on HDR input and trains a generator to produce tone-mapped images against a discriminator evaluating real and generated HDR-LDR pairs.The generator uses an encoder-decoder architecture, while the discriminator classifies image patches and averages their scores.Source: §I · p. 2
  • Training targets come from ranking 13 tone-mapping operators with TMQI for each HDR input and selecting the top-scoring output.This objective metric supplies targets in place of an extensive subjective evaluation for every scene and operator.Source: §I · p. 2
  • DeepTMO is a fast, parameter-free, end-to-end operator designed to map varied linear HDR scenes to high-resolution LDR outputs.The proposed model targets scenes including indoor and outdoor environments, people, structures, and day or night conditions.Source: §I · pp. 1–2
  • The authors compare four generator-discriminator scale combinations and examine loss functions and normalization layers to address artifacts such as blurring, tiling, and saturation.They report choosing a multi-scale generator and discriminator and compare the model with best tone-mapped outputs over 105 images, alongside a pair-wise subjective study.Source: §I · p. 2

II. RELATED WORK

Prior tone-mapping and learning-based approaches face trade-offs in contrast, artifacts, scene-specific tuning, or high-resolution output. The related work motivates conditional GANs and multi-scale designs for HDR-to-LDR mapping.

A. Tone Mapping Operators for HDR Content

  • Global tone-mapping methods can wash out contrast, while local methods preserve contrast and details but may create halos near high-frequency edges.The paper also describes human-visual-system-inspired operators as addressing different objectives.Source: A. Tone Mapping Operators for HDR… · p. 2

B. CNNs for HDR Scenes

  • Scene-specific parameter tuning makes optimization-based tone mapping time-consuming and limits its real-time applicability across varied scenes.The paper also notes that a deep bilateral tone mapper handles 16-bit linear images, while another end-to-end method is calibrated one scene at a time.Source: A. Tone Mapping Operators for HDR… · p. 3, B. CNNs for HDR Scenes · p. 3
  • For CNN tone mapping, formulating a loss for sharp, high-quality LDR outputs from linear HDR inputs is complex and ill-posed.The authors say their GAN-based approach avoids this particular loss-design issue.Source: B. CNNs for HDR Scenes · p. 3

C. Generative Adversarial Networks

  • Conditional GANs provide a structured loss suited to preserving HDR scene structure, but can produce blurry or noisy high-resolution outputs.The related work cites multi-scale generator-discriminator designs as improving high-resolution generation and contrasts its own 1024 × 2048 experiments with prior HDR-to-LDR tests on 256x256 crops.Source: C. Generative Adversarial Networks · p. 3

III. ALGORITHM

DeepTMO aims to produce high-resolution, high-quality tone-mapped images across varied HDR scenes without external parameter tuning. The section motivates adversarial learning and multi-scale architecture to address limitations of simple losses and artifact-prone outputs.

A. Problem Formulation

  • DeepTMO targets high-resolution, high-quality tone mapping across varied real-world HDR images without external parameter tuning.The model is intended to adapt to scene type, content, brightness, and contrast.Source: A. Problem Formulation · pp. 3–4
  • The model scales linear HDR input to [0,1] at 32-bit floating-point precision and performs tone mapping on luminance before reproducing color.Color reproduction uses Cout = Cin Lin · Lout.Source: A. Problem Formulation · p. 4
  • L1 loss can blur outputs, while perceptual loss can overcompress contrast; TMQI cannot directly serve as a backpropagation loss.TMQI’s naturalness component has characteristic discontinuity, and a prior approach’s separate SSIM and naturalness optimization is not readily incorporated into an end-to-end pipeline.Source: A. Problem Formulation · p. 4
  • A learned adversarial loss can encode desired tone-mapping characteristics from training data, avoiding manual loss design and favoring perceptually superior outputs over mean-valued or blurred ones.Source: A. Problem Formulation · p. 4
  • The paper investigates single- and multi-scale generator and discriminator designs to pursue artifact-free, high-resolution tone-mapped output.Source: A. Problem Formulation · p. 4

B. DeepTMO (Single-Scale)

The single-scale DeepTMO uses a supervised conditional GAN to map HDR luminance to tone-mapped LDR luminance. Its generator is an encoder–decoder, while its discriminator evaluates local image patches.

B. DeepTMO (Single-Scale)

  • The single-scale model is trained on corresponding HDR and tone-mapped LDR image pairs, with the generator producing LDR luminance and the discriminator distinguishing real from generated pairs.The generator receives normalized HDR luminance; the discriminator receives HDR and LDR luminance pairs.Source: B. DeepTMO (Single-Scale) · p. 5
  • The generator downsamples HDR input, applies nine two-layer residual blocks, then upsamples the compressed representation into a full-size LDR output.Its encoder–decoder comprises a convolutional front end, residual blocks, and a deconvolutional back end.Source: B. DeepTMO (Single-Scale) · p. 5
  • The 70 × 70 PatchGAN discriminator evaluates overlapping local patches across the image, enabling fully convolutional use on images of varying size.Its local patch focus also models high-frequency information.Source: B. DeepTMO (Single-Scale) · p. 5
  • Single-scale generator and discriminator yield globally high-quality reconstructions but can leave noisy artifacts around bright light sources.Source: B. DeepTMO (Single-Scale) · p. 5

C. DeepTMO (Multi-Scale)

The multi-scale model combines two-scale generator and discriminator designs to address artifacts while retaining local detail and global structure. The paper reports that the combined design produces structurally preserved, refined outputs without local noisy artifacts.

C. DeepTMO (Multi-Scale)

  • The two-scale discriminator processes original and 2× downsampled inputs, letting the coarse scale assess broader patch context and the fine scale emphasize local nuances.The coarser discriminator supports globally consistent patch-level details, while the finer one attends to minute details.Source: C. DeepTMO (Multi-Scale) · p. 5
  • Adding Multi-D suppresses Single-D’s noisy patterns to a large extent, but traces remain because of Single-G’s limitations; adding Multi-G produces globally consistent and locally refined output.Source: C. DeepTMO (Multi-Scale) · p. 6
  • Multi-G combines a full-resolution Go for fine detail with a Gd receiving a 2× downsampled input for a broader-view coarse prediction.The final output combines Go’s local prediction with input from Gd.Source: C. DeepTMO (Multi-Scale) · p. 6, Figure 6 · p. 6
  • Across the four generator–discriminator combinations, Single-D and Single-G leave noisy patterns, while using both multi-scale components gives the reported superior-quality result.Multi-D with Single-G suppresses patterns partly; Single-D with Multi-G removes them broadly but leaves localized artifacts.Source: C. DeepTMO (Multi-Scale) · pp. 6–7
  • The combined multi-scale generator and discriminator yield superior-quality, artifact-free images with structurally preserved, minutely refined output.Source: C. DeepTMO (Multi-Scale) · pp. 6–7

D. Tone Mapping Objective Function

DeepTMO trains a generator and discriminator on paired HDR-input and tone-mapped-output luminance channels, using a conditional adversarial objective. Feature-matching and VGG perceptual losses augment that objective to guide image quality.

  • The generator and discriminator learn from paired HDR-input and tone-mapped-output luminance channels, with the generator synthesizing outputs and the discriminator distinguishing real from generated images.The conditional adversarial objective has the generator minimize and the discriminator maximize the loss.Source: D. Tone Mapping Objective Function · p. 7
  • The multi-discriminator objective accounts for the output- and downsampled-scale discriminators.Source: D. Tone Mapping Objective Function · p. 7
  • Feature-matching loss compares intermediate discriminator features for real and generated images, while VGG perceptual loss compares features from a pretrained 19-layer VGG network.The feature-matching loss extracts features from individual discriminator layers; the VGG network was pretrained on ImageNet classification.Source: D. Tone Mapping Objective Function · p. 7
  • Without feature-matching and VGG losses, output quality is inferior; VGG suppresses noisy repetitive patterns, while feature matching tones down overexposed window regions.Together, the losses support overall contrast and preservation of local fine details.Source: D. Tone Mapping Objective Function · p. 7, Figure 7 · p. 7

E. Network Insight

The network compresses HDR input into an encoded representation, maps it into LDR space, and transforms that representation into an LDR output.

  • The convolutional front ends compress HDR tone into an encoded HDR-space representation, while convolutional layers downsample spatial resolution and Instance Norm and activations compress pixel intensity range.Source: E. Network Insight · p. 7
  • Residual layers map encoded information from HDR space to LDR space by adding residual information to the compressed HDR input.Source: E. Network Insight · pp. 7–8
  • Back-end layers transform the LDR-space representation into an LDR output, with transposed convolutions upsampling spatial resolution.Activation functions maintain a compressed tone described as perceptually appealing for the scene.Source: E. Network Insight · p. 8

IV. BUILDING THE HDR DATASET

The paper builds a diverse HDR training set and selects tone-mapped targets using TMQI. It also reports that instance normalization avoids the non-uniform luminance compression seen with batch normalization.

IV. BUILDING THE HDR DATASET

  • 698 diverse HDR images were collected for training, and 105 images from were reserved for testing.Training images were selected from multiple sources, with video frames manually chosen to avoid similar scenes; random cropping and flipping augmented training.Source: §IV · p. 8

A. Target Tone Mapped Images

  • For each training HDR scene, the highest-TMQI output among 13 classical TMOs and gamma and log mappings becomes the target, using default TMO parameters.The authors note that fine-tuning could further boost performance but is impractical across the large image set and parameter spaces.Source: A. Target Tone Mapped Images · p. 8

V. TRAINING AND IMPLEMENTATION DETAILS

  • Instance normalization learns mean and standard deviation per scene; plain batch normalization yields non-uniform luminance compression, while omitting normalization produces poor generation quality.The normalization comparison is shown in Fig. 8.Source: 1) Instance Vs. Batch Norm · p. 8, Figure 8 · p. 8

2) Implementation:

DeepTMO is evaluated on unseen real-world HDR scenes against highly ranked tone-mapping outputs, with quantitative, qualitative, and runtime comparisons. It scores highest on average TMQI across 105 test scenes and is preferred in most subjective comparisons, while showing occasional halos and oversaturated colors.

2) Implementation:

  • The implementation uses mini-batch SGD and ADAM, with batch size 1 for multi-scale training because of limited GPU memory.Training uses random resizing and cropping; all networks are trained from scratch.Source: 2) Implementation · pp. 8–9

VI. RESULTS AND EVALUATION

A. Comparison with the Best Quality Tone-Mapped Images

B. Quality Evaluation

The study reports a pairwise subjective comparison involving 20 participants with normal or corrected-to-normal vision, aged 23–38.

  • Twenty participants aged 23–38 with normal or corrected-to-normal vision took part in the pairwise subjective comparison.Source: B. Quality Evaluation · p. 11

1) Test Environment and Setup:

The subjective study compared DeepTMO with top TMQI-ranked outputs across selected scene categories. Participants chose the more realistic and appealing image in side-by-side comparisons, and DeepTMO was significantly preferred for most scenes.

1) Test Environment and Setup:

  • The test used 15 scenes from five categories, pairing each DeepTMO output with a top TMQI-ranked tone-mapped image.The categories were humans, dark/noisy, indoor, structures, and landscapes.Source: 1) Test Environment and Setup · p. 11

2) Procedures:

  • Observers chose the more realistic and appealing image in side-by-side comparisons, with each pair shown twice in reversed left-right order.Participants had unlimited time, and the procedure included a training session.Source: 2) Procedures · pp. 11–12

3) Results:

  • DeepTMO was significantly preferred over the best TMQI-rated images for most scenes, with observers generally favoring outputs that preserved contrast.The study used thresholds of at least 13 of 20 votes for significant preference and no more than 6 for least favored; interviews suggested some detailed TMQI targets looked like paintings.Source: 3) Results · p. 12

VII. CONCLUSION, LIMITATIONS AND FUTURE WORK

The authors present DeepTMO as a parameter-free, multi-scale cGAN approach that preserves contrast, lighting, and fine detail across HDR scenes. They identify limitations in training targets, halo artifacts, and color saturation, and discuss alternative training and color-correction approaches.

VII. CONCLUSION, LIMITATIONS AND FUTURE WORK

  • DeepTMO uses a multi-scale GAN to preserve contrast, lighting, and fine details while addressing blurry or tiling effects in high-resolution outputs.The paper reports comparisons of architectural choices, loss functions, and normalization methods.Source: §VII · p. 12

A. Limitations and Future Work

  • The model’s expressive power is limited by available training data and target quality, since TMQI targets are not as perfect as the human visual system.The authors note that some lower-ranked TMQI images appear more interesting than the top-ranked counterpart.Source: A. Limitations and Future Work · p. 13
  • Halo rings around bright regions are linked to scarce training examples and overly saturated targets, and log-scaled input does not remove them.Source: A. Limitations and Future Work · p. 13
  • The authors suggest weakly supervised or unsupervised learning as alternatives that could reduce reliance on noisy tone-mapped ground truths.Source: A. Limitations and Future Work · p. 13
  • DeepTMO can produce overly saturated colors; the authors propose existing color correction methods or learning HDR-to-LDR color mapping as possible remedies.Their example applies a color-correction method with a saturation-control parameter.Source: Figure 16 · p. 12, A. Limitations and Future Work · p. 13

Related papers

Loading 1908.04197v1…