PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications

Soo Ye Kim, Jihyong Oh, Munchurl Kim

arXiv:1904.11176v3eess.IVcs.CV

TL;DR

Legacy LR SDR video must be converted to HR HDR for UHD HDR displays, amid a lack of original UHD HDR content. Deep SR-ITM combines input decomposition and modulation for this joint conversion, and the paper reports restored details and enhanced contrast compared with cascaded methods.

  • Problem

    Legacy FHD SDR content must be converted for UHD HDR displays, while original UHD HDR visual content is scarce.

    Source: §1 · p. 1

  • Method

    Deep SR-ITM decomposes the input into base and detail layers and uses modulation blocks to directly predict HR HDR images in HDR display format.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Deep SR-ITM restores details and enhances contrast compared with cascaded SR and ITM methods, with natural colors reported against the compared joint method.

    Source: §5 · p. 8, §1.1 · p. 11

  • Takeaways & Limitations

    Directly reconstructing HR HDR from transmitted LR SDR video requires less bandwidth and can provide high-quality HR HDR content on high-end TVs.

    Source: §1 · p. 1

Abstract

from arXiv · show

Recent modern displays are now able to render high dynamic range (HDR), high resolution (HR) videos of up to 8K UHD (Ultra High Definition). Consequently, UHD HDR broadcasting and streaming have emerged as high quality premium services. However, due to the lack of original UHD HDR video content, appropriate conversion technologies are urgently needed to transform the legacy low resolution (LR) standard dynamic range (SDR) videos into UHD HDR versions. In this paper, we propose a joint super-resolution (SR) and inverse tone-mapping (ITM) framework, called Deep SR-ITM, which learns the direct mapping from LR SDR video to their HR HDR version. Joint SR and ITM is an intricate task, where high frequency details must be restored for SR, jointly with the local contrast, for ITM. Our network is able to restore fine details by decomposing the input image and focusing on the separate base (low frequency) and detail (high frequency) layers. Moreover, the proposed modulation blocks apply location-variant operations to enhance local contrast. The Deep SR-ITM shows good subjective quality with increased contrast and details, outperforming the previous joint SR-ITM method.

1. Introduction

The paper addresses converting legacy LR SDR video into HR HDR for UHD HDR displays, where SR and ITM must jointly restore spatial detail and contrast. Deep SR-ITM decomposes the input and uses modulation blocks, directly predicting HDR display-format images for 4K applications.

1. Introduction

  • Converting legacy FHD SDR video to UHD HDR is motivated by premium displays and the lack of original UHD HDR content.The paper also describes direct LR SDR transmission followed by on-device reconstruction as requiring less bandwidth.Source: §1 · p. 1
  • Joint SR-ITM must restore high-frequency detail lost with reduced resolution while recovering local contrast and detail lost with SDR’s reduced signal range.The target is an HR HDR image predicted from an LR SDR input.Source: §1 · p. 1
  • Deep SR-ITM separates the input into base and detail layers, with distinct feature-extraction passes that focus on low- and high-frequency components.The detail pass focuses on restoring details, while spatially variant modulation operations enhance local contrast.Source: §1 · p. 2
  • Unlike linear-domain ITM methods that predict scene luminance, Deep SR-ITM directly predicts HR HDR images in the pixel domain and HDR display format.The paper trains and tests on 4K HDR videos for practical applications.Source: §1 · p. 2
  • The modulation blocks apply location-specific operations to enhance local contrast, addressing a central challenge of joint SR-ITM.The paper identifies modulation blocks as a key contribution.Source: §1 · p. 2

2. Related work

Prior ITM approaches have limitations for joint SR-ITM, including restricted region processing and luminance-only prediction. Deep SR-ITM instead combines signal decomposition and modulation with residual and skip blocks for the joint task.

2.1. Inverse tone-mapping

  • For joint SR-ITM, saturated-region-only processing and luminance prediction do not address color changes across all regions or gamut expansion; auto-encoder structures may also lose spatial information important for SR.Deep SR-ITM is designed to restore details and enhance local contrast across pixel regions while predicting HR HDR in display format.Source: §2.1 · p. 2

2.2. Super-resolution

  • Deep SR-ITM uses residual, skip, and modulation blocks alongside guided-filter input decomposition to restore high-frequency detail and local contrast.The decomposition separates aspects of the input for the network to process.Source: §2.2 · pp. 2–3, Figure 3 · p. 3

2.3. Joint SR-ITM

  • Unlike cascaded SR and ITM, which can increase complexity and accumulate prediction error, Deep SR-ITM is a dedicated joint network using decomposed base and detail signals for spatially variant modulation.The earlier joint method used a multi-purpose CNN with separate SR, ITM, and joint-output branches.Source: §2.3 · p. 3

3. Proposed method

Deep SR-ITM maps a single LR SDR image to an HR HDR image using signal decomposition and modulation. Separate base and detail processing supports color and intensity expansion alongside high-frequency detail restoration.

3. Proposed method

  • Deep SR-ITM generates an HR HDR image from a single LR SDR image using signal decomposition and modulation.The output uses BT.2020 and PQ-OETF.Source: §3 · p. 3

3.1. Input decomposition

  • The guided filter produces a low-frequency base layer, and element-wise division yields a mostly colorless detail layer dominated by high-frequency edges and texture.The original image is also concatenated with the base and detail layers before they enter separate feature-extraction passes.Source: §3.1 · p. 3

3.2. Residual skip modulation blocks

  • Spatially variant, image-adaptive modulation supports pixel-by-pixel signal adjustment, while residual, modulation, and skip blocks form the network's feature-extraction paths.ResSkipBlocks bridge base-pass modulation features into the detail pass; a dimension-reduction layer selects information from the expanded input.Source: §3.2 · p. 4

3.3. Fusion and synthesis

  • The network concatenates base- and detail-pass features, processes them with residual blocks, then predicts HR HDR output using pixel shuffle and a bicubic-upscaled input residual.Global residual learning and local residual learning inside the four block types are used to ease training and enhance prediction accuracy.Source: §3.3 · p. 4

3.4. Toy network

  • The toy network is designed to support efficient experiments whose results mirror the original network, while retaining its decomposition, modulations, and skip and residual connections.It includes global and local residual connections and skip connections from the base to detail pass.Source: §3.4 · p. 4, Figure 4 · p. 5

4. Experiment results

Deep SR-ITM outperforms previous methods on all reported measures and scales except HDR-VDP at SF = 4. The experiments evaluate its data, layer decomposition, modulation, component ablations, and realistic-condition reconstructions.

4.1. Experiment conditions

  • The 4K HDR dataset contains 59,818 frames from ten YouTube videos; seven videos were used for training and three for testing.The authors chose YouTube-converted SDR pairs because the 19 tested tone-mapping operators produced unnatural colors and unstable dark-scene results.Source: §4.1 · p. 5, Figure 5 · p. 5

4.2. Input decomposition

  • Column (d), with separate base and detail passes and the image stacked into each, achieves the highest gain over column (a); it beats column (b) by 0.21 dB PSNR.For two-pass structures, stacking decomposed layers yields a 0.35 dB PSNR gain.Source: §4.2 · p. 6

4.3. Modulation

  • Stacking decomposed layers for modulation inputs yields at most 0.08 dB PSNR, less than the 0.35 dB gain for network inputs.The authors report that specific base- or detail-layer features can suffice for modulation, while the integrated image is needed in the main pass.Source: §4.3 · p. 6, Table 2 · p. 6

4.4. Ablation study

  • Guided-filter decomposition and modulation add 0.08 dB and 0.22 dB PSNR, respectively, for a total 0.3 dB gain over the baseline.Skip connections are effective only without modulation; their benefits appear to overlap with modulation.Source: §4.4 · p. 6, Table 3 · p. 6

4.5. Performance comparisons

  • Deep SR-ITM outperforms previous methods on all reported measures and scales except HDR-VDP at SF = 4.Table 4 gives the quantitative comparison for SF = 2 and SF = 4.Source: §4.5 · p. 7, Table 4 · p. 8

4.6. Reconstruction under realistic conditions

  • Without HDR ground truth, tests on originally 4K SDR videos show reconstructions with enhanced contrast and detail under realistic conditions.Figure 9 shows bicubic-upscaled LR SDR inputs above the HR HDR predictions.Source: §4.6 · p. 8, Figure 9 · p. 8

5. Conclusion

Deep SR-ITM decomposes SDR input into low- and high-frequency information and uses spatially variant modulation to enhance local contrast.

5. Conclusion

  • The detail layer assists spatial up-scaling by restoring high-frequency details, while the base layer assists ITM by expanding intensity and color.Modulation uses spatially variant operations to boost local contrast in the image signal.Source: §5 · p. 8
  • The framework directly produces HR HDR images in the pixel domain for UHD HDR consumer displays.The authors describe this as a handy application for generating premium visual content.Source: §5 · p. 8
  • The authors report that relevant code and test-set information are available on GitHub.Source: §5 · p. 8

1. Additional qualitative comparisons

The added comparisons report stronger contrast and detail from Deep SR-ITM against cascaded methods, and more natural colors than the prior joint CNN-based method.

1.1. Lack of contrast in the cascade of EDSR and Eilertsen et al.’s method

  • Deep SR-ITM shows enhanced contrast over the EDSR and Eilertsen et al. cascade.Figure 1 presents this qualitative comparison.Source: Figure 1 · p. 11, §1.1 · p. 11

1.2. More comparisons against the cascade of conventional methods

  • Against cascades of EDSR and ITM methods, Deep SR-ITM restores local contrast and details while increasing resolution and signal amplitudes.It directly generates HR HDR frames in display format, avoiding the manual conversions required by the cascaded pipeline.Source: §1.2 · p. 12, Figure 2 · p. 12

1.3. Color artifacts in the CNN-based joint SR-ITM method

  • Compared with Kim et al.’s joint CNN method, Deep SR-ITM shows natural colors, while that method exhibits unnatural colors and banding in homogeneous regions.The artifacts are noted especially in sky and water regions.Source: §1.3 · p. 13, Figure 3 · p. 13

1.4. Comparison videos

  • The comparison videos were generated frame-by-frame from 2K (1920×1080) SDR inputs in 4K (3840×2160) resolution and HDR display format, with ground-truth videos at each clip’s end.For Eilertsen et al.’s method, test frames were cropped to 3840×2144 and zero-padded at the bottom to produce 4K frames.Source: §1.4 · p. 13
  • The videos are intended for HDR10-capable consumer TVs; SDR viewing requires the specified MPC-HC and madVR setup and should not be judged with other settings.The authors say the comparisons are difficult to convey on paper because they are presented at actual 4K resolution and in HDR display format.Source: §1.4 · p. 13

2. Additional quantitative comparisons

Against LapSRN cascaded with ITM methods, Deep SR-ITM outperforms in all measures and scales except HDR-VDP at SF = 4. The authors conclude that color compliance matters more to final HR HDR prediction than high-frequency detail reconstruction.

2. Additional quantitative comparisons

  • The similar ranges of values across SR baselines suggest that the SR method’s influence is minimal compared with the ITM method’s influence.The comparison uses LapSRN with ITM methods and is described as similar to the EDSR comparison in Table 4.Source: §2 · p. 14
  • The authors conclude that overall color compliance matters more than high-frequency detail reconstruction for final HR HDR prediction.Source: §2 · p. 14
  • Deep SR-ITM outperforms LapSRN cascaded with ITM methods in all measures and scales except HDR-VDP for SF = 4.Table 1 presents the qualitative comparison against the LapSRN and ITM cascade.Source: Table 1 · p. 14, §2 · p. 14

Related papers

Loading 1904.11176v3…