PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time Image Enhancer via Learnable Spatial-aware 3D Lookup Tables

Tao Wang, Yong Li, Jingyang Peng, Yipeng Ma, Xian Wang, Fenglong Song, Youliang Yan

arXiv:2108.08697v1cs.CVeess.IV

TL;DR

Existing enhancement methods often fall short on perceptual quality or computational efficiency, particularly for high-resolution images. The paper combines learnable spatial-aware 3D LUTs with global and pixel-wise weighting, and reports advantages over state-of-the-art methods on public datasets, with 4K processing in about 4 ms on an NVIDIA V100.

  • Problem

    Many enhancement methods do not meet practical requirements for visual perception or computational efficiency, especially on high-resolution images.

    Source: §1 · p. 1

  • Method

    The method learns spatial-aware 3D LUTs and fuses them using a two-head predictor's image-level weight vector and pixel-wise weight map.

    Source: §1 · pp. 1–2

  • Results

    The authors report performance and efficiency advantages over state-of-the-art methods on public datasets, with about 4 ms to process a 4K image on an NVIDIA V100.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The approach combines image-level scenario adaptation with pixel-wise category fusion in an efficient LUT-based enhancer.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recently, deep learning-based image enhancement algorithms achieved state-of-the-art (SOTA) performance on several publicly available datasets. However, most existing methods fail to meet practical requirements either for visual perception or for computation efficiency, especially for high-resolution images. In this paper, we propose a novel real-time image enhancer via learnable spatial-aware 3-dimentional lookup tables(3D LUTs), which well considers global scenario and local spatial information. Specifically, we introduce a light weight two-head weight predictor that has two outputs. One is a 1D weight vector used for image-level scenario adaptation, the other is a 3D weight map aimed for pixel-wise category fusion. We learn the spatial-aware 3D LUTs and fuse them according to the aforementioned weights in an end-to-end manner. The fused LUT is then used to transform the source image into the target tone in an efficient way. Extensive results show that our model outperforms SOTA image enhancement methods on public datasets both subjectively and objectively, and that our model only takes about 4ms to process a 4K resolution image on one NVIDIA V100 GPU.

1. Introduction

Existing enhancement methods struggle to combine perceptual quality with real-time efficiency, particularly at high resolutions. The paper introduces spatial-aware 3D LUTs with a two-head predictor to adapt enhancement globally and locally.

1. Introduction

  • On HDR+ at 480p and 4K, the method achieves the highest PSNR and second-fastest execution speed among compared methods on an NVIDIA V100.DeepLPF [16] runs out of memory at 4K.Source: Figure 1 · p. 1
  • The proposed enhancer combines spatial-aware 3D LUTs with a two-head weight predictor to address local-information gaps in traditional LUT-based enhancement.The predictor supplies image-level scenario weights and pixel-wise category weights for LUT fusion.Source: §1 · pp. 1–2
  • The method processes a 4K image in about 4 ms on an NVIDIA V100 GPU.Source: §1 · p. 2
  • The authors report extensive quantitative and qualitative comparisons on two public datasets, finding advantages in performance and efficiency.Source: §1 · p. 2

2. Related Work

Related work is grouped into pixel-level, patch-level, and image-level enhancement methods, each with different performance and resource trade-offs.

2. Related Work

  • Pixel-level methods use CNNs for dense pixel-to-pixel reconstruction and achieve strong results, but their feature extraction and regeneration are costly, especially at high resolutions.Source: §2 · p. 2
  • Patch-level methods apply features from down-sampled inputs to local patches and can perform well in computation and memory, yet may still overload hardware at ultra-high resolutions.Source: §2 · pp. 2–3
  • Image-level methods efficiently apply global scaling factors or mapping curves, but image-adaptive 3D LUTs [25] omit spatial information and can produce poor local contrast or color.Source: §2 · pp. 2–3

3. Methodology

The method extends 3D-LUT enhancement with spatially aware LUTs and a two-head predictor, then trains the system with losses targeting content consistency, LUT regularity, color, and perception.

3.1. Network architecture

  • Each pixel’s RGB value indexes LUT mappings, and a pixel-wise category map weights their results to produce a spatially aware output.Traditional LUT mapping depends only on pixel values; the proposed approach adaptively fuses multiple LUTs using pixel-wise weights.Source: Figure 3 · p. 3, §3.1 · p. 3
  • The two-head predictor outputs global scene-adaptation probabilities and an M-channel, pixel-wise map for fusing LUTs.The global probabilities combine spatial-aware LUTs across scenes, while each map channel supplies a fusion weight for specific LUTs.Source: §3.1 · p. 4
  • Spatial-aware trilinear interpolation uses the pixel-wise category weights, while trilinear interpolation is selected to improve output smoothness.Source: §3.1 · p. 4

3.2. Loss function

  • The loss combines MSE for content consistency, smoothness and monotonicity terms for LUT regularity, CIE94 color difference, and LPIPS perceptual loss.The color term uses CIE94 in LAB space to match target colors; LPIPS is chosen to improve perceptual quality.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate the method on public image-enhancement datasets, compare it with other methods, and test its components. They report strong benchmark results and fast 4K inference, alongside ablations that guide the final design.

4. Experiments

  • Evaluation uses MIT-Adobe FiveK and HDR+ datasets, with PSNR, SSIM, and LPIPS; higher PSNR/SSIM and lower LPIPS indicate better results.The experiments also construct full-resolution FiveK and 480p and full-resolution HDR+ datasets; training uses PyTorch on a V100 for 400 epochs with batch size 1 and customized CUDA interpolation.Source: §4 · pp. 4–5, §4.1 · p. 5

4.1. Ablation study

  • Combining 1D and 3D weights in ours(3,10) improves PSNR by 2.82 dB over the original method; directly adding LUTs to Zeng’s [25] method does not improve performance effectively.The full-resolution variant has 0.17 dB PSNR degradation; removing resize operations gains 0.09 dB but increases FLOPS from 8.111G to 113.79G, so ours(3,10) is retained as the final architecture.Source: Table 1 · p. 5, §4.1 · p. 5
  • Increasing the pixel-wise category count M improves performance up to M = 10, with only minor improvement or deterioration at larger values, so subsequent experiments use M = 10.The tested values range from M = 1 to M = 64 with T = 3.Source: Figure 4 · p. 5, §4.1 · p. 5
  • Adding color loss and perception loss to the basic loss dramatically improves all three evaluation metrics; visual comparisons also report clearer detail and better color and local contrast with both losses.The basic loss combines MSE, smoothness, and monotonicity losses; Figure 5 describes plants as fuzzy with color loss alone and sharper detail after adding perception loss.Source: Table 2 · p. 5, §4.1 · p. 6, Figure 5 · p. 6

4.2. Analysis of pixel-wise category map

  • The pixel-wise category maps can reflect semantic groupings such as person, sky, and building, or illumination levels such as middle, low, and high.The maps are not explicitly loss-supervised; the authors report a 1.52 dB PSNR degradation when they tried explicit loss on the category map.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3. Comparison with State-of-the-Arts

  • Our approach outperforms other methods on MIT-Adobe FiveK in PSNR and LPIPS, and exceeds the second-best HDR+ model by 0.85 dB at 480p and 1.59 dB at full resolution.On 480p MIT-Adobe FiveK, its SSIM is less than 1% below DeepLPF, while other metrics are much better; DeepLPF cannot run at full resolution because of memory use.Source: §4.3 · p. 6, Table 3 · p. 7, Table 4 · p. 7
  • At 4K, our method takes about 4 ms on a V100 and is about two orders faster than other methods except 3DLUT.The reported running time remains approximately unchanged with image resolution, while competing methods other than 3DLUT take exponentially longer as resolution increases.Source: §4.3 · pp. 6–7, Table 5 · p. 7

5. Discussion and Conclusion

The spatial-aware 3D LUT method is reported to combine good local contrast with high efficiency and to outperform competing methods on performance and efficiency. Visual comparisons describe improved color, detail, and fewer artifacts, while HDRNet’s performance decreases as input resolution increases for a fixed grid size.

5. Discussion and Conclusion

  • A 0.91dB PSNR difference between 480p and 4K inference illustrates HDRNet’s resolution sensitivity for a fixed grid size.The authors conclude HDRNet performance decreases as input resolution grows.Source: §5 · p. 7
  • The method is reported to outperform many SOTA methods on both performance and efficiency, while producing good local contrast.The authors attribute local robustness to spatial-aware 3D LUTs with pixel-wise category maps.Source: §5 · p. 7
  • On the MIT-Adobe FiveK example, the proposed result is described as outperforming other methods in color and detail.The comparisons note red tint, insufficient background saturation, contour artifacts, darker output, and lost cloth texture in competing results.Source: Figure 7 · p. 8
  • On the HDR+ example, the proposed result is described as closer to ground truth, with clearer texture, better color, and fewer artifacts.Other results show yellowing, sky bending, or blur; traditional (f) leaves a red block dark rather than enhancing local areas differently.Source: Figure 8 · p. 8

Related papers

Loading 2108.08697v1…