Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement

Chongyi Li; Chun-Le Guo; Man Zhou; Zhexin Liang; Shangchen Zhou; Ruicheng Feng; Chen Change Loy

Source-linked AI summary

Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement

Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangchen Zhou, Ruicheng Feng, Chen Change Loy

arXiv:2302.11831v1Feb 23, 2023cs.CV

TL;DR

UHD LLIE requires joint luminance enhancement and noise removal without excessive computation, while existing methods struggle with real-world UHD images. UHDFour processes amplitude and phase separately in a Fourier-domain cascaded network, using low-resolution computation and high-resolution adjustment; experiments report superior UHD LLIE performance with appealing efficiency and introduce the 2,150-pair UHD-LL dataset.

Problem
Existing LLIE methods struggle with joint luminance enhancement, noise removal, and efficient processing for real-world UHD images.
Method
UHDFour embeds Fourier processing in a cascaded network that separately handles amplitude and phase, performs most computation at low resolution, and adjusts high-resolution features.
Results
UHDFour outperforms state-of-the-art methods in UHD LLIE with appealing efficiency, while its evaluation uses the UHD-LL benchmark.
Takeaways & Limitations
UHD-LL enables comparison of LLIE methods on real UHD low-light images, and UHDFour provides an efficient solution for the UHD setting.
Takeaways & Limitations
The experiments cover image enhancement rather than video, do not consider adversarial losses, and use sRGB data that may fail on extreme information-loss cases.

Abstract

from arXiv · show

Ultra-High-Definition (UHD) photo has gradually become the standard configuration in advanced imaging devices. The new standard unveils many issues in existing approaches for low-light image enhancement (LLIE), especially in dealing with the intricate issue of joint luminance enhancement and noise removal while remaining efficient. Unlike existing methods that address the problem in the spatial domain, we propose a new solution, UHDFour, that embeds Fourier transform into a cascaded network. Our approach is motivated by a few unique characteristics in the Fourier domain: 1) most luminance information concentrates on amplitudes while noise is closely related to phases, and 2) a high-resolution image and its low-resolution version share similar amplitude patterns.Through embedding Fourier into our network, the amplitude and phase of a low-light image are separately processed to avoid amplifying noise when enhancing luminance. Besides, UHDFour is scalable to UHD images by implementing amplitude and phase enhancement under the low-resolution regime and then adjusting the high-resolution scale with few computations. We also contribute the first real UHD LLIE dataset, \textbf{UHD-LL}, that contains 2,150 low-noise/normal-clear 4K image pairs with diverse darkness and noise levels captured in different scenarios. With this dataset, we systematically analyze the performance of existing LLIE methods for processing UHD images and demonstrate the advantage of our solution. We believe our new framework, coupled with the dataset, would push the frontier of LLIE towards UHD. The code and dataset are available at https://li-chongyi.github.io/UHDFour.

1 INTRODUCTION

UHD LLIE must jointly enhance luminance, remove noise, and remain computationally efficient, but existing methods struggle with real-world UHD inputs. UHDFour addresses these challenges through Fourier-domain processing and the UHD-LL benchmark.

UHD image processing increases efficiency demands because additional pixels make existing algorithms harder to run.
LLIE in the UHD regime requires simultaneous luminance enhancement, noise removal, and efficient processing.
Existing LLIE methods either focus on luminance, process luminance and noise jointly in the spatial domain, or rely on low-resolution training data.
UHDFour separately handles luminance and noise in the Fourier domain because luminance appears mainly in amplitudes while noise is closely related to phase.
2,150 real 4K low-noise/normal-clear image pairs comprise UHD-LL, covering diverse darkness, noise levels, and capture scenarios.
UHDFour and UHD-LL support systematic analysis of existing LLIE methods and evaluation of UHD enhancement effectiveness and efficiency.

2 OUR APPROACH

UHDFour embeds Fourier processing in a cascaded low- and high-resolution network. It separately enhances amplitude and phase, performs most computation at low resolution, and applies lightweight high-resolution adjustment.

2.1 OBSERVATIONS IN FOURIER DOMAIN: Fourier-domain amplitude swapping suggests luminance and noise can be decomposed to a certain extent.Swapping amplitudes between low-light/noisy and normal-clear images produces normal-noise and low-clear compositions.
2.1 OBSERVATIONS IN FOURIER DOMAIN: High-resolution and low-resolution normal-clear images share similar amplitude patterns, enabling extensive processing at low resolution and minor high-resolution adjustments.
2.2 THE UHDFOUR NETWORK: UHDFour maps an UHD low-noise input x to a normal-clear target y using an LRNet and an HRNet.
2.2 THE UHDFOUR NETWORK: The LRNet downsamples features to 1/8 resolution, processes them with an encoder-decoder, and estimates refined amplitude and phase features plus a low-resolution output.
2.2 THE UHDFOUR NETWORK: The HRNet preserves input information through PixelUnshuffle, adjusts high-resolution features, and restores the original resolution with Pixelshuffle and upsampling.
2.2 THE UHDFOUR NETWORK: The FouSpa Block parallelly enhances Fourier amplitude and phase components alongside spatial features.
2.2 THE UHDFOUR NETWORK: The Adjustment Block modulates high-resolution amplitude using refined low-resolution amplitude and concatenates refined and input phase features.
2.2 THE UHDFOUR NETWORK: Training combines L1 supervision for low- and high-resolution outputs, perceptual loss for the low-resolution output, and SSIM loss for the high-resolution output.

3 UHD-LL DATASET

UHD-LL is a real paired dataset of 2,150 4K low-light and normal-clear images, designed to represent diverse noise and darkness conditions. Its collection and split procedures address pairing stability and evaluation consistency.

UHD-LL contains 2,150 paired 4K images in 8-bit sRGB format, with low-noise and normal-clear counterparts.
The dataset uses two Sony cameras and captures normal-clear images at ISO ∈[100, 800] and low-noise images at ISO ∈[1000, 20000] with reduced exposure time.
Neutral-density filters help produce extremely dark images while increasing ISO, enriching the range of darkness and noise levels.
Tripod mounting, remote exposure control, and affine image alignment are used to reduce misalignment from camera shake and dynamic objects.
The dataset is split into 2,000 training pairs and 115 testing pairs, with exclusive scenes and data between partitions.
Compared with classic LLIE datasets, UHD-LL provides more images, diverse scenes and darkness/noise levels, and higher resolutions.

4 EXPERIMENTS

Experiments show that existing LLIE models struggle with UHD low-light restoration, while UHDFour achieves stronger quality and efficiency on the UHD-LL benchmark.

4.1 BENCHMARKING EXISTING MODELS: Existing released models improve luminance but leave noise, artifacts, over-exposure, or color deviations in UHD results.The evaluation uses models released from their original papers on UHD low-light images.
4.1 BENCHMARKING EXISTING MODELS: Non-reference metrics disagree with visual quality, as RUAS-LOL is over-exposed despite achieving the best NIMA score.The authors argue that generic non-reference metrics do not accurately assess enhanced UHD low-light images.
4.2 COMPARING RETRAINED MODELS: UHDFour achieves state-of-the-art PSNR, SSIM, and LPIPS performance on retrained UHD-LL comparisons.The method outperforms compared methods by a large margin according to Table 3.
4.2 COMPARING RETRAINED MODELS: UHDFour has the fastest UHD processing speed because most computation is conducted in the low-resolution space.Transformer-based SNR-Aware and Restormer rank second best in the retrained comparison.
4.3 ABLATION STUDY: Ablations show that separate amplitude-phase processing, spatial components, FouSpa Blocks, adjustment design, and LR-result estimation all contribute to full-model performance.Replacing the Fourier branch with spatial processing or replacing specialized blocks with residual blocks reduces performance.
5 CONCLUSION: The study is limited to image enhancement, excludes video benchmarks and adversarial losses, and may fail on extreme sRGB cases with information loss.The authors identify HDR data as potentially suitable for extreme cases.

A RELATED WORK

Existing LLIE and image-decomposition methods address related enhancement, denoising, or Fourier-domain problems, but UHDFour specifically targets joint luminance enhancement, noise removal, and UHD resolution constraints. The paper introduces Fourier-domain motivations and a real UHD benchmark to address these gaps.

Existing LLIE methods variously estimate illumination, use attention-guided enhancement, or apply frequency and band decompositions.
UHDFour differs by jointly addressing luminance enhancement, noise removal, and UHD resolution constraints in the Fourier domain.
Prior datasets often omit noise, use retouched or synthetic pairs, require specific sensor patterns, or exclude UHD images.
UHD-LL contributes real low-noise/normal-clear 4K pairs captured across diverse darkness, noise levels, and scenes.
The method is motivated by partial Fourier-domain decomposition of luminance and noise and similar amplitude patterns across high- and low-resolution images.

B ANALYSIS OF UHD-LL DATASET

UHD-LL is designed as a diverse real-world UHD LLIE benchmark with consistent train/test intensity distributions and challenging noise levels. It contains 2,150 paired 4K images spanning varied scenes, darkness, and signal-to-noise ratios.

Training and testing partitions have consistent pixel-intensity distributions, supporting a rational dataset split.
SNR values range from 1 to 30 and center in the [0,10] range, indicating challenging noise levels.
2,150 pairs of 4K UHD low-noise/normal-clear images cover real noise, diverse darkness levels, and many scenes.

C FURTHER ANALYSIS OF MOTIVATION

Further analysis supports the paper’s two Fourier-domain motivations: amplitude swapping separates luminance and noise to some extent, while high- and low-resolution amplitude patterns show a related tendency. Quantitative measurements support the first motivation, whereas the second is assessed visually because spectrum sizes cannot be directly interpolated.

Motivation analysis: Amplitude swapping between corresponding low-noise and normal-clear images produces compositional normal-noise and low-clear images.
Motivation analysis: Additional visual examples in Figures 10–12 show the same tendency as the main motivation examples.
Motivation analysis: Real normal-clear images resemble compositional normal-noise images in average luminance and compositional low-clear images in MUSIQ values.
Motivation analysis: Real low-noise images resemble compositional low-clear images in luminance and compositional normal-noise images in MUSIQ values.
Motivation analysis: The second motivation cannot be quantitatively evaluated with full-reference metrics because amplitude spectra of different sizes cannot be directly interpolated.

D VISUALIZATION IN THE NETWORK

Visualization of UHDFour shows that the final output and its low-resolution output have amplitude and phase components similar to their corresponding ground truths. The paper cautions that phase imagery does not directly visualize noise.

The final UHDFour result has amplitude and phase components similar to those of the ground truth.
The low-resolution output y8 also has amplitude and phase components similar to the corresponding ground truth y8.
Phase is periodic and cannot be accurately represented in imagery format, because it represents the initial position of the wave.

E MORE RESULTS ON RELEASED MODELS

Released state-of-the-art models cannot handle UHD low-light images well, and stitching introduces visible borders for SNR-Aware.

All existing released models perform poorly on UHD low-light images.SNR-Aware results also show obvious borders because full-resolution inference was unavailable and stitching was used.

F MORE RESULTS ON RETRAINED MODELS ON UHD-LL

Retrained state-of-the-art models on UHD-LL still produce unsatisfactory UHD low-light enhancements, with residual noise and artifacts. The results support separately processing luminance and noise in the Fourier domain.

Retrained models on UHD-LL still produce unsatisfactory UHD low-light results.Noise and artifacts remain visible in their outputs.
The results indicate that jointly enhancing luminance and removing noise in the spatial domain is difficult.UHDFour addresses this by embedding Fourier transform in a cascaded network and separately processing decomposed luminance and noise information.

G MORE RESULTS ON RETRAINED MODELS ON LOL-V1 AND LOL-V2 DATASETS

On LOL-v1 and LOL-v2, the method achieves satisfactory performance despite these datasets having mild noise and low-resolution images that limit evaluation of its full capabilities.

UHDFour achieves satisfactory performance on low-light images from the LOL-v1 and LOL-v2 datasets.The datasets’ mild noise and low resolution limit assessment of noise removal and high-resolution processing.
The LOL-v1 and LOL-v2 results suggest potential for the solution in different circumstances.

H ABLATION STUDY

Ablations show that Fourier and spatial processing components, modulation and guidance mechanisms, and low-resolution estimation all contribute to UHDFour’s visual quality and efficiency. Comparable-parameter residual replacements do not reproduce the full model’s performance.

Ablation findings: Removing the Fourier branch prevents effective luminance enhancement and noise removal.
Ablation findings: The Spatial branch affects the result, while replacing the FouSpa Block with a comparable-parameter Residual Block is unsatisfactory.This indicates the FouSpa Block’s performance is not explained by using more parameters.
Ablation findings: Removing Amplitude Modulation produces visually unpleasing results, while Phase Guidance and the Spatial branch contribute to the Adjustment Block’s performance.
Ablation findings: Replacing the Adjustment Block with a comparable-parameter Residual Block remains unsatisfactory, and estimating the low-resolution result produces a clearer output.
Computational analysis: FFT and IFFT have the same computational cost across the evaluated feature scales.The study evaluates 8×, 16×, and 32× downsampled features with 16 channels.