PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Light Image Enhancement via Structure Modeling and Guidance

Xiaogang Xu, Ruixing Wang, Jiangbo Lu

arXiv:2305.05839v1cs.CV

TL;DR

Low-light enhancement methods often leave dark-region structure poorly modeled while recovering appearance from noisy images. This paper jointly models appearance and edge-based structure, uses structure-guided feature synthesis for enhancement, and trains the components end-to-end. Experiments report SOTA performance on PSNR and SSIM across sRGB and RAW datasets with the same architecture.

  • Problem

    Existing low-light enhancement methods mainly optimize appearance distance and ignore explicit structural details in dark regions.

  • Method

    The framework uses a GAN-trained generative edge model with structure-aware features and a structure-guided enhancement module coupled to appearance modeling.

  • Results

    The framework achieves SOTA performance on both PSNR and SSIM across sRGB and RAW datasets with the same architecture.

  • Takeaways & Limitations

    Joint structure modeling and guidance produces sharp and realistic low-light enhancement results while retaining a common architecture across datasets.

Abstract

from arXiv · show

This paper proposes a new framework for low-light image enhancement by simultaneously conducting the appearance as well as structure modeling. It employs the structural feature to guide the appearance enhancement, leading to sharp and realistic results. The structure modeling in our framework is implemented as the edge detection in low-light images. It is achieved with a modified generative model via designing a structure-aware feature extractor and generator. The detected edge maps can accurately emphasize the essential structural information, and the edge prediction is robust towards the noises in dark areas. Moreover, to improve the appearance modeling, which is implemented with a simple U-Net, a novel structure-guided enhancement module is proposed with structure-guided feature synthesis layers. The appearance modeling, edge detector, and enhancement module can be trained end-to-end. The experiments are conducted on representative datasets (sRGB and RAW domains), showing that our model consistently achieves SOTA performance on all datasets with the same architecture.

1. Introduction

The paper addresses blurry low-light enhancement by jointly modeling appearance and structure, using restored structure maps to guide enhancement. Its framework combines generative edge modeling with structure-guided synthesis and achieves strong results across sRGB and RAW datasets.

  • Motivation: Most existing methods optimize appearance distance while ignoring explicit structural details in dark regions, producing blurry outcomes and low SSIM.Structural information such as edges can distinguish regions and reduce the ill-posedness of appearance reconstruction.
  • Framework: A modified generative model predicts edge-based structure maps, while a structure-guided enhancement module synthesizes features for appearance improvement.The generative model uses structure-aware features and GAN training; the enhancement module guides a simple appearance model with restored structure.
  • Training: The appearance model, structure model, and enhancement module are trained end-to-end.The framework includes explicit appearance modeling, structure modeling, and structure-guided enhancement in one jointly trainable system.
  • Results: Extensive experiments show SOTA performance on both PSNR and SSIM across sRGB and RAW datasets using the same architecture.The paper reports consistent performance across different datasets and domains without changing the network architecture.
  • Framework: The proposed framework jointly performs structure modeling and guidance to improve low-light appearance enhancement.The framework explicitly models structure and uses restored structure maps to guide appearance modeling.

2. Related Work

Prior low-light enhancement methods mainly optimize appearance reconstruction, while structural information has also been explored through edges, gradients, or separately trained networks.

  • Low-light enhancement: Deep learning methods mainly optimize appearance reconstruction error, but enhanced dark regions can remain blurry even with SOTA approaches.Related methods have therefore explored structural information to improve enhancement.
  • Structural information: Existing approaches obtain structural maps using offline-computed edges or gradients and offline-trained networks.The cited approaches do not describe the paper’s jointly trained end-to-end framework.
  • Paper framework: The paper presents an overview with explicit appearance modeling A, structure modeling S, and enhancement module E.Structure maps guide appearance enhancement, and the overall framework is trained end-to-end.

3. Method

The framework explicitly models both appearance and structure, using low-light edge maps to guide a U-Net-based enhancement module. A modified generative structure model and structure-guided feature synthesis are trained end-to-end to improve enhancement details and realism.

  • Framework Overview: The framework models appearance and structure separately, then uses predicted structure maps to guide appearance enhancement in an end-to-end pipeline.Appearance modeling uses a common U-Net, while structure modeling predicts edge maps and the enhancement module conditions feature synthesis on them.
  • Structure Modeling: The structure model uses edge information with a modified StyleGAN backbone composed of a Structure-Aware Feature Extractor and Structure-Aware Generator.The extractor receives content and gradient information, while the generator uses the extracted structure-aware features to produce structure maps.
  • Structure-Aware Feature Extractor (SAFE): SAFE computes multi-directional first-order gradients and combines long-range transformer features with short-range CNN features to extract structural representations.Gradient maps emphasize edge regions; long- and short-range features are merged through a Long-Short-Range Fusion module.
  • Structure-Guided Enhancement Module: SGEM enhances the initial appearance prediction by processing the concatenated appearance and input images under structure-map guidance and learning a residual output.The module can be implemented as a simple U-Net, with structure-guided feature synthesis providing the conditioning signal.
  • Structure-Guided Feature Synthesis: Each SGEM decoder layer generates spatially varying convolution kernels and normalization maps from resized structure maps before processing its feature map.Structure Guided Convolutions produce location-specific kernels, while Structure Guided Normalizations produce location-specific normalization parameters.
  • End-to-End Training: Training combines appearance reconstruction, structure regression, and GAN losses, with edge targets obtained from normal-light ground truth using the Canny detector.The GAN loss is introduced because dark-image noise and invisible content make structure estimation difficult with regression loss alone.

4. Experiments

Experiments evaluate the framework on SID and LOL datasets using standard training and comparisons against many low-light enhancement methods. The method surpasses the baselines, with visual results showing reduced noise and clearer visibility.

  • Implementation: The framework is trained in PyTorch on four RTX3090 GPUs using Adam with momentum set to 0.9.These implementation settings describe the experimental training configuration.
  • Datasets: The experiments use SID and LOL, including LOL-real and LOL-synthetic, to cover challenging sRGB and RAW low-light settings.SID includes heavily noisy short-exposure images captured in extremely dark environments.
  • Qualitative analysis: Visual comparisons on LOL-real and LOL-synthetic show that the method produces less noise and clearer visibility.The comparisons use the best-performing baseline on each dataset.
  • Evaluation: The method is compared with a broad collection of state-of-the-art low-light enhancement methods using PSNR and SSIM.The comparison includes methods such as SID, DeepUPE, KIND, MIR-Net, LLFlow, and SNR.
  • Quantitative analysis: The method surpasses all baselines on LOL-real and LOL-synthetic, with especially evident SSIM improvements attributed to structure modeling and guidance.SSIM is used to assess high-frequency details and structures in the enhanced results.

4.4. Comparison on RAW Domain

On SID’s RAW domain, the framework achieves state-of-the-art PSNR and SSIM performance while enhancing lightness and details and suppressing artifacts. Ablations show that its appearance and structure components jointly contribute to the results, and performance remains close under added noise.

  • RAW-domain comparison: The framework achieves state-of-the-art performance on SID’s RAW domain in both PSNR and SSIM.Its SSIM is best among the compared RAW-input methods and reflects image richness and sharpness.
  • RAW-domain comparison: Qualitative RAW results show improved image lightness and revealed details while suppressing artifacts.These visual results support the reported superiority in the RAW domain.
  • Ablation study: The ablation study removes appearance modeling, structure modeling, SAFE, structure-guided feature synthesis, alternative structure guidance, and GAN loss individually.These settings isolate the contributions of the framework’s main components.
  • Ablation study: The full setting yields higher PSNR and SSIM than all ablation settings, supporting joint appearance and structure modeling and structure guidance.The comparison between “Ours w/o A” and “Ours” tests the need for both modeling paths, while “Ours w/o S” tests structure guidance.
  • Robustness: With added Gaussian noise of variance 30 to 50, performance remains close to the setting without noise.The authors use this result to report robustness toward perturbations.

4.6. Evaluation for Structural Modeling

Structural-modeling ablations evaluate edge prediction using cross-entropy and L2 distance against normal-light edge maps. The full setting produces the most accurate structure modeling, while removing GAN loss increases uncertainty and artifacts.

  • Evaluation protocol: Structural modeling is evaluated against normal-light edge maps using cross-entropy and L2 distance.The comparison focuses on variants that retain structure modeling.
  • Structural-modeling results: The full setting achieves the most accurate structure modeling among the evaluated ablations.The comparison includes variants without GAN loss, the proposed structure guidance, or the proposed feature extractor.
  • Structural-modeling results: The proposed structure model outperforms alternative feature-extractor and structure-guidance choices in the ablation comparisons.These comparisons are made between “Ours” and “Ours w/o F” or “Ours w/o S.G.”.
  • GAN-loss analysis: Removing GAN loss increases uncertainty in edge predictions and causes artifacts in the predicted structure maps.The visual comparisons show that the full method has more edge details and fewer artifacts than the ablations.

4.7. User Study

A large-scale user study evaluates enhancement quality through ratings from 100 participants on 50 real-captured low-light iPhone 8 photos. The study covers varied indoor and outdoor environments and compares the method with five strong baselines.

  • Study design: The user study includes 100 participants who rate 50 real-captured low-light photos taken with an iPhone 8.The photos cover varied indoor and outdoor capturing environments.
  • Study design: The evaluation compares the method with five strongest baselines selected by mean PSNR on SID and LOL.All methods are trained on SID before evaluation on the real-captured photos.
  • Evaluation procedure: Human subjective evaluation is conducted through user ratings on six questions.The rating results are summarized in Figure 7.

4.8. Structure Modeling with More Data

The framework can use additional normal-light data to improve structure modeling without extra paired low-light enhancement data. Fine-tuning the structure model produces better structure in very dark areas and improves the result.

  • The experiments reported here use networks trained without extra data before the additional structure-model fine-tuning.
  • Paired low-light and normal-light enhancement data are difficult and expensive to collect, whereas edge supervision can be obtained readily from normal-light images.
  • Fine-tuning the structure model with additional normal-light data improves results by synthesizing better structure for very dark areas.

4.9. Ours with Other Structural Representations

Although the framework uses edges for structure modeling, it also supports segmentation and depth maps as alternative structural representations. These representations are extracted from normal-light data and used as supervision for the structure model.

  • Edges are used because they provide a general structural representation across various scenes.
  • The framework is also applicable to segmentation maps and depth maps as alternative structural representations.
  • Segmentation and depth ground truths are extracted from normal-light data with DPT and used to train the structure model.

5. Conclusion

The paper introduces joint structure modeling and appearance enhancement, using restored structure maps to guide appearance reconstruction. Its experiments demonstrate effectiveness across sRGB and RAW domains, while extremely dark areas remain a limitation without larger-scale training.

  • The framework jointly models structure and appearance, using restored structure maps to enhance appearance modeling results.
  • Structure modeling uses a modified generative model with structural features and GAN-loss training.
  • Appearance enhancement uses a structure-guided module with structure-guided feature synthesis layers.
  • Extensive experiments on sRGB and RAW domains demonstrate the framework's effectiveness.
  • Extremely dark areas can produce structure-generation artifacts with existing low-light datasets, although large-scale training can alleviate this issue.
Loading 2305.05839v1…