PaperlayerView exact revision on arXiv

Source-linked AI summary

EnlightenGAN: Deep Light Enhancement without Paired Supervision

Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang

arXiv:1906.06972v2cs.CVeess.IV

TL;DR

Low-light enhancement commonly lacks paired low/normal-light images, making conventional supervised training difficult. EnlightenGAN uses an unpaired one-path GAN with global-local discrimination and input-based self-regularization, and the paper reports strong cross-metric performance and flexible real-world adaptation.

  • Problem

    Low-light enhancement lacks practical paired low/normal-light training images, while paired supervision is difficult to obtain in uncontrolled scenes.

  • Method

    EnlightenGAN learns an unpaired low-to-normal-light mapping with a one-path GAN, global-local discriminators, self feature preservation, and self-regularized attention.

  • Results

    Experiments report that EnlightenGAN outperforms multiple state-of-the-art approaches under subjective and objective metrics across various low-light datasets.

  • Takeaways & Limitations

    Unpaired training makes EnlightenGAN easily adaptable to real noisy low-light images from different domains without paired normal-light data.

Abstract

from arXiv · show

Deep learning-based methods have achieved remarkable success in image restoration and enhancement, but are they still competitive when there is a lack of paired training data? As one such example, this paper explores the low-light image enhancement problem, where in practice it is extremely challenging to simultaneously take a low-light and a normal-light photo of the same visual scene. We propose a highly effective unsupervised generative adversarial network, dubbed EnlightenGAN, that can be trained without low/normal-light image pairs, yet proves to generalize very well on various real-world test images. Instead of supervising the learning using ground truth data, we propose to regularize the unpaired training using the information extracted from the input itself, and benchmark a series of innovations for the low-light image enhancement problem, including a global-local discriminator structure, a self-regularized perceptual loss fusion, and attention mechanism. Through extensive experiments, our proposed approach outperforms recent methods under a variety of metrics in terms of visual quality and subjective user study. Thanks to the great flexibility brought by unpaired training, EnlightenGAN is demonstrated to be easily adaptable to enhancing real-world images from various domains. The code is available at \url{https://github.com/yueruchen/EnlightenGAN}

I. INTRODUCTION

EnlightenGAN addresses low-light enhancement without paired low/normal-light training images by combining unpaired GAN learning with self-regularization and global-local processing. The paper reports stronger visual and objective results, alongside flexible adaptation to real-world domains.

  • Motivation: Paired low-light and normal-light images are difficult to capture simultaneously, while synthetic corruptions can produce non-photo-realistic artifacts.These constraints make paired supervision impractical for uncontrolled low-light enhancement.
  • Approach: EnlightenGAN learns an unpaired mapping between low-light and normal-light image spaces using a lightweight one-path GAN without cycle consistency.This avoids reliance on exactly paired images and provides shorter training time than cycle-consistent prior approaches.
  • Approach: The method combines dual global-local discriminators, self-regularized feature preservation, and illumination-based attention to constrain training without ground-truth supervision.The discriminators address spatially varying illumination, while self-regularization uses information from the input image.
  • Benefits: Unpaired training removes dependence on paired data and supports larger image varieties from different domains, improving real-world generalization.The paper contrasts this flexibility with overfitting to specific data-generation protocols or imaging devices.
  • Evaluation: Comprehensive evaluations report superiority across visual quality, no-reference image-quality assessment, and human subjective surveys.The experiments compare EnlightenGAN with several state-of-the-art methods.

II. RELATED WORKS

Prior low-light enhancement research includes paired-data collection, HDR fusion, traditional image-processing methods, and supervised learning. EnlightenGAN differs by targeting postprocessing of a single low-light image with unpaired adversarial training rather than paired or raw-sensor workflows.

  • Paired Datasets: Status Quo: Real paired low/normal-light datasets are difficult to scale because capture requires fixed cameras and immobile scenes.The LOL dataset contains only 500 real-scene pairs under this setup.
  • Paired Datasets: Status Quo: Changing exposure time can introduce local over- or under-exposure artifacts in paired data collection.This further complicates obtaining reliable low-light and normal-light correspondences.
  • Related Approaches: HDR methods capture and align multiple images under different illumination conditions, but they are not designed to postprocess one single low-light image.Their multi-image acquisition setting differs from single-image enhancement.
  • Adversarial Learning: The EnlightenGAN architecture uses an attention-guided U-Net generator and dual discriminators to direct global and local information.Its training also uses self feature preservation to maintain textures and structures.
  • Adversarial Learning: Most restoration and enhancement GAN applications use paired training data, whereas EnlightenGAN learns an inter-domain mapping with unpaired data and no cycle-consistency requirement.This setting distinguishes it from cycle-consistent unsupervised translation methods and from raw-sensor enhancement models requiring paired images.

III. METHOD

EnlightenGAN combines global and local adversarial discrimination with a relativistic global discriminator to enhance spatially varying low-light regions. Its ablation design also includes local-patch discrimination and least-square adversarial losses.

  • Global-Local Discriminators: The generator uses an attention-guided U-Net with dual discriminators directing global and local information.A self feature preserving loss also guides training to maintain textures and structures.
  • Global-Local Discriminators: A vanilla image-level discriminator can miss local enhancement needs when illumination varies across the image.A small bright region in an otherwise dark image illustrates this limitation.
  • Global-Local Discriminators: The global-local structure combines an image-level discriminator with a local discriminator trained on randomly cropped real and enhanced patches.This is intended to make local patches of enhanced images resemble realistic normal-light patches.
  • Global-Local Discriminators: The global discriminator uses a relativistic structure that estimates whether real data are more realistic than fake data.The implementation replaces the sigmoid function with a least-square GAN loss.
  • Global-Local Discriminators: The local discriminator randomly crops five patches from each output and real image and applies the original least-square GAN adversarial loss.The global and local components therefore use different adversarial-loss formulations.

B. Self Feature Preserving Loss

The self feature preserving loss adapts perceptual supervision to unpaired enhancement by matching VGG features of each low-light input and its enhanced output. The loss is applied globally and to local patches, with normalization used for training stability.

  • Self Feature Preserving Loss: Unlike paired perceptual loss, self feature preserving loss constrains VGG-feature distance between a low-light input and its enhanced output.This self-regularizes content features without requiring a ground-truth normal-light image.
  • Self Feature Preserving Loss: The loss preserves image content features before and after enhancement rather than matching the output to a ground-truth image.Its motivation is the paper’s unpaired training setting.
  • Self Feature Preserving Loss: The formulation compares feature maps extracted from the low-light input and generator output using a pretrained VGG-16 model.The feature-map dimensions and VGG layer indices determine the averaging in the loss.
  • Self Feature Preserving Loss: Local patches from the input and output are regularized with a similarly defined local self feature preserving loss.This extends self feature preservation beyond the full image.
  • Self Feature Preserving Loss: Instance normalization is added after the VGG feature maps to stabilize training, and the overall objective combines the resulting losses.The paper describes this as part of EnlightenGAN’s overall training loss.

C. U-Net Generator Guided with Self-Regularized Attention

EnlightenGAN uses a U-Net generator with a self-regularized attention map derived directly from input illumination. The map emphasizes darker regions throughout the generator to support spatially adaptive enhancement.

  • U-Net Generator: The generator backbone is a U-Net that extracts multi-level features and uses multi-scale context to preserve textures and synthesize images.U-Net is selected for its established use in image restoration and enhancement.
  • Self-Regularized Attention: The attention mechanism uses the normalized illumination channel of the input RGB image to guide spatially varying enhancement.The design targets stronger enhancement in dark regions than in bright regions.
  • Self-Regularized Attention: The attention map is defined as 1−I and multiplied with intermediate feature maps and the output image after resizing.Because it comes from the input rather than supervision, the map is a form of self-regularization.
  • U-Net Generator: The attention-guided U-Net uses eight convolutional blocks, and bilinear upsampling followed by convolution replaces deconvolution to mitigate checkerboard artifacts.Each block contains two 3 × 3 convolutional layers with LeakyReLU and batch normalization.

A. Dataset and Implementation Details

The study trains on separate low-light and normal-light image collections without maintaining pairs, using standardized preprocessing and a two-stage optimization schedule. Training uses a lightweight one-path GAN implementation.

  • Dataset: The unpaired training set contains 914 low-light and 1016 normal-light images collected from released datasets and HDR sources.The images are assembled without any requirement to preserve correspondence between the two groups.
  • Dataset: Images of medium brightness are manually removed, then all photos are converted to PNG and resized to 600 × 400 pixels.Testing uses standard images from prior low-light enhancement datasets.
  • Implementation Details: EnlightenGAN trains for 100 epochs at learning rate 1e-4, followed by 100 epochs with the learning rate linearly decayed to 0.The optimizer is Adam with batch size 32.
  • Implementation Details: The complete training process takes 3 hours on 3 Nvidia 1080Ti GPUs.The paper attributes the shorter training time to the lightweight one-path GAN without cycle-consistency.

B. Ablation Study

The ablation study removes the local discriminator and self-regularized attention mechanism separately, showing that the full model better avoids local color distortion and under-exposure.

  • The study visualizes input attention maps as guides indicating which regions should receive more or less enhancement.
  • Removing the local discriminator or self-regularized attention produces local color distortion or under-exposure in several image regions.Affected regions include skies, roofs, blossoms, tree boundaries, and a T-shirt.
  • The full EnlightenGAN produces more realistic colors and visually pleasing results than the ablated variants.The comparison supports the effectiveness of the global-local discriminator design and self-regularized attention mechanism.

C. Comparison with State-of-the-Arts

EnlightenGAN is compared with several enhancement methods using visual comparisons and no-reference IQA. It preserves details while avoiding artifacts and achieves the best overall averaged NIQE.

  • The visual comparison uses original low-light images, competing outputs, and EnlightenGAN outputs arranged across columns in Fig. 4.
  • EnlightenGAN preserves texture details and enhances dark areas without the over-exposure artifacts seen in LIME.SRIE and NPE remain darker, while CycleGAN and RetinexNet provide less satisfactory brightness and naturalness.
  • Lower NIQE indicates better visual quality, and EnlightenGAN wins on three of five image sets while achieving the best overall averaged NIQE.The evaluation covers MEF, NPE, LIME, VV, and DICM, plus the whole testing set.

3) Human Subjective Evaluation:

A human study compares five enhancement methods on 23 images through pairwise judgments from nine subjects. EnlightenGAN receives the most favorable overall rankings, while domain adaptation further improves real-world results.

  • Human Subjective Evaluation: The study randomly selects 23 testing images and asks 9 subjects to compare outputs from five methods pairwise.The methods are LIME, RetinexNet, NPE, SRIE, and EnlightenGAN.
  • Human Subjective Evaluation: EnlightenGAN ranks first on 10 of 23 images, second on 8, and third on 5 in the subjective evaluation.
  • Human Subjective Evaluation: EnlightenGAN achieves the overall most favored human ranking, with an average ranking of 1.78 over 23 images.RetinexNet and LIME score poorly because of over-exposures and occasional noise amplification.
  • Adaptation on Real-World Images: On BBD-100k, EnlightenGAN-N produces the most visually pleasing results by balancing brightness with artifact and noise suppression.The domain-adapted model is trained using low-light images from the target dataset.

E. Pre-Processing for Improving Classification

The paper tests EnlightenGAN as preprocessing for low-light object recognition on ExDark using a fixed pretrained classifier. Enhancement improves both top-1 and top-5 accuracy.

  • The ExDark test images are enhanced by pretrained EnlightenGAN and then classified with an ImageNet-pretrained ResNet-50 without domain adaptation or joint training.The high-level task performance serves as a semantic-aware metric for enhancement results.
  • 23.94% top-1 and 40.92% top-5 accuracy follow EnlightenGAN preprocessing, compared with 22.02% and 39.46% before enhancement.
  • The accuracy improvement supplies side evidence that EnlightenGAN preserves semantic details in addition to producing visually pleasing results.
Loading 1906.06972v2…