PaperlayerView exact revision on arXiv

Source-linked AI summary

Low-Light Image and Video Enhancement Using Deep Learning: A Survey

Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, Chen Change Loy

arXiv:2104.10729v3cs.CV

TL;DR

Low-light images can impair visual quality and information available to high-level tasks, while diverse mobile-phone data for testing LLIE generalization were lacking. This paper surveys LLIE, introduces a mobile-phone image and video dataset and comparison platform, and finds that method performance varies across datasets, metrics, and tasks, with no method always winning.

  • Problem

    Low-light images compromise aesthetic quality and information transmission for high-level tasks, and diverse mobile-phone data for evaluating LLIE generalization were lacking.

    Source: §1 · pp. 1–2

  • Method

    The paper surveys LLIE methods and open issues, introduces a diverse mobile-phone image and video dataset, and provides an online platform for comparing popular methods.

    Source: §1 · p. 2

  • Results

    Method performance varies significantly across test datasets and evaluation metrics, with different methods performing better for image quality, real-world phone images and videos, efficiency, and face detection.

    Source: §4.6 · pp. 14–15

  • Takeaways & Limitations

    Deep learning-based LLIE methods benefit face detection in the dark, but detection accuracy remains extremely low compared with normal-light images.

    Source: §4.6 · p. 16

    Designing LLIE loss functions that account for image characteristics remains an ongoing research challenge.

    Source: §3.4 · p. 7

Abstract

from arXiv · show

Low-light image enhancement (LLIE) aims at improving the perception or interpretability of an image captured in an environment with poor illumination. Recent advances in this area are dominated by deep learning-based solutions, where many learning strategies, network structures, loss functions, training data, etc. have been employed. In this paper, we provide a comprehensive survey to cover various aspects ranging from algorithm taxonomy to open issues. To examine the generalization of existing methods, we propose a low-light image and video dataset, in which the images and videos are taken by different mobile phones' cameras under diverse illumination conditions. Besides, for the first time, we provide a unified online platform that covers many popular LLIE methods, of which the results can be produced through a user-friendly web interface. In addition to qualitative and quantitative evaluation of existing methods on publicly available and our proposed datasets, we also validate their performance in face detection in the dark.This survey together with the proposed dataset and online platform could serve as a reference source for future study and promote the development of this research field. The proposed platform and dataset as well as the collected methods, datasets, and evaluation metrics are publicly available and will be regularly updated.

1 INTRODUCTION

This survey examines deep learning methods for low-light image and video enhancement and introduces a diverse-phone dataset and online platform to support evaluation and further research.

1 INTRODUCTION

  • Low-light enhancement supports surveillance, autonomous driving, and computational photography, while smartphone capture is constrained by small apertures, real-time processing, and limited memory.These constraints make smartphone photography in dim environments especially challenging.Source: §1 · p. 1
  • Retinex model-based methods,,,,,,, can lose detail or distort color, retain or amplify noise, produce prior-related artifacts, and require lengthy optimization.These limitations arise because the reflection component is treated as the enhanced result and noise is usually ignored.Source: §1 · p. 1
  • Images captured in poor lighting can have buried scene content, reduced contrast, boosted noise, and inaccurate color, impairing visual quality and information for high-level tasks.The affected tasks include object tracking, recognition, and detection.Source: Figure 1 · p. 1, §1 · p. 1
  • From 2017 to 2021, more than 100 deep-learning LLIE papers appeared, using supervised, reinforcement, unsupervised, zero-shot, and semi-supervised learning.Figure 2 presents representative methods across these learning strategies.Source: §1 · p. 2, Figure 2 · p. 2
  • The survey provides an in-depth review of deep-learning low-light image and video enhancement across learning strategies, architectures, losses, datasets, metrics, model sizes, speed, and performance.It focuses on recent deep-learning advances, unlike Wang et al., whose review mainly covers conventional methods.Source: §1 · p. 2
  • The authors introduce a diverse-phone image and video dataset for testing generalization and an online platform for running popular methods on user images without a GPU.The dataset covers varied illumination conditions, and the platform provides results through a user-friendly web interface.Source: §1 · p. 2

2 DEEP LEARNING-BASED LLIE

The section formulates LLIE as learning a trainable network that enhances a low-light image by minimizing loss against ground truth. It surveys five learning strategies, their representative methods, and their data and generalization challenges.

2.1 Problem Definition

  • LLIE maps a low-light image through a trainable network to an enhanced result, with parameters optimized to minimize loss against ground truth.The survey notes that supervised and unsupervised losses can be used.Source: §2.1 · pp. 2–3

2.2 Learning Strategies

  • Supervised methods include end-to-end networks such as LLNet, Retinex-based methods such as Retinex-Net, and realistic-data approaches trained on captured or synthesized data.Retinex methods separately enhance illumination and reflectance; the survey reports better enhancement than direct end-to-end learning in most cases,.Source: §2.2 · p. 3
  • Real-data efforts include SID raw-image pairs, static-video DRV, dynamic-scene SMOID, and SIDGAN synthesis of dynamic raw-to-RGB video data.The cited passage notes DRV contains static videos and SMOID has 179 video pairs.Source: §2.2 · pp. 3–4
  • DeepExposure learns local exposure sequentially with reinforcement learning and fuses retouched results, while EnlightenGAN uses adversarial losses and self-feature preservation without paired training data.EnlightenGAN uses an attention-guided U-Net generator and global-local discriminators.Source: §2.2 · p. 4
  • Zero-shot methods learn enhancement from test images without paired or unpaired training data, including curve-based Zero-DCE and Retinex decomposition methods such as RRDNet and RetinexDIP.Zero-DCE uses image-specific curves and non-reference losses, enabling lightweight image-to-curve mapping; ExCNet estimates an S-curve for back-lit restoration.Source: §2.2 · p. 4
  • Semi-supervised DRBN recovers enhanced-image bands under supervision, then recomposes them through unsupervised adversarial learning; an LSTM-based extension reports better enhancement.The survey also categorizes methods as supervised, reinforcement, unsupervised, zero-shot, and semi-supervised, with supervised methods further divided into end-to-end, Retinex-based, and realistic-data-driven approaches.Source: §2.2 · pp. 3–4
  • 73% of deep learning-based LLIE methods use supervised learning, which benefits from paired data but faces limited real-world coverage and generalization challenges.Other strategies also face hurdles: unsupervised and semi-supervised training stability and color deviations, reinforcement-learning reward design, and zero-shot loss design.Source: §2.2 · pp. 4–5, Figure 3 · p. 5

3 TECHNICAL REVIEW AND DISCUSSION

The section surveys representative deep learning-based LLIE methods and discusses their network designs, use of Retinex theory, data formats, losses, datasets, and evaluation practices.

3 TECHNICAL REVIEW AND DISCUSSION

  • The review summarizes representative deep learning-based LLIE methods in Table 1 and analyzes their technical characteristics.Table 1 summarizes essential characteristics of the methods, including whether they are Retinex-based and their number of trainable parameters.Source: §3 · p. 5, Table 1 · p. 6

3.1 Network Structure

  • U-Net-like structures are common because they integrate multiscale features, but vanishing gradients, noisy skip connections, and designs insufficiently tailored to low-light data remain concerns.The review also identifies pyramid, multi-stage, and frequency-decomposition networks among the structures used in LLIE.Source: §3.1 · p. 5

3.2 Combination of Deep Model and Retinex Theory

  • Almost one-third of methods combine deep networks with Retinex theory, but may retain reflectance-based assumptions and deep-network overfitting risks.These methods may estimate Retinex components with separate subnetworks or use an estimated illumination map to guide network learning.Source: §3.2 · p. 5

3.3 Data Format

  • RGB dominates LLIE, while raw-data models can recover detail and contrast, reduce noise, and improve brightness in extremely low-light images.The review suggests smooth transformation from raw patterns to RGB as a possible way to combine RGB convenience with raw-data enhancement quality.Source: §3.3 · p. 5

3.4 Loss Function

  • LLIE uses reconstruction, perceptual, smoothness, color, exposure, and adversarial losses; specially designed exposure loss motivates non-reference objectives.L1 preserves color and luminance, SSIM preserves structure and texture, perceptual loss compares feature representations, smoothness loss constrains noise or neighboring pixels, and adversarial loss encourages resemblance to reference images. Existing losses are often adapted from other vision tasks, and LLIE-specific metrics remain limited beyond exposure loss.Source: §3.4 · pp. 5, 7

3.5 Training Datasets

  • Paired training data include synthetic and real-world datasets, but synthesis assumptions and data limitations can impair generalization and pixel-wise training.Gamma correction can introduce artifacts and color deviations, while random-illumination synthesis avoids artifacts but assumes uniform local brightness and can neglect context. Examples include LOL’s 500 real-scene pairs, VE-LOL-L’s 2,500 pairs, SID’s 5,094 raw short-exposure images, and video datasets DRV and SMOID. Broader challenges include a synthetic-to-real gap, insufficient scale and diversity of real data, and input–ground-truth misalignment.Source: §3.5 · pp. 7–8

3.6 Testing Datasets

  • Testing often uses self-collected data because there is no acknowledged benchmark, and small, repetitive, or mismatched test sets can produce biased, unfair comparisons.High-level-task datasets include BBD-100K nighttime videos, ExDARK’s 7,363 low-light images, and DARK FACE’s 6,000 nighttime images.Source: §3.6 · p. 8

3.7 Evaluation Metrics

  • Evaluation combines image-quality metrics and application testing, but existing metrics often miss human perception, low-light-specific needs, video quality, or machine perception.PSNR and MSE are widely used but can misrepresent perceived quality; SSIM measures structural similarity, and lower MAE and LOE are better. The review notes that LOE is the main metric designed specifically for low-light images and that no metric evaluates low-light video enhancement.Source: §3.7 · pp. 8–9

4 BENCHMARKING AND EMPIRICAL ANALYSIS

This section introduces LLIV-Phone and LLIE-Platform and benchmarks enhancement methods across image and video datasets, computational efficiency, and face detection in the dark. It also discusses raw- and RGB-based methods, including their differing strengths and limitations.

4.1 A New Low-Light Image and Video Dataset

  • LLIV-Phone contains 120 videos and 45,148 images from 18 phone cameras, captured across diverse lighting and indoor and outdoor scenes.The challenging real-scene dataset is intended to test generalization and can also support unsupervised training or serve as a reference for synthesizing realistic low-light data.Source: §4.1 · p. 9

4.2 Online Evaluation Platform

  • LLIE-Platform addresses researchers’ software and hardware barriers by letting users run 14 popular deep learning-based enhancement methods through a web interface.The platform is intended to support running existing methods and developing new ones, with additional methods to be offered regularly.Source: §4.2 · pp. 9–10

4.3 Benchmarking Results

  • Supervised methods generally score better on LOL-test and MIT-Adobe FiveK-test, but no method leads on every metric or both datasets, and real-world LLIV-Phone images challenge most methods.MBLLEN leads all four reported metrics on MIT-Adobe FiveK-test, while LLNet leads MSE and PSNR on LOL-test; LLIV-Phone results show failures in brightness and noise handling, with artifacts or color deviations. The authors also caution that MIT-Adobe FiveK ground truths and relatively clean inputs may not suit LLIE evaluation.Source: §4.3 · pp. 12–13, Table 5 · p. 12
  • For raw-image enhancement, EEMEFN outperforms SID across tested patterns and metrics, but both methods remain far from ground truth, especially on APS-C X-Trans images.The visual comparison reports that both methods reduce noise, while EEMEFN’s more complex structure recovers brightness better than SID’s simple U-Net.Source: Table 6 · p. 12, §4.3 · p. 13
  • On real-world phone videos, TBEFN has the best temporal coherence by ALV, whereas ExCNet performs worst, with instability across consecutive frames.The authors caution that ALV alone is inadequate for evaluating low-light video enhancement. On phone images, non-reference IQA metrics can also conflict with visual quality: Retinex-Net scores best on PI and SPAQ despite visible artifacts and color deviations.Source: §4.3 · pp. 13–14, Table 7 · p. 13, Table 8 · p. 14

4.4 Computational Complexity

  • Zero-DCE is fastest because its lightweight network estimates curve parameters, while LLNet and KinD++ have extremely high FLOPs and ExCNet and RRDNet have long runtimes.The reported FLOPs reach 4124.177G for LLNet and 12238.026G for KinD++; ExCNet and RRDNet runtimes reflect time-consuming optimization.Source: §4.4 · p. 14, Table 9 · p. 15

4.5 Application-Based Evaluation

  • Deep learning-based enhancement improves face detection in the dark, with Retinex-Net, Zero-DCE, and TBEFN relatively robust, but accuracy remains limited and depends on the detector.Zero-DCE offers a balance between detection AP and perceptual quality, while Retinex-Net’s stronger AP results contain artifacts and unnatural textures; the evaluation uses a pretrained DSFD detector.Source: §4.5 · p. 14, Figure 10 · p. 14, Table 10 · p. 16

4.6 Discussion

  • Across tasks, no method always wins: performance depends on datasets and metrics, and visual quality may diverge from IQA scores; supervised methods often perform better but require paired data and greater resources.Zero-shot methods avoid paired or unpaired training data and may generalize better, but their quantitative performance is inferior. The authors also note that dark-scene face detection remains far below normal-light accuracy and that more comprehensive metrics are needed for video enhancement.Source: §4.6 · pp. 14–16
  • Raw-based LLIE usually recovers details, produces more vivid color, and reduces noise and artifacts better, but is sensor- and format-specific; RGB-based LLIE is more versatile yet struggles with low light and excessive noise.Raw data contain more information, including wider color gamut and higher dynamic range; RGB images are commonly the final imagery produced by mobile devices.Source: §4.6 · p. 16

5 OPEN ISSUES

The survey identifies six unresolved challenges for low-light image and video enhancement: generalization, unknown noise and artifacts, uneven illumination, semantic-region handling, and use of neighboring video frames.

Generalization Capability

  • LLIE models generalize poorly across datasets, and combining real and synthetic training data does not resolve this limitation.For example, a method trained on MIT-Adobe FiveK cannot effectively enhance LOL images.Source: §5 · p. 16

Removing Unknown Noises

  • Existing methods often fail to remove unknown real-camera noise and may amplify it, while training with Gaussian or Poisson noise does not ensure satisfactory real-world performance.Noise types in training data differ from real noise; removing unknown noise remains unsolved.Source: §5 · p. 16

Removing Unknown Artifacts

  • Unknown artifacts, including those from JPEG compression or editing, remain difficult for existing low-light image and video enhancement methods to suppress.Source: §5 · p. 16

Correcting Uneven Illumination

  • Methods may brighten both dark and normally lit or over-exposed regions, so enhancing dark regions while suppressing over-exposed areas remains unresolved.Night scenes can contain dark regions alongside normal-light or over-exposed light sources.Source: §5 · p. 16

Distinguishing Semantic Regions

  • Methods often ignore semantic regions, potentially turning black hair off-white; enhancement should target darkness caused by external illumination rather than semantic content.Source: §5 · p. 16

Using Neighbouring Frames

  • Video methods commonly process frames individually, leaving unresolved how neighboring frames can improve enhancement and speed.Well-lit regions in adjacent frames could help enhance the current frame, and estimated parameters could be reused to reduce estimation time.Source: §5 · p. 16

6 FUTURE RESEARCH DIRECTIONS

The section outlines future LLIE research directions, from learning strategies and model design to data, evaluation, generalization, video enhancement, and semantic guidance.

Effective Learning Strategies

  • Zero-shot learning could reduce reliance on paired data, with further work needed on zero-reference losses, deep priors, and optimization strategies.The paper also calls for clarifying the effectiveness of supervised and unsupervised learning in LLIE.Source: §6 · pp. 16–17

Specialized Network Structures

  • Develop LLIE-specific architectures beyond U-Net, balancing illumination, small pixel values, noise, color constancy, memory, and speed; explore efficient operations [95],, NAS,, and transformers [99], [100].The paper questions whether encoder-decoder structures are best suited to LLIE and also suggests leveraging local similarity.Source: §6 · p. 17

Loss Functions

  • Design loss functions better suited to LLIE, using research on neural networks that approximate human visual quality perception to inform their development,.Source: §6 · p. 17

Realistic Training Data / Standard Test Data

  • Build large, diverse real-world paired training sets or realistic synthetic data, alongside a standard image-and-video test benchmark with ground truths and challenging, varied scenes.Existing training data lack the authenticity, scale, and diversity of real low-light conditions; ad-hoc test partitions make fair comparisons difficult.Source: §6 · p. 17

Task-Specific Evaluation Metrics

  • Develop more accurate, task-specific evaluation metrics, especially for non-reference assessment, to address the limits of current image-quality metrics.Current metrics variously emphasize human visual quality or effects on high-level machine-vision tasks.Source: §6 · p. 17

Robust Generalization Capability

  • Improve generalization, since most methods fail on real-world test data due to factors including synthetic or small-scale training data, ineffective structures, and unrealistic assumptions.Source: §6 · p. 17

Extension to Low-Light Video Enhancement

  • Advance low-light video enhancement by reducing flicker, exploiting temporal information across neighboring frames, and speeding up enhancement.Directly applying existing LLIE methods to video often produces unsatisfactory results and flickering artifacts.Source: §6 · p. 17

Integrating Semantic Information

  • Integrate semantic priors to help networks distinguish regions during enhancement and avoid color shifts, such as black hair turning gray.The paper identifies semantic information as crucial to low-light enhancement.Source: §6 · p. 17

(Supplementary Material)

The supplementary material extends visual comparisons of low-light enhancement across image benchmarks and the proposed datasets. It also assesses video temporal coherence using luminance curves and variance values.

Video temporal coherence

  • Luminance curves for ten videos from different phones assess temporal coherence; smoother curves and smaller average luminance variance indicate better coherence, following Jiang and Zheng.The curves use average luminance values in bounding boxes across consecutive frames.Source: (Supplementary Material) · p. 21, Figure 15 · p. 35

Resources

  • The supplementary material releases the proposed dataset and provides an online platform for testing methods with user inputs.The dataset is available for download, and the platform accepts arbitrary inputs.Source: (Supplementary Material) · p. 21

Image visual comparisons

  • Figures 1–6 compare deep-learning enhancement results on LOL-test and MIT-Adobe FiveK-test images.The comparisons are shown in Figures 1–3 for LOL-test and Figures 4–6 for MIT-Adobe FiveK-test.Source: (Supplementary Material) · p. 21, Figure 1 · p. 22, Figure 4 · p. 25
  • Figures 7–9 show results from different deep learning-based low-light enhancement methods on LLIV-Phone-imgT images, and Figures 10–11 show such results on raw low-light images sampled from SID-test.Figure 10 shows raw Bayer-pattern images from the SID-test-Bayer test dataset; the inputs are amplified for visualization.Source: (Supplementary Material) · p. 21, Figure 10 · p. 31
  • Figures 12–14 show enhancement results and face-detection results for images from DARK FACE.The figure captions recommend zooming in to inspect face bounding boxes.Source: (Supplementary Material) · p. 21, Figure 12 · p. 32

Related papers

Loading 2104.10729v3…