PaperlayerView exact revision on arXiv

Source-linked AI summary

LEDNet: Joint Low-light Enhancement and Deblurring in the Dark

Shangchen Zhou, Chongyi Li, Chen Change Loy

arXiv:2202.03373v2eess.IVcs.CV

TL;DR

Night photography combines low light and motion blur, but existing independent methods and datasets do not adequately support their joint restoration. The paper synthesizes realistic degradations into LOL-Blur and trains LEDNet, a unified enhancement-deblurring network. It reports superior results on synthetic and real-world datasets and presents the dataset and model as a foundation for the task.

  • Problem

    Low-light enhancement and deblurring are usually treated independently, while paired data capturing their coexistence is difficult to collect.

  • Method

    The paper builds a synthesis pipeline with realistic low-light and saturated-region blur modeling, then proposes LEDNet with an enhancement encoder, deblurring decoder, and adaptive skip connections.

  • Results

    The method achieves superior results to prior arts on both synthetic and real-world datasets.

  • Takeaways & Limitations

    LOL-Blur and LEDNet provide a foundation for further exploration of low-light enhancement and deblurring in the dark.

  • Takeaways & Limitations

    Existing blur synthesis is inaccurate in saturated regions because sRGB clipping damages the underlying intensities used by the blur model.

Abstract

from arXiv · show

Night photography typically suffers from both low light and blurring issues due to the dim environment and the common use of long exposure. While existing light enhancement and deblurring methods could deal with each problem individually, a cascade of such methods cannot work harmoniously to cope well with joint degradation of visibility and textures. Training an end-to-end network is also infeasible as no paired data is available to characterize the coexistence of low light and blurs. We address the problem by introducing a novel data synthesis pipeline that models realistic low-light blurring degradations. With the pipeline, we present the first large-scale dataset for joint low-light enhancement and deblurring. The dataset, LOL-Blur, contains 12,000 low-blur/normal-sharp pairs with diverse darkness and motion blurs in different scenarios. We further present an effective network, named LEDNet, to perform joint low-light enhancement and deblurring. Our network is unique as it is specially designed to consider the synergy between the two inter-connected tasks. Both the proposed dataset and network provide a foundation for this challenging joint task. Extensive experiments demonstrate the effectiveness of our method on both synthetic and real-world datasets.

1 Introduction

Night images commonly combine low visibility, color distortion, and motion blur, while independently trained enhancement and deblurring methods fail to address these degradations together. LEDNet introduces a synthesis pipeline, LOL-Blur dataset, and unified network for joint restoration.

  • Motivation: Long exposure in night photography causes motion blur, while insufficient light still produces low visibility and distorted color.Handheld shooting limits the usable shutter speed, so low light and blur typically coexist.
  • Motivation: Independent enhancement and deblurring methods can enlarge saturated-region blur, remove blur cues, or fail on nighttime images.Enhancement may overexpose or oversmooth, while deblurring models trained on daytime data cannot directly handle dark scenes.
  • Data challenge: Collecting paired low-light blurry and normal-light sharp images is difficult because existing enhancement and deblurring datasets use incompatible capture processes.Camera shake and dynamic environments also introduce geometric and photometric misalignment during acquisition.
  • Proposed data: 12,000 low-blur/normal-sharp pairs form the LOL-Blur dataset, with diverse darkness and saturated and unsaturated blurs.The dataset is produced by a synthesis pipeline designed to model realistic low-light blurring degradation.
  • Proposed model: LEDNet uses an enhancement-specialized encoder, deblurring decoder, and adaptive skip connections to pass light-enhanced features for blur removal.The network is designed to treat low-light enhancement and deblurring within one context rather than as a forced cascade.

2 Related Work

Related work includes CNN-based and optimization-based deblurring, unsupervised low-light enhancement, and synthetic data construction. The paper addresses limitations in modeling saturated-region blur and natural low-light appearance.

  • Image Deblurring: CNN-based deblurring methods evolved from blur-kernel estimation toward end-to-end kernel-free networks, often using multi-scale or hierarchical structures.Training datasets enabled the dominance of end-to-end approaches for dynamic-scene deblurring.
  • Image Deblurring: Optimization-based low-light deblurring methods rely on light streaks or specially process saturated regions, but light-streak methods fail when suitable sources or kernel sizes are unavailable.Saturated-region handling can reduce artifacts around those regions.
  • Low-light Enhancement: Unsupervised enhancement methods use adversarial losses or image-specific curve estimation, and exposure-conditioned Zero-DCE can generate low-light images with diverse darkness.The paper uses this curve-adjustment framework for low-light simulation.
  • Synthesis limitations: Simple averaging of sharp sequences weakens saturated-region blurs because clipped sRGB intensities damage the blur model.The paper reverses clipping before blur synthesis to recover more realistic saturated blurs.

3 LOL-Blur Dataset

The LOL-Blur dataset is generated through a synthesis pipeline designed to model joint low-light and blur degradation, especially saturated-region blurs. It contains 12,000 paired examples derived from high-frame-rate video sequences.

  • Dataset construction: 12,000 paired data are generated from 170 training videos and 30 testing videos, each containing 60 frames.The videos are recorded at 250 fps and processed into sequence clips.
  • Dataset construction: The pipeline combines low-light simulation with blur synthesis to produce low-light blurry and normal-light sharp pairs.The mid-frame of each 7- or 9-frame sharp clip serves as the ground-truth image.
  • Low-light simulation: Gamma-based low-light simulation can introduce noticeable warm color deviations, whereas EC-Zero-DCE is used to produce more natural low-light images.The comparison is described against prior Gamma-correction approaches.
  • Blur simulation: Standard frame averaging is inaccurate for saturated pixels because sRGB clipping damages the latent intensities used for blur synthesis.This limitation is particularly relevant to saturated regions in dark blurry images.
  • Blur simulation: The proposed clipping-reversal blur simulation recovers clipped saturated intensities and generates more realistic saturated blurs than commonly used GoPro-style simulation.The modified simulation also improves handling of unsaturated and saturated blurs.

4 LEDNet

LEDNet treats low-light enhancement and deblurring as a joint non-blind restoration problem with spatially varying degradations. Its encoder enhances illumination, while adaptive connections transfer enhanced features to a deblurring decoder.

  • Architecture: LEDNet maps low-light blurry inputs to normal-light sharp outputs with an encoder-decoder architecture and filter adaptive skip connections.The architecture is designed for mixed visibility and texture degradations that vary spatially.
  • Low-light Enhancement Encoder: The LE-Encoder uses residual blocks, downsampling, pyramid pooling, and CurveNLU to embed inputs into normal-light feature space.An intermediate enhanced image is predicted at the smallest scale for supervision.
  • Low-light Enhancement Encoder: PPM aggregates hierarchical global context from pooling branches with bin sizes 1, 2, 3, and 6 to suppress artifacts caused by blur and noise.The module is inserted into the low-light enhancement encoder.
  • Low-light Enhancement Encoder: CurveNLU applies learnable, position-dependent curve transformations to feature intensities using concave-down increasing curves.Its parameters lie in [0, 1], and the same curve is applied across feature channels.
  • Deblurring Decoder: The D-Decoder contains convolutional blocks with residual upsampling and FAC layers, while FASC supplies dynamic filters for spatially varying blur removal.The decoder receives enhanced features from the LE-Encoder through adaptive connections.
  • Training losses: The overall objective combines enhancement and deblurring L1-plus-perceptual losses as L = λ_en L_en + λ_deb L_deb.The reported weights are λ_per = 0.01, λ_en = 0.8, and λ_deb = 1.

5 Experiments

Experiments evaluate LEDNet on synthetic and real-world night-blur data, including quantitative benchmarks, visual comparisons, and ablations of the synthesis pipeline and network modules. The reported comparisons show sharper, more perceptually pleasing outputs and support the roles of the proposed components.

  • Experimental settings: LEDNet is evaluated on LOL-Blur using 10,200 training pairs and 1,800 test pairs, with PSNR, SSIM, and LPIPS as quantitative metrics.Training uses 170 sequences and testing uses 30 sequences; Table 1 reports runtime and parameter counts alongside quality metrics.
  • LOL-Blur evaluation: On LOL-Blur, LEDNet produces much sharper and visually pleasing images compared with the evaluated baseline approaches.The visual comparisons assess joint enhancement and deblurring quality on the synthetic dataset.
  • Real-world evaluation: On a real-world night-blurred image, LEDNet achieves the best perceptual quality with more stable enhancement and better deblurring, while other methods retain saturated-region blur and artifacts.The comparison highlights failures associated with saturated regions and artifacts in competing approaches.
  • Real-world evaluation: The real test set contains 240 captured low-light blurry images and 160 night blurry images from RealBlur, evaluated with MUSIQ, NRQM, and NIQE.These perceptual metrics are used because real test images lack ground-truth images.
  • Data-synthesis ablation: Ablations show that LOL-Simulation yields more natural results with less noise and color distortion, while Clipping Reverse improves robustness to saturated-region blur.The synthesis pipeline is evaluated through alternative training datasets and blur-generation choices.
  • Real-world evaluation: Noise simulation through CycleISP helps LEDNet handle real-world noises well, and the real-data results strongly suggest the effectiveness of the dataset and network.The evidence concerns qualitative and perceptual evaluation on real images without ground truth.
  • Network ablation: Removing PPM introduces artifacts and degrades performance, while CurveNLU adapts feature enhancement spatially and FASC connections improve performance over simple concatenation.The ablations also identify enhancement loss as necessary for performance and show that spatially varying operations suit different saturated and unsaturated blur models.

6 Conclusion

The paper concludes that realistic low-light blur synthesis enables the LOL-Blur dataset, while LEDNet jointly enhances illumination and removes blur in one forward pass. It presents the dataset and model as a foundation for further work on low-light enhancement and deblurring in the dark.

  • 6 Conclusion: The synthesis pipeline models realistic low-light blurring and produces the large-scale, diverse LOL-Blur paired dataset.The dataset supports learning and benchmarking the joint task.
  • 6 Conclusion: LEDNet performs illumination enhancement and blur removal in a single forward pass.The conclusion also highlights PPM, CurveNLU, and FASC as components supporting stability, robustness, and deblurring.
  • 6 Conclusion: The dataset and network offer a foundation for further exploration of low-light enhancement and deblurring in the dark.

Supplementary Material

The supplementary material provides additional architecture details, LEDNet analyses, data-synthesis discussions, and visual comparisons on synthetic and real-world images.

  • Supplementary Material: Section A presents architecture details for modules in LEDNet.
  • Supplementary Material: Section B provides further analyses and results for CurveNLU and the loss function.
  • Supplementary Material: Section C analyzes the data-synthesis pipeline and LOL-Blur dataset, followed by additional visual comparisons on LOL-Blur and real-world images.

A Architecture Details

The supplementary architecture details describe the Residual Downsample/Upsample modules and Pyramid Pooling Module used in LEDNet, retaining their original configurations. Additional ablation experiments examine CurveNLU, PPM, and enhancement loss.

  • A Architecture Details: LEDNet adopts existing Residual Downsample/Upsample modules and a Pyramid Pooling Module with their original configurations.The supplementary figures provide detailed structures for both modules.
  • A Architecture Details: Additional ablation experiments analyze the effects of CurveNLU, PPM, and enhancement loss in LEDNet.

B.1 Analysis on CurveNLU

CurveNLU estimates different nonlinear adjustments for unsaturated and saturated regions, brightening dark areas while preserving saturated intensities. Using n = 3 balances performance and computational complexity.

  • Curve parameter behavior: CurveNLU estimates distinct curve parameters for unsaturated and saturated regions, producing stronger intensity increases in dark unsaturated areas and near-zero curvature in saturated regions.The contrasting curves help brighten dark features while avoiding overexposure in saturated regions.
  • Curve order: n = 3 is selected as a trade-off between computational complexity and performance because higher curve orders provide only slight PSNR/SSIM gains.LEDNet with CurveNLU substantially outperforms the n = 0 baseline without CurveNLU.

B.2 Effectiveness of Enhancement Loss

The enhancement loss is necessary for effective training, because removing it substantially degrades visual quality and produces severe artifacts and unsmooth regions.

  • Enhancement loss: Removing the enhancement loss Len significantly harms visual quality, producing severe artifacts and unsmooth regions.The comparison is shown in Fig. 13, consistent with the main-manuscript ablation results.

C More Discussions on LOL-Blur Dataset

The dataset pipeline uses EC-Zero-DCE and CycleISP to synthesize realistic low-light images across exposure levels while modeling spatially varying luminance and realistic dark-image noise.

  • Low-light simulation: EC-Zero-DCE generates more natural and realistic low-light images than Gamma correction, which introduces noticeable warm-toned color deviation.The comparison evaluates the low-light image synthesis pipeline against a prior correction method.
  • Low-light simulation: EC-Zero-DCE produces diverse darkness levels for different exposures and retains saturated-pixel intensities through spatially varying adjustment.This behavior simulates the light effects observed in real-world night images.
  • Noise simulation: CycleISP generates more natural and realistic dark-image noise than commonly used Gaussian and Poisson noise models.The noise is simulated in the RAW domain.

C.3 Luminance Distribution of Datasets

LOL-Blur is evaluated through luminance-distribution analysis and extensive qualitative comparisons on synthetic and real-world night blurry images. LEDNet handles diverse scenarios and blur patterns in these visual evaluations.

  • Luminance distributions: The RealBlur ground-truth images are concentrated at low intensities, making that dataset unsuitable for training a light-enhancement network.REDS contains many sunny scenes and therefore cannot be used directly to generate low-light images.
  • Qualitative comparisons: LEDNet produces sharper and visually pleasing results on the LOL-Blur dataset compared with the listed baseline methods.The visual comparisons include cascaded methods and networks trained with LOL-Blur.
  • Qualitative comparisons: On real-world night blurry images, LEDNet shows more stable light enhancement and better deblurring, especially in saturated regions where other methods leave large blurs or artifacts.The reported comparisons cover multiple real-world examples and identify saturated-region behavior as a key distinction.
  • Qualitative comparisons: LEDNet handles blur patterns of different shapes and performs well across different scenarios on the RealBlur dataset.Light streaks in the inputs reveal the varied blur patterns used in the evaluation.
Loading 2202.03373v2…