Source-linked AI summary
Low-Light Image Enhancement with Wavelet-based Diffusion Models
Hai Jiang, Ao Luo, Songchen Han, Haoqiang Fan, Shuaicheng Liu
TL;DR
Low-light enhancement is challenging because diffusion restoration can be computationally expensive and unstable, while conventional methods face limited robustness or perceptual fidelity. DiffLL uses wavelet-based conditional diffusion, stable training, and high-frequency restoration to improve efficiency, consistency, and detail reconstruction. Experiments report state-of-the-art distortion and perceptual quality with notable speedups over prior diffusion-based methods, though the method remains limited for extreme low light and direct real-time video enhancement.
Problem
Diffusion-based low-light enhancement offers visual-quality benefits but suffers from high computation, long inference times, and content inconsistency during sampling.
Method
DiffLL applies conditional diffusion to wavelet average coefficients, trains with forward diffusion and denoising, and uses HFRM for high-frequency detail reconstruction.
Results
DiffLL outperforms existing state-of-the-art methods on distortion and perceptual metrics while achieving a speed exceeding 70× that of DDIM.
Takeaways & Limitations
The method provides robust, efficient low-light enhancement with stable sampling, fine-grained restoration, and potential practical value for low-light face detection.
Takeaways & Limitations
DiffLL is less effective in extremely low-light environments, is not yet efficient enough for direct low-light video enhancement, and is limited to LLIE generalization.
Abstract
from arXiv · showhide
Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. To address these issues, we propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL. Specifically, we present a wavelet-based conditional diffusion model (WCDM) that leverages the generative power of diffusion models to produce results with satisfactory perceptual fidelity. Additionally, it also takes advantage of the strengths of wavelet transformation to greatly accelerate inference and reduce computational resource usage without sacrificing information. To avoid chaotic content and diversity, we perform both forward diffusion and denoising in the training phase of WCDM, enabling the model to achieve stable denoising and reduce randomness during inference. Moreover, we further design a high-frequency restoration module (HFRM) that utilizes the vertical and horizontal details of the image to complement the diagonal information for better fine-grained restoration. Extensive experiments on publicly available real-world benchmarks demonstrate that our method outperforms the existing state-of-the-art methods both quantitatively and visually, and it achieves remarkable improvements in efficiency compared to previous diffusion-based methods. In addition, we empirically show that the application for low-light face detection also reveals the latent practical values of our method. Code is available at https://github.com/JianghaiSCU/Diffusion-Low-Light.
1 INTRODUCTION
DiffLL addresses low-light enhancement challenges by combining diffusion models with wavelet transformation, stable training, and high-frequency restoration. It reports state-of-the-art distortion and perceptual quality with substantially improved efficiency over prior diffusion-based methods.
- Motivation: Low-light enhancement is ill-posed, and weak illumination impairs visual quality and downstream vision tasks.Existing traditional methods depend heavily on handcrafted priors, while learning-based methods face perceptual-fidelity trade-offs.
- Motivation: Diffusion-based restoration can provide strong visual quality but requires substantial computation and long inference times.On 600×400 LOLv1 images, diffusion-based methods take more than 10 seconds to restore an image.
- Contributions: DiffLL introduces WCDM, which performs diffusion on wavelet average coefficients to reduce spatial dimensions and computational cost.The framework also trains with both forward diffusion and denoising to improve stable, content-consistent inference.
- Results: DiffLL achieves state-of-the-art distortion and perceptual quality while offering notable efficiency improvements over previous diffusion-based methods.The reported contributions include a speed exceeding 70× that of DDIM.
- Contributions: DiffLL uses HFRM to reconstruct local details by combining vertical and horizontal information with diagonal wavelet details.This module complements the diffusion-based restoration of global information.
3 METHOD
DiffLL combines 2D-DWT with conditional diffusion to restore low-light images in a lower-dimensional wavelet representation. Its pipeline also reconstructs high-frequency details through HFRM to improve restoration efficiency and quality.
- Pipeline: DiffLL first converts the low-light input into the wavelet domain through K successive 2D-DWT operations.The pipeline produces an average coefficient used by the diffusion model.
- Wavelet-based Conditional Diffusion: WCDM performs diffusion on the wavelet average coefficient rather than the original image or latent space.This design leverages wavelet transformation to reduce the spatial dimension processed by diffusion.
- High-Frequency Restoration: HFRM reconstructs high-frequency coefficients to restore local image details and improve overall quality.The module complements the diffusion-based restoration of the wavelet average coefficient.
3.1 Discrete Wavelet Transformation
The discrete wavelet transformation decomposes an image into one average coefficient and three directional high-frequency coefficient sets. DiffLL prioritizes restoring the average coefficient while reconstructing local details separately.
- Coefficient Decomposition: 2D-DWT with Haar wavelets decomposes a low-light image into average, vertical, horizontal, and diagonal sub-bands.The transformation is applied recursively to the average coefficient for additional spatial reduction.
- Coefficient Roles: The average coefficient represents global image information, whereas the high-frequency coefficients contain sparse local details.This separation motivates treating global illumination and fine-grained structure with different modules.
- Coefficient Roles: Exchanging the average coefficient changes global illumination, while exchanging high-frequency coefficients approximately preserves image content.Mean-error comparisons indicate that the average coefficient contains richer information than the high-frequency coefficients.
- Restoration Strategy: DiffLL restores the average coefficient with diffusion and reconstructs the remaining high-frequency coefficients using HFRM.Repeated transformations yield a 4^K spatial-dimension reduction for the diffusion model.
3.2 Conventional Conditional Diffusion Models
Conventional conditional diffusion models progressively add Gaussian noise and learn reverse denoising transitions conditioned on an input. Their objective trains a network to predict noise during this iterative restoration process.
- Forward Diffusion: The forward diffusion process progressively transforms input x_0 into Gaussian noise over T steps using a fixed variance schedule.The schedule consists of β_1 through β_T, with x_T sampled from N(0,I).
- Denoising: The inverse process starts from randomly sampled Gaussian noise and gradually denoises it into a restored result.The learned reverse transitions are modeled as Gaussian denoising steps.
- Conditional Modeling: Conditional diffusion learns a denoising process conditioned on the input while leaving the forward diffusion process unchanged.This conditioning aims to maintain fidelity to the distribution associated with the input.
- Training Objective: The training objective optimizes a noise-prediction network using the squared error between sampled and predicted noise.The network receives the noisy state, conditional input, and diffusion timestep.
3.3 Wavelet-based Conditional Diffusion Models
WCDM performs diffusion on wavelet-domain average coefficients to reduce spatial cost while preserving information, and trains both diffusion directions to stabilize inference and maintain content consistency.
- Wavelet-based Conditional Diffusion Models: The conventional diffusion process requires costly inference because it typically uses T=1000 forward steps with small variance β_t.The small variance supports a near-Gaussian denoising assumption but increases computational demands.
- Wavelet-based Conditional Diffusion Models: WCDM converts low-light images into wavelet coefficients and performs diffusion on the average coefficient to reduce spatial dimensions efficiently.The wavelet representation separates global information from sparse high-frequency details.
- Wavelet-based Conditional Diffusion Models: During training, WCDM performs forward diffusion and denoising, then minimizes the L2 distance between restored and reference coefficients to enforce content consistency.The sampled noisy coefficient is denoised conditionally, and the restored average coefficient is compared with the training reference.
- Wavelet-based Conditional Diffusion Models: Wavelet transformation preserves all image information without the information loss and added parameters associated with VAE-based latent representations.The paper contrasts wavelets' linear operation with VAE encoding for spatial dimension reduction.
3.4 High-Frequency Restoration Module
HFRM reconstructs sparse high-frequency wavelet coefficients by combining vertical and horizontal information with diagonal details, then applies efficient local and long-range feature processing.
- High-Frequency Restoration Module: HFRM reconstructs wavelet high-frequency coefficients that encode vertical, horizontal, and diagonal image details.These coefficients provide sparse representations of fine image structure.
- High-Frequency Restoration Module: Three depth-wise separable convolutions efficiently extract features from the input coefficients.The module uses separable convolutions at both feature extraction and channel reduction stages.
- High-Frequency Restoration Module: Two cross-attention layers use vertical and horizontal information to complement diagonal details during reconstruction.The design specifically leverages V and H information to restore D details.
- High-Frequency Restoration Module: A progressive dilation Resblock supports local restoration and a larger receptive field while avoiding gridding effects through gradually changing dilation rates.The first and last convolutions extract local information, while middle dilated convolutions use longer-range context.
3.5 Network Training
Network training combines diffusion, detail-preservation, and content losses, and evaluates the model on paired and unpaired low-light benchmarks using distortion and perceptual metrics.
- Network Training: The content loss combines L1 and SSIM losses to minimize differences between the restored and reference images.It compares the restored low-light image with the corresponding high-quality reference.
- Network Training: The total training objective combines the diffusion objective, detail-preservation loss, and content loss.This jointly optimizes diffusion behavior, high-frequency reconstruction, and image content similarity.
- Network Training: Training uses PyTorch on four RTX 2080Ti GPUs for 1 × 10^5 iterations, with Adam, an initial learning rate of 1 × 10^-4, batch size 12, and 256×256 patches.The wavelet transformation scale K is set to 2.
- Network Training: Evaluation uses paired datasets with PSNR, SSIM, LPIPS, and FID, while five unpaired datasets use NIQE, BRISQUE, and PI.The metric groups separately assess distortion, visual quality, and non-reference perceptual quality.
4.2 Comparison with Existing Methods
DiffLL is compared with optimization-based, learning-based, transformer, and diffusion-based methods across paired, high-resolution, and unpaired benchmarks, showing strong quantitative, qualitative, and efficiency results.
- Quantitative Comparison: On LOLv1, DiffLL improves PSNR and SSIM over the second-best SNRNet by 1.726dB and 0.003, respectively.On LOLv2-real, it reaches 28.857dB PSNR and 0.876 SSIM, exceeding the second-best approaches by 3.947dB and 0.018.
- Quantitative Comparison: DiffLL achieves the lowest LPIPS and FID scores on all three paired datasets, with FID below 50 on each dataset.The reported datasets are LOLv1, LOLv2-real, and LSRW.
- Efficiency Comparison: DiffLL averages 0.157s inference time and 1.850G GPU memory per 600×400 image on an RTX 2080Ti.The paper characterizes this as a moderate efficiency level with a favorable performance-efficiency trade-off.
- Unpaired and Qualitative Comparison: On unpaired datasets, DiffLL obtains the lowest NIQE scores on MEF, LIME, and NPE and generalizes to unseen scenes with restored illumination and details.Qualitative comparisons report fewer exposure, color, noise, and artifact problems than competing methods in several cases.
- Qualitative Comparison: Qualitative comparisons show DiffLL improving global and local contrast, reconstructing sharper details, and suppressing noise across paired test cases.Competing methods are reported to produce underexposure, overexposure, noise amplification, or artifacts in the compared examples.
4.3 Low-Light Face Detection
The study evaluates low-light image enhancement as preprocessing for face detection using the DARK FACE dataset, with real-world nighttime images split into training and testing subsets.
- The evaluation uses 10,000 DARK FACE images captured in real-world nighttime scenes.6,000 images are selected for training and the remainder for testing.
- Because test-set bounding-box labels are unavailable, the study uses the training set for evaluation.
4.4 Ablation Study
Ablation studies examine wavelet scale, sampling steps, high-frequency restoration, loss terms, and training strategy. The experiments identify trade-offs between efficiency and reconstruction quality while supporting the proposed design choices.
- Wavelet Transformation Scale: K=2 is selected as the default wavelet scale because larger scales reduce inference time but can degrade performance through reduced information richness.At scale K, the average-coefficient spatial dimension decreases by 4^K relative to the original image.
- Wavelet Transformation Scale: K=3 remains comparable to state-of-the-art methods, indicating that the wavelet-based diffusion model can restore higher-resolution images.
- Sampling Step: Sampling steps from 5 to 30 have no major performance impact, while larger steps increase inference time.Training with denoising helps the model learn denoising across different sampling-step counts.
- High-frequency Restoration Module: HFRM improves reconstruction, with gains of 2.142dB and 1.440dB in PSNR and 0.064 and 0.044 in SSIM at wavelet scales k=1 and k=2, respectively.The default HFRM outperforms variants that omit or reverse vertical-horizontal complementarity for diagonal details.
- Loss Function: Removing diffusion loss decreases all perceptual metrics, while the auxiliary loss term improves restored-image visual fidelity and overall performance.The content loss also produces noticeable improvements, especially in the reported perceptual metrics.
- Training Strategy: Adding denoising to training reduces content diversity and chaotic or distorted outputs while achieving the best performance and minimal variance.Forward diffusion alone produces diverse results across five evaluations, whereas the default strategy yields more consistent images.
4.5 Limitations
DiffLL is less effective in extremely low-light environments and is not yet efficient enough for direct low-light video enhancement. Its generalization is also limited to the LLIE task observed during training.
- The method performs less well in extremely low-light environments because those images contain greater information loss and are harder to restore.
- Unlike some real-time methods, DiffLL is not yet efficient enough for direct low-light video enhancement.
- The model has limited generalization capacity beyond the LLIE task observed during training.Effectiveness on other image restoration tasks remains future work.
5 CONCLUSIONS
DiffLL combines wavelet-based conditional diffusion, a consistency-oriented training strategy, and high-frequency restoration for low-light enhancement. On public benchmarks it outperforms competitors quantitatively and qualitatively while remaining computationally efficient, and face-detection results indicate practical value.
- DiffLL uses wavelet transformation with conditional diffusion to produce visually satisfactory enhancement while reducing inference time and computational resource consumption.
- Denoising during both training and inference enables stable sampling and consistent restored results.
- The high-frequency restoration module complements diagonal details with vertical and horizontal information for finer detail reconstruction.
- Experiments on public benchmarks show that DiffLL outperforms competitors quantitatively and qualitatively while being computationally efficient.
- Low-light face-detection results show the practical value of the enhancement method.