PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Deep Context-Sensitive Decomposition for Low-Light Image Enhancement

Long Ma, Risheng Liu, Jiaao Zhang, Xin Fan, Zhongxuan Luo

arXiv:2112.05147v1eess.IVcs.CV

TL;DR

Low-light enhancement seeks to recover useful image information that is hidden in dark scenes, while existing decomposition networks can overlook scene-level context. This paper connects illumination and reflectance estimation with a context-sensitive decomposition architecture and illumination guidance. Across seven benchmarks, the authors report favorable visual and quantitative comparisons, and describe lightweight variants with nearly the same performance as CSDNet.

  • Problem

    Low-light images hide object shape and chrominance, while existing decomposition networks can omit scene-level context and traditional methods can yield exposure, detail, color, artifact, or noise problems.

    Source: §I · pp. 1–2

  • Method

    The method links illumination and reflectance estimation with a context-sensitive decomposition connection and spatially varying illumination guidance, using paired CSDNet or unpaired CSDGAN training.

    Source: A. Our Contributions · pp. 2–3

  • Results

    Evaluations across seven benchmarks report desirable visual quality and the best quantitative scores against state-of-the-art approaches for CSDNet and CSDGAN.

    Source: A. Our Contributions · p. 3

  • Takeaways & Limitations

    LiteCSDNet and encoder-sharing SLiteCSDNet are presented for efficiency, with almost the same performance as CSDNet and more than 98.5% fewer parameters for SLiteCSDNet than existing state-of-the-art methods.

    Source: A. Our Contributions · p. 3

Abstract

from arXiv · show

Enhancing the quality of low-light images plays a very important role in many image processing and multimedia applications. In recent years, a variety of deep learning techniques have been developed to address this challenging task. A typical framework is to simultaneously estimate the illumination and reflectance, but they disregard the scene-level contextual information encapsulated in feature spaces, causing many unfavorable outcomes, e.g., details loss, color unsaturation, artifacts, and so on. To address these issues, we develop a new context-sensitive decomposition network architecture to exploit the scene-level contextual dependencies on spatial scales. More concretely, we build a two-stream estimation mechanism including reflectance and illumination estimation network. We design a novel context-sensitive decomposition connection to bridge the two-stream mechanism by incorporating the physical principle. The spatially-varying illumination guidance is further constructed for achieving the edge-aware smoothness property of the illumination component. According to different training patterns, we construct CSDNet (paired supervision) and CSDGAN (unpaired supervision) to fully evaluate our designed architecture. We test our method on seven testing benchmarks to conduct plenty of analytical and evaluated experiments. Thanks to our designed context-sensitive decomposition connection, we successfully realized excellent enhanced results, which fully indicates our superiority against existing state-of-the-art approaches. Finally, considering the practical needs for high-efficiency, we develop a lightweight CSDNet (named LiteCSDNet) by reducing the number of channels. Further, by sharing an encoder for these two components, we obtain a more lightweight version (SLiteCSDNet for short). SLiteCSDNet just contains 0.0301M parameters but achieves the almost same performance as CSDNet.

I. INTRODUCTION

Low-light enhancement must recover image information hidden in dark scenes, yet existing approaches can produce unnatural exposure, color, detail, and noise. The paper proposes a context-sensitive, two-stream decomposition architecture and lightweight variants, reporting favorable evaluations across multiple benchmarks.

I. INTRODUCTION

  • Low-light conditions hide object shape and chrominance, which can hinder further operations; demand for clear, bright images is rising with portable-device use.The introduction identifies night scenes as one challenging lighting condition.Source: §I · p. 1
  • Prior Retinex-based methods and deep networks can produce exposure, detail, color, artifact, or noise problems; representative networks also use a redundant route that omits scene-level context.The paper contrasts their initialization, optimization, and exposure-suppression stages with a direct low-light-to-normal-exposure mapping.Source: Figure 1 · p. 1, §I · p. 2

A. Our Contributions

  • A two-stream estimator links illumination and reflectance through a context-sensitive connection, with spatially varying illumination guidance for edge-aware smoothness.The connection incorporates physical principles and contextual dependencies across spatial scales.Source: A. Our Contributions · pp. 2–3
  • Evaluations of paired CSDNet and unpaired CSDGAN across seven benchmarks report desirable visual quality and the best quantitative scores against state-of-the-art approaches.The reported visual qualities include sufficient detail, vivid colors, and fewer noises.Source: A. Our Contributions · p. 3
  • By reducing CSDNet’s channel count and sharing an encoder for its two components, LiteCSDNet and SLiteCSDNet achieve almost the same performance as CSDNet; SLiteCSDNet uses 0.0301M parameters, over 98.5% fewer than existing state-of-the-art methods.Source: A. Our Contributions · p. 3

II. RELATED WORKS

Earlier low-light enhancement methods span direct value mapping and Retinex-based optimization, but both face limitations in exposure, detail, color, or consistency across challenging scenes.

A. Value-based Mapping

  • Value-based methods such as histogram equalization and S-curve techniques are simple exposure adjustments, but can produce unnatural exposure, inadequate detail, distorted color, artifacts, or uneven exposure.HE variants target natural exposure or detail preservation, while the cited S-curve methods still struggle with uneven exposure.Source: A. Value-based Mapping · p. 3

B. Model-based Optimization

  • Retinex-based optimization models low-light images through reflectance and illumination, but reported methods exhibit ghosting, under- or over-exposure, insufficient structure, or over-smoothing.The cited limitations are associated with the methods' priors, constraints, or modeling procedures.Source: B. Model-based Optimization · p. 4
  • Model-based methods have restricted capacity and do not achieve consistent performance in some challenging scenarios.Source: B. Model-based Optimization · p. 4

C. Data-driven Deep Learning

Data-driven low-light enhancement has advanced with available training datasets, but existing deep methods leave spatial-scale context underused. The proposed approach uses a two-stream architecture and context-sensitive connection to directly map low-light inputs to enhanced results.

C. Data-driven Deep Learning

  • Existing data-driven methods ignore contextual information across spatial scales and apply physical principles at the image output, often following an under-exposure-to-over-exposure-to-normal-exposure route.Paired datasets helped advance deep networks for low-light enhancement, while other methods use synthetic or unpaired training approaches.Source: C. Data-driven Deep Learning · pp. 4–5

III. THE PROPOSED APPROACH

  • The proposed approach combines two-stream estimation, a context-sensitive connection, and spatially-varying illumination guidance, with CSDNet and CSDGAN corresponding to different training manners.Source: §III · p. 5

A. Two-Stream Estimation Mechanism

  • IENet and RENet simultaneously estimate illumination and reflectance, providing an explicit mapping that avoids the redundant intermediate-exposure pipeline described for prior methods.The networks are instantiated with identical standard U-Nets; the low-light input's grayscale image is used for illumination estimation.Source: A. Two-Stream Estimation Mechanism · p. 5

B. Context-Sensitive Decomposition Connection

The context-sensitive decomposition connection bridges reflectance and illumination feature streams by incorporating the Retinex physical principle into feature-space processing. The paper reports that it improves brightness, suppresses artifacts, and highlights structure.

B. Context-Sensitive Decomposition Connection

  • The connection is applied to decoder features that combine copied encoder features with decoder-generated features, and the final enhanced result is the updated reflectance.Source: B. Context-Sensitive Decomposition… · p. 5
  • The authors report that the connection improves brightness, suppresses artifacts, and highlights structural information, especially at the output terminal.Source: B. Context-Sensitive Decomposition… · pp. 5–6
  • The connection aggregates reflectance and illumination context in feature maps, incorporating the physical principle to characterize the network's inner structure across spatial scales.The paper contrasts this feature-level design with approaches that focus on the image domain.Source: B. Context-Sensitive Decomposition… · pp. 5–6

C. Spatially-Varying Illumination Guidance

The spatially-varying illumination guidance addresses structural and edge deficiencies in estimated illumination without relying on computationally expensive complex regularization. It is constructed from neighborhood-based input differences and is reported to present edge and texture information.

C. Spatially-Varying Illumination Guidance

  • Direct illumination estimation can produce artifacts and inexact exposure when structural and edge presentation is unsatisfactory; complex prior regularization can improve it but has high computational cost.Source: C. Spatially-Varying Illumination… · p. 6
  • The guidance is intended to provide edge-aware structural illumination and stabilize the context-sensitive decomposition connection; its visualization is described as presenting effective edge and texture information.Source: C. Spatially-Varying Illumination… · p. 6
  • The illumination guidance enlarges input pixel values using neighborhood regions, then takes the difference from the original input to obtain edge and texture information.The authors state that its computation is more efficient than similar maps in existing works,.Source: C. Spatially-Varying Illumination… · p. 6

D. CSDNet

CSDNet is trained on paired datasets with losses for the enhanced output, perceptual similarity, and smoothness of the illumination estimate.

D. CSDNet

  • The smooth L1 loss applies a piecewise penalty to pixel differences, quadratic when |u| ≤ 1 and linear otherwise.N is the number of pixels.Source: D. CSDNet · p. 6
  • CSDNet training combines MSE, perceptual, and smooth L1 losses.The MSE loss constrains the final enhanced output, the perceptual loss targets perceptual similarity, and smooth regularization constrains IENet's output.Source: D. CSDNet · p. 6

E. CSDGAN

CSDGAN evaluates the proposed architecture with unpaired supervision, using a discriminator as a training strategy to address the limitation of paired datasets in real-world scenarios.

E. CSDGAN

  • The supplied visuals cover MIT-Adobe FiveK examples, component comparisons, and quantitative PSNR/SSIM comparisons on LOL.The captions identify visual comparisons in FiveK and different components; Table II is captioned as a quantitative LOL comparison.Source: Figure 6 · p. 7, Figure 7 · p. 7, Table II · p. 8
  • Its adversarial loss provides supervision by sampling fake and real distributions, including global images and local patches.The discriminator uses a global-local design; CSDGAN also adds a residual connection between RENet's input and output.Source: E. CSDGAN · p. 7
  • Unlike CSDNet, without ground truth, the method uses perceptual loss to ensure perceptual similarity between the low-light input and the final output.The settings are i = 5 and j = 1.Source: E. CSDGAN · p. 7
  • CSDGAN uses the discriminator as a training strategy for evaluating the architecture under unpaired supervision, given paired datasets' real-world limitation.The architecture is the generator; the discriminator is not presented as the core architectural contribution.Source: E. CSDGAN · pp. 6, 8

IV. EXPERIMENTAL RESULTS

The experimental section evaluates CSDNet and CSDGAN, analyzes the methods, and presents lightweight CSDNet variants.

IV. EXPERIMENTAL RESULTS

  • The experiments evaluate the proposed CSDNet and CSDGAN, followed by extensive analyses of the methods.Source: §IV · p. 8
  • The section also presents two lightweight versions of CSDNet to address practical needs.Source: §IV · p. 8
  • Experiments ran on a PC with an Intel Core i7-8700 CPU, 32 GB RAM, and an NVIDIA GeForce GTX 1070 8GB GPU.Source: §IV · p. 8

A. Implementation Details

Implementation covers paired and unpaired evaluation across multiple benchmarks, with metrics and visual comparisons selected for the datasets.

A. Implementation Details

  • Comparisons include model-based, paired end-to-end, and unpaired end-to-end methods, generally matched to the same paired or unpaired training mechanism.Source: A. Implementation Details · p. 8
  • CSDNet is trained with paired supervision on MIT-Adobe FiveK and LOL; FiveK has 5,000 raw images with five expert-retouched references, using expert-C as labels.FiveK is presented with visual comparisons only, while LOL is evaluated visually and with PSNR and SSIM.Source: A. Implementation Details · pp. 8–9
  • The visual comparisons include NPE examples, unpaired networks, component comparisons, and challenging ExDark examples; LOL figures report PSNR/SSIM beneath images.The LOL denoising comparison contrasts images without denoising against methods followed by CBDNet, except KinD, which already includes denoising.Source: Figure 10 · p. 9, Figure 11 · p. 9, Figure 12 · p. 9
  • CSDGAN is trained on EnlightenGAN's unpaired dataset and tested on NPE, NASA, MEF, and LIME, with NIQE as the evaluation metric.NIQE is lower-is-better; ExDark provides an additional challenging example.Source: A. Implementation Details · p. 9

B. Performance Evaluation of CSDNet

CSDNet is evaluated qualitatively across paired and unpaired settings, including standard benchmarks and challenging low-light cases. The reported comparisons favor the proposed methods in exposure, color, structure, and noise handling.

MIT-Adobe FiveK Dataset

  • CSDNet outperformed the compared methods on MIT-Adobe FiveK, with saturated colors and clear visual presentation; the authors report using 500 training pairs versus DeepUPE’s 4,500.The comparison included LIME, WhiteBox, and DeepUPE; the authors also describe more precise illumination estimates than DeepUPE.Source: B. Performance Evaluation of CSDNet · p. 10

LOL Dataset

Four Standard Datasets

  • On four standard datasets, CSDGAN achieved the lowest average NIQE and the best reported visual expression, with clear structure and color.The comparisons also describe under-exposure, over-exposure, and structural-detail issues in other methods.Source: C. Performance Evaluation of CSDGAN · p. 11

Analytical Evaluations

  • Across additional visual comparisons, CSDGAN retained better exposure and expression than RetinexNet and DeepUPE, while EnlightenGAN showed color distortion and artifacts.The comparisons also report structural-edge and exposure shortcomings in DeepUPE.Source: C. Performance Evaluation of CSDGAN · p. 11

More Challenging Cases in ExDark Dataset

  • On challenging ExDark cases, CSDGAN achieved the best reported exposure and visual performance amid instability in other end-to-end methods.The cited issues in other methods included under-exposure, block artifacts, and noise.Source: C. Performance Evaluation of CSDGAN · p. 11

D. Effects of Context-Sensitive Decomposition Connection

The ablations compare context-sensitive decomposition connections across architectures and show advantages over commonly used connections. They also examine how connection placement affects visual results and feature maps.

Three Versions of CSDNet

  • Among three CSDNet connection variants, CSDNetc produced the best color and structural expression when Eq. (2) was applied at both upsample and addition layers.CSDNeta applied Eq. (2) only at upsample layers, while CSDNetb applied it only at addition layers.Source: D. Effects of Context-Sensitive… · p. 12
  • The context-sensitive decomposition mechanism strengthened structural edges and textures while removing most artifacts and noise in the examined feature maps.The maps were from the 32nd channel after the decoder’s third upsample layer.Source: D. Effects of Context-Sensitive… · p. 12

Ablation Study

  • The context-sensitive connection consistently outperformed the commonly used connection across three architectures on the LOL test images.The comparison contrasts S3, S7, and S11 with S1, S5, and S9; Fig. 17 is the visual comparison of cases in Table IV.Source: D. Effects of Context-Sensitive… · pp. 12–13

E. Ablation Study on Illumination Guidance

Illumination guidance improves scores across tested architectures, while a residual connection ameliorates visual problems in CSDGAN. Lightweight variants reduce parameters while retaining nearly the same visual expression as CSDNet.

E. Ablation Study on Illumination Guidance

  • Illumination guidance significantly improved numerical scores across architectures and connection types, with visual benefits including artifact removal or increased brightness.Its placement varied by architecture: it was added at the last convolution layer, at the illumination decoder input, or as in the proposed framework.Source: E. Ablation Study on Illumination… · p. 13

F. Effects of Residual Connection in CSDGAN

  • Adding a residual connection between input and output ameliorated color distortion, over-exposure, and detail loss in CSDGAN, supporting its stability.Without the connection, the authors report these visual drawbacks as inevitable.Source: F. Effects of Residual Connection in… · p. 13

G. LiteCSDNet, SLiteCSDNet — Lightweight CSDNet

  • SLiteCSDNet used 0.00301M parameters—half LiteCSDNet’s—and both lightweight versions’ visual performance nearly matched CSDNet’s; all versions scored significantly above other works on 100 randomly sampled MIT-Adobe FiveK test images.Source: G. LiteCSDNet, SLiteCSDNet —… · p. 13

V. CONCLUSIONS AND FUTURE WORK

  • The conclusion reports that experiments on seven datasets showed superiority over existing state-of-the-art approaches, and that two lightweight versions target high efficiency and accuracy.The paper also proposes exploring the approach in denoising, dehazing, and deraining in future work.Source: §V · pp. 13–14

Related papers

Loading 2112.05147v1…