Source-linked AI summary
Semantic-Guided Zero-Shot Learning for Low-Light Image/Video Enhancement
Shen Zheng, Gaurav Gupta
TL;DR
Low-light images degrade visual quality and semantic information, while conventional and supervised approaches require impractical capture, annotation, or optimization conditions. The paper proposes a semantic-guided zero-shot network that extracts pixel-wise enhancement factors, progressively enhances images, and preserves semantics without paired or unpaired training data or segmentation labels. Extensive experiments report advantages in perceptual quality, efficiency, and high-level vision tasks, including low-light detection and segmentation.
Problem
Low-light images suffer under-exposure and noise that degrade perceptual quality and semantic information, while existing approaches can require paired data or impractical acquisition and optimization.
Method
SGZ combines lightweight enhancement-factor extraction, recurrent image enhancement, and unsupervised semantic segmentation without paired images, unpaired datasets, or segmentation labels.
Results
The method demonstrates excellence in perceptual quality, model efficiency, and benefits for high-level vision tasks, while processing 1000 images of size 1200 × 900 within 1 second on a single GPU.
Takeaways & Limitations
Semantic guidance can be integrated into zero-shot low-light enhancement while supporting efficient image/video processing and downstream detection and segmentation.
Takeaways & Limitations
Traditional enhancement methods remain difficult to tune because they depend on manually selected priors and hand-crafted regularization, with intricate optimization causing long inference times.
Abstract
from arXiv · showhide
Low-light images challenge both human perceptions and computer vision algorithms. It is crucial to make algorithms robust to enlighten low-light images for computational photography and computer vision applications such as real-time detection and segmentation. This paper proposes a semantic-guided zero-shot low-light enhancement network (SGZ) which is trained in the absence of paired images, unpaired datasets, and segmentation annotation. Firstly, we design an enhancement factor extraction network using depthwise separable convolution for an efficient estimate of the pixel-wise light deficiency of an low-light image. Secondly, we propose a recurrent image enhancement network to progressively enhance the low-light image with affordable model size. Finally, we introduce an unsupervised semantic segmentation network for preserving the semantic information during intensive enhancement. Extensive experiments on benchmark datasets and a low-light video demonstrate that our model outperforms the previous state-of-the-art. We further discuss the benefits of the proposed method for low-light detection and segmentation. Code is available at https://github.com/ShenZheng2000/Semantic-Guided-Low-Light-Image-Enhancement
1. Introduction
Low-light conditions degrade exposure, noise, contrast, and semantic information, while existing capture and enhancement approaches face practical limitations. The paper proposes a semantic-guided zero-shot framework combining adaptive enhancement-factor extraction, recurrent enhancement, and semantic guidance without paired images, unpaired datasets, or segmentation labels.
- Low-light images suffer under-exposure and high ISO noise, degrading perceptual quality and semantic information for vision tasks.
- Higher ISO, longer exposure, and manual software adjustment respectively introduce noise or blur, require artistic skill, or scale poorly.
- Traditional methods depend on manually selected priors and hand-crafted regularization, while intricate optimization makes them unsuitable for real-time tasks.
- The proposed framework uses EFE to estimate pixel-wise light deficiency, RIE to progressively enhance images, and USS to preserve semantic information without segmentation annotation.
- The method fuses high-level semantic information with low-level enhancement without paired images, unpaired datasets, or segmentation labels.
- A lightweight convolutional network extracts the enhancement factor, while recurrent enhancement and five non-reference losses target diverse lighting conditions.
- The model processes 1000 images of size 1200 × 900 within 1 second on a single GPU, supporting low-light video enhancement.
2. Related Work
Prior low-light enhancement work includes histogram, Retinex, supervised, unsupervised, and zero-shot approaches. The paper distinguishes its method by automatically learning enhancement requirements and using semantic information without segmentation labels.
- Histogram equalization and Retinex methods can produce perceptual improvements but rely on manually selected priors and hand-crafted regularization.
- The proposed model automatically extracts enhancement factors from low-light images and enables recurrent enhancement with linear complexity.
- Supervised deep methods require paired low- and normal-light images, whereas unsupervised methods avoid paired-image collection.
- Zero-shot learning removes the need for both paired images and unpaired datasets through non-reference enhancement objectives.
- Unlike other deep methods, the model uses a pretrained segmentation network without labels to preserve semantic information without significantly increasing computational complexity.
3. Proposed Method
The proposed method combines lightweight enhancement-factor extraction, recurrent progressive brightening, semantic preservation, and non-reference losses for low-light enhancement.
- Enhancement Factor Extraction Network: The EFE network estimates pixel-wise light deficiency using depthwise separable convolutions and outputs an enhancement factor through Tanh activation.It uses symmetric skip connections and avoids batch normalization and up/downsampling to preserve spatial coherence.
- Recurrent Image Enhancement Network: RIE progressively enhances images by feeding each stage’s output and the enhancement factor into the next recurrence.The method selects Order 2 because Order 1 is insensitive to the enhancement factor, while Orders 3 and 4 can over-brighten images.
- Unsupervised Semantic Segmentation Network: The USS network preserves semantic information by segmenting enhanced images with frozen bottom-up and top-down feature pathways.The bottom-up pathway extracts features, while the top-down pathway converts semantic features into higher-resolution spatial representations.
- Loss Functions: The training objective combines spatial, RGB, brightness, total variation, and semantic losses without paired-image content or perceptual losses.The total loss weights Lspa, Lrgb, Lbri, and Ltv by 1, while Lsem is weighted by 0.1.
- Loss Functions: Spatial consistency preserves neighboring-region differences using adjacent and non-adjacent neighbors, with non-adjacent relationships weighted by α = 0.5.The local-region side length is set to A = 4.
- Loss Functions: Brightness and total variation losses constrain exposure, reduce noise, and improve smoothness while accounting for inter-channel color relations.The target exposure level is E = 0.60, and total variation uses horizontal and vertical gradients across channels.
4. Experiments
Experiments evaluate SGZ across enhancement quality, efficiency, ablations, and downstream detection, segmentation, and video settings. The results show strong benchmark performance, efficient inference, and improved high-level vision outcomes, while failure cases remain under strong motion blur and mirror reflection.
- Evaluation setup: The evaluation compares SGZ with traditional, supervised, unsupervised, and zero-shot enhancement methods across multiple low-light datasets.Comparisons include NPE, LIME, MEF, DICM, VV, LOL, DarkBDD, and DarkCityScape using reference and no-reference quality metrics.
- Ablation study: Removing Lrgb, Lbri, or Ltv causes severe color deviation, weak low-light enhancement, or unnatural artifacts, while removing Lspa or Lsem corrupts facial details and regional contrast.The ablation study reports that Lbri, Lrgb, and Ltv strongly influence enhancement, whereas Lspa and Lsem have smaller overall impacts but preserve local details and contrast.
- Ablation study: The proposed A and E settings perform best, and Gaussian USS initialization outperforms VOC pretraining on the larger DarkCityScape dataset.The paper attributes the initialization difference to possible data bias and concludes that Gaussian initialization is sufficient for promising results.
- Image enhancement results: SGZ achieves the best average UNIQUE and second-best average BRISQUE across NPE, LIME, MEF, DICM, and VV, while ranking second on LOL and first on DarkCityScape.The results identify DarkCityScape as the more challenging extreme low-light dataset.
- Efficiency and video: 0.001 second per image enables processing 1000 images within 1 second, while fewer FLOPs and small parameter counts support low-light video enhancement and mobile deployment.Efficiency measurements use 1200 × 900 images and a single NVIDIA GeForce RTX 2080 Ti GPU.
- Downstream vision tasks: SGZ helps detect the greatest number of cars and achieves the best mIOU and mPA in semantic segmentation comparisons.The visual segmentation comparison places SGZ closest to ground truth, while other methods introduce incorrect regions, holes, blur, noise, or artifacts.
- Failure cases: The proposed model cannot address strong motion blur or mirror reflection in its reported failure cases.These limitations are shown in the failure-case visualization.
5. Conclusion
The proposed semantic-guided zero-shot network enhances low-light images without paired images, unpaired datasets, or segmentation labels, while improving perceptual quality, efficiency, and high-level vision tasks.
- The network is trainable without paired images, unpaired datasets, or segmentation labels.
- Its enhancement factor extraction, recurrent enhancement, and unsupervised semantic segmentation components support the proposed training-free-data framework.
- Extensive experiments demonstrate excellence in perceptual quality, model efficiency, and benefits for high-level vision tasks.
- Future work will investigate motion blur removal and detection-driven enhancement for low-light image and video enhancement.