Source-linked AI summary
Learning Semantic-Aware Knowledge Guidance for Low-Light Image Enhancement
Yuhui Wu, Chen Pan, Guoqing Wang, Yang Yang, Jiwei Wei, Chongyi Li, Heng Tao Shen
TL;DR
LLIE methods often enhance images globally without accounting for region semantics, risking deviations from original colors. SKF integrates semantic priors through feature interactions and guided losses, with experiments reporting improved performance across baselines, datasets, models, and scenes.
Problem
Many LLIE methods improve images globally and uniformly without using semantic information from different regions, which can cause deviations from regional original colors.
Method
SKF uses a pretrained segmentation network as a semantic knowledge bank and integrates its priors through semantic-aware embedding, semantic-guided color histogram loss, and semantic-guided adversarial loss.
Results
SKF achieves superior performance across six baseline methods, while LLFlow-L-SKF outperforms all competitors.
Takeaways & Limitations
SKF preserves color consistency and visual details while enabling semantic-aware enhancement across different LLIE models and scenes.
Abstract
from arXiv · showhide
Low-light image enhancement (LLIE) investigates how to improve illumination and produce normal-light images. The majority of existing methods improve low-light images via a global and uniform manner, without taking into account the semantic information of different regions. Without semantic priors, a network may easily deviate from a region's original color. To address this issue, we propose a novel semantic-aware knowledge-guided framework (SKF) that can assist a low-light enhancement model in learning rich and diverse priors encapsulated in a semantic segmentation model. We concentrate on incorporating semantic knowledge from three key aspects: a semantic-aware embedding module that wisely integrates semantic priors in feature representation space, a semantic-guided color histogram loss that preserves color consistency of various instances, and a semantic-guided adversarial loss that produces more natural textures by semantic priors. Our SKF is appealing in acting as a general framework in LLIE task. Extensive experiments show that models equipped with the SKF significantly outperform the baselines on multiple datasets and our SKF generalizes to different models and scenes well. The code is available at Semantic-Aware-Low-Light-Image-Enhancement.
1. Introduction
LLIE improves underexposed images, but many methods enhance regions globally and uniformly without using semantic information. SKF addresses this gap by using semantic priors at feature and loss levels to preserve regional color consistency and improve image quality.
- Motivation: Low-light images have poor visibility and can degrade downstream vision tasks, motivating LLIE to reveal buried details.Traditional approaches include histogram equalization and Retinex-based methods.
- Research gap: Deep LLIE methods generally outperform traditional approaches, but commonly enhance images globally without accounting for region-specific semantic information.The introduction identifies semantic information as crucial for enhancement.
- Research gap: Existing semantic-guided methods do not fully exploit segmentation-network knowledge, and their task-specific interactions limit broader LLIE use.The paper frames these limitations through questions about obtaining diverse semantic knowledge and understanding its contribution to image quality.
- Approach: A pretrained segmentation network serves as a semantic knowledge bank providing intermediate features and semantic maps as diverse priors for LLIE.These priors can refine image features and guide enhancement during training.
- Approach: The semantic-aware embedding module uses cross-modal interactions between semantic and image features in representation space.It computes similarity between reference and target features to connect heterogeneous representations.
- Approach: The semantic-guided color histogram loss uses local scene geometry with global color information, while the semantic-guided adversarial loss targets fake regions for more natural textures.Together, these losses address regional color consistency and visual quality beyond global enhancement.
2. Related Work
Prior LLIE research includes traditional histogram and Retinex methods, deep Retinex-based and end-to-end models, and semantic guidance at loss or feature levels. SKF is presented as a general framework that more fully integrates semantic information at both levels.
- Traditional methods: Traditional LLIE methods use histogram equalization to extend dynamic range or Retinex decomposition to separate reflection and illumination.Retinex-based methods require explicit priors, which are difficult to design across varied scenes.
- Learning-based methods: Deep learning LLIE methods include Retinex-based frameworks that decompose and enhance images and end-to-end frameworks that directly learn enhanced results.End-to-end designs incorporate concepts such as Laplacian pyramids, filters, normalization flows, wavelets, and adversarial learning.
- Semantic-guided methods: Semantic-guided methods are divided into loss-level and feature-level approaches.Loss-level methods add semantic objectives, whereas feature-level methods combine intermediate segmentation features with image features.
- Loss-level methods: Loss-level semantic guidance has been used as an additional constraint in image denoising, super-resolution, and LLIE, including semantic brightness consistency.These methods introduce semantic-aware losses alongside original task objectives.
- Feature-level methods: Feature-level semantic guidance extracts intermediate segmentation-network features and introduces them into image-feature representation space across several restoration tasks.Applications include image restoration, deraining, super-resolution, LLIE, and depth estimation.
- Positioning: SKF responds to insufficient interaction between semantic priors and original tasks by combining two semantic-guided losses with a semantic-aware embedding module.The paper positions this combination as a general framework for LLIE compared with prior semantic-guided methods.
3. Method
SKF integrates semantic priors from a segmentation network into LLIE through feature refinement, instance-level color guidance, and semantic adversarial supervision. The framework is designed to preserve regional color consistency while improving enhancement quality.
- Motivation and Overview: Semantic priors enable region-aware enhancement, helping preserve original hues while treating smooth and detail-rich regions differently.The paper motivates semantic guidance because globally uniform enhancement can deviate from a region’s original color.
- Problem Definition: SKF models semantic guidance with a frozen segmentation network that supplies segmentation results and multi-scale intermediate features to the enhancement network.The semantic prior M is produced by the pre-trained segmentation network and used while updating the enhancement network.
- Semantic-Aware Embedding Module: The SE module bridges heterogeneous segmentation and enhancement features through cross-modal similarity and semantic-aware attention, producing refined decoder features.It performs pixel-wise interaction between semantic and image feature maps, and its output feeds the next enhancement decoder layer.
- Semantic-Guided Color Histogram Loss: The SCH loss divides the enhanced result into semantic instance patches and adjusts each instance’s differentiable color histogram to retain local color information.Boundary pixels are excluded during histogram estimation to reduce effects from semantic misclassification.
- Semantic-Guided Adversarial Loss: The SA loss uses segmentation maps and refined patches to guide global and local discriminators toward regions likely to be fake.Its local component selects the worst candidate patch for updating discriminator and generator parameters, while real patches remain randomly cropped.
4. Experiments
Experiments evaluate SKF across paired and unpaired low-light datasets, multiple baselines, quantitative metrics, visual comparisons, and ablations. SKF consistently improves baseline performance, while component studies attribute gains to semantic priors rather than parameter increases.
- Quantitative Evaluation: SKF improves baseline performance on LOL and LOL-v2, averaging 1.750 dB and 1.611 dB PSNR gains, respectively.LLFlow-L-SKF reaches 26.798 dB on LOL and 28.451 dB on LOL-v2.
- Quantitative Evaluation: Across unpaired datasets, SKF averages a 0.519 NIQE gain across methods and datasets, with only three worse cases.RetinexNet-SKF performs best on MEF, while KinD++-SKF performs best on the other five datasets.
- Qualitative Evaluation: Visual comparisons show SKF reducing noise, color inconsistencies, white artifacts, halos, and unnatural details across LOL and LIME.The reported improvements include more consistent colors, naturalistic details, and reduced haloing around lamps.
- Ablation Study: Adding SCH loss and SE improves PSNR by 0.243 dB and 0.841 dB on average, while combining them yields a 1.741 dB gain over baseline.Adding SA loss can slightly reduce some full-reference metrics but improves NIQE by 0.292 on average.
- Ablation Study: SCH improves PSNR by 0.512 dB on average, while semantic-guided SA loss improves NIQE by 0.411 over baseline.The results associate SCH with color-consistency preservation and SA with identifying fake regions and producing natural images.
- Ablation Study: SKF improves PSNR by 1.272 dB on average over larger versions with similar parameter counts, supporting semantic priors as the source of improvement.The comparison uses HWMNet-SKF, LLFlow-S-SKF, and LLFlow-L-SKF against larger baseline versions.
5. Conclusion
SKF incorporates semantic priors into enhancement through SE, SCH loss, and SA loss to preserve color consistency and visual details. It outperforms baseline methods broadly, while improvements remain limited for unknown categories.
- SKF integrates semantic priors through the SE module, SCH loss, and SA loss to preserve color consistency and visual details.SE uses semantic feature representations, SCH provides regional color constraints, and SA combines global and local adversarial loss with semantic priors.
- SKF achieves superior performance across all six baseline methods, with LLFlow-L-SKF outperforming all competitors.
- SKF’s improvement is limited for unknown categories, motivating stronger identification of unknown instances by the semantic knowledge bank.The authors also identify applying SKF to other low-level vision tasks as future work.