Source-linked AI summary
Iterative Prompt Learning for Unsupervised Backlit Image Enhancement
Zhexin Liang, Chongyi Li, Shangchen Zhou, Ruicheng Feng, Chen Change Loy
TL;DR
Backlit enhancement requires handling uneven illumination without paired training data, while directly using CLIP is difficult because accurate prompts are hard to specify. CLIP-LIT learns positive and negative prompts, trains an enhancement network with CLIP similarities, and iteratively refines both components; it outperforms state-of-the-art methods in visual quality and generalization.
Problem
Backlit enhancement must preserve well-lit regions while correcting underexposed areas, but existing methods may over-enhance bright regions or rely on restrictive assumptions.
Method
CLIP-LIT learns positive and negative prompts in CLIP’s latent space, trains an enhancement network with CLIP-aware constraints, and alternates prompt refinement with model fine-tuning.
Results
CLIP-LIT outperforms state-of-the-art methods in qualitative and quantitative metrics without paired training data.
Takeaways & Limitations
The method generates visually pleasing results with improved color and luminance, restored texture details, and no visible artifacts in challenging backlit regions.
Abstract
from arXiv · showhide
We propose a novel unsupervised backlit image enhancement method, abbreviated as CLIP-LIT, by exploring the potential of Contrastive Language-Image Pre-Training (CLIP) for pixel-level image enhancement. We show that the open-world CLIP prior not only aids in distinguishing between backlit and well-lit images, but also in perceiving heterogeneous regions with different luminance, facilitating the optimization of the enhancement network. Unlike high-level and image manipulation tasks, directly applying CLIP to enhancement tasks is non-trivial, owing to the difficulty in finding accurate prompts. To solve this issue, we devise a prompt learning framework that first learns an initial prompt pair by constraining the text-image similarity between the prompt (negative/positive sample) and the corresponding image (backlit image/well-lit image) in the CLIP latent space. Then, we train the enhancement network based on the text-image similarity between the enhanced result and the initial prompt pair. To further improve the accuracy of the initial prompt pair, we iteratively fine-tune the prompt learning framework to reduce the distribution gaps between the backlit images, enhanced results, and well-lit images via rank learning, boosting the enhancement performance. Our method alternates between updating the prompt learning framework and enhancement network until visually pleasing results are achieved. Extensive experiments demonstrate that our method outperforms state-of-the-art methods in terms of visual quality and generalization ability, without requiring any paired data.
1. Introduction
Backlit enhancement must preserve well-lit regions while correcting underexposed areas, a challenge existing supervised and unsupervised methods handle imperfectly. CLIP-LIT addresses this with learnable prompts, CLIP-guided enhancement, and iterative prompt refinement, achieving strong qualitative and quantitative results without paired data.
- Motivation: Backlit images have imbalanced illuminance, and manual correction must preserve well-lit regions while enhancing underexposed regions.These difficulties also affect visual quality and subsequent perception algorithms.
- Motivation: Existing supervised methods may process bright and dark areas similarly, causing over-enhancement in well-lit regions or under-enhancement in low-light regions.Unsupervised methods instead rely on assumptions such as average luminance and gray-world models, or adversarially learned reference-image distributions.
- Motivation: CLIP can distinguish well-lit from backlit images, but directly applying it to enhancement is non-trivial because prompt wording strongly affects similarity scores.The best wording may vary with complex illumination, motivating learnable positive and negative prompts.
- Method: CLIP-LIT initializes prompts by aligning backlit and well-lit images with learnable negative and positive text prompts in CLIP’s latent space.The initialized prompts then provide text-image similarity constraints for training the enhancement network.
- Method: Prompt refinement iteratively uses rank learning to distinguish backlit images, enhanced results, and well-lit images while alternating updates to prompts and the enhancement network.The authors report superiority over state-of-the-art methods in qualitative and quantitative metrics without paired training data.
2. Related Work
Related work spans backlit enhancement, low-light enhancement, and exposure correction, using segmentation, fusion, learned curves, Retinex models, and supervised or adversarial learning. CLIP-LIT differs by using CLIP prompt learning to extract low-level illumination representations rather than focusing on high-level semantics.
- Backlit Image Enhancement: Backlit enhancement methods use region segmentation, learning-based restoration, image fusion, or image-specific parametric S-curves.ExCNet is described as learning an S-curve to correct ill-exposed images.
- Light Enhancement: Related low-light and exposure-correction methods include Retinex-based restoration, supervised learning with paired data, and unsupervised adversarial approaches.These approaches connect backlit enhancement with broader light-enhancement tasks.
- Vision-Language Priors: CLIP’s visual-language prior has demonstrated zero-shot classification and image-quality or abstract-perception assessment from large-scale image-text data.These properties motivate applying CLIP to backlit image enhancement.
- Prompt Learning: Existing prompt-learning methods target high-level vision tasks, whereas CLIP-LIT uses prompt learning to extract low-level representations such as color, exposure, and saturation.The approach explicitly ignores high-level semantic information for this enhancement task.
3. Methodology
CLIP-LIT learns prompts that distinguish backlit from well-lit images, then uses them to train and iteratively refine an enhancement network. Prompt refinement incorporates current and previous enhanced results through ranking constraints so outputs move toward well-lit representations while preserving content and structure.
- Iterative Learning: Including previous enhanced results in the ranking constraints encourages each new result to be closer to positive samples than the previous result while remaining distant from negative samples.The method preserves previous-round outputs so prompts focus more on light and color distributions than on high-level image content.
- Overview: The two-stage pipeline first learns negative/positive prompts for backlit/well-lit images, then alternates prompt refinement with enhancement-network tuning.The initial prompts guide an initial enhancement network; refined prompts are subsequently used for further network tuning.
- Prompt Initialization: Prompt initialization minimizes classification loss on CLIP text-image similarities, associating backlit inputs with negative prompts and well-lit images with positive prompts.The image and prompt representations are encoded in CLIP’s latent space, and binary cross-entropy trains the prompt pair.
- Initial Enhancement Network: The initial enhancement network estimates an illumination map and produces the enhanced image through It = Ib/Ii, trained with CLIP-Enhance and identity losses.The identity loss encourages content and structural similarity to the backlit input, while the CLIP-Enhance loss measures similarity to the learned prompts.
- Prompt Refinement: Prompt refinement uses margin ranking to move current enhanced results toward well-lit images and away from backlit inputs in CLIP space.The refinement stage is performed iteratively and alternates with enhancement-network tuning using the refined prompts.
- Experiments: The method reports visually pleasing enhancement with improved color and luminance, preserved content and structure, and strong image-quality metrics without paired training data.The reported visual comparisons describe reduced over- and under-exposure, restored texture details, and no introduced artifacts; user-study results favored the method on every image by over 60%.
– Supplementary Material –
The supplementary material adds ablations, training details, further discussions, comparisons, extension discussions, and a video demonstration.
- The supplement covers ablation studies, extra training details, further discussions, additional comparisons, and extension discussions.
- A video demonstration is provided to showcase the method’s effectiveness.
A. More Ablation Studies
Additional experiments compare CLIP-LIT with supervised and adversarial-loss alternatives and examine prompt initialization. The method produces brighter, more detailed results, while prompt initialization mainly affects convergence speed rather than final performance.
- A.1. Superiority in Generalization Ability: Our method produces clearer details, sufficient luminance, and more realistic color distribution than Unetpair on Backlit300, whose outputs remain under-exposed.The comparison evaluates generalization against Unetpair trained with paired BAID data.
- A.2. Prompt Initialization Learning: Random prompt initialization requires more iterations to converge, while word-based initialization has little effect on final performance or training.The conclusion is supported by both the initialization comparison and its quantitative table.
- A.3. Visual Comparisons with Adversarial Loss: CLIP-LIT produces results more consistent with the input and brighter in backlit areas than a model trained with adversarial loss.Other training settings were kept fixed in the comparison.
B. Extra Training Details
The supplementary training details describe relaxed prompt-learning constraints, alternating threshold control, and comparison protocols for supervised and unsupervised methods.
- Prompt-learning constraints: The theoretical setting m1 = 0 is relaxed to m1 = 0.2 because prompt-pair learning otherwise fails to converge well.
- Alternating training control: Prompt learning is frozen and enhancement training begins when its loss falls below ThrA, while ThrB controls the reverse transition.The thresholds are set to ThrA = 60 and ThrB = 90.
- Comparison protocol: Existing supervised models are compared without retraining because CLIP-LIT does not require paired data, while retrained unsupervised methods may perform worse than their original models.The stated reasons include disruption of global enhancement by uneven brightness and training-data requirements.
D. Comparisons with Retrained Supervised Methods
The paper retrains state-of-the-art supervised methods on the same paired subset for a fair comparison. CLIP-LIT outperforms these retrained methods on most metrics and yields sharper, brighter visual results.
- D. Comparisons with Retrained Supervised Methods: The supervised baselines are retrained on the same 380 randomly selected paired images.This establishes a common training-data basis for the comparison.
- D. Comparisons with Retrained Supervised Methods: CLIP-LIT outperforms the retrained supervised methods on most metrics in Table 7.
- D. Comparisons with Retrained Supervised Methods: The retrained supervised methods produce relatively blurred and under-exposed results, whereas CLIP-LIT achieves superior visual quality.The visual comparisons cover the BAID and Backlit300 datasets.
E.1. Visual Comparisons on the BAID Test Dataset
On the BAID test dataset, CLIP-LIT effectively enhances backlit images without over- or under-exposure and produces a more natural appearance than compared methods.
- CLIP-LIT enhances BAID backlit images without causing over- or under-exposure.
- Compared with other methods, CLIP-LIT produces the most natural appearance.
E.2. Visual Comparisons on The Backlit300 Dataset
On the Backlit300 dataset, CLIP-LIT preserves backlit details and well-lit backgrounds while producing favorable night-scene results with natural color contrast.
- CLIP-LIT preserves color and detail content in backlit areas most clearly and realistically.
- Its enhanced details have strong natural color contrast while well-lit backgrounds remain unchanged.
- CLIP-LIT yields the most visually favorable result in the night scene.
F. More Comparisons on More Datasets
Additional evaluation on five unseen test datasets shows that CLIP-LIT retains state-of-the-art performance and produces natural results without artifacts or exposure problems.
- Quantitative and visual comparisons evaluate CLIP-LIT against unsupervised methods on five unseen extra test datasets.
- CLIP-LIT still reaches SOTA performance on the five unseen test datasets.
- Compared with other unsupervised methods, CLIP-LIT produces the most natural results without artifacts or over- or under-exposure problems.
G. Discussions of failure cases and potential extensions
The method has demonstrated favorable visual behavior across varied scenes, but its scope is limited by missing information in extreme sRGB exposures and the absence of denoising capability; the authors suggest broader restoration applications and HDR extension.
- Failure cases and potential extensions: Extreme over- or under-exposed sRGB regions may remain unprocessable when limited bit depth causes information loss.
- Failure cases and potential extensions: CLIP-LIT cannot denoise because its training data does not contain noise.
- Failure cases and potential extensions: The authors identify low-light enhancement and deraining as possible additional image-restoration applications.
- Visual behavior across datasets: Figure comparisons report natural enhancement, preserved color and content, and favorable results across faces, landscapes, animals, architecture, and night scenes.
- Visual behavior across datasets: On BAID and Backlit300 examples, CLIP-LIT is reported to preserve well-lit areas while naturally enhancing dark or backlit regions.