Source-linked AI summary
Bilevel Layer-Positioning LoRA for Real Image Dehazing
Yan Zhang, Long Ma, Yuxin Feng, Zhe Huang, Fan Zhou, Zhuo Su
TL;DR
Real image dehazing remains difficult because synthetic-to-real transfer requires unsupervised guidance and costly full-model adaptation. The paper introduces CLIP-based H2C semantic alignment and BiLaLoRA, which jointly learns low-rank weights and injection layers; the method reports strong performance across diverse real-world scenes and benchmarks.
Problem
Real dehazing lacks effective unsupervised mechanisms for unlabeled scenes and requires costly full-model fine-tuning, limiting adaptation across diverse real-world conditions.
Method
H2C uses CLIP to align haze-to-clear semantic transformations, while BiLaLoRA jointly optimizes LoRA weights and adapter injection layers for targeted adaptation.
Results
BiLaLoRA adapts across architectures and domains, maintains strong performance under diverse and dense haze, and substantially improves cross-domain performance with minimal parameter overhead.
Takeaways & Limitations
The framework provides a flexible plug-and-play approach for efficient real-image dehazing adaptation across scenes and pretrained models.
Takeaways & Limitations
Future work is needed for other low-level vision tasks, more refined semantic guidance, and severe-degradation restoration settings.
Abstract
from arXiv · showhide
Learning-based real image dehazing methods have achieved notable progress, yet they still face adaptation challenges in diverse real haze scenes. These challenges mainly stem from the lack of effective unsupervised mechanisms for unlabeled data and the heavy cost of full model fine-tuning. To address these challenges, we propose the haze-to-clear text-directed loss that leverages CLIP's cross-modal capabilities to reformulate real image dehazing as a semantic alignment problem in latent space, thereby providing explicit unsupervised cross-modal guidance in the absence of reference images. Furthermore, we introduce the Bilevel Layer-positioning LoRA (BiLaLoRA) strategy, which learns both the LoRA parameters and automatically search the injection layers, enabling targeted adaptation of critical network layers. Extensive experiments demonstrate our superiority against state-of-the-art methods on multiple real-world dehazing benchmarks. The code is publicly available at https://github.com/YanZhang-zy/BiLaLoRA.
1. Introduction
Real image dehazing must bridge the synthetic-to-real domain gap while learning from unlabeled scenes and avoiding the cost of full fine-tuning. The paper combines CLIP-based H2C guidance with BiLaLoRA for efficient, targeted adaptation.
- Synthetic haze modeling oversimplifies atmospheric conditions, creating a significant domain gap that degrades real-world generalization.
- Real-scene adaptation lacks effective unsupervised objectives for unpaired haze images and is burdened by the computational and memory cost of updating full models.
- 1. Introduction: BiLaLoRA jointly optimizes LoRA weights and injection-layer positions to target architecture- and scene-dependent performance bottlenecks.
- 1. Introduction: H2C uses CLIP’s cross-modal capability to reformulate dehazing as semantic alignment without requiring paired real data.
2. Related Works
Real image dehazing suffers from a synthetic-to-real domain gap, while PEFT and LoRA offer lower-cost adaptation by training only selected low-rank parameters. Prior vision applications show LoRA’s potential for controllable enhancement and cross-domain generalization.
- Motivation: Real-world dehazing performance degrades because models trained on synthetic datasets face a synthetic-to-real domain gap.Domain adaptation strategies seek to reduce the resulting distribution discrepancy.
- Parameter-Efficient Fine-Tuning: PEFT freezes the pretrained model and adapts a small number of additional parameters, reducing computational cost for new tasks.LoRA injects trainable low-rank matrices into selected weight layers while targeting performance comparable to full fine-tuning.
- LoRA in Vision: LoRA-based vision methods have supported controllable image super-resolution and improved cross-domain generalization in semantic segmentation.These applications motivate LoRA as a flexible adaptation mechanism for computer vision.
3. The H2C Unsupervised Loss
The H2C loss addresses the lack of paired clear references by aligning haze-to-clear image transformations with text-defined semantic directions in CLIP’s latent space. The paper combines this guidance with a bilevel procedure that searches injection layers before LoRA fine-tuning, and reports generalization across real scenes and domains.
- The H2C Unsupervised Loss: The method provides unsupervised guidance without paired clear images by treating dehazing as semantic mapping in a shared CLIP embedding space.This reframes the absence of pixel-wise correspondences as a need for semantic rather than reference-based supervision.
- The H2C Unsupervised Loss: The H2C formulation uses CLIP image features for the input and output images and CLIP text features for negative hazy and positive clear prompts.The negative prompt defines the hazy starting point, while the positive prompt defines the clear endpoint.
- Bilevel Layer-Positioning LoRA: BiLaLoRA alternates architecture-parameter and LoRA-weight updates, selects the top-k layers, then fine-tunes LoRA weights with the selected layers fixed.The procedure uses a bilevel positioning stage followed by a LoRA fine-tuning stage.
- The H2C Unsupervised Loss: The H2C loss aligns the image transformation from haze to clear with a target direction defined by CLIP text prompts.It represents the image shift as ∆Vimg = Vout − Vin and the text trajectory as ∆Ttext = Tpos − Tneg.
- The H2C Unsupervised Loss: The loss is designed to guide clearer image generation through latent semantic information without adding complex structures.The paper contrasts this with domain adaptation methods that constrain feature-space distributions.
- Evaluation: The evaluation examines cross-model flexibility on three real dehazing datasets and cross-domain stability using synthetic pretraining from four source datasets.These experiments use four non-reference metrics for flexibility and DEA pretrained on four synthetic datasets for domain robustness.
4. BiLaLoRA
BiLaLoRA adapts dehazing models by jointly learning low-rank updates and their injection layers through bilevel optimization, targeting architecture-dependent bottlenecks with limited fine-tuning.
- Model-Agnostic Layer-Positioning Modeling: Encoder adaptation provides most of the measured MUSIQ gain, with the encoder’s final block accounting for the majority of improvement on RTTS.The analysis grafted adapted modules into the original MSBDN and DEA models to quantify component contributions.
- Model-Agnostic Layer-Positioning Modeling: LoRA represents updates with trainable low-rank matrices A and B, where the rank r is much smaller than the input and output dimensions.This parameterization supports adapting selected weight layers without updating the full pretrained model.
- Model-Agnostic Layer-Positioning Modeling: A learnable sigmoid-constrained gate α continuously relaxes discrete layer selection and scales each module’s low-rank contribution with γ.The resulting update is modulated before the selected modules are fine-tuned.
- Bilevel Layer-Positioning LoRA: BiLaLoRA jointly optimizes LoRA injection positions and low-rank weights through a bilevel objective, rather than selecting layers heuristically.The upper objective optimizes architectural parameters α, while the lower objective optimizes low-rank increments ΔW.
- Bilevel Layer-Positioning LoRA: BiLaLoRA estimates bilevel hypergradients efficiently using implicit differentiation and a rank-one approximation based only on first-order derivatives.Direct Jacobian computation is described as prohibitively expensive, while the approximation yields a computationally efficient estimator.
- Model-Agnostic Layer-Positioning Modeling: Across four pretrained dehazing architectures and multiple source datasets, BiLaLoRA improves adaptation while automatically positioning bottleneck layers.The implementation ranks candidate injection sites, then fine-tunes the top-k modules; against full fine-tuning, it achieves comparable results with dramatically reduced training time and negligible inference overhead.
5. Experimental Results
Experiments evaluate BiLaLoRA on real dehazing benchmarks and additional generalization datasets using no-reference metrics and qualitative comparisons. The method ranks first or second on key metrics and remains effective under challenging haze and domain shifts.
- Experimental Setup: The evaluation uses RTTS, URHI, and Fattal as real datasets, with HazyDet, Dense-Haze, and O-Haze assessing generalization.The reported no-reference metrics are FADE, BIQME, Entropy, and MUSIQ.
- Performance Evaluation: BiLaLoRA ranks first or second across key metrics on RTTS, URHI, and Fattal, while qualitative results show better haze removal and fine-detail preservation.Table 2 reports quantitative evaluations on the three real datasets, and Fig. 8 compares specialized and all-in-one restoration methods.
- Generalization Evaluation: BiLaLoRA maintains consistent performance on Dense-Haze and O-Haze, where CoA and IPC deteriorate substantially under dense haze.All models were evaluated on the testing datasets without retraining.
- Generalization Evaluation: On HazyDet, BiLaLoRA recovers haze-obscured scene details while preventing color distortion in UAV-perspective images.The qualitative evaluation is presented without retraining on the testing dataset.
6. Algorithmic Analyses
The analyses show that H2C’s directional guidance preserves scene structure, while bilevel layer selection improves adaptation stability and identifies an efficient number of LoRA layers.
- 6.1. Effects of Text-Directed Loss: H2C’s negative guidance is necessary because removing it drives outputs toward a singular positive semantic target and neglects content consistency.The full loss therefore supports haze removal while preserving the original scene’s structural integrity.
- 6.2. Necessity of Bilevel Modeling: Bilevel modeling substantially outperforms heuristic layer selection and naïve joint learning, with more stable convergence and sustained performance improvements.The convergence analysis attributes this behavior to automatically pinpointing and fine-tuning bottleneck layers.
- 6.2. Necessity of Bilevel Modeling: Manual layer choices constrain flexibility and can produce suboptimal adaptation, whereas bilevel modeling dynamically targets architecture-dependent bottleneck layers.This distinction explains the observed convergence and performance advantage over fixed selection.
- 6.3. Influence of Layer Number: Performance improves as more LoRA layers are activated, peaks at three layers, and then shows diminishing marginal returns with added redundancy and overhead.The parameter analysis identifies three layers as the optimal selection.
7. Concluding Remarks
BiLaLoRA targets performance bottleneck layers for efficient real-image dehazing adaptation with minimal parameter overhead. The authors identify broader low-level vision applications and severe-degradation adaptation as future directions.
- 7. Concluding Remarks: BiLaLoRA automatically pinpoints and optimizes bottleneck layers, improving cross-domain performance with minimal parameter overhead.Its plug-and-play LoRA design provides a flexible adaptation solution for real image dehazing.
- 7. Concluding Remarks: Future work will extend BiLaLoRA to other low-level vision tasks and investigate refined semantic guidance for severe degradation.The planned extensions also include cross-domain adaptation techniques for restoration challenges.