PaperlayerView exact revision on arXiv

Source-linked AI summary

All-in-one Weather-degraded Image Restoration via Adaptive Degradation-aware Self-prompting Model

Yuanbo Wen, Tao Gao, Ziqi Li, Jing Zhang, Kaihao Zhang, Ting Chen

arXiv:2411.07445v1cs.CV

TL;DR

All-in-one weather restoration needs better ways to use degradation-aware priors across different weather conditions. ADSM combines CLIP-trained latent prompts with diffusion restoration and a wavelet-based noise estimator; ablations report performance decreases when prompt generators are removed.

  • Problem

    Existing all-in-one restoration approaches use degradation-aware priors inefficiently, limiting adaptation to different weather conditions.

    Source: §I · p. 1

  • Method

    ADSM trains CLIP-aligned latent prompts for degradation type, degradation property, and image caption, then uses them in diffusion restoration with a wavelet-oriented noise estimating network.

    Source: §I · p. 2, §VI · p. 11

  • Results

    Ablations report a 3.93 dB performance improvement over the model without LPGs and a 2.76 dB overall performance decrease when the caption prompt generator is removed.

    Source: E. Ablation Studies · p. 10

  • Takeaways & Limitations

    The conclusion states that the prompts help the model understand weather distortions and reconstruct image content, while the wavelet-based estimator improves frequency perception and inference speed.

    Source: §VI · p. 11

    Reconstruction fidelity depends heavily on accurate LPG prompts, and diffusion inference time and computational burden restrict real-world applicability.

    Source: §V · p. 11

Abstract

from arXiv · show

Existing approaches for all-in-one weather-degraded image restoration suffer from inefficiencies in leveraging degradation-aware priors, resulting in sub-optimal performance in adapting to different weather conditions. To this end, we develop an adaptive degradation-aware self-prompting model (ADSM) for all-in-one weather-degraded image restoration. Specifically, our model employs the contrastive language-image pre-training model (CLIP) to facilitate the training of our proposed latent prompt generators (LPGs), which represent three types of latent prompts to characterize the degradation type, degradation property and image caption. Moreover, we integrate the acquired degradation-aware prompts into the time embedding of diffusion model to improve degradation perception. Meanwhile, we employ the latent caption prompt to guide the reverse sampling process using the cross-attention mechanism, thereby guiding the accurate image reconstruction. Furthermore, to accelerate the reverse sampling procedure of diffusion model and address the limitations of frequency perception, we introduce a wavelet-oriented noise estimating network (WNE-Net). Extensive experiments conducted on eight publicly available datasets demonstrate the effectiveness of our proposed approach in both task-specific and all-in-one applications.

I. INTRODUCTION

Weather restoration methods face a trade-off between specialized models and limited adaptation to diverse degradations, while existing prompt approaches can lack degradation-specific and image-content cues. ADSM uses CLIP-trained latent prompts with diffusion restoration and a wavelet-oriented noise estimator.

I. INTRODUCTION

  • Task-specific and task-aligned methods address one degradation type, while all-in-one methods use one model for varied weather degradations.The introduction describes the first two settings as requiring different models or parameters and says both are limited in adapting to diverse weather.Source: §I · p. 1
  • The authors report improved naturalness and image quality over existing all-in-one approaches in the figure’s supervised and unsupervised comparisons.The introduction also describes improved supervised metrical scores.Source: Figure 1 · p. 1, §I · p. 2
  • Fixed discrete prompts are sensitive to prompt quality, while ProRes [20] lacks specific degradation and image-content information for complex weather images.The introduction identifies a need for prompts that characterize both degradation specifics and contextual details.Source: §I · p. 1
  • ADSM uses CLIP-trained latent prompt generators for degradation type, degradation property, and image caption, integrating degradation prompts into time embedding and caption prompts through cross-attention.The degradation prompts target individual and combined weather conditions; the caption prompt guides image-content reconstruction.Source: §I · p. 2
  • WNE-Net uses wavelet-oriented noise estimation to improve sampling efficiency and noise estimation, with frequency-aware blocks and wavelet down-sampling to reduce computation.Its WSRB separates frequency components and models long-range relationships, while WFSB uses frequency-sensitive branches.Source: §I · p. 2

II. RELATED WORK

Prior all-in-one weather-removal methods do not incorporate prompts to improve adaptability across degradation types. Visual prompt-learning approaches use paired or learnable prompts, which can yield dissimilar visual representations across prompt settings.

A. All-in-one Weather Removal

  • Earlier all-in-one weather-removal methods capture degradation representations through data-driven self-training without prompts to improve adaptability across weather types.The cited approaches include task-specific encoders, task-related queries, knowledge blending, patch-based diffusion, frequency-oriented restoration, and predefined guidelines.Source: A. All-in-one Weather Removal · p. 3

B. Visual Prompt Learning

  • Visual prompt methods rely heavily on high-quality paired or learnable prompts, with dissimilar visual representations across different prompt settings in reconstructed images.The subsection discusses MAE-VQGAN, CoCoOp, ControlNet, Painter, and ProRes as examples of prompt-learning approaches.Source: B. Visual Prompt Learning · p. 3

III. METHODOLOGY

The method trains latent prompt generators from weather-degraded images to produce degradation-type, degradation-property, and caption prompts. It uses degradation-aware guidance in a diffusion model’s time embedding and cross-attention, while restricting cross-attention to the noise estimator’s middle and decoder stages.

A. Latent Prompt Generators

  • Similar-meaning texts depicting the same image can have different CLIP latent encodings, making text-only reverse-sampling constraints unstable.The generators instead use fixed and learnable CLIP image encoders to derive prompts from weather-degraded images.Source: A. Latent Prompt Generators · p. 3, Figure 3 · p. 3
  • Unlocked CLIP features produce degradation-type, degradation-property, and caption prompts, while locked-encoder features join the caption prompt through cross-attention.Degradation prompts are combined, passed through MLP layers, and scaled for integration into the diffusion model’s time embedding.Source: A. Latent Prompt Generators · p. 3, Figure 4 · p. 4
  • The LPGs are trained with manually labeled degradation types and properties, synthetic BLIP captions for clean images, and cross-entropy prompt alignment.The authors state that the generators classify mixed weather conditions and avoid reliance on extra image pairs or prompts.Source: A. Latent Prompt Generators · p. 3

B. Adaptive Degradation-aware Self-prompting Model

  • The diffusion restoration model combines degradation-type and degradation-property prompts into a scaled degradation-aware prompt for its time embedding.The degradation prompts are intended to help identify and correct weather degradations.Source: B. Adaptive Degradation-aware… · pp. 3–4
  • Caption-prompt cross-attention controls contextual information flow, and is limited to the noise estimator’s middle and decoder stages.The stated reason for this restriction is the computational demand of self-attention on high-resolution weather-degraded images.Source: B. Adaptive Degradation-aware… · p. 4
  • The noise estimator receives degradation-type, degradation-property, and caption prompts; the authors assign degradation classification and specification to the first two, and reconstruction guidance to the caption prompt.The model conditions noise estimation on the noisy state and the degraded image.Source: B. Adaptive Degradation-aware… · p. 4

C. Wavelet-oriented Noise Estimating Network

WNE-Net modifies the diffusion noise-estimation network with wavelet operations to capture frequency components and long-range information while reducing sampling computation. Its blocks combine wavelet-based processing with self-attention, convolution, and degradation-aware prompts.

C. Wavelet-oriented Noise Estimating Network

  • WNE-Net uses discrete wavelet transforms to separate frequency components and reduce feature resolution before self-attention, then inverse transforms restore the original resolution.The design targets global information and frequency components that traditional diffusion residual blocks may overlook.Source: C. Wavelet-oriented Noise Estimating… · p. 4
  • The WSRB combines frequency-aware attention with a convolutional branch that integrates individual frequency variables and captures local and global information.The block also uses a skip convolution after a multi-layer perceptron for long-distance information exchange and training robustness.Source: C. Wavelet-oriented Noise Estimating… · pp. 4–5
  • The illustrated WNE-Net is U-shaped and takes the combined noisy signal state at time t and degraded image condition as input.The figure intentionally leaves out time embedding, degradation-aware prompts, and caption prompts.Source: Figure 5 · p. 5
  • Degradation-aware prompts and time embedding are integrated between WSRB branches and during WFSB sampling to help identify weather degradation types and properties.The architecture figure omits prompt and time-embedding complexities for simplification.Source: C. Wavelet-oriented Noise Estimating… · p. 5, Figure 5 · p. 5
  • WFSB uses wavelet feature down-sampling and up-sampling blocks for cross-stage progression, with DWT reducing feature resolution and IDWT enhancing it.The paper motivates this design by noting that conventional cross-stage sampling can overlook frequency characteristics and overlapping strategies can misalign features.Source: C. Wavelet-oriented Noise Estimating… · p. 5

A. Implementation Details

Training proceeds in two stages: the latent prompt generators are trained first, then their prompts are embedded in the diffusion restoration model for further training. The reported implementation specifies the optimizer, iteration counts, and learning rates.

A. Implementation Details

  • Training first optimizes the latent prompt generators, then embeds their prompts into the wavelet-aware noise estimating network for further training.The experiments use an NVIDIA Tesla A800 GPU and the AdamW optimizer with β1 = 0.9 and β2 = 0.99.Source: A. Implementation Details · p. 5
  • The latent prompt generators train for 10 000 iterations, while prompted image reconstruction runs for 1 000 000 iterations.Source: A. Implementation Details · p. 5
  • The learning rates are 5 × 10−6 for the latent prompt generators and 2 × 10−4 for the diffusion restoration model, with the latter cosine-annealed to 1 × 10−6.Source: A. Implementation Details · p. 5
  • Table I describes the training and testing sample partitions in AWIR120K across different weather degradations.Source: Table I · p. 6

B. Datasets and Evaluation Metrics

The evaluation covers eight single and mixed weather degradations and uses a composed training dataset of 119 146 image pairs. Image quality is assessed with distortion, perceptual similarity, and naturalness metrics.

B. Datasets and Evaluation Metrics

  • AWIR120K contains 119 146 image pairs spanning eight weather degradations, and evaluation covers rain, haze, raindrop, snow, low-lightness, and three mixed degradations.The evaluated datasets are Rain200H, RESIDE-6K, Raindrop-A, Snow100K-L, LOL, RS100K-L, Test1, and RainDS-syn.Source: B. Datasets and Evaluation Metrics · p. 6
  • PSNR, SSIM, and LPIPS compare reconstructed images with ground truth, with higher scores indicating better image quality.Source: B. Datasets and Evaluation Metrics · p. 6
  • NIQE evaluates naturalness, with lower values corresponding to better perceptual image quality.Source: B. Datasets and Evaluation Metrics · p. 6

C. Comparisons with Task-specific Setting

In separate task-specific training and testing across eight weather-degraded datasets, ADSM reports gains over recent methods on the reported comparisons. It also reports best SSIM for deraining and best quantitative performance for dehazing and enhancement.

C. Comparisons with Task-specific Setting

D. Comparisons with All-in-one Setting

In the all-in-one setting, ADSM achieves the best reported metric scores across eight weather-degraded datasets. Ablations report gains from the latent prompt generators and wavelet-based noise-estimation components, with wavelet-transformed self-attention reducing computational cost.

D. Comparisons with All-in-one Setting

  • ADSM achieves the best metric scores across eight weather-degraded datasets in the all-in-one setting.Reported PSNR gains over PromptIR [82] are 0.45 dB for deraining, 0.98 dB for dehazing, and 0.63 dB for raindrop removal; for desnowing, the reported gain is 0.53 dB over PromptIR [82].Source: D. Comparisons with All-in-one Setting · p. 6, Table III · p. 8

E. Ablation Studies

  • Using the proposed LPGs improves performance by 3.93 dB compared with the model without them; removing the caption prompt generator causes an overall performance decrease of 2.76 dB.All three latent prompt generators benefit overall performance.Source: E. Ablation Studies · p. 10
  • Replacing WSRB with a residual block decreases PSNR by 1.13 dB, while replacing WFSB cross-stage sampling with convolution causes a 1.38 dB drop.The authors report that incorporating discrete wavelet transform into the noise-estimating network is effective for noise estimation.Source: E. Ablation Studies · p. 10, Table V · p. 10
  • Wavelet-transformed features reduce self-attention computational burden by an order of magnitude in both addition and multiplication operations.The authors also report improved frequency-domain noise perception with wavelet-based self-attention.Source: Table VI · p. 11, E. Ablation Studies · p. 11

V. LIMITATIONS

The authors identify dependence on accurate latent prompts and diffusion-model inference costs as limitations of ADSM.

V. LIMITATIONS

  • Reconstruction fidelity depends heavily on latent prompt generators accurately characterizing degradation type, degradation property, and image captions.The stated prompt requirements span both degradation and image-caption information.Source: §V · p. 11
  • Significant diffusion-model inference time and computational burden restrict the approach’s real-world applicability.Source: §V · p. 11
  • The prompt generators must accurately characterize degradation type and property as well as image captions for high-fidelity reconstruction.Source: §V · p. 11

VI. CONCLUSION

The conclusion presents ADSM as an all-in-one weather-restoration model using prompts for degradation types, properties, and image captions. It also identifies faster reverse sampling and richer latent features as future directions.

VI. CONCLUSION

  • ADSM uses latent prompt generators to derive prompts for degradation types, degradation properties, and image captions.The authors state that these prompts help the restoration model understand prevalent weather distortions and reconstruct image contents.Source: §VI · p. 11
  • The wavelet-based noise-estimating network is intended to improve frequency perception and speed inference.Source: §VI · p. 11
  • Future work will explore extracting more latent features from degraded images, accelerating reverse sampling, and creating a more comprehensive dataset.Source: §VI · p. 11
Loading 2411.07445v1…