Source-linked AI summary
PromptIR: Prompting for All-in-One Blind Image Restoration
Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan
TL;DR
Deep restoration models often fail to generalize across degradation types and levels, forcing separate models or prior corruption knowledge. PromptIR uses lightweight, input-conditioned prompts to dynamically guide one restoration network, achieving state-of-the-art performance across denoising, deraining, and dehazing. The method's practical scope includes restoration at varying resolutions, with prompt components upsampled to match incoming features.
Problem
Existing restoration models generalize poorly across degradation types and levels, requiring degradation-specific models and prior knowledge of the input corruption.
Method
PromptIR uses a plug-in prompt block whose generated prompts encode degradation context and interact with restoration features to dynamically adapt processing.
Results
PromptIR achieves state-of-the-art performance on image denoising, deraining, and dehazing with one unified model.
Takeaways & Limitations
PromptIR provides an all-in-one blind restoration framework that uses only the input image and can integrate with existing restoration networks.
Takeaways & Limitations
Fixed-size prompt components cannot directly handle varying inference resolutions, so they must be bilinearly upsampled to match incoming features.
Abstract
from arXiv · showhide
Image restoration involves recovering a high-quality clean image from its degraded version. Deep learning-based methods have significantly improved image restoration performance, however, they have limited generalization ability to different degradation types and levels. This restricts their real-world application since it requires training individual models for each specific degradation and knowing the input degradation type to apply the relevant model. We present a prompt-based learning approach, PromptIR, for All-In-One image restoration that can effectively restore images from various types and levels of degradation. In particular, our method uses prompts to encode degradation-specific information, which is then used to dynamically guide the restoration network. This allows our method to generalize to different degradation types and levels, while still achieving state-of-the-art results on image denoising, deraining, and dehazing. Overall, PromptIR offers a generic and efficient plugin module with few lightweight prompts that can be used to restore images of various types and levels of degradation with no prior information on the corruptions present in the image. Our code and pretrained models are available here: https://github.com/va1shn9v/PromptIR
1 Introduction
Image restoration methods often specialize in particular degradations, limiting generalization and requiring separate models or prior knowledge. PromptIR addresses this with input-conditioned prompts that dynamically guide one unified restoration model across degradation types and levels.
- Image restoration is ill-posed because degraded images can have many feasible natural and unnatural reconstructions.
- Task-specific restoration networks generalize poorly, while separate models for degradation types, levels, and datasets are costly and require identifying the input corruption.
- Prompt guidance is injected at multiple decoding stages with few learnable parameters, supporting one model across denoising, deraining, and dehazing.
- PromptIR uses learnable prompts that encode discriminative degradation information and interact with restoration features to dynamically adjust network behavior.
- PromptIR is a blind, plug-in all-in-one framework that restores images from the input alone without prior degradation knowledge.
- PromptIR produces better-clustered degradation embeddings than AirNet, indicating more discriminative degradation context for restoration.
2 Related Works
Related work spans multi-degradation restoration, transformer architectures, and prompt learning. Existing approaches address scalability, computational complexity, or task-specific adaptation in different ways.
- Multi-degradation Image Restoration: Multi-degradation restoration remains relatively under-explored, with weather-specific pathways that require known degradation types and scale poorly.
- Transformer-based restoration: Transformers provide strong feature representations for restoration, but naive self-attention has quadratic complexity with image size.
- Prompt learning: Learnable prompts encode task-specific context for parameter-efficient adaptation in vision and multitask learning.
3 Method
PromptIR uses a UNet-style encoder-decoder with prompt blocks that generate input-conditioned degradation information and use it to guide restoration dynamically. The prompt block combines prompt generation and prompt interaction modules, integrating prompts with decoder features through transformer blocks.
- Overall Pipeline: PromptIR uses a UNet-style encoder-decoder with transformer blocks and inserts prompt blocks between decoder levels to guide clean-image recovery.The encoder progressively reduces spatial resolution and increases channel capacity, while the decoder restores high-resolution outputs.
- Prompt Block: The prompt block contains a Prompt Generation Module (PGM) and a Prompt Interaction Module (PIM).PGM produces input-conditioned prompts, while PIM uses them to adapt restoration features.
- Prompt Generation Module (PGM): PGM predicts attention-based weights from input features and applies them to learnable prompt components to produce input-conditioned prompts.It uses global average pooling, channel downscaling, softmax weights, prompt adjustment, and a 3 × 3 convolution; bilinear upsampling accommodates different input resolutions.
- Prompt Interaction Module (PIM): PIM concatenates generated prompts with input features and passes them through a transformer block to exploit degradation information during restoration.The prompts interact with decoder features at multiple levels, enriching degradation-specific context.
- Transformer Block: The architecture-agnostic prompt block uses an existing transformer block composed of MDTA and GDFN rather than introducing a new transformer design.MDTA performs channel-wise self-attention with linear complexity, while GDFN suppresses less informative features and propagates useful ones.
4 Experiments
PromptIR is evaluated as a unified all-in-one model and in single-task settings across denoising, deraining, and dehazing, with additional ablations examining prompting design and generalization.
- Experimental settings: PromptIR is evaluated on dehazing, deraining, and denoising under all-in-one and single-task settings.The all-in-one setting uses one model trained across degradation types, whereas single-task experiments train separate models.
- All-in-one results: 0.86 dB average gain over AirNet is achieved across restoration tasks in the all-in-one setting.PromptIR also improves over DL by 2.08 dB on average.
- All-in-one results: 2.64 dB PSNR improvement is reported for dehazing, while deraining improves by 1.47 dB over AirNet.On denoising, PromptIR provides a 1.16 dB boost over DL at σ=50.
- Single-task results: 8.13 dB improvement over AirNet is reported for single-task dehazing, alongside 2.13 dB for deraining and 0.51 dB for denoising at σ=50 on Urban100.The dehazing result also exceeds Restormer by 0.44 dB.
- Ablations: Dynamic prompts improve Rain100L deraining by 0.19 dB over fixed prompt components, while the prompt block gains 0.3 dB over the baseline.Prompt blocks between every consecutive decoder level perform best, whereas using only one latent-space block degrades performance.
- Generalization: PromptIR exceeds AirNet by approximately 7 dB PSNR when tested at unseen noise level σ=100 after training on σ∈{15,25,50}.For spatially varying noise levels σ=[0,15,25,50], PromptIR improves over AirNet by 0.23 dB.
- Degradation combinations: Increasing the number of combined degradation types causes performance to drop, while rainy-and-noisy training achieves good performance and suggests task compatibility.Hazy images appear to negatively affect the combined model.
5 Conclusion
The paper concludes that PromptIR uses a drop-in prompt block to adapt restoration representations to relevant degradations. It reports improvements across denoising, deraining, and dehazing and identifies broader corruption coverage as future work.
- Conclusion: PromptIR proposes a drop-in prompt block that dynamically adjusts feature representations for the relevant degradation.The block interacts with input features so the restoration process adapts to the corruption.
- Conclusion: PromptIR improves image denoising, deraining, and dehazing when integrated into a state-of-the-art restoration model.The conclusion frames this as evidence for the utility of prompt blocks in all-in-one restoration.
- Future work: Future work will extend PromptIR to a broader set of corruptions toward universal image-restoration models.
A Additional Ablation Studies
Additional ablations compare prompt tokens with contrastive embeddings, test where prompt blocks are inserted, and detail the Transformer block used in PromptIR.
- Prompt design: Contrastive embeddings from AirNet’s degradation encoder perform significantly worse than PromptIR prompt tokens.Good performance with contrastive embeddings also requires a custom-designed restoration network, whereas PromptIR prompt blocks are designed for seamless integration.
- Prompt placement: Prompt blocks should be used only on the decoder side rather than on both encoder and decoder branches.The study identifies decoder-only prompting as important for performance.
- Transformer block: The Transformer block comprises Multi-Dconv head Transposed Attention (MDTA) and Gated Dconv Feed-forward Network (GDFN) submodules.MDTA computes multi-head attention across channel dimensions, while GDFN uses parallel convolutional paths and gated element-wise combination.
C Qualitative results:
Additional qualitative results are presented for single-task models to further examine prompt-block effectiveness.
- Qualitative results: Single-task qualitative results are provided to further elucidate prompt-block effectiveness.
C.1 Dehazing
The figure compares image deraining methods on SOTS images and reports that PromptIR effectively removes haze, producing visually better images.
- The comparison uses images from the SOTS dataset.
- PromptIR effectively removes haze in the displayed examples.
- The resulting images are reported as visually better than those from previous methods.
C.2 Deraining
The figure compares image deraining methods on Rain100L images and shows PromptIR removing rain streaks to generate rain-free images.
- The comparison uses images from the Rain100L dataset.
- PromptIR effectively removes rain streaks in the displayed examples.
- The method generates rain-free images.
C.3 Denoising
The figure compares image deraining methods on URBAN100 images with σ = 50 and reports visually better results from PromptIR.
- The comparison uses images from the URBAN100 dataset.
- The displayed comparison uses σ = 50.
- PromptIR produces visually better images than previous methods in the selected patches.