Source-linked AI summary
InstructIR: High-Quality Image Restoration Following Human Instructions
Marcos V. Conde, Gregor Geigle, Radu Timofte
TL;DR
Image restoration must recover high-quality images from degraded observations across multiple degradation types, while prior approaches rely on specialized models or learned degradation guidance. InstructIR uses natural human-written instructions to guide a multi-task restoration model, achieving state-of-the-art results across several tasks and establishing a benchmark for text-guided restoration. Its scope remains limited for images containing multiple degradations or unknown out-of-distribution degradations.
Problem
Image restoration must recover high-quality images from degraded observations, while existing All-in-One methods use learned guidance vectors rather than raw human-written prompts.
Method
InstructIR guides a blind multi-task restoration model with natural human-written instructions and uses a text-embedding classification loss to learn meaningful degradation-aware representations.
Results
InstructIR achieves state-of-the-art results on denoising, deraining, deblurring, dehazing, and low-light image enhancement.
Takeaways & Limitations
The dataset and results establish a benchmark for text-guided image restoration and enhancement.
Takeaways & Limitations
The model struggles with images containing more than one degradation or unknown out-of-distribution degradations.
Abstract
from arXiv · showhide
Image restoration is a fundamental problem that involves recovering a high-quality clean image from its degraded observation. All-In-One image restoration models can effectively restore images from various types and levels of degradation using degradation-specific information as prompts to guide the restoration model. In this work, we present the first approach that uses human-written instructions to guide the image restoration model. Given natural language prompts, our model can recover high-quality images from their degraded counterparts, considering multiple degradation types. Our method, InstructIR, achieves state-of-the-art results on several restoration tasks including image denoising, deraining, deblurring, dehazing, and (low-light) image enhancement. InstructIR improves +1dB over previous all-in-one restoration methods. Moreover, our dataset and results represent a novel benchmark for new research on text-guided image restoration and enhancement. Our code, datasets and models are available at: https://github.com/mv-lab/InstructIR
1 Introduction
Image restoration seeks to recover high-quality images from degraded observations, but diverse degradations and multiple possible restorations make the problem challenging. InstructIR addresses this gap by using human-written instructions to guide one model across multiple restoration tasks.
- Image restoration recovers a high-quality image from a degraded counterpart, but multiple solutions may exist for each observation.
- Specialized restoration methods handle noise, blur, or haze effectively but generalize poorly across degradation types.
- All-in-One models reduce resource demands by using one blind restoration model for multiple degradation types and levels.
- Unlike learned prompt embeddings in prior All-in-One methods, InstructIR uses raw user prompts in natural language.
- InstructIR is presented as the first approach using real human-written instructions for multi-task image restoration.
- The method reports state-of-the-art performance across denoising, deraining, deblurring, dehazing, and low-light enhancement.
2 Related Work
Prior restoration research includes task-specific networks, general-purpose models trained independently per task, and All-in-One models guided by degradation information. InstructIR builds on an efficient NAFNet backbone while extending restoration guidance toward human-written text instructions.
- General-purpose restoration models such as SwinIR, MAXIM, and Uformer are trained independently for diverse tasks including denoising, deraining, and deblurring.
- Restormer uses non-local blocks to capture complex features across an image.
- InstructIR uses NAFNet as its backbone because of its efficient, simple design and strong performance across restoration tasks.
- Single-task restoration is resource-intensive because applications require different models and task selection on demand.
- All-in-One restoration uses a single blind model to tackle different degradation types and levels.
- Prior All-in-One methods guide restoration with auxiliary degradation classifiers or learned multidimensional guidance vectors.
- Text-guided image manipulation methods use prompts to describe images or actions, but InstructPix2Pix does not address inverse problems.
3 Image Restoration Following Instructions
InstructIR uses natural-language instructions to control blind, multi-task image restoration. Its pipeline generates diverse training prompts, adapts a text encoder, and routes instruction-specific image features through a single restoration model.
- 3.1 Generating Prompts for Training: Human instructions provide expressive control without requiring example clean images or visual-content descriptions, supporting users with varying language and domain expertise.The training prompt collection includes GPT-4-generated and real user prompts, while ambiguous requests are filtered.
- 3.2 Text Encoder: InstructIR uses a pure sentence encoder rather than CLIP because degradation prompts generally contain little or no visual content.The text encoder maps each prompt to a fixed-size representation for the restoration model.
- 3.2 Text Encoder: The encoder is frozen while a learned projection adapts its embeddings, reducing overfitting risk and producing task-oriented representations.The projection maps the text dimension dt to the restoration model input dimension dv and applies l2 normalization.
- 3.2 Text Encoder: Training greatly improves instruction clustering, clearly separating deraining, denoising, dehazing, deblurring, and low-light enhancement embeddings.Super-resolution and enhancement instructions remain more dispersed, matching their broader linguistic applicability.
- 3.2 Text Encoder: An intent-classification loss trains a lightweight classification head alongside the projection, yielding over 95% accuracy for identifying degradation from prompts.The classification head is a two-layer MLP, and only the projection and head need training to capture task-relevant language information.
- 3.3 InstructIR: The model combines a NAFNet U-Net image backbone with an encoded instruction and instruction-conditioned blocks for multi-task restoration.The ICB mechanism approximates task routing so the model can condition image processing on the user’s instruction.
- 3.3 InstructIR: Each ICB converts the instruction embedding into channel-wise soft masks, filters image features, and enhances them with a convolutional NAFBlock.The resulting differentiable routing selects relevant channels and provides a degree of interpretability through feature weights.
4 Experimental Results
InstructIR is evaluated as a unified model across multiple restoration tasks and benchmarks, using human-written instructions to guide restoration. Experiments cover 3D, 5D, 6D, and 7D task settings, with instruction sensitivity also assessed.
- Evaluation Setup: InstructIR evaluates image denoising, deblurring, deraining, dehazing, low-light enhancement, and photo-realistic enhancement across nine benchmarks.
- Model and Training: The unified model uses a NAFNet-based image backbone and a frozen sentence encoder, with training combining image L1 loss and intent-classification cross-entropy.
- Multi-Task Settings: The 5D setting combines denoising, deblurring, dehazing, deraining, and low-light enhancement in one model.
- Results: InstructIR achieves competitive results in the 5D setting and improves reconstruction performance further in the 3D setting.
- Multi-Task Settings: The 6D and 7D variants add super-resolution and MIT5K photo retouching, respectively, to the multi-task setup.
- Instruction Evaluation: Instruction sensitivity is evaluated with repeated PSNR/SSIM measurements using varied prompts for each image.
5 Multi-Task Discussion and Study
The multi-task studies examine how adding tasks affects restoration, how instructions control processing, and how InstructIR compares qualitatively with other methods. Adding unrelated enhancement causes a slight restoration decline, while instruction-based routing preserves broad multi-task performance.
- 6D Variant: Adding super-resolution in 6D increases performance on deblurring, a related degradation, without notably harming the other tasks.
- 7D Variant: Adding photo-realistic enhancement in 7D slightly decreases restoration performance, while the model retains state-of-the-art results and notable MIT5K enhancement performance.
- Multi-Task Ablation: The multi-task ablation indicates that instruction-based task routing supports multiple tasks without notable performance loss.
- Instruction Control: Human instructions let users control image enhancement and apply multiple instructions sequentially.
- Qualitative Results: The qualitative evaluation compares InstructIR with all-in-one and task-specific methods across low-light enhancement, deblurring, dehazing, deraining, and denoising.
- Limitations: InstructIR struggles with images containing multiple degradations or unknown out-of-distribution degradations, consistent with a limitation of related all-in-one methods.
6 Conclusion
InstructIR uses natural human-written instructions to guide a multi-task restoration model across multiple degradations. The paper reports state-of-the-art results on several restoration tasks and presents the results as a benchmark for text-guided restoration.
- InstructIR guides image restoration with natural human-written instructions.
- The multi-task model recovers high-quality images from degraded counterparts while considering multiple degradation types.
- The paper reports state-of-the-art results for denoising, deraining, deblurring, dehazing, and low-light enhancement.
A Additional Training Details and Ablations
Training combines image restoration and intent-classification objectives, while ablations examine text encoders, prompt behavior, and multi-task extensions. The results support lightweight text guidance and prompt-conditioned task routing, with encoder choice showing little performance difference.
- Training Objectives: InstructIR trains with L = L1 + Lce, combining image reconstruction loss with prompt-based intent classification loss.
- Training Objectives: The intent-classification loss predicts the restoration task or degradation from the prompt embedding and tends toward zero quickly.
- Text Encoder Ablation: Text-encoder ablations find no significant performance difference among BGE-micro-v2, all-MiniLM-L6-v2, and CLIP under a fixed image model.
- Text Encoder Ablation: BGE-micro-v2 is selected because it uses 17M parameters compared with 40–60M for the other tested encoders.
- Model Design: Given a trained text-guided image model, the language model can be switched without performance loss.
- Multi-Task Extensions: The 6D super-resolution setup uses bicubic downsampling and re-upsampling, and its addition helps the deblurring task.
A.1 Additional Ablation Studies
The additional ablation section documents comparisons for denoising, deblurring, and dehazing, including both all-in-one and task-specific baselines.
- A.1 Additional Ablation Studies: Ablation and comparison studies are reported in Tables 9 and 10.Table 9 covers image denoising, while Table 10 covers image deblurring and dehazing.
- A.1 Additional Ablation Studies: Table 9 compares general restoration and all-in-one methods for image denoising.The table reports PSNR across benchmark datasets and multiple noise levels σ.
- A.1 Additional Ablation Studies: Table 10 compares InstructIR with task-specific classical methods for deblurring and dehazing.The comparisons are conducted on benchmark datasets.
B Additional Visual Results
The paper presents qualitative restoration examples across the studied degradation types, with results generally outperforming reference all-in-one methods.
- B Additional Visual Results: InstructIR produces high-quality qualitative results across the studied image degradations.Figures 17 and 18 provide diverse visual samples, while Figure 19 evaluates real-world image dehazing.
- B Additional Visual Results: In most cases, InstructIR outperforms the reference all-in-one methods AirNet and PromptIR in visual comparisons.The paper also compares InstructIR with diffusion-based InstructPix2Pix on real-world cases.
B.1 Efficiency Analysis
The efficiency analysis reports fast full-HD inference on consumer GPUs and lower computational cost than PromptIR, alongside qualitative examples spanning several degradation settings.
- B.1 Efficiency Analysis: Under 1s inference for FHD images is reported on consumer-grade GPUs with 12–24GB memory.This is stated as an inference capability of InstructIR.
- B.1 Efficiency Analysis: 16M parameters versus 35M and 1.6x fewer operations make InstructIR more efficient than PromptIR.The comparison identifies PromptIR as the SOTA reference method.
- B.1 Efficiency Analysis: The denoising examples use BSD68 images with noise level σ = 25.Figure 17 presents these results for all-in-one methods.
- B.1 Efficiency Analysis: The deraining examples compare all-in-one methods on the Rain100L dataset.These results are shown in Figure 18.
- B.1 Efficiency Analysis: Real-world dehazing examples from RTTS are evaluated without ground-truth using the instruction “remove and haze and mist from this photo please”.The caption reports pleasant results compared with generative restoration methods such as RIDCP.