PaperlayerView exact revision on arXiv

Source-linked AI summary

InstructIR: High-Quality Image Restoration Following Human Instructions

Marcos V. Conde, Gregor Geigle, Radu Timofte

arXiv:2401.16468v5cs.CVcs.LGeess.IV

TL;DR

Image restoration must recover high-quality images from degraded observations across multiple degradation types, while prior approaches rely on specialized models or learned degradation guidance. InstructIR uses natural human-written instructions to guide a multi-task restoration model, achieving state-of-the-art results across several tasks and establishing a benchmark for text-guided restoration. Its scope remains limited for images containing multiple degradations or unknown out-of-distribution degradations.

  • Problem

    Image restoration must recover high-quality images from degraded observations, while existing All-in-One methods use learned guidance vectors rather than raw human-written prompts.

    Source: §1 · pp. 1–2

  • Method

    InstructIR guides a blind multi-task restoration model with natural human-written instructions and uses a text-embedding classification loss to learn meaningful degradation-aware representations.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 7

  • Results

    InstructIR achieves state-of-the-art results on denoising, deraining, deblurring, dehazing, and low-light image enhancement.

    Source: §1 · p. 2, §5 · p. 10, §6 · p. 13

  • Takeaways & Limitations

    The dataset and results establish a benchmark for text-guided image restoration and enhancement.

    Source: §1 · p. 2, §6 · p. 13

    The model struggles with images containing more than one degradation or unknown out-of-distribution degradations.

    Source: §5.1 · p. 13

Abstract

from arXiv · show

Image restoration is a fundamental problem that involves recovering a high-quality clean image from its degraded observation. All-In-One image restoration models can effectively restore images from various types and levels of degradation using degradation-specific information as prompts to guide the restoration model. In this work, we present the first approach that uses human-written instructions to guide the image restoration model. Given natural language prompts, our model can recover high-quality images from their degraded counterparts, considering multiple degradation types. Our method, InstructIR, achieves state-of-the-art results on several restoration tasks including image denoising, deraining, deblurring, dehazing, and (low-light) image enhancement. InstructIR improves +1dB over previous all-in-one restoration methods. Moreover, our dataset and results represent a novel benchmark for new research on text-guided image restoration and enhancement. Our code, datasets and models are available at: https://github.com/mv-lab/InstructIR

1 Introduction

Image restoration seeks to recover high-quality images from degraded observations, but diverse degradations and multiple possible restorations make the problem challenging. InstructIR addresses this gap by using human-written instructions to guide one model across multiple restoration tasks.

  • Image restoration recovers a high-quality image from a degraded counterpart, but multiple solutions may exist for each observation.Source: §1 · p. 2
  • Specialized restoration methods handle noise, blur, or haze effectively but generalize poorly across degradation types.Source: §1 · p. 2
  • All-in-One models reduce resource demands by using one blind restoration model for multiple degradation types and levels.Source: §1 · p. 2
  • Unlike learned prompt embeddings in prior All-in-One methods, InstructIR uses raw user prompts in natural language.Source: §1 · p. 2
  • InstructIR is presented as the first approach using real human-written instructions for multi-task image restoration.Source: §1 · p. 2
  • The method reports state-of-the-art performance across denoising, deraining, deblurring, dehazing, and low-light enhancement.Source: §1 · p. 2

2 Related Work

Prior restoration research includes task-specific networks, general-purpose models trained independently per task, and All-in-One models guided by degradation information. InstructIR builds on an efficient NAFNet backbone while extending restoration guidance toward human-written text instructions.

  • General-purpose restoration models such as SwinIR, MAXIM, and Uformer are trained independently for diverse tasks including denoising, deraining, and deblurring.Source: §2 · p. 2
  • Restormer uses non-local blocks to capture complex features across an image.Source: §2 · p. 3
  • InstructIR uses NAFNet as its backbone because of its efficient, simple design and strong performance across restoration tasks.Source: §2 · p. 3
  • Single-task restoration is resource-intensive because applications require different models and task selection on demand.Source: §2 · p. 3
  • All-in-One restoration uses a single blind model to tackle different degradation types and levels.Source: §2 · p. 3
  • Prior All-in-One methods guide restoration with auxiliary degradation classifiers or learned multidimensional guidance vectors.Source: §2 · p. 3
  • Text-guided image manipulation methods use prompts to describe images or actions, but InstructPix2Pix does not address inverse problems.Source: §2 · p. 3

3 Image Restoration Following Instructions

InstructIR uses natural-language instructions to control blind, multi-task image restoration. Its pipeline generates diverse training prompts, adapts a text encoder, and routes instruction-specific image features through a single restoration model.

3.1 Generating Prompts for Training

  • Human instructions provide expressive control without requiring example clean images or visual-content descriptions, supporting users with varying language and domain expertise.The training prompt collection includes GPT-4-generated and real user prompts, while ambiguous requests are filtered.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Text Encoder

  • InstructIR uses a pure sentence encoder rather than CLIP because degradation prompts generally contain little or no visual content.The text encoder maps each prompt to a fixed-size representation for the restoration model.Source: §3.2 · p. 5
  • The encoder is frozen while a learned projection adapts its embeddings, reducing overfitting risk and producing task-oriented representations.The projection maps the text dimension dt to the restoration model input dimension dv and applies l2 normalization.Source: §3.2 · p. 5
  • Training greatly improves instruction clustering, clearly separating deraining, denoising, dehazing, deblurring, and low-light enhancement embeddings.Super-resolution and enhancement instructions remain more dispersed, matching their broader linguistic applicability.Source: §3.2 · p. 5, Figure 3 · p. 6
  • An intent-classification loss trains a lightweight classification head alongside the projection, yielding over 95% accuracy for identifying degradation from prompts.The classification head is a two-layer MLP, and only the projection and head need training to capture task-relevant language information.Source: §3.2 · p. 5

3.3 InstructIR

  • The model combines a NAFNet U-Net image backbone with an encoded instruction and instruction-conditioned blocks for multi-task restoration.The ICB mechanism approximates task routing so the model can condition image processing on the user’s instruction.Source: §3.3 · p. 6, Figure 4 · p. 6
  • Each ICB converts the instruction embedding into channel-wise soft masks, filters image features, and enhances them with a convolutional NAFBlock.The resulting differentiable routing selects relevant channels and provides a degree of interpretability through feature weights.Source: §3.3 · pp. 6–7

4 Experimental Results

InstructIR is evaluated as a unified model across multiple restoration tasks and benchmarks, using human-written instructions to guide restoration. Experiments cover 3D, 5D, 6D, and 7D task settings, with instruction sensitivity also assessed.

Evaluation Setup

  • InstructIR evaluates image denoising, deblurring, deraining, dehazing, low-light enhancement, and photo-realistic enhancement across nine benchmarks.Source: §4 · p. 7

Model and Training

  • The unified model uses a NAFNet-based image backbone and a frozen sentence encoder, with training combining image L1 loss and intent-classification cross-entropy.Source: §4.1 · pp. 7–8

Multi-Task Settings

  • The 5D setting combines denoising, deblurring, dehazing, deraining, and low-light enhancement in one model.Source: §4.3 · p. 9

Results

  • InstructIR achieves competitive results in the 5D setting and improves reconstruction performance further in the 3D setting.Source: §4.3 · pp. 9–10

Multi-Task Settings

  • The 6D and 7D variants add super-resolution and MIT5K photo retouching, respectively, to the multi-task setup.Source: §4.3 · p. 10

Instruction Evaluation

  • Instruction sensitivity is evaluated with repeated PSNR/SSIM measurements using varied prompts for each image.Source: §4.3 · p. 10, Table 4 · p. 10

5 Multi-Task Discussion and Study

The multi-task studies examine how adding tasks affects restoration, how instructions control processing, and how InstructIR compares qualitatively with other methods. Adding unrelated enhancement causes a slight restoration decline, while instruction-based routing preserves broad multi-task performance.

6D Variant

  • Adding super-resolution in 6D increases performance on deblurring, a related degradation, without notably harming the other tasks.Source: §5 · p. 10

7D Variant

  • Adding photo-realistic enhancement in 7D slightly decreases restoration performance, while the model retains state-of-the-art results and notable MIT5K enhancement performance.Source: §5 · p. 10

Multi-Task Ablation

  • The multi-task ablation indicates that instruction-based task routing supports multiple tasks without notable performance loss.Source: §5 · p. 11, Table 6 · p. 11

Instruction Control

Qualitative Results

Limitations

  • InstructIR struggles with images containing multiple degradations or unknown out-of-distribution degradations, consistent with a limitation of related all-in-one methods.Source: §5.1 · p. 13

6 Conclusion

InstructIR uses natural human-written instructions to guide a multi-task restoration model across multiple degradations. The paper reports state-of-the-art results on several restoration tasks and presents the results as a benchmark for text-guided restoration.

  • InstructIR guides image restoration with natural human-written instructions.Source: §6 · p. 13
  • The multi-task model recovers high-quality images from degraded counterparts while considering multiple degradation types.Source: §6 · p. 13
  • The paper reports state-of-the-art results for denoising, deraining, deblurring, dehazing, and low-light enhancement.Source: §6 · p. 13, Figure 10 · p. 14, Figure 11 · p. 14

A Additional Training Details and Ablations

Training combines image restoration and intent-classification objectives, while ablations examine text encoders, prompt behavior, and multi-task extensions. The results support lightweight text guidance and prompt-conditioned task routing, with encoder choice showing little performance difference.

Training Objectives

Text Encoder Ablation

Model Design

Multi-Task Extensions

A.1 Additional Ablation Studies

The additional ablation section documents comparisons for denoising, deblurring, and dehazing, including both all-in-one and task-specific baselines.

A.1 Additional Ablation Studies

  • Ablation and comparison studies are reported in Tables 9 and 10.Table 9 covers image denoising, while Table 10 covers image deblurring and dehazing.Source: §A.1 · p. 17
  • Table 9 compares general restoration and all-in-one methods for image denoising.The table reports PSNR across benchmark datasets and multiple noise levels σ.Source: Table 9 · p. 17
  • Table 10 compares InstructIR with task-specific classical methods for deblurring and dehazing.The comparisons are conducted on benchmark datasets.Source: Table 10 · p. 17

B Additional Visual Results

The paper presents qualitative restoration examples across the studied degradation types, with results generally outperforming reference all-in-one methods.

B Additional Visual Results

  • InstructIR produces high-quality qualitative results across the studied image degradations.Figures 17 and 18 provide diverse visual samples, while Figure 19 evaluates real-world image dehazing.Source: B Additional Visual Results · p. 18
  • In most cases, InstructIR outperforms the reference all-in-one methods AirNet and PromptIR in visual comparisons.The paper also compares InstructIR with diffusion-based InstructPix2Pix on real-world cases.Source: B Additional Visual Results · p. 18

B.1 Efficiency Analysis

The efficiency analysis reports fast full-HD inference on consumer GPUs and lower computational cost than PromptIR, alongside qualitative examples spanning several degradation settings.

B.1 Efficiency Analysis

  • Under 1s inference for FHD images is reported on consumer-grade GPUs with 12–24GB memory.This is stated as an inference capability of InstructIR.Source: §B.1 · p. 18
  • 16M parameters versus 35M and 1.6x fewer operations make InstructIR more efficient than PromptIR.The comparison identifies PromptIR as the SOTA reference method.Source: §B.1 · p. 18
  • The denoising examples use BSD68 images with noise level σ = 25.Figure 17 presents these results for all-in-one methods.Source: Figure 17 · p. 18
  • The deraining examples compare all-in-one methods on the Rain100L dataset.These results are shown in Figure 18.Source: Figure 18 · p. 18
  • Real-world dehazing examples from RTTS are evaluated without ground-truth using the instruction “remove and haze and mist from this photo please”.The caption reports pleasant results compared with generative restoration methods such as RIDCP.Source: Figure 19 · p. 19

Related papers

Loading 2401.16468v5…