PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion

Yitong Jiang, Zhaoyang Zhang, Tianfan Xue, Jinwei Gu

arXiv:2310.10123v5cs.CV

TL;DR

Real-world images can have multiple unknown degradations that separate restoration methods struggle to address. AutoDIR combines semantic-agnostic assessment with text-guided latent-diffusion restoration, and consistently outperforms state-of-the-art methods across seven tasks.

  • Problem

    Real-world images often undergo multiple unknown degradations, while single-task restoration methods struggle with complex cases involving multiple unknown degradations or enhancement steps.

    Source: §1 · p. 1

  • Method

    AutoDIR pairs semantic-agnostic BIQA, which detects degradations and generates text guidance, with AIR, a structurally corrected latent-diffusion restoration stage.

    Source: §1 · p. 2

  • Results

    Across seven image restoration tasks, AutoDIR consistently outperforms state-of-the-art methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Joint training across restoration datasets positions AutoDIR as a foundation model that supports few-shot learning for novel tasks.

    Source: §5 · p. 14

    AutoDIR's computational cost is similar to other diffusion-based restoration methods and often significantly higher than non-generative networks such as NAFNet.

    Source: §5 · p. 14

Abstract

from arXiv · show

We present AutoDIR, an innovative all-in-one image restoration system incorporating latent diffusion. AutoDIR excels in its ability to automatically identify and restore images suffering from a range of unknown degradations. AutoDIR offers intuitive open-vocabulary image editing, empowering users to customize and enhance images according to their preferences. Specifically, AutoDIR consists of two key stages: a Blind Image Quality Assessment (BIQA) stage based on a semantic-agnostic vision-language model which automatically detects unknown image degradations for input images, an All-in-One Image Restoration (AIR) stage utilizes structural-corrected latent diffusion which handles multiple types of image degradations. Extensive experimental evaluation demonstrates that AutoDIR outperforms state-of-the-art approaches for a wider range of image restoration tasks. The design of AutoDIR also enables flexible user control (via text prompt) and generalization to new tasks as a foundation model of image restoration. Project is available at: \url{https://jiangyitong.github.io/AutoDIR_webpage/}.

1 Introduction

AutoDIR targets images with multiple unknown degradations that single-task restoration methods struggle to handle. It combines semantic-agnostic degradation assessment with text-guided, structurally corrected latent-diffusion restoration and reports strong results across seven tasks.

1 Introduction

  • Real-world images can contain multiple unknown degradations, while single-task methods struggle with complex cases requiring multiple enhancement steps.The paper frames the goal as a model that distinguishes unknown degradations, restores them task-agnostically, and lets users tune results.Source: §1 · pp. 1–2
  • AutoDIR uses SA-BIQA to detect degradations and guide AIR, and can decompose unknown degradations into multiple prompt-supported restoration steps.The overview describes handling unseen tasks as well as user interaction through text prompts.Source: §1 · p. 2, Figure 1 · p. 2
  • SA-CLIP regularization shifts CLIP toward assessing structural quality rather than semantic content, supporting open-vocabulary identification of unknown artifacts.SA-BIQA text embeddings also serve as restoration instructions and allow runtime user control through open-vocabulary prompts.Source: §1 · pp. 2–3
  • AIR jointly trains a multitask restoration model and combines diffusion's texture-hallucination ability with structural inductive bias to preserve image consistency.The paper contrasts super-resolution's need to hallucinate textures with low-light enhancement's need to preserve content except brightness.Source: §1 · p. 3
  • AutoDIR consistently outperforms state-of-the-art methods across seven image restoration tasks.The evaluation also includes images from under-display and underwater cameras with multiple unknown degradations.Source: §1 · p. 3

2 Related Work

Prior work unifies restoration through generative priors or supervised models, and separately advances text-based editing. The paper describes constraints in both areas; Fig. 2 is identified as the AutoDIR diagram.

2 Related Work

  • Generative-prior restoration methods rely on known degradation functions, while supervised methods are limited in degradation categories and handling multiple unknown degradations.AutoDIR is presented as unifying complex degradations without assumptions about specific degradation functions.Source: §2 · p. 3
  • Text-based editing methods can require semantic editing focus, per-image diffusion fine-tuning, or interior maps unavailable for real-image editing.The paper distinguishes AutoDIR as supporting real-image enhancement without per-image fine-tuning.Source: §2 · p. 4
  • Fig. 2 is identified as the diagram of AutoDIR, with further details deferred to Section 3.Source: Figure 2 · p. 4

3 Our Method

AutoDIR pairs automatic degradation assessment with prompt-guided latent-diffusion restoration, adding a structural-correction path to preserve image context.

3 Our Method

  • AutoDIR pairs SA-BIQA, which identifies predominant degradations and generates prompts, with AIR, which restores images using SC-LDM and automatic or user text guidance.SA-BIQA supplies the automatic prompt; users can also provide open-vocabulary instructions.Source: §3 · p. 5

3.1 Semantic-Agnostic Blind Image Quality Assessment (SA-BIQA)

  • SA-CLIP uses semantic-agnostic regularization to focus on image quality rather than image semantics when identifying unknown degradations.The loss penalizes predicting a degradation for the corresponding clean image.Source: §3.1 · pp. 5–7

3.2 Effectiveness of Semantic-Agnostic CLIP

  • SA-CLIP separates foggy from clean embeddings and outperforms ViT, original CLIP, and naive fine-tuning in degradation detection.Its semantic-agnostic constraint directs fine-tuning toward image quality rather than image semantics.Source: Figure 3 · p. 6, §3.2 · p. 7, §3.1 · p. 7

3.3 All-in-one Image Restoration (AIR)

  • A lightweight structural-correction module addresses latent diffusion’s difficulty reconstructing complex, small structures caused by VAE compression and reconstruction.The paper identifies faces and text as examples of structures LDM may fail to preserve.Source: §3.3 · p. 8
  • SCM injects original-image context residually, with an adjustable weight balancing structural consistency against latent diffusion’s generation capacity.A larger weight is useful for structure-sensitive low-light enhancement; a smaller one is often used for super-resolution.Source: §3.3 · pp. 8–9
  • On held-out TOLED and EUVP datasets, AutoDIR handles UDC restoration as brightening then deblurring, and underwater restoration as deblurring then dehazing.Both datasets contain multiple unknown degradations and were excluded from training.Source: Figure 7 · p. 10
  • Text prompts redirect diffusion cross-attention: dehazing spreads attention broadly, while deraindrop focuses on artifacts and low-resolution attends to edges or textures.The maps provide a qualitative view of how text conditions distinguish restoration tasks.Source: Figure 5 · p. 9, §3.3 · p. 10

4 Experiments

Experiments evaluate AutoDIR on seven restoration tasks and images with multiple unknown degradations, then examine component contributions and transfer to unseen tasks.

4 Experiments

  • The experiments compare AutoDIR with CNN-based single-task and all-in-one methods and diffusion-based multitasking backbones on two unknown-degradation settings and seven restoration tasks.The comparison includes super-resolution, where the authors highlight generative models' detail-hallucination ability.Source: §4 · p. 10

4.1 Results on Multiple Unknown Degradations

  • On unseen TOLED and EUVP data, AutoDIR decomposes UDC restoration into brightening then deblurring, and underwater restoration into deblurring then dehazing; 96.9% of user-study votes favored it.The datasets were excluded from training, and AutoDIR outperformed the other black-box and classifier-based all-in-one methods on quantitative perceptual results.Source: §4.1 · p. 11, Table 3 · p. 11

4.2 Results on Multi-task Image Restoration

  • Across seven tasks AutoDIR is competitive; on super-resolution it outperforms all-in-one and task-specific discriminative baselines and black-box LDM, with more appealing visuals on six other tasks.The reported super-resolution comparison includes non-reference metrics because full-reference metrics may not align with human evaluation for that task.Source: Table 2 · p. 11, Figure 8 · p. 11, §4.2 · p. 12

4.3 Ablation Studies

  • Adding SA-BIQA guidance improves multitasking restoration, while the Structural-Correction Module yields gains and preserves fine structures versus original latent-diffusion restoration.The ablation is reported on deraindrop and dehazing tasks.Source: §4.3 · p. 13, Table 5 · p. 14
  • Pretrained AIR adapts faster to de-snowing than AIR without pretrained weights and outperforms NAFNet trained specifically for de-snowing.The unseen de-snowing and JPEG experiments fine-tune AIR initialized from weights trained on seven restoration tasks.Source: §4.3 · pp. 13–14, Figure 10 · p. 13

5 Conclusions and Discussions

AutoDIR combines automatic restoration of multiple unknown degradations with open-vocabulary editing and is presented as a foundation model for image restoration. Its current computational cost and emphasis on global restoration are limitations.

5 Conclusions and Discussions

  • AutoDIR automatically restores images with multiple unknown degradations and supports open-vocabulary image-editing instructions.The authors describe it as a foundation model trained jointly from multiple image-restoration datasets, with effective few-shot learning for novel tasks.Source: §5 · p. 14
  • The authors describe AutoDIR as a foundation for work on complex real-world images, with potential applications in local enhancement and multi-task video restoration.These applications are stated as potential extensions.Source: §5 · p. 14
  • AutoDIR has computational costs similar to diffusion-based restoration methods and focuses more on global restoration than local editing.The authors note that its computational cost is often significantly higher than non-generative networks, and identify local editing as future work.Source: §5 · p. 14

A Open-Vocabulary User Interaction

Open-vocabulary text prompts let users iteratively customize image edits to their preferences.

A Open-Vocabulary User Interaction

  • Users can edit an image with successive open-vocabulary instructions, using each edited result as the input for the next edit.The figure describes the original image, the first edit, and a second edit based on the first.Source: A Open-Vocabulary User Interaction · p. 15

B Trade-off of adjustable coefficient w

The adjustable structural-correction weight trades recovery of image structure against latent diffusion’s generation capability.

B Trade-off of adjustable coefficient w

  • A larger structural-correction weight helps recover complex original structures, while a smaller weight is needed in tasks such as super-resolution to maintain generation capability.The figure illustrates the opposing effects across its two rows.Source: B Trade-off of adjustable coefficient w · p. 15

C Extensive Experimental Results

The supplementary results provide visualizations for unseen real-world images with multiple degradations and for seven joint-learned restoration tasks. They also illustrate user editing and the structural-correction trade-off.

C Extensive Experimental Results

  • The user-editing figure presents an original image followed by two edits, each guided by an open-vocabulary instruction.The second instruction is applied to the first edited image.Source: Figure 11 · p. 16
  • A larger structural-correction weight favors recovering original structure, whereas a smaller weight favors the latent diffusion model’s generation capability.The figure illustrates these opposing effects in its first and second rows, respectively.Source: Figure 12 · p. 17

C.1 Results on Images with Multiple Unknown Degradations in Unseen Real-world Datasets

  • Additional visualizations on UCD, EVUP, LOL-Blur, and RainDS show AutoDIR results for multiple unknown degradations in unseen real-world datasets.The authors present these visualizations to further demonstrate performance in complex scenarios.Source: §C.1 · p. 17

C.2 Results on Seven Joint-learned Tasks

  • Additional visualizations cover seven joint-learned tasks: denoising, super-resolution, deblurring, deraining, dehazing, low-light enhancement, and deraindrop.The supplementary material also provides zoom-in visualizations.Source: §C.2 · p. 17

C.3 Ablation Studies

The ablations examine degradation detection and structural correction, while implementation details describe training and task settings. In an unseen-real-image user study, AutoDIR received more than 96% of votes for best denoising results.

C.3 Ablation Studies

  • Semantic-agnostic fine-tuning shifted SA-BIQA attention from foreground objects to background areas that may contain artifacts.The authors report that this focus supports more accurate BIQA results.Source: §C.3 · p. 17, Figure 26 · p. 31
  • Structural correction preserved fine details and reduced face and text distortion; it also corrected undesirable hallucinated textures in latent-diffusion results.The comparisons cover dehazing, deraindrop, and denoising.Source: §C.3 · p. 31, Figure 27 · p. 33, Figure 28 · p. 33
  • SA-CLIP outperformed ViT Classifier, pre-trained CLIP, and naively fine-tuned CLIP in F-scores across all seven degradation tasks.Table 6 reports F-scores for seven degradation tasks and clean images.Source: §C.3 · p. 31, Table 6 · p. 31

D Implementation Details

  • Training proceeds through BIQA image-encoder fine-tuning, AIR-backbone fine-tuning, and separate structural-correction-module training.Inference uses correction coefficient 1 for denoising, deraining, dehazing, deraindrop, low-light enhancement, and deblurring, and 0.1 for super-resolution.Source: D Implementation Details · p. 32
  • The experiments cover seven restoration tasks, with unseen-degradation tests on TOLED and EUVP datasets.The seven tasks are denoising, deblurring, super-resolution, low-light enhancement, dehazing, deraining, and deraindrop.Source: D Implementation Details · p. 32

E User Study

  • More than 96% of 616 user-study responses favored AutoDIR for best denoising results on 28 images with unknown artifacts.The comparison included AirNet and PromptIR.Source: E User Study · p. 32, Figure 30 · p. 34

F Comparison on unseen real-world Super-res and Denoise dataset

AutoDIR is evaluated on unseen real-world super-resolution and denoising datasets. The reported comparisons include qualitative results and quantitative tables.

F Comparison on unseen real-world Super-res and Denoise dataset

G Evaluation of SA-BIQA on unseen real-world datasets.

On unseen real-world super-resolution, deblur, and low-light enhancement datasets, SA-BIQA reportedly predicted the dominant artifact with over 90% accuracy.

G Evaluation of SA-BIQA on unseen real-world datasets

  • SA-BIQA achieved over 90% accuracy in predicting the dominant artifact across unseen real-world super-resolution, deblur, and low-light enhancement tasks.The accompanying table headers label the tasks low-res, blur, and low-light.Source: G Evaluation of SA-BIQA on unseen… · p. 35

Related papers

Loading 2310.10123v5…