PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlling Vision-Language Models for Multi-Task Image Restoration

Ziwei Luo, Fredrik K. Gustafsson, Zheng Zhao, Jens Sjölund, Thomas B. Schön

arXiv:2310.01018v2cs.CV

TL;DR

Low-level image restoration exposes a gap in pretrained VLMs because corrupted inputs misalign visual features with degradation descriptions. DA-CLIP adds a controller that predicts degradation and high-quality content embeddings, integrates them into restoration networks, and is reported to improve both degradation-specific and unified tasks. The method also introduces a ten-task mixed-degradation dataset with synthetic captions, while increasing memory requirements despite nearly unchanged test-time computation.

  • Problem

    Pretrained VLMs have limited impact on image restoration because corrupted inputs create feature mismatches across fine-grained degradation types, while unified restoration methods cover few degradation types and datasets.

  • Method

    DA-CLIP trains an image controller around fixed CLIP encoders to predict degradation embeddings and high-quality content embeddings, then integrates them into restoration networks through cross-attention and prompting.

  • Results

    DA-CLIP consistently improves restoration performance across degradation-specific and unified tasks, with the reported degradation-specific evaluation achieving the best perceptual results across tasks and state-of-the-art results for all metrics on deraining.

  • Takeaways & Limitations

    The results support using degradation-aware control of pretrained vision-language representations as a framework for multi-task image restoration.

  • Takeaways & Limitations

    DA-CLIP significantly increases memory requirements compared with baseline models, although test-time FLOPs and runtime are virtually unaffected; mixed-degradation language effects also remain difficult to justify because of a shadowed raindrop example.

Abstract

from arXiv · show

Vision-language models such as CLIP have shown great impact on diverse downstream tasks for zero-shot or label-free predictions. However, when it comes to low-level vision such as image restoration their performance deteriorates dramatically due to corrupted inputs. In this paper, we present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks as a multi-task framework for image restoration. More specifically, DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself will also output a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. In addition, we construct a mixed degradation dataset with synthetic captions for DA-CLIP training. Our approach advances state-of-the-art performance on both \emph{degradation-specific} and \emph{unified} image restoration tasks, showing a promising direction of prompting image restoration with large-scale pretrained vision-language models. Our code is available at https://github.com/Algolzw/daclip-uir.

1 INTRODUCTION

Existing vision-language models struggle with low-level image restoration because corrupted inputs obscure fine-grained degradation differences. DA-CLIP addresses this gap with degradation-aware control, mixed-degradation training data, and a unified restoration framework.

  • 1 INTRODUCTION: Figure 1 illustrates a single unified model producing the paper’s restoration results while predicting degradation embeddings and high-quality features.The figure contrasts the approach with CLIP and presents its multi-task restoration behavior.
  • 1 INTRODUCTION: Existing VLMs often misalign corrupted image features with degradation texts because they do not capture fine-grained differences between degradation types.This limitation reduces their impact on low-level vision tasks such as image restoration.
  • 1 INTRODUCTION: Unified restoration models reduce repeated task-specific training but remain limited to a small number of degradation types and datasets, without using information embedded in VLMs.DA-CLIP is positioned as a way to incorporate large-scale pretrained vision-language knowledge into this setting.
  • 1 INTRODUCTION: DA-CLIP adapts pretrained CLIP for both degradation-specific and unified image restoration through an image controller that predicts degradation and high-quality content embeddings.The framework integrates content embeddings into restoration networks and uses degradation information for unified restoration.
  • 1 INTRODUCTION: The proposed framework constructs a mixed-degradation dataset with ten restoration tasks and synthetic captions for training DA-CLIP or unified restoration models.The dataset supports learning both high-quality features and degradation types from low-quality inputs.

2 BACKGROUND AND RELATED WORK

Image restoration research spans task-specific and unified approaches, while pretrained VLMs provide aligned visual-text representations and synthetic-caption training. DA-CLIP connects these directions by using a controlled VLM for restoration.

  • 2 BACKGROUND AND RELATED WORK: Unified image restoration uses one model across multiple degradation types, but existing methods cover few tasks and omit auxiliary degradation information.These methods use additional encoders or visual prompts to implicitly cluster inputs by degradation.
  • 2 BACKGROUND AND RELATED WORK: DA-CLIP’s method overview freezes CLIP’s text and image encoders, trains an image controller contrastively, and adds cross-attention and degradation prompting to restoration models.The controller predicts degradation features and controls the image encoder to produce high-quality content features.
  • 2 BACKGROUND AND RELATED WORK: Blind image restoration addresses unknown degradation levels using practical GAN-based degradation settings or diffusion priors to generate realistic outputs.The cited approaches include BSRGAN, Real-ESRGAN, StableSR, and DiffBIR.
  • 2 BACKGROUND AND RELATED WORK: VLMs commonly learn aligned image-text features with contrastive learning, while BLIP improves data quality by bootstrapping synthetic captions from noisy web data.These properties motivate using pretrained multimodal representations and synthetic captions for restoration.
  • 2 BACKGROUND AND RELATED WORK: Text-to-image systems have introduced task-specific controls and instruction-based image-to-image translation on top of diffusion models.ControlNet adds conditioning controls, while InstructPix2Pix combines language instructions with image translation.

3 DEGRADATION-AWARE CLIP

DA-CLIP controls a fixed CLIP image encoder to produce clean-content and degradation embeddings from corrupted inputs. Cross-attention integrates content guidance into restoration networks, while degradation-aware prompt learning supports unified restoration.

  • 3 DEGRADATION-AWARE CLIP: DA-CLIP controls a pretrained CLIP encoder to extract high-quality content features and predict the input degradation type.The controller aligns image-based embeddings from low-quality inputs with clean-caption and real-degradation text embeddings.
  • 3.1 IMAGE CONTROLLER: The controller is a CLIP image-encoder copy with zero-initialized controls that modify transformer-block outputs during training.Its outputs include degradation embeddings and hidden controls added to corresponding encoder blocks.
  • 3.1 IMAGE CONTROLLER: The joint contrastive objective aligns both content and degradation image embeddings with their paired text embeddings while separating mismatched pairs.The CLIP weights remain frozen, and only the image controller is fine-tuned.
  • 3.2 IMAGE RESTORATION WITH DA-CLIP: Cross-attention injects clean-content embeddings into the image-restoration network, while degradation embeddings and prompt learning target degradation classification in unified restoration.The base restoration framework uses an IR-SDE U-Net without self-attention layers.
  • 3.2 IMAGE RESTORATION WITH DA-CLIP: BLIP-generated captions from high-quality images are combined with low-quality images and degradation types to provide training tuples.The clean captions are accurate but do not encode degradation information.

4 DATASET CONSTRUCTION

The dataset combines ten image degradation types with high-quality images and synthetic captions for training DA-CLIP. Its training and testing composition is summarized by degradation type in Table 1.

  • 4 DATASET CONSTRUCTION: The collected dataset covers ten degradation types: blurry, hazy, JPEG-compression, low-light, noisy, raindrop, rainy, shadowed, snowy, and inpainting.Examples of the ten degradations are shown in Figure 1.
  • 4 DATASET CONSTRUCTION: BLIP generates synthetic captions for high-quality images, which are paired with low-quality images and their corresponding degradation types.These image-text-degradation tuples support DA-CLIP training on the mixed-degradation dataset.
  • 4 DATASET CONSTRUCTION: Table 1 summarizes the training and testing image counts for each degradation type.The table provides dataset details organized by degradation category.

5 EXPERIMENTS

Across degradation-specific and unified restoration, DA-CLIP improves restoration models through predicted content and degradation embeddings, with gains also shown in classification and model variants.

  • 5.1 DEGRADATION-SPECIFIC IMAGE RESTORATION: DA-CLIP achieves the best perceptual results across four degradation-specific tasks and consistently improves IR-SDE on every dataset and metric.It also sets a new state-of-the-art for all reported metrics on image deraining.
  • 5.2 UNIFIED IMAGE RESTORATION: DA-CLIP obtains the best perceptual results across ten unified degradations, significantly outperforming IR-SDE on all four metrics while retaining good distortion performance.On JPEG and noise removal, outputs are realistic but somewhat noisy, producing lower distortion metrics for those tasks.
  • 5.2 UNIFIED IMAGE RESTORATION: NAFNet with DA-CLIP surpasses PromptIR across all metrics, showing compatibility with both diffusion-based and direct restoration models.Adding degradation context significantly improves the NAFNet results.
  • 5.2 UNIFIED IMAGE RESTORATION: The controller perfectly predicts every degradation except blurry, where it reaches 91.6% accuracy, unlike original CLIP's near-zero recognition on several degradations.Original CLIP achieves below 2% accuracy for noisy and raindrop images and 0% for inpainting.
  • 5.3 DISCUSSION AND ANALYSIS: Ablations show that either HQ content or degradation embeddings improves unified restoration, while combining both yields further gains and ground-truth embeddings provide an upper bound.Using original CLIP content embeddings fails to substantially improve unified restoration, whereas DA-CLIP clearly outperforms that baseline in degradation-specific restoration.
  • 5.3 DISCUSSION AND ANALYSIS: DA-CLIP increases memory requirements compared with NAFNet and IR-SDE, although test-time FLOPs and runtime are virtually unaffected.This is identified as a limitation of the method's model complexity and computational cost.

6 CONCLUSION

DA-CLIP provides a universal framework for image restoration by combining pretrained vision-language models with degradation-aware control, prompt learning, and cross-attention. It improves restoration across degradation-specific and unified tasks, while remaining limited by mixed-degradation data and real-world robustness challenges.

  • DA-CLIP consistently improves restoration performance across degradation-specific and unified tasks and a variety of degradation types.
  • The controller predicts degradation embeddings and clean content embeddings, while prompt learning and cross-attention integrate these signals into restoration models.
  • The current dataset makes it difficult to restore multiple degradations in the same scene, motivating more robust models for real-world captured photos.

A MORE DETAILS ABOUT DATASETS

The mixed degradation dataset covers ten image restoration tasks using collected datasets and synthetic corruptions, with visual examples illustrating the degradation types. DA-CLIP is trained on normalized 224 × 224 inputs with a pretrained ViT-B-32 CLIP encoder.

  • The dataset spans 10 restoration tasks, including blur, haze, JPEG compression, low light, noise, raindrops, rain, shadows, snow, and inpainting.
  • The component datasets provide task-specific training and testing images, including synthetic JPEG compression with quality factor 10 and Gaussian noise with noise level 50.
  • Figure 10 provides visual examples for the ten degradations and their corresponding datasets.
  • DA-CLIP uses a ViT-B-32 image encoder pretrained on LAION-2B and trains for 50 epochs on four NVIDIA A100 GPUs after normalization and resizing to 224 × 224.

B.2 ADDITIONAL ANALYSIS OF DA-CLIP

Additional analyses show that DA-CLIP benefits from zero-initialized controller layers, generalizes to unseen light rain, and improves both diffusion-based and MSE-based restoration models. The method also supports unified restoration through degradation classification and prompting.

  • Fine-tuned CLIP improves degradation classification over direct retraining, but cannot predict high-quality content embeddings from low-quality inputs for downstream restoration.
  • Zero-initializing the controller’s dense layers further improves classification accuracy across datasets without additional cost.
  • DA-CLIP is evaluated in unified and degradation-specific settings, including integration into NAFNet and analysis on multiple restoration tasks.
  • DA-CLIP surpasses all other unified approaches by a significant margin on the unseen Rain100L light-rain dataset.
  • DA-CLIP’s degradation context benefits NAFNet and Restormer, while adding content embeddings further improves their unified restoration performance.

B.5 TRAINING CURVES ON SINGLE DEGRADATION TASKS

Training-curve analyses show that adding DA-CLIP embeddings improves the baseline model across four degradation-specific restoration tasks. Larger patches also improve unified-task training, while the added modules increase memory requirements without materially changing test-time computation.

  • DA-CLIP embeddings produce better training than IR-SDE on deraining, low-light enhancement, deblurring, and dehazing tasks.
  • DA-CLIP increases model memory requirements compared with NAFNet and IR-SDE, while FLOPs and runtime remain virtually unaffected.
  • Increasing the patch size from 128 × 128 to 256 × 256 improves the unified restoration training process.

C.1 DETAILED QUANTITATIVE RESULTS FOR UNIFIED IMAGE RESTORATION

The paper reports detailed unified image-restoration comparisons across four metrics and supplements the quantitative results with visual examples, including real-world and compressed or blurry images.

  • C.1 DETAILED QUANTITATIVE RESULTS FOR UNIFIED IMAGE RESTORATION: Figures 14 and 16 provide additional visual comparisons between the proposed method and other unified image-restoration approaches.
  • C.1 DETAILED QUANTITATIVE RESULTS FOR UNIFIED IMAGE RESTORATION: Figure 15 compares the proposed method with Real-ESRGAN and StableSR on compressed and blurry images.
  • C.1 DETAILED QUANTITATIVE RESULTS FOR UNIFIED IMAGE RESTORATION: Figure 17 illustrates tests on real-world images, while the comparisons include PromptIR, IR-SDE, and AirNet.

D LIMITATION ON MIXED DEGRADATIONS

The mixed-degradation setup assigns one degradation label per image, so the model is not trained for scenes containing multiple degradations. The authors identify this as limiting both restoration coverage and evaluation of linguistic components, while proposing broader real-world and linguistic extensions for future work.

  • D LIMITATION ON MIXED DEGRADATIONS: A single degradation label per image prevents the current model from being trained to restore multiple degradations in the same scene.The stated scope excludes mixed cases such as an image containing both raindrops and shadows.
  • D LIMITATION ON MIXED DEGRADATIONS: In a raindrop example containing a shadow area, the model removes the raindrop degradation but not the shadow.
  • D LIMITATION ON MIXED DEGRADATIONS: The limitation prevents the paper from fully exploring and justifying linguistic components for mixed degradations.
  • D LIMITATION ON MIXED DEGRADATIONS: Future work targets practical handling of real-world and more complex degradations, alternative pretrained VLM backbones, and instruction-based image restoration.
Loading 2310.01018v2…