Source-linked AI summary
Degradation-Aware Feature Perturbation for All-in-One Image Restoration
Xiangpeng Tian, Xiangyu Liao, Xiao Liu, Meng Li, Chao Ren
TL;DR
All-in-one restoration must handle diverse degradations with shared parameters, which can create task interference. DFPIR reshapes features through degradation-guided channel shuffling and attention masking in a perturbation block between encoding and decoding. The method reports state-of-the-art performance across multiple restoration tasks, including a five-task average improvement over prior methods.
Problem
Shared parameters make universal restoration difficult because different degradation tasks can have inconsistent or opposing gradient update directions.
Method
DFPIR uses degradation-guided channel-wise and attention-wise feature perturbations through a Degradation-Guided Perturbation Block between encoder and decoder stages.
Results
DFPIR achieves state-of-the-art performance across several all-in-one restoration tasks and improves average performance by 0.45 dB over InstructIR across five tasks.
Takeaways & Limitations
Degradation-aware perturbations provide a unified restoration framework that preserves image features while reducing mutual influence among degradation types.
Takeaways & Limitations
Parameter-space approaches can require many additional degradation parameters or more complex network structures, increasing computational complexity.
Abstract
from arXiv · showhide
All-in-one image restoration aims to recover clear images from various degradation types and levels with a unified model. Nonetheless, the significant variations among degradation types present challenges for training a universal model, often resulting in task interference, where the gradient update directions of different tasks may diverge due to shared parameters. To address this issue, motivated by the routing strategy, we propose DFPIR, a novel all-in-one image restorer that introduces Degradation-aware Feature Perturbations(DFP) to adjust the feature space to align with the unified parameter space. In this paper, the feature perturbations primarily include channel-wise perturbations and attention-wise perturbations. Specifically, channel-wise perturbations are implemented by shuffling the channels in high-dimensional space guided by degradation types, while attention-wise perturbations are achieved through selective masking in the attention space. To achieve these goals, we propose a Degradation-Guided Perturbation Block (DGPB) to implement these two functions, positioned between the encoding and decoding stages of the encoder-decoder architecture. Extensive experimental results demonstrate that DFPIR achieves state-of-the-art performance on several all-in-one image restoration tasks including image denoising, image dehazing, image deraining, motion deblurring, and low-light image enhancement. Our codes are available at https://github.com/TxpHome/DFPIR.
1. Introduction
All-in-one restoration reduces the need for separate degradation-specific networks but faces interference from shared parameters and task differences. DFPIR addresses this through degradation-guided channel and attention perturbations that reshape features for unified restoration.
- Separate networks for each degradation type achieve strong results but increase model size and computational demands.
- Shared parameters in all-in-one methods can produce inconsistent or opposing gradient directions across degradation tasks.
- Existing all-in-one methods modify either parameter space or feature space, with both approaches incorporating degradation information to mitigate task interference.
- DFPIR adjusts feature space using degradation-guided channel-wise and attention-wise perturbations aligned with unified network parameters.
- Channel shuffling preserves inherent image features while reducing degradation-feature influence, and selective attention masking further limits cross-degradation interference.
2. Related Works
Related work progresses from single-degradation restoration toward unified multi-task restoration. The central challenge is balancing degradation-specific interference reduction with preservation of inherent image features.
- Single Task Image Restoration: Single-task restoration models have advanced substantially but remain primarily designed for single-degradation scenarios.
- All-In-One Image Restoration: Multi-task restoration uses one network design to address multiple tasks, making interference reduction and feature preservation central challenges.
- All-In-One Image Restoration: One established direction modifies the parameter space to accommodate different degradation types.
3. Proposed Method
DFPIR adjusts encoded features before shared decoding by applying degradation-guided channel and attention perturbations. Its DGPB uses high-dimensional channel shuffling and top-K attention masking to reduce cross-degradation interference while preserving image information.
- Motivation: DFPIR targets mutual influence among degradation features in a shared-parameter all-in-one restoration model.Different degradation tasks can produce inconsistent or opposing gradient update directions under shared parameters.
- Overall Pipeline: DFPIR uses Restormer as an encoder-decoder backbone and inserts DGPB between encoding and decoding in the skip-connection stage.The block perturbs encoded features before they are aligned with the shared decoder.
- Degradation-Guided Perturbation Block: DGPB combines DGCPM channel shuffling and CAAPM top-K attention masking, both guided by degradation-type prompts.The prompts are obtained by encoding textual degradation descriptions with a pre-trained CLIP model.
- Degradation-Guided Channel Perturbation Module: DGCPM doubles feature channels, performs degradation-guided top-K channel shuffling, then halves the channels to preserve the original channel count.The high-dimensional perturbation is designed to avoid excessive disruption and difficult convergence from directly shuffling low-dimensional features.
- Experimental Analysis: On three tasks, DFPIR gains 0.45 dB average PSNR over InstructIR, while its perturbed features show tighter task-specific t-SNE clustering than PromptIR.The reported multi-task comparison uses PSNR and SSIM on full RGB images; the clustering observation is made under the three-task setting.
- Channel-Adapted Attention Perturbation Module: CAAPM combines shuffled-feature queries with original-feature keys and values, applying a selection mask to create perturbed attention features before feed-forward processing.This interaction preserves original feature information while adapting attention to degradation-specific features.
4. Experiments
DFPIR is evaluated in unified restoration settings spanning three and five degradation tasks, with quantitative, visual, and ablation studies supporting its effectiveness.
- Experimental Settings: Experiments use PSNR and SSIM under all-in-one settings covering three and five degradation types, alongside separate single-task models.The three-task setting evaluates dehazing, deraining, and denoising; the five-task setting adds deblurring and low-light enhancement.
- Three-Task Results: 0.45 dB average improvement over InstructIR and 0.82 dB over PromptIR are reported across three restoration tasks.DFPIR also improves over InstructIR by 0.67 dB on deraining and 1.65 dB on dehazing.
- Five-Task Results: 1.09 dB average improvement over InstructIR is reported across five tasks, while DFPIR exceeds Restormer and NAFNet by 3.04 dB and 2.88 dB in average PSNR.The five tasks are dehazing, deraining, denoising, deblurring, and low-light enhancement.
- Visual Results: Visual comparisons report stronger challenging-case dehazing, deraining results closer to ground truth, and more detail recovery for heavily degraded noisy inputs.The visual analysis covers dehazing, deraining, and denoising.
- Visual Results: Feature visualizations indicate that DGCPM suppresses degradation-specific features, while DGCPM with CAAPM further enhances intrinsic details and suppresses degradation features.Channel shuffling changes task-specific channel order, and the combined perturbations are reported to improve feature separation.
- Ablation Studies: Average PSNR rises from 32.49 to 32.88 with DGCPM and CAAPM, while channel shuffle outperforms direct channel attention by 0.15 dB.The ablation attributes the strongest result to the combined channel- and attention-wise perturbations.
- Ablation Studies: Attention perturbation requires a balanced γ because excessive masking increases information loss, whereas insufficient perturbation leaves stronger cross-task interference.The reported non-optimal settings are γ = 0.5 and γ = 1.0.
5. Conclusion
DFPIR aligns feature representations with a unified parameter space through degradation-guided channel and attention perturbations. Its DGPB integrates these operations between encoding and decoding, and the ablation framing highlights their combined design.
- Conclusion: DFPIR introduces degradation-aware feature perturbations to align the feature space with a unified parameter space.The framework targets all-in-one image restoration.
- Conclusion: The ablation study reports separate effects for key components and γ, with the component comparison using average PSNR and SSIM on three tasks.Table 3 reports average PSNR in dB and SSIM for the ablations.
- Conclusion: Channel-wise perturbation shuffles channels in high-dimensional space, while attention-wise perturbation selectively masks attention features.Both operations are dynamically guided by degradation type prompts.
- Conclusion: The Degradation-Guided Perturbation Block integrates channel shuffling and attention masking between the encoder and decoder.The block is designed to realize both perturbation mechanisms within the encoder-decoder architecture.