Source-linked AI summary
AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and Modulation
Yuning Cui, Syed Waqas Zamir, Salman Khan, Alois Knoll, Mubarak Shah, Fahad Shahbaz Khan
TL;DR
Existing restoration methods either specialize in particular degradations or overlook the distinct frequency effects of degradation types in all-in-one settings. AdaIR adaptively mines and modulates frequency components using degradation-guided spectral information. It reports state-of-the-art performance across five restoration tasks, including consistent gains over competing all-in-one methods in evaluated settings.
Problem
Existing task-specific methods generalize poorly across degradation types, while all-in-one approaches operate purely in the spatial domain despite degradations affecting different frequency subbands.
Method
AdaIR uses adaptive frequency mining guided by degraded-image spectra, bidirectional frequency modulation, and feature merging within an all-in-one restoration network.
Results
AdaIR achieves state-of-the-art performance across denoising, dehazing, deraining, motion deblurring, and low-light enhancement, with a 0.63 dB average PSNR gain over PromptIR across three evaluated tasks and settings.
Takeaways & Limitations
The framework adaptively emphasizes informative frequency subbands according to the degradation present in the input image.
Abstract
from arXiv · showhide
In the image acquisition process, various forms of degradation, including noise, haze, and rain, are frequently introduced. These degradations typically arise from the inherent limitations of cameras or unfavorable ambient conditions. To recover clean images from degraded versions, numerous specialized restoration methods have been developed, each targeting a specific type of degradation. Recently, all-in-one algorithms have garnered significant attention by addressing different types of degradations within a single model without requiring prior information of the input degradation type. However, these methods purely operate in the spatial domain and do not delve into the distinct frequency variations inherent to different degradation types. To address this gap, we propose an adaptive all-in-one image restoration network based on frequency mining and modulation. Our approach is motivated by the observation that different degradation types impact the image content on different frequency subbands, thereby requiring different treatments for each restoration task. Specifically, we first mine low- and high-frequency information from the input features, guided by the adaptively decoupled spectra of the degraded image. The extracted features are then modulated by a bidirectional operator to facilitate interactions between different frequency components. Finally, the modulated features are merged into the original input for a progressively guided restoration. With this approach, the model achieves adaptive reconstruction by accentuating the informative frequency subbands according to different input degradations. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on different image restoration tasks, including denoising, dehazing, deraining, motion deblurring, and low-light image enhancement. Our code is available at https://github.com/c-yn/AdaIR.
1 Introduction
Image restoration seeks clean images from degraded inputs, but task-specific models generalize poorly and existing all-in-one methods overlook frequency-domain differences. AdaIR addresses this with adaptive frequency mining and modulation, achieving state-of-the-art results across several restoration tasks.
- Image restoration generates high-quality clean images by removing degradations such as noise, haze, blur, and rain.
- Task-specific methods often fail to generalize across degradation types and levels, while separate models are costly and difficult to deploy on resource-constrained devices.
- Different degradations affect distinct frequency subbands, with noise and rain contaminating high frequencies and low-light and haze affecting low frequencies.
- AdaIR mines low- and high-frequency signals using adaptive spectra decomposition, modulates their interactions bidirectionally, and merges them into the original features for restoration.
- AdaIR learns discriminative degradation contexts more effectively and achieves state-of-the-art performance on denoising, dehazing, deraining, motion deblurring, and low-light enhancement.
2 Related Work
Related work includes task-specific restoration methods and unified all-in-one models. Existing unified approaches either require degradation-type information or learn degradation representations and integrate restoration knowledge within a single model.
- Single-Task Image Restoration: Single-task restoration methods span denoising, dehazing, deraining, and motion deblurring, but lack generalization beyond particular degradation types and levels.
- All-in-One Image Restoration: Early all-in-one models use separate encoder and decoder heads but require prior knowledge of the degradation type to select the restoration pathway.
- All-in-One Image Restoration: Blind all-in-one methods learn degradation representations from corrupted images or organize task-oriented knowledge collection and progressive restoration.
3 Method
AdaIR uses a Transformer encoder-decoder with Adaptive Frequency Learning Blocks to separate degradation-related frequency content and progressively restore clean images. Its modules adaptively mine low- and high-frequency features, exchange complementary information between them, and refine spatial features.
- Overall Pipeline: AdaIR processes degraded images through a four-level Transformer encoder-decoder and inserts AFLBs between decoder levels for progressive reconstruction.The encoder transforms high-resolution features into a lower-resolution latent representation, while the decoder reconstructs high-resolution output.
- Progressive Restoration: The modulated frequency features are merged into the original input features to accentuate informative subbands for different degradation types.AFLB is designed to adaptively segregate degradation content from clean image content in the frequency domain before spatial-domain refinement.
- Adaptive Frequency Learning Block: Each AFLB uses FMiM to extract low- and high-frequency representations from input features guided by the degraded image’s adaptively decoupled spectra.FMiM performs domain transformation, mask generation, and feature extraction, using FFT-derived spectral guidance.
- Frequency Mining Module: The Mask Generation Block produces a degradation-adaptive frequency boundary whose low- and high-frequency masks separate spectral components before inverse Fourier transformation.Two factors ranging from 0 to 1 determine mask size, and the high-frequency mask covers the remaining spectral region.
- Frequency Modulation Module: FMoM exchanges complementary information across branches, using high-frequency details to enrich low-frequency features and low-frequency global information to enhance high-frequency features.The H-L unit produces spatial attention from high-frequency features, while the L-H unit generates channel attention from low-frequency features.
4 Experiments
Experiments evaluate AdaIR under all-in-one and single-task settings across multiple restoration tasks, with ablations examining its frequency components, input guidance, generalization, and computational behavior.
- AdaIR is evaluated under all-in-one and single-task protocols, using unified or task-specific models respectively.
- All-in-One Results: 0.63 dB PSNR over PromptIR and 1.49 dB over AirNet are AdaIR’s average gains across three all-in-one restoration tasks.
- All-in-One Results: 2.27 dB on deraining and 0.48 dB on dehazing are AdaIR’s gains over PromptIR in the three-degradation setting.
- Single-Task Results: 0.49 dB over PromptIR on dehazing, 8.62 dB over AirNet on dehazing, and 1.86 dB over PromptIR on deraining are reported in single-task tests.
- Five-Degradation Results: 1.86 dB averaged across five restoration tasks is AdaIR’s gain over IDR, with particularly large improvement on dehazing.
- Ablation Studies: 3.03 dB over the baseline is achieved by the full AdaIR design, with 2.64M additional parameters and 6.21 GFlops overhead.
- Ablation Studies: Replacing degraded-image spectral guidance with input embedding features reduces performance from 30.52 dB to 29.29 dB.
- Generalization: AdaIR reports favorable results on unseen desnowing and mixed degradations when directly tested after three-task training.
5 Conclusion
AdaIR adaptively restores images by identifying and refining degradation-relevant frequency components within an all-in-one Transformer framework.
- AdaIR combines frequency mining and modulation to identify and enhance frequency components associated with different degradation patterns.
- Integrated into a U-shaped Transformer backbone, AdaIR achieves state-of-the-art performance across multiple image restoration tasks.
A Additional Ablation Studies
Additional ablations study AFLB placement, FMoM design, degradation combinations, and Transformer-block composition.
- AFLB Placement: Using AFLBs in both encoder and decoder performs worse than integrating them solely into the decoder.
- AFLB Placement: 29.58 dB with one AFLB is below the best result obtained by placing AFLBs between every consecutive decoder level.
- FMoM Design: The proposed FMoM exchanges complementary information across frequency features, outperforming spatial attention that does not discriminate frequency inputs.
- FMoM Design: 29.67 dB PSNR for spatial attention is 0.85 dB below the proposed FMoM in single-task dehazing.
- Degradation Combinations: Adding more degradation types makes restoration increasingly difficult, with hazy images causing a larger performance drop than rainy or noisy images.
- Transformer Block: Transformer blocks in AdaIR comprise MDTA and GDFN components.
B Computational Comparisons
Computational comparisons indicate that AdaIR offers a favorable accuracy–complexity tradeoff among all-in-one restoration methods.
- AdaIR strikes a better tradeoff between accuracy and complexity than competing all-in-one methods.
C Transformer Block in the AdaIR Framework
The AdaIR framework uses Transformer Blocks built from transposed attention and a gated feed-forward network. These components normalize features, apply convolutional projections, and combine branch outputs for restoration processing.
- Transformer Block Components: Each Transformer Block comprises multi-dconv head transposed attention and a gated-dconv feed-forward network.The two components are applied successively.
- Multi-Dconv Head Transposed Attention: MDTA normalizes input features, generates query, key, and value projections, and forms a C × C transposed-attention map using Softmax-normalized dot products.The projections use combinations of 1 × 1 and 3 × 3 depth-wise convolutions.
- Multi-Dconv Head Transposed Attention: The MDTA output adds the attention result to the original input through a residual connection.The attention computation uses reshaped query, key, and value tensors.
- Gated-Dconv Feed-Forward Network: GDFN normalizes features, expands channels through two convolutional branches, applies depth-wise convolutions, and multiplies the branches after GELU activation in one branch.The two branches are combined element-wise.
- Gated-Dconv Feed-Forward Network: Within GDFN, layer normalization, element-wise multiplication, depth-wise convolution, and GELU are denoted by LN, ⊙, DW 3×3, and ϕ, respectively.These symbols specify the operations used in the feed-forward computation.
D Additional Visual Results
Additional visual results compare AdaIR with other methods across all-in-one and single-task deraining, dehazing, and denoising settings. The reported visual examples also include degradation-context separation and PSNR comparisons.
- Five-Degradation Setting: Under the five-degradation setting, t-SNE features discriminate contexts for five degradation types, with low-light enhancement closer to dehazing than to other clusters.The proximity is consistent with both degradations mainly affecting low-frequency image content.
- Additional Visual Comparisons: Additional qualitative results cover deraining, dehazing, and denoising in both all-in-one and single-task settings.The comparisons include Rain100L, SOTS, and BSD68 with σ = 50.
- All-in-One Visual Results: 36.31 dB PSNR is reported for the referenced comparison against AirNet and PromptIR in one visual restoration example.The displayed values also include 16.90 dB, 36.02 dB, and 34.08 dB for the degraded input, AirNet, and PromptIR, respectively.
- All-in-One Visual Results: 35.57 dB PSNR is reported for the displayed rainy-image comparison against AirNet.The corresponding displayed values are 21.61 dB for the rainy image and 31.30 dB for AirNet.
- All-in-One Visual Results: 29.61 dB PSNR is reported for the displayed hazy-image comparison against AirNet.The corresponding displayed values are 10.86 dB for the hazy image and 16.98 dB for AirNet.
- All-in-One Visual Results: 28.69 dB PSNR is reported for the displayed noisy-image comparison, alongside 14.49 dB and 27.49 dB values.The passage does not identify the associated methods for these displayed values.