Source-linked AI summary
Multi-Stage Progressive Image Restoration
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao
TL;DR
Image restoration must recover clean images while balancing contextual understanding and spatial detail, but existing multi-stage designs can lose information or rely on weak stage interactions. MPRNet progressively combines contextual encoder-decoder processing, a high-resolution branch, supervised attention, and cross-stage feature fusion. It achieves strong gains across ten datasets and multiple restoration tasks while remaining lightweight and efficient.
Problem
Image restoration requires balancing high-level contextual information with spatial details, while existing multi-stage designs may lose information through inadequate stage interactions.
Method
MPRNet uses progressive multi-stage restoration with encoder-decoder stages, a high-resolution final stage, supervised attention, and cross-stage feature fusion.
Results
MPRNet delivers strong performance gains and new state-of-the-art results across ten datasets covering image deraining, deblurring, and denoising.
Takeaways & Limitations
The tightly interlinked architecture provides contextually enriched and spatially accurate restoration while maintaining low model size and efficient runtime.
Abstract
from arXiv · showhide
Image restoration tasks demand a complex balance between spatial details and high-level contextualized information while recovering images. In this paper, we propose a novel synergistic design that can optimally balance these competing goals. Our main proposal is a multi-stage architecture, that progressively learns restoration functions for the degraded inputs, thereby breaking down the overall recovery process into more manageable steps. Specifically, our model first learns the contextualized features using encoder-decoder architectures and later combines them with a high-resolution branch that retains local information. At each stage, we introduce a novel per-pixel adaptive design that leverages in-situ supervised attention to reweight the local features. A key ingredient in such a multi-stage architecture is the information exchange between different stages. To this end, we propose a two-faceted approach where the information is not only exchanged sequentially from early to late stages, but lateral connections between feature processing blocks also exist to avoid any loss of information. The resulting tightly interlinked multi-stage architecture, named as MPRNet, delivers strong performance gains on ten datasets across a range of tasks including image deraining, deblurring, and denoising. The source code and pre-trained models are available at https://github.com/swz30/MPRNet.
1. Introduction
Image restoration is ill-posed, while existing architectures struggle to balance contextual information with spatial detail and effective multi-stage information exchange. MPRNet addresses these issues with progressive stages, supervised attention, and cross-stage feature fusion, achieving state-of-the-art results across restoration tasks.
- Motivation: Image restoration has infinite feasible solutions, making generalizable priors necessary for recovering clean images from degraded inputs.Handcrafted priors are challenging to design and often not generalizable, while CNNs learn priors from natural image statistics.
- Motivation: Existing multi-stage methods trade contextual reliability for spatial accuracy or use suboptimal sequential information transfer.Encoder-decoder designs may lose spatial details, single-scale pipelines may be semantically weaker, and naive stage-to-stage passing can underperform.
- Proposed approach: MPRNet combines encoder-decoder stages for contextual information with a full-resolution final stage for preserving fine spatial details.The architecture progressively decomposes restoration into manageable subtasks while retaining local image information.
- Proposed approach: Supervised attention uses each stage’s restored image and ground-truth guidance to refine incoming features before propagation.The module generates attention maps that reweight local features, supporting progressive restoration at every stage.
- Proposed approach: Cross-stage feature fusion aggregates multi-scale features across stages to preserve contextual information and improve information flow.The design includes lateral feature connections in addition to sequential exchange between stages.
- Results: MPRNet achieves new state-of-the-art results on ten synthetic and real-world datasets spanning deraining, deblurring, and denoising.The paper also reports low complexity, ablations, qualitative results, and generalization tests.
2. Related Work
Image restoration methods have evolved from handcrafted priors toward CNNs, with architectures broadly divided into single-stage and multi-stage designs. Multi-stage methods progressively solve smaller subtasks, but identical subnetworks across stages can produce suboptimal results.
- Architectural background: CNN-based restoration methods use learned image statistics and have achieved state-of-the-art results across low-level vision tasks.Architectural designs include residual learning, encoder-decoders, dilated convolutions, and attention mechanisms.
- Single-Stage Approaches: Single-stage restoration methods commonly adapt components originally developed for high-level vision tasks.Examples include residual learning, encoder-decoders, and dilated convolutions for denoising, deblurring, and deraining.
- Multi-Stage Approaches: Multi-stage methods progressively recover clean images by assigning smaller restoration subtasks to lightweight subnetworks.This decomposition is intended to make the challenging restoration problem easier to solve.
- Multi-Stage Approaches: Using an identical subnetwork at every stage may yield suboptimal restoration results.The paper identifies stage-specific architectural design as an important consideration for multi-stage systems.
- Attention: Attention modules in low-level vision capture long-range dependencies along spatial dimensions, channel dimensions, or both.They have been applied to image deraining, deblurring, super-resolution, and denoising.
3. Multi-Stage Progressive Restoration
MPRNet progressively restores images through complementary multi-stage subnetworks: encoder-decoders capture contextual information, while an original-resolution stage preserves fine spatial details. Supervised attention and cross-stage feature fusion refine and propagate information between stages.
- Architecture: MPRNet uses three progressive stages: the first two encoder-decoders learn broad contextual information, while the last original-resolution stage preserves fine texture.The final stage avoids downsampling to maintain pixel-to-pixel spatial correspondence.
- Architecture: The encoder-decoder subnetworks use channel attention blocks and bilinear upsampling followed by convolution instead of transposed convolution.This design extracts features at multiple scales while reducing checkerboard artifacts during decoding.
- Architecture: ORSNet forms the last stage with original-resolution blocks containing channel attention blocks to generate spatially enriched high-resolution features.It operates without downsampling to preserve fine details from input to output.
- Cross-Stage Feature Fusion: Cross-stage feature fusion refines earlier-stage features with 1 × 1 convolutions before aggregating them into later stages.The mechanism preserves contextualized information, enriches later features, and facilitates more stable optimization across stages.
- Supervised Attention: Supervised attention modules use ground-truth-guided intermediate predictions to generate per-pixel masks that recalibrate features before propagation.SAM combines attention-guided features with an identity path and passes the resulting representation to the next stage.
4. Experiments and Analysis
MPRNet is evaluated on deraining, deblurring, and denoising across multiple synthetic and real-world datasets using PSNR and SSIM, with ablations examining its components. It consistently improves restoration quality and generalizes across datasets while retaining efficiency advantages in several comparisons.
- Datasets and Evaluation Protocol: MPRNet is evaluated on image deraining, deblurring, and denoising across 10 datasets using PSNR and SSIM.The evaluation includes synthetic and real-world data, with separate training setups for each task.
- Image Deraining Results: 1.98 dB average PSNR gain over MSPFN on five deraining datasets accompanies 20% error reduction, 3.7× fewer parameters, and 2.4× faster runtime.Improvements reach 4 dB on Rain100L.
- Image Deblurring Results: 9% PSNR and 21% SSIM improvements over the previous best method on GoPro are reported for image deblurring.On HIDE, MPRNet reports 11% and 13% error reductions and achieves state-of-the-art performance despite training only on GoPro.
- Image Deblurring Results: 0.29 dB and 0.28 dB gains over DMPHN are obtained on RealBlur-R and RealBlur-J when applying the GoPro-trained model directly.When trained and tested on RealBlur, gains over SRN are 0.66 dB and 0.38 dB on the same subsets.
- Image Denoising Results: 0.19 dB and 0.21 dB gains over CycleISP and SADNet are reported on SIDD and DND, respectively, while the SIDD-trained model generalizes to DND without training images.Qualitative results report preserved structural and textural details while removing real noise.
- Ablation Studies: 30.49 dB falls to 30.07 dB without SAM, 30.31 dB without CSFF, and 29.86 dB without both components.The ablation study also reports better performance when earlier stages use encoder-decoders and the last stage uses ORSNet.
5. Resource Efficient Image Restoration
MPRNet supports resource-efficient restoration by producing useful predictions at different stages, allowing earlier outputs for faster systems and later outputs for higher accuracy. Its stage-wise deblurring results remain competitive while stage complexity varies.
- Resource Efficiency: A single multi-stage network can provide early predictions for compute-efficient systems and later predictions for higher accuracy.This addresses the accuracy–computational-efficiency trade-off in CNN restoration models.
- Stage-wise Performance: Stage-wise deblurring performance is reported on GoPro, with runtimes computed using an Nvidia Titan Xp GPU.The stage-wise results are summarized in Table 7.
- Stage-wise Performance: MPRNet maintains competitive restoration performance at each stage, with stage 1 described as light and fast.Stage 1 performs better than SRN and DeblurGANv2, while stage 2 gains 0.51 PSNR over DMPHN.
6. Conclusion
The paper concludes that MPRNet progressively improves degraded images through supervised multi-stage processing that combines contextual and spatially accurate features. Cross-stage feature fusion and attention-guided exchange support performance gains while keeping the model lightweight and runtime-efficient.
- Conclusion: MPRNet progressively improves degraded inputs by injecting supervision at each stage.The architecture combines complementary feature processing across multiple stages.
- Conclusion: Cross-stage feature fusion and attention-guided output exchange connect contextualized and spatially accurate stages.The stated design targets flexible information exchange between stages.
- Conclusion: MPRNet achieves performance gains on numerous benchmark datasets while remaining lightweight in model size and efficient in runtime.The conclusion identifies limited-resource devices as an application context.