Source-linked AI summary
Learning Enriched Features for Fast Image Restoration and Enhancement
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao
TL;DR
Image restoration must recover clean content from degraded measurements while balancing precise spatial detail against broad contextual representation. MIRNet-v2 maintains high-resolution features throughout the network and combines them with context from parallel multi-resolution streams using information exchange and attention-based fusion. Across six real-image benchmarks and several restoration and enhancement tasks, it reports state-of-the-art results, with improved accuracy and substantially lower computational cost than MIRNet.
Problem
Existing CNN restoration designs typically prioritize either spatially precise high-resolution features or semantically strong low-resolution context, while restoration requires both.
Method
MIRNet-v2 uses multi-resolution convolution streams, coarse-to-fine information exchange, attention-based aggregation, and residual contextual blocks while maintaining a high-resolution main branch.
Results
MIRNet-v2 achieves state-of-the-art results across defocus deblurring, denoising, super-resolution, and enhancement on six real-image datasets, while improving PSNR from 39.72 dB to 39.84 dB over MIRNet.
Takeaways & Limitations
The architecture combines contextual information from multiple scales with preservation of high-resolution spatial details for image restoration and enhancement.
Takeaways & Limitations
The reported quantitative results include a correction because scores previously reported in one cited work were incorrect and later revised by its original authors.
Abstract
from arXiv · showhide
Given a degraded input image, image restoration aims to recover the missing high-quality image content. Numerous applications demand effective image restoration, e.g., computational photography, surveillance, autonomous vehicles, and remote sensing. Significant advances in image restoration have been made in recent years, dominated by convolutional neural networks (CNNs). The widely-used CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatial details are preserved but the contextual information cannot be precisely encoded. In the latter case, generated outputs are semantically reliable but spatially less accurate. This paper presents a new architecture with a holistic goal of maintaining spatially-precise high-resolution representations through the entire network, and receiving complementary contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing the following key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) non-local attention mechanism for capturing contextual information, and (d) attention based multi-scale feature aggregation. Our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on six real image benchmark datasets demonstrate that our method, named as MIRNet-v2 , achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNetv2
1 INTRODUCTION
Image restoration must recover clean content from ill-posed, degraded measurements while preserving spatial details and encoding sufficient context. MIRNet-v2 addresses this tension by maintaining high-resolution features and progressively exchanging and selectively fusing multi-scale contextual features, with reported accuracy and efficiency gains over MIRNet.
- Motivation: Image restoration is an ill-posed inverse problem because camera and lighting conditions produce degraded measurements with many possible clean-image solutions.Degradations include noise, low contrast, excessive darkness, and excessive brightness.
- Motivation: Encoder-decoder CNNs capture broad context through resolution reduction, whereas high-resolution networks preserve spatial details but have limited receptive fields and contextual encoding.The two common designs therefore emphasize different aspects of restoration quality.
- Proposed approach: MIRNet-v2 maintains original high-resolution features while parallel convolution streams encode multi-scale context for position-sensitive restoration.The architecture is designed to preserve true edges and textures while removing undesired degraded content.
- Proposed approach: The method progressively exchanges information from coarse to fine resolutions and uses selective kernel fusion to dynamically combine features with varying receptive fields.Self-attention selects useful kernels while preserving complementary characteristics across branches.
- Proposed approach: MIRNet-v2 removes high-to-low fusion, replaces the dual attention unit with a residual contextual block, adds group convolutions, and uses progressive learning to reduce computational cost and improve performance.Progressive learning trains on small patches early and gradually larger patches later.
- Results: 39.84 dB PSNR is achieved by MIRNet-v2 versus 39.72 dB for MIRNet, while parameters and FLOPs decrease by approximately 81%.Convolutions decrease by 36%, and training and inference speed increase by 2.2× and 3.6×, respectively.
2 RELATED WORK
Image restoration and enhancement research spans deblurring, denoising, super-resolution, and enhancement, with methods tailored to distinct degradation sources and image-quality goals. MIRNet-v2 instead presents a shared multi-scale framework that preserves full-resolution details while fusing contextual information from parallel branches.
- Scope: MIRNet-v2 targets dual-pixel defocus deblurring, denoising, super-resolution, and image enhancement within one architecture.The method processes features at the original resolution and fuses contextual information from multiple branches.
- Defocus deblurring: Defocus deblurring seeks an all-in-focus image from a defocused input, using either direct deblurring or disparity-guided procedures.Dual-pixel sensors provide two sub-aperture views at each pixel location.
- Image denoising: Denoising research includes transform modification, neighborhood averaging, self-similarity methods such as NLM and BM3D, and more recent deep-learning models.Patch-based methods exploit redundancy in natural images.
- Super-resolution: Super-resolution approaches have used sampling theory, edge-guided interpolation, natural-image priors, patch exemplars, sparse representations, and deep-learning architectures.Current data-driven methods are actively explored because they improve results over conventional algorithms.
- Image enhancement: Image enhancement addresses missing vivid details or contrast caused by lighting and camera limitations, while histogram equalization can produce under- or over-enhanced images.Retinex-inspired methods model aspects of human vision for enhancement.
3 PROPOSED METHOD
MIRNet-v2 combines high-resolution spatial representations with complementary multi-scale context through recurrent multi-scale residual blocks. Its design uses information exchange, selective attention-based fusion, contextual feature extraction, and progressive training to balance restoration accuracy and efficiency.
- Multi-Scale Residual Block: Multi-scale residual blocks use parallel streams to produce semantically richer coarse features and spatially precise fine features.The streams operate across fine-to-coarse and coarse-to-fine resolutions.
- Overall Pipeline: The overall pipeline extracts low-level features, processes them through recursive residual groups, predicts a residual image, and adds it to the input.The restored image is formed as Î = I + R.
- Selective Kernel Feature Fusion: SKFF dynamically selects and aggregates features from different resolution streams using self-attention and varying receptive fields.It recalibrates each stream with softmax attention before aggregation and uses approximately five times fewer parameters than concatenation.
- Residual Contextual Block: Residual contextual blocks suppress less useful features and retain informative ones through contextual processing within convolutional streams.Their convolutional front end uses two 3x3 group convolutions, which are described as more resource efficient than standard convolutions.
- Progressive Training Regime: Progressive learning trains on smaller patches early and gradually larger patches later to balance training speed with test-time accuracy.The strategy is also described as curriculum learning and supports inputs of different sizes at test time.
4 EXPERIMENTS
MIRNet-v2 is evaluated on six real-image datasets across defocus deblurring, denoising, super-resolution, and enhancement. It achieves strong quantitative and qualitative results, while ablations support its multi-scale design and progressive learning strategy.
- Dual-pixel defocus deblurring: MIRNet-v2 achieves state-of-the-art dual-pixel defocus deblurring results, improving PSNR over RDPD by 0.86 dB indoors and 0.77 dB outdoors.
- Dual-pixel defocus deblurring: 0.81 dB and 1.07 dB PSNR gains are reported over competing methods when indoor and outdoor DPDD scenes are combined.
- Dual-pixel defocus deblurring: MIRNet-v2 removes spatially varying defocus blur while producing sharper, more visually faithful images and preserving fine details.
- Image denoising: On SIDD, MIRNet-v2 gains 0.32 dB over CycleISP and, on DND, gains 0.11 dB over DAGL in denoising comparisons.
- Image denoising: MIRNet-v2 generalizes from SIDD training to DND testing, whose images come from different cameras with different noise characteristics.
- Super-resolution: For RealSR scaling factors ×2, ×3, and ×4, MIRNet-v2 consistently achieves better image-quality scores than the compared super-resolution methods.
- Image enhancement: MIRNet-v2 reports gains of 3.44 dB over KinD++ on LoL and 0.93 dB over DeepUPE on Adobe-FiveK enhancement.
- Ablation studies: Ablations show that RCB improves over the EDSR ResBlock, while group convolutions provide a parameter- and FLOP-efficient trade-off.
5 CONCLUDING REMARKS
The paper proposes MIRNet-v2 to combine full-resolution spatial precision with contextual information from parallel multi-scale branches. Its state-of-the-art results across six datasets and four tasks support this architecture.
- Full-resolution processing preserves spatial details, while parallel branches provide contextualized features for restoration and enhancement.
- MIRNet-v2 learns relationships within and across multi-scale branches and dynamically adapts its receptive field without sacrificing original feature details.
- The method achieves state-of-the-art results on six datasets spanning four image restoration and enhancement tasks.