Source-linked AI summary
Learning Enriched Features for Real Image Restoration and Enhancement
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao
TL;DR
Image restoration must remove degradation while preserving spatial details, yet existing CNNs tend to favor either high-resolution precision or low-resolution context. MIRNet combines persistent high-resolution representations with multi-scale contextual exchange through multi-scale residual blocks and attention-based fusion, achieving state-of-the-art results across five real-image datasets and three restoration tasks.
Problem
Existing image restoration CNNs typically favor either spatially precise high-resolution representations or contextually strong low-resolution representations, although restoration requires both.
Method
MIRNet uses a high-resolution main branch, parallel multi-resolution streams, repeated cross-scale exchange, attention mechanisms, and selective kernel feature fusion.
Results
MIRNet achieves state-of-the-art results on five real-image benchmark datasets for image denoising, super-resolution, and image enhancement.
Takeaways & Limitations
MIRNet learns enriched representations that combine multi-scale contextual information with preserved high-resolution spatial details.
Abstract
from arXiv · showhide
With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In a nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for a variety of image processing tasks, including image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNet.
1 Introduction
MIRNet addresses the tension between spatial precision and contextual representation in real image restoration by combining persistent high-resolution processing with multi-scale feature exchange. Its architecture uses complementary branches, selective fusion, and recursive residual design, and achieves state-of-the-art results across five real-image datasets and multiple tasks.
- Motivation: Real image restoration is an ill-posed inverse problem requiring removal of degradation while preserving true edges and textures.Camera limitations and unsuitable lighting can produce noisy, dark, bright, or low-contrast images.
- Motivation: Existing CNNs trade spatial precision for contextual strength by using either high-resolution processing or progressively reduced-resolution representations.Encoder-decoder designs obtain broad context through spatial reduction, whereas high-resolution designs better preserve precise details.
- Proposed approach: MIRNet maintains original high-resolution features while parallel lower-resolution streams encode complementary multi-scale context.This design targets precise pixel correspondence and contextually enriched representations simultaneously.
- Proposed approach: Selective kernel fusion dynamically selects useful kernels and progressively exchanges information across scales in both top-down and bottom-up directions.The mechanism combines features with varying receptive fields while preserving their complementary characteristics.
- Proposed approach: A recursive residual design progressively breaks down the input signal, simplifying learning and supporting construction of very deep networks.The architecture repeatedly fuses features across multi-resolution branches.
- Results: MIRNet achieves state-of-the-art results on five real-image benchmark datasets spanning denoising, super-resolution, and image enhancement.The experiments also evaluate practical generalization across datasets.
2 Related Work
The paper situates MIRNet among classical and deep-learning approaches for denoising, super-resolution, and enhancement. These task-specific methods include self-similarity, sampling, residual, attention, Retinex-inspired, encoder-decoder, and GAN-based techniques.
- Overview: MIRNet targets image denoising, super-resolution, and enhancement within a broader need for effective restoration algorithms.The paper frames these tasks as responses to rapidly growing image content and degraded camera captures.
- MIRNet: MIRNet differs from these approaches by processing original-resolution features while fusing contextual information from multiple parallel branches.Its framework uses a recursive residual design and multi-scale residual blocks.
- Image denoising: Classical denoising methods modify transform coefficients, average neighborhood pixels, or exploit image self-similarity through patch-based algorithms.NLM and BM3D are cited as examples of self-similarity-based denoising.
- Super-resolution: Super-resolution research spans sampling theory, edge-guided interpolation, natural-image priors, patch exemplars, sparse representations, and diverse deep-learning architectures.Deep-learning designs include residual, recursive, progressive, dense, attention, multi-branch, and GAN-based approaches.
- Image enhancement: Image enhancement methods include histogram equalization, Retinex-inspired algorithms, CNNs, encoder-decoder networks, and GANs.Histogram equalization is common but can produce under- or over-enhanced images.
3 Proposed Method
MIRNet combines parallel multi-resolution processing, repeated cross-scale information exchange, and attention-based fusion to preserve high-resolution detail while incorporating contextual representations. Its building blocks include selective kernel feature fusion, dual attention, and residual resizing modules within recursive residual groups.
- Multi-scale Residual Block: MIRNet processes features through parallel multi-resolution convolution streams that exchange information across scales and resolutions.The streams provide fine-to-coarse semantically richer and coarse-to-fine spatially precise representations.
- Overall Pipeline: The overall pipeline extracts low-level features, applies recursive residual groups, predicts a residual image, and adds it to the input.Each recursive residual group contains multiple multi-scale residual blocks; the network uses Charbonnier loss with ε=10^-3.
- Selective Kernel Feature Fusion: Selective kernel feature fusion nonlinearly aggregates multi-resolution features by fusing global descriptors and selecting softmax-weighted stream responses.For three streams, the output is U = s1 · L1 + s2 · L2 + s3 · L3, using approximately 6× fewer parameters than concatenation.
- Dual Attention Unit: Dual attention recalibrates convolutional features by combining spatial and channel attention so less useful features are suppressed and informative features are propagated.The channel-attention branch uses global average pooling, excitation layers, and sigmoid gating to rescale feature maps.
- Residual Resizing Modules: Residual resizing modules provide the downsampling and upsampling operations used in the multi-scale architecture.These modules are part of the proposed block’s design for handling representations at different resolutions.
4 Experiments
MIRNet is evaluated across five real-image datasets spanning denoising, super-resolution, and enhancement, with quantitative, qualitative, and generalization tests. It achieves strong results across these tasks while preserving structures, textures, and natural appearance.
- Experimental Setup: MIRNet is evaluated on five real-image datasets for denoising, super-resolution, and image enhancement.The experiments include qualitative and quantitative comparisons across all three tasks.
- Image Denoising: MIRNet removes real noise while preserving homogeneous-region smoothness, structural content, and fine texture without artifacts.Competing methods are described as either oversmoothing details or introducing chroma artifacts and blotchy textures.
- Image Denoising: MIRNet trained on SIDD generalizes to DND despite different camera sets and noise characteristics.DND has no training data, so this evaluation tests cross-dataset transfer from SIDD.
- Super-Resolution: 0.45 dB, 0.74 dB, and 0.22 dB are MIRNet’s PSNR gains over LP-KPN for ×2, ×3, and ×4 RealSR super-resolution, respectively.The method consistently obtains better image-quality scores than competing approaches for all three scaling factors, with a similar SSIM trend.
- Super-Resolution: MIRNet recovers structural patterns and edges in RealSR super-resolution while producing natural images with better color reproduction.The visual comparisons also report artifacts or structure loss for competing methods.
5 Ablation Studies
Ablation studies show that skip connections, multi-stream information exchange, dual-attention units, and selective-kernel feature fusion each contribute to MIRNet’s performance. More streams and DAU columns improve restoration quality, while SKFF is both more effective than concatenation and substantially lighter.
- Component Analysis: Removing skip connections causes the largest performance drop and makes the network difficult to converge, producing high training errors and low PSNR.This ablation identifies skip connections as the most consequential tested component.
- Component Analysis: SKFF information exchange among parallel convolution streams improves performance, and DAU positively influences final image quality.Both mechanisms are evaluated as architectural components of the multi-scale residual block.
- Feature Aggregation: SKFF generates more favorable results than summation and concatenation while using ∼6× fewer parameters than concatenation.The comparison evaluates alternative feature aggregation strategies.
- MRB Layout: Increasing the number of parallel streams provides significant improvements in image restoration quality.The result supports the importance of processing multi-scale features.
- MRB Layout: Increasing the number of DAU columns yields better scores, indicating the significance of information exchange among parallel streams for feature consolidation.The layout study varies both the number of streams and the number of DAU-containing columns.
6 Concluding Remarks
MIRNet combines full-resolution processing with contextualized features from parallel multi-scale branches to address the competing demands of spatial precision and context. Its receptive field can adapt dynamically without sacrificing original feature details, and it achieves state-of-the-art results across five datasets and three tasks.
- Concluding Remarks: MIRNet’s main branch preserves full-resolution features while parallel branches provide contextualized multi-scale representations.The architecture is designed to combine precise spatial details with broader context.
- Concluding Remarks: Its feature-fusion strategy dynamically adapts the receptive field without sacrificing original feature details.The conclusion links adaptive context aggregation with preservation of spatial information.
- Concluding Remarks: MIRNet consistently achieves state-of-the-art results on five datasets across image denoising, super-resolution, and image enhancement.The authors present this consistency as evidence supporting the approach’s effectiveness.