Source-linked AI summary
Restormer: Efficient Transformer for High-Resolution Image Restoration
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang
TL;DR
High-resolution image restoration needs long-range pixel modeling, but standard self-attention is computationally prohibitive and local attention restricts context. Restormer uses efficient channel-wise attention and gated feed-forward blocks with local context mixing, achieving state-of-the-art results across multiple restoration tasks.
Problem
Self-attention captures long-range pixel dependencies, but its quadratic spatial complexity and local-window alternatives limit efficient global modeling for high-resolution restoration.
Method
Restormer combines linear-complexity transposed attention and gated feed-forward blocks with depth-wise convolutions in a multi-scale encoder-decoder.
Results
Restormer achieves consistent gains across deraining and motion-deblurring benchmarks, including 1.05 dB averaged improvement over SPAIR for deraining and 0.47 dB over MIMO-UNet+ for deblurring.
Takeaways & Limitations
The model provides a computationally efficient way to exploit global and local image context for high-resolution restoration.
Takeaways & Limitations
Restricting self-attention to non-overlapping 48×48 patches limits spatial context and conflicts with capturing true long-range pixel relationships.
Abstract
from arXiv · showhide
Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.
1. Introduction
Restormer is an efficient Transformer for high-resolution image restoration that models global pixel connectivity while remaining applicable to large images. It addresses CNN limitations and restricted-attention approaches through linear-complexity attention, gated feed-forward processing, and progressive learning.
- Motivation: CNNs provide strong learned image priors, but limited receptive fields and static filters hinder long-range dependency modeling and adaptation to input content.Image restoration reconstructs high-quality images by removing degradations such as noise, blur, and rain drops, making strong priors important.
- Motivation: Restricted self-attention uses 8×8 windows or 48×48 patches, limiting the long-range pixel relationships needed for high-resolution restoration.These restrictions reduce computational load but contradict the goal of capturing true long-range pixel relationships.
- Architecture: Restormer introduces MDTA, which computes self-attention across feature dimensions rather than spatial dimensions and has linear complexity.MDTA replaces vanilla multi-head self-attention and avoids explicitly modeling pairwise pixel interactions by computing cross-covariance across features.
- Architecture: Restormer reformulates the Transformer feed-forward network with a GELU-based gating mechanism formed from the element-wise product of two linear projections.The gated transformation is designed to improve information flow through the network.
- Training: Progressive learning trains Restormer with small patches and large batches early, then larger patches and smaller batches, improving test-time performance.The strategy helps the network learn context from large images during training.
2. Background
Data-driven CNNs, especially computationally efficient hierarchical UNets and skip-connection designs, have become prominent for image restoration. Local self-attention reduces complexity but restricts context aggregation to neighborhoods, motivating a Transformer approach for restoration.
- Image Restoration: Data-driven CNN architectures have outperformed conventional image-restoration approaches.
- Image Restoration: Hierarchical multi-scale UNet architectures are widely studied because they remain computationally efficient.
- Transformers for Image Restoration: Local self-attention strategies reduce complexity but restrict context aggregation to local neighborhoods, making them less suited to image restoration.
3. Method
Restormer is designed for efficient high-resolution image restoration through a multiscale encoder-decoder and Transformer blocks that reduce self-attention cost while modeling global and local context. Its components include channel-wise attention, gated feed-forward processing, and progressive learning with increasingly larger training patches.
- Overall Pipeline: Restormer uses a 4-level symmetric encoder-decoder whose hierarchical reduction of spatial size and expansion of channel capacity improves efficiency.Pixel-unshuffle and pixel-shuffle perform downsampling and upsampling, while skip connections concatenate encoder and decoder features.
- Multi-Dconv Head Transposed Attention: Conventional self-attention has O(W^2H^2) time and memory complexity, making it unsuitable for many high-resolution restoration tasks.The computational bottleneck arises from key-query interactions across the spatial resolution of W×H images.
- Multi-Dconv Head Transposed Attention: MDTA achieves linear complexity by computing self-attention across channels, using cross-covariance to encode global context implicitly.Depth-wise convolutions emphasize local context before feature covariance produces the global attention map.
- Gated-Dconv Feed-Forward Network: GDFN incorporates depth-wise convolutions and gating to encode neighboring-pixel information and control feature information flow across hierarchical levels.Its role is complementary to MDTA: GDFN focuses on fine details, whereas MDTA enriches features with contextual information.
- Progressive Learning: Progressive learning trains Restormer on smaller patches early and gradually larger patches later to better capture global image statistics.This strategy addresses the risk that training Transformers only on small cropped patches produces suboptimal full-resolution test performance.
4. Experiments and Analysis
Restormer is evaluated on deraining, motion and defocus deblurring, and synthetic and real image denoising benchmarks. It consistently achieves state-of-the-art performance, with substantial gains over competing CNN- and Transformer-based methods while reducing computational cost in several comparisons.
- Evaluation scope: Experiments cover image deraining, single-image motion deblurring, single-image and dual-pixel defocus deblurring, and synthetic and real image denoising.The evaluation uses benchmark datasets and task-specific experimental settings.
- Image deraining: 1.05 dB average improvement over SPAIR is achieved across five deraining datasets, with gains reaching 2.06 dB on Rain100L.PSNR/SSIM are computed using the Y channel in YCbCr color space.
- Single-image motion deblurring: 0.47 dB and 0.26 dB average gains over MIMO-UNet+ and MPRNet, respectively, are obtained across four motion-deblurring datasets.Restormer is trained only on GoPro and directly evaluated on HIDE and RealBlur; it also uses 81% fewer FLOPs than MPRNet.
- Defocus deblurring: Restormer sets state-of-the-art results for both single-image and dual-pixel defocus deblurring on DPDD.It improves over IFAN by ∼0.6 dB on the combined scene category and over Uformer by 1.01 dB PSNR.
- Real image denoising: Among competing real-image denoising methods, only Restormer surpasses 40 dB PSNR, despite training only on SIDD and testing directly on DND.The real-image evaluation uses the SIDD and DND datasets.
- Gaussian image denoising: 0.37 dB and 0.31 dB gains over DRUNet and SwinIR, respectively, are achieved for noise level 50 on Urban100 grayscale denoising.Restormer achieves state-of-the-art performance under both single-model and separate-model settings, while using 3.14× fewer FLOPs and running 13× faster than SwinIR.
5. Conclusion
Restormer is a computationally efficient Transformer for high-resolution image restoration. Its redesigned Transformer blocks improve feature aggregation and transformation, with MDTA modeling global context through channel-wise self-attention at linear rather than quadratic complexity.
- 5. Conclusion: Restormer is an image restoration Transformer designed to handle high-resolution images computationally efficiently.The model is presented as a Transformer architecture for image restoration that remains applicable to high-resolution inputs.
- 5. Conclusion: The model introduces key designs in Transformer-block components to improve feature aggregation and transformation.These changes target the core components of the Transformer block.
- 5. Conclusion: MDTA models global context by applying self-attention across channels rather than spatial dimensions, yielding linear instead of quadratic complexity.This attention formulation is intended to capture global context while reducing complexity for high-resolution image restoration.