Source-linked AI summary
GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions
Tao Wang, Kaihao Zhang, Ziqian Shao, Wenhan Luo, Bjorn Stenger, Tong Lu, Tae-Kyun Kim, Wei Liu, Hongdong Li
TL;DR
Adverse weather introduces complex degradations that make image restoration difficult and can impair downstream vision tasks. GridFormer proposes a unified grid-structured transformer backbone with residual dense transformer blocks and compact-enhanced attention, achieving state-of-the-art results across five restoration tasks. The authors note that its current evaluation is limited to image restoration, leaving video restoration for future work.
Problem
Complex rain, haze, and snow degradations reduce image quality and create difficulties for downstream computer vision tasks.
Method
GridFormer is a unified grid-structured transformer using residual dense transformer blocks, compact-enhanced attention, and local feature enhancement for adverse-weather restoration.
Results
GridFormer achieves state-of-the-art performance across five tasks: deraining, dehazing, deraining and dehazing, desnowing, and multi-weather restoration.
Takeaways & Limitations
The framework provides a general backbone for both weather-specific and multi-weather image restoration.
Takeaways & Limitations
GridFormer is evaluated for image restoration, while extending it to video restoration remains future work.
Abstract
from arXiv · showhide
Image restoration in adverse weather conditions is a difficult task in computer vision. In this paper, we propose a novel transformer-based framework called GridFormer which serves as a backbone for image restoration under adverse weather conditions. GridFormer is designed in a grid structure using a residual dense transformer block, and it introduces two core designs. First, it uses an enhanced attention mechanism in the transformer layer. The mechanism includes stages of the sampler and compact self-attention to improve efficiency, and a local enhancement stage to strengthen local information. Second, we introduce a residual dense transformer block (RDTB) as the final GridFormer layer. This design further improves the network's ability to learn effective features from both preceding and current local features. The GridFormer framework achieves state-of-the-art results on five diverse image restoration tasks in adverse weather conditions, including image deraining, dehazing, deraining \& dehazing, desnowing, and multi-weather restoration. The source code and pre-trained models are available at https://github.com/TaoWangzj/GridFormer.
1 Introduction
Image restoration in adverse weather is challenging because rain, haze, and snow cause complex degradations that reduce visual quality and hinder downstream vision tasks. GridFormer addresses this with a unified grid-based transformer design and reports state-of-the-art performance across weather-specific and multi-weather restoration.
- Motivation: Adverse weather causes color distortion, blur, noise, and low contrast, reducing image quality and complicating object recognition and scene understanding.
- Related challenges: Traditional handcrafted-prior methods generalize poorly to complex weather, while many CNN approaches target only one restoration task.
- Related challenges: Unified multi-weather methods address several degradations in one architecture, but TransWeather is constrained by limited feature fusion across network scales.
- Proposed framework: GridFormer combines residual dense transformer blocks with a grid structure to exploit hierarchical features and integrate contextual information across spatial scales.
- Proposed framework: Its compact-enhanced self-attention uses sampling and compact self-attention for efficiency, while local enhancement strengthens local details.
- Results: GridFormer achieves a new state-of-the-art on five tasks: deraining, dehazing, deraining and dehazing, desnowing, and multi-weather restoration.
2 Related Work
Related work spans task-specific restoration methods, unified multi-weather architectures, and vision transformers for low-level image restoration. GridFormer builds on these directions while aiming for a more general transformer-based restoration backbone.
- Restoration in Adverse Weather Conditions: Adverse-weather restoration targets high-quality images degraded by rain, fog, or snow, including deraining, dehazing, desnowing, and multi-weather restoration.
- Task-specific restoration: Deraining methods learn mappings from rainy images or detail layers to restored images using recurrent context, attention, dense, GAN, and variational designs.
- Task-specific restoration: Dehazing approaches evolved from estimating transmission and atmospheric light to directly restoring clear images with attention, multi-scale, GAN, and transformer architectures.
- Positioning GridFormer: GridFormer differs from a related CNN dehazing method by using a general transformer design with compact-enhanced layers and residual-dense integration to promote feature reuse.
- Task-specific restoration: Desnowing research uses translucency, residual generation, stacked dense multi-scale networks, and size- and transparency-aware adversarial learning.
- Multi-weather restoration: Unified multi-weather systems use architectures such as multi-encoder-decoder networks and TransWeather's transformer-based paths and decoder with weather-type embeddings.
- Vision Transformers in Image Restoration: Vision-transformer restoration methods include IPT, SwinIR, Uformer, DenSformer, recursive transformers, and Restormer, but IPT requires costly large-scale pre-training.
3 Method
GridFormer is a transformer-based restoration framework that embeds residual dense transformer blocks in a multi-resolution grid to restore adverse-weather images. Its architecture combines hierarchical feature interaction, efficient compact-enhanced attention, local enhancement, and multi-scale loss supervision.
- Motivation and Architecture: GridFormer embeds residual dense transformer blocks in a grid structure to exploit hierarchical image features for adverse-weather restoration.The framework is motivated by restoring images degraded by haze, rain, and snow.
- Architecture: Three resolution paths interact dynamically, with higher-resolution paths refining details and lower-resolution paths supplying global information through larger receptive fields.Each path contains seven GridFormer layers and restoration occurs at different image resolutions.
- Grid Fusion Module: The grid fusion module organizes three rows and five columns of GridFormer layers, using sampling operations and weighted attention to fuse complementary features across scales.Weighted feature fusion uses trainable channel-wise vectors for features from different rows and columns.
- Grid Tail and Training: GridFormer reconstructs multi-scale outputs with long skip connections and is trained using multi-scale Charbonnier loss combined with perceptual loss.The perceptual-loss weight is set to 0.1, and the perceptual features are extracted from VGG19 Conv5_4.
- Residual Dense Transformer Block: Each residual dense transformer block contains three densely connected transformer layers, 1 × 1 local feature fusion, and a local skip connection for residual learning.These dense connections are intended to support feature reuse and information propagation.
- Compact-Enhanced Attention: Compact-enhanced attention uses feature sampling, channel-wise compact self-attention, and local enhancement to reduce transformer cost while strengthening local information.Channel-wise attention avoids the spatial key-query interaction whose complexity grows as O(N × N).
4 Experiments and Analysis
GridFormer is evaluated across five adverse-weather restoration tasks using diverse datasets and comparisons with state-of-the-art methods. Results show strong restoration performance, cross-dataset generalization, real-world effectiveness, and contributions from its architectural components.
- Experimental Setup: GridFormer is evaluated on dehazing, desnowing, raindrop removal, deraining and dehazing, and multi-weather restoration tasks.The experiments introduce task-specific datasets and comparison methods, followed by restoration results and ablation studies.
- Dehazing Results: GridFormer achieves the best performance on SOTS-indoor, including a 4.23 dB PSNR gain over MAXIM-2S.It also achieves the best performance across all reported metrics on Haze4K.
- Desnowing Results: GridFormer improves over SnowDiff64 by 2.3 dB PSNR and 0.0072 SSIM on Snow100K-S, and by 2.66 dB and 0.0195 on Snow100K-L.The visual comparisons report effective snow-corruption removal and perceptually pleasing results.
- Raindrop Removal Results: GridFormer achieves the best PSNR for raindrop removal and nearly matches RainDropDiff64 in SSIM, differing by 0.0022.Visual comparisons report successful raindrop removal and realistic image generation.
- Deraining and Dehazing Results: GridFormer handles haze and rainfall simultaneously, with visual results described as vivid and effective.The comparison is reported for image deraining and dehazing experiments.
- Cross-Dataset and Real-World Evaluation: GridFormer and GridFormer-S outperform other approaches in cross-dataset testing after training on All-weather and testing on Rain100L and Test100.The models are applied directly to specific deraining datasets without task-specific retraining.
- Cross-Dataset and Real-World Evaluation: GridFormer effectively removes haze in a real-world hazy-image comparison and produces a clearer result than current methods.The evidence is qualitative and based on an image collected from the Internet.
- Ablation Study: Local enhancement improves PSNR by 0.59 dB on RainDrop and 1.33 dB on SOTS-Indoor, while computational complexity is significantly reduced.The ablation results support the contribution of local enhancement operations.
5 Limitations and Future Work
GridFormer outperforms previous methods for adverse-weather image restoration but remains limited to image restoration and leaves several improvement directions open.
- Scope: GridFormer is evaluated for image restoration, while extending it to video restoration remains future work.The authors identify video restoration in adverse weather conditions as an important direction.
- Future improvements: Pre-training and contrastive learning are proposed as strategies for further exploring GridFormer's performance potential.
- Future improvements: The current multi-scale feature fusion uses simple weighted attention, which the authors suggest could be improved with more sophisticated attention modules.
6 Conclusion
GridFormer is a unified Transformer architecture for adverse-weather image restoration that combines grid-based information exchange with compact-enhanced residual dense processing. Comprehensive experiments show strong results across weather-specific and multi-weather restoration tasks.
- Architecture: GridFormer uses a grid structure to communicate information across streams and exploit hierarchical features from input images.
- Architecture: Its compact-enhanced Transformer layer is integrated in a residual dense manner to encourage feature reuse and enhance feature representation.
- Results: GridFormer significantly surpasses state-of-the-art methods on both weather-specific and multi-weather restoration tasks.