Source-linked AI summary
TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions
Jeya Maria Jose Valanarasu, Rajeev Yasarla, Vishal M. Patel
TL;DR
Adverse weather removal remains challenging because many methods target only one degradation and the multi-weather All-in-One model uses multiple encoders. TransWeather uses a single transformer encoder-decoder with intra-patch blocks and learnable weather queries, improving results across multiple datasets and real-world images while reducing inference cost. High-intensity real rain and splattering remain limitations.
Problem
Most weather-removal methods address one degradation, while All-in-One uses multiple encoders and still has scope for performance improvement.
Method
TransWeather is a single encoder-decoder transformer using intra-patch blocks and learnable weather type queries for all-weather restoration.
Results
TransWeather improves performance across multiple rain, snow, fog, and mixed-weather test datasets, including against individually fine-tuned methods, and performs faster at 0.14 seconds per 256 × 256 image.
Takeaways & Limitations
A single model instance can provide efficient all-weather restoration and serve as an efficient backbone for generic weather-removal frameworks.
Takeaways & Limitations
TransWeather does not perform well on some real-world images with high-intensity rain, and splattering effects remain unresolved.
Abstract
from arXiv · showhide
Removing adverse weather conditions like rain, fog, and snow from images is an important problem in many applications. Most methods proposed in the literature have been designed to deal with just removing one type of degradation. Recently, a CNN-based method using neural architecture search (All-in-One) was proposed to remove all the weather conditions at once. However, it has a large number of parameters as it uses multiple encoders to cater to each weather removal task and still has scope for improvement in its performance. In this work, we focus on developing an efficient solution for the all adverse weather removal problem. To this end, we propose TransWeather, a transformer-based end-to-end model with just a single encoder and a decoder that can restore an image degraded by any weather condition. Specifically, we utilize a novel transformer encoder using intra-patch transformer blocks to enhance attention inside the patches to effectively remove smaller weather degradations. We also introduce a transformer decoder with learnable weather type embeddings to adjust to the weather degradation at hand. TransWeather achieves improvements across multiple test datasets over both All-in-One network as well as methods fine-tuned for specific tasks. TransWeather is also validated on real world test images and found to be more effective than previous methods. Implementation code can be accessed at https://github.com/jeya-maria-jose/TransWeather .
1. Introduction
Adverse weather degrades images and undermines vision systems, while existing approaches often target individual conditions or use costly multi-encoder designs. TransWeather addresses all-weather removal with one transformer encoder-decoder, intra-patch attention, and learned weather queries.
- Motivation: Rain, fog, and snow reduce visibility and corrupt image information, impairing detection, segmentation, and depth estimation in navigation and surveillance.Clean images are also desired for photography.
- Prior limitations: Traditional weather-removal priors must be modeled separately for each weather condition, while separate algorithms complicate the restoration pipeline.
- Approach: All-in-One handles multiple weather conditions with task-specific encoders and a generic decoder, but TransWeather uses a single encoder and decoder with learned weather type queries.
- Approach: Intra-patch transformer blocks attend within sub-patches to extract fine details and remove small degradations such as rain streaks, raindrops, and snow.Efficient self-attention is used to keep computational complexity low.
- Results: TransWeather achieves superior performance across rain, snow, fog, and mixed-degradation datasets, including against methods fine-tuned for individual tasks.The authors also report fast inference and strong results on real-world degraded images.
2. Related Works
Prior work largely addresses individual weather-removal tasks, while All-in-One introduced a multi-weather CNN with multiple task-specific encoders. TransWeather instead uses a single transformer encoder-decoder with intra-patch processing and weather queries for all-weather restoration.
- Single-weather removal: Research has extensively studied deraining, dehazing, desnowing, and raindrop removal using specialized methods.
- All-in-One weather removal: All-in-One combines multiple task-specific encoders with a common decoder and uses neural architecture search to optimize encoder feature extraction.A discriminator classifies degradation types and routes loss to specific encoders.
- Transformers in low-level vision: Transformers have been adopted for low-level vision, including image processing and restoration architectures such as U-former and Swin-IR.
- TransWeather: TransWeather differs by solving all adverse-weather tasks with one transformer encoder-decoder, intra-patch blocks, and weather-query conditioning.
3. Proposed Method - TransWeather
TransWeather restores images affected by varied adverse weather using a single transformer encoder-decoder, combining hierarchical image features with learned weather-specific task features. Its encoder augments attention within patches for fine details, while the decoder and convolutional tail reconstruct the clean image.
- Overall architecture: TransWeather formulates all-weather restoration as a single weather-agnostic encoder-decoder network, replacing All-in-One's weather-specific encoders with learned weather type queries.The queries attend to encoder features and produce task features used for restoration.
- Transformer Encoder: The encoder divides the degraded image into patches and extracts hierarchical features across stages at progressively reduced resolutions.This provides both coarse and fine information for the restoration pipeline.
- Reconstruction: A convolutional projection block combines decoder task features with encoder features to output the clean image at the original dimensions.The convolutional tail uses four layers, upsampling before each layer, encoder skip connections, and a final tanh activation.
- Transformer Encoder: Intra-PT blocks process smaller sub-patches inside each stage and combine their features with the main transformer block to capture fine degradation details.Sub-patches are half the original patch height and width; efficient attention uses a high reduction ratio.
- Transformer Decoder: The decoder uses learnable weather type embeddings as queries and final-stage encoder features as keys and values to generate a task feature vector.The task feature is fused with hierarchical encoder features before image reconstruction.
- Loss: Training uses an end-to-end smooth L1 loss between the prediction and ground truth, augmented by a VGG16-based perceptual loss.The total objective is Ltotal = LsmoothL1 + λLperceptual, with λ controlling the perceptual-loss contribution.
4. Experiments
Experiments evaluate TransWeather on synthetic and real-world adverse-weather datasets against task-specific methods, All-in-One, and Swin-IR. Results show strong restoration quality across weather combinations and fine details, including real rain streaks and drops.
- Experimental Setup: The experiments use 9,000 Snow100K images, 1,069 Raindrop images, and 9,000 Outdoor-Rain images combined as the All-Weather training set.The training distribution follows All-in-One for fair comparison.
- Experimental Setup: Evaluation covers Test1 for fog-plus-rain removal, Snow100K-L for snow removal, RainDrop for raindrop removal, and real-world rain images.The study tests both synthetic and real-world degradation.
- Comparison Methods: Comparisons include task-specific methods, Swin-IR, and All-in-One, with task-specific networks fine-tuned for their respective datasets.All-in-One and TransWeather are both evaluated as single-model solutions across the weather-removal tasks.
- Quantitative Evaluation: PSNR and SSIM quantify performance on Test1, Snow100K-L, and RainDrop, with higher values indicating better results.The quantitative comparisons are reported in Tables 1, 2, and 3.
- Quantitative Results: TransWeather achieves superior performance across multiple rain, snow, fog, and combined-weather test datasets, including comparisons with individually fine-tuned methods.The reported results also identify cases where task-specific CCN and DDMSNet outperform All-in-One, while TransWeather remains competitive in the comparison.
- Visual Quality Comparison: Visual comparisons show TransWeather removing fog, rain streaks, small snow particles, and fine rain drops more effectively than previous methods on synthetic and real-world images.The visual examples include Test1, Snow100K-L, RainDrop, and real-world images.
5. Discussions
TransWeather’s ablations show that its hierarchical encoder, Intra-PT blocks, weather queries, and transformer decoder each improve restoration. The model also reduces inference cost and parameters relative to All-in-One, while remaining limited on intense real-world rain and splattering.
- Ablation Study: Each proposed component improves performance on the Raindrop test dataset when added progressively to the base transformer configuration.The progression adds hierarchical encoding, Intra-PT blocks, and learnable weather queries with a transformer decoder.
- Weather Queries: Weather queries activate differently for fog, rain, and snow, with mixed-weather images activating queries associated with both degradations.Fog queries attend broadly to afflicted regions, rain queries focus sparsely on rain details, and snow queries respond more strongly to snow images.
- Efficiency: 0.14 seconds per 256 × 256 image and 31 M parameters make TransWeather faster and smaller than All-in-One’s 44 M parameters.The comparison reports feed-forward inference time and parameter counts for the competing models.
- Comparison with All-in-One: TransWeather differs from All-in-One by using a transformer backbone, one encoder, stable L1 and perceptual losses, and no adversarial training or neural architecture search.The transformer is designed for low-level vision with added emphasis on smaller patches.
- Limitations: High-intensity real-world rain and rain splattering remain unresolved limitations for TransWeather and other methods.The authors attribute difficulty to real-rain streak size and intensity, while splattering occurs when intense rain hits objects or people.
6. Conclusion
TransWeather addresses all-weather image restoration with a single encoder-decoder model and learnable weather queries. Across synthetic and real-world datasets, it reports good restoration, faster inference, and better visual results, while high-intensity rain remains challenging.
- Conclusion: TransWeather uses a single encoder-decoder network with learnable weather type queries to identify degradation and guide weather removal.Its transformer encoder operates on sub-patches to help remove small weather degradations efficiently.
- Conclusion: TransWeather obtains good results across multiple synthetic and real-world datasets while also achieving faster inference.The conclusion presents these as outcomes of using one model instance for varied weather degradations.
- Conclusion: TransWeather produces better visual results on real-world adverse-weather images than previous methods.The conclusion reports this result after experiments on multiple synthetic and real-world datasets.