TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions

Introduction

• Rain, fog, and snow reduce image visibility and can impair detection, segmentation, and depth estimation systems used in autonomous navigation and surveillance [3] [5] [41] [59] [35].The introduction also identifies clean weather-free images as desirable in photography.Source: §1 · p. 1
• Earlier methods often addressed one degradation at a time, requiring separate models, while All-in-One used multiple encoders and remained computationally complex.Other generic restoration networks discussed in the introduction were fine-tuned on individual datasets rather than using one model for all weather-removal tasks.Source: §1 · pp. 1–2
• TransWeather uses one encoder and one decoder, with learned weather-type queries that match encoder features to adapt restoration to the degradation present.The decoder fuses decoded and hierarchical encoder features, then projects them to image space with a convolutional block.Source: Figure 1 · p. 1, §1 · p. 2
• Its Intra-PT encoder blocks attend within sub-patches to capture fine details such as small rain or snow artifacts, using efficient attention to keep computational complexity low.The paper reports that adding Intra-PT blocks improves transformer performance and adaptation to weather removal.Source: §1 · p. 2
• TransWeather outperforms All-in-One and task-specific methods across multiple test datasets, runs quickly at inference, and performs better than previous methods on real-world images.The reported test conditions include rain, snow, fog, and combinations of weather degradations.Source: §1 · p. 2

Weather-specific restoration

• Rain, fog, and raindrop removal have been studied with recurrent, GAN, CNN, attention, and transfer-learning approaches.Examples include rain-layer decomposition, dehazing with atmospheric light and transmission maps, and dual attention for raindrop removal.Source: §2 · pp. 2–3
• Snow-removal methods include CNN-based Desnow-Net [29], stacked dense networks, JSTASR, and DDMSNet.JSTASR uses size- and transparency-aware processing, while DDMSNet uses semantic and geometric priors.Source: §2 · p. 3

All-in-One weather removal

• All-in-One addresses multiple weather degradations with task-specific encoders, a common decoder, degradation classification, and neural architecture search.Its discriminator routes loss to specific encoders.Source: §2 · p. 3

Transformers in low-level vision

• Low-level vision transformers include U-former [55] and Swin-IR [27], alongside transformer methods developed specifically for dehazing [74] and deraining [48].Image processing transformer [4] demonstrated the value of large-scale pretraining for low-level applications.Source: §2 · p. 3

TransWeather

• TransWeather uses a single encoder-decoder model for all adverse-weather tasks, with intra-patch transformer blocks and weather-type queries guiding restoration.The encoder is modified for low-level tasks, and the decoder learns the task from weather-type queries.Source: §2 · p. 3

3. Proposed Method - TransWeather

• Unlike All-in-One [23], which uses a different encoder for each weather type, TransWeather uses a weather-agnostic encoder-decoder with learned queries to share information across tasks efficiently.The paper motivates this shared formulation as a way to use complementary information between weather-removal tasks.Source: §3 · p. 3

3.1. Network Architecture

• The pipeline patchifies a degraded H × W × 3 image, progressively reduces resolution through transformer stages, then projects decoded and hierarchical features to a clean H × W × 3 output.Source: §3.1 · p. 3

3.1.1 Transformer Encoder

• The encoder builds hierarchical coarse-to-fine features with overlapped patch merging, while intra-patch blocks attend over half-size subpatches to capture fine details for smaller degradations.Intra-patch features are added to the main transformer-block features at each stage.Source: §3.1.1 · pp. 3–4

3.1.2 Transformer Decoder

• Learnable weather-type queries serve as decoder queries, attending to last-stage encoder features as keys and values; the resulting task feature is fused with hierarchical encoder features.Source: §3.1.2 · p. 5, Figure 3 · p. 5

3.1.3 Convolutional Projection Block

• The convolutional projection block uses four convolutional layers, upsampling before each layer and encoder skip connections, with tanh in the final layer.Source: §3.1.3 · p. 5

3.2. Loss

• Training combines smooth L1 reconstruction loss with VGG16 perceptual loss computed from layers 3, 8, and 15.The VGG16 feature extractor is pretrained on ImageNet, and the perceptual-loss contribution is controlled by λ.Source: §3.2 · p. 5

4.1. Datasets

• Training uses 9,000 Snow100K images, 1,069 Raindrop images, and 9,000 Outdoor-Rain images; testing uses Test1, Snow100k-L, RainDrop, and real-world rain images.The combined training data is termed “All-Weather”; Snow100K and Outdoor-Rain images are synthetic, while Raindrop training images are real.Source: §4.1 · p. 5

4.2. Implementation Details

• Training uses Adam at learning rate 0.0002 for 200 epochs with batch size 32, on an NVIDIA RTX 8000 GPU.The learning rate is halved after 100 and 150 epochs.Source: §4.2 · p. 5

4.3. Comparison with state-of-the-art methods

• Comparisons include task-specific rain-drop, snow, and rain-plus-fog methods, Swin-IR, and All-in-One; TransWeather and All-in-One each use one model instance for all tasks.The task-specific methods are fine-tuned for specific datasets.Source: §4.3 · pp. 5–6

4.3.1 Referenced Quality Metrics

• PSNR and SSIM evaluate Test1, Snow100K-L, and RainDrop; Test1 comparisons apply deraining and dehazing in both orders, while Swin-IR and MPRNet receive task-specific training.The passage notes that CCN and DDMSNet fine-tuned on individual datasets outperform All-in-One, but does not specify TransWeather’s relative quantitative performance.Source: §4.3.1 · p. 6

4.3.2 Visual Quality Comparison

• On synthetic images, TransWeather removes fog and rain streaks that other methods, including All-in-One, fail to remove together, and also removes very small snow particles.These comparisons are shown for Test1 and Snow100K-L in Figures 4 and 5.Source: §4.3.2 · p. 6
• On RainDrop and real-world images, TransWeather removes finer rain streaks or drops than the previous methods.These comparisons are shown in Figures 6 and 7.Source: §4.3.2 · p. 6

Ablation Study

• Each successive addition—hierarchical encoding, intra-patch blocks, and learnable weather queries with a transformer decoder—improves performance on the RainDrop test dataset.The ablation models were trained on All-Weather; Table 4 reports results on the RainDrop test dataset.Source: §5 · p. 7, Table 4 · p. 7

What do the weather queries learn?

• Different weather queries activate for fog, rain, or snow, while multiple query types activate together for images degraded by multiple weather conditions.Fog queries attend broadly across the image, while rain-query attention is sparse around rain details.Source: §5 · p. 7, Figure 8 · p. 8

Inference Time

• 0.14 seconds per 256 × 256 image is TransWeather’s reported inference time; its 31 M parameters are fewer than All-in-One Network’s 44 M.The paper says TransWeather is faster than previous weather-removal methods, without giving their specific times here.Source: §5 · p. 7

Differences from All-in-One

• Unlike All-in-One, TransWeather uses a single transformer encoder, and trains with L1 and perceptual losses rather than multiple encoders, adversarial training, and neural architecture search.The paper also describes TransWeather as faster, smaller, and better in quantitative and qualitative performance than All-in-One.Source: §5 · p. 7

Limitations

• TransWeather struggles with some real-world images containing high-intensity rain, especially when streaks differ in size or intensity or rain splatters on surfaces and people.The paper notes that removing this splattering effect remains a limitation for all methods, including TransWeather.Source: §5 · pp. 7–8

• TransWeather addresses all adverse-weather removal with a single encoder-decoder network and learnable weather-type queries that inform the removal process.Source: §6 · p. 8
• Figure 9 identifies a limitation: high-intensity rain and rain splattering are not removed by any method.Source: Figure 9 · p. 8
• Experiments across synthetic and real-world datasets report good results with one model instance, faster inference, and better visual results on real-world adverse-weather images.Source: §6 · p. 8
• Its transformer encoder works on sub-patches to help remove small weather degradations more efficiently.Source: §6 · p. 8