Source-linked AI summary
Efficient All-in-One Weather Restoration using Spectral Harmonization
Paula Garrido-Mellado, Daniel Feijoo, Yuning Cui, Alvaro Garcia, Marcos V. Conde
TL;DR
Adverse weather restoration must handle diverse and simultaneous degradations, while existing approaches can require substantial computation and may process spectral components too uniformly. FReSH-IR uses multiscale specialized high- and low-frequency processing within a lightweight all-in-one architecture. It achieves competitive restoration quality with 80% fewer parameters, up to 90% lower computational cost, and 5× faster inference than leading approaches, supporting constrained-resource deployment.
Problem
Diverse and simultaneous weather degradations make separate models impractical, while existing spectral approaches process components too uniformly and struggle with localized artifacts.
Method
FReSH-IR is a lightweight all-in-one model that decomposes multiscale features into high- and low-frequency components and processes them with specialized blocks.
Results
FReSH-IR achieves restoration quality competitive with leading AIO methods while using 80% fewer parameters, up to 90% fewer MACs, and 5× faster inference than SSGFormer.
Takeaways & Limitations
FReSH-IR offers an efficiency-quality trade-off suitable for edge and on-device weather restoration.
Takeaways & Limitations
Performance on real-world scenes remains limited for demanding downstream tasks such as autonomous driving because realistic weather-restoration training datasets are scarce.
Abstract
from arXiv · showhide
Adverse weather conditions such as rain, haze, and snow significantly degrade image quality, posing challenges for both human perception and physical AI. Existing restoration methods require large computational budgets, struggling to process high-resolution images and handle different degradations. In this paper, we present Frequency Reconstruction via Spectral Harmonization, a novel lightweight all-in-one restoration method that explicitly decomposes feature representations into high- and low-frequency components at each scale of a hierarchical encoder-decoder architecture. By combining spectral decomposition with spatial processing through Fourier-based skip connections, FReSH-IR captures complementary frequency information without sacrificing spatial detail. Our approach achieves similar restoration quality with 80% fewer parameters and operations than transformer-based models. Extensive experiments demonstrate that our method offers a great efficiency-performance trade-off, highlighting its practical applications in constrained-resource systems.
1. Introduction
Adverse weather and overlapping degradations make unified image restoration difficult, while existing spectral methods process frequencies too uniformly. FReSH-IR addresses this with specialized high- and low-frequency processing and achieves competitive quality at substantially lower computational cost.
- Adverse weather severely degrades visual information, motivating restoration methods for rain, snow, raindrops, and haze.
- Multiple simultaneous degradations make separate task-specific models impractical and complicate generalization across diverse weather conditions.
- Existing frequency-domain methods either process the spectrum globally or apply identical operations to all components, limiting handling of localized artifacts.
- FReSH-IR decomposes features into high- and low-frequency components at each scale and processes them with specialized blocks in an asymmetric UNet.
- 80% fewer parameters, up to 90% fewer MACs, and 5× faster inference accompany restoration quality competitive with leading AIO methods, including SSGFormer.
2. Related Work
Weather restoration research evolved from single-degradation models toward all-in-one systems that address multiple real-world degradations with one model. Related work spans CNN, GAN, Transformer, Mamba, frequency-domain, and prior-based approaches.
- Early adverse-weather restoration used task-specific models, each targeting a single degradation.
- Raindrop Removal: Raindrop removal progressed from video temporal information to CNN and GAN approaches that learn degradation priors and use edge information.
- Dehazing: Dehazing methods range from CNNs and atmospheric maps to attention, density-aware networks, and spatial-frequency baselines.
- Deraining: Deraining increasingly uses frequency-domain techniques, alongside Transformer and emerging Mamba-based methods.
- Desnowing: Desnowing research incorporates transparency, context, semantic and geometric priors, wavelets, Transformers, and multimodal optimization.
- All-in-One Restoration: All-in-one restoration emerged to replace several computationally challenging task-specific models with one model for diverse degradations, often using Transformers.
3. Method
FReSH-IR uses a hierarchical U-shaped architecture that explicitly separates and processes low- and high-frequency features at each scale. Frequency-aware skip fusion, attention, and specialized branches combine structural information with fine-grained detail for weather restoration.
- Overall Framework: FReSH-IR uses a U-shaped CNN backbone with hierarchical encoder-decoder processing and explicit low- and high-frequency decomposition at each scale.The encoder progressively downsamples features, while the decoder progressively recovers them.
- Frequency-Aware Encoder: At each encoder level, a Spectral Decomposition Block uses a smooth Fourier mask to separate low and high frequencies.The low-frequency component captures coarse structure, while the high-frequency component retains fine-grained details and localized degradation patterns.
- Frequency-Aware Encoder: The encoder refines low-frequency features with Residual Blocks and sums them with high-frequency features to preserve fine structural details throughout encoding.The design explicitly preserves high-frequency information that is critical for weather artifact removal.
- Dual-Attention Module: A Dual-Attention Module combines spatial and frequency-domain operations to emphasize important regions in the deepest feature representation.It uses pooled frequency components to form spatial attention and applies channel-wise depthwise-convolution refinement before final fusion.
- Spectral Skip Connections: The Spectral Harmonization Block decomposes upsampled decoder features and fuses each component with the corresponding encoder skip feature instead of merging all channels uniformly.This selectively integrates structural low-frequency information and high-frequency detail at each scale.
- Frequency Transformer Blocks: Frequency Transformer Blocks process low- and high-frequency branches independently before fusing them, using dilated convolutions for multiscale structure and a transformer block for fine-grained dependencies.The branches are fused in the final Frequency Transformer Block at each level.
4. Experimental Results
FReSH-IR is evaluated on standard synthetic and real-world all-in-one weather benchmarks, with qualitative and downstream-task results alongside computational comparisons. It delivers competitive restoration while substantially reducing resource requirements.
- Experimental Setup: The benchmark combines RainDrop, Outdoor-Rain, and Snow100K datasets for multi-weather restoration evaluation.The training set contains 1,069 RainDrop, 9,000 Outdoor-Rain, and 9,000 Snow100K images.
- Quantitative Results: 80% fewer parameters, up to 90% fewer MACs, and 5× faster inference are achieved relative to top-performing approaches such as SSGFormer.The paper reports competitive restoration quality alongside these efficiency reductions.
- Downstream Tasks: Applying FReSH-IR before downstream processing consistently improves performance on Rainy Cityscapes.The study evaluates FastSAM-s, SegFormer-B3, and YOLO11 across segmentation and detection tasks.
- Downstream Tasks: Restoration improves detector confidence and increases the number of correctly identified objects on degraded images.These qualitative detection comparisons are reported in Figure 6.
5. Discussion
The discussion links FReSH-IR’s efficiency to edge deployment and examines architectural choices through qualitative comparisons and ablations. It also identifies limited realistic training data as a remaining constraint for demanding real-world tasks.
- Efficiency Discussion: FReSH-IR is designed for edge deployment, combining comparable efficiency with memory, speed, and performance advantages.The discussion contrasts it with MWFormer-L’s 182.81M parameters and WGWS’s doubled runtime.
- Qualitative Comparisons: FReSH-IR provides visual results comparable to state-of-the-art methods across synthetic and real-world weather benchmarks.The comparisons cover RainDrop, Snow100K, Outdoor-Rain, and RealSnow.
- Ablation Studies: Reversing the Fourier-mask cutoff reduces performance, supporting restoration of high-frequency information at upper decoder levels.The ablation indicates that processing high frequencies with downsampled features is ineffective for image restoration.
- Ablation Studies: Restoring low- and high-frequency components in the encoder provides no improvement over restoring low frequencies in the encoder and high frequencies in the decoder.This ablation reinforces the network’s asymmetric frequency-restoration strategy.
- Ablation Studies: Replacing the dual-attention module with a standard NAFBlock decreases performance, supporting the module’s role in the architecture.The discussion associates this result with the GAP operation described in the design preliminaries.
- Limitations: Real-world performance remains limited for demanding downstream tasks such as autonomous driving because realistic weather-restoration training data are scarce.The paper identifies dataset creation, rather than further architectural development, as a future priority.
6. Conclusion
FReSH-IR is presented as an efficient all-in-one weather restoration method for edge applications. It combines spectral decomposition and specialized frequency branches to achieve competitive restoration quality with substantially lower resource use.
- FReSH-IR decomposes and fuses spectral information at each network scale through the Spectral Harmonization Block and specialized frequency branches.
- Up to 90% lower computational cost in memory, operations, and runtime is reported while maintaining restoration quality competitive with heavier models.
- The method is designed for edge applications and is described as a practical pathway toward real-world, on-device image restoration.
A. Dataset overview
The evaluation uses multi-weather datasets covering synthetic and real rain, haze, raindrops, and snow, with additional unseen downstream perception benchmarks.
- The training set combines 9,000 Outdoor-Rain images, 1,069 RaindropData images, and 9,000 Snow100K images.Outdoor-Rain contains synthetic haze and rain streaks, RaindropData contains real raindrops, and Snow100K contains synthetic snow.
- Rain and fog are tested on 750 synthetic Outdoor-Rain image pairs with varying rain-streak and haze intensities and directions.
- Raindrop removal is evaluated on 58 real image pairs containing raindrops with different shapes and intensities.
- Desnowing is evaluated on 16,801 synthetic Snow100K-L pairs, alongside 1,329 realistic snow images without ground truth for visual results.
- Downstream evaluation uses unseen Rainy Cityscapes and CSD datasets for semantic segmentation and object detection under synthetic rain or snow.Rainy Cityscapes provides urban driving scenes with rain, while CSD is a synthetic snow benchmark.
B. Frequency analysis.
The frequency analysis finds FFT most precise for isolating weather artifacts, while deeper representations require GAP-based extraction because Fourier decomposition loses meaningful information.
- FFT decomposition yields the most precise isolation of weather artifacts across the three evaluated degradation types.
- Resampling-based extraction fails to provide meaningful information for isolating weather-induced degradations.
- Beyond a certain network depth, Fourier decomposition loses meaningful information, prompting a transition to GAP-based extraction.
- GAP-based extraction retains structural image edges despite operating on deep feature representations.
C. Additional efficiency results.
Efficiency experiments compare inference across resolutions on an RTX 4090 and Jetson Orin NX. FReSH-IR handles high resolutions and scales better than alternatives, which encounter out-of-memory errors.
- The runtime study evaluates image-resolution scaling on a 24GB RTX 4090 GPU and a 16GB Jetson Orin NX.
- Only FReSH-IR and WGWS handle resolutions of at least 1440p.
- FReSH-IR demonstrates superior inference-time scalability relative to image resolution compared with other methods.
- Alternative methods encounter out-of-memory errors in the resolution experiments.
D. Additional ablation studies
Additional ablations support processing both frequency components across encoder and decoder stages, using specialized branches, depth-specific masks, additive skip fusion, and 24 channels for efficiency.
- Processing only high frequencies in the encoder significantly decreases performance, supporting low-frequency restoration in the encoder.
- Processing only high frequencies in the decoder significantly decreases performance, supporting restoration of both frequency components there.
- Replacing specialized frequency branches with separate NAF-Blocks adds computational cost for negligible performance improvement, while Restormer processing lowers restoration performance.
- Concatenating skip features slightly increases cost and severely degrades performance compared with additive fusion.
- Using different Fourier masks at each depth improves restoration performance compared with assigning the same cutoff value across all depths.
- A 24-channel configuration achieves an optimal balance between performance and computational efficiency.
E. Qualitative results
Qualitative comparisons on synthetic and real benchmarks show that FReSH-IR achieves results similar to state-of-the-art methods at substantially lower computational cost.
- FReSH-IR produces qualitative results comparable to state-of-the-art methods across synthetic and real benchmarks.The comparisons include RainDrop, Snow100K, and OutdoorRain datasets.
- Additional qualitative results are reported for realistic Snow100K images.