PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient All-in-One Weather Restoration using Spectral Harmonization

Paula Garrido-Mellado, Daniel Feijoo, Yuning Cui, Alvaro Garcia, Marcos V. Conde

arXiv:2609.02839v1cs.CV

TL;DR

A unified model for diverse, interacting weather degradations remains challenging, while storing and selecting separate models is impractical. FReSH-IR decomposes features into frequency components at each scale and achieves competitive restoration quality with 80% fewer parameters, up to 90% fewer MACs, and 5× faster inference than top-performing approaches such as SSGFormer.

  • Problem

    A unified model for diverse, interacting weather degradations remains challenging, while storing and dynamically selecting separate models is impractical.

    Source: §1 · p. 1

  • Method

    FReSH-IR decomposes features into high- and low-frequency components at each scale and processes them through specialized blocks.

    Source: §1 · p. 2

  • Results

    80% fewer parameters, up to 90% fewer MACs, and 5× faster inference than top-performing approaches such as SSGFormer, with competitive restoration quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FReSH-IR offers a pathway toward real-world, on-device image restoration.

    Source: §6 · p. 9

    Real-world performance remains limited for demanding downstream tasks such as autonomous driving, which the authors attribute to limited realistic weather-restoration training data.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

Adverse weather conditions such as rain, haze, and snow significantly degrade image quality, posing challenges for both human perception and physical AI. Existing restoration methods require large computational budgets, struggling to process high-resolution images and handle different degradations. In this paper, we present Frequency Reconstruction via Spectral Harmonization, a novel lightweight all-in-one restoration method that explicitly decomposes feature representations into high- and low-frequency components at each scale of a hierarchical encoder-decoder architecture. By combining spectral decomposition with spatial processing through Fourier-based skip connections, FReSH-IR captures complementary frequency information without sacrificing spatial detail. Our approach achieves similar restoration quality with 80% fewer parameters and operations than transformer-based models. Extensive experiments demonstrate that our method offers a great efficiency-performance trade-off, highlighting its practical applications in constrained-resource systems.

1. Introduction

FReSH-IR is a lightweight all-in-one weather restoration model that processes high- and low-frequency features with specialized blocks at each scale. It achieves competitive restoration quality with fewer computational resources than leading AIO methods.

  • Snow, rain, raindrops, and haze degrade visual information, and restoration research has used image priors such as edges and Fourier frequency patterns.Source: §1 · p. 1
  • Early methods targeted single weather types, while concurrent degradations and the impracticality of storing and selecting multiple models make a unified restoration framework challenging.Source: §1 · p. 1
  • AIRFormer, Fourmer, and AdaIR [10] use frequency-domain methods, but global or uniform processing [10] limits their handling of localized artifacts such as raindrops and snowflakes.Source: §1 · pp. 1–2
  • FReSH-IR uses an asymmetric UNet to decompose features into high- and low-frequency components at each scale and process them through specialized blocks.Source: §1 · p. 2
  • FReSH-IR achieves competitive restoration quality with 80% fewer parameters, up to 90% fewer MACs, and 5× faster inference than top-performing approaches such as SSGFormer [18].Source: §1 · p. 2

2. Related Work

Early adverse-weather restoration used task-specific models, while computational challenges from deploying several models helped motivate single-model all-in-one restoration. Related work spans raindrop removal, dehazing, deraining, and desnowing, alongside unified models using varied architectures and mechanisms.

2.1. Single-task Adverse Weather Removal

  • Early adverse-weather removal models targeted one degradation type, forming the task-specific approach to restoration.This contrasts with the later direction of using one model to restore all degradation types.Source: §2.1 · p. 2
  • Raindrop removal progressed from video temporal information to CNN- and GAN-based learned priors, edge information, and day/night drop- and background-focused data [21].The dataset in [21] contains both day and night images, including drop-focused and background-focused images.Source: §2.1 · p. 2
  • Dehazing research ranges from CNNs [1] to Vision Transformers, with approaches using atmospheric luminosity and transmission maps, content-guided attention [7], hierarchical density awareness, and spatial-frequency processing.The cited work is described as a strong baseline operating in both spatial and frequency domains.Source: §2.1 · p. 2
  • Deraining methods have used learning-based approaches, frequency-domain techniques [16], Transformers [6], and more recently Mamba-based methods.Frequency-domain work is described as common and effective, with improved visual and quantitative results.Source: §2.1 · p. 2
  • Desnowing work has explored transparency awareness, context-aware CNNs, semantic and geometric priors, wavelets, Transformers [3], and multimodal optimization and language models.Lai et al. used Multi-Model Optimization and Multimodal Large Language Models.Source: §2.1 · p. 2

2.2. All-in-One Image Restoration

  • Using several task-specific models for real-world degradations is computationally challenging, motivating single-model all-in-one restoration, first proposed for adverse weather by Li et al..AIO methods include Transformer architectures, frequency mining [10], Mixture-of-Experts [34], histogram-based attention, prompt-based methods [8], and a new training paradigm with degradation analysis.Source: §2.2 · p. 2

3. Method

FReSH-IR is a U-shaped CNN that separates and processes low- and high-frequency features through its encoder, bottleneck, decoder, and spectral skip connections. Its design combines frequency-aware restoration with pixel, perceptual, and spectral supervision.

3.1. Preliminaries: Frequency Spectrum Analysis

  • The encoder uses stricter FFT masks early, relaxes them at deeper levels, and switches to GAP-based extraction at the bottleneck to retain structural edges for fine-detail reconstruction.The analysis compares four high-frequency extraction methods; the selected hybrid strategy uses FFT decomposition early and GAP at the bottleneck.Source: §3.1 · pp. 2–3

3.2. Overall Framework

  • FReSH-IR uses a U-shaped CNN that projects the input into shallow features, processes them hierarchically, applies bottleneck attention, and progressively recombines encoder features in the decoder.The decoder keeps high- and low-frequency components separate within each level until its final block.Source: §3.2 · p. 3

3.3. Frequency-Aware Encoder

  • At each encoder level, SDB decomposes the input feature map into low- and high-frequency components with a smooth Fourier mask; high-frequency output retains fine-grained details and localized degradation patterns, while residual blocks refine the low-frequency component.The cutoff ratios are α = [0.4, 0.32, 0.23, 0.12] across encoder levels.Source: §3.3 · p. 3

3.4. Dual-Attention Module

  • At the bottleneck, DAM derives spatial attention from pooled low- and high-frequency features, refines channels with depthwise paths, and uses GAP-based high-frequency extraction to emphasize structural content.The spatial attention map modulates one depthwise path.Source: §3.4 · p. 3

3.5. Spectral Skip Connections

  • At each decoder level, SHB replaces additive skips with frequency-aware fusion, matching decomposed upsampled features to encoder features so the decoder can selectively integrate structural low frequencies and high-frequency detail.The decomposition uses the encoder counterpart’s mask.Source: §3.5 · pp. 3–4, Figure 3 · p. 4

3.6. Frequency Transformer Blocks (FTBs)

  • FTBs process low- and high-frequency branches independently before fusion: parallel dilated depthwise convolutions capture multi-scale structure, while a transformer block models long-range dependencies in fine details.A shared FFN refines the concatenated branches; the final block at each level projects them back to the original channel dimension.Source: §3.6 · pp. 4–5

3.7. Regularization and Loss Functions

  • Training combines pixel L1, bottleneck enhance, and Fourier edge losses weighted 1, 1, and 0.05, respectively, omitting enhance loss during the first 10 epochs.Enhance loss supervises the bottleneck against ground truth downsampled by a factor of 16 and includes a VGG-19 perceptual term; Fourier edge loss supervises high-frequency detail.Source: §3.7 · p. 5

4. Experimental Results

Experiments compare FReSH-IR on synthetic and real-world weather-restoration benchmarks, assess visual restoration, and evaluate effects on downstream perception tasks.

4.1. Implementation Details

  • Training uses a three-dataset all-in-one benchmark, 384 × 384 crops, batch size 16, and 500 epochs with AdamW.Training data comprise RainDrop, Outdoor-Rain, and Snow100K; testing uses their specified RainDrop, Test1, and Snow100K-L test sets.Source: §4.1 · p. 5

4.2. Quantitative Results

  • FReSH-IR uses 52× and 10× fewer parameters than MWFormer and TransWeather, with overall performance gains of +1.67dB and +0.08dB; it improves WGWS by +0.49dB and runs 2× faster.The paper also reports competitive results on a real-world benchmark.Source: §4.2 · p. 5, Table 1 · p. 6

4.3. Qualitative Results

  • Visual results are comparable to state-of-the-art methods on synthetic and real-world benchmarks while saving 80% of computational cost against SSGFormer [18].The comparisons include TransWeather, WGWS, and Histoformer.Source: §4.3 · p. 5

4.4. Downstream tasks

  • On Rainy Cityscapes, restoration preprocessing consistently improves detection and segmentation over degraded input, with detection confidence and correctly identified objects also increasing.The evaluated models are FastSAM-s, SegFormer-B3, and YOLO11.Source: §4.4 · pp. 5–6

5. Discussion

The discussion examines FReSH-IR’s efficiency and architectural choices, then identifies limited realistic training data as a challenge for demanding real-world tasks.

5.2. Ablation Studies

  • Qualitative comparisons show restoration results comparable to SOTA across RainDrop, Snow100k, OutDoor Rain, and RealSnow, with notable efficiency on RealSnow.On downstream object detection, FReSH-IR recovers fine details that help the detector recognize objects missed in degraded inputs; Table 3 covers downstream task performance on Rainy Cityscapes before and after restoration.Source: Table 3 · p. 6, Figure 4 · p. 7, Figure 5 · p. 7

5.1. Efficiency Concerns

  • +1.67dB average performance over TransWeather accompanies comparable efficiency and better perceptual metrics on real-world data.The paper also reports that MWFormer-L uses 182.81M parameters and WGWS doubles FReSH-IR’s runtime; it describes FReSH-IR as efficient across all indicators and suited to edge deployment.Source: §5.1 · p. 6, Figure 7 · p. 8

5.2. Ablation Studies

  • Processing high-frequency information with downsampled features reduced performance, while restoring high frequencies at upper decoder levels supports the design choice.Restoring both low- and high-frequency information in the encoder yielded no improvement; replacing dual attention with NAFBlock [2] decreased performance. The paper also reports studying training losses in Table 5 and further ablations in supplementary material.Source: §5.2 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5.3. Limitations

  • Limited realistic training data constrains performance on real-world scenes for demanding downstream tasks such as autonomous driving.The paper notes that collecting real weather-restoration samples is difficult and recommends prioritizing dataset creation over further architectural development.Source: §5.3 · p. 8

6. Conclusion

FReSH-IR is an efficient all-in-one weather restoration method designed for edge applications, using frequency-aware processing to achieve competitive restoration quality at substantially lower computational cost.

6. Conclusion

  • FReSH-IR achieves restoration quality competitive with far heavier models at up to 90% lower memory, operations, and runtime cost.The conclusion frames this trade-off as a deliberate design choice to recover most state-of-the-art quality with fewer resources.Source: §6 · p. 8
  • Its Spectral Harmonization Block and specialized frequency branches decompose and fuse spectral information at each scale.Source: §6 · p. 8
  • The work points toward practical on-device image restoration and further efficiency-first frameworks that treat deployment constraints as a core design objective.Source: §6 · p. 9

A. Dataset overview

The study trains on synthetic and real weather-degradation images, evaluates restoration on separate rain, raindrop, and snow test sets, and tests downstream perception on unseen rainy-city and snow datasets.

A. Dataset overview

  • Training uses 9,000 Outdoor-Rain images, 1,069 RaindropData images, and 9,000 Snow100K training images, following prior studies.Outdoor-Rain includes synthetic haze and rain-streak images; RaindropData contains real raindrop images, and Snow100K images are synthetically snow-affected.Source: A. Dataset overview · p. 11
  • Rain and fog restoration are tested on 750 synthetic degraded-image pairs from Outdoor-Rain Test1.The pairs vary in rain-streak and haze intensity and direction.Source: A. Dataset overview · p. 11
  • Deraining of raindrops is tested on 58 real-image pairs from the Raindrop A subset of RaindropData.The raindrops vary in shape and intensity.Source: A. Dataset overview · p. 11
  • Desnowing is tested on 16,801 synthetic snow-image pairs from Snow100K-L, with realistic visual results shown on 1,329 Snow100K realistic images without ground truth.The Snow100K-L test set provides paired images; the realistic subset does not include ground truth.Source: A. Dataset overview · p. 11
  • Downstream perception is evaluated on unseen Rainy Cityscapes and CSD, synthetic rain and snow datasets with 1,188 and 2,000 test images, respectively.Rainy Cityscapes includes semantic-segmentation and object-detection ground truth; CSD is a synthetic snow benchmark.Source: A. Dataset overview · p. 11

B. Frequency analysis.

FFT most precisely isolates weather artifacts while suppressing clean-image structure, but at greater depth it loses meaningful information and the analysis transitions to GAP extraction.

B. Frequency analysis.

  • FFT decomposition most precisely isolates weather artifacts while suppressing structural edges from the underlying clean image.The analysis is motivated by weather artifacts appearing mainly in degraded images’ high-frequency domain.Source: B. Frequency analysis · p. 11
  • GAP also extracts degradation strongly while retaining clean-image edges; Sobel makes artifacts less visible with stronger edge responses, and resampling yields no meaningful information.GAP’s retained edges help preserve fine structural detail during restoration; resampling is unsuitable for this extraction purpose.Source: B. Frequency analysis · p. 11
  • Beyond a certain depth, Fourier decomposition loses meaningful information, so extraction transitions to GAP, which retains structural image edges in deep features.The analysis illustrates both behaviors in Figure 8(b).Source: B. Frequency analysis · p. 11

C. Additional efficiency results.

FReSH-IR handles resolutions of at least 1440p and scales inference time with image resolution better than other methods, which encounter out-of-memory errors.

D. Additional ablation studies

The ablations support restoring both frequency components, specialized frequency branches, additive skip fusion, and depth-specific masks; the reported 24-channel configuration balances restoration performance and computational efficiency.

D. Additional ablation studies

  • Using only high frequencies in either encoder or decoder significantly decreases performance, supporting restoration of both frequency components at both stages.The encoder and decoder experiments are reported in Table 7(a).Source: D. Additional ablation studies · p. 11, Table 7 · p. 12
  • Separate NAF-Blocks for each decoder frequency bring negligible improvement at higher cost, while Restormer processing of high frequencies costs slightly less but significantly reduces restoration performance.These decoder experiments support specialized branches for high- and low-frequency components.Source: D. Additional ablation studies · p. 11, Table 7 · p. 12
  • Changing SHB skip fusion from addition to concatenation slightly increases cost and severely degrades performance.The comparison is included in Table 7(a).Source: D. Additional ablation studies · p. 11, Table 7 · p. 12
  • Using different masks at each depth improves restoration performance compared with setting the same cutoff value at every depth.This experiment supports depth-specific cutoff values.Source: D. Additional ablation studies · p. 11, Table 7 · p. 12
  • The ablation also tested replacing the proposed skip-connection mechanism with traditional skip-connections.The supplied passage does not state the experiment's outcome.Source: D. Additional ablation studies · p. 11
  • The 24-channel configuration achieves an optimal balance between restoration performance and computational efficiency.The channel-count ablation is reported in Table 7(b).Source: D. Additional ablation studies · p. 12, Table 7 · p. 12

E. Qualitative results

The section presents qualitative comparisons across synthetic and real benchmarks, including additional results for the realistic Snow100K subset.

Related papers

Loading 2609.02839v1…