Source-linked AI summary
Restoring Images in Adverse Weather Conditions via Histogram Transformer
Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, Xiaochun Cao
TL;DR
Adverse-weather Transformers often restrict attention spatially or by channel, limiting long-range feature capture. Histoformer uses intensity-binned histogram attention, dynamic-range convolution, and correlation loss for unified restoration, with reported state-of-the-art performance and real-world restoration benefits.
Problem
Existing adverse-weather Transformers restrict self-attention to fixed spatial ranges or channels, limiting long-range spatial feature capture.
Method
Histoformer uses intensity-binned histogram self-attention, dynamic-range convolution, multi-range feed-forward processing, and correlation loss for unified weather removal.
Results
The method attains state-of-the-art performance across various datasets and is reported effective for real-world restoration and downstream detection.
Takeaways & Limitations
Dynamic-range spatial attention enables the model to process similarly degraded pixels across long spatial ranges while retaining efficient unified restoration.
Abstract
from arXiv · showhide
Transformer-based image restoration methods in adverse weather have achieved significant progress. Most of them use self-attention along the channel dimension or within spatially fixed-range blocks to reduce computational load. However, such a compromise results in limitations in capturing long-range spatial features. Inspired by the observation that the weather-induced degradation factors mainly cause similar occlusion and brightness, in this work, we propose an efficient Histogram Transformer (Histoformer) for restoring images affected by adverse weather. It is powered by a mechanism dubbed histogram self-attention, which sorts and segments spatial features into intensity-based bins. Self-attention is then applied across bins or within each bin to selectively focus on spatial features of dynamic range and process similar degraded pixels of the long range together. To boost histogram self-attention, we present a dynamic-range convolution enabling conventional convolution to conduct operation over similar pixels rather than neighbor pixels. We also observe that the common pixel-wise losses neglect linear association and correlation between output and ground-truth. Thus, we propose to leverage the Pearson correlation coefficient as a loss function to enforce the recovered pixels following the identical order as ground-truth. Extensive experiments demonstrate the efficacy and superiority of our proposed method. We have released the codes in Github.
1 Introduction
Adverse-weather restoration is important for visual quality and downstream vision safety, but existing Transformer methods restrict attention to channels or fixed spatial ranges. Histoformer addresses this limitation with dynamic-range histogram attention, complementary modules, and correlation-aware training.
- Adverse weather degrades visual quality and harms downstream tasks including object detection and depth estimation.
- Existing Transformer restorers limit self-attention to fixed spatial ranges or channel dimensions, weakening long-range spatial feature modeling.
- Histoformer targets unified adverse-weather removal with histogram self-attention that focuses dynamically on weather-induced obstructions.
- Histogram self-attention sorts spatial features into intensity-based bins and applies attention across bin or frequency dimensions.
- Dynamic-range convolution sorts pixels horizontally and vertically before separable convolution, while DGFF captures multi-scale and multi-range information.
- The method adds a correlation loss to improve linear association between recovered and ground-truth images and reports state-of-the-art performance across datasets.
- The approach is also reported to restore real-world images and support downstream detection.
2 Related Work
Related work spans task-specific restoration for rain, raindrops, snow, and fog, alongside unified all-in-one models. Histoformer differs from prior Transformer approaches by using dynamic-range spatial attention for similarly patterned degradation.
- Adverse-weather restoration research covers deraining, dehazing, desnowing, raindrop removal, and all-in-one weather removal.
- Rain Streak Removal: Rain-streak methods progressed from bilateral-filter decomposition to CNNs, recurrent models, and physics-based adversarial approaches.
- Raindrop Removal: Raindrop-removal methods include temporal information, shallow CNNs, and attention GANs with dedicated datasets.
- Snow Removal: Snow-removal research developed from CNNs and dense networks toward size- and transparency-aware, multi-scale, wavelet, and cross-attention methods.
- Fog Removal: Fog-removal methods use atmospheric luminosity, transmission maps, multiple input modalities, and density-aware architectures.
- All-in-One Weather Removal: All-in-one systems include task-specific encoders with shared decoders, unified Transformer encoder-decoders, data selection, and weather-general or weather-specific stages.
- Transformer-based Image Restoration: Unlike prior Transformer methods using fixed spatial ranges or channel-only attention, Histoformer applies dynamic-range spatial attention to similarly patterned degradation.
3 Method
Histoformer uses Histogram Transformer Blocks that combine dynamic-range histogram attention with dual-scale feed-forward processing. Its attention sorts features by intensity, reshapes them into complementary histogram organizations, and applies attention to aggregate dynamically located weather-related features.
- Overall Architecture: The network backbone uses Histogram Transformer Blocks in encoder and decoder stages, with skip connections and pixel-unshuffle/shuffle for feature-scale changes.Each block is built from DHSA and DGFF components.
- Dynamic-range Histogram Self-Attention: DHSA reorders spatial features with dynamic-range convolution before dual-path histogram attention, then restores their original spatial locations.The module combines dynamic-range convolution with global and local dynamic feature aggregation.
- Dynamic-range Histogram Self-Attention: Dynamic-range convolution sorts features horizontally and vertically so convolution operates across pixels organized by intensity rather than only neighboring pixels.A 1 × 1 convolution and depth-wise 3 × 3 convolution process the recombined features.
- Histogram Reshaping: After sorting Value features, Histoformer gathers corresponding Query-Key features and reshapes them into bins for two complementary attention paths.The method uses BHR for large-scale information and FHR for fine-grained information based on intensity proximity.
- Histogram Reshaping: BHR assigns B bins of HW/B elements, whereas FHR uses bins of frequency B and HW/B bins; their attention outputs are multiplied elementwise.The two reshaping schemes respectively emphasize large dynamically located groups and smaller intensity-neighboring groups.
- Dual-scale Feed-Forward Processing: DGFF enriches multi-range representations within each Histogram Transformer Block alongside layer normalization and residual feature fusion.The architecture also describes dual-scale processing for visual feature modeling.
4 Experiments
Histoformer is evaluated across synthetic and real adverse-weather datasets, against task-specific and unified baselines, with qualitative comparisons and component ablations. The experiments report strong restoration performance and improvements from dynamic-range convolution, dual-branch histogram attention, DGFF, and correlation loss.
- Experimental settings: The evaluation uses Snow100K, Raindrop, Outdoor-Rain, Test1, RainDrop, and Snow100K test sets, including a 1,329-image real-world subset.Training includes 9,000 Snow100K images, 1,069 Raindrop images, and 9,000 Outdoor-Rain images.
- Experimental settings: Histoformer is compared with task-specific restoration methods and unified models trained for snow, rain-and-fog, raindrop, and all-in-one weather removal.The unified comparison includes All-in-One, Chen et al., TransWeather, WGWS-Net, WeatherDiff, and AWRCP.
- Qualitative evaluation: Qualitative results show comprehensive snow removal, while WeatherDiff retains some snow degradation and restores details less effectively; Histoformer also removes complex haze and rain streaks.The comparisons cover three tasks and a real-world weather-removal case.
- Ablation studies: Dynamic-range convolution improves performance by 0.14 dB over vanilla convolution, with no significant difference between horizontal-first and vertical-first sorting.The result comes from the dynamic-range convolution ablation on Outdoor-Rain.
- Ablation studies: Dual-branch histogram self-attention with dynamic-range convolution improves PSNR by 0.96 dB over TKSA, whereas single BHR or FHR branches neglect inter-bin or inner-bin relationships.MDTA and TKSA emphasize channel information and may lose long-range spatial information.
- Ablation studies: DGFF gains 0.3 dB PSNR over MSFN, while correlation loss consistently improves performance and its default weight is set to 1.Increasing C × B improves performance until values above 44 cause an out-of-memory error.
5 Conclusion
The paper introduces Histoformer for all-in-one adverse-weather removal, centered on histogram self-attention and supporting modules. Extensive experiments demonstrate its effectiveness and superiority.
- Histoformer addresses all-in-one weather removal through histogram self-attention, which segments spatial features into multiple bins.
- Histogram self-attention allocates attention along bin or frequency dimensions to selectively focus on weather-related features across dynamic ranges.
- DGFF and correlation loss support learning multi-range and multi-scale information, while experiments demonstrate the approach’s effectiveness and superiority.