Source-linked AI summary
Deep Dense Multi-scale Network for Snow Removal Using Semantic and Geometric Priors
Kaihao Zhang, Rongqing Li, Yanjiang Yu, Wenhan Luo, Changsheng Li, Hongdong Li
TL;DR
Snow degrades visibility and downstream vision performance, while existing desnowing methods often overlook semantic and geometric information and struggle with occluded details. The paper proposes DDMSNet, a coarse-to-fine framework that uses semantic and depth maps as priors for attention-guided restoration. Experiments on three datasets report state-of-the-art snow-removal performance and improvements in semantic segmentation and depth estimation.
Problem
Snow reduces scene visibility and downstream vision performance, while existing desnowing methods overlook semantic and geometric information and struggle with snow-occluded details.
Method
DDMSNet combines coarse snow removal, semantic segmentation, depth estimation, and dense multi-scale attention-guided restoration using semantic and geometric priors.
Results
The method achieves state-of-the-art snow-removal performance on three datasets, and desnowed images improve semantic segmentation and depth estimation.
Takeaways & Limitations
Semantic and geometric priors provide the basis for restoring clean snowy images while supporting downstream scene-understanding applications.
Takeaways & Limitations
Existing desnowing methods, including the paper’s motivating baselines, can fail to restore details in heavily snow-occluded regions.
Abstract
from arXiv · showhide
Images captured in snowy days suffer from noticeable degradation of scene visibility, which degenerates the performance of current vision-based intelligent systems. Removing snow from images thus is an important topic in computer vision. In this paper, we propose a Deep Dense Multi-Scale Network (\textbf{DDMSNet}) for snow removal by exploiting semantic and geometric priors. As images captured in outdoor often share similar scenes and their visibility varies with depth from camera, such semantic and geometric information provides a strong prior for snowy image restoration. We incorporate the semantic and geometric maps as input and learn the semantic-aware and geometry-aware representation to remove snow. In particular, we first create a coarse network to remove snow from the input images. Then, the coarsely desnowed images are fed into another network to obtain the semantic and geometric labels. Finally, we design a DDMSNet to learn semantic-aware and geometry-aware representation via a self-attention mechanism to produce the final clean images. Experiments evaluated on public synthetic and real-world snowy images verify the superiority of the proposed method, offering better results both quantitatively and qualitatively.
1 INTRODUCTION
Snow reduces scene visibility and harms downstream vision tasks, while existing desnowing methods overlook semantic and geometric information and struggle with snow-occluded details. DDMSNet addresses these issues using map-guided semantic and geometric representations, dense multi-scale processing, and attention-guided restoration.
- Snow reduces image visibility and degrades subsequent tasks including object detection, tracking, and scene analysis.
- Opaque snow obscures regions, making desnowing harder than deraining and requiring models to distinguish airborne snow from snow on surfaces.
- Existing methods generally ignore semantic and geometric information and fail to restore details in heavily occluded regions.
- DDMSNet combines coarse snow removal, semantic segmentation, depth estimation, and dense multi-scale restoration with attention-guided semantic- and geometry-aware representations.
- The method uses semantic and geometric features obtained at different stages as priors to remove snow and recover clean images.
- Experiments on three datasets report state-of-the-art snow-removal performance, while desnowed images improve semantic segmentation and depth estimation.
2 RELATED WORK
Prior snow-removal research includes handcrafted image priors and deep networks, while related deraining and dehazing methods model weather effects using learned, physical, or auxiliary information. The paper positions DDMSNet against these approaches by emphasizing multi-scale feature processing and semantic and geometric priors.
- 2.1 Snow Removal: Traditional snow-removal methods detect snow using foreground-background separation, color and shape features, frequency separation, or color assumptions.
- 2.1 Snow Removal: Deep snow-removal methods include DesnowNet, GAN-based restoration, pixel-level multi-scale networks, and NAS, but recent methods may ignore feature-space, semantic, or geometric information.
- 2.2 Image Deraining: Snow is opaque, so snow removal must restore occluded details more difficultly than image deraining, whose methods model rain characteristics using handcrafted or learned approaches.
- 2.3 Image Dehazing: Image dehazing methods use atmospheric cues and depth information, with traditional priors estimating transmission and atmospheric light.
- 2.3 Image Dehazing: Deep dehazing research includes coarse-to-fine multi-scale, pyramid, and AOD-Net approaches for estimating transmission and atmospheric light.
3 OUR METHOD
The framework uses a coarse-to-fine snow-removal pipeline that combines multi-scale RGB and latent features with semantic and geometric guidance. Attention-based representations and multi-scale losses produce refined snow-free images.
- Overall framework: DDMSNet receives coarsely desnowed images, semantic labels, and depth maps to learn semantic-aware and geometry-aware representations for finer restoration.The map-guided design uses semantic and geometric cues in the final snow-removal stage.
- Overall framework: The pipeline first coarsely desnows snowy inputs, then predicts semantic segmentation and depth maps from the intermediate results.The coarse stage reduces snow before semantic understanding and depth estimation, while the predicted maps provide cues for later restoration.
- Deep dense multi-scale network: DDMSNet extracts multi-scale features from both multi-scale RGB images and single-scale RGB images through scale-specific subnetworks.Each subnetwork contains semantic-guided attention, a transfer module, and depth-guided attention; the transfer module extracts three-scale features.
- Geometry-aware representation: The geometry-aware module combines transfer features with depth labels and uses convolutional layers and residual dense blocks to recover three-channel RGB images.The geometry information is concatenated in the last DDMSNet layer, and the geometry-aware representation is processed by two CNN layers for image recovery.
- Semantic-aware representation: Semantic attention converts object-type responses into attention weights, divides 30-channel features into 30 groups, and merges them with group and 1 × 1 convolutions.The semantic labels contain about 30 object types, and the weights are applied element-wise to feature channels.
- Loss function: The coarse and fine networks are trained with smooth L1 and perceptual losses, with the fine loss extending the formulation across three adopted scales.The perceptual loss uses feature maps from pretrained VGG16, while the smooth L1 formulation applies to both snow-removal networks.
4 EXPERIMENTS
Experiments evaluate DDMSNet on three synthetic or real-world snowy-image datasets, including two newly created street-scene datasets. Ablations and comparisons show that dense multi-scale features, coarse-to-fine processing, semantic and geometric representations, and their combination improve snow removal results.
- Datasets: Experiments use SnowKITTI2012, SnowCityScapes, and Snow100K, with the first two created to address the lack of snowy street scenes in existing datasets.SnowKITTI2012 contains 1,500 training and 1,000 testing image pairs; SnowCityScapes contains 2,000 training and 2,000 testing pairs.
- Ablation Study: The ablation compares SnowCNN, MSNet, DDMSNet, coarse-to-fine DDMSNet(+), semantic and geometry variants, and the combined DDMSNet(S+G).These variants isolate dense multi-scale features, coarse-to-fine processing, and semantic-aware or geometry-aware representations.
- Ablation Study: DDMSNet variants outperform the plain SnowCNN and MSNet, while DDMSNet(+) improves over DDMSNet, supporting the coarse-to-fine strategy.The comparison is reported on SnowKITTI2012 using PSNR and SSIM.
- Ablation Study: Adding semantic-aware or geometry-aware representations improves DDMSNet(+), and DDMSNet(S+G) achieves the best ablation performance.The reported comparison attributes the gains to semantic-aware and geometry-aware attentions and their combination.
5 CONCLUSION
The paper proposes DDMSNet for single-image snow removal using semantic and geometric information as global priors. It also introduces two synthetic snowy street-scene datasets and reports state-of-the-art performance against previous methods.
- Conclusion: DDMSNet is a deep dense multi-scale network that uses semantic and geometric information as global priors for snow removal and clean-image restoration.The method is presented as a new multi-scale network for snow removal.
- Conclusion: Two large-scale snowy datasets are synthesized from the public KITTI and Cityscapes datasets for snow-removal research.The datasets target snowy street-scene imagery.
- Conclusion: Experimental results report that the proposed method performs better than previous methods and achieves state-of-the-art performance.The conclusion summarizes the reported experimental outcome without specifying a numerical metric.