Source-linked AI summary
DesnowNet: Context-Aware Deep Network for Snow Removal
Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, Jenq-Neng Hwang
TL;DR
Snow removal is difficult because snow varies in density, size, shape, trajectory, and transparency, while existing approaches have limited generalization. DesnowNet uses multistage, attribute-aware, multi-scale recovery and outperforms existing atmospheric-removal methods and a semantic-segmentation baseline on Snow100K, although realistic images can exhibit speckle-like artifacts.
Problem
Snow removal is challenging because snow has uneven density, varied sizes and shapes, irregular trajectories, and differing transparency, while hand-crafted approaches often generalize weakly.
Method
DesnowNet uses multistage translucency recovery and residual generation, separately models translucency and chromatic aberration, and applies context-aware multi-scale features and loss functions.
Results
DesnowNet significantly improves prediction accuracy on Snow100K and outperforms state-of-the-art atmospheric-particle removal approaches and a semantic-segmentation baseline.
Takeaways & Limitations
The multistage design recovers details obscured by opaque snow, while chromatic aberration modeling and multi-scale designs support accurate prediction across snow variations.
Takeaways & Limitations
On realistic images, confusing and blurry backgrounds can cause speckle-like artifacts when recovering obscured regions.
Abstract
from arXiv · showhide
Existing learning-based atmospheric particle-removal approaches such as those used for rainy and hazy images are designed with strong assumptions regarding spatial frequency, trajectory, and translucency. However, the removal of snow particles is more complicated because it possess the additional attributes of particle size and shape, and these attributes may vary within a single image. Currently, hand-crafted features are still the mainstream for snow removal, making significant generalization difficult to achieve. In response, we have designed a multistage network codenamed DesnowNet to in turn deal with the removal of translucent and opaque snow particles. We also differentiate snow into attributes of translucency and chromatic aberration for accurate estimation. Moreover, our approach individually estimates residual complements of the snow-free images to recover details obscured by opaque snow. Additionally, a multi-scale design is utilized throughout the entire network to model the diversity of snow. As demonstrated in experimental results, our approach outperforms state-of-the-art learning-based atmospheric phenomena removal methods and one semantic segmentation baseline on the proposed Snow100K dataset in both qualitative and quantitative comparisons. The results indicate our network would benefit applications involving computer vision and graphics.
1 INTRODUCTION
Atmospheric particles can obstruct computer-vision interpretation, while snow is especially difficult to remove because its density, size, shape, trajectory, and transparency vary. DesnowNet addresses translucent and opaque snow through multistage, attribute-aware, context-aware, and multi-scale processing.
- Motivation: Atmospheric particles can obstruct object-centric labeling and increase risks of false alarms and unstable machine interpretation.The introduction illustrates this with pedestrian and vehicle labeling supported by Google Vision API.
- Motivation: Snow removal is harder than rain or haze removal because snow has uneven density, diverse particle sizes and shapes, irregular trajectories, and varying transparency.These characteristics make existing learning-based atmospheric-particle methods difficult to adapt to snow.
- Motivation: Existing snow-removal approaches rely mainly on hand-crafted features, which often produce weak generalization.The paper contrasts this limitation with its learned network design.
- Approach: DesnowNet uses a multistage network to handle translucent and opaque snow particles and recover details obscured by opaque coverage.The design aims to acquire recovered image content for more accurate estimation and restoration.
- Approach: The network models snow using per-coordinate translucency, an independent chromatic aberration map, context-aware features and loss functions, and multi-scale processing.These design choices address color distortions and variation in snow-particle size and shape.
- Evaluation: Experiments on Snow100K report improved prediction accuracy over state-of-the-art learning-based atmospheric-phenomena removal methods and a semantic-segmentation baseline.The comparison includes both qualitative and quantitative results.
2 RELATED WORKS
Prior rain and haze removal methods use hand-crafted priors or learned mappings tailored to specific particle properties. Snow violates these assumptions through greater variation in particle shape and size, limiting direct transfer of existing approaches.
- Rain Removal: Early rain-removal methods model similar gradient orientations, high spatial frequency, sparse components, or recurring particle patterns.These approaches include image decomposition, sparse coding, HOG, depth-of-field, Eigen-color, and Gaussian-mixture modeling.
- Rain Removal: DerainNet separates rainy images into high-frequency detail and low-frequency base layers, but its spatial-frequency assumption fails for coarse-grained snow particles.The cited work reports improved rain-removal accuracy and clarity over earlier hand-crafted methods, while snow remains a failure case.
- Haze Removal: Haze-removal methods use contrast maximization, albedo and transmission estimation, transmission refinement, and dark-channel priors.These methods rely on haze-specific physical or heuristic cues.
- Haze Removal: DehazeNet learns a mapping from hazy images to medium transmission maps using an atmospheric-scattering model with an equal-brightness haze mask.Its haze-mask brightness is determined by the global image maximum, and the architecture assumes translucent haze without opaque corruption.
- Snow Removal: Snow is more complex than rain or haze because particle shape and size vary, yet existing snow methods inherit rainfall-driven HOG, frequency-separation, and color assumptions.The passage identifies these inherited features as only partial models of snow particles.
3 PROPOSED METHOD
DesnowNet uses multi-scale, context-aware features and separate recovery stages to estimate translucency, chromatic aberration, and missing content caused by snow. Its training combines snow-free image, final output, and snow-mask losses, while Snow100K synthesizes varied snow masks for evaluation.
- Network formulation: DesnowNet models a snowy image using a snow-free image, snow mask, and chromatic aberration map.The snow mask represents translucency, while the chromatic aberration map captures color aberration at each coordinate.
- Descriptor: Multi-scale receptive fields and a dilation pyramid extract context-aware features across the varied sizes and shapes of snow particles.The dilation pyramid concatenates features from dilated convolutions rather than directly summing them, preserving spatial information.
- Recovery submodule: The translucency recovery module estimates a snow mask and chromatic aberration map, then recovers details behind translucent snow.Pyramid maxout selects robust feature maps from different receptive fields for snow-mask and aberration estimation.
- Recovery submodule: The residual generation module uses estimated snow locations, chromatic aberration, and recovered output to generate residual complements for opaque snow regions.Its pyramid-sum design models snow variation across scales, while the residual distribution is intended to remain zero-mean for compensation.
- Loss function: The overall loss jointly weights the intermediate snow-free estimate, final snow-free estimate, snow-mask loss, and l2-norm regularization.The two snow-free estimates receive equal importance, while the snow-mask and regularization terms use separate weights.
4 DATASET
The Snow100K dataset combines synthesized snowy images, snow-free ground truths, snow masks, and realistic snowy images, with varied particle simulations for training and evaluation.
- Dataset construction: Snow100K contains 100k synthesized snowy images, corresponding snow-free ground truths, snow masks, and 1,329 realistic snowy images.Images were sourced through the Flickr API, manually categorized, and resized with aspect ratios preserved.
- Dataset construction: The synthesis pipeline creates 5.8k base masks spanning small, medium, and large particles with varied densities, shapes, trajectories, and transparencies.These masks simulate diversity in falling snow particle sizes and appearances.
- Dataset construction: Snow100K-S, Snow100K-M, and Snow100K-L represent increasing snowfall complexity by superposing one, two, or three category-specific base masks.The subsets use progressively more combinations of small, medium, and large masks.
- Dataset construction: Additional randomness comes from snow brightness sampled within [max(y_i) × 0.7, max(y_i)] and random cropping of oversized base masks.These factors are applied during the superposition process.
- Dataset validation: A survey evaluates whether synthesized snowy images are semantically reasonable by comparing them with realistic snowy images.The experiment collected 500 synthesized images for the survey.
5 EXPERIMENTS
Experiments evaluate DesnowNet’s components, losses, activations, snow-mask estimation, and comparisons against established methods on Snow100K. Results favor the proposed multi-scale and recovery designs, while realistic images still expose speckle-like artifacts.
- Ablation studies: The dilation-pyramid descriptor achieves the highest PSNR and SSIM among evaluated descriptor backbones.Its concatenation operation aggregates multi-scale dilated-convolution feature maps.
- Ablation studies: Pyramid maxout yields the highest PSNR and SSIM because it robustly models variation in snow-particle size and shape.Pyramid sum performs similarly to a basic conv3×3 layer and lacks sufficient sensitivity for this task.
- Ablation studies: Removing the TR module substantially harms performance, while removing AE reduces PSNR by 1.09 dB.The TR module uses estimated snow-mask and chromatic-aberration information during recovery.
- Ablation studies: The pyramid l2-norm outperforms a single-scale l2-norm, and PReLU achieves significantly better estimation accuracy than the alternative activations.These comparisons support multi-scale loss design and the selected activation for SE and AE outputs.
- Comparison: DesnowNet outperforms DerainNet, DehazeNet, and DeepLab quantitatively on Snow100K and produces the fewest artifacts in qualitative comparisons.DerainNet removes almost nothing, while DehazeNet and DeepLab introduce visible artifacts; DesnowNet achieves the most visually appealing outputs.
- Comparison: DesnowNet obtains superior PSNR and SSIM similarity scores for estimated snow masks compared with the other evaluated methods.The reported advantage is attributed to modeling variation in spatial frequency, trajectory, translucency, particle size, and shape.
- Failure cases: On realistic snowy images, DesnowNet sometimes generates speckle-like artifacts when blurry or confusing backgrounds hinder recovery of obscured regions.The authors state that these artifacts usually do not detract from distinguishing obscured image details.
6 CONCLUSIONS
The paper presents a learning-based single-image snow-removal method whose multistage design recovers details obscured by opaque snow. It also identifies chromatic aberration mapping and multi-scale design as important components, while acknowledging possible speckle-like artifacts.
- DesnowNet is presented as the first learning-based method for removing snow particles from single images.
- Its translucency recovery and residual generation modules recover details obscured by opaque snow particles and compensate for potential artifacts.
- The chromatic aberration map is identified as a key milestone for accurate prediction by modeling subtle snow-related color inconsistencies across three channels.
- Multi-scale designs provide interpretability for accounting for variations in snow particles.
- In certain situations, DesnowNet may introduce speckle-like artifacts, leaving improved visual quality as a possible direction for future research.