Source-linked AI summary
Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
Guanbin Li, Xiang He, Wei Zhang, Huiyou Chang, Le Dong, Liang Lin
TL;DR
Single-image de-raining remains challenging because prior approaches rely on local or handcrafted representations, while long streaks and dense patterns require broader spatial context. The paper proposes NLEDN, an encoder-decoder combining non-local weighting, densely connected blocks, pooling striding, and long-range skip-connections. Experiments on synthetic and real datasets report effective removal across rain densities with preserved image details and improvements over state-of-the-art methods.
Problem
Existing de-raining methods struggle with long rain streaks and dense streaks resembling rain-free patterns because CNN convolutions have limited local receptive fields.
Method
NLEDN uses an encoder-decoder framework with non-locally enhanced dense blocks, pooling striding, and long-range skip-connections for feature and spatial-context modeling.
Results
Experiments on synthetic and real datasets show effective removal of rain streaks across densities while preserving image texture and outperforming state-of-the-art methods.
Takeaways & Limitations
The proposed architecture provides a single-image de-raining method that models long-distance dependencies while preserving image details.
Takeaways & Limitations
The entrance and exit layer architecture is not unique and was selected to incorporate the core encoding and decoding modules.
Abstract
from arXiv · showhide
Single image rain streaks removal has recently witnessed substantial progress due to the development of deep convolutional neural networks. However, existing deep learning based methods either focus on the entrance and exit of the network by decomposing the input image into high and low frequency information and employing residual learning to reduce the mapping range, or focus on the introduction of cascaded learning scheme to decompose the task of rain streaks removal into multi-stages. These methods treat the convolutional neural network as an encapsulated end-to-end mapping module without deepening into the rationality and superiority of neural network design. In this paper, we delve into an effective end-to-end neural network structure for stronger feature expression and spatial correlation learning. Specifically, we propose a non-locally enhanced encoder-decoder network framework, which consists of a pooling indices embedded encoder-decoder network to efficiently learn increasingly abstract feature representation for more accurate rain streaks modeling while perfectly preserving the image detail. The proposed encoder-decoder framework is composed of a series of non-locally enhanced dense blocks that are designed to not only fully exploit hierarchical features from all the convolutional layers but also well capture the long-distance dependencies and structural information. Extensive experiments on synthetic and real datasets demonstrate that the proposed method can effectively remove rain-streaks on rainy image of various densities while well preserving the image details, which achieves significant improvements over the recent state-of-the-art methods.
1 INTRODUCTION
Single-image de-raining is a practical but ill-posed computer-vision problem. The paper proposes an end-to-end non-locally enhanced encoder-decoder with dense blocks and pooling-striding to improve rain modeling while preserving details.
- Motivation: Rain streaks degrade outdoor-surveillance vision systems and multimedia applications, motivating automatic single-image removal.The task has applications including driverless technology and content-based image editing.
- Problem: Single-image de-raining is more ill-posed than video de-raining because it lacks temporal information.Earlier video methods commonly exploit luminance-domain temporal correlations between successive frames.
- Existing Approaches: Deep methods learn pixel-wise mappings from rainy inputs to clean images or negative residual maps through convolution, pooling, and nonlinear operations.These approaches have produced significant performance improvements over traditional methods.
- Proposed Method: NLEDN combines an encoder and decoder built from non-locally enhanced dense blocks with pooling striding for increasingly abstract feature representation.The encoder and decoder each contain three cascaded blocks, while each block uses non-local weighting, dense convolutional encoding, and residual inference.
- Results: Experiments on synthetic and real datasets report significant improvements over state-of-the-art methods while removing rain streaks and preserving texture details.The paper identifies these outcomes as a primary contribution.
2 RELATED WORKS
Related work frames single-image de-raining as an ill-posed separation or filtering problem and notes that local convolutions struggle with long streaks and complex backgrounds. The paper motivates non-local operations to model long-range dependencies and spatial context.
- Traditional Methods: Traditional methods use handcrafted priors, decomposition, sparse coding, dictionary learning, or filtering, but often oversmooth rain-free scene details.These approaches model rain streaks and clean scenes in separate subspaces or rely on nonlocal mean smoothing.
- Proposed Architecture: Figure 2 presents NLEDN as an encoder-decoder architecture using NEDBs and pooling-indices guidance, with long-range skip-connections from the input and low-level activations to the output.The figure caption identifies these as core architectural components.
- Deep Learning Methods: Recent CNN methods improve de-raining, but remain weak at removing long streaks in complex backgrounds and distinguishing dense streaks from similar clean-image patterns.The cited explanation is that convolutional operations have small local spatial receptive fields.
- Non-local Networks: Non-local operations compute each position’s response as a weighted sum of features across spatial positions, enlarging the receptive field beyond neighboring locations.The paper relates this design to traditional non-local mean filtering used in earlier single-image de-raining.
3 METHOD
NLEDN combines a fully convolutional encoder-decoder with long-range skip connections and non-locally enhanced dense blocks. Pooling indices guide decoding, while dense and residual connections support hierarchical feature learning, long-distance dependencies, detail preservation, and stable optimization.
- 3.1 Entrance and Exit Layers: NLEDN takes a rainy image as input and produces its rain-free version through a fully convolutional encoder-decoder.The entrance uses shallow feature extraction, and the exit predicts a rain map for reconstructing the rain-free image.
- 3.1 Entrance and Exit Layers: Long-range skip connections link the input and shallow features to near-exit layers, preserving low-level information and enabling residual learning.These connections also facilitate gradient back-propagation and pixel-wise prediction.
- 3.2 Non-locally Enhanced Encoding and Decoding: Each encoder and decoder stage contains a non-locally enhanced dense block that combines non-local feature weighting, dense hierarchical encoding, and local residual learning.Dense layers receive activations from all preceding layers, while residual learning addresses gradient vanishing or exploding.
- 3.2 Non-locally Enhanced Encoding and Decoding: Non-local operation enhances each position using feature information from all positions, capturing spatial dependencies beyond local neighborhoods.The operation computes pairwise relationships between feature positions and aggregates representations using a normalization factor.
- 3.2 Non-locally Enhanced Encoding and Decoding: Three encoder NEDBs use strided max-pooling, while three decoder NEDBs use max-unpooling and corresponding pooling indices to restore spatial resolution.Skip connections link encoder activations with their counterpart decoder layers, and multi-scale non-local processing accommodates different resolutions.
- 3.3 Loss Function: Training minimizes the mean absolute error between the predicted rain-free image and its ground truth.The loss averages absolute pixel differences over the image height, width, and channels.
4 EXPERIMENTAL RESULTS
NLEDN is evaluated on synthetic and real rainy images using quantitative metrics for synthetic data and visual comparison for real data. It consistently outperforms existing methods while preserving structural details, and ablations support the contributions of dense connections, pooling-index receptive-field control, and non-local weighting.
- Evaluation Setup: NLEDN is evaluated on four synthetic benchmark datasets and on real rainy images collected from online sources, released images, and the authors’ cameras.The synthetic evaluation uses DDN-Data, DIDMDN-Data, Rain100L, and Rain100H.
- Evaluation Setup: Synthetic evaluation uses PSNR and SSIM on the luminance channel, whereas real-world results are assessed through visual comparison because rain-free ground truth is unavailable.The luminance channel is selected for its stated impact on human visual perception.
- Comparison with the State-of-the-Art: 1.07db average PSNR improvement is reported for NLEDN over existing best-performing algorithms across the four synthesized datasets, with SSIM gains on each dataset.The reported PSNR improvements are 2.81db, 1.34db, 4.03db, and 1.07db respectively as presented across the cited results passage; SSIM improvements are 2.70%, 4.90%, 0.73%, and 7.87%.
- Comparison with the State-of-the-Art: NLEDN produces the best reported visual results on synthesized and real rainy images, removing streaks while preserving image structure details.On difficult synthetic cases, competing outputs include failed streak removal, over-smoothing, or artifacts; NLEDN is described as cleaner with preserved structure.
- Comparison with the State-of-the-Art: NLEDN outperforms methods using additional rain masks, annotations, or density information without relying on that additional data.JORDER and DID-MDN use extra information during training on specified datasets, whereas the paper reports NLEDN still greatly outperforms them.
- Ablation Study: Dense connections, pooling-index receptive-field control, and non-local weighting each improve ablation performance, with non-local weighting adding 0.77db PSNR and 0.32% SSIM on average.Multiple dense blocks also improve representation, while more than six blocks causes performance deterioration in the reported validation experiments.
5 CONCLUSION
The paper introduces NLEDN, an encoder-decoder framework built from non-locally enhanced dense blocks for single-image rain-streak removal. Experiments on synthetic and real datasets show removal across varied rain densities while preserving image texture.
- NLEDN combines encoder and decoder networks composed of tailored non-locally enhanced dense blocks.These blocks exploit hierarchical features and capture long-distance dependencies and structural information through non-local weighting.
- Experiments on synthetic and real datasets show effective rain-streak removal across images with various rain densities.
- The method promisingly preserves image texture similar to the rain streaks while removing rain artifacts.