PaperlayerView exact revision on arXiv

Source-linked AI summary

FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors

Tai-Xiang Jiang, Ting-Zhu Huang, Xi-Le Zhao, Liang-Jian Deng, Yao Wang

arXiv:1803.07487v3cs.CV

TL;DR

FastDeRain targets the ill-posed separation of clean video and rain streaks in outdoor vision data. It models directional and temporal gradient priors with sparsity and total-variation regularization, then solves the model efficiently with SALSA. Experiments on synthetic and real data report effective, quantitatively competitive, and markedly faster rain removal than compared methods.

  • Problem

    Video rain removal must distinguish clean content and rain streaks from an observed video, an ill-posed task important because rain degrades multiple vision-system functions.

  • Method

    FastDeRain combines sparsity, directional unidirectional-TV priors, temporal continuity, and a SALSA-based solver for video rain-streak removal.

  • Results

    FastDeRain outperforms compared state-of-the-art methods visually and quantitatively on synthetic and real data and is markedly faster.

  • Takeaways & Limitations

    The reported results support using directional gradient priors and efficient optimization for effective and fast video rain-streak removal.

Abstract

from arXiv · show

Rain streak removal is an important issue in outdoor vision systems and has recently been investigated extensively. In this paper, we propose a novel video rain streak removal approach FastDeRain, which fully considers the discriminative characteristics of rain streaks and the clean video in the gradient domain. Specifically, on the one hand, rain streaks are sparse and smooth along the direction of the raindrops, whereas on the other hand, clean videos exhibit piecewise smoothness along the rain-perpendicular direction and continuity along the temporal direction. Theses smoothness and continuity results in the sparse distribution in the different directional gradient domain, respectively. Thus, we minimize 1) the $\ell_1$ norm to enhance the sparsity of the underlying rain streaks, 2) two $\ell_1$ norm of unidirectional Total Variation (TV) regularizers to guarantee the anisotropic spatial smoothness, and 3) an $\ell_1$ norm of the time-directional difference operator to characterize the temporal continuity. A split augmented Lagrangian shrinkage algorithm (SALSA) based algorithm is designed to solve the proposed minimization model. Experiments conducted on synthetic and real data demonstrate the effectiveness and efficiency of the proposed method. According to comprehensive quantitative performance measures, our approach outperforms other state-of-the-art methods especially on account of the running time. The code of FastDeRain can be downloaded at https://github.com/TaiXiangJiang/FastDeRain.

I. INTRODUCTION

FastDeRain addresses video rain-streak removal as an ill-posed decomposition problem by exploiting directional and temporal gradient priors for rain streaks and clean video. It combines a sparsity-based model with efficient optimization and evaluates the method on synthetic and real rainy videos.

  • Rain degrades outdoor vision tasks including detection, tracking, recognition, and scene analysis, motivating video rain-streak removal.
  • The task decomposes an observed rainy video into clean background, rain streaks, and noise, requiring prior information to make the inverse problem tractable.
  • Clean videos have sparse temporal gradients because they are continuous over time, whereas rain streaks do not share this property.
  • The model uses directional smoothness priors: vertical rain streaks and horizontally piecewise-smooth clean content motivate unidirectional TV regularization.
  • FastDeRain uses a SALSA-based solver, GPU acceleration, automatic direction detection, and experiments on synthetic and real-world rainy videos.

II. NOTATION AND PRELIMINARIES

This section establishes tensor notation and basic operations used to formulate the video rain-removal model. It defines tensor dimensions, fibers, slices, inner products, and the Frobenius norm.

  • An N-mode tensor is represented as X ∈ R^(I1×I2×···×IN), with x_i1,i2,···,iN denoting an individual component.
  • Tensor fibers fix every index except one, while third-order tensors contain column, row, and tube fibers oriented as column vectors.
  • A slice fixes all but two indices; third-order tensors have horizontal, lateral, and frontal slices, with frontal slice X::k also written Xk.
  • The section defines the inner product of same-sized tensors and introduces the corresponding Frobenius norm.

III. MAIN RESULTS

FastDeRain formulates rain removal as decomposition of a rainy tensor into clean video, rain streaks, and noise. The method uses prior information about these components to regularize the ill-posed inverse problem.

  • Problem formulation: A rainy video O is modeled as the sum of clean video B, rain streaks R, and noise or error N.
  • Problem formulation: The decomposition assigns distinct latent roles to the observed video, rain-free content, rain streaks, and noise term.
  • Problem formulation: The goal is to recover B and R from O, which is ill-posed and therefore requires analyzed priors and corresponding regularizers.

B. Priors and regularizers

The method uses directional priors to distinguish rain streaks from clean video: rain streaks are sparse and vertically structured, while clean video is smooth horizontally and continuous temporally.

  • The model assumes rain streaks are approximately vertical when establishing directional regularizers.
  • The ℓ1 norm of R promotes sparse rain streaks and can restrain their magnitude under heavier, non-extreme rain.The sparsity term’s effect can be reduced by tuning its parameter.
  • Horizontal unidirectional TV regularization models the clean video’s piecewise-smooth intensity variation perpendicular to approximately vertical rain streaks.
  • Minimizing the ℓ1 norm of the vertical gradient of R helps distinguish rain streaks from the clean video.
  • Temporal ℓ1 regularization models the clean video’s continuity, whose temporal gradients contain more zero and small non-zero values than rainy-video and rain-streak gradients.

C. The proposed model

The proposed model combines directional sparsity and smoothness priors for rain streaks and clean video, while accounting for small deviations between the assumed and actual rain directions.

  • The model treats the rain direction as vertical and the rain-perpendicular direction as horizontal when the directional angle is small.The actual falling direction may differ from the vertical direction by a small angle.
  • The formulation uses vertical rain-streak gradients, rain-streak sparsity, horizontal clean-video gradients, and temporal clean-video gradients.∇1, ∇2, and ∇t denote the vertical, horizontal, and temporal differential operators, respectively.

D. Optimization

FastDeRain solves a convex constrained formulation with SALSA-based alternating updates for auxiliary variables, the clean video, the rain streaks, and multipliers.

  • SALSA, adopted within ADMM, solves the proposed concise convex model for large-scale video rain-streak removal.
  • Four auxiliary tensors convert the model into an equivalent constrained problem linking rain streaks and clean-video gradients to auxiliary variables.
  • The constrained formulation enforces data fidelity, auxiliary-variable consistency, and nonnegative bounds on the clean video and rain streaks.
  • The auxiliary-variable subproblems have closed-form soft-thresholding solutions.
  • The algorithm alternates updates of auxiliary tensors, B and R, and Lagrange multipliers before returning estimates of the rain-free video and rain.
  • O(mnt log(mnt)) is the overall time complexity for a video of dimensions m × n × t.Each auxiliary subproblem costs O(mnt), while FFT-based B or R updates cost O(mnt · log(mnt)).

E. Discussion of the oblique rain streaks

Because real rain streaks may be oblique, FastDeRain detects their angle and shifts video rows to make the streaks approximately vertical before applying the algorithm.

  • The vertical directional prior becomes a double-edged sword because real rainfall streaks are not always vertical.
  • The direction detector median-filters horizontal slices, forms a residual, rotates it across candidate angles, and selects the angle minimizing the vertical-gradient norm.
  • For simulated 45° streaks, the detector recovered 45°; angle ranges can also be normalized by flipping or transposing frames.
  • Figure 3 plots the magnitude of y_i against candidate angle θ_i, while Figure 4 illustrates the two shift operations.
  • For detected angles from 15° to 45°, the shift strategy uses two row-sliding operations to make streaks approximately vertical without interpolation degradation.

IV. EXPERIMENTAL RESULTS

Experiments evaluate FastDeRain on synthetic and real rainy videos using quantitative, visual, and efficiency comparisons against recent methods.

  • Evaluation setup: Experiments assess FastDeRain on synthetic data and real-world rainy videos.The evaluation includes implementation details, compared methods, and performance measurements.
  • Implementation details: Color videos are converted to YUV, and the method is applied only to the Y channel.The Y channel represents luma, while U and V represent chrominance.
  • Implementation details: FastDeRain is implemented on Windows 10 and Matlab with CPU and GPU computing support.The reported setup uses an Intel i5-4590 CPU, 16 GB RAM, and a GTX1080 GPU.
  • Compared methods: The comparison includes DDN, TCL, SE, and MS-CSC as recent state-of-the-art single-image and video-based methods.DDN is single-image based; TCL, SE, and MS-CSC are video-based methods.

A. Synthetic data

Synthetic experiments compare FastDeRain across multiple rain-generation settings and show strong quality, detail-preservation, robustness, and efficiency results.

  • Synthetic data generation: Synthetic rain is generated in three cases spanning sparse same-angle streaks, denser multi-angle streaks, and streaks from photographed data.The synthetic backgrounds include dynamic-camera videos and a static-camera video.
  • Quantitative comparisons: FastDeRain considerably outperformed four state-of-the-art methods across all selected quality assessment indexes in Table II.The assessment reports PSNR, mean SSIM, FSIM, VIF, UIQI, GMSD, and running time.
  • Efficiency: FastDeRain took less than 10 seconds on all synthetic data and has approximate time complexity O(mntlog(mnt)).The method uses closed-form solutions to its sub-problems.
  • Visual comparisons: Across synthetic cases, FastDeRain removed almost all rain streaks while preserving clean-video details.Other methods sometimes extracted background details as rain streaks or produced over- or under-derained results.
  • Component analysis: All four model components contribute to rain-streak removal, with temporal continuity described as overwhelmingly important.Removing the temporal regularization term nearly caused the method to fail.
  • Noise robustness: Including the noise term improves robustness when videos contain both rain streaks and noise.Without noise, the model without N performs better; with rain and noise, the model with N performs better.
  • Comparison with prior version: FastDeRain made substantial progress over the previous conference version on the reported synthetic data.The comparison uses both quantitative results and visual results.

B. Real data

On four real rainy videos with varied scenes, FastDeRain removes streaks, preserves spatial details, and runs faster than the compared methods.

  • Real-data setup: The real-data evaluation uses four videos with different scene complexity and backgrounds, including wall, yard, the Matrix, and crossing.The wall video has regular background patterns, while yard has a more complex background.
  • Visual comparisons: On wall, FastDeRain removes almost all rain streaks and preserves the background better than the other compared methods.Adjacent frames help distinguish bright rain streaks from vertical background patterns.
  • Visual comparisons: On yard, FastDeRain removes most rain streaks while preserving the background, whereas other methods miss streaks or remove fine structures.The comparison uses a single displayed frame because the scene contains little texture resembling rain streaks.
  • Overall real-data results: Across the four videos, FastDeRain achieves the best combined streak removal and spatial-detail retention, with substantially lower running time than other video-based methods.The scenes differ substantially in content and structure.
  • Visual comparisons: FastDeRain obtains the best result on the Matrix, particularly for the obvious rain streak on Neo’s face.The comparison uses two adjacent frames showing rapidly changing luminance.
  • Visual comparisons: On crossing, FastDeRain produces the visually best extracted rain streaks while other methods remove or extract structural details incorrectly.The compared errors include curb-line extraction and removal of line-pattern textures.

C. Oblique rain streaks

For oblique rain streaks, FastDeRain uses a shift strategy to make streaks approximately vertical before applying its algorithm.

  • Evaluation: The shift strategy is evaluated on streak angles of [15°, 35°] and [35°, 55°] in dynamic- and static-camera videos.The experiments use waterfall for the dynamic-camera case and highway for the static-camera case.

V. CONCLUSION

FastDeRain uses directional gradient priors and sparsity to outperform state-of-the-art video rain-streak removal methods visually and quantitatively, with notably faster running time. The method remains limited by residual rain artifacts in natural scenes mixed with haze.

  • V. CONCLUSION: FastDeRain combines directional gradient priors with sparsity for video rain-streak removal.The authors attribute its performance to analyzing characteristic priors of rainy videos, clean videos, and rain streaks.
  • V. CONCLUSION: FastDeRain outperforms the compared state-of-the-art methods visually and quantitatively.The quantitative comparison includes methods [34], [26], [39], and [40] under a shift strategy when rain streaks are far from vertical.
  • V. CONCLUSION: FastDeRain is markedly faster than the compared methods, including a fast single-image-based method.Figure 17 compares its shift-strategy results with TCL, DDN, SE, MS-CSC, and ground truth on two highway videos.
  • V. CONCLUSION: Residual rain artifacts in natural scenes mixed with haze remain an open problem for FastDeRain.The authors identify handling these residual artifacts as future work.
Loading 1803.07487v3…