Source-linked AI summary
Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework
Kaihao Zhang, Dongxu Li, Wenhan Luo, Wenqi Ren, Wei Liu
TL;DR
Video deraining must recover clean videos while modeling spatial and temporal information efficiently, because rain degrades visibility and outdoor-system robustness. ESTINet combines residual spatial extraction, convolutional recurrent temporal modeling, and enhanced spatio-temporal refinement. Experiments on three public datasets report better performance and higher speed than state-of-the-art methods.
Problem
Video deraining needs to exploit continuous-frame information for powerful spatio-temporal features while restoring high-quality videos at high speed.
Method
ESTINet combines spatial feature extraction, an Interaction-CBLSTM for temporal correlations, and enhanced spatio-temporal refinement in an end-to-end framework.
Results
Experiments on three public datasets show state-of-the-art video-deraining performance while outperforming competing methods in speed.
Takeaways & Limitations
ESTINet provides a video-deraining framework that improves both deraining quality and processing speed relative to state-of-the-art methods.
Abstract
from arXiv · showhide
Video deraining is an important task in computer vision as the unwanted rain hampers the visibility of videos and deteriorates the robustness of most outdoor vision systems. Despite the significant success which has been achieved for video deraining recently, two major challenges remain: 1) how to exploit the vast information among continuous frames to extract powerful spatio-temporal features across both the spatial and temporal domains, and 2) how to restore high-quality derained videos with a high-speed approach. In this paper, we present a new end-to-end video deraining framework, named Enhanced Spatio-Temporal Interaction Network (ESTINet), which considerably boosts current state-of-the-art video deraining quality and speed. The ESTINet takes the advantage of deep residual networks and convolutional long short-term memory, which can capture the spatial features and temporal correlations among continuing frames at the cost of very little computational source. Extensive experiments on three public datasets show that the proposed ESTINet can achieve faster speed than the competitors, while maintaining better performance than the state-of-the-art methods.
1 INTRODUCTION
Video deraining must exploit temporal correlations across successive frames while improving the speed–quality trade-off. ESTINet combines spatial and temporal feature extraction with enhanced spatio-temporal refinement and reports state-of-the-art quality and speed.
- Motivation: Video deraining is harder than image deraining because it must model temporal correlations among continuing frames.Existing methods can achieve state-of-the-art performance slowly, while fast methods may lag in performance.
- Proposed framework: ESTINet is an end-to-end network designed to extract stronger spatio-temporal information for rain-streak removal.Its framework includes spatial, spatio-temporal interaction, and enhanced spatio-temporal modules.
- Proposed framework: SICM extracts high-level spatial features from rainy frames before STIM uses an Interaction-CBLSTM to model temporal correlations with little computational increase.STIM also provides coarse derained frames for subsequent enhancement.
- Proposed framework: Interaction-CBLSTM connects features from the last frame to the input and replaces tanh with convolution to accommodate different input-frame scales.
- Results: Experiments on three public rainy-video datasets report state-of-the-art deraining performance while also outperforming competing methods in speed.
2 RELATED WORK
Prior video-deraining work exploits temporal context through handcrafted, prior-based, and deep-learning methods. However, much of the deep-learning literature emphasizes performance over computational time, motivating a method that improves both performance and speed.
- Existing approaches: Single-image deraining uses only one frame’s visual information, whereas video deraining exploits temporal correlations across video sequences.
- Existing approaches: Early video-deraining methods modeled temporal context, motion, or rain appearance using prior-based and learn-based techniques.
- Deep-learning methods: Recent deep-learning methods substantially advanced video deraining through architectures including robust CNNs and recurrent neural networks.
- Research gap: Most deep deraining methods focus on performance while overlooking computational time, leaving a need for higher-speed video deraining with improved performance.
3 ESTINET
ESTINet combines spatial extraction, temporal interaction learning, and enhanced spatio-temporal refinement to recover clean videos from rainy frame sequences. Its two-stage losses supervise coarse and refined deraining results.
- Overall Architecture: ESTINet divides video deraining into SICM for spatial representation, STIM for temporal information, and ESTM for spatio-temporal consistency refinement.Rainy videos are grouped into frame sequences and processed through the three-module framework.
- Frame-based Spatial Representation: SICM uses a lightweight encoder-decoder with ResBlocks and multi-scale feature fusion to extract spatial features from rainy frames.The encoder downsamples feature maps, while the decoder restores resolution through up-projection operations.
- Spatial-Temporal Interaction Learning: STIM uses Interaction-CBLSTM to model temporal correlations with convolutions, bidirectional processing, and last-frame spatial features.The design replaces elementwise operations with convolutions and connects features from the previous frame to the input.
- Enhanced Spatio-Temporal Refinement: ESTM receives coarse deraining results from STIM and combines them with rainy frames using 3D convolutions to refine rain removal and image details.The module targets the difficulty of removing heavy rain while preserving realistic content details.
- Loss Functions: Training combines a spatio-temporal interaction loss for SICM and STIM with an enhanced spatio-temporal loss for ESTM.The interaction loss supervises coarse deraining, while the enhanced loss assesses refined results against clean images; α is set to 1.
4 EXPERIMENTS
Experiments evaluate ESTINet on three public rainy-video datasets, comparing quality, speed, visual results, and architectural components. The results report strong deraining performance and faster processing than competing methods, while ablations examine temporal information and enhanced spatio-temporal refinement.
- Comparison with Existing Methods: Qualitative comparisons on RainSynLight25 and NTURain report better results than existing methods, while real-world sequences show rain removal to some extent.The exemplar comparisons are produced without alignment, and the real-world outputs are shown for input and processed sequences.
- Ablation Study: All SICM + STIM variants outperform SICM + 2DCNN, and increasing the number of input frames improves performance.The ablation defines n as the number of consecutive frames and uses Table 3 for quantitative comparison.
- Ablation Study: SICM + STIM (#5) + ESTM outperforms ConvLSTM and bidirectional ConvLSTM variants, while ESTM’s 3D convolution and full spatio-temporal interaction provide additional gains.These comparisons support the effectiveness of STIM, ESTM, and their combination.
- Efficiency Analysis: ESTINet is significantly faster than other state-of-the-art methods, including FastDeRain, in speed tests on NTURain using a GeForce GTX 1080 Ti.The speed table reports seconds and also provides parameters and FLOPs for deep deraining methods.
5 CONCLUSION
The paper concludes with ESTINet, an end-to-end video-deraining framework that combines spatial extraction, temporal correlation modeling, and enhanced refinement. Experiments show that it outperforms state-of-the-art methods in both quality and speed.
- ESTINet addresses video deraining with a faster scheme that provides better quantitative and qualitative results.
- SICM extracts spatial features, Interaction-CBLSTM-based STIM captures temporal correlations, and ESTM refines coarse outputs into finer derained results.SICM and STIM interact during training before their coarse results are processed by the 3D-DenseNet-based ESTM.
- Extensive experiments verify that the framework outperforms state-of-the-art methods in both quality and speed.