PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework

Jie Chen, Cheen-Hau Tan, Junhui Hou, Lap-Pui Chau, He Li

arXiv:1803.10433v1cs.CV

TL;DR

Rain removal methods struggle with opaque occlusions, fast camera motion, and dynamic scenes, despite the importance of clean visual input for outdoor vision systems. SPAC-CNN uses superpixel-level temporal alignment and CNN detail compensation, achieving up to 5dB reconstruction advantage over state-of-the-art methods. The authors report cleaner removal especially for highly dynamic scenes with heavy, opaque rainfall from fast-moving cameras.

  • Problem

    Existing rain-removal methods are limited by opaque occlusions, fast camera motion, and dynamic scene content, while rain degradation threatens outdoor vision-system robustness.

  • Method

    SPAC-CNN aligns superpixels across video frames, generates match tensors and an intermediate derain output, then uses a CNN to restore details lost through mis-alignment blur.

  • Results

    Up to 5dB reconstruction advantage is reported over state-of-the-art rain-removal methods, with cleaner removal under highly dynamic scenes and heavy, opaque rainfall.

  • Takeaways & Limitations

    Superpixel processing provides a unified way to align background and moving objects while supporting rain removal in challenging fast-camera video scenes.

  • Takeaways & Limitations

    At camera speeds up to 50 km/h, larger search ranges increase computation and can lower PSNR through mis-alignment blur and alignment errors.

Abstract

from arXiv · show

Rain removal is important for improving the robustness of outdoor vision based systems. Current rain removal methods show limitations either for complex dynamic scenes shot from fast moving cameras, or under torrential rain fall with opaque occlusions. We propose a novel derain algorithm, which applies superpixel (SP) segmentation to decompose the scene into depth consistent units. Alignment of scene contents are done at the SP level, which proves to be robust towards rain occlusion and fast camera motion. Two alignment output tensors, i.e., optimal temporal match tensor and sorted spatial-temporal match tensor, provide informative clues for rain streak location and occluded background contents to generate an intermediate derain output. These tensors will be subsequently prepared as input features for a convolutional neural network to restore high frequency details to the intermediate output for compensation of mis-alignment blur. Extensive evaluations show that up to 5 dB reconstruction PSNR advantage is achieved over state-of-the-art methods. Visual inspection shows that much cleaner rain removal is achieved especially for highly dynamic scenes with heavy and opaque rainfall from a fast moving camera.

1. Introduction

Rain streaks degrade outdoor visual systems, while existing image- and video-based methods struggle with opaque occlusions, fast camera motion, and dynamic content. SPAC-CNN addresses these challenges through superpixel-level alignment and CNN-based detail compensation.

  • Rain streaks reduce contrast and visibility, obscure scene features, and can be misconstrued as scene motion, threatening outdoor vision-system robustness.
  • Image-based methods lack temporal information and therefore struggle with torrential rain containing large, opaque occlusions.
  • Video-based alignment is difficult because camera motion and moving objects shift scene content, while global alignment can poorly align scenes with large depth ranges.
  • SPAC-CNN aligns superpixels across neighboring frames, handling background and moving objects together without prior assumptions about object motion.
  • Up to 5dB reconstruction advantage is reported over state-of-the-art rain-removal methods, with cleaner removal in highly dynamic scenes under heavy, opaque rainfall.
  • Strong local superpixel properties improve alignment under heavy rain and establish temporal correspondence for recovering heavily occluded backgrounds.

2. Related Work

Prior rain-removal methods use either single-frame visual cues or temporal information, but video methods remain weak in complex dynamic scenes captured by fast-moving cameras.

  • Image-based methods distinguish rain from backgrounds using visual features, priors, and models that separate rain and background layers.
  • Video-based methods exploit temporal, intensity, chromatic, geometric, motion-region, optical-flow, or matrix-decomposition cues to separate rain from scene content.
  • Temporal methods handle occlusions better than image-based methods but perform poorly for complex dynamic scenes recorded from fast-moving cameras.

3. Proposed Model

The proposed model aligns video content at the superpixel level, forms an intermediate derain result from matched tensors, and uses a CNN to compensate lost high-frequency details.

  • The notation distinguishes scalars, 2D matrices, 3D tensors, functions, and operators through different typographic conventions.
  • Given a target frame, the method uses neighboring past and future frames in a sliding buffer and derains only the luminance channel.
  • The history buffer is updated with the derain output to support cleaner deraining under heavy rainfall.
  • Superpixel-level matching produces an optimal temporal match tensor and a sorted spatial-temporal match tensor for subsequent processing.
  • The sorted spatial-temporal tensor is averaged into an intermediate derain output, while both tensors provide CNN input features for compensating mis-alignment blur.

3.1. Robust Content Alignment via Superpixel Spatial-Temporal Matching

The framework replaces global or pixel-level correspondence with superpixel-based spatial-temporal matching, using rain-aware templates to align scene content and suppress occlusions. Matched tensors are averaged to estimate rain-free content for subsequent detail recovery.

  • Content Alignment: Global vs. Superpixel: Video deraining depends on accurate correspondence across frames, but global alignment fails in dynamic scenes with large depth ranges and pixel-level matching is vulnerable to sparse features and rain occlusion.These limitations motivate alignment over larger, depth-consistent regions.
  • Content Alignment: Global vs. Superpixel: Superpixels group perceptually meaningful pixels whose boundaries often follow scene content, making them more adaptive than rectangular units for content alignment.The method adopts superpixels as basic units because they are more likely to segment uniform-depth regions.
  • Optimal Temporal Matching for Rain Detection: Each target superpixel is represented by a bounding box and searched within a spatial-temporal buffer covering the possible motion range across neighboring frames.Template matching searches candidate locations in the buffer, and selected matches form temporal slices of the optimal temporal match tensor T0.
  • Optimal Temporal Matching for Rain Detection: The optimal temporal matches provide clues for rain detection, while chroma-gradient edge information reduces false rain classifications caused by background mis-alignment.The method combines temporal intensity evidence with a rain-free edge map derived from the Cb and Cr channels.
  • Sorted Spatial-Temporal Template Matching for Rain Occlusion Suppression: Rain-free superpixel pixels guide a second template-matching stage that sorts candidate matches into T1, whose slices are expected to remain aligned and resist rain interference.When enough candidates are stacked, averaging T1 slices suppresses rain-induced fluctuations and recovers occluded background content for CNN feature preparation.

3.2. Detail Compensation for Mis-Alignment Blur

The method uses complementary temporal and spatial-temporal features to recover occluded content while compensating for blur and rain-mask errors with a CNN.

  • Occluded Background Feature: Averaging sorted spatial-temporal matches estimates a rain-free image and restores information from rain-occluded regions, but unavoidable mis-alignment causes blur.The blur is especially noticeable under fast camera motion.
  • Occluded Background Feature: The first feature replaces pixels marked as rain with corresponding pixels from the averaged estimate while retaining unmasked pixels from the target frame.This feature is already a reasonable derain output, but its quality depends on rain-mask correctness.
  • Occluded Background Feature: False-positive rain labels lose content detail, whereas false-negative labels reintroduce rain streaks, motivating more informative CNN inputs.False-positive pixels are background pixels classified as rain; false-negative pixels are rain pixels classified as background.
  • Temporal Consistency Feature: Temporal consistency identifies false-negative pixels because correctly recovered pixels should show only positive rain-induced intensity fluctuations across neighboring frames.An obvious negative temporal intensity drop indicates a likely false-negative pixel.
  • Temporal Consistency Feature: The optimal temporal match tensor supplies temporal correspondence clues, while sorted spatial-temporal matches provide detail references with supposedly small mis-alignment.These tensors form complementary feature groups for CNN-based compensation.
  • CNN Structure and Training Details: A four-layer fully convolutional CNN with kernel sizes 11, 5, 3, and 1 predicts content detail, using ReLU after every convolutional layer.Training minimizes L2 distance to the clean ground-truth scene.

4. Performance Evaluation

SPAC-CNN is evaluated on synthesized rain videos spanning different camera motions, using quantitative, visual, edge-precision, and runtime comparisons. It consistently improves reconstruction and rain removal, including under fast motion and heavy opaque rain.

  • Experimental setup: The evaluation uses eight held-out videos with synthesized rain, grouped by panning or unstable cameras versus fast-moving cameras.Fast-camera scenes use speeds between 20 and 30 km/h.
  • Rain-streak edge precision-recall: SPAC-CNN shows consistent advantages over SPAC-Avg in rain-streak edge precision-recall curves for both camera-motion groups.The comparison indicates CNN compensation suppresses rain-edge influences while restoring scene details.
  • Rain-streak edge precision-recall: SPAC-CNN remains ahead of image-based methods at all recall rates for fast-camera scenes, while VMD deteriorates and introduces background distortion.For slower-motion scenes, video-based methods outperform image-based methods because temporal correspondence is more accurately established.
  • Scene reconstruction PSNR/SSIM: 5 dB higher reconstruction PSNR and at least 0.06 higher SSIM than SPAC-Avg are reported for both camera-motion groups.Across groups, SPAC-CNN also exceeds image-based baselines by approximately 3–5 dB in the reported comparisons.
  • Feature analysis: Combining all three CNN input features gives the highest derain PSNR, with F1 identified as the most important feature.The feature ablation evaluates testing data a1 and b4.
  • Visual comparison: Visual comparisons show image-based methods fail on large opaque streaks, whereas temporal information restores occluded details and SPAC-CNN preserves high-frequency content.SPAC-CNN generalizes from synthetic training rain to real-world rain and remains robust under heavy rain and fast camera motion.
  • Execution efficiency: SPAC-CNN is much faster than the compared video-based method and has runtime comparable to DDN for a VGA frame.SPAC-Avg is faster than all other methods.

5. Discussion

The discussion identifies motion-model and speed boundaries for SPAC-CNN. Superpixels are important for matching accuracy, while faster cameras increase computation and can increase residual misalignment.

  • Superpixel decomposition: Replacing superpixels with rectangular decomposition deteriorates matching accuracy and introduces averaging blur, especially at object boundaries.The choice of superpixels is therefore a key performance factor in SPAC-CNN.
  • Motion handling: SP template matching handles translational motion, while global frame alignment and CNN compensation mitigate errors from rotation, scaling, and non-rigid transforms.These additional motions are not handled by SP template matching alone.
  • High-speed limitation: At camera speeds up to 50 km/h, enlarging the SP search range increases computation, while larger misalignment blur can lower PSNR and cause alignment errors.The authors suggest retraining the CNN with data from such faster cameras to improve performance.

6. Conclusion

The paper concludes that SPAC-CNN uses superpixel-based alignment and CNN compensation to remove torrential, opaque rain from video captured by fast-moving cameras. Experiments support its efficiency and robustness relative to state-of-the-art methods.

  • Conclusion: SPAC-CNN uses superpixels for content alignment and occlusion removal, then a CNN compensates blur introduced by deraining operations.The proposed video-based system is designed for torrential rainfall with opaque streak occlusions from fast-moving cameras.
Loading 1803.10433v1…