PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatially-Adaptive Image Restoration using Distortion-Guided Networks

Kuldeep Purohit, Maitreya Suin, A. N. Rajagopalan, Vishnu Naresh Boddeti

arXiv:2108.08617v1cs.CVcs.LG

TL;DR

Spatially rigid, degradation-specific restoration struggles with images containing both severely distorted and clean regions. SPAIR localizes degraded pixels and uses distortion-guided sparse processing to adapt restoration spatially, outperforming strong degradation-specific baselines across four tasks and 11 datasets.

  • Problem

    Existing restoration methods often use degradation-specific architectures and spatially invariant processing, limiting restoration when distortions vary across image regions.

  • Method

    SPAIR combines a localization network that predicts degraded regions with a restoration network using spatial modulation, sparse convolution, and sparse non-local attention.

  • Results

    SPAIR outperforms strong degradation-specific baselines across four restoration tasks and 11 datasets, with a 2.16 dB average gain over MSPFN for rain-streak removal.

  • Takeaways & Limitations

    Explicit distortion localization enables selective restoration of degraded regions while reducing processing of unaffected regions across diverse degradation types.

  • Takeaways & Limitations

    The sparse convolution module has a receptive field limited to degraded pixels, motivating the complementary SNL module for global context aggregation.

Abstract

from arXiv · show

We present a general learning-based solution for restoring images suffering from spatially-varying degradations. Prior approaches are typically degradation-specific and employ the same processing across different images and different pixels within. However, we hypothesize that such spatially rigid processing is suboptimal for simultaneously restoring the degraded pixels as well as reconstructing the clean regions of the image. To overcome this limitation, we propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts computation to difficult regions in the image. SPAIR comprises of two components, (1) a localization network that identifies degraded pixels, and (2) a restoration network that exploits knowledge from the localization network in filter and feature domain to selectively and adaptively restore degraded pixels. Our key idea is to exploit the non-uniformity of heavy degradations in spatial-domain and suitably embed this knowledge within distortion-guided modules performing sparse normalization, feature extraction and attention. Our architecture is agnostic to physical formation model and generalizes across several types of spatially-varying degradations. We demonstrate the efficacy of SPAIR individually on four restoration tasks-removal of rain-streaks, raindrops, shadows and motion blur. Extensive qualitative and quantitative comparisons with prior art on 11 benchmark datasets demonstrate that our degradation-agnostic network design offers significant performance gains over state-of-the-art degradation-specific architectures. Code available at https://github.com/human-analysis/spatially-adaptive-image-restoration.

1. Introduction

Spatially varying degradations affect different image regions with different severity, making spatially rigid restoration inadequate. SPAIR addresses this by localizing distortions and selectively guiding restoration while preserving clean regions.

  • Motivation: Rain, raindrops, motion blur, and shadows arise from different imaging conditions but share spatially varying effects.Their affected regions depend on factors such as size, location, scene depth, and motion.
  • Motivation: Fixed CNN filters can perform poorly when severe distortions coexist with unaffected regions, introducing unwanted changes or artifacts in clean pixels.Image-dependent distortion location and magnitude further challenge static processing.
  • SPAIR framework: SPAIR uses a distortion-localization network, NetL, to estimate a binary mask that steers NetR toward degraded regions.The design targets both restoration of degraded pixels and reconstruction of clean regions.
  • SPAIR framework: NetR combines spatial feature modulation, sparse convolution, and sparse non-local attention for distortion-guided feature processing.SFM modulates feature statistics, while SC and SNL focus extraction on degraded regions and gather local or global context.
  • Scope: The framework was designed to address diverse spatially varying degradations without relying on task-specific network engineering.The paper reports detailed analysis, qualitative results, and generalization tests.

2. Related Works

Prior adaptive inference methods alter computation, but they do not provide the fine-grained spatial control required for restoration across intermediate layers. SPAIR instead uses explicit degradation masks throughout filtering, attention, and feature guidance.

  • Adaptive inference: Existing adaptive inference methods dynamically skip layers, sample pixels, or exploit input sparsity to alter CNN computation.These approaches include layer skipping, pixel sampling with interpolation, and input-sparsity exploitation.
  • Adaptive inference: Such methods lack fine-grained spatial control across multiple intermediate layers and often retain shared parameters for degraded and clean regions.Layer or network pruning is also difficult to apply to encoder-decoder restoration architectures.
  • SPAIR distinction: SPAIR uses degradation masks for sparse filtering, attention-weight calculation, and guidance toward non-degraded regions.This extends mask use beyond input concatenation and supports selective processing of degraded pixels.
  • SPAIR distinction: Compared with rain-guided models, SPAIR adds intermediate feature-statistics transfer, global context through SNL, and spatially selective processing.The comparison identifies these as three differences from prior rain-guided approaches.
  • Multiple degradations: SPAIR is evaluated on realistic datasets spanning physically unrelated degradations, unlike approaches limited to simple synthetic degradation combinations.The related-work discussion contrasts this scope with OWAN’s synthetic degradations.

3. Proposed Network Architecture

SPAIR separates distortion localization from restoration, then uses the predicted mask and localization features to adapt normalization, convolution, and non-local context aggregation within NetR. Its decoder selectively processes degraded regions while preserving clean-region information and uses global clean context to complement local sparse filtering.

  • Architecture: SPAIR divides restoration into NetL, which localizes degraded pixels, and NetR, which uses that knowledge to guide restoration.NetL predicts a binary mask, while NetR combines the degraded input with localization guidance.
  • Architecture: Both networks use U-Net-style encoder-decoder backbones, with NetL lightweight because localization is simpler than intensity restoration.The backbone is a densely connected encoder-decoder adopted for both localization and restoration.
  • Spatial Feature Modulation: SFM fuses NetL features with NetR features at multiple levels and adaptively shifts feature statistics at degraded locations.The final SFM output applies modulation only where the mask indicates degradation, leaving clean locations unchanged.
  • Mask-Guided Sparse Convolution: SC performs sparse convolution on masked degraded pixels, reducing the need for spatially invariant filters to jointly learn restoration and reconstruction.The sparse operation samples mask-selected pixels and applies convolution using the kernel support region.
  • Region-Guided Sparse Non-Local Module: Because SC sees only degraded pixels locally, SNL adds distortion-guided global context aggregation from valid clean or less-degraded locations.SNL operates on degraded locations, assigns dynamic weights to usable context, and leaves clean-region features unaltered.
  • Region-Guided Sparse Non-Local Module: SNL uses two horizontal-vertical scanning stages and sparse 1 × 1 refinement to aggregate full-image context while processing only mask-selected feature locations.The two stages repeat directional aggregation, and the sparse 1 × 1 convolution refines selected point-wise features instead of the full H×W grid.

4. Datasets and Implementation Details

SPAIR is evaluated across four spatially varying restoration tasks using established datasets and task-specific training configurations. Deraining uses a single model trained on 13,712 clean-rain pairs and tests across five benchmarks.

  • Rain-Streaks: Deraining trains on 13,712 clean-rain image pairs gathered from multiple datasets and evaluates on Rain100H, Rain100L, Test100, Test2800, and Test1200.
  • Relative error reduction is computed by converting PSNR to RMSE and SSIM to DSSIM.
  • Deraining reports PSNR and SSIM on the Y channel, while shadow removal uses Lab-space RMSE over shadow and non-shadow regions.
  • Rain-Streaks: Real-world rain evaluation uses SPANet, containing 2 × 10^5 training images and 1000 testing images.
  • Raindrop: Raindrop removal uses AGAN with 861 training and 58 test samples, generated using raindrop-covered glass between camera and scene.
  • Shadow: Shadow removal uses ISTD with 1300 training and 540 test images containing real shadows and diverse textured scenes.

5. Experimental Evaluation

Across deraining, raindrop removal, shadow removal, and deblurring, SPAIR reports stronger quantitative and qualitative results than competing methods. The evaluations also highlight adaptive restoration, artifact reduction, and cross-dataset generalization.

  • Rain-Streak Removal: 2.16 dB averaged across all deraining datasets is SPAIR’s gain over MSPFN, while gains reach 3 dB on low-rain-density datasets.The improvement is 0.41 dB on very heavy rain in Rain100H.
  • Rain-Streak Removal: SPAIR recovers background textures more effectively than existing deraining methods and avoids artifacts across scenes, rain densities, streak orientations, and magnitudes.
  • Raindrop Removal: SPAIR outperforms raindrop-removal baselines by a large margin, producing results visually closer to ground truth with fewer artifacts and color distortions.
  • Shadow Removal: SPAIR has minimal shadow-boundary artifacts and outperforms traditional and learning-based baselines qualitatively and quantitatively on ISTD.
  • Deblurring: SPAIR performs favorably across GoPro, HIDE, and RealBlur-J, outperforming all methods including [49] on HIDE despite training only on GoPro.
  • Deblurring: 0.39 dB over DMPHN and 0.44 dB over the existing best method are SPAIR’s gains on RealBlur-J under cross-dataset and in-dataset settings, respectively.
  • Deblurring: Visual deblurring comparisons show non-uniform restoration that preserves sharpness and more faithfully restores text, boundaries, and textures.

6. Network Analysis

The ablation study isolates the contributions of degradation guidance, sparse convolution, and sparse non-local processing. The final SPAIR model improves over dense CNN baselines while retaining one architecture across restoration tasks.

  • Table 6 evaluates design choices using PSNR on AGAN raindrop and GoPro motion-blur benchmarks.
  • Spatial Feature Modulator: SFM-guided Net2 substantially improves accuracy over the dense Net1 baseline, supporting degradation guidance through masks and localization features at intermediate levels.
  • Sparse Non-Local Module: SNL replaces the general non-local layer with explicit distortion guidance, dynamically weighting features in degraded regions while better serving degraded and clean regions.
  • Net5 versus Net3 demonstrates the utility of global context aggregation, while Net5’s improvement over Net1 supports the overall solution over static CNNs.
  • Practical Benefit: The same architecture can support rain, shadows, blur, and related applications by changing learned weights rather than re-engineering the network.

7. Conclusions

SPAIR combines degraded-region localization with region-guided sparse restoration for spatially varying image degradations. Across four restoration tasks and 11 datasets, it outperforms strong degradation-specific baselines.

  • SPAIR models restoration as degraded-region localization followed by region-guided sparse restoration.
  • The framework uses NetL features to spatially modulate NetR through the SFM module.
  • Sparse convolution and sparse non-local attention introduce distortion-localization awareness into the restoration network.
  • Across 11 datasets and four restoration tasks, SPAIR outperforms strong degradation-specific baselines.The tasks involve raindrop, rain-streak, shadow, and motion-blur removal.
  • Ablation analyses and visualizations validate the effectiveness of SPAIR’s key components.
Loading 2108.08617v1…