Source-linked AI summary
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring
Maitreya Suin, Kuldeep Purohit, A. N. Rajagopalan
TL;DR
The paper addresses inefficient and spatially inflexible CNN processing for non-uniform motion blur in dynamic scenes. It introduces a content-adaptive, patch-hierarchical architecture combining efficient global attention, adaptive local filtering, and cross-attention. On two benchmarks, the approach reports state-of-the-art results and is 3× faster than the nearest competitor.
Problem
CNN-based deblurring methods use spatially invariant weights and geometrically uniform receptive fields, making efficient handling of spatially varying blur difficult.
Method
The method combines a multi-patch hierarchical structure with global-local content-aware filtering, efficient attention, adaptive local filters, and cross-attention across levels.
Results
The approach achieves state-of-the-art results on two dynamic-scene deblurring benchmarks while being 3× faster than the nearest competitor [26].
Takeaways & Limitations
The design delivers content-aware spatially varying restoration with consistent performance across diverse blur magnitudes and suitability for computationally limited environments.
Takeaways & Limitations
High memory requirements limited the ablation network to one decoder attention module per level and prevented adding more blocks or cross-attention.
Abstract
from arXiv · showhide
This paper tackles the problem of motion deblurring of dynamic scenes. Although end-to-end fully convolutional designs have recently advanced the state-of-the-art in non-uniform motion deblurring, their performance-complexity trade-off is still sub-optimal. Existing approaches achieve a large receptive field by increasing the number of generic convolution layers and kernel-size, but this comes at the expense of of the increase in model size and inference speed. In this work, we propose an efficient pixel adaptive and feature attentive design for handling large blur variations across different spatial locations and process each test image adaptively. We also propose an effective content-aware global-local filtering module that significantly improves performance by considering not only global dependencies but also by dynamically exploiting neighbouring pixel information. We use a patch-hierarchical attentive architecture composed of the above module that implicitly discovers the spatial variations in the blur present in the input image and in turn, performs local and global modulation of intermediate features. Extensive qualitative and quantitative comparisons with prior art on deblurring benchmarks demonstrate that our design offers significant improvements over the state-of-the-art in accuracy as well as speed.
1. Introduction
Dynamic-scene motion deblurring remains difficult because spatially varying blur challenges fixed convolutional processing and large receptive fields increase computation. The paper proposes adaptive global-local filtering and attentive patch-hierarchical processing to improve the accuracy-speed trade-off.
- Dynamic scenes produce spatially varying blur from moving objects, camera shake, and depth variations, limiting the generalization of hand-crafted priors.
- Fixed, spatially invariant CNN weights are sub-optimal for differently blurred pixels, while stacking filters increases computational cost and memory consumption.
- Large receptive fields improve deblurring but create a difficult trade-off among inference speed, receptive field, and accuracy.
- A multi-patch hierarchical architecture replaces depth-based cascading with global-local attentive modules and residual attention across layers.
- The proposed design uses global attention, adaptive local filters, and cross-attention across encoder-decoder levels to transform features content-adaptively.
- 3× faster than the nearest competitor [26], while extensive benchmark evaluations report state-of-the-art dynamic-scene deblurring results.
2. Proposed Architecture
The proposed architecture replaces static depth-and-filter expansion with a dynamic framework that adapts filtering and receptive fields across locations and input images. Its three-level multi-patch hierarchy and content-aware processing modules target efficient learning and faster processing.
- Static receptive-field expansion through deeper CNNs and larger filters can be suboptimal because effective receptive fields remain smaller than theoretical ones.
- The architecture dynamically changes filtering and receptive fields across spatial locations and input images.
- Experiments report consistent performance across diverse blur magnitudes and utility for computationally limited environments.
- The network uses a three-level multi-patch hierarchy, reducing patches at higher levels while adaptively fusing lower-level features through attention.The input is sliced into four non-overlapping patches at the bottom level, and level 1 produces the final deblurred image.
- Each level combines an encoder and decoder with residual blocks containing convolutional layers and a content-aware processing module.The processing module has global and local feature branches that are dynamically fused.
3. Content-Aware Processing Module
The content-aware processing module combines global attention with pixel-dependent local filtering, then fuses both branches adaptively to handle spatially varying motion blur.
- Global feature processing: Self-attention gathers global information for each pixel while avoiding the quadratic memory cost of standard attention.Reordering the matrix multiplications changes complexity from O(d_a(HW)^2) to O(d_a d_c HW), enabling attention across encoder-decoder levels.
- Global feature processing: The module generates spatial attention masks and adaptively distributes complementary global features to pixels.Attention maps weight spatial features, aggregate global descriptors, and emphasize important feature embeddings before redistribution.
- Cross attention: Cross encoder-decoder and cross-level attention transfers information from other layers or levels while making attention decisions from decoder or higher-level features.The attended feature source differs from the feature source that generates the attention maps.
- Local feature processing: Pixel-Dependent Filtering uses kernels and offsets that vary by pixel but remain shared across channels for efficient local processing.A maximum offset threshold constrains local processing, while standard convolution is recovered when kernels are constant and offsets are zero.
- Attentive fusion: A learned fusion mask dynamically adjusts the contribution of the global and local branches at each pixel.The fused output contains global and local information distributed adaptively along pixels.
4. Experiments
Experiments evaluate the proposed deblurring model on GoPro and HIDE benchmarks, compare qualitative restoration and efficiency, and analyze the contributions and interpretability of its adaptive modules.
- Quantitative Evaluation: The evaluation uses GoPro and HIDE test benchmarks, comparing the model with conventional and learning-based deblurring methods.GoPro contains 1103 test images and HIDE contains 2025 test images; comparisons include state-of-the-art learning-based approaches and two representative traditional methods.
- Quantitative Evaluation: On HIDE, the approach outperforms all compared methods without requiring human bounding-box supervision.The authors attribute this superiority to the robustness of the proposed adaptive modules.
- Qualitative Evaluation: Visual comparisons show that prior methods can leave incomplete deblurring or artifacts, whereas the proposed network restores details in text and edge regions more faithfully.The model also avoids parameter tuning during testing required by some compared methods.
- Quantitative Evaluation: The method achieves better PSNR and SSIM with lower inference time than DMPHN on both datasets, using a comparable number of parameters.The evaluation reports consistently stronger quantitative and visual results while maintaining a favorable efficiency comparison.
- Ablation Studies: Beyond three residual blocks, quantitative improvement becomes marginal, motivating the selected encoder-decoder depth.Training performance and quantitative results improve with more blocks, but gains beyond three are limited.
- Ablation Studies: Ablations show complementary benefits from spatial attention and dynamic filtering, while adaptive offsets and weights improve directional local filtering.Cross-attention also improves information flow between encoder-decoder layers and levels relative to simple addition.
- Interpretability Analysis: The visualizations relate attention weights to dominant blurred regions and show spatial masks suppressing homogeneous regions before self-attention.Additional visualizations expose spatially varying filter offsets and predicted-kernel variance for different blur patterns.
5. Conclusions
The paper concludes that its content-adaptive architecture addresses spatially varying blur in dynamic scenes while balancing accuracy, memory, and processing time. Its adaptive filtering and attention modules also provide interpretability and may extend to other image-processing tasks.
- Contributions: The proposed content-adaptive architecture targets spatially varying blur in dynamic-scene images.It combines efficient self-attention, cross-attention, and content-aware dynamic filtering.
- Conclusions: Experiments show better qualitative and quantitative results than state-of-the-art methods on two benchmarks.The authors report an accuracy, memory, and time balance for the proposed approach.
- Conclusions: The method is presented as more interpretable than existing deep learning-based approaches and applicable to other image-processing tasks.The conclusion identifies interpretability as a key strength and proposes broader task applicability.