Source-linked AI summary
Deep Stacked Hierarchical Multi-patch Network for Image Deblurring
Hongguang Zhang, Yuchao Dai, Hongdong Li, Piotr Koniusz
TL;DR
Current multi-scale and scale-recurrent deblurring architectures face expensive runtime and performance saturation when adding finer levels. The paper introduces a hierarchical multi-patch network and stacked variants, achieving state-of-the-art deblurring performance, real-time 720p processing, and improved results with stacking.
Problem
Multi-scale and scale-recurrent architectures incur expensive runtime and cannot improve deblurring performance by simply adding finer levels.
Method
DMPHN exploits a Spatial Pyramid Matching-like multi-patch hierarchy that passes local residual cues toward coarser regions, with stacked variants addressing depth limitations.
Results
The model achieves state-of-the-art PSNR and SSIM performance, runs at 30fps for 720p images, and stacked variants further improve results over shallower and competing approaches.
Takeaways & Limitations
The same stacked network can trade performance against runtime by varying depth, while the basic model supports efficient real-time motion deblurring.
Takeaways & Limitations
The authors observe a stacking-depth limitation in multi-scale and multi-patch models, where adding additional coarser or finer grids does not improve overall deblurring performance.
Abstract
from arXiv · showhide
Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.
1. Introduction
Existing multi-scale and scale-recurrent deblurring models incur expensive runtime and show limited gains from increased depth. DMPHN addresses these issues with a fine-to-coarse multi-patch hierarchy and stacked variants.
- Challenges: Multi-scale and scale-recurrent methods have expensive runtime and struggle to improve deblurring quality.Their coarse-to-fine designs use many training parameters and costly processing.
- Challenges: Increasing depth at very low-resolution inputs does not seem to improve multi-scale deblurring performance.
- Proposed Approach: DMPHN uses an SPM-inspired fine-to-coarse multi-patch hierarchy with residual-like links between levels.Local residual cues are passed toward coarser regions, allowing levels to focus on different blur scales.
- Proposed Approach: Equal spatial resolution across levels enables small-filter residual learning and fast inference, while finer levels receive relatively more training data.
- Stacking: The authors identify depth limitations in existing models and introduce stacked DMPHN variants to overcome performance saturation.
- Additional Contributions: The work also investigates weight sharing and proposes a memory-friendly DMPHN variant for efficient motion deblurring.
2. Related Work
Prior deblurring research includes conventional optimization, CNN-based regression, recurrent architectures, and GANs. These methods target non-uniform blur, temporal information, or photorealistic detail through different modeling strategies.
- Conventional Methods: Conventional methods often fail on non-uniform blur because they assume a spatially invariant deblurring kernel.Their complex computational inference also leads to long processing times.
- Deep Deblurring: CNN methods learn mappings from blurry inputs to sharp images, including non-blind kernel-conditioned and patch-based approaches.
- Deep Deblurring: Multi-scale CNNs restore sharp images across resolutions using multi-scale losses that mimic conventional coarse-to-fine processing.
- Recurrent Methods: RNN-based deblurring models process sequential information, while SRN-DeblurNet aggregates feature maps across coarse-to-fine scales with ConvLSTM cells.
- Generative Methods: GAN-based methods target high-quality deblurred images by emphasizing texture details and photorealistic generation.
3. Our Framework
DMPHN uses a fine-to-coarse hierarchical multi-patch architecture with residual information flow, avoiding costly multi-scale upsampling and supporting stacked depth for improved deblurring.
- Hierarchical multi-patch architecture: The model uses feature concatenation and residual-like links instead of the deconvolution/upsampling links used by prior multi-scale methods.These operations are enabled by the multi-patch setup.
- Efficiency: 3.6 MB parameters versus 303.6 Mb for the multi-scale network in [17] supports DMPHN’s fast inference.The comparison is attributed to the residual nature of the model.
- Hierarchical multi-patch architecture: DMPHN processes non-overlapping image patches hierarchically, passing finer-level residual information to coarser levels through encoder-decoder pairs.The (1-2-4-8) notation denotes patch counts from coarsest to finest levels.
- Stacked multi-patch network: Adding finer grids does not further improve performance because coarser levels quickly attain low empirical training loss and exclude the finest levels’ residual contributions.This motivates increasing depth horizontally by stacking multiple DMPHN models.
- Stacked multi-patch network: Stack-DMPHN connects multiple bottom-to-top DMPHNs, while Stack-VMPHN reverses information flow in a top-bottom-top configuration.Both stacked designs evaluate MSE at each sub-model output; the paper reports that Stack-VMPHN outperforms DMPHN.
- Stacked multi-patch network: Stacked networks improve deblurring while allowing runtime and model size to remain within a reasonable range by using efficient DMPHN units.The framework supports varying stacking depth for different applications.
4. Experiments
The experiments evaluate DMPHN on GoPro and VideoDeblurring-related settings, showing strong deblurring quality alongside efficient inference on high-resolution inputs.
- Evaluation Setup: DMPHN is evaluated on GoPro, whose 3,214 blurred/clean pairs come from 33 sequences at 720×1280 resolution.The protocol uses 2,103 pairs for training and 1,111 for testing.
- Performance: DMPHN (1-2-4-8) achieves the best reported PSNR and SSIM among compared state-of-the-art models.The paper also notes that patches below 1/16 image size are not helpful for motion-blur removal.
- Runtime: 30ms per 720×1280 image enables real-time 720p deblurring at 30fps.The authors caution that I/O overhead still requires fast transfers and sufficient GPU or SSD resources.
- Runtime: DMPHN’s fast runtime is attributed to shallow encoders and decoders, small filters, fewer links, and reduced upsampling or deconvolution.These design choices reduce computation relative to coarse-to-fine architectures.
- Model Size: The model uses a much smaller parameter budget than the compared multi-scale network, whose size is reported as 303.6MB versus 3.6MB for each DMPHN encoder-decoder.The comparison passage describes the DMPHN model size as 10× smaller, while the parameter passage reports 3.6 MB and 303.6 Mb.
5. Conclusions
The paper concludes that hierarchical multi-patch processing provides state-of-the-art non-uniform motion deblurring with real-time 720p inference, while stacked variants further improve results.
- Conclusions: DMPHN achieves state-of-the-art PSNR and SSIM performance and runs at 30fps for 720p images.The conclusion presents these as outcomes of the multi-patch SPM- and residual-like design.
- Conclusions: Stack(4)-DMPHN and Stack(2)-VMPHN improve results over shallower DMPHN and competing approaches while being approximately 4× faster than those methods.The conclusion states that stacking overcomes the depth limitation observed in competing approaches.
- Conclusions: Qualitative comparisons across GoPro and VideoDeblurring scenes report that the proposed model produces the sharpest details across different scenes.Figure 9 compares blurry inputs, two prior methods, and the proposed result.
A. Outputs of Stacked Network
The stacked-network outputs improve progressively across sub-models, with later outputs producing higher-quality deblurring results.
- VMHPN: Figure 10 presents Stack(3)-VMHPN outputs from M1 through M3 from left to right.The figure caption identifies the displayed sub-model order.
- DMHPN: Figure 11 presents Stack(3)-DMHPN outputs from M1 through M3 from left to right.The figure caption identifies the displayed sub-model order.
B. Extension to Saliency Detection
The proposed network is also evaluated for saliency detection on MSRA-B, where it can be trained from scratch rather than depending on ImageNet pretraining.
- Dataset and Setup: The saliency-detection experiment uses MSRA-B, containing 3,000 training images and 2,000 testing images.The experiment investigates the model’s generalization to a different task.
- Training: Unlike current deep saliency methods described here, the proposed network can be trained from scratch without ImageNet pretraining.The passage states that those other methods often do not converge without such pretraining.
- Quantitative Results: Table 5 reports quantitative saliency-detection results on MSRA-B, with higher Fβ and lower MAE indicating better scores.The caption specifies the directionality of both metrics.
- Qualitative Results: Figure 12 shows instances of saliency detection on the MSRA-B dataset.The caption identifies the figure’s content but does not state a quantitative comparison.