PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions

Hyeongseok Son, Junyong Lee, Sunghyun Cho, Seungyong Lee

arXiv:2108.09108v1cs.CV

TL;DR

Single-image defocus deblurring is difficult because blur size and shape vary spatially. The paper proposes a KPAC network that models defocus through shared-weight inverse-kernel layers and attention. It reports state-of-the-art performance with a lighter-weight network than previous work.

  • Problem

    Single-image defocus deblurring must handle blur that varies spatially in both size and shape.

    Source: §1 · p. 1

  • Method

    The proposed KPAC block shares convolution weights across atrous layers and uses scale and shape attention to model spatially varying inverse kernels.

    Source: §1 · p. 2

  • Results

    The lightweight network achieves state-of-the-art performance and is lighter-weighted than previous work [1].

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that stacking KPAC blocks offers an effective lightweight way to handle spatially varying defocus blur.

    Source: §6 · p. 8

    The method may still fail on large-scale blur, irregular blur shapes, and bokeh with sharp boundaries.

    Source: §6 · p. 8

Abstract

from arXiv · show

This paper proposes a novel deep learning approach for single image defocus deblurring based on inverse kernels. In a defocused image, the blur shapes are similar among pixels although the blur sizes can spatially vary. To utilize the property with inverse kernels, we exploit the observation that when only the size of a defocus blur changes while keeping the shape, the shape of the corresponding inverse kernel remains the same and only the scale changes. Based on the observation, we propose a kernel-sharing parallel atrous convolutional (KPAC) block specifically designed by incorporating the property of inverse kernels for single image defocus deblurring. To effectively simulate the invariant shapes of inverse kernels with different scales, KPAC shares the same convolutional weights among multiple atrous convolution layers. To efficiently simulate the varying scales of inverse kernels, KPAC consists of only a few atrous convolution layers with different dilations and learns per-pixel scale attentions to aggregate the outputs of the layers. KPAC also utilizes the shape attention to combine the outputs of multiple convolution filters in each atrous convolution layer, to deal with defocus blur with a slightly varying shape. We demonstrate that our approach achieves state-of-the-art performance with a much smaller number of parameters than previous methods.

1. Introduction

The paper addresses the difficulty of deblurring spatially varying blur sizes and shapes. It proposes an inverse-kernel KPAC design with shared convolution weights and attention, reporting a lightweight network with better performance than prior work [1].

1. Introduction

  • Real-world blur shapes can exceed disc or Gaussian models, hindering accurate defocus-map estimation and deblurring in conventional two-step approaches [18] [4].These approaches estimate a per-pixel blur-size map, then perform non-blind deconvolution.Source: §1 · p. 1
  • DPDNet avoids a restrictive blur model and outperforms two-step methods on real-world images, but can leave ringing artifacts or residual blur [1].The paper attributes this to the encoder-decoder architecture's limited handling of spatially variant blur.Source: §1 · p. 1
  • The approach exploits the observation that changing blur size while preserving shape leaves the inverse-kernel shape unchanged, allowing spatially varying blur to be modeled without assuming a specific blur shape.A few layers represent discrete inverse-kernel sizes, whose outputs are aggregated for arbitrary sizes.Source: §1 · pp. 1–2
  • KPAC shares convolution kernels across atrous layers with different dilation rates, then uses per-pixel scale attention and shape attention to handle varying blur size and slight shape changes.The shared kernels represent invariant inverse-kernel shape; scale attention aggregates layer outputs.Source: §1 · p. 2
  • The lightweight network achieves state-of-the-art performance and is lighter-weighted than previous work [1].Kernel sharing enables the smaller parameter count.Source: §1 · p. 2

2. Related Work

Prior work addresses defocus deblurring through blur-model-based two-step pipelines, end-to-end learning, and inverse-kernel methods. The paper distinguishes its approach by modeling spatially varying inverse kernels rather than a single kernel for uniform deconvolution.

2.1. Defocus deblurring

  • Conventional two-step methods estimate a defocus map using parametric blur models, but restricted models and map-estimation errors can prevent faithful deblurring [18] [4].Methods use handcrafted, learned, or combined features to estimate the map.Source: §2.1 · p. 2
  • The first end-to-end defocus-deblurring model outperforms conventional two-step approaches, and dual-pixel input significantly improves performance, but its network does not explicitly model spatially varying blur [1].The cited work presents a dataset for supervised training.Source: §2.1 · p. 2

2.2. Inverse kernel

  • Inverse-kernel methods have addressed artifacts through progressive processing, priors, postprocessing, or feature-space processing; prior CNN approaches target uniform deconvolution, unlike this paper's spatially varying inverse kernels.The prior CNN approaches adapt a separable property of large inverse kernels.Source: §2.2 · p. 2

3. Key Idea

The section motivates inverse-kernel deconvolution for spatially varying defocus blur by exploiting scale changes that preserve inverse-kernel shape, then validates a discrete-scale combination approximation.

3.2. Experimental validation of the key idea

  • Figure 1's two Eq. (4) outputs score 21.64 dB and 21.59 dB against the sharp image, with 51.09 dB between the outputs.The figure uses Wiener deconvolution to compute the inverse kernels.Source: Figure 1 · p. 3

3. Key Idea

  • The section frames deblurring as convolution with a spatial pseudo-inverse kernel, connecting inverse-kernel deconvolution to CNN spatial operations.The latent sharp image is derived by convolving the blurry image with the pseudo-inverse kernel.Source: §3.1 · p. 3

3.1. Inverse kernel-based deconvolution for spatially varying defocus blur

  • Changing a blur kernel's support size while preserving its shape leaves the corresponding inverse-kernel shape unchanged, with its scale changing accordingly.The inverse kernel is the spatial pseudo-inverse used for deconvolution.Source: §3.1 · p. 3

3.2. Experimental validation of the key idea

  • 87.4% with equal weights and 92.5% with non-negative least-squares weights: Fig. 2's discrete-scale mixtures approximate deconvolution at another inverse-kernel size.The equal-weight result uses αi = 0.5.Source: Figure 2 · p. 3, §3.2 · pp. 3–4

3.1. Inverse kernel-based deconvolution for spatially varying defocus blur

  • Spatially varying deconvolution can be approximated by combining results from inverse kernels with a discrete set of sizes using per-pixel weight maps.The weights αi apply to the result image for each inverse-kernel upsampling factor si.Source: §3.1 · p. 3

4. Network Design

The network extends inverse-kernel deblurring to a CNN by combining outputs from a discrete set of scales. Its KPAC block shares atrous-convolution weights across scales while using scale and shape attention to accommodate spatially varying blur.

4. Network Design

  • KPAC replaces direct inverse-kernel combination with a convolutional aggregation of features from multiple inverse-kernel scales.The aggregation is learned rather than limited to a linear combination.Source: §4 · p. 4

4.1. Kernel-sharing Parallel Atrous Convolutional (KPAC) block

  • Parallel atrous-convolution layers share kernel weights, with different dilation rates representing inverse kernels at different sizes.Dilation expands a kernel’s spatial support without changing its number of weights, enabling direct weight sharing.Source: §4.1 · p. 4
  • Per-pixel scale attention aggregates atrous-layer outputs to represent inverse kernels at arbitrary scales.A different spatial attention map weights each layer’s feature map.Source: §4.1 · pp. 4–5
  • Shape attention combines convolution-filter outputs to accommodate variation in inverse-kernel shape.The atrous layers share the channel-wise shape-attention weights.Source: §4.1 · p. 5

4.2. Defocus deblurring network

  • The encoder–decoder network places KPAC blocks between its encoder and decoder and uses LeakyReLU in all but the final convolution.The authors report that KPAC works with nonlinear features, citing the local linearity of CNNs.Source: §4.2 · p. 5
  • Multiple KPAC blocks can be stacked to iteratively remove residual blur, although the block does not explicitly implement conventional inverse kernels.Training uses MAE loss and, for more realistic textures, a perceptual loss combined with MAE.Source: §4.2 · p. 5

5. Experiments

Experiments analyze the KPAC components and evaluate deblurring quality, computational cost, and generalization. The proposed models outperform the compared single-image methods, including dual-pixel DPDNet, while using fewer parameters; the three-level model also uses fewer FLOPs than DPDNet.

5.1. Analysis

  • Weight sharing both reduces the number of learning parameters and improves deblurring quality.The comparison uses two-block models with 5 × 5 kernels.Source: §5.1 · p. 6, Table 1 · p. 6
  • The ablation study reports that each KPAC component improves deblurring quality, while scale and shape attention help handle spatially varying blur and restore fine structures.The qualitative comparison shows models without attention suffering from spatially varying blur.Source: §5.1 · p. 6, Table 2 · p. 6, Figure 7 · p. 6
  • Dilation rates 1, 3, and 5 help remove blur of any, medium, and large sizes, respectively.The analysis manually modulates attention maps to isolate the feature maps from specified scales.Source: §5.1 · p. 6, Figure 8 · p. 7
  • A single KPAC block raises PSNR by 0.90 dB; quality rises with additional blocks before declining after three, so the final model uses two.The authors attribute the decline possibly to increased training complexity and select two blocks for relatively high quality with a small model.Source: §5.1 · p. 7, Table 3 · p. 7

5.2. Evaluation

  • The proposed models outperform classical two-step methods and single-image DPDNet, and also outperform dual-pixel DPDNet without its dual-pixel cue.The two- and three-level models perform similarly; the three-level model tends to handle extremely large blur better.Source: Table 4 · p. 7, §5.2 · p. 7
  • The DPDD qualitative comparison reports sharper results with more detail, including detail clarity comparable to dual-pixel DPDNet.The illustrated result is produced by the three-level model.Source: §5.2 · pp. 7–8, Figure 10 · p. 8
  • 10 times fewer FLOPs than DPDNet are required by the three-level model.The comparison measures FLOPs and average running time on 1280 × 720 images.Source: Table 5 · p. 7, §5.2 · p. 8
  • On images from other cameras in the CUHK dataset, the method restores fine details with fewer visual artifacts than single-image DPDNet.This generalization comparison is qualitative because the dataset has no all-in-focus ground truth.Source: §5.2 · p. 8, Figure 11 · p. 8

6. Conclusion

The conclusion presents inverse-kernel deblurring with KPAC as a lightweight framework that achieves state-of-the-art performance. It also notes that challenging blur cases may still defeat the method.

6. Conclusion

  • The method achieves state-of-the-art deblurring performance with much fewer parameters than previous methods.The conclusion attributes this performance to KPAC’s lightweight structure and says multiple KPAC blocks can be stacked.Source: §6 · p. 8
  • KPAC handles spatially varying inverse kernels using shared convolutional weights at different dilation rates and per-pixel scale attention.The framework uses a small number of atrous convolution layers and can be built by stacking KPAC blocks.Source: §6 · p. 8
  • The method may still fail on large-scale blur, irregular blur shapes, and bokeh with sharp boundaries.Source: §6 · p. 8

Supplementary Material

The supplementary material collects architecture details and additional analyses and experiments. Network and KPAC block architectures are presented in Tables 6 and 7.

Supplementary Material

  • The supplementary material covers the network architecture, proof of a main-paper equation, kernel upsampling, inverse-kernel sampling, blur detection, noise sensitivity, irregular blur, additional results, and dual-pixel extensions.Source: Supplementary Material · p. 10

A. Detailed Network Architecture

  • Tables 6 and 7 give the detailed architectures of the deblurring network and KPAC block, respectively.Table 7 uses AC to denote an atrous convolution layer.Source: A. Detailed Network Architecture · p. 10

B. Proof of ( 1

The supplementary proof establishes a relation between kernel upsampling and inverse-kernel upsampling, with scaling needed for normalization. The discussion supports Gaussian scale changes but cautions that arbitrary kernels may be modeled less accurately.

B. Proof of ( 1

  • Zero-padding the discrete Fourier transform implements spatial upsampling, which is mathematically equivalent to convolution with a sinc kernel.The supplementary proof defines frequency-domain zero padding for an image and relates it to spatial upsampling.Source: B. Proof of ( 1 · p. 10
  • The inverse kernel is computed from the blur kernel using Wiener deconvolution, with ϵ as a noise parameter.When ϵ = 0, the inverse kernel is equivalent to the direct inverse F^-1(1/F(k)).Source: B. Proof of ( 1 · p. 10

C. Discussion on Kernel Upsampling

  • Tables 6 and 7 present the detailed architectures of the deblurring network and KPAC block.Table 7 identifies AC as an atrous convolution layer.Source: Table 6 · p. 10, Table 7 · p. 11

B. Proof of ( 1

  • The proof establishes that upsampling a kernel and then inverting it is equivalent to upsampling its inverse kernel.The equivalence is shown for zero-padding-based upsampling in the supplementary proof.Source: B. Proof of ( 1 · pp. 10–11
  • Both sides of the upsampling relation scale total intensity by s^2, so the proof applies a 1/s^2 factor to normalize the kernels.Source: B. Proof of ( 1 · p. 11

C. Discussion on Kernel Upsampling

  • For Gaussian blur kernels, upsampling corresponds to increasing the kernel’s standard deviation.The supplementary discussion presents this as an observation about Gaussian kernels, a commonly used assumption in existing approaches.Source: C. Discussion on Kernel Upsampling · p. 11
  • For arbitrary non-Gaussian kernels, upsampling may not accurately model real-world blur-scale changes; KPAC’s attentions and other convolution layers are intended to address the gap.Source: C. Discussion on Kernel Upsampling · p. 11

D. Inverse Kernel-based Deconvolution for Spatially Varying Defocus Blur

The section tests approximating spatially varying deconvolution by combining results from inverse kernels of discrete sizes. Across the shown scale settings, the combinations closely approximate deconvolution at the target scale.

D. Inverse Kernel-based Deconvolution for Spatially Varying Defocus Blur

  • Eq. (5) approximation accuracies for result (d) were 98.68%, 97.84%, 98.96%, 95.37%, and 97.03% across the five tested scale triplets.Source: Figure 13 · p. 12
  • Eq. (5) modified approximation accuracies for result (e) were 98.55%, 97.87%, 99.10%, 96.12%, and 97.64% across the five tested scale triplets.Source: Figure 13 · p. 12
  • Although neither sampled inverse kernel matches the target blur size, their linearly combined deconvolution results are visually pleasing and almost equivalent to target-scale deconvolution.The examples use sampled factors s1 and s2 around target factor st, with s1 < st < s2.Source: D. Inverse Kernel-based Deconvolution… · pp. 11–12, Figure 13 · p. 12

E. Inverse Kernel Sampling for Atrous Convolution

The supplementary examples support replacing dense inverse-kernel sampling with sparse atrous sampling for deblurring in spatially smooth blurred regions. A modified version of Eq. (5) using dilated inverse kernels produces results almost equivalent to the original.

E. Inverse Kernel Sampling for Atrous Convolution

F. Blur Detection Using a Scale Attention Map

The scale attention map detects blur comparably to a recent defocus-map method, while noise augmentation improves noise robustness. The method is most suited to blur variation dominated by size and may not handle severely irregular shapes or strong highlights.

F. Blur Detection Using a Scale Attention Map

G. Sensitivity to Noise

  • Training with additive Gaussian noise in the 0%–3% range makes the model more robust to noise, with more consistent PSNRs around 25 dB.The comparison is against a model trained without noise augmentation.Source: G. Sensitivity to Noise · p. 14

H. Handling Irregular Blur

  • The network moderately handles unseen smartphone images that usually have small blurs, but may not properly handle severely irregular shapes or strong highlights, which are rare in training.Its design is more effective when most blur variation is in size; Figure 16 shows removal in both center and peripheral regions.Source: H. Handling Irregular Blur · p. 14, Figure 16 · p. 14

I. Additional Results

Dual-pixel input further improves the model and outperforms DPDNet with dual-pixel input by a large margin. Additional results also show failures on irregular bokeh that was not included in training.

I. Additional Results

  • Additional qualitative results are presented on the DPDD and CUHK blur detection datasets.Source: §I · p. 14
  • The model fails on bokeh with sharp boundaries and swirly bokeh, irregular blurs not contained in the training dataset.Source: Figure 17 · p. 15

J. Our Model Using Dual-Pixel Images

Loading 2108.09108v1…