PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Estimation from Salient Structure for Robust Motion Deblurring

Jinshan Pan, Risheng Liu, Zhixun Su, Xianfeng Gu

arXiv:1212.1073v2cs.CV

TL;DR

Large blur and complex structures make blind image deblurring and kernel estimation challenging, and incorrect kernels can make restored results unreliable. The paper selects reliable image structures and estimates kernels with sparsity, continuity, and noise suppression in view; it reports deblurring images with small and large blur while preserving image details.

  • Problem

    Large blur and complex structures can prevent state-of-the-art blind deblurring methods from obtaining correct kernels, making final results unreliable.

    Source: §1 · pp. 2–3

  • Method

    The method selects image structures adaptively for kernel estimation and uses structural information in an adaptive weighted regularization term for latent-image restoration.

    Source: §1 · p. 4, §6 · p. 34

  • Results

    The method deblurs images with both small and large blur kernels, especially when blurred images contain rich details, while preserving image details.

    Source: §6 · p. 34

  • Takeaways & Limitations

    The paper concludes that its structure-based kernel estimation and restoration approach can preserve image details across small- and large-blur cases.

    Source: §6 · p. 34

    Kernel estimation can fail when the blurred image is textureless or contains severe saturated pixels.

    Source: §6 · p. 34

Abstract

from arXiv · show

Blind image deblurring algorithms have been improving steadily in the past years. Most state-of-the-art algorithms, however, still cannot perform perfectly in challenging cases, especially in large blur setting. In this paper, we focus on how to estimate a good kernel estimate from a single blurred image based on the image structure. We found that image details caused by blurring could adversely affect the kernel estimation, especially when the blur kernel is large. One effective way to eliminate these details is to apply image denoising model based on the Total Variation (TV). First, we developed a novel method for computing image structures based on TV model, such that the structures undermining the kernel estimation will be removed. Second, to mitigate the possible adverse effect of salient edges and improve the robustness of kernel estimation, we applied a gradient selection method. Third, we proposed a novel kernel estimation method, which is capable of preserving the continuity and sparsity of the kernel and reducing the noises. Finally, we developed an adaptive weighted spatial prior, for the purpose of preserving sharp edges in latent image restoration. The effectiveness of our method is demonstrated by experiments on various kinds of challenging examples.

1. Introduction

Large blur and complex image structures make reliable kernel estimation difficult, undermining restoration. The paper selects reliable structures and estimates kernels designed to preserve sparsity and continuity while suppressing noise.

1. Introduction

  • Large blur and complex structures make useful sharp-edge selection difficult, leading to incorrect kernels and unreliable deblurred results.The introduction contrasts these cases with the performance of several state-of-the-art methods.Source: §1 · p. 3
  • An adaptive structure-selection method removes detrimental structures and selects reliable image structures for kernel estimation.The selected structures are intended to provide useful sharp edges based on image characteristics.Source: §1 · pp. 3–4
  • The proposed kernel estimator preserves blur-kernel sparsity and continuity while reducing noise.The paper identifies noisy kernels as another source of unreliable deblurred results.Source: §1 · p. 4
  • An adaptive regularization term combines final salient structures to guide latent-image restoration and preserve sharp edges.The introduction also reports restored images with fine textures.Source: §1 · pp. 3–4
  • On challenging images with tiny structures or large blur kernels, the method produces reliable, noiseless kernels with sparsity and continuity and high-quality restored images.The introduction reports these outcomes across challenging examples without giving quantitative comparisons.Source: §1 · p. 4

2. Related Work

Prior work estimates blur kernels using image statistics, kernel priors, and selected edges, but reported challenges include noisy estimates and difficulty with large blur. The section also reviews methods for non-blind restoration and their artifact or regularization issues.

2. Related Work

  • Kernel-estimation approaches using priors or spectral irregularities can yield noisy estimates, and hard thresholding can destroy kernels’ inherent structure.The section describes several prior-based methods before noting these kernel-estimate problems.Source: §2 · p. 5
  • Explicit edge-prediction methods select structures for kernel estimation, but some have difficulty with large blur or produce noisy results.The cited examples include edge detection followed by Radon-transform estimation and iterative edge selection.Source: §2 · p. 5
  • Other edge-selection and refinement methods adaptively select useful edges, but kernel continuity may be ignored and estimates can still contain noise.The section also notes that sparsity and continuity are not guaranteed in one learned-region approach.Source: §2 · p. 6
  • After kernel estimation, non-blind deconvolution methods include approaches that can suffer from noise and ringing artifacts, while later methods address ringing or guide latent-image restoration with image statistics.The section also notes that isotropic TV regularization can result in an unspecified issue.Source: §2 · p. 6

3. Kernel Estimation from Salient Structure

The section builds a reliable image structure for kernel estimation, then selects salient edges and analyzes why structure signals are more useful than texture signals.

3. Kernel Estimation from Salient Structure

  • The structure-extraction pipeline computes a main structure, enhances it with a shock filter, and selects salient edges for kernel estimation.As deblurring proceeds, the edge-selection thresholds decrease to include more edges and subtle structures.Source: §3 · p. 7, §3.1 · pp. 10–11

3.1.1. Analysis on Structure Selection Method

  • Texture contains fine-scale details and noise, and its shape is seriously damaged by blur, making a sharp texture signal difficult to recover.The authors explain that texture-based kernel estimates are therefore hard to obtain reliably.Source: §3.1 · p. 8, §3.1.1 · pp. 12–13

3.1. Extracting Reliable Structure

  • Model (3)'s adaptive weight penalizes flat areas and narrow strips while preserving useful structures for kernel estimation.Its kernel estimates and deblurred results are reported as significantly better than model (2)'s.Source: §3.1 · pp. 9–10
  • Model (3) produces significantly better kernel estimates than model (2) while retaining useful structures for kernel estimation.The authors also report that model (2) improves kernel accuracy but may cause staircasing.Source: §3.1 · pp. 8, 10

3.1.1. Analysis on Structure Selection Method

  • The structure component can be sharpened with a shock filter, whereas the texture component usually fails to support kernel estimation.The authors contrast this with step signals, which they report have succeeded in kernel estimation in prior work.Source: §3.1.1 · p. 12

3.2. Kernel Estimation

This section estimates kernels with a spatial term designed to suppress noise while preserving sparsity and continuity. The authors compare its kernel and deblurring results with alternative models.

3.2. Kernel Estimation

  • Noisy interim kernels can damage latent-image estimates and subsequently degrade later kernel estimates during refinement.The authors contrast this with model (10), whose spatial term removes noise from estimates that remain noisy under model (8).Source: §3.2 · pp. 13, 15, 17
  • The result in (c) outperforms the result in (b), and the results in (c) and (f) of our method are reported as the best.Source: §3.2 · p. 14
  • The spatial term C(k) counts kernel pixels with nonzero gradients and helps remove noise while preserving kernel structure.The authors say it discourages discontinuous points, promoting continuity while also making the kernel sparse.Source: §3.2 · p. 15
  • Model (10) is described as robust to noise and as preserving both sparsity and continuity of the kernel.The parameter µ controls kernel smoothness.Source: §3.2 · p. 15
  • The authors optimize model (11) with constrained iterative reweighted least squares for three iterations, then use alternating optimization for model (12).The inner IRLS system becomes a quadratic program solved with a dual active-set method.Source: §3.2 · p. 16

3.3. Interim Latent Image Estimation

The interim latent-image stage uses anisotropic total variation to guide restoration, focusing on recovering sharp edges from the blurred image.

3.3. Interim Latent Image Estimation

  • The restoration stage focuses on recovering sharp edges from the blurred image.Anisotropic TV guides this latent-image restoration.Source: §3.3 · p. 17
  • The authors solve the latent-image model with IRLS, computing weights from the image recovered in the previous iteration.They empirically run three IRLS iterations.Source: §3.3 · p. 17
  • The inner IRLS system uses 30 conjugate-gradient iterations.This is the reported inner-solver iteration count.Source: §3.3 · p. 17

3.4. Multi-scale Implementation Strategy

The implementation estimates kernels across a coarse-to-fine image pyramid, with the number of levels set according to the blur-kernel size.

3.4. Multi-scale Implementation Strategy

  • The algorithm uses a coarse-to-fine image pyramid for multi-scale kernel estimation.The number of pyramid levels is chosen based on kernel size.Source: §3.4 · p. 17
  • The coarsest pyramid level is selected so the kernel is around 3 to 7 pixels wide or high.The number of levels is adaptively determined by kernel size.Source: §3.4 · pp. 17–18
  • At each pyramid level, the algorithm downsamples the blurred image, estimates the kernel, then estimates the latent image.It performs a few iterations at each level.Source: §3.4 · p. 18

3.5. Analysis on Kernel Estimation

The analysis evaluates structure selection and kernel estimation separately: salient-edge selection improves kernel estimates, while model (10) is reported to reduce noise and preserve blur-kernel continuity.

3.5.1. Effectiveness of the Proposed Structure Selection Method

  • For the blur kernels in Fig. 7, Figure 8 compares estimation accuracy using SSDE; the proposed method outperforms the other selection methods in that comparison.Source: §3.5.1 · pp. 19–20, Figure 8 · p. 20
  • Without salient edges ∇S, kernel-estimate SSDE values increase over iterations, whereas results using ∇S are better.The comparison uses an example from the “im02 ker08” test case in.Source: §3.5.1 · p. 20
  • Among the runs using ∇S, model (3) improves kernel-estimate quality, accuracy, and convergence over omitting that model.The authors compare the cyan and dashed-green curves in Figure 9.Source: §3.5.1 · pp. 20, 22
  • The proposed salient edges perform comparably to salient edges extracted from ground-truth images.The authors compare the cyan curve with the dotted-black curve in Figure 9.Source: §3.5.1 · pp. 21–22

3.5.2. Effectiveness of the Proposed Kernel Estimation Model

  • Model (10) improves kernel-estimation accuracy in SSDE and removes noise while preserving blur-kernel continuity.Figure 10 presents the kernel-estimation comparison.Source: §3.5.2 · p. 22

4. Final Latent Image Estimation

The final latent-image model uses predicted structure to guide restoration with spatially varying smoothness. The algorithm estimates a kernel and then restores the latent image.

4. Final Latent Image Estimation

  • Model (13) may cause stair-casing and destroy textures; adaptive regularization terms are proposed to address this while preserving edges.The approach is inspired by prior edge-preserving regularization.Source: §4 · p. 22
  • Model (14) uses predicted structure to vary smoothness spatially, contributing to edge preservation in latent-image restoration.The smoothness requirement depends on salient edges ∇S.Source: §4 · p. 23
  • The solver uses three IRLS iterations with gradient-based weights and 100 conjugate-gradient iterations in each inner system.The weights are assigned for ∂xI and ∂yI; the supplied formula specifies wy.Source: §4 · p. 23
  • The completed algorithm takes a blurred image and kernel size, estimates the kernel, then estimates and outputs the latent image.Source: §4 · p. 24

5. Experiments

Experiments evaluate deconvolution, kernel estimation, and image restoration on synthetic and challenging blurred images, including textured, large-blur, outlier, and saturated-region cases. They also report runtime comparisons and assess the effects of structure selection and adaptive spatial weighting.

5.1. Experimental Results and Evaluation

  • For Fig. 11, our method achieves higher PSNR for the restored image and lower SSDE for the kernel estimate.Table 1 compares SSDE and PSNR for the kernel estimates and restored images, respectively.Source: §5.1 · p. 25, Table 1 · p. 26
  • On a real captured image, our method improves kernel estimation and latent-image restoration, while the result without structure-selection model (3) is lower quality.The comparison reports noise in other methods’ deblurred results and in Xu and Jia’s kernel estimate.Source: §5.1 · p. 26, Figure 12 · p. 27
  • For a 45 × 45 blur kernel, our method outperforms the compared methods in kernel estimation and latent-image restoration; its adaptive weighted spatial prior preserves sharper edges and finer textures.Source: §5.1 · pp. 26, 28
  • For large blur, the approach generates a better kernel estimate and a visually comparable deblurred result; in another large-blur example, it provides a clearer image with finer textures.The estimated blur kernel size is 53 × 53.Source: §5.1 · p. 29, Figure 15 · p. 30
  • On the synthetic dataset, our method provides more reliable deconvolution results than the compared methods.The evaluation uses cumulative histograms of deconvolution error ratios.Source: §5.1 · pp. 29–30
  • Figure 16 presents a cumulative histogram of deconvolution error ratios across test examples.Source: Figure 16 · p. 31

5.2. Computational Cost

  • Our Matlab implementation estimates a 27 × 27 kernel from a 255 × 255 image in about 2 minutes (Intel Xeon CPU@2.53GHz, 12GB RAM), versus about 7, 3, and 4 minutes for methods, respectively.The C++ implementation of [2] takes about 50 seconds; compared with, our method needs more computational time due to involving non-convex models in kernel estimation.Source: §5.2 · p. 31

5.3. Handling Blurred Images with Outliers

  • For blurred images with non-uniformly distributed outliers, the kernel-estimation method estimates a blur kernel from a patch without obvious outliers, then applies non-blind deblurring for restoration.Figures 17 and 18 further demonstrate the kernel-estimation method’s effectiveness.Source: §5.3 · pp. 32–33
  • On a noisy blurred example, our estimated result is comparable with, and on a saturated image our method provides a reliable kernel from a cropped region.The restored image still shows visual ringing from saturated areas, and model (14) is not robust to outliers.Source: §5.3 · p. 33

6. Conclusion and Discussion

The paper presents a kernel-estimation algorithm that removes detrimental structures, preserves kernel characteristics, and uses structural information to guide latent-image restoration. It reports deblurring small and large kernels, especially for richly detailed images, while identifying textureless or saturated images and spatially varying blur as limitations.

6. Conclusion and Discussion

  • A self-adaptive method removes structures that may harm kernel estimation, while the kernel model removes noise and preserves kernel characteristics.The model's stated kernel characteristics include continuity and sparsity, reducing adverse effects from incorrectly chosen structures.Source: §6 · p. 34
  • In final deconvolution, structural information guides an adaptive weighted regularization term for latent-image restoration, which preserves image details.Source: §6 · p. 34
  • The algorithm deblurs images with both small and large blur kernels, especially when the blurred images contain rich details.The authors report extensive testing of the algorithm.Source: §6 · p. 34
  • The method can fail on textureless images, where salient edges are unavailable, or on images with severe saturation that breaks the convolution model's linearity.Uniformly distributed saturated areas may be selected for kernel estimation because of their saliency.Source: §6 · p. 34
  • Spatially varying blur is not properly handled by the method and is left for future work.Source: §6 · pp. 34–35

Appendix A. Relationship to the Structure Extraction Method [36]

The appendix relates the texture-removal method in to relative total variation (RTV), then distinguishes the paper’s structure-extraction method by its use of an adaptive smoothness weight.

Appendix A. Relationship to the Structure Extraction Method [36]

  • [36] used local information for texture removal and introduced relative total variation (RTV) as an adaptive regularization term.RTV is defined with structure S and a spatial-affinity weighting function g.Source: Appendix A · p. 35
  • When g is a scalar weight, the effect of Eq. (A.1) is similar to that of 1/r(x) or w(x).Source: Appendix A · p. 35
  • The paper also considers the variation form of Eq. (4) as a special regularizer for extracting image structures, and distinguishes its method from by using w(x) as an adaptive smoothness weight.The authors state that this approach would extract useful structures for kernel estimation.Source: Appendix A · p. 35
Loading 1212.1073v2…