PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Restoration Using Joint Statistical Modeling in Space-Transform Domain

Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, Wen Gao

arXiv:1405.3173v1cs.MMcs.CV

TL;DR

The paper addresses image restoration as an ill-posed inverse problem where local smoothness and nonlocal self-similarity are difficult to characterize jointly. It introduces JSM in an adaptive hybrid space-transform domain and formulates a corresponding optimization method. Experiments across inpainting, deblurring, and mixed-noise removal report improved restoration quality, including about 2.7 dB PSNR and 0.016 FSIM over BPFA on average.

  • Problem

    Image restoration requires effective modeling of both local smoothness and nonlocal self-similarity, while traditional nonlocal regularization can inaccurately characterize self-similarity through weighted graphs.

  • Method

    The paper combines local statistical modeling in the space domain with nonlocal statistical modeling in the transform domain through JSM and solves the resulting regularized inverse problem.

  • Results

    The proposed algorithm achieves significant improvements over current state-of-the-art schemes across image inpainting, deblurring, and mixed Gaussian plus salt-and-pepper noise removal; in one evaluation, it improves over BPFA by about 2.7 dB PSNR and 0.016 FSIM on average.

  • Takeaways & Limitations

    Joint statistical modeling provides a unified restoration strategy that simultaneously exploits local consistency and nonlocal image repetition.

Abstract

from arXiv · show

This paper presents a novel strategy for high-fidelity image restoration by characterizing both local smoothness and nonlocal self-similarity of natural images in a unified statistical manner. The main contributions are three-folds. First, from the perspective of image statistics, a joint statistical modeling (JSM) in an adaptive hybrid space-transform domain is established, which offers a powerful mechanism of combining local smoothness and nonlocal self-similarity simultaneously to ensure a more reliable and robust estimation. Second, a new form of minimization functional for solving image inverse problem is formulated using JSM under regularization-based framework. Finally, in order to make JSM tractable and robust, a new Split-Bregman based algorithm is developed to efficiently solve the above severely underdetermined inverse problem associated with theoretical proof of convergence. Extensive experiments on image inpainting, image deblurring and mixed Gaussian plus salt-and-pepper noise removal applications verify the effectiveness of the proposed algorithm.

I. INTRODUCTION

Image restoration is an ill-posed inverse problem whose quality depends on effective image priors. The paper proposes joint statistical modeling to combine local smoothness and nonlocal self-similarity in an adaptive space-transform framework.

  • Problem: Image restoration reconstructs a high-quality image from degraded observations in an ill-posed linear inverse-problem framework.The model covers denoising, deblurring, inpainting, and related degradation settings.
  • Scope: The paper focuses experimentally on image inpainting, image deblurring, and image denoising applications.The inverse-problem formulation represents these tasks through different degradation operators.
  • Prior limitations: Classical regularizers exploit local smoothness but can smear details, while nonlocal methods improve edges yet may suffer from weighted-graph inaccuracies.These limitations motivate modeling both local and nonlocal image statistics more effectively.
  • Proposed approach: The proposed JSM combines local smoothness and nonlocal self-similarity in a unified statistical model for more reliable and robust restoration.It uses local statistical modeling in the space domain and nonlocal statistical modeling in a transform domain.
  • Proposed approach: JSM is formulated within a regularization-based minimization framework for solving image inverse problems.The framework treats image priors as regularization terms controlled by a regularization parameter.

A. Local Statistical Modeling for Smoothness in Space Domain

The paper models image structure statistically at both local pixel level and nonlocal block level, using gradient distributions and 3D transform coefficients to support restoration.

  • Local smoothness: Local smoothness is modeled through neighboring-pixel gradients in the two-dimensional space domain.Vertical and horizontal finite-difference filters capture locally similar intensities and derivatives near zero.
  • Local smoothness: Horizontal gradients of Lena have a sharp distribution concentrated near zero, motivating statistical modeling of local image smoothness.Figure 2 presents the gradient image and its histogram.
  • Local smoothness: Generalized Gaussian distributions provide a family for modeling gradient statistics, including Gaussian, Laplacian, and hyper-Laplacian cases.The shape parameter determines the distributional form.
  • Local smoothness: The selected Laplacian model yields a convex local statistical regularizer that supports efficient optimization.The paper relates this formulation to anisotropic total variation while using it to characterize smoothness.
  • Nonlocal self-similarity: Nonlocal self-similarity is modeled by stacking similar image blocks into 3D arrays, applying an orthogonal 3D transform, and modeling the resulting coefficients.The transform-coefficient histogram is sharp and concentrated near zero, so the paper uses a Laplacian model for NLSM.
  • Nonlocal self-similarity: The proposed NLSM uses a fixed number of best-matching blocks, supports inverse reconstruction by block averaging, and is incorporated into image inverse problems.The method is presented as simpler and more robust to similarity criteria than threshold-based block selection.

C. Joint Statistical Modeling (JSM)

JSM combines local smoothness in the two-dimensional space domain with nonlocal self-similarity in a three-dimensional transform domain. The resulting regularized inverse problem is solved by a Split Bregman algorithm whose convergence is established under convexity conditions.

  • C. Joint Statistical Modeling (JSM): JSM combines pixel-level local statistical modeling in the space domain with block-level nonlocal statistical modeling in the transform domain.The nonlocal model uses distributions of transform coefficients obtained from 3D arrays formed by stacking similar image blocks.
  • C. Joint Statistical Modeling (JSM): The JSM regularizer is incorporated into a new image-restoration minimization functional with observation, local-prior, and nonlocal-prior constraints.The control parameters τ and λ weight the local and nonlocal constraints.
  • C. Joint Statistical Modeling (JSM): Split Bregman Iteration transforms the optimization into separate u, w, and x sub-problems that can be solved iteratively.The complete algorithm is presented through sequential variable updates and a stopping criterion.
  • C. Joint Statistical Modeling (JSM): The proposed algorithm converges to a solution of the JSM restoration objective because it is an instance of Split Bregman Iteration with closed, proper, convex functions.The proof also uses a full-column-rank matrix condition.
  • C. Joint Statistical Modeling (JSM): The u-subproblem has a closed-form solution, while inpainting and deblurring exploit the structures of their degradation operators for efficient computation.Inpainting costs O(N), and deblurring uses diagonalization with FFT-based transforms.
  • C. Joint Statistical Modeling (JSM): Residual coefficients are modeled as independently distributed, zero-mean variables with iteration-dependent variance, with generalized Gaussian distributions fitting the observed residuals.The residual distribution is reported as suitable for generalized Gaussian characterization, and its variance is estimated at different iterations.

D. Summary of Proposed Algorithm

The paper summarizes the derived JSM restoration procedure in a complete algorithmic description. The final algorithm is presented in Table II alongside the experimental image set.

  • D. Summary of Proposed Algorithm: The proposed image-restoration algorithm is assembled from the preceding derivations and presented in a complete description in Table II.The derivation addresses all three Split Bregman sub-problems before presenting the final algorithm.
  • D. Summary of Proposed Algorithm: The experimental test images used to evaluate the method are shown in Fig. 7.The paper identifies Fig. 7 as the display of all experimental images.

IV. EXPERIMENTAL RESULTS

Experiments evaluate the proposed algorithm on image inpainting, deblurring, and mixed Gaussian plus salt-and-pepper noise removal using objective and visual-quality metrics. PSNR measures objective image quality, while FSIM assesses visual quality.

  • IV. EXPERIMENTAL RESULTS: The experiments cover image inpainting, image deblurring, and mixed Gaussian plus salt-and-pepper noise removal.The evaluations were conducted in Matlab on a specified desktop system.
  • IV. EXPERIMENTAL RESULTS: PSNR is used for objective image quality, whereas FSIM is used to evaluate visual quality.PSNR is reported in dB, and FSIM values lie between 0 and 1, with higher values indicating better visual quality.
  • IV. EXPERIMENTAL RESULTS: For color images, results are computed from the luminance component while leaving chrominance components unchanged.The paper notes that only part of the experimental results is shown.

A. Image Restoration from Partial Random Samples

On restoration from partial random samples, the proposed method achieves the strongest reported objective and visual scores across the tested cases. Visual comparisons attribute this performance to accurate recovery of edges and textures with fewer artifacts.

  • A. Image Restoration from Partial Random Samples: The proposed method achieves the highest PSNR and FSIM scores in every case reported in Table III.The comparison includes SALSA, SKR, MCA, BPFA, and FoE across three color images and multiple sampling ratios.
  • A. Image Restoration from Partial Random Samples: 2.7 dB PSNR and 0.016 FSIM are the proposed algorithm’s average improvements over BPFA, the second-best algorithm.For House at Ratio=20%, the reported average improvements are 4.2 dB PSNR and 0.02 FSIM.
  • A. Image Restoration from Partial Random Samples: At Ratio=20%, the proposed method achieves 27.54 dB/0.9264 on Barbara and 33.28 dB/0.9631 on Foreman.These figures are reported as PSNR/FSIM values in the visual comparisons.
  • A. Image Restoration from Partial Random Samples: The proposed JSM recovers edges and textures while suppressing noise-caused artifacts, whereas competing methods show blur, incorrect textures, or striped artifacts.The visual assessment reports that all methods perform well on smooth regions, but differ in texture and artifact recovery.
  • A. Image Restoration from Partial Random Samples: For text removal, the proposed algorithm reaches 37.99 dB/0.9899 on Barbara and 34.45 dB/0.9770 on Parthenon.The corresponding comparisons include SKR and FoE restoration results.

B. Image Restoration for Text Removal

The proposed algorithm is evaluated for text removal from color images using a known text mask. It achieves the highest reported PSNR and FSIM values and visually recovers edges and textures more accurately than competing methods.

  • Text removal treats the degradation mask as a known text region and aims to infer the original image by removing that region.
  • The proposed algorithm achieves the highest PSNR and FSIM values in all tested text-removal cases.
  • Visual comparisons show more accurate edges and textures, with better visual quality than SKR, FoE, and BPFA.

C. Image Deblurring

The JSM approach is evaluated for deblurring under multiple blur kernels and noise settings, including a large-blur case. It outperforms the compared methods quantitatively and produces cleaner, sharper reconstructions with almost unnoticeable ringing artifacts.

  • The experiments use 9×9 uniform, Gaussian, and motion blur kernels with added Gaussian noise, comparing JSM against SALSA, SA-DCT, and BM3D.
  • 31.03 dB/0.9394: the proposed method versus SALSA at 30.30 dB/0.9300 and BM3D at 28.73 dB/0.8959 on Butterfly with 9×9 uniform blur.
  • 32.18 dB/0.9610: the proposed method versus SALSA at 30.32 dB/0.9518 and BM3D at 30.61 dB/0.9342 on Leaves with Gaussian blur.
  • JSM significantly outperforms competing methods in PSNR and FSIM across all three tested blur-kernel types.
  • Visual results show cleaner, sharper image edges and textures with almost unnoticeable ringing artifacts, attributed to simultaneous local and nonlocal regularization.
  • 26.51 dB/0.8724: the proposed method versus AKTV at 25.19 dB/0.8109 on Cameraman with 19×19 uniform blur and BSNR=40.

D. Mixed Gaussian plus Salt-and-Pepper Noise Removal

The paper addresses images corrupted by both Gaussian and salt-and-pepper noise by converting the problem into restoration from partial samples before applying the proposed method. Across benchmark images, the method outperforms FTV and IFASDA in quantitative and visual results.

  • Mixed Gaussian and salt-and-pepper noise combines distinct degradation processes that single-noise denoising methods do not handle well.
  • Adaptive median filtering identifies a mask, transforming mixed-noise removal into image restoration from partial random samples with Gaussian noise.
  • The proposed method achieves the highest PSNR and FSIM in every tested case, surpassing IFASDA by average improvements of 1.8 dB and 0.01, respectively.
  • FTV suppresses noise but oversmooths details, whereas IFASDA recovers structures but introduces artifacts in smooth regions.
  • 31.04 dB/0.9383: the proposed method versus FTV at 25.40 dB/0.8728 and IFASDA at 27.45 dB/0.9129 on Barbara with r = 50%.

E. Parameter Optimization

The parameter strategy reduces optimization of four parameters to a one-parameter search. Experiments identify a parameter setting that balances noise suppression and detail preservation and can be transferred across test images within a case.

  • The proposed algorithm has four parameters, but their optimization is simplified to searching over one parameter.
  • The model uses a one-to-six ratio between μ1 and μ2 to balance local smoothness modeling and nonlocal self-similarity modeling.
  • A rule of thumb for τ and λ is derived from the Gaussian-noise standard deviation, with relationships linking the four parameters to μ.
  • The remaining μ optimization is obtained by simply searching a set of values for each image-processing application.
  • The PSNR-evolution and visual-comparison figures assess how μ affects restoration quality across motion-blur test settings.
  • An optimal μ balances image-noise suppression against image-detail preservation; values that are too small leave noise, while values that are too large lose details.
  • For each case, the optimal μ is almost identical across test images, allowing one image to determine μ for the others.

F. Algorithm Complexity and Computational Time

The proposed algorithm’s main computational burden is the x-subproblem involving 3D transforms, while initialization substantially affects total runtime. Using SKR results as initialization reduces computation to about one seventh of the original time.

  • The x-subproblem dominates complexity because each 3D array requires 3D and inverse 3D transforms.
  • For image House, each iteration takes about 1.25 s on a 3.25GHz CPU.
  • With degraded-image initialization, 100 iterations at Ratio=80% take about 130 s, while 400 iterations at Ratio=20% take about 510 s.
  • Using SKR results instead of degraded images as initialization reduces computational time to about one seventh of the original.

G. Algorithm Convergence and Robustness

The proposed method is evaluated for convergence and robustness under different initializations. Across Lena and Barbara inpainting tests, it produces nearly identical results from multiple starting points and is reported to have nice convergence properties.

  • The convergence and robustness tests use image inpainting at Ratio=30% on Lena and Barbara.
  • Initializations from SALSA, FoE, SKR, and BPFA are used to evaluate PSNR evolution across iterations.
  • The method provides almost the same results when starting from various initializations.
  • Experiments across image inpainting, deblurring, and mixed Gaussian plus salt-and-pepper noise removal show significant improvements over current state-of-the-art schemes and nice convergence properties.
Loading 1405.3173v1…