PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Deblurring and Super-resolution by Adaptive Sparse Domain Selection and Adaptive Regularization

Weisheng Dong, Lei Zhang, Guangming Shi, Xiaolin Wu

arXiv:1012.1184v1cs.CVcs.MM

TL;DR

The paper addresses sparse-representation image restoration, where dictionary selection is critical and smoothing can erase fine details. It proposes ASDS-AReg, combining adaptive sparse-domain selection with adaptive regularization, and reports better results than many state-of-the-art methods.

  • Problem

    Dictionary Φ is a critical issue in sparse-representation modeling, while restoration algorithms can smooth fine image details.

  • Method

    The ASDS-AReg approach adaptively selects sparse domains from example patches and introduces adaptive regularization using piecewise autoregressive models and additional regularization terms.

  • Results

    The proposed ASDS-AReg approach outperforms many state-of-the-art methods on image restoration in both evaluated dimensions.

  • Takeaways & Limitations

    Adaptive sparse-domain selection improves sparse modeling and consequently image-restoration results, while adaptive regularization further improves reconstructed-image quality.

Abstract

from arXiv · show

As a powerful statistical image modeling technique, sparse representation has been successfully used in various image restoration applications. The success of sparse representation owes to the development of l1-norm optimization techniques, and the fact that natural images are intrinsically sparse in some domain. The image restoration quality largely depends on whether the employed sparse domain can represent well the underlying image. Considering that the contents can vary significantly across different images or different patches in a single image, we propose to learn various sets of bases from a pre-collected dataset of example image patches, and then for a given patch to be processed, one set of bases are adaptively selected to characterize the local sparse domain. We further introduce two adaptive regularization terms into the sparse representation framework. First, a set of autoregressive (AR) models are learned from the dataset of example image patches. The best fitted AR models to a given patch are adaptively selected to regularize the image local structures. Second, the image non-local self-similarity is introduced as another regularization term. In addition, the sparsity regularization parameter is adaptively estimated for better image restoration performance. Extensive experiments on image deblurring and super-resolution validate that by using adaptive sparse domain selection and adaptive regularization, the proposed method achieves much better results than many state-of-the-art algorithms in terms of both PSNR and visual perception.

I. Introduction

Image restoration is an ill-posed inverse problem whose quality depends on suitable image priors and sparse domains. The paper addresses limited dictionary adaptivity by selecting local sub-dictionaries and adding adaptive regularization for deblurring and super-resolution.

  • Motivation: Image restoration reconstructs a high-quality image from degraded measurements, but the inverse problem is generally non-unique.Natural-image priors are therefore used to regularize the reconstruction.
  • Motivation: TV regularization favors piecewise-constant structures and tends to smooth fine image details.Later methods improve TV models to preserve edges more effectively.
  • Sparse representation: Sparse representation models natural images with dictionaries whose coefficients are mostly zero, but l0-minimization is NP-hard.The convex l1 alternative is widely used for sparse coding, and reweighted l1 regularization can improve restoration results.
  • Sparse representation: Analytically designed dictionaries are fast but lack adaptivity to local image structures, while redundant learned dictionaries can produce unstable decompositions and visual artifacts.This motivates selecting compact sub-dictionaries matched to individual patches.
  • Proposed approach: ASDS learns compact PCA sub-dictionaries from clustered example patches and adaptively selects the most relevant sub-dictionary for each patch.The selected local sparse domain is intended to represent patches more accurately than a universal dictionary.
  • Proposed approach: The method adds adaptively selected AR models and non-local self-similarity as regularizers, estimates local sparsity parameters, and solves the l1 problem with iterative shrinkage.The non-local term is described as helpful for preserving edge sharpness and suppressing noise.
  • Results: Experiments on deblurring and super-resolution report that ASDS-AReg reconstructs image details and outperforms many state-of-the-art methods in PSNR and visual perception.The framework combines adaptive sparse-domain selection with adaptive regularization.

II. Related Works

Related work motivates adaptive sparse modeling by identifying limitations in universal dictionaries and conventional regularization, while highlighting autoregressive and non-local priors as useful image-restoration tools.

  • Natural images can be represented using sparse structural primitives such as edges and line segments.
  • Standard sparse coding commonly replaces non-convex l0 minimization with convex l1 or weighted l1 regularization.
  • Dictionary choice is central because image contents vary across images and local patches.
  • Universal over-complete dictionaries may include irrelevant atoms, reducing sparse-coding efficiency and representation accuracy.
  • TV regularization has limited flexibility for local structures and can produce over-smoothed results, motivating alternative priors.
  • The paper combines learned AR regularization with an NL self-similarity term to regularize image restoration.

III. Sparse Representation with Adaptive Sparse Domain Selection

The ASDS framework learns compact sub-dictionaries offline and assigns the best-fitting one to each local patch, then reconstructs the image through sparse coding and iterative updates.

  • ASDS learns a series of compact sub-dictionaries and adaptively assigns each local patch its sparse domain.
  • Each image patch is extracted from the image, represented over a selected orthonormal sub-dictionary, and reconstructed by averaging patches.
  • The inverse-restoration objective combines degraded-image fidelity with a weighted l1 sparse-representation penalty.

A. Learning the sub-dictionaries

The method learns compact sub-dictionaries from clustered, edge-focused training patches, using PCA to retain the most useful components for each local pattern group.

  • Patches with intensity variance below the threshold Δ are excluded, retaining meaningful patches with edge structures for dictionary learning.
  • Each cluster is assigned a compact sub-dictionary, allowing the most suitable dictionary to be selected for each local patch.
  • Training patches are high-pass filtered and clustered with K-means so sub-datasets capture distinctive local image patterns.
  • PCA constructs each sub-dictionary from principal components of its cluster covariance matrix.
  • The retained component count r balances reconstruction error against l1 sparsity, with the optimal r selected by minimizing their combined objective.
  • Figure 1 compares cluster centroids with the first eight atoms of their corresponding learned sub-dictionaries.

B. Adaptive selection of the sub-dictionary

Adaptive selection uses projected patch–centroid similarity, while adaptive reweighting estimates coefficient variability from non-local patches to improve sparsity regularization.

  • B. Adaptive selection of the sub-dictionary: The method forms centroid–sub-dictionary pairs and assigns each patch the dictionary associated with its best-matching centroid.
  • B. Adaptive selection of the sub-dictionary: Projected centroid distances provide a more robust selection criterion than directly comparing noisy patch estimates.
  • B. Adaptive selection of the sub-dictionary: The selected dictionaries and image estimate are alternately updated until the adaptive sparse-domain process converges.
  • C. Adaptively reweighted sparsity regularization: Adaptive reweighting estimates local sparsity through a MAP formulation under Gaussian noise and Laplacian coefficient priors.
  • C. Adaptively reweighted sparsity regularization: Coefficient-weight estimates use the standard deviation of representations from non-local similar patches.
  • C. Adaptively reweighted sparsity regularization: The proposed adaptive reweighting improved average performance by about 0.2dB over the method in for deblurring and super-resolution within ASDS.

IV. Spatially Adaptive Regularization

The method adds spatially adaptive regularization to ASDS by selecting locally fitted AR models and exploiting non-local image redundancy. AR models capture local structure, while similar patches provide complementary enhancement.

  • IV. Spatially Adaptive Regularization: ASDS-AReg extends adaptive sparse-domain selection with two adaptive regularizers: local AR modeling and non-local similarity.The AR term exploits local stationarity, while non-local redundancy supplies complementary information from repetitive structures.
  • Training the AR models: AR models are learned separately from clustered high-quality training patches and used to regularize corresponding image patches.Each cluster produces one AR model, yielding a set of models for adaptive regularization.
  • Non-local similarity regularization: Non-local regularization searches the image for similar patches and uses their weighted predictions to enhance each local patch.The method relies on repetitive structures and combines this term with the local AR constraint.
  • Training the AR models: Each AR model predicts a patch’s central pixel from neighboring pixels within a square support window.The experiments use a 3×3 support window, corresponding to an AR model of order 8.

B. Adaptive selection of the AR model for regularization

The AR model is selected adaptively for each patch using an estimated patch representation, while non-local similarity supplies a complementary regularization signal from repetitive image structures.

  • Adaptive selection of the AR model for regularization: For each patch, the method selects the AR model using the same adaptive procedure used for sub-dictionary selection.The selection uses an estimate of the patch and compares it with candidate representations.
  • Adaptive AR regularization: The selected AR model predicts the patch’s central pixel from neighboring pixels within the patch.The prediction error is incorporated into the sparse-representation objective as a local regularization term.
  • Non-local similarity regularization: Non-local similarity identifies patches with small estimated distance from the current patch and uses weighted neighboring-patch predictions.The weights are normalized exponential functions of patch distances, with h controlling their concentration.
  • Combined regularization: The combined objective couples observation fidelity, local AR regularization, non-local similarity, and sparse representation.The non-local term is added to the ASDS-based sparse model as a complementary adaptive regularizer.

V. Summary of the Algorithm

The algorithm alternates between solving a reweighted l1 problem and updating the adaptive sparse domain, AR models, and non-local weights until convergence or a maximum iteration count.

  • Objective function: The full objective combines degradation fidelity, local AR regularization, non-local similarity regularization, and a weighted l1 sparsity penalty.These terms respectively fit the observation, enforce local stationarity, exploit non-local redundancy, and promote sparse coefficients.
  • Optimization: The resulting optimization is a reweighted l1-minimization problem solved with an iterative shrinkage algorithm.The paper outlines this solver in Algorithm 1.
  • Initialization: Initialization uses wavelet shrinkage to estimate the image, then selects sub-dictionaries, AR models, and non-local weights.The corresponding matrices and algorithm parameters are initialized before iterations begin.
  • Iteration: Each iteration updates the image estimate, solves for sparse coefficients with soft thresholding, and reconstructs the image by averaging patches.The adaptive domain and regularization matrices are refreshed every P iterations.
  • Stopping rule: Iterations stop when the normalized estimate change falls below e or when Max_Iter is reached.The scalar e controls convergence, while Max_Iter caps the number of iterations.
  • Practical settings: The method uses an empirically fixed r=4.7 because determining its optimal value for every local patch is difficult.Adaptive components are updated only every P iterations to reduce computational cost.

VI. Experimental Results

Experiments evaluate the proposed approach for image deblurring and super-resolution using learned patch dictionaries, adaptive regularization, and comparisons with established restoration methods.

  • Training datasets: Training uses two datasets, each containing five high-quality images, to assess robustness to the choice of training images.The datasets provide patches for learning dictionaries and AR models.
  • Training data construction: The training procedure randomly crops 727,615 patches of size 7×7 from each training-image set after excluding smooth patches.Patches are selected using a variance threshold Var(si)>Δ with Δ=16.
  • Deblurring experiments: Deblurring tests use Gaussian and 9×9 uniform blur kernels with additive Gaussian white noise, comparing against five recent methods.The baselines include wavelet shrinkage, two TV methods, l0-norm sparsity, and BM3D deblurring.
  • Super-resolution experiments: Super-resolution tests apply a truncated 7×7 Gaussian blur with standard deviation 1.6 before down-sampling by a factor of 3.The proposed method is compared with four state-of-the-art methods, with iterative back-projection added for one baseline’s deblurring limitation.
  • Evaluation protocol: The evaluation reports PSNR and SSIM on luminance, while chromatic components use bicubic interpolation for color images.The proposed method is tested in ASDS, ASDS-AR, and ASDS-AR-NL variants.

C. Experimental results on de-blurring

Deblurring experiments show that adaptive sparse-domain selection and adaptive regularization improve reconstruction quality, especially edge preservation and suppression of ringing, noise, and other artifacts. Across test images and blur settings, the proposed method generally achieves strong PSNR and SSIM performance against competing methods.

  • Ablation and visual results: ASDS is effective for deblurring, while combining adaptive regularization further improves results by reducing ringing artifacts around edges.The reported comparisons found nearly identical results for TD1 and TD2, with further improvement after adding AR and non-local regularization.
  • Quantitative results: 0.50 dB and 0.40 dB are the average PSNR improvements of ASDS-AR-NL-TD2 over BM3D for uniform blur at the two tested noise levels.For Gaussian blur, the corresponding improvements over BM3D are 0.15 dB and 0.18 dB.
  • Quantitative results: The proposed method achieves the highest SSIM index in the reported deblurring comparisons.This result is reported alongside the average PSNR improvements for uniform and Gaussian blur kernels.
  • Visual comparisons: The proposed method produces sharper edges and more recovered fine structures than competing super-resolution methods on reconstructed images.The paper attributes the improved edge preservation to adaptive sparse-domain selection and adaptive regularizations.
  • Robustness to noise: The method remains robust to noise, suppressing noise while reconstructing fine edges in noisy super-resolution images.Competing methods are described as producing noise-caused edge artifacts or over-smoothed results in the noisy case.

E. Experimental results on a 1000-image dataset

Experiments on a 1000-image natural-image dataset evaluate robustness across deblurring and super-resolution settings, dictionary-class counts, and patch sizes. The proposed method consistently outperforms competing methods, while performance is relatively insensitive to class count and patch size, with smaller patches producing artifacts in smooth regions.

  • Dataset: The dataset contains 1000 natural images with varied contents, including images collected from Flickr and the Berkeley Segmentation Database.A 256×256 sub-image rich in edge and texture structures was cropped from each image for testing.
  • Deblurring: The proposed method constantly outperforms competing methods in the large-scale deblurring experiments.For uniform blur, its average PSNR gain over BM3D reaches 0.85 dB; for non-truncated Gaussian blur, the gain reaches 0.19 dB.
  • Super-resolution: The proposed method performs constantly better than competing methods in the super-resolution experiments.The comparison uses average PSNR and SSIM values together with PSNR-gain distributions over methods and.
  • Class-count robustness: 100, 200, and 400 dictionary and AR-model classes produce very similar deblurring and super-resolution performance.The authors interpret this result as robustness to the number of classes.
  • Patch-size analysis: 3×3, 5×5, and 7×7 patch sizes produce similar PSNR and SSIM results, but smaller patches create artifacts in smooth regions.The implementation adopts 7×7 patches after these comparisons.

F. Discussions on the computational cost

The proposed iterative shrinkage algorithm has parallelizable patch-coding steps and precomputable components, but its runtime remains iterative and nontrivial. Reported acceleration strategies can further reduce computational cost.

  • Acceleration: Patch-coding steps can run in parallel, and deblurring matrix calculations can use FFT instead of direct matrix computation.These implementation choices are described as ways to speed up the algorithm.
  • Acceleration: Sub-dictionary and AR-model updates are performed every P=100 iterations to reduce computational overhead.Their updates require nearest-neighbor searches over the patches.
  • Convergence: The algorithm converges in 700~1000 iterations in most cases.The method is implemented as an iterative shrinkage algorithm.
  • Runtime: For a 256×256 image, the proposed algorithm requires about 2~5 minutes for deblurring and super-resolution on the reported Matlab implementation.The measurement uses an Intel Core2 Duo 2.79G PC under Matlab R2010a.
  • Acceleration: Several accelerating techniques can further reduce the proposed method's computational cost.The paper cites existing acceleration techniques as possible additions to improve convergence speed.
Loading 1012.1184v1…