Source-linked AI summary
Deblurring via Stochastic Refinement
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, Peyman Milanfar
TL;DR
Image deblurring is ill-posed, while deterministic, distortion-optimized methods can produce unrealistic reconstructions. The paper introduces a predict-and-refine conditional diffusion model that generates diverse plausible outputs, improving perceptual quality while retaining competitive distortion performance and more efficient sampling.
Problem
Single-image deblurring admits multiple plausible sharp images, but existing deterministic methods optimize distortion metrics that only partially reflect human perception.
Method
A predict-and-refine conditional diffusion model jointly trains a deterministic predictor and a stochastic sampler that refines its output into diverse reconstructions.
Results
The method achieves state-of-the-art perceptual quality across multiple standard datasets while retaining competitive distortion metrics; on GoPro, it achieves an FID of 4.04.
Takeaways & Limitations
Stochastic diffusion-based deblurring provides a practical alternative to single deterministic reconstructions and can traverse the Perception-Distortion curve by aggregating different numbers of samples.
Takeaways & Limitations
Slow sampling and large network size make diffusion models too computationally expensive for consumer-level devices.
Abstract
from arXiv · showhide
Image deblurring is an ill-posed problem with multiple plausible solutions for a given input image. However, most existing methods produce a deterministic estimate of the clean image and are trained to minimize pixel-level distortion. These metrics are known to be poorly correlated with human perception, and often lead to unrealistic reconstructions. We present an alternative framework for blind deblurring based on conditional diffusion models. Unlike existing techniques, we train a stochastic sampler that refines the output of a deterministic predictor and is capable of producing a diverse set of plausible reconstructions for a given input. This leads to a significant improvement in perceptual quality over existing state-of-the-art methods across multiple standard benchmarks. Our predict-and-refine approach also enables much more efficient sampling compared to typical diffusion models. Combined with a carefully tuned network architecture and inference procedure, our method is competitive in terms of distortion metrics such as PSNR. These results show clear benefits of our diffusion-based method for deblurring and challenge the widely used strategy of producing a single, deterministic reconstruction.
1. Introduction
Image deblurring is ill-posed because one blurry observation can correspond to multiple plausible sharp images, yet most methods produce a single deterministic reconstruction. The paper proposes a stochastic predict-and-refine diffusion model that improves perceptual quality, preserves competitive distortion, and supports efficient, tunable sampling.
- A blurry observation can correspond to multiple plausible sharp images, but existing methods typically output one deterministic clean-image estimate.
- Pixel-level losses optimize PSNR, although distortion metrics only partially match human perception and can produce visibly lower-quality reconstructions.
- The proposed predict-and-refine model jointly trains a deterministic data-adaptive predictor and a stochastic sampler that refines its output.
- Residual modeling makes diffusion sampling more efficient, while stochastic generation produces diverse, realistic reconstructions without sacrificing pixel-level distortion.
- The method achieves state-of-the-art distortion and perceptual results across standard datasets and traverses the Perception-Distortion curve without retraining or finetuning.
- The dual-network architecture uses an initial predictor to generate a deterministic candidate that the denoiser then models through its residual.
2. Related Work
Related deblurring methods commonly produce deterministic point estimates optimized for distortion metrics, despite the ambiguity of the inverse problem. Generative models offer posterior sampling and diversity, providing an alternative to regression-to-the-mean reconstructions.
- Deblurring can target posterior samples, a conditional mean, or a maximum a posteriori point estimate of the sharp image.
- End-to-end neural methods use blurry-sharp pairs and specialized architectures or losses to optimize pixel-level metrics such as PSNR and SSIM.
- Distortion-loss point estimators suffer regression to the mean, averaging multiple valid solutions and often losing image details.
- Likelihood-based generative models can sample diverse candidates from a learned posterior, unlike deterministic point estimates.
3. Diffusion Probabilistic Models
Diffusion probabilistic models define image generation as a fixed forward noising process and a learned stochastic reverse process. The denoiser estimates clean images from noisy states, enabling practical posterior sampling through arbitrary noise levels.
- Forward diffusion: A diffusion probabilistic model uses a T-step Markov chain that repeatedly injects Gaussian noise into a clean sample.The latent variables retain the original data dimensionality, and the noise schedule controls the variance added at each step.
- Forward diffusion: The forward process has closed-form marginal and reverse-step distributions, allowing partially noisy samples to be drawn at arbitrary time steps.This arbitrary-marginal sampling makes diffusion training practical because the training objective relies on it.
- Reverse process: The learned denoiser fθ estimates x0 from a partially noisy image, replacing the unavailable clean image in the reverse diffusion step.The reverse process runs backward from Gaussian noise toward a clean sample, while training seeks to approximate the true reverse transitions.
- Reverse process: An alternative parameterization trains fθ to predict Gaussian noise ϵ, which is deterministically related to xt and x0 through the diffusion equation.The formulation also supports conditioning on a continuous noise level.
- Sampling: Standard diffusion sampling starts from xT drawn from a Gaussian distribution and requires T denoiser calls before returning a single x0 sample.Each reverse transition samples a slightly less noisy image from the current state.
4. Predict-and-Refine Diffusion Model
The predict-and-refine model reduces diffusion sampling cost by combining a deterministic initial predictor with a stochastic residual denoiser. Its inference procedure trades perceptual quality and distortion through noise schedules and sample averaging, while a fully convolutional design supports arbitrary image sizes.
- Motivation: Diffusion models may require thousands of denoiser forward passes, motivating sampling strategies that reduce the number of iterative steps.The proposed method addresses this computational cost by exploiting a cheap initial guess for conditional generation.
- Model: A deterministic initial predictor supplies a data-adaptive clean-image candidate, while the denoiser models only the residual correction.The initial predictor is trained jointly through gradients flowing from the denoiser loss, without extra loss or pretraining.
- Sampling procedure: Predict-and-refine sampling initializes xinit from the blurry input, runs stochastic reverse diffusion on a latent residual, and returns xinit + z0.At each reverse step, the sampler draws Gaussian noise and applies the learned denoising transition.
- Efficiency: The residual model is up to 7× faster than the non-residual model at comparable sample quality.Because the initial predictor runs once, reducing denoiser cost is amplified across the repeated sampling steps.
- Inference trade-offs: Using many small-noise steps generally improves perceptual quality, whereas fewer large-noise steps lower distortion; sample averaging provides the complementary trade-off.The LPIPS-oriented setting uses T = 500 without averaging, while the distortion-oriented Ours-SA setting uses T = 10 with averaging.
- 4.2. Resolution-agnostic Architecture: A fully convolutional architecture supports arbitrary input shapes, with preliminary experiments reporting little to no sample-quality degradation at non-native resolutions.Self-attention, positional encoding, and group normalization were removed to make the architecture agnostic to image resolution.
5. Experiments
Experiments evaluate the method on GoPro and HIDE using perceptual, distortion, qualitative, and human-preference measures. The method achieves strong perceptual quality while retaining competitive distortion performance, and its stochastic outputs vary with input blur.
- Evaluation setup: The evaluation uses GoPro and HIDE datasets, with perceptual metrics complemented by PSNR and SSIM and qualitative human evaluation.The perceptual metrics include LPIPS, NIQE, FID, and KID; human ratings compare side-by-side GoPro crops.
- GoPro results: The GoPro results achieve state-of-the-art performance across perceptual metrics while maintaining competitive PSNR and SSIM.The sample-averaging variant reaches a new SOTA PSNR of 33.23, while the method obtains an FID of 4.04, nearly 70% lower than DeblurGANv2.
- GoPro results: The method establishes a new perception-distortion Pareto frontier, and sample aggregation controls the trade-off without retraining or finetuning.The framework can traverse the Perception-Distortion curve by aggregating different numbers of generated samples.
- HIDE results: On HIDE, models trained on GoPro significantly outperform baselines across perceptual metrics while maintaining competitive distortion values.This evaluates generalization to out-of-distribution input.
- Human study: Human raters preferred both model variants over competing methods across 480 pairwise comparisons.Raters modestly preferred sample averaging for flatter crops but individual samples for highly textured crops.
- Qualitative analysis: Blurrier inputs produce higher variance across generated samples, indicating that output diversity adapts to the strength of the blur.Qualitative samples also show improved detailed textures, although some reconstructions contain additional noise.
6. Discussion and Analysis
The analysis examines why residual modeling improves diffusion deblurring, finding substantial sampling-efficiency gains and evidence that residual images are simpler to model. It also shows that the initial predictor alone can achieve strong distortion performance.
- More efficient sampling: Up to 7× faster sampling is achieved at comparable sample quality with a large initial predictor and smaller denoiser.The residual model requires less sampling time than a regular conditional diffusion model despite having more total parameters.
- Initial predictor: The initial predictor often produces a reasonable reconstruction that is less blurry than the input, despite having no explicit reference-matching loss.The authors leave alternative parameterizations and further investigation of the predictor for future work.
- Residual images are simpler to model: Residual pixel values have 3.91 bpd entropy versus 7.42 bpd for reference images, supporting the hypothesis that residuals are simpler to model.Entropy is computed over pixel values aggregated across all pixel locations.
- Initial predictor: The initial predictor alone reaches state-of-the-art PSNR of 33.07 when trained with a simple L2 loss.The ablation identifies EMA weights, large batch size, and network size as critical hyperparameters.
7. Conclusion and Future Directions
The paper concludes that stochastic conditional diffusion improves perceptual blind deblurring while retaining competitive distortion performance. It identifies computational cost as a remaining barrier and points to more efficient sampling and network designs as future directions.
- Conclusion: The framework achieves improved perceptual quality and competitive distortion metrics across standard datasets compared with state-of-the-art methods.The conclusion presents this as evidence for stochastic diffusion-based deblurring.
- Future Directions: Diffusion models remain too computationally expensive for consumer-level devices because of slow sampling and large network size.Suggested directions include DDIM, distillation, and U-Net architectures optimized for distortion and runtime.
- Conclusion: Perception-distortion trade-offs persist across combinations of perceptual metrics and PSNR or SSIM.The additional plots include perceptual metrics versus PSNR and SSIM comparisons.
- Conclusion: Blurrier input images produce greater diversity among generated deblurred samples.The relationship is analyzed using input sharpness and sample diversity.
C. Synthetic DIV2K Deblurring Dataset
The supplementary material describes a synthetic DIV2K deblurring dataset and the diffusion formulation, architecture, schedules, and evaluation plots used to analyze the method.
- Synthetic dataset: The custom dataset applies synthetic camera-shake blur and noise to DIV2K images for controlled qualitative evaluation.Existing paired datasets have low-quality ground truth images, making qualitative assessment difficult.
- Noise schedule: A linear variance schedule is used, while continuous noise-level conditioning permits a different inference schedule from the training schedule.The model predicts Gaussian noise ε, and inference schedules can be varied across steps and noise levels.
- Network architecture: The fully convolutional U-Net removes self-attention, group normalization, and positional encoding, and concatenates xt with conditioning input y.Both the denoiser and initial predictor use this architecture, with different starting channel counts and parameter sizes.
- Analysis plots: Additional plots show that perception-distortion trade-offs persist across perceptual metrics paired with PSNR or SSIM.Figure 8 also relates blur severity to diversity in generated samples.
F. Evaluation Details
The evaluation details describe inference hyperparameter sweeps and the procedures used to obtain baseline restoration results and metrics across GoPro, HIDE, and DIV2K.
- Inference settings: Inference steps were swept over T ∈ {10, 20, 30, 50, 100, 200, 300, 500}.The sweep was performed for experiments on GoPro, HIDE, and DIV2K.
- Inference settings: The final forward-process variance was swept over {0.01, 0.02, 0.05, 0.1, 0.2, 0.5}, with initial variance fixed at 1 × 10^-6.Intermediate schedule values were linearly interpolated.
- Baseline comparisons: Author-produced restorations were used whenever possible; otherwise, official implementations and pretrained models generated the comparisons.This procedure was intended to ensure fair metric comparisons.
- Baseline comparisons: For SimpleNet, only reported PSNR, SSIM, and LPIPS metrics were used because restorations and code were unavailable.For HINet, MPRNet, and SAPHNet, author-produced restorations were used; MIMO-UNet+ and DeblurGANv2 used author releases.
G. Large GoPro and HIDE Results
The appendix provides enlarged GoPro and HIDE deblurring restorations previously shown in the main text.
- Figures 11–12 show larger versions of the GoPro restorations from the main text.
- Figure 13 shows a larger version of the HIDE restoration from the main text.
- The appendix groups the enlarged visual comparisons across GoPro and HIDE datasets.
H. Additional Results
Additional experiments document the shared U-Net architecture, visual comparisons against established methods, and improved textural detail over an L2 regression baseline.
- DIV2K results: On the synthetically generated DIV2K deblurring dataset, the method produces more realistic textural details than an over-smoothed L2 regression baseline.
- Architecture: The U-Net architecture is used for both the denoiser network and the initial predictor.
- GoPro and HIDE comparisons: GoPro and HIDE figures compare the method, including sampling averaging, with SAPHNet, DeblurGAN-v2, MIMO-Unet+, MPRNet, and HINet.