Source-linked AI summary
Deep Image Deblurring: A Survey
Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Bjorn Stenger, Ming-Hsuan Yang, Hongdong Li
TL;DR
Image deblurring aims to recover sharp images from blur, but the rapid growth and diversity of deep-learning methods makes the field difficult to overview. This survey synthesizes recent approaches, their formulations, datasets, metrics, architectures, losses, and applications, while identifying challenges including perceptual evaluation, data scarcity, and computational cost.
Problem
The expanding number of deep-learning deblurring methods and differing network designs make it challenging to obtain a rapid overview of the field.
Method
The paper surveys recent deep-learning image and video deblurring methods across problem formulations, architectures, loss functions, datasets, metrics, and domain-specific applications.
Results
The survey provides a taxonomy and detailed review of deep-learning deblurring methods, including applications to face, text, and stereo images.
Takeaways & Limitations
The paper serves as a literature review and reference point while outlining challenges and future research opportunities in image deblurring.
Takeaways & Limitations
PSNR and SSIM may not reflect perceptual quality, while MOS is difficult to reproduce and consistent human-aligned evaluation remains challenging.
Abstract
from arXiv · showhide
Image deblurring is a classic problem in low-level computer vision with the aim to recover a sharp image from a blurred input image. Advances in deep learning have led to significant progress in solving this problem, and a large number of deblurring networks have been proposed. This paper presents a comprehensive and timely survey of recently published deep-learning based image deblurring approaches, aiming to serve the community as a useful literature review. We start by discussing common causes of image blur, introduce benchmark datasets and performance metrics, and summarize different problem formulations. Next, we present a taxonomy of methods using convolutional neural networks (CNN) based on architecture, loss function, and application, offering a detailed review and comparison. In addition, we discuss some domain-specific deblurring applications including face images, text, and stereo image pairs. We conclude by discussing key challenges and future research directions.
1 INTRODUCTION
Image deblurring seeks to recover sharp images from blur caused by factors such as focus, camera motion, and target motion. This survey organizes recent deep-learning methods to provide an overview and reference for researchers.
- Image deblurring recovers a sharp image from a blurred input affected by causes including lack of focus, camera shake, and fast target motion.
- The growth of deep-learning methods for single-image and video deblurring has made the field difficult to survey rapidly because network designs differ.
- The survey reviews preliminaries including problem definitions, blur causes, deblurring approaches, quality metrics, and benchmark datasets.
- It discusses recent deep-learning models for single-image and video deblurring and provides a taxonomy for categorizing existing methods.
- The paper analyzes image-deblurring challenges and discusses research opportunities after reviewing methods and evaluation resources.
2 Preliminaries
The preliminaries define deblurring as inverse recovery of a latent sharp image and distinguish methods by whether the blur function is known. They also review blur models and image-quality assessment metrics.
- Blur Causes: Image blur may arise from camera shake, in-scene motion, or out-of-focus effects, and real scenes can contain multiple blur factors.
- Problem Formulation: The blur model maps a latent sharp image to a blurred image, while deblurring estimates the latent image through an inverse blur model.
- Motion Blur: Motion-blur models often use convolution, with blur kernels representing degradation and additive Gaussian noise representing sensor noise.
- Out-of-focus Blur: Out-of-focus blur depends on distance from the focal plane, and deep networks can detect blur regions or predict depth to guide deblurring.
- Evaluation Metrics: Image-quality assessment includes subjective MOS and objective full-reference or no-reference metrics, with PSNR and SSIM commonly used for restoration.
3 Non-Blind Deblurring
Non-blind deblurring assumes the blur kernel is known, yet noise and lost high-frequency information still make recovery difficult. Deep approaches combine deconvolution, denoising, learned priors, or direct network reconstruction.
- Problem Formulation: Non-blind deblurring recovers a latent sharp image from a blurry image when the blur kernel is given, but sensor noise and lost high-frequency information remain challenging.
- Method Categories: Deep non-blind methods broadly either perform deconvolution followed by denoising or directly employ deep networks.
- Deconvolution with Denoising: Deconvolution-with-denoising methods first recover a sharp image through regularized inversion and then remove artifacts produced during deconvolution.
- Learning Priors: Learned-prior methods represent image distributions or estimate noise while integrating priors into optimization-based deblurring.
- Limitations: Without outlier handling, non-blind approaches tend to generate ringing artifacts even when the estimated blur kernel is accurate.
4 Blind Deblurring
Deep deblurring methods address both single-image and video inputs through varied network layers, architectures, and temporal aggregation strategies. The survey organizes these approaches by their mechanisms and highlights trade-offs between perceptual realism and distortion-based image-quality metrics.
- Problem setting: Blind deblurring estimates both the latent sharp image and the unknown blur kernel, including non-uniform blur caused by different regional motions.Early methods mainly addressed uniform blur, while later approaches model non-uniform blur using camera motion.
- Input and aggregation: Video deblurring aggregates neighboring frames either by stacking images at the input or by fusing features before reconstruction.Image-level methods estimate the sharp central frame, whereas feature-level methods extract and fuse temporal information from neighboring frames.
- Basic layers and blocks: Deblurring networks commonly use residual, dense, attention, and convolutional components to improve optimization, feature propagation, spatial adaptation, or region prioritization.Convolutions may directly recover sharp images or incorporate depth and semantic guidance; attention can separately target foreground, background, and global regions.
- Basic layers and blocks: Recurrent layers transfer information across scales for single images or across neighboring frames for videos, supporting coarse-to-fine reconstruction and temporal feature extraction.Video recurrent methods either pass feature maps between steps or generate sharp frames directly from recurrent processing.
- Network architectures: The survey groups architectures into deep auto-encoders, GANs, cascaded, multi-scale, and reblurring networks, with DAE and GAN variants covering single-image and video deblurring.Video GAN generators additionally model neighboring-frame information, while 3DCNNs exploit spatial and temporal information.
- Network architectures: GAN-based deblurring can produce realistic images but typically performs worse on distortion metrics such as PSNR or SSIM.The generator is sufficient during inference, while a discriminator evaluates realism during training.
5 Loss Functions
Deep deblurring methods use loss functions that target pixel fidelity, perceptual similarity, realism, or motion modeling, with distinct strengths and trade-offs.
- Pixel Loss: Pixel loss measures reconstruction error against ground-truth images using L1 or L2 differences.L2 is widely used because it leads to high PSNR, while some models optimize L1.
- Perceptual Loss: Perceptual loss compares deep-network feature representations of sharp and deblurred images rather than matching individual pixels.Features may come from classification networks such as VGG19, producing visually pleasing results.
- Adversarial Loss: Adversarial loss jointly trains generator and discriminator networks so generated deblurred images resemble real sharp images.The discriminator predicts whether a deblurred image is real, guiding the generator toward photo-realistic outputs.
- Relativistic Loss: Relativistic loss evaluates whether generated images are more realistic than synthesized images and updates generator and discriminator components differently.Only the second part of the loss updates the generator, while the first updates the discriminator.
- Advantages and Drawbacks: Loss functions trade off different objectives: pixel loss can oversmooth, perceptual loss can remain unlike real images, and optical-flow loss targets motion blur.Adversarial and optical-flow losses may not improve PSNR/SSIM effectively, and optical-flow loss applies only to motion-blurred images.
6 Benchmark Datasets for Image Deblurring
Image-deblurring benchmarks span synthetic and real blur, single images and videos, and specialized domains such as text, faces, and stereo imagery.
- Dataset Overview: Benchmark datasets are designed to represent varied real-world blur and include domain-specific collections for face and text deblurring.Table 4 summarizes representative datasets for single-image, video, and domain-specific algorithms.
- Early Datasets: Early datasets use controlled capture or synthetic kernels, but uniform-blur collections can be insufficient for robust CNN training.The Levin dataset contains 4 sharp images and 8 uniform kernels, while Sun et al. extend it to 640 blurred images and retain the uniform-blur limitation.
- Realistic Blur: Later benchmarks model nonuniform and mixed scenarios using camera trajectories, real and synthetic images, and varied settings including outdoor, face, text, and low-light scenes.Köhler et al. provide 48 nonuniformly blurred images, while Lai et al. combine 100 real and 200 synthetic blurry images.
- Video Deblurring Datasets: Video-oriented datasets synthesize blur by averaging high-frame-rate sharp frames, with GoPro containing 3,214 pairs and HIDE containing 8,422 pairs.GoPro uses 240fps footage and HIDE averages 11 continuing frames, using the central frame as ground truth.
- Real-Blur Datasets: RealBlur provides 9,476 image pairs from JPEG and RAW subsets for training and benchmarking on real blurry images.The RAW subset applies white balance, demosaicking, and denoising operations.
- Domain-Specific Datasets: Specialized datasets support event-based, text, face, and stereo deblurring with task-specific blur synthesis and annotations.Examples include Blur-DVS with 2,178 synthesized pairs, 3M text-training patches, 130 million blurry face-training images, and 20,637 stereo pairs.
7 Performance Evaluation
The survey compares representative deblurring methods across datasets, metrics, architectures, loss functions, and blind versus non-blind settings. Results show that conclusions depend on evaluation criteria and blur-kernel availability, while multiscale and video-specific designs are often advantageous in their supported settings.
- Evaluation setup: Representative methods are evaluated on the GoPro, Köhler et al., Shen et al., and DVD datasets using quantitative tables and comparison figures.The reported results come from the respective papers, with bar graphs added for GoPro single-image and DVD video deblurring.
- Single image deblurring: Multiscale networks generally outperform single-scale networks, while GAN-based models can produce more realistic images but achieve poorer GoPro PSNR and SSIM.Coarse-to-fine processing provides a better prior at higher resolutions, whereas GAN objectives emphasize realism rather than necessarily improving distortion metrics.
- Evaluation metrics: Tao et al. outperforms Nah et al. and Kupyn et al. on GoPro PSNR and SSIM but performs worse under LPIPS.This metric-dependent reversal reflects the distinction between perceptual similarity and pixel-wise similarity, motivating evaluation with multiple quantitative measures.
- Loss functions: Combining reconstruction and perceptual losses generally improves performance, whereas GAN-based losses do not necessarily improve PSNR or SSIM.Using GAN loss or RaGAN loss with the same model yields similar deblurring performance.
- Blind versus non-blind deblurring: Blind methods outperform non-blind methods on RWBI when kernels must be estimated, but non-blind methods perform better when ground-truth kernels are provided.The comparison attributes the RWBI gap to the difficulty of explicitly estimating blur kernels and evaluates the controlled-kernel setting separately.
- UHD and video deblurring: UHD motion deblurring is more challenging, and multiscale architectures achieve better PSNR and SSIM by using downsampled inputs that retain sufficient detail.The survey also reports that video-based methods exploiting temporal information outperform single-image methods applied independently to video frames.
8 Domain-specific Deblurring
Domain-specific deblurring methods address faces, text, and stereo imagery by exploiting structure or information unavailable to generic image deblurring systems. These applications target specialized reconstruction needs, including facial semantics, OCR quality, and depth- or correspondence-based blur handling.
- Face deblurring: Face deblurring methods exploit facial structures and characteristics, with semantic parsing labels used as priors to guide restoration.Shen et al. extract pixel-wise semantic labels through a parsing network before deblurring.
- Text deblurring: Blurred text harms OCR, and generic deblurring methods are poorly suited to text images.The survey reviews both classical and deep-learning approaches for restoring text images.
- Stereo image deblurring: Stereo deblurring uses two blurry views to generate corresponding sharp images while exploiting depth and differences in corresponding-pixel blur.The stereo setting is motivated by blur caused by vibration when a stereo camera is mounted on a moving platform.
- Text deblurring: The deep-learning method in achieves better text-deblurring results than the reviewed non-deep methods.The comparison includes methods and the deep-learning method.
- Stereo image deblurring: Depth provides a prior for spatially varying blur, and stereo correspondence allows selecting the sharper pixel when the two views differ in blur.For translational motion, nearer points are more blurry than farther points, providing a usable depth-related cue.
- Other applications: Other specialized tasks include extracting video sequences from a single blurred image, synthesizing high-framerate sharp frames, and joint deblurring with super-resolution.
9 Challenges and Opportunities
The survey identifies persistent gaps between benchmark performance and practical image deblurring, especially for real-world data, perceptual evaluation, dataset quality, and computational efficiency.
- Real-world data: Real-world blurry images remain challenging despite strong benchmark performance because synthetic blur does not capture real-world degradation.Syntheticly trained models can perform excellently on synthetic tests yet worse on real images.
- Real-world data: Real images contain additional corruption from quantization, sensor noise, and low resolution, motivating unified restoration models.
- Real-world data: General deblurring models may fail on domain-specific images when sharpness must be recovered while preserving facial identity or text characters.
- Loss functions: Loss selection is scenario-dependent: L1 can outperform L2 on GoPro, while combining perceptual loss reverses that comparison for the same network.
- Evaluation metrics: PSNR and SSIM may misrepresent perceptual quality, whereas MOS better measures perception but requires user studies and is difficult to reproduce consistently.
- Data and models; Computational cost: High-quality real paired datasets are scarce and lack diversity, while many current methods remain too slow for high-speed 4K restoration on mobile devices.On a Tesla V100, three methods required approximately 26.76, 28.41, and 31.62 seconds per 4K image.