Source-linked AI summary
Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring
Seungjun Nah, Tae Hyun Kim, Kyoung Mu Lee
TL;DR
Dynamic-scene blind deblurring is challenged by complex, non-uniform motion that existing kernel assumptions and synthetic training data do not capture well. The paper proposes a kernel-free multi-scale CNN, multi-scale losses, and a realistic high-speed-camera dataset; experiments report superior qualitative and quantitative performance, with the dataset using an approximate gamma response when the true CRF is unavailable.
Problem
Complex dynamic-scene blur makes kernel estimation inaccurate, while existing deblurring methods and training data rely on restricted or unrealistic blur models.
Method
A multi-scale CNN directly restores latent images without explicit kernel estimation, trained with multi-scale losses on realistic blurry–sharp pairs generated from high-speed sharp-frame sequences.
Results
The approach outperforms state-of-the-art methods in qualitative and quantitative evaluations while being much faster.
Takeaways & Limitations
Kernel-free learning and realistic blur–sharp supervision support dynamic-scene deblurring without requiring explicit per-pixel blur-kernel estimation.
Takeaways & Limitations
When the ground-truth camera response function is unavailable, the dataset generation approximates it with a gamma curve using γ = 2.2.
Abstract
from arXiv · showhide
Non-uniform blind deblurring for general dynamic scenes is a challenging computer vision problem as blurs arise not only from multiple object motions but also from camera shake, scene depth variation. To remove these complicated motion blurs, conventional energy optimization based methods rely on simple assumptions such that blur kernel is partially uniform or locally linear. Moreover, recent machine learning based methods also depend on synthetic blur datasets generated under these assumptions. This makes conventional deblurring methods fail to remove blurs where blur kernel is difficult to approximate or parameterize (e.g. object motion boundaries). In this work, we propose a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various sources. Together, we present multi-scale loss function that mimics conventional coarse-to-fine approaches. Furthermore, we propose a new large-scale dataset that provides pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera. With the proposed model trained on this dataset, we demonstrate empirically that our method achieves the state-of-the-art performance in dynamic scene deblurring not only qualitatively, but also quantitatively.
1. Introduction
Dynamic-scene deblurring is difficult because spatially varying blur from camera motion, object motion, depth variation, occlusion, and motion boundaries is hard to model accurately. The paper addresses these limitations with a kernel-free multi-scale CNN, multi-scale training losses, and a realistic blurry–sharp image dataset.
- Problem: Dynamic-scene blur combines camera shake, object motion, depth variation, occlusion, and motion boundaries, making the unknown blur kernel difficult to estimate.Kernel estimation is severely ill-posed, especially when multiple objects move alongside the camera.
- Problem: Simple or locally linear blur-kernel assumptions remain inaccurate near abrupt motion discontinuities and occlusions, producing ringing artifacts in restored images.Existing methods approximate kernels using restricted motion models or local linearity.
- Related work: CNN deblurring methods also commonly rely on synthetic blurry images generated from uniform or locally linear kernels, limiting their applicability to complex real-world blur.The paper identifies unrealistic kernel models and unavailable real blurry–sharp pairs as persistent issues.
- Proposed approach: The proposed multi-scale CNN directly restores latent images without estimating explicit blur kernels or assuming a restricted kernel model.Its coarse-to-fine architecture preserves fine detail and long-range dependency, while multi-scale losses train intermediate outputs explicitly.
- Dataset: The authors construct a realistic dataset by capturing high-speed sharp-frame sequences and integrating them, with gamma correction, to generate blurry images paired with ground truth.This kernel-free acquisition process avoids designing complex motion kernels for training data.
- Results: The method achieves state-of-the-art dynamic-scene deblurring performance in both qualitative and quantitative evaluations.The reported improvements are obtained after training on the proposed dataset with augmentation.
2. Blur Dataset
The dataset simulates realistic blur by integrating high-speed sharp frames over an exposure interval, while accounting for the camera response function. It contains 3,214 blurry–sharp image pairs generated from GOPRO4 videos.
- Blur accumulation: Blur is synthesized by accumulating sharp sensor signals over the exposure time rather than convolving a sharp image with a predefined kernel.The process approximates camera exposure using sampled high-speed video frames.
- Blur accumulation: The accumulation model averages M sampled sharp frames, with T denoting exposure time and S[i] the i-th frame signal.The camera response function g maps latent sensor signals to observed image values.
- Camera response: Because the original sensor signal and camera response function are unknown, the method approximates the response with a gamma curve using γ = 2.2.The text states that nonlinear camera response functions complicate non-uniform deblurring.
- Camera response: Latent frame signals are recovered by inverse gamma correction before successive frames are combined to synthesize the blurry image.This correction uses S[i] = g^-1(Ŝ[i]).
- Dataset construction: The GOPRO dataset uses 240 fps videos and averages 7–13 successive frames to create blur strengths, yielding 3,214 pairs at 1280×720 resolution.Each sharp target is defined as the mid-frame among the frames averaged for its corresponding blur.
- Dataset construction: Frame averaging preserves motion-dependent non-uniform blur, unlike uniform-kernel convolution that also blurs a stationary background.The comparison motivates a kernel-free blur-generation approach for dynamic scenes.
3. Proposed Method
The proposed deblurring model combines a Gaussian-pyramid CNN with modified residual blocks and multi-scale content and adversarial losses. It processes blurry images across scales, trains intermediate outputs toward sharp targets, and restores the finest-scale image.
- Multi-scale architecture: Gaussian-pyramid inputs and outputs let coarser features aid finer-scale deblurring while preserving fine-level information.The model uses three scales with a 0.5 ratio between consecutive levels.
- Residual blocks: Modified residual blocks remove the post-shortcut rectified linear unit, which empirically improves convergence speed and performance.Residual structure also supports deeper networks and lets parameters learn differences between similar blurry and sharp images.
- Multi-scale architecture: Each finer-scale network receives its blurry input concatenated with sharp features from the previous stage, and the finest scale restores the original-resolution image.
- Training objectives: The multi-scale content loss applies mean squared error at every pyramid level so intermediate outputs match sharp images at corresponding scales.Each scale loss is normalized by its channels, width, and height.
- Training objectives: Training jointly combines multi-scale content loss with adversarial loss, using a discriminator that distinguishes finest-scale outputs from ground-truth sharp images.The generator minimizes adversarial loss while the discriminator maximizes it.
4. Experimental Results
Experiments evaluate the model on GOPRO and additional deblurring datasets using qualitative and quantitative comparisons. On GOPRO, the K = 2 system gives the best PSNR and SSIM, while K = 3 is fastest; qualitative comparisons also show reduced artifacts.
- GOPRO dataset: On the GOPRO test set, the proposed model improves image quality over state-of-the-art competitors in qualitative and quantitative evaluation.The test set contains 1111 blurry–sharp pairs, approximately one-third of the total dataset.
- GOPRO dataset: K = 2 produces the best GOPRO results for both PSNR and SSIM, whereas K = 3 provides the fastest runtime.
- GOPRO dataset: Compared with Sun et al. and Kim and Lee, the proposed method avoids failures around nonlinear blur shapes, motion boundaries, and weak edges.The competing methods are described as relying on kernel-estimation behavior in these difficult regions.
- Köhler dataset: On the Köhler dataset, the paper reports multi-scale SSIM because the dataset supplies its own evaluation code.The paper also reports that K = 3 gives the best PSNR and K = 2 the best MSSIM in its system.
- Additional dataset: On dataset, the proposed results avoid ringing artifacts while preserving details such as wave ripple.The images are not aligned for direct PSNR or SSIM comparison, so the paper reports qualitative comparisons.
5. Conclusion
The paper presents a blind deblurring neural network that combines coarse-to-fine multi-scale processing, realistic data, and supervised learning. It outperforms state-of-the-art methods qualitatively and quantitatively while being faster.
- 5. Conclusion: The proposed network estimates sharp images without relying on explicit blur-kernel estimation.This avoids problems associated with kernel estimation.
- 5. Conclusion: The model follows a coarse-to-fine approach and is trained in multi-scale space.
- 5. Conclusion: A realistic ground-truth blur dataset enables efficient supervised learning and rigorous evaluation.
- 5. Conclusion: The approach outperforms state-of-the-art methods qualitatively and quantitatively while being much faster.
A. Appendix
The appendix reports additional comparative experiments designed to demonstrate the effectiveness of the proposed deblurring method.
- A. Appendix: The appendix presents additional comparative experimental results.
- A. Appendix: These experiments provide further evidence concerning the proposed deblurring method.
- A. Appendix: The stated purpose of the appendix is to demonstrate the method’s effectiveness.
A.1. Comparison of loss function
This appendix compares training losses for the proposed model on the GOPRO test dataset and examines both quantitative and qualitative effects of adversarial loss.
- A.1. Comparison of loss function: Adding adversarial loss does not increase PSNR but increases SSIM on the GOPRO test dataset.The evaluation assumes linear CRF with K = 3 and λ = 1 × 10−4.
- A.1. Comparison of loss function: The SSIM increase indicates that adversarial loss encourages more natural and structure-preserving images.
- A.1. Comparison of loss function: Figures A.1 and A.2 qualitatively compare networks trained with Lcont against Lcont + λLadv.
A.2. Comparison on GOPRO dataset
The appendix provides qualitative GOPRO test-dataset comparisons between two prior methods and the proposed method.
- A.2. Comparison on GOPRO dataset: Qualitative deblurring results are provided on the GOPRO test dataset.
- A.2. Comparison on GOPRO dataset: Figures A.3 and A.4 compare the deblurring results of Kim and Lee, Sun et al., and the proposed method.
- A.2. Comparison on GOPRO dataset: The blurry images used for these visual comparisons come from the proposed dataset.
A.3. Comparison on Lai et al. [20] dataset
The paper evaluates competing deblurring methods on real images from the Lai et al. dataset, which contains both synthetic and real blurry images.
- The Lai et al. dataset contains synthetic and real blurry images.
- A synthetically generated blurry image from the dataset was evaluated earlier in Section 4.3.
- Real-image deblurring results from competing methods are presented in Figures A.5 and A.6.
A.4. Comparison on real dynamic scenes
The paper also compares qualitative deblurring results on real dynamic scenes captured with a SONY RX100 M4 camera.
- The real dynamic scenes were captured using a SONY RX100 M4 camera.
- Figures A.7 and A.8 compare Kim and Lee, Sun et al., and the proposed method on these scenes.