Source-linked AI summary
Deblurring by Realistic Blurring
Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, Hongdong Li
TL;DR
Existing deblurring methods commonly train on synthetically blurred image pairs, but these approximations leave a gap from complex real-world blur. The paper learns realistic blurring with BGAN and uses it to guide DBGAN deblurring, reporting superior quantitative performance and perceptual quality while introducing RWBI.
Problem
Existing methods use artificially blurred paired data, although real blur involves complex factors and differs from synthetic blur.
Method
A BGAN learns realistic blur from unpaired sharp and blurry images, while DBGAN learns to recover sharp images using generated blurry data and relativistic blur loss.
Results
The method achieves consistently superior perceptual quality and outperforms state-of-the-art methods quantitatively on the reported evaluations.
Takeaways & Limitations
Learning the blurring process alongside deblurring and using the RWBI dataset provides the paper's framework for handling realistic blur.
Abstract
from arXiv · showhide
Existing deep learning methods for image deblurring typically train models using pairs of sharp images and their blurred counterparts. However, synthetically blurring images do not necessarily model the genuine blurring process in real-world scenarios with sufficient accuracy. To address this problem, we propose a new method which combines two GAN models, i.e., a learning-to-Blur GAN (BGAN) and learning-to-DeBlur GAN (DBGAN), in order to learn a better model for image deblurring by primarily learning how to blur images. The first model, BGAN, learns how to blur sharp images with unpaired sharp and blurry image sets, and then guides the second model, DBGAN, to learn how to correctly deblur such images. In order to reduce the discrepancy between real blur and synthesized blur, a relativistic blur loss is leveraged. As an additional contribution, this paper also introduces a Real-World Blurred Image (RWBI) dataset including diverse blurry images. Our experiments show that the proposed method achieves consistently superior quantitative performance as well as higher perceptual quality on both the newly proposed dataset and the public GOPRO dataset.
1. Introduction
Image deblurring models often rely on artificially blurred training pairs, although real blur arises from complex exposure and scene factors that synthetic averaging does not capture. The paper addresses this gap by learning realistic blurring with BGAN before deblurring with DBGAN, using unpaired data and the RWBI dataset.
- Motivation: Real-world blur integrates sharp snapshots over an exposure period, while conventional training often approximates this process with simple frame averaging.Camera shake, object motion, and depth-of-field effects further complicate realistic blur modeling.
- Motivation: Artificially blurred images differ from real blurry images, creating a mismatch that can impair deblurring performance on testing data.Prior synthesis avoids the difficulty of obtaining paired real blurry and sharp images but does not reproduce realistic blur sufficiently.
- Method: The proposed framework combines BGAN, which learns realistic blurring from unpaired sharp and blurry images, with DBGAN, which recovers sharp images.Sharp images and random noises are supplied to BGAN to generate blurry images that preserve the input content.
- Method: A relativistic blur loss helps BGAN reduce the gap between synthesized and real blur by making real blurry images relatively more realistic to the discriminator.The loss uses complementary push and pull effects to update the generator and discriminator objectives.
- Contributions and results: The RWBI dataset supports training BGAN and evaluating the proposed deblurring model, alongside state-of-the-art quantitative results and superior perceptual quality.The reported evaluation covers the public GOPRO benchmark and real-world blurry images.
2. Related Works
Prior deblurring work largely focuses on recovering sharp images, while blur synthesis commonly relies on simplified models that fail to reproduce realistic blurry images.
- Image Blurring: Image blurring is modeled as convolution with a blur kernel plus additive noise, using IB = K ∗IS + N.K may be known or unknown, depending on whether the problem is non-blind or blind deblurring.
- Image Blurring: Training pairs are often synthesized from multiple sharp frames or simplified averaging, creating a gap between real and artificially blurred images.These approximations avoid the difficulty of obtaining paired real blurry and sharp images.
- Image Deblurring: Traditional and deep learning methods primarily recover sharp images from blurry inputs rather than modeling the blurring process itself.Existing neural methods include kernel estimation, sharp-image prediction, and multi-scale deblurring approaches.
- Image Blurring: GAN-based methods are introduced to generate varied blurry images, but earlier approaches did not generate realistic blur from real blurred images.The proposed direction is inspired by SinGAN's use of random noise to produce related image variations.
3. Deblurring by Blurring
The framework learns realistic blur with BGAN before training DBGAN to recover sharp images, using unpaired data for blurring and paired data for deblurring. Relativistic losses, perceptual and content losses, and varied blur inputs support the two-stage training process.
- BGAN: Learning to Blur: BGAN learns to mimic real-world blurry images from unpaired sharp and blurry datasets, generating paired data for DBGAN.Sharp images come from a public dataset, while real blurry images come from the RWBI dataset.
- DBGAN: Learning to Deblur: DBGAN is trained on paired blurry-sharp images produced through the BGAN module to recover sharp images from blurry inputs.Only the DBGAN generator is required during inference.
- Loss Functions: BGAN combines perceptual and adversarial losses, while DBGAN combines perceptual, adversarial, and L1 losses computed from generated and real sharp images.The perceptual loss uses features before, rather than after, the ReLU activation layer.
- Relativistic Blur Loss: The relativistic blur loss trains BGAN to make synthesized blur more realistic by pushing generated images toward 0.5 while pulling real-image predictions toward 0.5.The loss incorporates the discriminator’s predictions for both synthesized and real blurry images.
- BGAN: Learning to Blur: Different blur inputs and noise maps are used to increase the variation of synthesized blurry images and cover different real-world blur conditions.The BGAN generator concatenates a sharp image with a noise map to model different blurring factors.
4. Experiments
Experiments evaluate the framework on GOPRO and the newly collected RWBI dataset, including qualitative comparisons and ablations of its components. The ablation results show gains from both relativistic loss and learning-to-blur augmentation.
- 4.1. Datasets: The GOPRO dataset contains 3,214 image pairs, with 2,103 for training and 1,111 for testing.
- 4.1. Datasets: RWBI contains 3,112 blurry images across 22 sequences captured with four different hand-held devices.The devices are an iPhone XS, Samsung S9 Plus, Huawei P30 Pro, and GoPro Hero 5 Black.
- 4.1. Datasets: RWBI is evaluated qualitatively because it has no ground-truth sharp images, while GOPRO supports quantitative and qualitative comparison.
- 4.3. Ablation Study: DBGAN takes a blurry image as input and produces a deblurred image using generator and discriminator networks with perceptual, content, and relativistic blur losses.
- 4.3. Ablation Study: DBGAN(−) replaces relativistic blur loss with traditional adversarial loss, whereas DBGAN(+) adds BGAN-generated blurry images during fine-tuning.
- 4.3. Ablation Study: DBGAN outperforms DBGAN(−), and DBGAN(+) further improves over DBGAN in qualitative ablations.These comparisons are reported as evidence for the effectiveness of relativistic loss and the learning-to-blur module.
- 4.4. Comparison with Existing Methods: Figure 6 compares blurry inputs and outputs from Nah et al., Tao et al., and DBGAN(+) using magnified patches.
- 4.4. Comparison with Existing Methods: Table 1 presents GOPRO Large performance for seven prior methods, DBGAN, and DBGAN(+).
5. Conclusion
The paper presents a two-module framework that learns realistic blurring before deblurring and introduces the RWBI dataset. Experiments report consistently superior perceptual quality and quantitative performance over state-of-the-art methods.
- The framework uses BGAN to transfer sharp images into realistic blurry images and DBGAN to recover sharp images from blurry inputs.
- BGAN learns from unpaired sharp and blurry images, while a relativistic blur loss reduces the gap between synthesized and real blur.
- The RWBI dataset supports training and testing deblurring models.
- Experiments report consistently superior perceptual quality and quantitative outperformance of the proposed method against state-of-the-art methods.