Source-linked AI summary
Learning to Deblur using Light Field Generated and Real Defocus Images
Lingyan Ruan, Bin Chen, Jizhou Li, Miuling Lam
TL;DR
Defocus deblurring is difficult because blur varies spatially and accurately aligned training pairs are hard to collect. The paper trains a dynamic-residual network first on LFDOF and then fine-tunes it on DPDD with feature loss; the authors report gains when this strategy is applied to several existing methods.
Problem
Spatially varying defocus blur and the difficulty of obtaining accurately corresponding defocused and all-in-focus pairs complicate single-image deblurring.
Source: §1 · p. 1Method
A multiscale network with dynamic residual blocks is trained on aligned LFDOF pairs and fine-tuned on DPDD using feature loss to address domain differences and pair misalignment.
Source: §4 · pp. 3–4Results
On DPDD test samples, LFDOF-and-DPDD training gained 0.511dB for DPDNetS, 0.249dB for KPAC, and 0.253dB for the proposed architecture in PSNR.
Source: §5.3 · p. 6Takeaways & Limitations
The authors report that the two-stage strategy can improve several existing learning methods, in addition to their proposed method.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Defocus deblurring is a challenging task due to the spatially varying nature of defocus blur. While deep learning approach shows great promise in solving image restoration problems, defocus deblurring demands accurate training data that consists of all-in-focus and defocus image pairs, which is difficult to collect. Naive two-shot capturing cannot achieve pixel-wise correspondence between the defocused and all-in-focus image pairs. Synthetic aperture of light fields is suggested to be a more reliable way to generate accurate image pairs. However, the defocus blur generated from light field data is different from that of the images captured with a traditional digital camera. In this paper, we propose a novel deep defocus deblurring network that leverages the strength and overcomes the shortcoming of light fields. We first train the network on a light field-generated dataset for its highly accurate image correspondence. Then, we fine-tune the network using feature loss on another dataset collected by the two-shot method to alleviate the differences between the defocus blur exists in the two domains. This strategy is proved to be highly effective and able to achieve the state-of-the-art performance both quantitatively and qualitatively on multiple test sets. Extensive ablation studies have been conducted to analyze the effect of each network module to the final performance.
1. Introduction
Defocus blur varies across image locations, making restoration challenging, while collecting aligned defocused and all-in-focus training pairs is difficult. The paper combines light-field and two-shot datasets and proposes a dynamic-residual network for deblurring.
1. Introduction
- Spatially varying blur complicates restoration, while kernel estimation and non-blind deconvolution can suffer from inaccurate kernels and ringing artifacts.Blur diameter depends on scene-point depth, and circle-of-confusion shape varies with position relative to the optical axis.Source: §1 · p. 1
- Accurate paired training data are difficult to collect because sequential two-shot captures can misalign, especially in outdoor scenes with motion or illumination changes.Light-field refocusing offers an alternative, but its blur differs from conventional-camera blur.Source: §1 · pp. 1–2
- The paper analyzes LFDOF and DPDD and proposes a training strategy that uses their complementary characteristics.It also compares light-field-generated and conventional-camera point spread functions.Source: §1 · p. 2
- The proposed end-to-end architecture uses dynamic residual blocks to reconstruct sharp images coarse-to-fine.Source: §1 · p. 2
- The authors report extensive module ablations and state-of-the-art quantitative and qualitative performance on multiple test sets.Source: §1 · p. 2
2. Related Work
Prior defocus-deblurring work includes kernel- or map-based methods, learned single-image networks, and datasets with different pairing and realism trade-offs. The paper focuses on single-image deblurring using real-scene DPDD and LFDOF data.
Related Work
- Conventional methods estimate a per-pixel defocus map before non-blind deconvolution, but inaccurate intermediate maps limit performance and the approach is computationally intensive.Source: §2 · p. 2
- DPDD provides sequentially captured defocused and all-in-focus pairs, whereas LFDOF generates pairs from single light-field captures; simulated datasets can lack realism.Other real-image datasets support blur detection only or provide benchmark test scenes.Source: §2 · p. 2
- Learned single-image deblurring methods use architectures including U-Net-like, multi-decoder, iterative-filter, and atrous-convolution designs, with the cited networks trained on DPDD.Source: §2 · p. 2
- Dynamic filtering learns per-pixel kernels for spatially varying degradation, while this paper instead learns image-domain dynamic residuals in a coarse-to-fine manner.The proposed block combines learned dynamic residuals with a direct residual rather than applying kernels directly to produce the output.Source: §2 · p. 2
3. Conventional Digital Camera vs. Light Field Camera
Conventional and light-field cameras form defocus blur differently because light-field sensors record ray direction as well as illuminance. The resulting light-field blur has a distinct point spread function from conventional-camera blur.
3. Conventional Digital Camera vs. Light Field Camera
- A conventional camera maps an in-focus scene point to one sensor pixel, while an out-of-focus point spreads into a circular circle of confusion.Source: §3 · p. 3
- A light-field camera's microlens array preserves ray-direction information, enabling synthetic aperture selection and refocusing by integrating samples across views or epipolar directions.Each Lytro Illum sub-aperture view records about 1/14×14 of the full aperture.Source: §3 · p. 3
- Conventional-camera point spread functions generally resemble a single Airy disk, whereas light-field-camera functions resemble multiple Airy disks.The authors estimate PSFs for front-focus, in-focus, and back-focus cases.Source: §3 · p. 3
4. Methodology
The method uses a multiscale encoder-decoder with cascaded dynamic residual blocks to reconstruct sharp images progressively. Training begins on accurately aligned LFDOF pairs and then uses feature loss on DPDD to address the domain difference.
4. Methodology
- An encoder extracts multiscale pyramidal features for decoder skip connections, and four cascaded dynamic residual blocks progressively restore the latent sharp image.Two residual blocks are added at each decoder scale.Source: §4 · p. 3, Figure 3 · p. 4
- Each dynamic residual block combines an upsampled image input with decoder features to estimate a dynamic kernel volume and a separate residual.The kernel volume is applied to the input to obtain the dynamic residual; the next block receives the previous output upsampled by 2×.Source: §4 · p. 3
- The block output combines its input, dynamic residual, and residual, whose contributions recover high-frequency details and low-frequency content, respectively.Source: §4 · p. 4
- LFDOF provides the main training stage for accurate image correspondence, followed by DPDD fine-tuning to alleviate differences between the datasets' blur domains.The first stage uses an L1 loss; the second uses VGG-based feature loss because two-shot pair misalignment makes pixel-wise loss unsuitable.Source: §4 · p. 4
- The training losses are applied at all scales, and the authors report that their two-stage strategy contributes significantly to restoration performance.Source: §4 · p. 4
5. Experiments
Experiments compare the proposed network with prior methods across five datasets and test its training strategy and architectural choices. The results favor the proposed method and the LFDOF-then-DPDD training strategy, with ablations examining dataset, loss, and DRB components.
5.2. Comparison to the State-of-the-Art Methods
- On RealDOF, our network beats DPDNetS, AIFNet, IFANet, and KPAC by 2.88dB, 2.65dB, 1.04dB, and 1.76dB in PSNR, respectively.RealDOF is not used for training, making it a fairer benchmark test set for comparing network performance.Source: §5.2 · p. 5, Table 2 · p. 6
- Qualitative comparisons report better restoration of text, texture, and object boundaries than IFAN, with strong visual generalization on CUHK and PixelDP despite absent all-in-focus ground truth.The experiments cover CUHK, DPDD, LFDOF, PixelDP, and RealDOF; LFDOF and DPDD are used for training, with the remaining datasets used for evaluation.Source: §5.1 · p. 4, §5.2 · p. 5, Figure 6 · p. 7
5.3. Analysis and Discussion
- LFDOF pretraining followed by DPDD fine-tuning outperforms training on either dataset alone on RealDOF, while LFDOF-only training gives sharper details but introduces artifacts.The authors attribute the artifacts to the difference between light-field-generated and real defocus blur.Source: §5.3 · p. 5, Table 3 · p. 6
- The RealDOF and DPDD evaluations compare networks trained on LFDOF, DPDD, or both, and the proposed two-stage strategy performs best among these choices.The table reports quantitative results on RealDOF; Figure 7 also compares visual results.Source: §5.3 · p. 5, Table 3 · p. 6, Figure 7 · p. 7
- On 76 DPDD test samples, LFDOF-and-DPDD training improves PSNR by 0.511dB for DPDNetS, 0.249dB for KPAC, and 0.253dB for our network.Visual quality also improves for these retrained networks.Source: §5.3 · p. 6, Table 4 · p. 7
- Using L1 loss in both training stages gives comparable PSNR and SSIM to the final strategy but worse DPDD LPIPS and 0.62dB lower RealDOF PSNR.On the mixed dataset, L1 alone and L1 combined with LVGG perform similarly, but worse than DPDD-only training.Source: §5.3 · p. 7, Table 5 · p. 7
- The DRB ablation finds that only the full block with all components restores realistic details in the shown RealDOF examples.The study compares component-removed variants, a multi-scale baseline, and a version with one full DRB.Source: §5.3 · pp. 7–8, Table 6 · p. 8, Figure 10 · p. 8
6. Conclusion
The proposed method combines light-field-generated and real defocus data with an end-to-end network to address single-image defocus deblurring. It outperforms other methods on multiple test sets, but irregularly shaped blur and blur mixed with object motion remain challenging.
6. Conclusion
- The method outperforms others by a significant margin on multiple test sets, with extensive comparisons and ablation studies supporting its effectiveness.The conclusion does not provide specific benchmark values.Source: §6 · p. 8
- Training first on light-field data with MAE loss and then on real defocus data with feature loss uses the strengths of both data types.The light-field data provides superior pixel-wise correspondence.Source: §6 · p. 8
- The end-to-end network uses a dynamic residual block and is reported effective for removing spatially varying defocus blur.Source: §6 · p. 8
- Irregularly shaped blur and defocus blur mixed with object motion remain limitations, and the authors identify them as future-work challenges.Source: §6 · p. 8