Source-linked AI summary
Defocus Deblurring Using Dual-Pixel Data
Abdullah Abuolaim, Michael S. Brown
TL;DR
Defocus blur from wide-aperture imaging is spatially varying and difficult to estimate. The paper uses discarded dual-pixel sub-aperture views with a DNN and a 2000-image paired dataset, achieving state-of-the-art quantitative and perceptual results while also demonstrating benefits for other vision tasks.
Problem
Wide-aperture capture creates spatially varying defocus blur, but recovering an all-in-focus image remains difficult.
Method
A DNN uses left and right dual-pixel views alongside a carefully captured dataset of blurred, sharp, and sub-aperture images.
Results
The method achieves state-of-the-art results on signal-processing and perceptual metrics and benefits other computer vision tasks.
Takeaways & Limitations
Dual-pixel data discarded after autofocus can provide effective additional information for defocus deblurring.
Takeaways & Limitations
The dataset is captured with a Canon DSLR because available smartphone DP data lack adjustable apertures and full-frame measurements.
Abstract
from arXiv · showhide
Defocus blur arises in images that are captured with a shallow depth of field due to the use of a wide aperture. Correcting defocus blur is challenging because the blur is spatially varying and difficult to estimate. We propose an effective defocus deblurring method that exploits data available on dual-pixel (DP) sensors found on most modern cameras. DP sensors are used to assist a camera's auto-focus by capturing two sub-aperture views of the scene in a single image shot. The two sub-aperture images are used to calculate the appropriate lens position to focus on a particular scene region and are discarded afterwards. We introduce a deep neural network (DNN) architecture that uses these discarded sub-aperture images to reduce defocus blur. A key contribution of our effort is a carefully captured dataset of 500 scenes (2000 images) where each scene has: (i) an image with defocus blur captured at a large aperture; (ii) the two associated DP sub-aperture views; and (iii) the corresponding all-in-focus image captured with a small aperture. Our proposed DNN produces results that are significantly better than conventional single image methods in terms of both quantitative and perceptual metrics -- all from data that is already available on the camera but ignored. The dataset, code, and trained models are available at https://github.com/Abdullah-Abuolaim/defocus-deblurring-dual-pixel.
1 Introduction
Wide apertures admit more light but create shallow depth of field and spatially varying defocus blur, motivating methods that preserve fast-shutter capture while recovering sharp images. This paper uses discarded dual-pixel views with a DNN and introduces a paired dataset for training and evaluation.
- Motivation: Wide apertures provide sufficient light but produce shallow depth of field and defocus blur outside the focused region.This trade-off matters for applications such as self-driving and mapping cameras that use fixed shutter speeds.
- Motivation: Dual-pixel sensors capture left and right sub-aperture views whose disparity is correlated with defocus blur.Cameras normally use this phase-shift information for autofocus and then discard it.
- Approach: The proposed DNN estimates a sharp image from the blurred input’s left and right dual-pixel views.The method exploits data already available during image capture rather than relying only on a single blurred image.
- Dataset: The dataset contains 500 blurred scenes with corresponding dual-pixel views and all-in-focus images, totaling 2000 full-frame images.The dataset includes 1000 sub-aperture views and images at 6720 × 4480-pixel resolution.
- Evaluation: The method is evaluated against conventional single-image defocus deblurring approaches using signal-processing and perceptual metrics.The related work includes defocus-map estimation and direct image-deblurring methods, while existing light-field datasets target different tasks or use less representative imagery.
3 DP image formation
Dual-pixel sensors record paired sub-aperture views whose phase differences reveal spatially varying defocus. The dataset captures matched blurred and sharp images while preserving camera settings needed for supervised learning.
- 3 DP image formation: Each dual-pixel unit independently records light from two halves of the main lens as left and right views.The paired views are formed by separate photodiodes beneath a microlens.
- 3 DP image formation: The final camera image aggregates the left and right signals, while the separate views retain additional information for deblurring.The combined image can have more blur than an individual sub-aperture signal in the out-of-focus case.
- 3 DP image formation: In-focus scene points produce no disparity between the dual-pixel views, whereas out-of-focus points produce phase differences correlated with blur.Out-of-focus regions are blurred in opposite directions in the two views through a point-spread function and its flipped version.
- 4 Dataset collection: Cross-correlation of left/right patches reveals different disparity patterns for in-focus and out-of-focus regions.A DNN uses these patterns to learn the extent of spatially varying blur.
- 4 Dataset collection: The dataset uses a Canon EOS 5D Mark IV because the available smartphone DP data lack adjustable apertures and full-frame measurements.Most smartphone images also exhibit little optical defocus because of their small fixed apertures.
- 4 Dataset collection: Each scene is captured at f/4 and f/22 with fixed lens position and focal length, while exposure compensation adjusts shutter speed.The f/4 image is the defocused input and the f/22 image supplies the sharper reference.
- 4 Dataset collection: The paired captures use static tripod-mounted scenes, remote operation, fixed ISO, and left/right views extracted from the blurred image.The sharp image serves as the all-in-focus ground truth, so its dual-pixel views are visually identical for this purpose.
5 Dual-pixel defocus deblurring DNN (DPDNet)
DPDNet is a U-Net-like encoder-decoder that takes the two DP views as a six-channel input and predicts a sharp image. Its design uses skip connections, downsampling, and MSE training, while Table 1 reports that the DP-view model performs best across all tested categories and metrics.
- Architecture: The symmetric encoder-decoder uses skip connections between corresponding feature maps to combine feature levels.The architecture adapts a U-Net-like design and passes feature maps to mirrored convolutional layers without cropping.
- Architecture: DPDNet takes the two three-channel DP views as a six-channel input and generates a three-channel sharp sRGB image.The network processes the views through encoder, bottleneck, and decoder stages.
- Architecture: Max pooling expands the receptive field needed to handle large defocus blur, while dropout layers reduce overfitting risk.The network uses 2 × 2 max pooling in encoder blocks and dropout before pooling in the fourth block and at the bottleneck.
- Training: The model predicts the sharp image as DPDNet(I_L, I_R; θ_DPDNet), where θ_DPDNet denotes the network weights and parameters.Training uses extracted 512×512 patches, Adam optimization, mini-batches of five, and MSE between output and ground truth.
- Quantitative results: DPDNet using two L/R DP views achieves the best results for indoor, outdoor, and combined test categories across all reported metrics.The comparison includes a same-architecture single-input variation, DPDNet-Single, trained without DP views.
6 Experimental results
Experiments compare DPDNet with conventional and learning-based deblurring methods using quantitative, qualitative, timing, and aperture-robustness evaluations. DPDNet performs best in the reported comparisons, processes images much faster than the second-best method, and generalizes to tested aperture settings.
- Evaluation setup: The evaluation reports PSNR, SSIM, MAE, and LPIPS on indoor, outdoor, and combined test categories.The testing set is divided by scene category, and LPIPS is included as a learned perceptual similarity metric.
- Quantitative results: DPDNet significantly outperforms competing methods across all metrics and scene categories, with DP views providing an advantage over DPDNet-Single.DPDNet-Single uses the same architecture settings and training procedure but accepts only the blurred image.
- Qualitative results: Qualitatively, DPDNet most closely resembles the ground-truth sharp image and handles spatially varying blur better than the compared methods.The comparison includes enlarged patches, while EBDB, DMENet, and JNB show limited ability to handle spatially varying blur.
- Time analysis: 1.2×10^3 times faster than DMENet, DPDNet processes a 1680×1120 testing image in a single pass.The compared methods use separate defocus-map estimation and non-blind deblurring passes, whereas DPDNet directly computes the sharp image.
- Robustness: DPDNet generalizes well to aperture settings different from its f/4 training blur, including the tested f/10 and f/16 settings.The robustness test uses less defocus blur at f/10 and f/16 than at f/4.
7 Applications
The paper examines how defocus blur affects image segmentation and monocular depth estimation, finding that DPDNet deblurring improves both downstream results.
- Image segmentation: Defocus blur causes PSPNet to segment a large portion of the image as unknown, whereas the deblurred image yields more correctly segmented categories.
- Monocular depth estimation: Defocus blur produces a completely wrong monocular depth map with the monodepth model.
- Monocular depth estimation: Testing the deblurred image significantly improves monodepth depth estimation results.
8 Conclusion
The paper presents a dual-pixel-based DNN method and dataset for reducing defocus blur in shallow-depth-of-field images. Experiments report state-of-the-art quantitative and perceptual results, along with benefits for other computer vision tasks.
- The method uses dual-pixel data available in modern camera sensors within a DNN framework to reduce defocus blur.
- The authors capture a dataset of blurred and sharp image pairs together with their dual-pixel images.
- The method achieves state-of-the-art results on signal processing and perceptual metrics.
- The deblurring method can benefit other computer vision tasks, and the dataset and method are intended to support further research.
Supplemental Materials
The supplemental materials document ablations, use cases, smartphone dual-pixel results, additional quantitative results, and animated qualitative examples.
- The supplement includes ablations of DPDNet variations involving inputs, network blocks, input sizes, patch filtering, and data types.
- It discusses defocus and motion blur, describes use cases, and reports results on smartphone dual-pixel data.
- The repository includes 14 animated qualitative examples and additional quantitative results.
S1 Ablation study
The ablation study tests extra inputs, network capacity, input size, homogeneous-patch filtering, and image bit depth, identifying several design choices with limited or modest effects.
- DPDNet with extra input image: Adding the combined blurred image IB to the left/right views provides no noticeable improvement because IB is their combination.
- DPDNet with less blocks: The full DPDNet performs better than DPDNet-Light, while the lighter model retains acceptable results for limited computational resources.
- DPDNet with different input sizes: 256×256 and 512×512 input patches perform similarly when the patch size is larger than the blur size.
- DPDNet with different filtering ratios: Homogeneous patches are ambiguous about blur size, and filtering 30% of patches gives the best quantitative results.
- DPDNet with different data types: 16-bit input data performs slightly better than 8-bit data, including lower LPIPS distance across all categories.
S2 Defocus and motion blur discussion
Defocus and motion blur arise from different physical image-formation processes, so motion-deblurring methods are not expected to perform well on defocus blur.
- Defocus and motion blur both blur the latent image, but their physical image-formation processes differ.
- Motion-deblurring methods are therefore not expected to provide good results for defocus blur.
- SRNet was nevertheless tested on the defocus-deblurring testing set as a validity check.
S3 Use cases
Defocus deblurring is useful when exposure choices trade off noise, motion blur, and defocus blur, including moving-camera and moving-object scenarios.
- Exposure trade-off: The exposure triangle reflects a trade-off among ISO-related noise, shutter-speed-related motion blur, and aperture-related defocus blur.
- Moving camera: For moving cameras, increasing ISO can reduce motion blur but introduce noise, while increasing aperture can introduce defocus blur.
- Moving camera: The proposed defocus deblurring method produces a sharper and cleaner result for the wide-aperture image in the moving-camera example.
- Moving object: For a moving object, widening the aperture handles motion blur but introduces defocus blur, which the proposed method deblurs into a sharper image.
S4 DPDNet performance for a smartphone DP sensor
Smartphone experiments provide a proof of concept for applying DPDNet to other DP sensors, but smartphone constraints prevent quantitative evaluation against ground truth.
- Experimental scope: The smartphone experiments examine DP data from Google Pixel 3 and 4 smartphones, whose sensor data differs from the Canon training setup.
- Sensor limitations: Pixel smartphones lack adjustable apertures, provide only one green raw-Bayer channel, and usually exhibit little defocus blur.
- Experimental scope: The experiments serve as a proof of concept that DPDNet should generalize to other DP sensors.
- Input preparation: For Pixel inputs, the single green channel is replicated to three channels, and the combined image is obtained by averaging the two views.
- Evaluation: Because smartphone aperture cannot be adjusted to capture ground truth, the reported results are qualitative rather than quantitative.
S5 More results
Additional results evaluate single-view behavior, aperture robustness, and smartphone generalization through LPIPS measurements and qualitative examples.
- Single-view evaluation: Single-DP-view evaluation reports average LPIPS separately for the left and right views.
- Single-view evaluation: The single-view blur is modeled as a half-disc point spread function, whereas combining both views produces a blur kernel closer to a full-disc kernel.
- Aperture robustness: Robustness testing evaluates DPDNet on images captured with different aperture settings using average LPIPS.
- Aperture robustness: LPIPS values in the robustness test are lower than those in the main-paper evaluation because f/10 and f/16 produce less defocus blur than f/4.
- Smartphone results: Smartphone examples qualitatively show DPDNet deblurring Pixel-camera images and generalizing to smartphone captures.