PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Depth-of-Field with a Single-Camera Mobile Phone

Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch, Marc Levoy

arXiv:1806.04171v1cs.CVcs.GR

TL;DR

Standard mobile cameras cannot optically produce the shallow depth-of-field used to isolate subjects, while existing alternatives may require extra hardware or camera motion. This paper combines person segmentation with dual-pixel depth when available to render synthetic defocus on single-camera phones, and all six users preferred its results in a 64-image study.

  • Problem

    Standard mobile cameras capture nearly all-in-focus images, limiting shallow depth-of-field without additional hardware or camera motion.

  • Method

    The system combines person-and-accessory segmentation with dense dual-pixel depth when available to render synthetic shallow depth-of-field images through three mobile-camera pipelines.

  • Results

    All 6 users preferred the method over two ablations and two baselines across 64 images, and all three pipelines ran in less than 4 seconds.

  • Takeaways & Limitations

    The modular system supports synthetic shallow depth-of-field on single-camera mobile phones across scenes with people or objects and with or without dual-pixel hardware.

  • Takeaways & Limitations

    Optical aberrations and pixel-split variation can cause blur kernels to vary significantly across the frame, limiting the imaging model’s accuracy.

Abstract

from arXiv · show

Shallow depth-of-field is commonly used by photographers to isolate a subject from a distracting background. However, standard cell phone cameras cannot produce such images optically, as their short focal lengths and small apertures capture nearly all-in-focus images. We present a system to computationally synthesize shallow depth-of-field images with a single mobile camera and a single button press. If the image is of a person, we use a person segmentation network to separate the person and their accessories from the background. If available, we also use dense dual-pixel auto-focus hardware, effectively a 2-sample light field with an approximately 1 millimeter baseline, to compute a dense depth map. These two signals are combined and used to render a defocused image. Our system can process a 5.4 megapixel image in 4 seconds on a mobile phone, is fully automatic, and is robust enough to be used by non-experts. The modular nature of our system allows it to degrade naturally in the absence of a dual-pixel sensor or a human subject.

1 INTRODUCTION

We introduce a fully automatic system that synthesizes shallow depth-of-field images on a wide range of mobile cameras with a single button press, combining person segmentation with dual-pixel depth when available. The system supports natural fallback modes for selfies and object photos while rendering depth-dependent blur efficiently on mobile devices.

  • 1 INTRODUCTION: The system enables untrained photographers to capture convincing shallow depth-of-field images with a single button press, without moving the camera, extra controls, or specialized hardware.Its requirements include fast processing, high-resolution output, plausible blur with the subject sharp, and operation across a wide range of scenes.
  • 1 INTRODUCTION: The method combines neural person-and-accessory segmentation with dual-pixel imagery, which provides a two-sample light field with an approximately 1 millimeter baseline when available.The segmentation network uses the image and face position to infer pixels belonging to people or held objects, while dual-pixel data supplies depth cues.
  • 1 INTRODUCTION: The system degrades naturally by using segmentation alone for front-camera selfies, augmented segmentation and dual-pixel depth for rear-camera people photos, and dual-pixel depth alone for object photos.Uniform background blur is sufficient for typical selfies with distant backgrounds, whereas rear-camera scenes require depth variation to avoid unnatural blur.
  • 1 INTRODUCTION: Dual-pixel disparities drive synthetic shallow-depth blur after burst alignment, stereo estimation, and optical-aberration calibration, while layered disk splatting and compositing produce smooth depth-dependent rendering.Synthetic noise is added to defocused regions because rendering reduces camera noise next to in-focus areas that retain it.
  • 1 INTRODUCTION: A forced 1.5× digital zoom improves portrait perspective by moving photographers farther from subjects and reduces processing from 12 megapixels to 5.4 megapixels.The wide field of view of typical mobile cameras otherwise encourages close shooting that can distort faces.

2 RELATED WORK · 3 PERSON SEGMENTATION

The paper situates synthetic shallow-depth-of-field rendering among parallax- and monocular-depth methods, then presents a mobile-efficient person-segmentation system that improves accuracy while requiring far fewer computations. Its pipeline combines large-scale training data, staged learning, face-guided inference, and edge-aware mask refinement.

  • 2 RELATED WORK: 2 RELATED WORK: Prior single-camera methods use deliberate translation or accidental hand shake for parallax, but often fail when camera motion provides insufficient parallax.Monocular methods instead infer depth through inverse rendering or supervised learning, but the problem remains highly underconstrained.
  • 2 RELATED WORK: 2 RELATED WORK: Shen et al. [2016a] and [2016b] segment people and blur backgrounds under simplified depth assumptions, but such approaches are computationally expensive and fail without people or in complex scenes.Their later method adds a differentiable matting layer, while Zhu et al. [2017] use smaller networks.
  • 3 PERSON SEGMENTATION: 3 PERSON SEGMENTATION: The system segments people and accessories both independently and to augment noisy dual-pixel disparity, using a fast and accurate mobile model with dedicated training and edge-aware filtering contributions.The method targets the substantial fraction of mobile images depicting people.
  • 3.1 Data Collection: 3.1 Data Collection: Training uses 122k Flickr images with polygon masks plus 73k pose-labeled images containing 227k person instances, and successive data improvements improved defocused-portrait quality.The authors emphasize varied poses, removal of poor images, and cleaning inaccurate masks.
  • 3.2 Training: 3.2 Training: The architecture stacks three U-Nets with intermediate supervision, takes a 4-channel 256 × 256 RGB-plus-face-location input, and uses staged segmentation and pose training with error-based pruning.Training used 660k images later pruned to 511k, an evaluation set of 1700 images, and one month on 40 GPUs across 10 machines.
  • 3.3 Inference; 3.4 Edge-Aware Filtering of a Segmentation Mask: 3.3 Inference and 3.4 Edge-Aware Filtering: Face-detector rectangles seed segmentation, while low-resolution masks are upsampled and refined using image-edge guidance to improve boundaries efficiently.Faces must exceed specified relative-size thresholds to reduce bystander inclusion; filtering also refines approximate training masks and uses a bilateral solver with subsequent smoothing and joint bilateral upsampling.
  • 3.5 Accuracy and Efficiency: 3.5 Accuracy and Efficiency: 3.07 Giga-flops per inference versus 607 for PortraitFCN+ and 3160 for Mask-RCNN, while the model achieves higher accuracy than PortraitFCN+ and inferior Mask-RCNN results.The comparisons use mean IoU on evaluation datasets; Mask-RCNN is also described as significantly larger.

4 DEPTH FROM A DUAL-PIXEL CAMERA

Dual-pixel sensors provide two synchronized, same-exposure views with approximately 1 mm baseline, enabling dense disparity estimation despite their narrow stereo geometry. The system denoises burst frames, estimates and corrects disparity, and uses segmentation cues to keep subjects in focus while synthesizing shallow depth of field.

  • Dual-pixel properties: DP views share exposure, white balance, timing, and a sensor reference frame, while their small baseline limits occlusions and disparity searches to a few pixels.The same single sensor makes standard stereo rectification unnecessary, but the narrow baseline also makes correspondence challenging.
  • Disparity estimation: The algorithm temporally denoises DP bursts, matches the two views, and applies a spatially varying correction for lens aberrations before producing dense depth.Correspondences extend Anderson et al. [2016], and the correction compensates for depth-dependent disparity changes caused by lens aberrations.
  • Denoising DP frames: Burst denoising recovers background disparity in low-light scenes, enabling the background to be blurred when single-frame DP data is too noisy.Six aligned and robustly averaged frames provide sufficient signal to distinguish a flower from its farther background.
  • Disparity estimation: Each 8 × 8 DP tile is matched over shifts from −3 to 3 pixels using SSD, with a quadratic fit yielding a sub-pixel disparity estimate.Tile confidence uses SSD loss, horizontal gradients, competing minima, and related heuristics.
  • Subject-aware depth: For people, the method assigns the interior of the segmentation mask the weighted average face disparity, keeping the subject in focus and reducing objectionable subject-depth errors.Pixels with CNN output Mc(x) > 0.94 are treated as interior pixels; two people at very different depths may consequently both appear in focus.

5 RENDERING

The rendering stage synthesizes shallow depth of field by applying disparity-dependent disk blurs to disparity subimages, then compositing, gamma-correcting, and adding noise. It adapts focus and blur strength for pleasing subject isolation while using efficient approximations for mobile processing.

  • Rendering pipeline: The pipeline precomputes disk kernels, applies them to disparity-range subimages, composites the results, gamma-corrects, and adds synthetic noise.Rendering uses the smoothed disparity and linear-color input image, which helps preserve highlights in defocused regions.
  • Focus and blur mapping: The method selects focus disparity from a well-focused detected face, user tap, or autofocus, then preserves a band around it with zero blur radius.This extended sharp region keeps an entire subject in focus and makes the effect easier for novices to control.
  • Focus and blur mapping: Blur radius is adjusted by focus distance, capped at rmax = 30 pixels, and applied through overlapping disparity bands for smoother transitions.The blur-strength function compensates for smaller disparity ranges in scenes focused farther away; adjacent bands typically overlap by 25%.
  • Efficient disk blurring: To avoid artifacts from spatially varying kernels, large disk blurs use sparse-gradient processing rather than prohibitively expensive full-range gather operations.Naive single-pass or gather-based approaches can create foreground-background color halos, whereas the proposed method targets efficient scatter-like behavior.
  • Efficient disk blurring: Blurring at half resolution provides an 8× speedup with little perceptible quality loss before upsampling to full resolution.The blur radius is also halved to preserve the equivalent appearance.

6 RESULTS

Across three pipelines, the system is preferred by all users in a 64-image study, remains robust to complementary segmentation and disparity errors, and processes 2688 × 2016 images in under 4 seconds. Its production deployment still has failure modes involving face detection, backgrounds, and non-person scenes.

  • 6 RESULTS: The three pipelines cover people with dual-pixel hardware, objects with dual-pixel hardware, and people without dual-pixel data using segmentation alone.DP + Segmentation integrates disparity and a segmentation mask; DP only omits the mask; Segmentation only applies without dual-pixel hardware.
  • 6 RESULTS: All users preferred the proposed method over two ablations, Shen et al. [2016a], and Barron et al. [2015] across 64 images.The study involved 6 people comparing 5 algorithms; the ablations were consistently the second and third most preferred.
  • 6 RESULTS: Face-detection failures can blur a person into the background, while disparity errors can remain in backgrounds or images without people.Dual-pixel data often mitigates face-detection failures on people at the subject’s depth, but accurate face detections remain important without DP data.
  • 6 RESULTS: Combining segmentation masks with disparity makes the system robust to cushion-blurring segmentation errors and incorrect depth estimates on textureless arms.Segmentation-only methods blur a cushion near the subject, while depth-based methods misestimate disparity on a dark, untextured arm; the combined method avoids both errors.
  • 6 RESULTS: 325 examples at 2688 × 2016 resolution all process in less than 4 seconds on a high-end mobile phone.The system was implemented in Halide and manually scheduled for an eight-core CPU with four cores at 2.35 GHz and four at 1.9 GHz.

7 DISCUSSION AND FUTURE WORK

The system combines person segmentation and dense dual-pixel depth to compute synthetic shallow depth-of-field images on single-camera mobile phones, and is marketed as “Portrait Mode” on Google Pixel smartphones. Future work includes segmenting more subject types and exploring non-photorealistic foreground–background separation.

  • 7 DISCUSSION AND FUTURE WORK: The system combines person segmentation with dense dual-pixel depth to produce synthetic shallow depth-of-field images on phones with only one camera.It was presented across varied examples and compared with other synthetic shallow depth-of-field methods.
  • 7 DISCUSSION AND FUTURE WORK: The method is marketed as “Portrait Mode” on Google Pixel and Pixel XL smartphones.
  • 7 DISCUSSION AND FUTURE WORK: Future work could extend segmentation to pets, food, and other photographed objects, while exploring desaturation, stylization, or other non-photorealistic foreground–background separation.These alternatives aim to draw attention to the subject rather than realistically simulate depth-of-field.

1 USEFULNESS OF SYNTHETIC DATA

Synthetic data augmentation improves segmentation-network accuracy, reducing error rates compared with training without augmentation. The evaluation emphasizes mask-boundary accuracy using the mean sum of squared error between gradients of predicted and reference masks.

  • 1 USEFULNESS OF SYNTHETIC DATA: Synthetic data augmentation decreases segmentation error rates compared with training without augmentation, as shown by evaluation accuracy over training iterations in Fig. 1.The metric emphasizes mask boundaries through the mean sum of squared error between gradients of the predicted and reference masks.

2 DISPARITY COMPUTATION DETAILS

The system compensates for brightness differences between dual-pixel views through local normalization, then computes disparity using a smooth optical-flow method adapted for calibration images and horizontal motion.

  • Calibration disparity: For planar calibration targets, the method applies Liu’s optical flow [2009] because disparity should be smooth without depth edges.The dual-pixel images are normalized before optical flow is applied.
  • Brightness normalization: Local normalization reduces dual-pixel brightness variations by matching each view’s local mean and standard deviation in 9 × 9 windows.Each image is normalized by subtracting its local mean and dividing by its local standard deviation plus 0.001; Fig. 2 visualizes the resulting reduction.
  • Calibration disparity: The calibration flow is restricted to horizontal motion and uses an L2 smoothness loss with an L1 data loss.

3 DISPARITY TO DEPTH CALCULATION

This section derives the relationship between dual-pixel disparity and scene depth, showing that disparity is approximately proportional to signed blur and linearly related to inverse depth under typical smartphone-camera conditions.

  • 3 DISPARITY TO DEPTH CALCULATION: Disparity d is approximately proportional to signed blur size ¯b, and combining the geometric equations yields a linear relationship between disparity d and inverse depth 1/D.The approximation is justified because the focus distance exceeds 10 cm while a typical smartphone focal length is roughly 5 mm.
  • 3 DISPARITY TO DEPTH CALCULATION: Using the thin-lens equation and similar-triangle geometry, the derivation relates object distance, focus distance, lens parameters, sensor position, and blur size.An object at distance D forms a sharp image at distance D_i, while defocus produces a blur determined by the lens geometry.
  • 3 DISPARITY TO DEPTH CALCULATION: Controlled dual-pixel experiments show that aligning and merging the data reduces disparity noise, with the lowest depth noise occurring in scene regions that are in focus.The experiments vary light level and target depth across several focus distances.

4 CONTROLLED EXPERIMENTS ON DP DATA

Controlled experiments on dual-pixel data measured disparity accuracy across lighting and focus conditions. Averaging six aligned frames reduced noise, while depth noise was lowest when the target was in focus.

  • Experimental setup: A fronto-parallel textured target was imaged at varying light levels and focus distances, with disparity standard deviation over the central field of view measuring error.The central fourth of the field of view was used to avoid calibration issues because disparity should remain constant across images.
  • Noise and light levels: Two times less noisy at 5 lux and 1.5 times less noisy in brighter scenes, six aligned and averaged frames outperformed single-frame disparities (Fig. 3(a)).The comparison used disparity values from six aligned frames versus one frame.
  • Focus distance: Depth noise was lowest when the textured target was in focus (Fig. 3(b)), showing sensitivity to focus-dependent image texture.This sensitivity presents an additional challenge compared with wide-baseline stereo, which can rely on coarser image content.

5 BLUR PARAMETERS

The rendering parameters primarily control blur aesthetics and are tuned empirically, with limited sensitivity to their exact values. The system adjusts blur scaling, in-focus disparity ranges, segmentation-boundary transitions, and convolution method selection to balance visual quality and efficiency.

  • Blur parameters were tuned by visualizing thousands of rendered images and mainly affect blur strength and depth of field, while the system remains relatively insensitive to their exact values.
  • The blur scale κ(·) combines image-size scaling, focus-distance disparity amplification, and a misfocus correction factor.
  • To reduce segmentation-boundary artifacts, frontal pixels with d(x) − d_focus > 0 receive 0.6× the usual blur scale.
  • Pixels within [d_focus − d∅, d_focus + d∅] remain unblurred, with d∅ scaled by κ and reduced for photos containing people.
  • The system switches from brute-force convolution to accelerated gradient-domain blur at a 2.75-pixel working-resolution radius, balancing computation against kernel-discretization error.

6 NOISE GENERATION

The system injects noise into blurred regions to make synthetic defocus appear natural, using a non-parametric approach because the camera’s complex imaging and post-processing pipeline resist accurate analytical noise modeling.

  • 6 NOISE GENERATION: Synthetic defocus requires noise injection in blurred regions, but the camera pipeline is too complex for an accurate parametric model, so the system adopts a non-parametric approach.The paper notes that common models such as Gaussian noise do not adequately describe the naturally occurring noise produced by the imaging and post-processing pipeline.

7 USER STUDY

The user study compared the proposed method, its ablations, and prior techniques through pairwise preference judgments by six unaffiliated participants. Renderings were evaluated both holistically and through automatically selected boundary crops emphasizing inter-image differences.

  • User Study: Participants selected which of two side-by-side renderings they would prefer their camera to produce, using successive pairwise comparisons across scenes.The winning rendering was subsequently compared against another randomly selected algorithm’s rendering.
  • User Study: Each rendering was shown in full alongside three non-overlapping 128 × 128 crops selected for maximal inter-image variance and, when present, human-subject boundary coverage.This presentation supported both holistic and localized evaluation of scene composition.
Loading 1806.04171v1…