Learning Single Camera Depth Estimation using Dual-Pixels

1. Introduction

• Dual-pixel sensors capture two stereo-like images within one camera, but classic stereo matching can fail when disparity and focus interact.The two green pixel halves form separate views; Figure 1 illustrates differences between the views for out-of-focus background content.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2
• The paper analyzes dual-pixel optics, identifies an affine ambiguity in depth, and develops training procedures and losses for affine-invariant depth estimation.The resulting depth remains affine-ambiguous but can support depth ordering and defocus rendering.Source: §1 · p. 2
• 30% lower error rates than previous dual-pixel and monocular depth estimation approaches are reported for the proposed model.The dataset pairs in-the-wild dual-pixel imagery with alternate views for view supervision and multi-view depth estimation.Source: §1 · p. 2

2. Related Work

• Deep learning accelerated monocular depth estimation using RGB-D datasets, while multi-view geometry enabled self-supervised training from video or stereo [18] [20] [22] [42] [57].Earlier monocular approaches consolidated cues such as shading, texture, and contours.Source: §2 · p. 2
• Depth-from-focus methods estimate depth from focal stacks or learned models [26], and focus cues can also supervise monocular depth prediction [52].The paper notes that dual-pixel views have different point-spread functions and combine focus and stereo cues.Source: §2 · p. 2
• Light-field cameras sample angular as well as spatial information, but sacrifice spatial resolution for angular resolution.Light fields have also been used for depth recovery.Source: §2 · p. 2
• Dual-pixel cameras sample two angles while sacrificing negligible spatial resolution, offering a compromise between conventional cameras and light-field cameras.The passage notes their adoption in consumer cameras and space-constrained applications such as endoscopy.Source: §2 · p. 3

3. Dual-Pixel Geometry and Ambiguity

• Dual-pixel pixels integrate light from opposite aperture halves, creating stereo-like disparity that is proportional to defocus blur.The sum of both views forms the ordinary full-pixel image, and disparity depends on focus distance as well as baseline.Source: §3 · p. 3
• Unknown focus distance leaves dual-pixel depth with both scale and offset ambiguities, unlike stereo depth with only scale ambiguity when extrinsics are unknown.Different camera parameters and scene depths can produce the same disparity.Source: §3 · pp. 3–4
• Under paraxial and thin-lens approximations, disparity is affine-related to inverse depth, with coefficients depending on aperture diameter, focal length, and focus distance.The passage states that recovering metric depth from disparity requires knowing those camera parameters.Source: §3 · pp. 3–4
• Although absolute depth is unavailable from dual-pixel cues alone, affine-transformed depth preserves relative ordering and suits synthetic defocus rendering.The ambiguity can also be resolved using heuristics such as likely sizes of known objects.Source: §3 · p. 4

4. View supervision for Affine Invariant Depth

• View supervision warps a second RGB view using predicted depth and minimizes a differentiable per-pixel color difference, but affine ambiguity requires modified training losses.The warp uses estimated inverse depth and known camera poses.Source: §4 · p. 4

4.1. 3D Assisted Loss

• 3D Assisted Loss solves affine parameters a and b by differentiable least squares against confident ground-truth inverse depth, then applies standard view supervision.A sparse ground-truth depth map with a few confident samples suffices to solve for the two scalars.Source: §4.1 · p. 4

4.2. Folded Loss

• Folded Loss removes the need for ground-truth depth by optimizing per-example a and b jointly with network parameters.Its parameterization is intended to avoid degeneracies as a approaches zero.Source: §4.2 · p. 4

5. Data Collection

• Each central dual-pixel capture is accompanied by four simultaneous moderate-baseline views, enabling multi-view geometry to produce partial, confidence-tagged depth estimates.Two stereo algorithms are used, and high-confidence depth locations are described as accurate; the dataset includes varied indoor and outdoor scenes but excludes faces.Source: §5 · p. 5, Figure 4 · p. 5
• The dataset contains 3,573 scenes and 17,865 RGB and dual-pixel images captured using Google Pixel 2 or Pixel 3 phones.For each capture, all five images come from the same phone model.Source: §5 · p. 5
• Only the center camera's dual-pixel signal is used; RGB and dual-pixel images are processed at 1512×2016, while ground-truth depth is half-resolution.The depth range is converted to inverse depth using inverse perspective sampling from 0.2 to 100 meters.Source: §5 · p. 5
• Because one fixed camera calibration did not work across captures, structure from motion estimates each capture's intrinsics and extrinsics, leaving variable scale ambiguity.The authors state that this scale ambiguity is compatible with their training and evaluation procedures.Source: §5 · p. 5
• The 3D Assisted Loss tends to outperform Folded Loss on DPNet accuracy, with ground-truth depth fully constraining the ambiguity.The alternative folds optimization of affine parameters into the loss computation.Source: Table 1 · p. 6

6.1. Data Setup

• The dataset split contains 2,757 training images and 718 test images, with capture sessions kept entirely within one split.Evaluation uses a center crop and reports COLMAP-depth errors on 671 test images after SfM failed on 47.Source: §6.1 · p. 6

6.2. Training a Neural Net for Depth Prediction

• The experiments use VGG and lightweight DPNet architectures, with about 19.8 million and 0.24 million parameters, respectively.Their reported inference costs are about 295 billion and 5.5 billion flops, respectively.Source: §6.2 · p. 6
• View-supervision training combines DSSIM and Charbonnier losses at weights 0.8 and 0.2, and aggregates losses across multiple depth resolutions.For multiple neighboring views, it takes the per-pixel minimum difference, assuming each pixel is visible in at least one neighboring view.Source: §6.2 · p. 6

6.3. Evaluation Metrics

• Because dual-pixel depth is only recoverable up to an unknown affine transform, the authors evaluate with affine-invariant metrics rather than conventional depth metrics.They use weighted Spearman rank correlation and affine-invariant weighted MAE and RMSE.Source: §6.3 · p. 6

6.4. Folded Loss vs 3D Assisted Loss

• The 3D-assisted loss converges to a better solution than the folded loss, so the authors use it in subsequent experiments..Source: §6.4 · pp. 6–7

6.5. Comparison to Other Methods

• Affine-invariant training improves performance across tested models; DPNet with RGB+DP achieves the best results, and dual-pixel input significantly outperforms RGB-only input.The comparison also reports lower accuracy for [19] than for the authors’ model and many dual-pixel-trained baselines.Source: Table 2 · p. 7, §6.5 · pp. 7–8
• The affine-invariant loss also enables view-supervised monocular methods [20] [22] to use dual-pixel data, with stereo-view supervision adapted to the camera rig.Their adaptation replaces a one-dimensional bilinear warp with a two-dimensional warp based on camera intrinsics and extrinsics.Source: §6.5 · p. 7
• Figure 5 shows better DPNet results with RGB+DP than RGB alone, and with affine invariance than scale invariance.The figure also contrasts these outputs with [19] and [55]; the caption notes depth errors in [55] and limited scene generalization in DORN [19].Source: Figure 5 · p. 8

• The paper presents the first learning-based approach for estimating depth from dual-pixel cues.Source: §7 · p. 8
• The approach lets neural networks estimate depth from dual-pixel imagery despite the fundamental affine ambiguity.Source: §7 · p. 8
• The authors construct a large in-the-wild dataset pairing five-view RGB images with dual-pixel data and demonstrate accurate depth maps from their technique and prior view-supervised models.Source: §7 · p. 8

A. Derivation of Equation 2

• The setup places a point light source at depth Z and defines its image distance Zi relative to the lens.Source: A. Derivation of Equation 2 · p. 11
• Similar triangles are used to derive the blur size b.Source: A. Derivation of Equation 2 · p. 11
• Substituting Equation 9 into Equation 10 yields Equation 2 in the main paper.Source: A. Derivation of Equation 2 · p. 11

B.1. Depth from Multi View Stereo

• The plane-sweep pipeline selects depth from a filtered cost volume, while confidence is based on consistency across views and requires visibility in at least two other cameras.It sweeps 256 inverse-perspective-sampled planes from 0.2m to 100m; the two highest neighbor-consistency scores are multiplied for final confidence.Source: §B.1 · p. 11
• Figure 6 compares depth from the authors’ multi-view pipeline with COLMAP; the authors report more conservative confidence labels and avoidance of edge-fattening artifacts.Low-confidence regions are shown in white.Source: Figure 6 · p. 11

B.2. Preprocessing of RGB and Dual-Pixel Data

• RGB and dual-pixel data are resized and combined as a five-channel input, with the dual-pixel data quantized from square-rooted 10-bit raw values to 8 bits.The dual-pixel data is upsampled to match the downsampled RGB image dimensions.Source: §B.2 · p. 11

C.1. Affine Invariant Weighted Error

• AIWE(2) and AIWE(1) measure confidence-weighted inverse-depth error after optimizing over affine transformations, using least squares and five-iteration iteratively reweighted least squares, respectively.These correspond to affine-invariant weighted RMSE and MAE.Source: §C.1 · p. 12

C.2. Spearman’s Rank Correlation Coefficient

• The weighted Spearman metric measures ordinal depth correctness, is invariant to positive affine transformations, and reports 1 −|ρs| to handle negative scaling with lower values better.Ground-truth confidence weights the rank correlation.Source: §C.2 · p. 12

D. Model Architecture

• DPNet uses encoder and decoder blocks connected by skip connections, with predictions produced at five resolutions.The encoder blocks control downsampling; decoder blocks receive encoder outputs as skip connections.Source: D. Model Architecture · p. 12

E.1. VGG vs DPNet

• DPNet slightly outperforms the larger VGG model, which the authors attribute to VGG’s tendency to overfit.The models’ qualitative results are reported as very similar.Source: §E.1 · p. 13

E.2. Multi-View vs Stereo

• Training with all four alternative views outperforms training with stereo views alone.The comparison is reported in Table 3.Source: §E.2 · p. 13, Table 3 · p. 13

E.3. Generalization Across Devices

• On devices absent from training, performance degrades only slightly without per-device calibration.The extended test set contains seven devices not used to generate the training set, while training used three of the ten capture devices.Source: §E.3 · p. 13, Table 4 · p. 13

E.4. Additional Metrics

• With WRMSE evaluated after resolving affine ambiguity using the 1/3 and 2/3 percentiles, DP input is important and affine invariance helps learning with DP input.Table 5 reports this metric for a subset of the main-paper comparisons.Source: §E.4 · pp. 13–14, Table 5 · p. 13

E.5. Additional Comparisons

• The additional comparisons include the method’s top and worst five results under 1 −|ρs|; poor cases tend to have textureless surfaces or far-off objects.One particularly bad prediction collapses all depths to a single value through an incorrect affine mapping.Source: §E.5 · p. 14

E.6. Results on data from Wadhwa et al. [55]

• The authors use DPNet trained with DP input only, choosing the unknown affine mapping through the focus depth and amount of synthetic defocus to render.They apply bilateral filtering followed by joint bilateral upsampling to align depth maps with RGB edges before rendering.Source: §E.6 · p. 14
• The additional figures show further results, including the method’s best and worst five predictions ranked by 1 −|ρs|.The best cases tend to be textured scenes with close focus distances.Source: Figure 8 · p. 15, Figure 9 · p. 16, Figure 10 · p. 17
• The authors’ depth maps produce cleaner synthetic defocus than the maps from [55], whose errors include sharp background highlights, uneven background blur, and incorrect foreground blur.Figure 12 compares the rendered results.Source: Figure 12 · p. 19