Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries

1. Introduction

• Depth discontinuities are difficult to preserve because boundary pixels may belong to either surface, and conventional regression losses can smear depth across them.Maintaining discontinuities matters for object shape and pose estimation.Source: §1 · p. 1

2. Related Works

• Prior approaches address depth mixing or boundary recovery, but some require substantial computation or memory, rely on sparse boundary labels, or have inference-time and generalization issues.The cited approaches include depth coefficients [12], spatial propagation [19], and semantic or depth-map priors [40] [7] and [23].Source: §1 · pp. 1–2

1. Introduction

• TWISE represents foreground and background depths separately at ambiguous boundaries, then fuses them into one depth estimate.The method is presented as a more efficient alternative to many-channel depth representations and is evaluated on KITTI, NYU2, and Virtual KITTI.Source: §1 · p. 2
• Asymmetric losses bias the two surface estimates toward foreground and background, while a fusion channel selects or blends them into a final estimate.Source: §1 · p. 2, §2 · p. 3
• The paper reports superiority on several KITTI metrics and competitive inference times, while identifying MAE as more consistent than RMSE when ground truth contains outliers.The introduction also states that the claim is validated with extensive experiments on VKITTI.Source: §1 · p. 2

2. Related Works

• The paper contrasts its single-view two-layer depth estimation with layered representations built from multiple viewpoints or depth maps.Its boundary illustration compares smeared depth from with fused estimates and foreground/background surface slices.Source: §2 · pp. 2–3

3. Methodology

• The method separates depth completion into twin-surface extrapolation and boundary fusion, using color information to help select a single depth.The foreground surface extends beyond object boundaries, while the background surface extends behind the occluding object.Source: §3 · p. 3

3.1. Ambiguities and Expected Loss

• Ambiguity means multiple scene depths are consistent with sparse measurements; the method models it explicitly, while treating pixel ambiguities independently.Expected loss is used to predict how a network trained on representative data behaves at ambiguous pixels.Source: §3.1 · pp. 3–4

3.2. Asymmetric Linear Error

• ALE and RALE are reflected asymmetric generalizations of absolute error; their negative and positive error sides receive different weights, controlled by γ.Both reduce to absolute error when γ = 1; the analysis restricts γ ≥ 1.Source: §3.2 · p. 4

3.3. Foreground and Background Estimators

• At binary ambiguities, ALE estimates the foreground surface and RALE estimates the background; increasing γ lowers the foreground probability needed to favor the foreground depth.Away from discontinuities and at sparse input pixels, the estimated foreground and background depths are nearly identical.Source: §3.3 · p. 4

3.4. Fused Depth Estimator

• A per-pixel weight combines the foreground and background estimates, with the fusion loss favoring an endpoint when the pixel is assumed to have one of two depths.The network predicts σ per pixel to choose foreground, background, or a combination.Source: §3.4 · pp. 4–5

3.5. Depth Surface Representation

• The network predicts three channels for foreground depth, background depth, and fusion weight, then combines them into one depth estimate.The method combines the component losses into one objective.Source: §3.5 · p. 5

3.6. Implementation Details

• The implementation adds three output channels and the proposed loss to a multistack network, with multiscale supervision during training.Training proceeds for 30 epochs, progressively reducing supervision at half and quarter resolution.Source: §3.6 · p. 5

4.1. Results

• On KITTI, TWISE outperforms the compared methods in MAE and iMAE, while NLSPN has the lowest RMSE at the expense of inference time.The exception is RMSE, the metric used to rank methods on the KITTI leaderboard.Source: §4.1 · p. 6
• Qualitative comparisons show sharper object boundaries with TWISE than the compared methods, whose estimates exhibit boundary mixing, blur, holes, jagged edges, or streaking.The comparison covers poles, trees, cars, and traffic signs at close and long ranges.Source: §4.1 · p. 6, §4 · p. 6
• As foreground-background ambiguity increases, the fused estimator is biased toward the foreground surface, which the authors attribute to denser supervision in close-up regions.The reported ambiguity is associated with a larger depth gap between surfaces.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
• On NYU2, TWISE ranks second in all standard metrics, and the authors report generalization to indoor scenes.They also report inference on KITTI is 10× faster than NLSPN [19].Source: §4.1 · p. 7

4.2. Ablation Studies

• Across two backbone networks, TWISE performs better than standard L1 on MAE and TMAE; the authors describe the loss as network agnostic.They interpret its MAE and TMAE performance as a positive side effect of addressing boundary smearing.Source: §4.2 · p. 7
• Fixing σ at 0.5 averages the two surfaces and gives decent MAE and RMSE, while binary selection performs worse; σ also learns ineffectively without color input.The authors conclude that high-resolution imagery helps σ resolve boundary ambiguities.Source: §4.2 · pp. 7–8
• With γ = 2.0, the authors choose a compromise: smaller γ favors interpolation metrics but raises RMSE, while larger γ increases surface separation and makes interpolation harder.At γ = 1, the loss is equivalent to MAE.Source: §4.2 · p. 8
• Under increasing input sparsity, TWISE generalizes better and has fewer errors across metrics than compared methods; on Virtual KITTI, MAE rankings transfer from semi-dense to clean GT, unlike RMSE rankings.The Virtual KITTI analysis also reports MAE as smooth and monotonic across the two GT settings, whereas RMSE zigzags.Source: §4.2 · p. 8, Table 6 · p. 8, Figure 9 · p. 8

5. Conclusion

• TWISE combines foreground and background depth estimates with a fusion channel to produce a single surface estimate while better maintaining step-wise depth discontinuities.The authors attribute its state-of-the-art depth estimates to simplifying the learning of depth discontinuities.Source: §5 · p. 8
• The authors’ analysis suggests MAE is superior to RMSE for ranking depth completion methods when ground-truth datasets are noisy.Source: §5 · p. 9
• Far-away depth estimates remain a target for improvement because learning there suffers from sparse ground-truth pixels.Source: §5 · p. 9

• The listed authors are Saif Imran, Xiaoming Liu, and Daniel Morris, affiliated with Michigan State University.Source: Supplementary Material · p. 11
• The listed contact addresses use the msu.edu domain.Source: Supplementary Material · p. 11
• The document is identified as arXiv:2104.02253v3, dated 25 July 2021.Source: Supplementary Material · p. 11

Abstract

• The supplement provides further insights into results comparing the method with state-of-the-art methods.Source: Abstract · p. 11
• It presents evidence of boundary outliers in KITTI semi-dense ground truth and discusses their effect on depth-completion performance.Source: Abstract · p. 11
• It discusses the KITTI and Virtual KITTI data-generation process used for the main-paper ablation study.Source: Abstract · p. 11

1. Relative Error Maps

• TWISE errors often arise from boundary pixels where foreground or background depth is selected incorrectly, and these large-error outliers are strongly weighted by RMSE.The figure compares TWISE and MultiStack error maps for absolute error and squared error.Source: §1 · p. 11, Figure 1 · p. 11
• The 200-image validation analysis reports that TWISE estimates depth better on most pixels than MultiStack, apart from a few boundary outliers.Source: §1 · p. 11
• Across pixels, TWISE has less error than MultiStack for about 2.70 × 10^6 pixels, versus about 6,100 where MultiStack is better; the average image in this set shows 13,500 versus 31 winning pixels, respectively.Source: §1 · p. 12
• The authors report substantially larger TWISE errors around object boundaries than inside objects in their region analysis.The analysis uses KITTI semantic segmentation labels on 140 images and defines edge pixels as within three pixels of object boundaries.Source: §1 · p. 12, Table 1 · p. 12
• Incorrect KITTI ground-truth depth labels are another source of outliers, and losses sensitive to outliers such as MSE can be negatively affected by noise.Source: §1 · p. 12

2. Outlier Errors and Analysis on KITTI Semi-Dense GT

• Boundary depth labels can spread foreground and background depths across poles, signs, and trees in KITTI semi-dense ground truth.The paper identifies this as an additional source of noise alongside larger errors on dynamic objects and large-range pixels reported by Uhrig.Source: §2 · p. 12
• The identified sources of semi-dense ground-truth outliers are noisy IMU rotation and translation, camera–LiDAR timing mismatch, stereo-global-matching consistency-check artifacts, and accumulated points from dynamic objects.The analysis introduces these as a variety of causes.Source: §2 · p. 12
• Metric-unit evaluation shows that noisy semi-dense ground truth can have significant errors, and lowering its RMSE may encourage learning its inherent noise.The authors convert disparity errors to depth errors using KITTI intrinsics and suggest the RMSE implication as a possibility.Source: §2 · p. 12

3. Sparse Patterns in KITTI

• TWISE is reported to generalize better than other state-of-the-art methods across sparsity levels.The section introduces this claim before describing how the lower-resolution patterns are made.Source: §3 · p. 12
• The simulated row-subsampling patterns reflect structured sparse-sensor inputs rather than uniform subsampling.The section contrasts real-world structured LiDAR patterns with uniform subsampling reported by Ma et al..Source: §3 · p. 13
• The 32R, 16R, and 8R patterns are formed by selecting evenly spaced rows from KITTI’s 64R LiDAR data, split by azimuth angle and projected into the image.The method simulates lower-resolution LiDARs; Figure 5 shows 64R, 32R, 16R, and 8R patterns.Source: §3 · p. 13, Figure 5 · p. 13

4. Network Architecture

• TWISE uses Li et al. ’s network as its backbone.The architecture is based on the network described in.Source: §4 · p. 13
• The modified final layer outputs three channels: foreground estimate d1, background estimate d2, and σ.Figure 7 depicts the three-channel output.Source: §4 · p. 13
• The three-channel strategy is repeated in the hourglass networks at all three multi-resolution levels.Further network details are referred to Li et al..Source: §4 · p. 13

5.1. VKITTI Results on Different Weathers

• Performance is largely maintained across Virtual KITTI weather conditions, with minor degradations in fog and rain.The study evaluates a model trained on the ‘clone’ set under different weather conditions; Table 3 reports the weather results.Source: §5.1 · p. 13, Table 3 · p. 14

5.2. Creating Semi-Dense and Sparse Depth from Dense VKITTI GT

• Sparse Virtual KITTI depth is made by LiDAR-like azimuth-elevation subsampling and cross-camera projection, while semi-dense supervision simulates KITTI-style accumulated points and noisy IMU extrinsics.The process also suppresses supervision depths beyond 90 m; Figure 6 compares sparse, semi-dense, and dense examples.Source: §5.2 · p. 14, Figure 6 · p. 14

5.3. Relation to KITTI GT by Outliers

• With outliers defined as depth errors greater than 1 m, KITTI semi-dense ground truth has outliers at 4.4% of inlier depths; adding extrinsic noise makes Virtual KITTI semi-dense ground truth more comparable to KITTI’s.Table 4 compares the semi-dense errors; its caption notes higher Virtual KITTI RMSE from dense far-range pixels versus sparse KITTI stereo data.Source: Table 4 · p. 14, §5.3 · p. 15

• The supplementary video renders point clouds from depth maps estimated by TWISE and MultiStack.Source: §6 · p. 15
• MultiStack’s point cloud contains significantly more mixed-depth pixels than TWISE’s.Source: §6 · p. 15
• Floating depth pixels in the point clouds are the visual cue for comparing mixed-depth artifacts.Source: §6 · p. 15