Source-linked AI summary
Learning Joint 2D-3D Representations for Depth Completion
Yun Chen, Bin Yang, Ming Liang, Raquel Urtasun
TL;DR
Depth completion must infer dense depth from ambiguous RGB cues and noisy, sparse depth observations. The paper introduces a block that learns 2D and 3D features with separate convolutions and repeatedly fuses them in image space, outperforming prior methods on KITTI.
Problem
Depth completion remains difficult because RGB depth is inherently ambiguous and sparse depth observations are noisy and uncertain.
Method
The method stacks 2D-3D fuse blocks that learn multi-scale 2D features and continuous-convolution 3D features, then fuse them in image space.
Results
The approach outperforms all previous state-of-the-art methods in KITTI depth-completion RMSE, despite training from scratch on KITTI data only.
Takeaways & Limitations
Multi-level fusion of complementary 2D appearance and 3D geometric representations provides a joint representation for depth completion.
Takeaways & Limitations
The reported approach uses a single-task loss rather than multi-task objectives and is evaluated on KITTI depth completion.
Abstract
from arXiv · showhide
In this paper, we tackle the problem of depth completion from RGBD data. Towards this goal, we design a simple yet effective neural network block that learns to extract joint 2D and 3D features. Specifically, the block consists of two domain-specific sub-networks that apply 2D convolution on image pixels and continuous convolution on 3D points, with their output features fused in image space. We build the depth completion network simply by stacking the proposed block, which has the advantage of learning hierarchical representations that are fully fused between 2D and 3D spaces at multiple levels. We demonstrate the effectiveness of our approach on the challenging KITTI depth completion benchmark and show that our approach outperforms the state-of-the-art.
1. Introduction
Depth completion estimates dense depth from sparse LiDAR observations and a dense RGB image, addressing ambiguity in image-based depth and uncertainty in sparse measurements. The paper proposes multi-level fusion of 2D and 3D representations and reports superior KITTI performance.
- Motivation: Depth completion combines sparse LiDAR depth with a dense RGB image to estimate dense depth more affordably than high-end dense depth sensors.Sparse active-sensor measurements can be inaccurate near object boundaries.
- Challenge: Image-only depth estimation is ambiguous, while sparse depth observations contain noise and uncertainty, making completion non-trivial.
- Existing Limitations: 2D-only methods struggle with precise 3D geometry after projection, whereas 3D-only methods fail to exploit dense appearance cues.
- Contribution: The proposed 2D-3D fuse block learns domain-specific 2D and 3D representations and fuses them back in image space at multiple network levels.It uses multi-scale 2D convolutions and continuous convolutions on 3D points.
- Results: The method outperforms previous state-of-the-art methods in KITTI depth completion RMSE, including methods using external data or multi-task learning.The model is trained from scratch using KITTI training data only.
2. Related Work
Prior depth-completion and RGBD representation methods variously rely on 2D projections, 3D representations, or limited fusion. The proposed block instead supports multi-level fusion of domain-specific 2D and 3D features in image space.
- Depth Estimation from RGB: Earlier RGB depth-estimation methods used probabilistic graphical models or CNNs, but image ambiguity limits high-quality dense depth.
- Depth Completion from RGBD: Depth completion methods combine sparse depth with images or learn from sparse input, while differing in how they represent and process the data.
- RGBD Representation Learning: RGBD representation methods have combined 3D volumes, multi-view projections, PointNet features, and image features using varied fusion strategies.
- Proposed Representation: The proposed fuse block uses multi-scale 2D convolution and 3D continuous convolution branches whose outputs are combined through a fusion layer.
- Proposed Representation: Unlike methods that fuse sensor features once, this approach fuses 2D and 3D features back into image space densely at multiple levels.
3. Learning Joint 2D-3D Representations
The paper introduces a modular 2D-3D fuse block that learns appearance and geometric features in separate branches, fuses them in image space, and supports stacked depth-completion networks. Continuous convolution supplies 3D geometric neighborhoods while multi-scale 2D convolution captures image features, enabling multilevel fusion for dense depth prediction.
- Network Construction: Stacking 2D-3D fuse blocks builds a depth-completion network whose joint representations are fully fused at multiple levels and whose size can be modified for performance-efficiency trade-offs.The example architecture combines RGBD-derived inputs, processes them through N stacked blocks, and predicts dense depth after output-stage convolution and upsampling.
- 2D-3D Fuse Block: The 2D-3D fuse block processes a 2D feature map and projected 3D points through separate sub-networks, then returns a same-resolution 2D feature map.The block uses a multi-scale 2D convolution network and a 3D continuous convolution network, with outputs combined by a fusion layer.
- 2D-3D Fuse Block: Multi-scale 2D convolution extracts appearance features using parallel full-resolution and downsampled-then-upsampled branches.The branches use conv(3, 1, C) and conv(3, 2, C), conv(3, 1, C), upsample(2), followed by element-wise summation.
- 2D-3D Fuse Block: Continuous convolution learns 3D geometric features by selecting K nearest neighbors in Euclidean space and parameterizing neighbor weights with an MLP.Its implementation performs a weighted sum of transformed point features and separates depth-wise convolution from a linear 1 × 1-like transformation to reduce memory and computation overhead.
- 2D-3D Fuse Block: The two branches capture complementary correlations because 2D convolution follows image-grid neighborhoods whereas continuous convolution follows exact 3D geometric correlation.This geometric neighborhood selection is intended to help represent non-smooth shapes at object boundaries, where depth estimation is difficult.
- Learning and Inference: The default training objective averages a weighted sum of ℓ2 and smooth ℓ1 losses over pixels with depth labels.The coefficient γ controls the balance between the two losses, while the smooth ℓ1 definition uses a quadratic regime for errors below 1 and a linear regime otherwise.
4. Experimental Evaluation
Experiments on KITTI evaluate the proposed model against prior methods and through ablations of its data, architecture, network configuration, and objective function. The results report state-of-the-art performance, favorable accuracy–size trade-offs, complementary block components, and sharper qualitative boundaries.
- Experimental Setting: KITTI experiments compare depth-completion methods on held-out data, using RMSE as the primary ranking metric alongside MAE, iRMSE, and iMAE.The benchmark provides 86,898 training, 1,000 validation, and 1,000 testing frames; each frame pairs RGB imagery with sparse LiDAR depth and dense ground truth.
- Comparison with State-of-the-art: The proposed model surpasses previous state-of-the-art methods on KITTI RMSE while training only on KITTI data, including methods using external data or multi-task learning.The best single model stacks N = 12 blocks with C = 64 feature channels.
- Ablation Studies: Multi-scale 2D convolution and continuous convolution are complementary, while stride 1, stride 2, and continuous-convolution branches are all necessary for the fuse block.The branches respectively provide local features, larger-field propagation and global features, and 3D geometric features.
- Ablation Studies: Varying feature-channel width C and block depth N produces accuracy–model-size trade-offs, with the proposed model achieving better performance at smaller size than other methods.Figure 5 summarizes the comparison across network configurations.
- Ablation Studies: The objective function controls metric-specific performance: ℓ2 favors RMSE, smooth ℓ1 favors MAE, and a two-stage loss balances all four metrics.The best single model uses ℓ2 loss for 100 epochs, followed by the sum of ℓ2 and smooth ℓ1 losses for 50 epochs.
- Qualitative Results: Qualitative comparisons show cleaner and sharper object boundaries in near and distant regions, especially where 2D convolution has limited appearance clues.The comparison uses NConv-CNN and Sparse2Dense on the KITTI test set.
5. Conclusion
The paper concludes that its architecture fuses 2D and 3D representations at multiple levels and demonstrates effectiveness on KITTI depth completion. It identifies fusing additional sensors and reasoning over video sequences as future directions.
- Experiments on KITTI depth completion demonstrate that the approach outperforms state-of-the-art methods.
- The proposed architecture fuses information between 2D and 3D representations at multiple levels.