PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Depth Completion with Calibrated Backprojection Layers

Alex Wong, Stefano Soatto

arXiv:2108.10531v2cs.CVcs.AIcs.LG

TL;DR

Sparse range points do not provide the dense depth map needed from an image, and ground-truth depth is often unavailable or expensive. KBNet uses calibration-aware layers and unsupervised training to infer dense depth, reporting gains across indoor and outdoor benchmarks and when test calibration differs.

  • Problem

    Sparse range measurements alone may be insufficient for planning, while ground-truth dense depth is often unavailable and prohibitively expensive.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    KBNet combines a sparse-to-dense module and calibrated backprojection layers, using camera intrinsics as input and photometric reprojection for unsupervised training.

    Source: §1 · p. 1, §2.2 · p. 4

  • Results

    13.7% and 8.8% average gains over the baseline [47] and state of the art [45], respectively, outdoors; indoors, gains are 51.7% and 30.5%, respectively; different calibrations yield 83% and 62% relative-error gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Using calibration as an input supports transfer to a different sensor platform at inference while geometric bias permits a smaller model that still achieves state of the art.

    Source: §4 · p. 9

    Incorrect camera calibration can backfire and yield distorted depth maps.

    Source: §4 · p. 9

Abstract

from arXiv · show

We propose a deep neural network architecture to infer dense depth from an image and a sparse point cloud. It is trained using a video stream and corresponding synchronized sparse point cloud, as obtained from a LIDAR or other range sensor, along with the intrinsic calibration parameters of the camera. At inference time, the calibration of the camera, which can be different than the one used for training, is fed as an input to the network along with the sparse point cloud and a single image. A Calibrated Backprojection Layer backprojects each pixel in the image to three-dimensional space using the calibration matrix and a depth feature descriptor. The resulting 3D positional encoding is concatenated with the image descriptor and the previous layer output to yield the input to the next layer of the encoder. A decoder, exploiting skip-connections, produces a dense depth map. The resulting Calibrated Backprojection Network, or KBNet, is trained without supervision by minimizing the photometric reprojection error. KBNet imputes missing depth value based on the training set, rather than on generic regularization. We test KBNet on public depth completion benchmarks, where it outperforms the state of the art by 30.5% indoor and 8.8% outdoor when the same camera is used for training and testing. When the test camera is different, the improvement reaches 62%. Code available at: https://github.com/alexklwong/calibrated-backprojection-network.

1. Introduction

The introduction frames depth completion as turning sparse range measurements and RGB into dense depth, then proposes calibration-aware, unsupervised learning. It reports gains across indoor and outdoor benchmarks, including when test calibration differs from training.

1. Introduction

  • Depth completion combines an image with sparse 3D range points to infer a depth value for every pixel, since range measurements alone may be insufficient for planning.The target is a dense map with depth at millions of image pixels rather than only the sensor's thousands of returns.Source: §1 · p. 1

1.1. Related Work and Contributions

  • KB layers use camera intrinsics to lift image and depth features into 3D positional encodings, making calibration an input that can change at inference.This contrasts with treating intrinsics as fixed parameters in the training loss.Source: §1 · p. 1, §1.1 · p. 3

1. Introduction

  • 13.7% and 8.8% average gains over the baseline [47] and state of the art [45], respectively, outdoors; indoors, gains are 51.7% and 30.5%, respectively.With different calibrations, the method generalizes better than the baseline and state of the art by 83% and 62%, respectively, in relative error.Source: §1 · p. 2
  • Unannotated videos and sparse point clouds train the model through photometric reprojection, sparse-point reconstruction, and total variation; its 6.9M parameters allow continued learning after deployment.At test time, inference uses each image and sparse point cloud independently, without video.Source: §1 · pp. 1–2, §1.1 · p. 2

1.1. Related Work and Contributions

  • The differentiable sparse-to-dense module addresses sparse-input convolution by learning a density-detail trade-off that retains near and far structures.Prior deep networks used many layers to densify sparse signals; alternatives included a non-differentiable mesh and pooling that decimated nearby-object detail.Source: §1.1 · pp. 2–3
  • The architecture imposes Euclidean geometric bias, while training-set statistics and model parameters also act as regularization for imputing missing depth.This contrasts with generic nearby-point regularization, which can treat adjacent pixels across occlusion boundaries as similar.Source: §1.1 · pp. 2–3

2. Method Formulation

KBNet combines sparse-depth densification with camera-calibrated backprojection to infer dense depth from an image and sparse point cloud without ground-truth depth annotations. Its training loss combines photometric consistency, sparse-depth consistency, and local smoothness.

2. Method Formulation

  • KBNet formulates depth completion as recovering a 3D scene from an image and sparse projected points without ground-truth depth annotations.The input includes camera intrinsics.Source: §2 · p. 3

2.1. Sparse-to-Dense Module (S2D)

  • The S2D module combines multi-scale min and max pooling to produce a dense or quasi-dense depth representation for the rest of the network.Its pooling outputs are fused with the original sparse depth through a 3 × 3 convolution and residual connection.Source: §2 · p. 3, §2.1 · pp. 3–4
  • Using both small and large kernels balances density against local detail, while combining min and max pooling preserves near and far structures.Large max-pooling kernels can remove nearby details; min pooling can corrupt far regions and blur adjacent structures.Source: §2.1 · p. 4, Figure 2 · p. 4

2.2. KBNet Architecture

  • Unlike approaches that implicitly learn camera calibration, KBNet takes intrinsics as input, allowing training and test cameras to use different calibrations and improving generalization.The authors connect implicit calibration learning to overfitting to the training camera.Source: §2.2 · p. 4
  • KB layers use camera intrinsics and a compressed depth descriptor to backproject pixels into 3D positional encodings, fused with image features and passed onward and to decoder skip connections.This RGB 3D representation replaces the typical RGB skip connection.Source: §2 · p. 3, §2.2 · p. 5, Figure 3 · p. 5
  • The 3D positional encoding represents scene topology rather than 2D image proximity, which the authors say better delineates adjacent objects and regularizes points on the same surface.They also report that KB layers enable a faster, slimmer network with fewer layers and parameters while achieving better performance.Source: §2.2 · p. 5

2.3. Loss Function

  • The training loss combines photometric consistency, sparse-depth consistency, and local smoothness; photometric consistency reconstructs the target from temporally adjacent frames.A pose network can be jointly learned with KBNet without extra supervision; pose is not needed at test time.Source: §2.3 · p. 5
  • Sparse-depth consistency grounds predictions to metric scale, while image-gradient-weighted depth smoothness allows discontinuities along object boundaries.The smoothness term penalizes depth gradients in the horizontal and vertical directions.Source: §2.3 · p. 6

3. Experiments and Results

The experiments evaluate KBNet on KITTI and VOID, then test transfer from VOID-trained models to NYUv2. Results cover benchmark accuracy, component ablations, calibration sensitivity, and cross-sensor generalization.

3.2. Datasets

  • The experiments evaluate KITTI outdoors, VOID indoors and outdoors, and NYUv2 indoors, with a separate VOID-to-NYUv2 sensor-transfer test.VOID uses about 1,500 sparse points, covering approximately 0.5% of each image; KITTI sparse maps have approximately 5% density.Source: §3 · p. 6, §3.2 · p. 6

3.1. Implementation Details

  • Implementation reports 16 ms end-to-end inference per frame and dataset-specific training schedules and image crops.Training used Adam on an Nvidia GTX 1080Ti; KITTI training took 70 hours for 60 epochs.Source: §3.1 · p. 6

3.3. KITTI Depth Completion Benchmark

  • Removing S2D or KB layers, or replacing S2D with scaffolding or SPP, degrades KITTI validation performance; sparse geometry still beats [47] in the stated rows.Without S2D, positional features encode only sparse geometry; without KB layers, VGG blocks replace the 3D spatial position encoding.Source: Table 3 · p. 7, §3.3 · p. 8
  • 8.8% average improvement over [45] on KITTI accompanies an 11.5% reduction in model size; the method also improves 13.7% over [47] across metrics.KITTI test results report a 29% model-size reduction against [47]; inference takes 16 ms per image.Source: §3.3 · pp. 6–7, Table 2 · p. 6
  • Calibration perturbations on KITTI leave performance robust up to approximately 10%, after which it degrades.The study alters focal length and principal point.Source: Figure 5 · p. 7, §3.3 · p. 8
  • Qualitative KITTI comparisons show better performance near street-sign/wall depth discontinuities and on distant road regions; VOID comparisons report better overall results than [47].The KITTI discussion associates these gains with 3D positional encoding that separates image-neighboring points far apart in scene topology.Source: §3.3 · p. 7, Figure 4 · p. 7, Figure 6 · p. 8

3.4. VOID Depth Completion Benchmark

  • 30.5% improvement over [45] on VOID accompanies a 51.7% improvement over [47], with gains across all metrics.The authors attribute the gains in part to S2D's learned dense representation and KB layers' 3D topology encoding.Source: Table 4 · p. 8, §3.4 · p. 8
  • On NYUv2 after training on VOID, KBNet's error increases by 10% and improves relative error over [47] by 83% and over [45] by 62%.The transfer evaluation changes from Intel RealSense to Microsoft Kinect; the authors note this setting also changes scene distribution.Source: §3.4 · p. 8, Table 5 · p. 8

4. Discussion

KBNet builds Euclidean reconstruction biases into its architecture, using calibration-aware backprojection to support cross-platform inference with a smaller model. The authors identify miscalibration, non-central-perspective optics, and unconstrained imputation as limitations.

4. Discussion

  • The architecture uses Euclidean reconstruction inductive biases rather than relying on a generic model, and its calibrated backprojection explicitly models 3D transformations and central perspective projection.The authors say this design reduces model size while still achieving state of the art.Source: §4 · p. 9
  • Feeding calibration as input allows a model trained on one sensor platform to be used with another at inference.This benefit depends on supplying the camera calibration.Source: §4 · p. 9
  • Incorrect calibration can distort depth maps, while omnidirectional optics require modifying the core backprojection module.The stated limitation reflects the rudimentary calibration model used.Source: §4 · p. 9
  • Where evidence is insufficient, the imputation regularizer can dominate and yield wildly wrong inferences; estimating depth uncertainty remains open.Source: §4 · p. 9

Supplementary Materials

The supplementary material documents implementation and loss details, then reports feature, calibration, sparsity, and cross-platform analyses. It also includes comparisons with supervised methods.

Supplementary Materials

  • Section A describes the full system, loss function, S2D kernel sizes, training augmentations, and learning-rate schedule for KITTI, VOID, and NYUv2.Source: Supplementary Materials · p. 12
  • Section B compares S2D features with typical convolutional features and reports a denser representation from S2D.Source: Supplementary Materials · p. 12
  • Sections C.1 and C.2 examine calibration-error and sparse-input-density sensitivity, reporting robustness when sparse points cover 0.15% of image space.Source: Supplementary Materials · p. 12
  • Sections D and E address transfer to test-time sensor platforms and compare results with supervised methods, including a fifth-place KITTI iMAE ranking.Source: Supplementary Materials · p. 12

A. System Overview

The system densifies sparse depth before combining it with RGB features and calibration-aware 3D positional encodings. Training uses adjacent video frames to construct reprojection-based losses; inference needs only an image and sparse point cloud.

A. System Overview

  • The system accepts RGB image, sparse depth, and camera intrinsics, then uses S2D to create a dense or quasi-dense depth representation.Source: A. System Overview · p. 12
  • KB layers backproject pixel coordinates using intrinsics and depth features to form a 3D positional encoding of scene structure.Source: A. System Overview · p. 13
  • The depth encoding is passed laterally to RGB features so the decoder can distinguish image-near points that are far apart in 3D.The decoder produces the final depth prediction.Source: A. System Overview · p. 13

A.1. Loss Function

  • Training estimates relative pose between adjacent frames and combines photometric reconstruction, sparse-depth reconstruction, and local-smoothness losses.The photometric term may be replaced by more sophisticated reprojection measures; pose estimation is not needed at inference.Source: §A.1 · p. 13

A.2. Implementation and Training Details

The appendix reports optimization schedules, image crops, and sparse-point augmentation for KITTI, VOID, and NYUv2. It also lists the dataset-specific pooling kernels used by S2D.

A.2. Implementation and Training Details

  • Training ran for 60 epochs on KITTI and 15 epochs each on VOID and NYUv2, using batch size 8.The corresponding crops were 768×320 for KITTI, 640×480 for VOID, and 576×416 for NYUv2.Source: §A.2 · p. 13
  • Table 7 lists dataset-specific min- and max-pooling kernel sizes, with larger kernels for VOID and NYUv2 because their VIO point clouds are sparser than KITTI’s LIDAR input.The learning-rate schedule is listed separately in Table 6.Source: Table 7 · p. 13, Table 6 · p. 13
  • Sparse-point removal ranged from 60% to 70% for KITTI and 30% to 60% for VOID and NYUv2.Augmentation was applied on every batch; for KITTI its frequency was reduced after epoch 50.Source: §A.2 · p. 13

B. Features Learned by Sparse-to-Dense

S2D converts sparse depth inputs into a dense or quasi-dense representation before the completion network. Its pooling and convolution design balances density with structural detail, while the section also introduces sensitivity studies.

B. Features Learned by Sparse-to-Dense

  • S2D uses multi-scale min and max pooling, followed by learned convolutions, to densify sparse depth while balancing near/far structures and density against detail.Pooled outputs are concatenated, processed by three 1×1 convolutions, then fused with the sparse input using a 3×3 convolution.Source: B. Features Learned by Sparse-to-Dense · pp. 13–14
  • With about 900 parameters, S2D produces dense or quasi-dense depth features before completion, whereas typical convolution blocks with about 10K–20K parameters remain sparse.The denser representation lets later layers learn scene geometry rather than continue densifying features.Source: B. Features Learned by Sparse-to-Dense · p. 14, Figure 8 · p. 14
  • S2D differs from prior pooling approaches: [45] uses max pooling that decimates nearby detail, while standard SPP uses pooling for dense-input tasks.The appendix notes that uses max pooling for consistent feature-map sizes and [2] uses average pooling to expand the receptive field.Source: B. Features Learned by Sparse-to-Dense · p. 14

C. Sensitivity Studies

  • The next subsection presents additional studies of sensitivity to incorrect calibration and sparse-depth density levels.Source: C. Sensitivity Studies · p. 14

C.1. To Incorrect Calibration

Calibration perturbations show that KBNet tolerates changes in principal point more readily than decreases in focal length. Its sensitivity is asymmetric, with focal-length reductions distorting backprojected surfaces more than comparable increases.

C.1. To Incorrect Calibration

  • Principal-point changes of up to 25% have little effect on performance in both VOID and KITTI, while changes beyond reasonable perturbations can reduce performance.The figures examine changes to focal length and principal point; the reported principal-point robustness applies to increasing or decreasing those parameters.Source: §C.1 · pp. 14–15, Figure 10 · p. 15
  • A focal-length decrease beyond 10% increases error, while increases are tolerated up to 20%; decreases distort surfaces by widening the backprojected field of view.Further increases of 20% to 25% produce a small error uptick as points become squashed together.Source: §C.1 · pp. 15–16
  • The reported perturbations are well beyond typical calibration error: one off-the-shelf calibration example had about 0.6% standard error and a ±1.1% 95% confidence margin.The authors nevertheless note a risk from using the wrong calibration.Source: §C.1 · p. 16
  • When tested on NYUv2 after training on VOID, KBNet’s point-cloud predictions are described as reasonable though imperfect, unlike [47]’s bowed scene.Source: Figure 14 · p. 17
  • With incorrect intrinsics, larger errors often appear along depth-map borders; a 25% decrease causes a slight scene-wide error increase, while a 25% increase affects output less.Source: Figure 9 · p. 15, §C.1 · p. 17

C.2. To Various Density Levels

On VOID, one model trained with 0.50% sparse-depth density is evaluated at three input densities. Performance declines as points become sparser, but the comparison with prior methods remains favorable overall, with an RMSE exception at the sparsest level.

C.2. To Various Density Levels

  • At 0.05% density, [45] beats the method on RMSE; overall, the method outperforms [45] [47] across the tested density levels.A single VOID model trained at 0.50% density was evaluated at 0.50%, 0.15%, and 0.05%.Source: §C.2 · p. 17
  • As input density falls, performance degrades, and the 0.05% test input is 10× sparser than the density used to select the model’s kernel sizes.The sparser representation limits the potential of the calibrated backprojection layers.Source: §C.2 · p. 17
  • Qualitative predictions show increasing blur and errors in homogeneous regions at 0.15%, then artifacts at 0.05%, where textureless surfaces lack sparse-depth anchors and may be assigned arbitrary depths.Source: §C.2 · p. 17

D. Generalization to Other Sensor Platforms

The authors test transfer from VOID, captured with an Intel RealSense, to NYUv2, captured with a Microsoft Kinect; neither model was trained on NYUv2. Their predictions indicate better cross-camera generalization for KBNet than for the baseline.

D. Generalization to Other Sensor Platforms

E. KITTI Depth Completion Benchmark

On KITTI, KBNet ranks first among unsupervised methods across all metrics and is competitive with supervised methods, despite using fewer parameters than several supervised models.

E. KITTI Depth Completion Benchmark

  • The KITTI comparison includes both supervised and unsupervised leaderboard benchmarks, with the paper's method listed as KNBet.Source: E. KITTI Depth Completion Benchmark · p. 19
  • Despite being unsupervised, KBNet outperforms some supervised methods [5] [6] across most metrics and its iMAE ranks fifth overall.The authors report comparable close-range performance and 6.9M parameters, versus 25.84M for [30], 53.4M for [32], and 28.99M for [49].Source: E. KITTI Depth Completion Benchmark · p. 19
  • KBNet ranks first among unsupervised KITTI methods across all metrics, beating state-of-the-art [45] by an average of 8.8% while using 11.5% fewer parameters.Method names and results are taken from the KITTI online leaderboard; [45] also uses Virtual KITTI training data.Source: E. KITTI Depth Completion Benchmark · p. 19

Related papers

Loading 2108.10531v2…