Unsupervised Depth Completion from Visual Inertial Odometry

Alex Wong; Xiaohan Fei; Stephanie Tsuei; Stefano Soatto

Source-linked AI summary

Unsupervised Depth Completion from Visual Inertial Odometry

Alex Wong, Xiaohan Fei, Stephanie Tsuei, Stefano Soatto

arXiv:1905.08616v4May 15, 2019cs.CVcs.AIcs.LGstat.ML

TL;DR

Dense depth completion is difficult when visual-inertial odometry supplies only a few hundred to a few thousand sparse points that do not reveal scene topology. The paper builds a piecewise-planar scaffolding and refines it with image-based cross-modal consistency, introducing a visual-inertial-plus-depth dataset and reporting state-of-the-art KITTI performance.

Problem
Sparse visual-inertial depth measurements are insufficient to determine scene topology, making dense shape inference ill-posed without assumptions about the environment.
Method
The method constructs a piecewise-planar scaffolding from sparse depth and refines it with images using photometric, sparse-depth, pose-consistency, and smoothness constraints.
Results
The lightweight two-stage method achieves state-of-the-art performance on the unsupervised KITTI depth-completion benchmark and introduces the first visual-inertial-plus-depth dataset.
Takeaways & Limitations
The paper demonstrates a route to dense depth completion that combines sparse metric-scale visual-inertial measurements with image-based refinement without pseudo-ground-truth annotations.

Abstract

from arXiv · show

We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to `self-supervision,' measuring photometric consistency across time, forward-backward pose consistency, and geometric compatibility with the sparse point cloud. We also launch the first visual-inertial + depth dataset, which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, and show state-of-the-art performance on it. Code available at: https://github.com/alexklwong/unsupervised-depth-completion-visual-inertial-odometry.

I. INTRODUCTION

Sparse visual-inertial reconstructions support motion estimation but provide too little scene topology for dense depth completion. The paper addresses this with a piecewise-planar scaffolding, cross-modal self-supervision, a new visual-inertial-plus-depth dataset, and state-of-the-art KITTI performance.

Motivation: Visual-inertial systems estimate metric-scale motion by combining image sequences with inertial measurements, but their sparse features poorly represent scene shape and topology.Sparse point clouds may contain hundreds to thousands of points, leaving gaps whose geometry is ambiguous.
Motivation: Inferring dense shape from sparse depth is ill-posed because filling gaps requires assumptions about the environment.The method uses images to restrict scenes compatible with the sparse measurements rather than relying only on generic priors.
Approach: The paper introduces a predictive cross-modal criterion combining temporal photometric consistency with forward-backward pose consistency for dense-depth learning without pseudo-ground-truth annotations.The criterion is described as a form of self-supervision.
Approach: The method first constructs a piecewise-planar scaffolding from sparse points, then refines it using images and priors derived from temporal and geometric constraints.This transfers supervisory information from sparse measurements to neighboring image regions through a two-stage process.
Contributions: The first visual-inertial-plus-depth dataset is introduced to support metric-scale reconstruction research using complementary visual and inertial sensors.The authors motivate inertials as widely available and typically co-located with cameras in mobile devices.
Results: The two-stage model uses far fewer parameters than competing methods while achieving state-of-the-art performance on the unsupervised KITTI depth-completion benchmark.Evaluation uses KITTI’s monocular stream for fair comparison, although its sparse points come from lidar rather than VIO.

II. RELATED WORK

Prior depth-completion methods commonly require dense annotations or specialized sparse-depth propagation, while this paper learns cross-modal fusion through a lightweight scaffolding-and-refinement pipeline. The formulation uses RGB images, sparse depth, and monocular temporal context to predict dense depth.

Supervised depth completion: Supervised depth-completion methods learn from per-pixel ground-truth depth while varying network topology, optimization, and sparse-input modeling.Examples include early or late fusion and specialized layers for propagating sparse depth.
Unsupervised depth completion: Unsupervised methods combine sparse-depth discrepancy with photometric reconstruction losses, but PnP and RANSAC alignment can fail in textureless indoor regions.The paper avoids pseudo-ground-truth annotations and learns cross-modal fusion from unannotated data.
Proposed approach: The proposed approach reduces network parameters through a two-stage design while retaining state-of-the-art performance on the unsupervised KITTI benchmark.Scaffolding propagates sparse information before neural refinement.
Contributions: Its contribution set includes exponential rotation parameterization, forward-backward pose consistency, and a visual-inertial depth-completion benchmark with indoor and outdoor scenes.These additions complement the lightweight two-stage architecture.
Method formulation: The method reconstructs a 3D scene from an RGB image and sparse depth by constructing scaffolding and refining it with an encoder-decoder network.The predicted depth is produced from the image and scaffolding, with monocular neighboring frames used during training.

A. A Two-Stage Approach

The method addresses extremely sparse depth by first building a coarse piecewise-planar scaffolding, then refining it with RGB information in a compact network trained without ground-truth depth.

Scaffolding: The first stage lifts sparse points into 3D, forms a triangulated mesh, and projects piecewise-linear surface interpolation back into the image plane as scaffolding.Non-coplanar regions remain approximate and are subsequently refined by the network.
Refinement: The second stage combines the RGB image with the scaffolding and refines the coarse depth using an encoder-decoder network with separate image and depth branches.The proposed VGG11 and VGG8 encoders contain approximately 5.7M and 2.4M parameters, respectively.
Refinement: The network learns to copy the input scaffolding initially, then fuse RGB information to improve regions such as pedestrians and street signs.This progression is illustrated during training in Fig. 3.
Outcome: The two-stage design achieves state-of-the-art unsupervised KITTI performance with roughly half as many parameters as prior art.Scaffolding reduces the need for the network to learn directly from sparse inputs.
Training objective: Training uses four constraints: photometric consistency, sparse-depth consistency, forward-backward pose consistency, and local smoothness.The loss is a weighted linear combination of these terms, while exponential-coordinate layers support pose representation.

A. Photometric Consistency

Photometric consistency trains depth and pose by reconstructing the target image from temporally adjacent frames and penalizing reconstruction discrepancies.

Temporal reconstruction: The method compares each observed image I_t with reconstructions from temporally adjacent images at t−1 and t+1.The reconstruction uses the estimated relative camera pose between the frames.
Reprojection: Relative pose, camera intrinsics, homogeneous coordinates, and perspective projection define how pixels are mapped between frames.These components establish the reprojection used to synthesize the target view.
Photometric loss: The photometric term combines average per-pixel L1 reprojection error with SSIM to reduce sensitivity to local illumination changes.SSIM is computed on 3 × 3 patches centered at each pixel.

B. Sparse Depth Consistency

Sparse-depth consistency anchors predictions to metric measurements, while pose, smoothness, and qualitative evaluation connect sparse supervision to geometric refinement.

Sparse-depth loss: The sparse-depth loss is the averaged L1 difference between predicted depth and available metric sparse depth over measured pixels.This term uses the support Ω_s of the sparse measurements.
Metric scale: Metric sparse depth supplies scale from lidar in KITTI and sparse visual-inertial reconstruction in the proposed dataset.Inertials are crucial because visual-inertial odometry provides sparse points in metric scale.
Pose consistency: Forward-backward pose consistency penalizes deviation of the composed forward and backward poses from the identity, using the logarithmic map on SE(3).The backward pose is expected to invert the forward pose.
Local smoothness: Local smoothness applies an L1 penalty to depth gradients in x and y, with edge-aware weights allowing discontinuities at image boundaries.The weights decrease the penalty near image gradients.
Qualitative evaluation: Qualitative KITTI results show stronger performance on thin structures and far regions, where the comparison method exhibits scanline- and circle-like artifacts.Warmer error-map colors indicate higher error.

V. DATASETS

The paper evaluates on KITTI and introduces VOID, a visual-inertial-plus-depth dataset designed for metric reconstruction and cross-modal validation across outdoor and indoor settings.

KITTI: KITTI provides about 80,000 image frames with lidar sparse depth at approximately 5% density and ground truth accumulated from 11 neighboring scans.The benchmark’s dense ground truth covers the bottom 30% of images.
Dataset motivation: Existing visual-inertial datasets lacked dense per-frame depth and were relatively small for training deep models, motivating the new dataset.The paper also reports qualitative TUM-VI results at a sparse-depth density of 0.015%.
VOID: VOID combines RGB video, inertial measurements for metric reconstruction, and dense per-frame depth for cross-modal validation.The dataset uses synchronized 400 Hz inertial and 30 Hz VGA RGB-depth streams from an Intel RealSense D435i.
VIO input: The VIO system estimates a very sparse structure, typically 20–30 in-state feature points, supplemented by tracked out-of-state features for reconstruction.This highlights the extreme sparsity of the visual-inertial input.
Implementation: The implementation reports inference at approximately 22 ms per image, with training times varying by model, dataset, and epoch count.On KITTI, training takes about 42 hours for VGG11 and 34 hours for VGG8 over 30 epochs.

VII. EXPERIMENTS AND RESULTS

On KITTI, the method achieves state-of-the-art unsupervised depth-completion performance while using substantially fewer parameters than competing approaches. Ablations show benefits from scaffolding, exponential pose parameterization, and forward-backward pose consistency, with degradation beyond 80 meters.

Our VGG11 model outperforms state-of-the-art on every metric by as much as 12.8% while using 48.4% fewer parameters.
Combining scaffolding with refinement performs better than scaffolding alone or a baseline without scaffolding, and the full VGG11 model gives the best overall results.
Performance begins to degrade beyond 80 meters because sparse measurements and parallax become insufficient.
Exponential pose parameterization consistently improves all metrics, while forward-backward pose consistency improves depth prediction across the evaluated ablations.
TUM-VI qualitative results use monochrome, highly distorted fisheye images and sparse depth at 0.015% density, with distortion compensated during training.
The VGG8 model uses approximately 6.4M parameters and retains near-VGG11 performance while achieving state-of-the-art unsupervised KITTI depth completion.

C. VOID Depth Completion Benchmark

On VOID, the method is evaluated across sparse-depth densities and challenging indoor and outdoor motion. Replacing the learned pose with SLAM poses consistently improves performance, while pose consistency substantially narrows the gap to that upper bound.

Replacing the pose network with SLAM poses systematically improves performance across all evaluation metrics on VOID.
The learned pose network performs worse than SLAM poses on challenging motion, indicating difficulty predicting accurate poses under non-trivial 6 DoF motion.
Adding pose consistency improves the exponential-parameterization model by as much as 23.4% and brings it closer to the SLAM-pose model.
VOID contains VGA images from indoor and outdoor scenes with challenging motion, and comparisons distinguish SLAM-estimated poses from pose-network predictions.
The results motivate combining classical SLAM with deep learning for depth completion and pose estimation.

APPENDIX A VOID DATASET

The appendix presents VOID qualitative depth-completion results and a pose-network evaluation protocol on KITTI Odometry. Pose accuracy is assessed with trajectory and relative-motion metrics, including a rotation-specific error.

VOID dataset: VOID contains sampled video images alongside sparse VIO point-cloud reconstructions and camera trajectories.The dataset examples illustrate the visual and sparse geometric inputs available for depth completion.
VOID results: Qualitative VOID results show recovered scene structure across indoor and outdoor settings at varied ranges.Completed depth maps are back-projected to 3D and viewed from another vantage point.
Pose evaluation: The pose study jointly trains depth and pose networks on KITTI depth completion data, then evaluates chained poses on KITTI Odometry sequences 09 and 10.The evaluation isolates the pose network after training the combined system.
Pose metrics: ATE measures whole-trajectory translational accuracy, whereas RPE focuses on local relative-pose accuracy with a sliding window of Δ=1.The appendix also introduces RRE because RPE does not reflect rotational accuracy.
Pose metrics: RRE evaluates relative rotation using the rotational part of an SE(3) transformation and the logarithmic map from SO(3) to R^3.This metric is designed to study rotation estimation and pose-parameterization effects.

B. Ablation Study on KITTI Odometry

The KITTI Odometry ablation compares Euler angles with exponential pose parameterization and tests the added pose-consistency term. Exponential parameterization performs better on global and rotation-sensitive measures, especially with pose consistency.

Evaluation setup: The pose ablation evaluates chained camera trajectories and reports both quantitative metrics and qualitative trajectory alignment.The associated study uses sequences 09 and 10 to assess pose-network variants.
Pose parameterization: Exponential parameterization is comparable to Euler angles on ATE-5F and RPE for sequence 09 but improves ATE by ≈20% and RRE by ≈3.4%.It outperforms Euler angles on all metrics for sequence 10.
Pose consistency: Adding pose consistency to exponential parameterization yields the best results in the reported ablation.The comparison is performed on KITTI Odometry sequences 09 and 10 after joint training on the depth-completion dataset.

APPENDIX D MORE RESULTS ON KITTI DEPTH COMPLETION BENCHMARK

Additional KITTI test-set visualizations broaden inspection beyond the benchmark’s first 20 online samples. The model recovers pedestrians, thin structures, and structures without associated sparse lidar points.

Evaluation scope: The appendix adds qualitative results across varied KITTI test scenes because the online benchmark displays only the first 20 samples.This supplements the main paper’s quantitative and qualitative evaluations.
Qualitative results: The VGG11 model recovers pedestrians and thin structures such as guard rails and street poles.These results use the full loss function and exponential rotation parameterization.
Qualitative results: The network predicts structures without associated sparse lidar points, including structures in the upper image domain.The paper attributes this behavior to the photometric data-fidelity term.
Qualitative results: During training, the network first copies the input scaffolding and later fuses image information to add scene elements missing from that scaffolding.This describes the progression reported for the full-loss model.

APPENDIX E NETWORK ARCHITECTURE

The appendix describes late-fusion VGG11 and VGG8 depth-completion architectures with a shared decoder and an auxiliary pose network used only during training. The lighter model substantially reduces parameters while performing similarly to VGG11.

Depth-completion networks: The models use late fusion with separate image and depth encoder branches, concatenated latent representations, skip connections, and a shared decoder.The main and lightweight variants use VGG11 and VGG8 encoders, respectively.
Lightweight model: The VGG8 model contains ≈2.4M encoder parameters and performs similarly to the VGG11 model.Its encoder is compared with approximately 23.8M and 14.8M parameters in and.
Pose network: The auxiliary pose network contains ≈1M parameters and is used only during training to construct photometric and pose-consistency losses.Its six-element output provides three rotation and three translation values.
Parameterization: The VGG11 depth-completion network contains ≈9.7M parameters, while the VGG8 model contains ≈6.4M.These totals include the encoder and decoder components.
Parameterization: The VGG11 model reduces parameters by 65.1% versus and 48.4% versus.The cited comparison uses approximately 27.8M and 18.8M parameters for the competing models.