Source-linked AI summary
Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera
Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa Narasimhan, Jan Kautz
TL;DR
Depth sensing is important but active sensors and image-based methods have practical and geometric limitations. The paper estimates per-pixel depth distributions from monocular video, represents them as DPVs, and integrates them over time with Bayesian filtering. It reports more accurate, robust, temporally stable, and cross-dataset-generalizable depth estimates, with outputs usable for RGB-D 3D scanning.
Problem
Depth sensing is important for 3D reconstruction and scene understanding, but active sensors and image-based methods have limitations including restricted range, interference, scale ambiguity, drift, and correspondence costs.
Method
The method predicts a per-pixel depth probability volume from monocular video and integrates successive volumes over time using a learnable Bayesian-filtering network.
Results
The method outperforms prior DL-based depth estimators in accuracy, robustness, and temporal stability, generalizes better across datasets, and supports RGB-D 3D reconstruction.
Takeaways & Limitations
Monocular RGB video can provide depth and uncertainty suitable for directly feeding classical RGB-D scanning methods for 3D scene reconstruction.
Takeaways & Limitations
Monocular camera pose scale drift may affect depth accuracy, and the method uses a local time window rather than all frames globally.
Abstract
from arXiv · showhide
Depth sensing is crucial for 3D reconstruction and scene understanding. Active depth sensors provide dense metric measurements, but often suffer from limitations such as restricted operating ranges, low spatial resolution, sensor interference, and high power consumption. In this paper, we propose a deep learning (DL) method to estimate per-pixel depth and its uncertainty continuously from a monocular video stream, with the goal of effectively turning an RGB camera into an RGB-D camera. Unlike prior DL-based methods, we estimate a depth probability distribution for each pixel rather than a single depth value, leading to an estimate of a 3D depth probability volume for each input frame. These depth probability volumes are accumulated over time under a Bayesian filtering framework as more incoming frames are processed sequentially, which effectively reduces depth uncertainty and improves accuracy, robustness, and temporal stability. Compared to prior work, the proposed approach achieves more accurate and stable results, and generalizes better to new datasets. Experimental results also show the output of our approach can be directly fed into classical RGB-D based 3D scanning methods for 3D scene reconstruction.
1. Introduction
The paper addresses limitations of active and image-based depth sensing by estimating depth and uncertainty continuously from monocular video. It represents per-pixel depth distributions as DPVs, integrates them over time, and reports improved accuracy, robustness, temporal stability, and cross-dataset generalization.
- Active depth sensors provide dense metric depth but can have limited range and resolution, consume more power, and suffer sensor interference.
- Image-based depth avoids some active-sensor limitations but faces scale ambiguity, drift, correspondence difficulties, and high computational cost.
- The method estimates a depth probability distribution for each pixel, producing a Depth Probability Volume with both depth estimates and per-pixel uncertainty.
- DPVs are accumulated sequentially using a learnable Bayesian-filtering network to reduce uncertainty and improve accuracy, robustness, and temporal stability.
- Across multiple datasets, the method outperforms selected DL-based methods and supports direct input to classical RGB-D scanning for 3D reconstruction.
2. Related Work
Prior depth-sensing approaches rely on active sensors, geometric image methods, or learned models with distinct limitations. This work places uncertainty estimation and Bayesian temporal integration at the center of learned monocular video depth sensing.
- Active sensors provide dense metric depth and confidence measures but suffer from limited range, low resolution, interference, and high power consumption.
- Single-view methods require strong scene assumptions, while stereo and multi-view methods face correspondence failures and computational expense.
- Learned depth methods span single-view, video, two-view, and multi-view settings, but their robustness and generalization remain insufficiently tested.
- The paper defines uncertainty as a posterior depth probability estimated from several consecutive frames, targeting measurement rather than model uncertainty.
- An additional network integrates depth distributions over time in a Bayesian-filtering manner to improve video depth accuracy and robustness.
3. Our Approach
The approach represents per-pixel depth as a probability volume, estimates it from local video frames, and updates it over time using Bayesian filtering. A learnable adaptive correction handles inconsistent information, while refinement produces higher-resolution depth and confidence maps.
- Network overview: The network comprises D-Net for per-frame DPV estimation, K-Net for temporal integration, and R-Net for image-guided spatial refinement.The full pipeline outputs a refined DPV from which depth and confidence are computed.
- DPV representation: A DPV represents the probability that pixel (u, v) has each candidate depth within the camera’s 3D view frustum.The volume is discretized into 64 planes uniformly over inverse depth.
- Temporal integration: Bayesian filtering predicts the DPV under camera motion and updates it with the newly estimated local DPV as subsequent frames arrive.This predict-update process is intended to reduce uncertainty through temporal accumulation.
- DPV estimation: The D-Net estimates a conditional DPV from each frame and its five-frame local temporal window, using warped neighboring-frame features to construct depth costs.The resulting DPV is refined and upsampled to the input image resolution by R-Net.
- Limitations and outputs: Direct Bayesian filtering reduces noise but can propagate incorrect depth near occlusion boundaries, motivating adaptive damping before final DPV refinement.The paper illustrates reduced noise on back-wall and floor regions with filtering, while identifying occlusion and disocclusion as failure cases for direct integration.
- Adaptive damping: K-Net adaptively transforms the measurement-prediction residual into a correction, preserving consistent regions while correcting depth probabilities that disagree.Unlike global damping, adaptive damping changes prediction weights for different DPV voxels and improves disoccluded-region estimation.
4. Experimental Results
The method is evaluated across indoor, outdoor, and cross-dataset settings, emphasizing accuracy, robustness, temporal consistency, and usefulness for 3D reconstruction. Results show improved depth quality, confidence-aware reconstruction, and stronger cross-dataset generalization, with performance depending on camera-pose estimation.
- Evaluation setup: The evaluation spans multiple indoor and outdoor datasets, using depth estimation metrics, qualitative comparisons, and direct RGB-D-based 3D reconstruction.Cross-dataset tests assess robustness without fine-tuning, while reconstruction evaluates metric accuracy, consistency, and usefulness.
- Confidence and temporal consistency: Confidence maps correlate with depth errors over time: confidence increases for reliable regions and decreases for uncertain regions as more frames arrive.Examples include books becoming more confident with lower error and glass becoming less confident with higher error.
- Indoor scenarios: On 7-Scenes, the method significantly outperforms DeMoN and DORN on common depth-estimation metrics, including when no IMU is used.The comparison uses models trained on different datasets and a separate indoor test set.
- Indoor scenarios: The estimated indoor depths are less noisy, sharper, and more temporally consistent, producing higher-quality 3D meshes than competing methods and sometimes real RGB-D sensing.The reported reconstruction advantage includes better coverage and accuracy on some monitors and glossy surfaces.
- Cross-dataset evaluation: The method performs better in the reported cross-dataset task, supporting stronger robustness and generalization ability than the comparison methods.The task includes training on KITTI and testing on virtual KITTI, and training on indoor datasets before testing on KITTI.
- Camera-pose ablation: Accurate camera poses are important: initializing only the first time window with DSO performs better than initializing all frames, partly because monocular VO can err in textureless regions.Dense-depth optimization may overcome some of these pose-estimation errors.
- Confidence-aware reconstruction: Confidence maps can correct low-confidence regions and mask them before voxel hashing, improving depth maps and 3D reconstruction.The cited example focuses on errors caused by specular reflection and low-confidence voxels.
5. Conclusions and Limitations
The paper concludes that continuous monocular-video depth sensing can estimate depth probability volumes and integrate them over time for RGB-D-style reconstruction. It identifies scale drift and the restriction to local temporal windows as limitations.
- Conclusion: The proposed method continuously estimates a depth probability volume from a monocular video camera and integrates it over time.The conclusion frames this as continuous depth sensing from a monocular video camera.
- Conclusion: The method is intended to support RGB-D-based 3D reconstruction using depth and uncertainty estimated from video.The reported pipeline includes confidence-aware processing for reconstruction.
- Limitations: Monocular-video camera poses can suffer from scale drift, which may affect depth-estimation accuracy.This is identified as a limitation for future work.
- Limitations: The current method focuses on a local time window rather than a global solution using all frames.The authors list global-context processing as future work.
A. Relation of K-Net to the Kalman filter
The appendix relates the proposed K-Net update to a Kalman filter by mapping image observations, negative-log depth probabilities, warping, and learned updates to the filter’s corresponding components.
- Assumptions: The correspondence is discussed under assumptions of an accurate measurement matrix and temporally invariant gain and measurement matrices.These assumptions support the stated relation between the proposed update and Kalman filtering.
- State and observation mapping: In the Kalman-filter analogy, input images are observations, while negative-log depth probabilities represent hidden states.The mapping compares the paper’s update equations with the standard filter formulation.
- Temporal transition: The warping operator corresponds to the transition matrix that maps the previous hidden state to the current state.This identifies the temporal prediction component of the proposed update.
- Residual update: K-Net corresponds to the combined gain-and-measurement mapping that transforms observation residuals into hidden-state updates.The comparison associates K-Net with the product of the gain and measurement matrices.
B.1. Complete metrics for Comparisons
The paper directs readers to Tables 5 and 6 for the complete depth-estimation comparison metrics.
- Complete comparison metrics: Tables 5 and 6 provide the complete metrics used for depth-estimation comparisons.The passage identifies these tables as the source for the full comparison results.
B.2. Results on KITTI without GPS or IMU
Table 7 evaluates the method on KITTI with reduced pose and hardware inputs, comparing IMU-only rotation against an optimization-based pose setting.
- Evaluation settings: Table 7 evaluates KITTI performance when only IMU measurements are available, denoted “GT R”.The table also considers a setting with neither IMU nor GPU available.
- Evaluation settings: The “opt. pose” setting evaluates KITTI performance without IMU or GPU measurements.This setting tests an optimization-based alternative to direct IMU measurements.
- Network configuration: The pipeline includes D-Net, K-Net, and R-Net structures, with D set to 64 in the described networks.The supplied implementation passages identify these network components and their shared depth setting.
C.3. R-Net
The R-Net is documented as one of the pipeline’s network components, with its structure provided in Table 9 and D set to 64.
- R-Net structure: The paper presents the R-Net structure in Table 9.The supplied table caption identifies Table 9 as the R-Net structure.
- R-Net structure: The paper sets D = 64 for the R-Net configuration.This value is stated alongside the R-Net structure description.