Source-linked AI summary
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath, Dieter Fox
TL;DR
Transparent objects often lack reliable depth in standard RGB-D sensing, leaving depth completion from a single RGB-D image as a challenge. This paper uses ray-voxel local implicit depth estimation with iterative refinement and synthetic training data; it reports better accuracy and speed than prior state-of-the-art methods, including generalization from synthetic training to real-world scenarios.
Problem
Commodity depth sensors fail to produce correct depth for transparent objects, limiting RGB-D methods used in applications such as robot manipulation.
Source: §1 · p. 1Method
The approach estimates depth with a local implicit function on ray-voxel pairs, followed by iterative refinement, and trains the pipeline on a large synthetic dataset.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The Omniverse synthetic dataset can boost real-world performance for both this approach and competing methods.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Majority of the perception methods in robotics require depth information provided by RGB-D cameras. However, standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. In this paper, we introduce a new approach for depth completion of transparent objects from a single RGB-D image. Key to our approach is a local implicit neural representation built on ray-voxel pairs that allows our method to generalize to unseen objects and achieve fast inference speed. Based on this representation, we present a novel framework that can complete missing depth given noisy RGB-D input. We further improve the depth estimation iteratively using a self-correcting refinement model. To train the whole pipeline, we build a large scale synthetic dataset with transparent objects. Experiments demonstrate that our method performs significantly better than the current state-of-the-art methods on both synthetic and real world data. In addition, our approach improves the inference speed by a factor of 20 compared to the previous best method, ClearGrasp. Code and dataset will be released at https://research.nvidia.com/publication/2021-03_RGB-D-Local-Implicit.
1. Introduction
Commodity depth sensors often fail on transparent objects, limiting RGB-D perception in household settings. The paper introduces ray-voxel local implicit depth estimation, iterative refinement, and synthetic training data for fast depth completion from a single RGB-D image.
1. Introduction
- ClearGrasp can produce undetermined depth without contact edges and is too slow for real-time use, while another reconstruction method requires an environment map and refractive index [28].These prior-work constraints motivate a single-image, faster approach.Source: §1 · p. 1
- LIDF estimates depth from color and local depth on ray-voxel pairs, using shared local structures to generalize to unseen objects and querying only occupied voxels intersected by a camera ray.Color supplies shape and curvature cues, while local depth helps locate transparent objects spatially.Source: §1 · p. 2
- The two-stage system combines LIDF with iterative self-correcting refinement, and its Omniverse Object dataset contains over 60,000 synthetic images of transparent and opaque objects.The dataset varies object models, poses, lighting, viewpoints, and background textures; training with it can boost real-world performance for the authors' method and competing methods.Source: §1 · p. 2
- The full pipeline is reported to outperform current state-of-the-art methods in accuracy and speed, with good generalization to real-world scenarios despite training only on synthetic data.The paper presents the pipeline as both fast and generalizable to unseen objects.Source: §1 · p. 2
2. Related Work
The related work distinguishes transparent-object depth completion from color-only depth prediction and completion from sparse measurements. It positions this method as single-view RGB-D completion without object priors, trained on synthetic data, and based on local implicit functions.
2. Related Work
- Unlike color-only depth prediction or sparse-depth completion, this method targets completing depth maps from noisy RGB-D images, reporting better results and 20× speedup over.The cited global optimization approach can be slow and perform poorly when its network predictions are inaccurate.Source: §2 · p. 2
- Many transparent-object methods assume known backgrounds, known 3D models, or multi-view input; this approach instead estimates depth from a single RGB-D view without priors.Compared with, it is trained purely on synthetic data and is reported to achieve better performance.Source: §2 · pp. 2–3
- Local voxel-based implicit representations address scene-scale limits of single latent-vector methods, while this task lacks the point clouds that such representations typically use.NeRF-based approaches can require scene-specific retraining or lengthy rendering; sparse voxel methods still sample points heuristically, whereas this method estimates ray-voxel termination positions.Source: §2 · p. 3
3. Method
The system completes missing depth from RGB-D input in two stages: it learns local geometry on ray-voxel pairs, then iteratively refines depth using updated voxel and ray embeddings. Its method also specifies how ray-voxel pairs are formed, pooled, and trained.
3. Method
- The system takes an RGB image, incomplete depth, and camera intrinsics as input and predicts a full depth map.The refinement stage recomputes voxel and ray embeddings from the input and predicted depth, then combines them with the original RGB embedding.Source: §3 · p. 3
3.1. Generating Ray-voxel Pairs
- For each pixel, the method back-projects valid depth points, marks voxels containing them as occupied, and finds occupied-voxel intersections with the camera ray.The workspace is divided into a fixed voxel grid, and ray-voxel intersections are computed with a ray-AABB test.Source: §3.1 · p. 3
3.2. Learning LIDF for Ray-voxel Pairs
- LIDF predicts a ray’s termination probability and position within an intersected voxel from RGB, voxel, and ray embeddings.The voxel embedding comes from valid points in the voxel, while the ray embedding encodes high-frequency ray-direction and intersection-point information.Source: §3.2 · p. 4
3.3. Ray Pooling
- Argmax pooling gives better depth accuracy than weighted sum, which can add noise from distant voxels even when their termination probabilities are low.Argmax selects the ray-voxel pair with the highest termination probability; weighted sum combines depth estimates across the ray.Source: §3.3 · pp. 4–5
3.4. Depth Refinement Model
- The refinement model recomputes voxel embeddings from valid and previously predicted points, then iteratively adjusts depth; two iterations are used, with more yielding only marginal improvement.The refinement networks do not share weights with the first-stage networks.Source: §3.4 · p. 5
3.5. Loss Functions
- Training combines depth-position L1 loss, termination-probability cross-entropy, and a surface-normal cosine-distance loss that regularizes learned object shapes.Source: §3.5 · p. 5
3.6. Implementation Details
- Training predicts depth only for corrupted pixels, while testing estimates depth at all pixels; the two network stages are trained separately for 60 epochs each.Training data simulates missing transparent-object depth, partial opaque-object depth, background holes, and color noise and blur.Source: §3.6 · p. 5
4. Experiments
The experiments compare the full method with prior approaches and test its components on transparent-object depth completion. The method leads the reported comparisons, while ablations examine refinement, inputs, representation choices, and training data.
4. Experiments
- Evaluation uses ClearGrasp’s four known- and novel-object splits across synthetic and real images, with depth errors measured in transparent-object regions.The pipeline is trained on ClearGrasp and the Omniverse Object dataset; predictions and ground truth are resized to 144×256 for evaluation.Source: §4 · pp. 5–6
4.1. Comparison to State-of-the-art Methods
- Our method achieves the best result on all datasets, including against baselines retrained on the same data.Table 1 reports the quantitative comparison; the paper also describes more meaningful depth in qualitative comparisons.Source: §4.1 · p. 6, Table 1 · p. 6
- 0.09 seconds per image versus ClearGrasp’s 1.82 seconds makes our method 20x faster in the reported test.Both methods were timed on a single P100 GPU over the testing set.Source: §4.1 · p. 6
4.2. Ablation Studies
- Depth refinement improves performance on real unseen objects and corrects tilted errors in first-stage predictions.The comparison evaluates predictions with and without refinement.Source: §4.2 · p. 6, Table 2 · p. 7, Figure 7 · p. 8
- Neither RGB-only nor depth-only input predicts transparent-object depth accurately; the first lacks depth estimation accuracy, while the second lacks visual cues identifying transparent objects.The experiments retrained the first-stage model using one modality at a time.Source: §4.2 · pp. 6–7, Table 3 · p. 7
- Ray information guides depth prediction, positional encoding changes fine-detail accuracy, and too many voxels reduce performance more than using fewer voxels.The positional-encoding effect is reported for δ1.05 and δ1.10.Source: §4.2 · p. 7, Table 4 · p. 7
- Training on ClearGrasp and Omniverse together improves performance over either dataset alone, and Omniverse complements ClearGrasp for transparent-object learning.Table 5 examines generalization to real novel objects.Source: §4.2 · pp. 7–8, Table 5 · p. 8
- Argmax ray pooling correctly classifies 84.77% of pixels and places 97.71% within ±1 voxel; learned candidate positions outperform heuristic sampling.The reported mean distances for misclassified −1 and +1 voxels are 0.064 and 0.048 meters, respectively; argmax also outperforms weighted sum on real-novel objects.Source: §4.2 · p. 8, Figure 8 · p. 8, Table 6 · p. 8
5. Conclusion
The conclusion presents a ray-voxel local implicit depth function with iterative refinement, alongside a synthetic dataset for transparent-object learning. The authors report generalization from synthetic training to real-world scenarios and improvements in accuracy and speed over prior art.
5. Conclusion
- The framework combines a local implicit depth function on ray-voxel pairs with an iterative depth-refinement model and introduces a large-scale synthetic dataset.The dataset can boost performance for both the proposed approach and competing methods.Source: §5 · p. 8
- Although trained only on synthetic datasets, the pipeline generalizes well to real-world scenarios.Source: §5 · p. 8
- Quantitative and qualitative results show substantial improvements over prior art in accuracy and speed.Source: §5 · p. 8
Appendix
The two-stage voxel-based PointNet encoder embeds valid points using their positions and colors, pools these embeddings into voxels, then repeats the process with point and voxel embeddings.
A. Voxel-based PointNet Encoder
- Valid-point positions relative to voxel centers and point colors produce initial point embeddings through a shared MLP.Max-pooling the point embeddings within each voxel, followed by another MLP, produces initial voxel embeddings.Source: A. Voxel-based PointNet Encoder · p. 12
- For the second stage, each valid point’s embedding is concatenated with the embedding of its voxel.The resulting input is processed using the same point-to-voxel embedding steps.Source: A. Voxel-based PointNet Encoder · p. 12
- Repeating the process produces the final voxel embedding Hvox.The encoder applies the same process as in the first stage to obtain this embedding.Source: A. Voxel-based PointNet Encoder · p. 12
B. Omniverse Object Dataset
The Omniverse Object Dataset combines selected ShapeNet and ClearGrasp objects with varied textures and provides scene-level labels generated through a two-pass rendering strategy.
B. Omniverse Object Dataset
- The dataset uses six ShapeNet categories and five ClearGrasp object types, with only ClearGrasp training objects selected so test objects remain unseen during training.Background textures come from CC0 TEXTURES; opaque-object textures come from CC0 TEXTURES and the Describable Textures Dataset.Source: B. Omniverse Object Dataset · p. 12
- Each image includes depth, segmentation, camera parameters, bounding boxes, and 6D object poses, with ground-truth depth produced by a two-pass rendering strategy.The first pass renders all objects as opaque for ground-truth data; the second renders candidate transparent objects as glass to produce RGB images.Source: B. Omniverse Object Dataset · p. 12
C. Additional Results for Ablation Studies
The ablations report benefits from refinement, RGB and depth inputs, ray information, and positional encoding, while showing trade-offs for voxel count and improved results from selected training and pooling choices.
Depth refinement model
- Depth refinement boosts performance on synthetic novel objects while achieving similar results on synthetic known and real known objects.The authors interpret this as evidence that refinement can increase generality.Source: C. Additional Results for Ablation… · p. 12
Input Modalities
- Both RGB and depth contribute to depth accuracy; without RGB the method predicts flat planes, and without depth its prediction is far from the table.RGB provides visual cues about object shape, while depth helps localize objects in metric space.Source: C. Additional Results for Ablation… · p. 12
Ray Information
- Ray information helps the model reason about transparent objects’ location and orientation.The passage reports quantitative comparisons in Table 10 and visual examples in Figure 11.Source: C. Additional Results for Ablation… · p. 12
Positional encoding
- Positional encoding improves performance on synthetic and real cases and helps learn fine details of small or heavily occluded objects.The qualitative examples are shown in Figure 12.Source: C. Additional Results for Ablation… · p. 12
Number of voxels
- Accuracy drops when the voxel count is too large; fewer voxels make offset regression harder, while more voxels can split objects.The examples also show that too few voxels can produce incorrect object orientation.Source: C. Additional Results for Ablation… · p. 12
Training Data
- Training on both datasets achieves the best results among the compared training-data choices.The passage points to Table 11 and Figure 14 for quantitative and qualitative comparisons.Source: C. Additional Results for Ablation… · pp. 12–13
Ray Pooling and Candidate Points Selection
- Argmax outperforms weighted sum across all testing-data types, and learned candidate-point offsets outperform heuristic sampling.The passage also reports that learned offsets are more robust to strong background textures in real images.Source: C. Additional Results for Ablation… · p. 13
D. Qualitative Results on NYUV2 Dataset
On NYUV2 scenes with large randomly created depth holes, the method predicts reasonable missing depth from the corrupted depth map and RGB image.
D. Qualitative Results on NYUV2 Dataset
- On NYUV2 general scenes and non-transparent objects, the method predicts reasonable missing depth from RGB images and depth maps corrupted with large random holes.The models were trained for this evaluation to complete the corrupted depth map.Source: D. Qualitative Results on NYUV2 Dataset · p. 13
E. Failure Cases
The approach has two stated failure modes: cracks from inconsistent voxel assignments and floating objects when table contact is unconstrained.
E. Failure Cases
- Pixels from one object can be assigned to different terminating voxels, creating reconstruction cracks; without a table-contact constraint, objects can float.Source: E. Failure Cases · p. 13
F. Discussion and Future Works
The discussion identifies sensor noise and heavy occlusion as future-work targets; accompanying figures and tables describe qualitative examples and ablation-study scopes.
F. Discussion and Future Works
- Future work proposes modeling each pixel projection as a cone for lateral noise, using realistic depth-noise training data, and studying cluttered scenes with heavy occlusion.Source: F. Discussion and Future Works · p. 13
- A refinement-model table distinguishes first-stage predictions from predictions made by the refinement model.Source: Table 8 · p. 13
- Ablation tables cover embedding modalities, ray information, positional encoding, voxel count, training data, ray pooling, and candidate-point selection, without captioned outcomes.Source: Table 9 · p. 13, Table 10 · p. 13, Table 11 · p. 14
- Qualitative figures compare input modalities, ray information, positional encoding, voxel counts, training data, ray pooling, and candidate-point selection.Source: Figure 10 · p. 14, Figure 11 · p. 14, Figure 12 · p. 15
- Failure examples show cracks when one object's pixels map to different terminating voxels and floating objects when table contact is unconstrained.Source: Figure 17 · p. 16
- The NYUV2 examples show input RGB and depth alongside predicted and ground-truth depth and point clouds.Source: Figure 18 · p. 17