Source-linked AI summary
3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser
TL;DR
Matching local geometry in noisy, low-resolution, incomplete 3D scans remains difficult for existing hand-crafted descriptors. 3DMatch learns a volumetric local descriptor from correspondence labels automatically obtained from RGB-D reconstructions, and it outperforms prior approaches across matching and registration while generalizing across tasks and spatial scales.
Problem
Matching local geometric features in real-world 3D scans is challenging because the data are noisy, low-resolution, and incomplete, limiting existing histogram-based descriptors.
Method
3DMatch is a data-driven 3D ConvNet that learns local volumetric patch descriptors using correspondence labels from existing RGB-D reconstructions.
Results
3DMatch consistently outperforms state-of-the-art methods for keypoint matching and geometric registration and generalizes to object alignment and mesh surface correspondence.
Takeaways & Limitations
Automatically obtained reconstruction correspondences can train a local descriptor that matches partial 3D geometry across applications and spatial resolutions.
Abstract
from arXiv · showhide
Matching local geometric features on real-world depth images is a challenging task due to the noisy, low-resolution, and incomplete nature of 3D scan data. These difficulties limit the performance of current state-of-art methods, which are typically based on histograms over geometric properties. In this paper, we present 3DMatch, a data-driven model that learns a local volumetric patch descriptor for establishing correspondences between partial 3D data. To amass training data for our model, we propose a self-supervised feature learning method that leverages the millions of correspondence labels found in existing RGB-D reconstructions. Experiments show that our descriptor is not only able to match local geometry in new scenes for reconstruction, but also generalize to different tasks and spatial scales (e.g. instance-level object model alignment for the Amazon Picking Challenge, and mesh surface correspondence). Results show that 3DMatch consistently outperforms other state-of-the-art approaches by a significant margin. Code, data, benchmarks, and pre-trained models are available online at http://3dmatch.cs.princeton.edu
1. Introduction
Matching local geometry in real-world 3D scans is difficult because the data are noisy, low-resolution, and partial, while hand-crafted descriptors are unstable on incomplete surfaces. 3DMatch addresses this with a learned local descriptor trained from correspondence labels in RGB-D reconstructions.
- Real-world 3D scans are difficult for local geometric matching because they are low-resolution, noisy, and partial.
- Hand-crafted descriptors based on histograms over geometric properties work well on complete models but are unstable on partial scan surfaces and difficult to adapt.
- 3DMatch is a data-driven 3D ConvNet that maps a local volumetric patch to a descriptor, with smaller descriptor distance indicating higher correspondence likelihood.
- The method obtains massive supervision from correspondence labels in existing RGB-D reconstructions rather than manual 3D keypoint annotations.
- 3DMatch is trained on 8 million correspondences from 62 RGB-D reconstructions and evaluated across keypoint matching, geometric registration, object alignment, and mesh correspondence.
2. Related Work
Prior work includes hand-crafted local descriptors, learned 2D and global 3D representations, and learned local descriptors. 3DMatch differs by learning spatially coherent volumetric features for noisy, partial real-world 3D data.
- Hand-crafted 3D Local Descriptors: Hand-crafted 3D descriptors struggle with noisy, low-resolution, incomplete range-sensor data and often generalize poorly across modalities.
- Learned 2D Local Descriptors: Learned 2D descriptors benefit from large labeled image datasets, but image correspondences and multi-view stereo are unsuitable for learning a general 3D surface descriptor.
- Learned 3D Global Descriptors: Learned global 3D methods focus on complete object models, whereas 3DMatch targets local geometric features in real-world RGB-D scans.
- Learned 3D Local Descriptors: Compared with a related 2D ConvNet approach, 3DMatch handles real-world partial data and uses spatially coherent 3D convolutions on volumetric input.
- Self-supervised Deep Learning: 3DMatch follows self-supervised learning work that obtains training labels automatically, using reconstruction correspondences for geometric descriptor learning.
3. Learning From Reconstructions
3DMatch learns local geometric descriptors from existing RGB-D reconstructions. These reconstructions provide many cross-view correspondence examples whose variation exposes the model to noise, viewpoint changes, occlusion, and geometric diversity.
- The learned function maps a local volumetric 3D patch to a descriptor, where smaller ℓ2 distance indicates a higher likelihood of correspondence.
- RGB-D reconstructions provide millions of training correspondences because points are observed from multiple scanning views.
- Training patches and match/non-match labels are extracted by projecting reconstructed interest points into visible, non-occluded RGB-D frames.
- Sensor noise and reconstruction imperfections introduce patch drift that provides an opportunity to learn small amounts of translation invariance.
- Using reconstructions from different sensors and algorithms exposes the descriptor to inconsistent jitter and supports robustness to that variation.
4. Learning A Local Geometric Descriptor
3DMatch represents local geometry as TDF voxel grids and processes them with a 3D ConvNet trained in a Siamese configuration. The descriptor is intended to cluster patches with similar geometric structure despite noisy and partial observations.
- 3D Data Representation: Each local region is converted into a 30 × 30 × 30 TDF voxel grid spanning 0.3m3, providing a volumetric encoding compatible with meshes, point clouds, and depth maps.
- 3D Data Representation: Removing TDF signs concentrates the largest voxel-value gradients around surfaces, a property described as crucial for robustness on partial data.
- Descriptor Behavior: The learned descriptors cluster local patches by geometric structures such as edges, planes, and corners in noisy and partial data.
- Network Architecture: The 3D ConvNet processes a 30×30×30 TDF grid through eight convolutional layers and one pooling layer to produce a 512-dimensional descriptor.
- Network Training: Siamese training shares weights across two streams and optimizes descriptors to be similar for corresponding patches and dissimilar for non-corresponding patches.
5. Evaluation
The evaluation tests 3DMatch for keypoint matching, geometric registration, reconstruction, and generalization across tasks and spatial scales. Across these settings, the learned descriptor supports challenging real-world scan alignment and improves registration and reconstruction quality.
- Evaluation scope: The evaluation covers local patch matching, geometric registration, scene reconstruction, 6D object pose estimation, and mesh surface correspondence.The experiments progress from descriptor discrimination to practical registration and cross-domain applications.
- Keypoint matching: 30,000 3D patches with a 1:1 match-to-non-match ratio are evaluated using false-positive rate at 95% recall, where lower error is better.The benchmark uses local 3D patches extracted from depth frames.
- Keypoint matching: 35.3 versus 38.5 error shows the 3D ConvNet outperforms a comparable 2D depth-patch ConvNet on keypoint matching.The 3D representation can encode real-world spatial scale and occluded regions, unlike 2D depth patches.
- Keypoint matching: 33.1% error is achieved by a metric-network variant, but the ℓ2-metric descriptor is retained because learned metrics require O(n^2) test-time comparisons.The retained version can be combined with metric-based acceleration structures such as KD-trees.
- Geometric registration: 66.8% recall and 40.1% precision are obtained by 3DMatch + RANSAC on fused fragments of real-world scans, versus 44.2% and 30.7% for Rusu et al. + RANSAC.Registration estimates a rigid transformation from mutually closest descriptors and uses RANSAC over matched 3D keypoint positions.
- Generalization and reconstruction: 61.0% rotation and 71.7% translation accuracy are reported for model fitting, while reconstruction examples show geometric matches complement color features.The pose metric counts predictions within 15° orientation error and 5 cm translation error; without reconstruction pre-training, performance falls to 53.8% and 69.1%.
6. Conclusion
3DMatch is presented as a learned descriptor for matching partial 3D data across applications, supported by automatically obtained RGB-D correspondences and released research resources.
- 3DMatch is a 3D ConvNet-based local geometric descriptor for matching partial 3D data across varied applications.
- RGB-D reconstructions provide vast correspondence datasets that enable training a descriptor outperforming existing geometric descriptors by a significant margin.
- The authors release code, pretrained models, and real-world correspondence-matching and surface-registration benchmarks to support further research.
A. Appendix
The appendix summarizes dataset statistics, network implementation details, and runtime measurements for the experiments.
- The appendix reports statistics for RGB-D reconstruction datasets used to generate 3DMatch training correspondences.
- It documents the network implementation used in the experiments.
- It also provides runtime statistics relevant to the reported experiments.
A.1. RGB-D Reconstruction Datasets
Training uses registered depth frames from 62 real-world scenes across five RGB-D datasets, with 54 scenes for training and 8 for testing.
- Registered depth frames from 62 real-world scenes supply the RGB-D reconstruction data.
- The scenes come from Analysis-by-Synthesis, 7-Scenes, SUN3D, RGB-D Scenes v.2, and Halber et al.
- The dataset split uses 54 scenes for training and 8 scenes for testing.
- Training-set correspondence size depends on overlap between visible surfaces across scanning views.
A.2. Implementation Details
The network is trained with SGD and on-the-fly sampled matching and non-matching 3D patch pairs, using a batch size of 128 and a contrastive margin of 1.
- 3DMatch is implemented in Marvin, with Xavier-initialized weights and zero-initialized biases.
- Training uses SGD with momentum, a fixed learning rate of 10^-3, momentum 0.99, and weight decay of 54.
- Matching and non-matching 3D patches are sampled on-the-fly during training with batch size 128 and contrastive margin 1.
- The reference model trained for approximately eight days on one NVIDIA Tesla K40c over 16 million patch pairs.
A.3. Run-time Information
The reported implementation runtimes include TDF conversion for local 3D patches represented as 30×30×30 voxel grids. Converting a 0.3m3 depth-frame point cloud takes 3–20 milliseconds with CUDA-enabled GPU acceleration, depending on point density.
- TDF Conversion: 3–20 milliseconds are required to convert a 0.3m3 depth-frame point cloud into a TDF voxel grid using CUDA-enabled GPU acceleration.The runtime depends on the point-cloud density, and the implementations were not optimized for speed.