Source-linked AI summary
2D3D-MatchNet: Learning to Match Keypoints Across 2D Image and 3D Point Cloud
Mengdan Feng, Sixing Hu, Marcelo Ang, Gim Hee Lee
TL;DR
Visual pose estimation lacks direct descriptors for matching image keypoints with LiDAR point-cloud keypoints, limiting use of more accurate 3D maps. The paper introduces 2D3D-MatchNet to jointly learn cross-domain descriptors and establish correspondences for pose estimation, achieving feasible localization with reported errors of 1.41 m and 6.40 degrees on Oxford RobotCar.
Problem
LiDAR point-cloud maps are seldom used for visual pose estimation because direct 2D image-to-3D point-cloud keypoint correspondences are difficult to obtain.
Method
2D3D-MatchNet jointly learns descriptors for 2D image patches and 3D local point-cloud volumes, using their matches to establish correspondences for pose estimation.
Results
The method achieved average translation and rotation errors of 1.41m and 6.40 degree on the standard Oxford RobotCar dataset.
Takeaways & Limitations
The network directly estimates camera pose from a point-cloud reference map through learned 2D image-to-3D point-cloud descriptor matching.
Takeaways & Limitations
Localization is likely to fail in scenes dominated by trees or flat building walls because image and point-cloud keypoint distinctiveness can differ across these structures.
Abstract
from arXiv · showhide
Large-scale point cloud generated from 3D sensors is more accurate than its image-based counterpart. However, it is seldom used in visual pose estimation due to the difficulty in obtaining 2D-3D image to point cloud correspondences. In this paper, we propose the 2D3D-MatchNet - an end-to-end deep network architecture to jointly learn the descriptors for 2D and 3D keypoint from image and point cloud, respectively. As a result, we are able to directly match and establish 2D-3D correspondences from the query image and 3D point cloud reference map for visual pose estimation. We create our Oxford 2D-3D Patches dataset from the Oxford Robotcar dataset with the ground truth camera poses and 2D-3D image to point cloud correspondences for training and testing the deep network. Experimental results verify the feasibility of our approach.
I. INTRODUCTION
Visual pose estimation depends on reliable 2D–3D correspondences, but LiDAR point-cloud maps lack descriptors that directly match image and 3D keypoints. The paper proposes jointly learned cross-domain descriptors and introduces a correspondence dataset.
- Visual pose estimation estimates camera pose relative to a reference 3D point-cloud map and supports vSLAM, SfM, autonomous driving, and augmented reality.
- Standard pose estimation first establishes 2D–3D keypoint correspondences, then applies PnP using at least three correspondences.
- LiDAR maps are seldom used because direct matching descriptors between 2D image keypoints and 3D point-cloud keypoints are unavailable.
- The proposed 2D3D-MatchNet jointly learns descriptors for image patches and local point-cloud volumes using existing SIFT and ISS keypoint detectors.
III. APPROACH
The approach maps image patches and local 3D point-cloud volumes into jointly learned descriptor spaces, then uses their matches to support camera-pose estimation.
- The pipeline takes a 2D query image and a LiDAR-built 3D point-cloud reference map, then learns descriptors for their keypoints.
- The resulting descriptor matches establish 2D–3D correspondences that are used for camera-pose estimation.
- SIFT extracts 2D keypoints and ISS extracts 3D keypoints before the network processes centered image patches and local point-cloud volumes.
- The network produces 2D and 3D descriptors whose matching-pair distance is much smaller than the distance for nonmatching pairs.
B. Our 2D3D-MatchNet: Network Architecture
2D3D-MatchNet is a triplet-like architecture that learns comparable descriptors from image patches and local point-cloud volumes using positive and negative training examples.
- The network has one image branch and two shared-weight point-cloud branches, taking image patches and fixed-radius local point-cloud spheres as inputs.
- Training tuples contain an anchor image patch, a positive point-cloud volume, and a negative point-cloud volume.
- The image and point-cloud branches learn descriptor functions that map their respective inputs into descriptor vectors.
- The image descriptor uses VGG16 convolutional blocks with global average pooling, while the point-cloud descriptor uses PointNet with L2 normalization.
- Training uses a weighted soft-margin triplet loss to reduce positive-pair distance relative to negative-pair distance without selecting an optimal margin.
C. Pose Estimation
The pose-estimation stage matches descriptors across all detected 2D and 3D keypoints, retains nearest candidates, and estimates pose robustly with EPnP inside RANSAC.
- SIFT and ISS detect the query-image and reference-map keypoints, respectively, and the network computes descriptors from their local inputs.
- For every 2D keypoint, the method selects the top K closest 3D keypoints according to descriptor similarity.
- EPnP estimates the camera pose from the putative 2D–3D correspondences, while RANSAC removes outliers for robust estimation.
IV. DATASET
The Oxford 2D-3D Patches Dataset contains 432,982 image-patch to point-cloud pairs for training and evaluating 2D-3D feature matching.
- 432,982 image-patch to point-cloud pairs comprise the dataset for 2D-3D feature matching.
A. The Oxford 2D-3D Patches Dataset
The dataset is built from Oxford RobotCar camera, Lidar, and GPS/INS data, organized into 60m submaps with filtered ISS-SIFT correspondences and normalized local inputs.
- Oxford RobotCar supplies monocular images, front 2D Lidar scans, and GPS/INS data for dataset construction.
- Traversals are split into disjoint 60m submaps to simplify point-cloud processing and reduce long-distance GPS-jump effects.
- Keypoint Detection: Ground-plane points are removed before ISS detects 3D keypoints, while SIFT detects 2D keypoints in images.
- 2D-3D Correspondences: ISS keypoints are projected into visible images and paired with nearest SIFT keypoints subject to a 3-pixel distance and multi-view correspondence requirement.
- ISS Volume and SIFT Patch Extraction: Selected ISS neighborhoods use a 1m radius and require at least 100 neighboring points, while SIFT scales above 4 are discarded and patches are 256 × 256.
- Data Pre-processing: SIFT patches are resized to 128×128 and zero-centered, while point-cloud volumes are centered, normalized, and padded to 1024 points.
C. Testing Data Generation
Testing evaluates localization on reference submaps and images excluded from training, using GPS/INS poses and generated ground-truth 2D-3D correspondences for verification.
- The trained network is tested on reference submaps and images not used during training.
- GPS/INS image poses provide ground-truth poses for verifying localization and computing ground-truth 2D-3D correspondences.
- The evaluation section covers network training, camera pose estimation, method comparison, and localization analysis.
A. Network Training
Training uses disjoint submap splits and an end-to-end triplet setup, while localization is assessed with pose thresholds, top-K recall, and descriptor-dimension comparisons.
- Data splitting and evaluation metric: One traversal is reserved for testing; for the other 35, the first 90% of submaps train the model and the final 10% remain unseen for testing.
- Data splitting and evaluation metric: Localization precision counts queries within a (10m, 45°) position-and-rotation threshold and reports average position and rotation errors.
- Network Training: Each triplet uses an image patch as anchor, its corresponding 3D volume as positive, and a randomly sampled point-cloud volume as negative.
- Network Training: The network is trained with Adam at an initial learning rate of 6×10^-5, requiring around two days of training.
- Network results: Localization is tested with descriptor dimensions D ∈ {64,128,256}.
B. Results
The method localizes query images by matching learned 2D image descriptors to 3D point-cloud descriptors and solving camera pose with EPnP. Results are reported for full traversals and unseen submaps, including success-rate curves and qualitative failure visualizations.
- The system extracts query-image descriptors, retrieves top-K nearest point descriptors, and establishes 2D-3D correspondences for localization.The experiments choose K = 5 as a trade-off between candidate coverage and localization performance.
- Camera pose is estimated with EPnP, and localization results are reported for full tests and submap tests across traversals.The full test excludes submaps with bad point-cloud maps caused by GPS inaccuracy, while submap tests cover several traversal times.
- Qualitative localization results are visualized with output feature dimension D set to 128.The reported visualization includes localization outcomes for the selected descriptor dimension.
- Figure 5 identifies ORB-SLAM2 failure positions and shows an image from a failure case.The figure provides a qualitative view of where the ORB-SLAM2 localization result fails.
- The ratio of successfully localized frames is evaluated as a function of error thresholds for both full-test and submap-test settings.Figure 6 compares the proposed method with ORB-SLAM2 on the full test and reports average submap-test results.
C. Evaluation and Comparison
The evaluation uses a large Oxford RobotCar traversal dataset to compare the proposed point-cloud-based localization with ORB-SLAM2 and PoseNet. The method localizes more than 40% of images across the whole testing area, generalizes closely from seen to unseen areas, and has scene-dependent failure modes.
- Evaluation setup: The evaluation uses 8,500 images from a 3 km path covering 845m × 617m, comparing ORB-SLAM2 and PoseNet with the proposed method.The images were collected in overcast afternoon conditions on July 14, 2015 at 16Hz.
- Comparison: ORB-SLAM2 cannot build the whole testing-area map, and its localization errors increase when mapping is inaccurate near trees or turns.PoseNet also shows huge localization error and does not generalize to unseen test sequences under the described vehicle-camera data.
- Comparison: More than 40% of images are successfully localized throughout the whole testing area, whereas existing algorithms fail in large-scale urban environments.The paper presents this as the principal whole-area localization result.
- Generalization: Submap-test results are slightly worse than full-test results but remain close, indicating generalization from seen to totally unseen testing areas.The paper interprets the proximity between the two settings as good generalization of the proposed network.
- Descriptor dimension: Descriptor dimension D = 128 localizes more images than D = 64 or D = 256 in the tested submaps.The paper attributes the choice to a trade-off between representation capacity, over-fitting risk, accuracy, and inference efficiency.
- Failure analysis: Localization is likely to fail in tree-heavy scenes and flat-wall scenes because trees produce confusing matches while smooth walls provide few 3D keypoints.These failure modes arise from mismatches between image texture, detected keypoints, and available discriminative point-cloud descriptors.
VI. CONCLUSION
The method estimates camera pose from a 3D point-cloud reference map by jointly learning image and point-cloud descriptors to obtain 2D-3D correspondences. On Oxford RobotCar, it achieved average translation and rotation errors of 1.41m and 6.40 degree.
- The qualitative visualization marks estimated cameras in red, ground-truth cameras in yellow, and predicted correspondences with purple lines.
- The method jointly learns image and point-cloud descriptors to obtain 2D-3D correspondences for camera pose estimation.
- 1.41m translation error and 6.40 degree rotation error were achieved on the standard Oxford RobotCar dataset.