Source-linked AI summary
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition
Mikaela Angelina Uy, Gim Hee Lee
TL;DR
Point-cloud place recognition lacks local descriptors that can be aggregated into global retrieval representations, despite LiDAR’s useful geometric robustness. PointNetVLAD combines PointNet and NetVLAD and trains the resulting descriptor with lazy metric-learning losses. Benchmark experiments show the feasibility and usability of this approach for large-scale LiDAR place recognition.
Problem
Point-cloud place recognition lacks local feature descriptors that can be encoded into global descriptors for retrieval, limiting the standard image-based pipeline.
Method
PointNetVLAD combines PointNet and NetVLAD for end-to-end global-descriptor extraction and uses lazy triplet and quadruplet metric-learning losses.
Results
Experimental results on benchmark datasets show the feasibility and usability of PointNetVLAD for large-scale point-cloud place recognition.
Takeaways & Limitations
The proposed network provides a demonstrated approach to the largely unexplored problem of point-cloud-based retrieval for place recognition.
Abstract
from arXiv · showhide
Unlike its image based counterpart, point cloud based retrieval for place recognition has remained as an unexplored and unsolved problem. This is largely due to the difficulty in extracting local feature descriptors from a point cloud that can subsequently be encoded into a global descriptor for the retrieval task. In this paper, we propose the PointNetVLAD where we leverage on the recent success of deep networks to solve point cloud based retrieval for place recognition. Specifically, our PointNetVLAD is a combination/modification of the existing PointNet and NetVLAD, which allows end-to-end training and inference to extract the global descriptor from a given 3D point cloud. Furthermore, we propose the "lazy triplet and quadruplet" loss functions that can achieve more discriminative and generalizable global descriptors to tackle the retrieval task. We create benchmark datasets for point cloud based retrieval for place recognition, and the experimental results on these datasets show the feasibility of our PointNetVLAD. Our code and the link for the benchmark dataset downloads are available in our project website. http://github.com/mikacuy/pointnetvlad/
1. Introduction
LiDAR-based place recognition is important for localization but remains underdeveloped because robust local descriptors for outdoor point clouds are unavailable. PointNetVLAD addresses this gap with an end-to-end network for extracting global descriptors from 3D point clouds.
- Localization retrieves a matching scene from a reference map to support autonomous robots such as self-driving cars and drones.
- Image-based place recognition aggregates local features such as SIFT into global descriptors for efficient database retrieval.
- LiDAR place recognition lacks a local feature extractor comparable to SIFT, preventing the usual bag-of-words global-descriptor pipeline.
- LiDAR offers precise depth and geometric information that is robust to lighting changes, but existing systems often rely on GPS before point-cloud registration.
- PointNetVLAD combines PointNet and NetVLAD for end-to-end global-descriptor extraction, using metric learning and lazy triplet and quadruplet losses.
2. Related Work
Prior 3D descriptors are limited by surface stability or density, while deep point-cloud methods address unordered inputs but had not demonstrated large-scale place recognition. PointNetVLAD combines PointNet and NetVLAD to extend end-to-end retrieval to 3D point clouds.
- Existing handcrafted 3D descriptors are unsuitable for broad outdoor LiDAR retrieval because some require stable surfaces and others require high point density.
- Deep learning is harder to apply to point clouds than images because 3D points are unordered rather than arranged on a regular lattice.
- PointNet directly accepts point clouds and uses symmetric max pooling to produce permutation-invariant representations, but its demonstrated tasks were classification and segmentation.
- NetVLAD aggregates learned local image features into global descriptors, but it is not designed to accept 3D points as input.
- PointNetVLAD combines or modifies PointNet and NetVLAD to perform end-to-end 3D point-cloud place recognition.
3. Problem Definition
The task divides a reference map into comparable submaps and retrieves the submap structurally most similar to a query. A learned descriptor function supports this by placing similar clouds closer than dissimilar ones under a distance metric.
- The reference map M is divided into submaps with approximately equal area coverage and far fewer points than the full map.
- Downsampling ensures that all submaps have the same number of points before retrieval.
- Given a query with comparable coverage and point count, retrieval seeks the database submap that is structurally most similar.
- The learned function f(.) maps a downsampled point cloud to a fixed-size global descriptor.
- For structurally similar and dissimilar clouds, descriptor distances should satisfy d(f(p̄),f(p̄r)) < d(f(p̄),f(p̄s)); retrieval then selects the minimum-distance database descriptor.
4. Our PointNetVLAD
PointNetVLAD combines PointNet, NetVLAD, and a fully connected layer to map unordered 3D point clouds to compact global descriptors. Lazy triplet and quadruplet losses focus training on hard negatives, while NetVLAD's symmetric aggregation preserves permutation invariance.
- Network Architecture: PointNetVLAD uses PointNet to learn D-dimensional local descriptors independently for each input 3D point.These descriptors are then passed to NetVLAD for aggregation.
- Network Architecture: NetVLAD learns K cluster centers and aggregates local descriptors into a (D × K)-dimensional VLAD vector.The cluster centers act as visual words, and learned weights and biases determine local-feature contributions.
- Network Architecture: A fully connected layer compresses the high-dimensional VLAD vector, and L2 normalization produces the compact descriptor f(P) ∈ R^O for efficient retrieval.The output dimension satisfies O ≪ (D × K).
- Metric Learning: Lazy triplet and quadruplet losses minimize anchor-positive descriptor distance while maximizing distance from hard negatives.The quadruplet variant adds a distance term involving a randomly sampled negative dissimilar to all clouds in the training tuple.
- Metric Learning: The max operator selects the closest negative, and the authors report that both lazy variants outperform the original losses, with lazy quadruplet slightly better than lazy triplet.The authors attribute this to concentrating training on the hardest negative rather than summing contributions from all negatives.
- Permutation Invariance: PointNetVLAD is permutation invariant because PointNet processes points independently and NetVLAD sums their contributions symmetrically.Reordering input points therefore leaves the global descriptor unchanged.
5. Experiments
Experiments evaluate PointNetVLAD on benchmark LiDAR datasets, comparing it with baselines, training choices, descriptor dimensions, and image-based retrieval. The results show stronger generalization and retrieval performance for PointNetVLAD, including under extended databases and changing scene conditions.
- 5.1. Benchmark Datasets: Four LiDAR place-recognition datasets comprise Oxford and three in-house regions: U.S., R.A., and B.D.Routes span 10km, 10km, 8km, and 5km, respectively.
- 5.1. Benchmark Datasets: Ground removal, voxel downsampling to 4096 points, normalization to [-1, 1], and distance-based tuple construction prepare the submaps for supervised learning.Structurally similar clouds are at most 10m apart, while dissimilar clouds are at least 50m apart.
- 5.1. Benchmark Datasets: The evaluation uses disjoint training and testing reference maps, with 28,382 training submaps and 7,572 test submaps; retrieval succeeds within 25m.Queries come from one testing reference map and retrieve against another reference map.
- 5.2. Results: When trained only on Oxford, PN VLAD outperforms PN STD and PN MAX across the reported baseline datasets and generalizes better to unseen environments.The comparison uses average recall at top 1%, with recall curves covering the top 25 database matches.
- 5.2. Results: A 256-dimensional PN VLAD descriptor is selected because performance improves from 128 to 256 dimensions but not further to 512 dimensions.At 128 dimensions, PN VLAD is on par with 512-dimensional PN MAX on Oxford and marginally better on the in-house datasets.
- 5.2. Results: Lazy triplet and quadruplet losses improve performance, and adding U.S. and R.A. training data improves generalization to unseen B.D. data.PN VLAD also outperforms PN MAX on an approximately 33km extended database and significantly outperforms image-based NetVLAD for overcast-day to night retrieval.
- 5.2. Results: Point clouds make retrieval more robust to illumination and weather changes, while failures remain on visually similar continuous roads and heavily occluded areas.Qualitative examples also show the network ignoring ground snow and parked or moving cars.
- 5.2. Results: Inference takes approximately 9ms on an NVIDIA GeForce GTX 1080Ti, and submap retrieval uses O(log n) search.The reported implementation is intended for real-time robotics systems.
6. Conclusion
The paper presents PointNetVLAD for large-scale point-cloud place recognition and reports permutation-invariant, discriminative global descriptors, with benchmark experiments showing feasibility and usability.
- The network maps unordered 3D point clouds to compact, discriminative global descriptors while remaining permutation invariant.
- Lazy triplet and quadruplet losses produce more discriminative and generalizable global descriptors.
- Benchmark-dataset experiments demonstrate the feasibility and usability of point-cloud retrieval for place recognition.
A. Benchmark Datasets
The benchmark datasets draw on the Oxford RobotCar dataset and three in-house datasets, using regional submaps and disjoint reference-map splits.
- The benchmark datasets comprise the Oxford RobotCar dataset and three in-house datasets.
- Each region includes a reference map and submaps representing local areas within that map.
- Data splitting uses disjoint reference maps created by randomly selecting 150m × 150m regions.
B. Implementation Details
Training uses online tuple construction with hard-negative mining, while the benchmark figures illustrate representative submaps and the spatial split between training and testing maps.
- Implementation Details: Each training iteration uses a batch size of 3 tuples, each containing one anchor, one positive, and 18 negative point clouds.
- Implementation Details: Hard-negative mining selects the hardest negatives from 2000 randomly sampled candidates for each anchor.
- Benchmark Visualization: The benchmark figures show reference maps, local submaps, and their corresponding preprocessed submaps across four regions.
- Data Splitting: Training and testing reference maps are represented by blue and red submaps, respectively, after randomly selecting regions from the full map.