PaperlayerView exact revision on arXiv

Source-linked AI summary

DH3D: Deep Hierarchical 3D Descriptors for Robust Large-Scale 6DoF Relocalization

Juan Du, Rui Wang, Daniel Cremers

arXiv:2007.09217v1cs.CV

TL;DR

Large-scale point-cloud relocalization research had addressed global place recognition and local 6DoF pose refinement separately. DH3D jointly learns local feature detection and description and aggregates those descriptors for global retrieval. It reports state-of-the-art results across multiple benchmarks and favorable transfer to visual-SLAM point clouds without fine-tuning.

  • Problem

    Point-cloud relocalization research had treated global place recognition and local 6DoF pose refinement in isolation, despite both depending on shared low-level geometric clues.

    Source: §1 · pp. 1–2

  • Method

    DH3D learns local feature detection and description from raw 3D points and aggregates the local descriptors into a global descriptor in one forward pass.

    Source: §1 · pp. 2–3

  • Results

    DH3D achieves state-of-the-art performance for local and global descriptors across multiple benchmarks and favorable LiDAR-to-visual-SLAM generalization without fine-tuning.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The unified workflow supports large-scale outdoor 6DoF relocalization, and LiDAR-trained local descriptors also perform favorably on visual-SLAM point clouds.

    Source: §1 · pp. 2–3, §5 · p. 14

Abstract

from arXiv · show

For relocalization in large-scale point clouds, we propose the first approach that unifies global place recognition and local 6DoF pose refinement. To this end, we design a Siamese network that jointly learns 3D local feature detection and description directly from raw 3D points. It integrates FlexConv and Squeeze-and-Excitation (SE) to assure that the learned local descriptor captures multi-level geometric information and channel-wise relations. For detecting 3D keypoints we predict the discriminativeness of the local descriptors in an unsupervised manner. We generate the global descriptor by directly aggregating the learned local descriptors with an effective attention mechanism. In this way, local and global 3D descriptors are inferred in one single forward pass. Experiments on various benchmarks demonstrate that our method achieves competitive results for both global point cloud retrieval and local point cloud registration in comparison to state-of-the-art approaches. To validate the generalizability and robustness of our 3D keypoints, we demonstrate that our method also performs favorably without fine-tuning on the registration of point clouds that were generated by a visual SLAM system. Code and related materials are available at https://vision.in.tum.de/research/vslam/dh3d.

1 Introduction

Point-cloud relocalization research had treated global place recognition and local 6DoF pose refinement separately, despite their reliance on shared geometric clues. DH3D unifies these tasks and reports strong benchmark performance, including favorable transfer from LiDAR-trained descriptors to visual-SLAM point clouds.

1 Introduction

  • Point-cloud relocalization research was less mature than image-based approaches, and global recognition and local pose refinement had been studied separately.Both tasks depend on the same low-level geometric clues.Source: §1 · pp. 1–2
  • An unsupervised describe-and-detect approach estimates local-feature discriminativeness to select keypoints suited for matching.The method explicitly learns a 3D keypoint detector.Source: §1 · pp. 2–3
  • DH3D unifies global place recognition and local 6DoF pose refinement, extracting local features and a global descriptor in one forward pass.The global descriptor supports retrieval; local feature matching supports pose refinement.Source: Figure 1 · p. 2, §1 · p. 3
  • FlexConv and Squeeze-and-Excitation integrate multi-level context and channel-wise relations into local features.The paper reports stronger feature matching and a boosted global descriptor.Source: §1 · p. 3
  • The descriptors achieve state-of-the-art performance across multiple point-cloud registration and retrieval benchmarks, with favorable LiDAR-to-visual-SLAM generalization.The visual-SLAM evaluation is reported without fine-tuning despite different point-cloud distributions.Source: §1 · p. 3

2 Related Work

Prior work spans handcrafted and learned local descriptors, keypoint detectors, and global place descriptors. The reviewed approaches differ in their use of contextual information and whether they detect features or rely on sampling.

2 Related Work

  • Handcrafted local descriptors encode geometric properties as histograms, but are reported to generalize poorly to large-scale scenarios and noisy real-world data.Examples include Spin Image, USC, PFH, and FPFH.Source: §2 · p. 3
  • Learned local descriptors include indoor RGB-D methods and outdoor relocalization systems, while many obtain features by sampling rather than learning detection.The surveyed outdoor approaches include 3DFeatNet, L3-Net, 3DSmoothNet, and DeepVCP.Source: §2 · p. 3
  • Only a few methods learn 3D keypoint detection; USIP is identified as the then-current state of the art for keypoint detectors.Other examples include handcrafted ISS, SIFT-3D, and Harris-3D, and learned attention-based approaches.Source: §2 · p. 4
  • Global descriptors range from handcrafted statistics to learned systems, with PointNetVLAD and PCAN using PointNet features that lack explicit contextual information.PCAN learns point attentions, while PointNetVLAD uses a NetVLAD layer over PointNet features.Source: §2 · p. 4

3 Hierarchical 3D Descriptors Learning

The hierarchical network jointly supplies local descriptors, keypoint scores, and a global descriptor from one point-cloud pass. It uses global retrieval for coarse submap search and local matching for 6DoF pose estimation.

3 Hierarchical 3D Descriptors Learning

  • The network delivers local descriptors, keypoint scores, and a global descriptor in one pass, enabling coarse global search followed by local matching for pose estimation.Source: §3 · p. 4

3.1 3D Local Feature Encoder and Detector

  • The encoder combines two spatial resolutions, FlexConv neighborhood structure, and SE channel recalibration to form L2-normalized local descriptors.Source: §3.1 · pp. 5–6
  • Local descriptors are trained with N-tuple loss over positive and negative patch pairs, normalized by true and false match counts to counter negative-pair imbalance.Source: §3.1 · pp. 6–7
  • The detector predicts saliency from learned descriptors and trains keypoint confidences using matching success, measured by AR over k nearest candidates and threshold κ.Source: §3.1 · p. 7

3.2 Global Descriptor Learning

  • For retrieval, local descriptors feed two FlexConv projections, a three-layer softmax attention map, NetVLAD aggregation, and fully connected compression.Source: §3.2 · pp. 7–8

4 Experiments

Experiments evaluate keypoint repeatability, local registration, global retrieval, visual-SLAM generalization, and component robustness. DH3D performs strongly across these tests, with performance varying by dataset and perturbation.

4 Experiments

  • The experiments train on Oxford RobotCar LiDAR data and use ETH and Stereo DSO point clouds to test generalization.Local training uses 20,731 submaps; a point cloud with 8,192 points takes 80ms for local and global descriptor extraction in one forward pass.Source: §4 · pp. 8–9

4.1 3D Keypoint Repeatability

  • At 256 keypoints, DH3D’s repeatability is roughly 1.75x that of 3DFeatNet, and it outperforms competitors when detecting more than 64 keypoints.USIP dominates the overall repeatability benchmark, though its keypoints are highly clustered.Source: §4.1 · p. 9, Figure 4 · p. 10

4.2 Point Cloud Registration

  • On Oxford RobotCar, DH3D delivers competitive or better registration results despite USIP’s stronger keypoint repeatability.Registration success requires RTE below 2m and RRE below 5°; Table 1 reports results by detector and descriptor combination.Source: §4.2 · pp. 10–11, Table 1 · p. 10
  • On ETH, DH3D’s detector performs fairly well and paired with its descriptor achieves the smallest rotation error, while 3DSmoothNet performs best overall.The authors note that 3DSmoothNet uses finer voxel downsampling (0.02m versus 0.1m).Source: §4.2 · p. 11, Table 2 · p. 11

4.3 Point Cloud Retrieval

  • Both DH3D settings outperform the compared methods on Oxford RobotCar at top 1% and top 1 retrieval, and their top-25 recall curves also lead.The point-halved setting shows only a very small performance drop.Source: Table 3 · p. 11, Figure 6 · p. 11, §4.3 · p. 12

4.4 Application to Visual SLAM

  • On Stereo DSO point clouds without fine-tuning, DH3D achieves the best rotation error (1.58°) and success rate (90.6%), and second-best translation error (0.36m).The evaluation uses 318 manually annotated pairs from eight Oxford RobotCar traversals.Source: Table 4 · p. 12, Figure 7 · p. 12, §4.4 · p. 13

4.5 Ablation Study

  • Ablations show that detector confidence-map learning and SE blocks improve matching descriptors, while replacing the proposed supervision causes the largest performance decrease.The proposed supervision synthesizes transformations.Source: §4.5 · p. 13, Table 5 · p. 14
  • Local descriptors retain over 90% registration success for noise below 0.15m and for downsampling factors α ≤ 1.5; global descriptors are less robust to rotation.Local registration performance drops significantly above 0.2m noise, while the global descriptor remains relatively robust to noise.Source: §4.5 · pp. 13–14, Figure 8 · p. 14, Figure 9 · p. 14

5 Conclusion

DH3D unifies local feature detection and description with global descriptor extraction in one forward pass. The authors report state-of-the-art performance across multiple benchmarks and favorable generalization from LiDAR training to visual-SLAM point clouds.

5 Conclusion

  • DH3D performs local feature detection, local feature description, and global descriptor extraction in a single forward pass.This replaces separate detect-then-describe processing and separate local and global descriptor computation.Source: §5 · p. 14
  • The authors report state-of-the-art performance for both local and global descriptors across multiple benchmarks.Source: §5 · p. 14
  • A model trained on LiDAR points generalizes favorably to point clouds generated by visual-SLAM methods.The authors identify further robustness to perturbations as future work.Source: §5 · p. 14

Supplementary Material DH3D: Deep Hierarchical 3D Descriptors for

The supplied supplementary-material passages identify the DH3D paper, its institutional affiliations, and its arXiv version date.

Supplementary Material DH3D: Deep Hierarchical 3D Descriptors for

1 Overview

The supplementary document aims to support reproducibility and provides additional experiments and technical details omitted from the main paper.

1 Overview

  • The authors provide the supplementary document alongside code to address reproducibility concerns.Source: §1 · p. 20
  • Additional results cover keypoint repeatability, qualitative registration examples including failures, global aggregation operators, and qualitative retrieval.The supplementary overview reports attention-based NetVLAD as the best tested aggregation choice.Source: §1 · p. 20
  • The document supplies technical details on network structures, training data, two-phase training, Stereo DSO point-cloud generation, and weak supervision.Source: §1 · p. 20

2 Additional Results on Local Feature

The detector maintains strong repeatability across the tested point-cloud settings, while qualitative registrations show both robustness to varied visual-SLAM conditions and failures on repetitive building facades.

2.1 Keypoint Repeatability

  • On ETH, our detector has the highest repeatability, while USIP degrades significantly; on StereoDSO-Oxford, it outperforms other methods except USIP.Handcrafted detectors and 3DFeatNet have repeatability below 0.2 on StereoDSO-Oxford.Source: §2.1 · p. 21, Figure 1 · p. 21

2.2 Qualitative Results on Oxford RobotCar

  • Two Oxford RobotCar registration examples fail when repetitive building-facade features dominate the local matches.The facades contain repeated structures and limited local textures.Source: §2.2 · p. 21, Figure 2 · p. 22

2.3 Qualitative Results on Stereo DSO Points

  • Without fine-tuning, LiDAR-trained keypoints and descriptors achieve fairly good matchings on Stereo DSO point clouds across varied conditions and spatial distributions.The examples include night, dusk, direct sun, snow, rain, and roadworks; the point clouds vary in density and distribution.Source: §2.3 · p. 21, Figure 3 · p. 23

3 Additional Results on Global Descriptor

Among the tested global aggregation approaches, attention-based NetVLAD performs best. Qualitative retrieval examples also show correct target-location matches under rotation and occlusion.

3.1 Comparison of Different Global Aggregators

  • The comparison tests NetVLAD, PointNet++, DGCNN, PointCNN, FlexConv, Max-Pool, and Avg-Pool as global descriptor assemblers.All tested models produce a 256-dimensional global descriptor.Source: §3.1 · p. 24
  • Attention-based NetVLAD has the best average recall among the tested global aggregators on Oxford RobotCar.PointCNN is the strongest baseline, while Max-Pool performs worst.Source: §3.1 · pp. 24–25, Figure 4 · p. 25

3.2 Quanlitative Results on Point Cloud Retrieval

  • In three retrieval examples, each best match correctly overlaps the target location, and the descriptors are reported as robust to rotation and occlusion.Most unretrieved submaps have relatively high descriptor distances.Source: §3.2 · p. 25, Figure 5 · p. 26

4 Technical Details

The technical details specify the detector and attention modules, data preparation, staged training, Stereo DSO evaluation set, and weak-supervision ablation setup.

4.1 Additional Network Details

  • The keypoint detector uses four 1 × 1 convolutions and sigmoid outputs for local-descriptor matching reliability; the attention predictor uses three 1 × 1 convolutions and softmax weights.The attention distribution sums to 1, and its inputs are learned local descriptors.Source: §4.1 · p. 27, Figure 6 · p. 27, Figure 7 · p. 27

4.2 Training Data Preparation

  • Local and global descriptor learning use different Oxford RobotCar data splits, and local training uses synthetic rotations and Gaussian noise with σnoise = 0.02m.Submaps use a 20m radius, 0.2m voxel downsampling, and 8192 randomly selected points; 20,731 clouds train local features and 21,026 submaps train global descriptors.Source: §4.2 · pp. 28–29, Figure 8 · p. 28

4.3 Two-Phase Training

  • Training proceeds in two phases: local descriptor matching first, followed by global descriptor learning with the local feature encoder frozen.The second phase trains later feature layers, the attention map, and NetVLAD using a lazy quadruplet loss.Source: §4.3 · pp. 29–30

4.4 Testing Set Generation by Stereo DSO

  • The Stereo DSO test set contains 332 point clouds and 318 pairwise poses from eight sequences, with point clouds cropped, downsampled, and randomly rotated.LiDAR scans and ICP provide ground-truth transformations; the set spans day-time and weather conditions.Source: §4.4 · p. 30

4.5 Weak Supervision Used in the Ablation Study

  • The weak-supervision ablation trains local descriptors with a triplet loss using closest positive and negative descriptors for each local cluster.The negative descriptor is the hardest negative, and the margin γ is 0.2.Source: §4.5 · pp. 30–31
Loading 2007.09217v1…